Fail-Safe Sweepers, Flag Precedence, and Auth Identity

Introduction

In a system that controls physical doors, the scary bugs are rarely the loud ones. A crashed request gets retried. A 500 gets paged. The bugs that erode trust are the quiet ones: a background job that cancels a paid reservation at 3 a.m., an authorization rule that silently evaluates to false, a timeline that sorts events into the wrong order because two timestamps spell the same instant differently.

This article walks through a set of engineering patterns we applied over a recent iteration of the UnlockOS SDK. Every pattern here is generalizable: guard your background writers with invariants, make precedence rules deterministic, keep auth state referentially stable, and recover instead of blanking.

---

1. Time is an instant, not a string

A guest timeline showed messages in the wrong order. The root cause was comparing ISO-8601 strings lexicographically:

// ❌ Lexicographic comparison of ISO strings
items.sort((a, b) => (a.createdAt < b.createdAt ? -1 : 1));

This works right up until two producers spell the same instant differently:

String	Instant
2026-08-09T22:51:56+09:00	1786…
2026-08-09T13:51:56.000Z	identical
2026-08-09T13:51:56Z	identical

All three are the same moment. Sorted as text, they scatter. In an audit log or an access-event timeline, "the order things happened" is not cosmetic — it is the evidence trail.

The fix is to normalize to an instant at the boundary, and to fail loudly on unparsable input:

export function toInstant(value: string): number {
  const ms = Date.parse(value);
  if (Number.isNaN(ms)) {
    throw new TypeError(`Invalid timestamp: ${value}`);
  }
  return ms;
}
export function byInstantAsc<T>(
  getTime: (item: T) => string,
  getId: (item: T) => string,
) {
  return (a: T, b: T): number => {
    const delta = toInstant(getTime(a)) - toInstant(getTime(b));
    // Stable tie-break: identical instants must still have a total order
    return delta !== 0 ? delta : getId(a).localeCompare(getId(b));
  };
}

Two details matter beyond the sort itself:

1. A deterministic tie-break. Events written in the same millisecond must not reorder between renders, otherwise UI diffing and snapshot tests become flaky.
2. Branded types at the boundary. If a value has been validated, encode that in the type so it cannot be confused with raw user input:

declare const brand: unique symbol;
export type Instant = number & { readonly [brand]: 'Instant' };
export const parseInstant = (value: string): Instant => toInstant(value) as Instant;

Rule of thumb: strings are for transport, instants are for logic. Never let a serialization format leak into a comparison operator.

---

2. One derivation function for lifecycle state

A stay has a lifecycle: reserved → checked-in → checked-out, with cancellation and no-show as terminal branches. When three surfaces (host app, admin console, billing job) each compute that status from raw columns, they will drift — and then the door panel and the invoice disagree about whether someone is in the room.

The fix is a single pure derivation function, exhaustively typed:

export type StayStatus =
  | 'reserved'
  | 'checked_in'
  | 'checked_out'
  | 'cancelled'
  | 'no_show';
export interface StayFacts {
  readonly cancelledAt: Instant | null;
  readonly checkedInAt: Instant | null;
  readonly checkedOutAt: Instant | null;
  readonly startsAt: Instant;
  readonly endsAt: Instant;
}
export function deriveStayStatus(facts: StayFacts, now: Instant): StayStatus {
  if (facts.cancelledAt !== null) return 'cancelled';
  if (facts.checkedOutAt !== null) return 'checked_out';
  if (facts.checkedInAt !== null) return 'checked_in';
  if (now > facts.endsAt) return 'no_show';
  return 'reserved';
}

The ordering of the guards is the specification: cancellation dominates, physical facts (check-in/out) dominate clock-derived guesses, and "no_show" is only ever inferred, never stored as truth.

Because the function is pure and total, it is cheap to pin down with a table test:

describe('deriveStayStatus', () => {
  const base = { startsAt: t('10:00'), endsAt: t('18:00') } as const;
  it.each([
    ['cancelled beats everything', { ...base, cancelledAt: t('09:00'), checkedInAt: t('10:05') }, 'cancelled'],
    ['checked-out beats checked-in', { ...base, checkedInAt: t('10:05'), checkedOutAt: t('11:00') }, 'checked_out'],
    ['expired without arrival is no_show', { ...base }, 'no_show'],
  ])('%s', (_name, facts, expected) => {
    expect(deriveStayStatus(normalize(facts), t('19:00'))).toBe(expected);
  });
});

A state machine does not have to be a library. It has to be one function, total over its input, with the precedence written down.

---

3. Background sweepers need invariants, not just a WHERE clause

A TTL sweep job releases abandoned reservation holds. It is exactly the kind of job that runs unattended, touches money, and nobody watches — and it cancelled prepaid reservations.

The original predicate matched on status = 'reserved' AND expires_at < now(). That is a description of time, not a description of safety. A prepaid reservation can legitimately sit in reserved past its hold window.

The hardened version makes every safety condition explicit and colocated:

UPDATE reservations
SET status = 'cancelled',
    cancel_reason = 'ttl_sweep',
    cancelled_at = now()
WHERE status = 'reserved'
  AND hold_expires_at < now()
  AND payment_state = 'unpaid'   -- never touch authorized/captured money
  AND checked_in_at IS NULL      -- never cancel someone already inside
  AND created_at < now() - interval '10 minutes'  -- grace for in-flight checkout
RETURNING id, facility_id;

Mirror the same predicate in application code so it can be unit-tested without a database:

export type PaymentState = 'unpaid' | 'authorized' | 'captured' | 'refunded';
export interface SweepCandidate {
  readonly status: StayStatus;
  readonly holdExpiresAt: Instant;
  readonly paymentState: PaymentState;
  readonly checkedInAt: Instant | null;
}
export function isSweepable(c: SweepCandidate, now: Instant): boolean {
  if (c.status !== 'reserved') return false;
  if (c.checkedInAt !== null) return false;
  if (c.paymentState !== 'unpaid') return false;
  return c.holdExpiresAt < now;
}

Then lock the invariants — not the happy path — with tests that read like policy statements:

describe('TTL sweep invariants', () => {
  const states: PaymentState[] = ['authorized', 'captured', 'refunded'];
  it.each(states)('never cancels a %s hold', (paymentState) => {
    expect(isSweepable(candidate({ paymentState }), NOW)).toBe(false);
  });
  it('never cancels a stay that already checked in', () => {
    expect(isSweepable(candidate({ checkedInAt: NOW }), NOW)).toBe(false);
  });
});

A related insight: KPI counters must be independent of lifecycle status. If "billable check-ins" is computed by counting rows whose current status is checked_in, then any later status transition silently rewrites history. Count the event, not the current row state:

// ❌ history mutates when status changes later
const billable = reservations.filter((r) => r.status === 'checked_in').length;
// ✅ an immutable event is counted exactly once, forever
const billable = events.filter((e) => e.type === 'checkin.completed').length;

This is worth a CI guard rather than a code-review convention — a test that fails the build if a counting query references a mutable status column.

Repairing damage: idempotent, dry-run-first backfills

When a bad sweeper has already written events, the repair script is itself a security-sensitive tool. Three non-negotiables:

interface BackfillOptions {
  readonly dryRun: boolean;       // default true
  readonly limit: number;         // bounded blast radius per run
  readonly reasonFilter: string;  // only rows written by the known-bad writer
}
export async function repairSpuriousCancels(opts: BackfillOptions) {
  const rows = await findCancelledBy(opts.reasonFilter, opts.limit);
  const targets = rows.filter((r) => r.paymentState !== 'unpaid' || r.checkedInAt !== null);
  logger.info('backfill.plan', { scanned: rows.length, targets: targets.length, dryRun: opts.dryRun });
  if (opts.dryRun) return { planned: targets.length, applied: 0 };
  const applied = await restoreAll(targets, { auditReason: 'backfill:ttl_sweep_repair' });
  return { planned: targets.length, applied };
}

Dry-run by default, bounded batches, and every write carries an audit reason that identifies which script made the change. If you cannot answer "who wrote this row and why" from the audit log alone, the log is decoration.

---

4. Authorization: evaluation order is the policy

Feature flags in this codebase gate menus, APIs, and operational tooling — which makes the flag evaluator part of the authorization surface. Two bugs there are instructive.

Bug A: a de-duplication pass ran before the admin bypass. The evaluator collapsed duplicate scope overrides first, and the collapse happened to drop the record that granted admin access. Access control was being decided by an incidental step ordering.

Bug B: duplicate scope overrides silently disabled a feature. When two rows targeted the same scope with different values, the reducer picked "last one wins" by array order — which is database-order, i.e. arbitrary. A feature could flip off after an unrelated insert.

The fix is to make the resolution rule explicit and total:

export type Scope = 'global' | 'environment' | 'facility' | 'user';
const PRECEDENCE: Record<Scope, number> = { global: 0, environment: 1, facility: 2, user: 3 };
export interface Override {
  readonly scope: Scope;
  readonly enabled: boolean;
  readonly updatedAt: Instant;
  readonly id: string;
}
export interface Decision {
  readonly enabled: boolean;
  readonly reason: 'admin_bypass' | 'override' | 'default';
  readonly source?: string;
}
export function evaluateFlag(
  overrides: readonly Override[],
  ctx: { isAdminBypass: boolean; defaultEnabled: boolean },
): Decision {
  // 1. Bypass is evaluated FIRST and cannot be shadowed by later passes.
  if (ctx.isAdminBypass) return { enabled: true, reason: 'admin_bypass' };
  if (overrides.length === 0) {
    return { enabled: ctx.defaultEnabled, reason: 'default' };
  }
  // 2. Deterministic winner: narrowest scope, then newest, then stable id.
  const winner = [...overrides].sort((a, b) => {
    const byScope = PRECEDENCE[b.scope] - PRECEDENCE[a.scope];
    if (byScope !== 0) return byScope;
    const byTime = b.updatedAt - a.updatedAt;
    return byTime !== 0 ? byTime : a.id.localeCompare(b.id);
  })[0];
  return { enabled: winner.enabled, reason: 'override', source: winner.id };
}

Three properties worth copying:

• Order is declared, not emergent. PRECEDENCE is a table you can review; array order is not.
• Every decision returns a reason. { enabled: false, reason: 'override', source: 'ovr_123' } turns a support ticket into a one-query investigation. A bare boolean is unauditable.
• Conflicts are observable. When two overrides share a scope, emit a warning with both ids rather than silently picking one.

const sameScope = overrides.filter((o) => o.scope === winner.scope);
if (sameScope.length > 1) {
  logger.warn('flag.conflicting_overrides', {
    scope: winner.scope,
    ids: sameScope.map((o) => o.id),
    chosen: winner.id,
  });
}

---

5. Auth state: preserve object identity when nothing changed

Auth SDKs emit events liberally — token refresh, tab focus, storage sync. Many of those events carry a session that is value-equal to the one you already hold. If your store replaces the object anyway, every subscriber re-renders, effects keyed on session re-run, and you get refetch storms, duplicated authorization calls, and occasionally a reconnect loop against a device gateway.

The fix is a reducer that returns the previous reference when nothing meaningful changed:

export interface Session {
  readonly userId: string;
  readonly accessToken: string;
  readonly expiresAt: Instant;
}
function isSameSession(a: Session | null, b: Session | null): boolean {
  if (a === b) return true;
  if (a === null || b === null) return false;
  return (
    a.userId === b.userId &&
    a.accessToken === b.accessToken &&
    a.expiresAt === b.expiresAt
  );
}
export function sessionReducer(prev: Session | null, next: Session | null): Session | null {
  // Returning `prev` keeps referential identity stable for subscribers.
  return isSameSession(prev, next) ? prev : next;
}

The same class of bug appeared in a device-connection path: a refresh race produced a spurious "reconnected" transition because a no-op state update was treated as a real change. A state container should represent transitions, not notifications. If the value did not change, no transition occurred.

---

6. Recover, don't blank

A continuously deployed SPA has an unavoidable failure mode: a client holds an old HTML shell that references route chunks which no longer exist. The dynamic import rejects, an error boundary catches nothing useful, and the operator sees a white screen — while standing at a door.

White screen is the worst possible outcome. Detect the specific failure and self-heal, with a loop guard:

const RELOAD_KEY = 'app:chunk-reload-at';
const RELOAD_WINDOW_MS = 30_000;
function isStaleChunkError(error: unknown): boolean {
  const message = error instanceof Error ? error.message : String(error);
  return /Loading (CSS )?chunk .* failed|Failed to fetch dynamically imported module|Importing a module script failed/i.test(message);
}
export function recoverFromStaleChunk(error: unknown): boolean {
  if (!isStaleChunkError(error)) return false;
  const last = Number(sessionStorage.getItem(RELOAD_KEY) ?? 0);
  if (Date.now() - last < RELOAD_WINDOW_MS) {
    // Already tried recently — a reload loop is worse than an error screen.
    return false;
  }
  sessionStorage.setItem(RELOAD_KEY, String(Date.now()));
  window.location.reload();
  return true;
}

The same principle showed up in payments. When an upstream customer record had been deleted out-of-band, the SDK surfaced a raw 502. The correct behaviour is to classify the upstream error, re-provision the missing resource, and return a localized, actionable message:

export type SetupFailure =
  | { kind: 'stale_customer'; customerId: string }
  | { kind: 'card_declined'; declineCode: string }
  | { kind: 'upstream_unavailable'; retryable: true };
export async function ensurePaymentSetup(userId: string): Promise<Result<SetupIntent, SetupFailure>> {
  const customerId = await getStoredCustomerId(userId);
  const customer = customerId ? await fetchCustomer(customerId) : null;
  if (customerId && (customer === null || customer.deleted)) {
    logger.warn('payment.stale_customer', { userId, customerId });
    const fresh = await createCustomer(userId);
    await replaceStoredCustomerId(userId, fresh.id);
    return ok(await createSetupIntent(fresh.id));
  }
  return ok(await createSetupIntent(customer?.id ?? (await createCustomer(userId)).id));
}

And for asynchronous integrations, a dead-letter queue is only half a design. The other half is a recovery worker that can replay "fossil" entries idempotently and reconcile any counters the failed attempts corrupted:

export async function recoverDeadLetters(batch: DeadLetter[]): Promise<RecoveryReport> {
  const report: RecoveryReport = { replayed: 0, dropped: 0, failed: 0 };
  for (const item of batch) {
    if (await isAlreadyApplied(item.idempotencyKey)) {
      await dropDeadLetter(item.id, 'already_applied');
      report.dropped += 1;
      continue;
    }
    try {
      await applyWithIdempotency(item.payload, item.idempotencyKey);
      await reconcileCounters(item.aggregateId);
      report.replayed += 1;
    } catch (error) {
      logger.error('dlq.replay_failed', { id: item.id, error });
      report.failed += 1;
    }
  }
  return report;
}

Every replay path must be idempotent-by-key, must reconcile derived counters, and must report what it did. Anything else turns a recovery tool into a second incident.

---

7. Make dangerous edits impossible, not discouraged

Applied database migrations are append-only by contract: editing one that has already run in production means environments silently diverge. Code review catches this most of the time; "most of the time" is not an access-control model.

A pre-write tool hook turns the convention into an enforced guard:

#!/usr/bin/env bash
set -euo pipefail
file="$1"
case "$file" in
  */migrations/*.sql) ;;
  *) exit 0 ;;
esac
version="$(basename "$file" | cut -d_ -f1)"
if grep -qx "$version" .migrations-applied; then
  echo "BLOCKED: migration $version is already applied. Add a new migration instead." >&2
  exit 1
fi
exit 0

The same philosophy applies to API surface: maintain an explicit registry of which edge functions are publicly reachable, and default everything else to internal. An endpoint that is public because nobody listed it as private is a vulnerability waiting for a scanner.

{
  "public": ["guest-inbox-timeline", "checkin-host-session"],
  "internal": ["auto-checkout-sweep", "gcal-dlq-recover", "billing-reconcile"],
  "policy": "deny-by-default: functions absent from this registry fail CI"
}

---

8. Typed failures beat generic errors

When a booking is rejected, "conflict" is not enough information for the caller to do the right thing. A hard overlap with an existing stay is a real rejection; a collision that only touches the cleaning buffer may be resolvable by shifting fifteen minutes. Model that distinction in the type system:

export type BookingConflict =
  | { kind: 'overlap'; conflictingId: string }
  | { kind: 'buffer_only'; conflictingId: string; bufferMinutes: number; suggestedStart: Instant }
  | { kind: 'quota_exceeded'; limit: number; used: number };
export function describeConflict(conflict: BookingConflict): string {
  switch (conflict.kind) {
    case 'overlap':
      return 'errors.booking.overlap';
    case 'buffer_only':
      return 'errors.booking.buffer_only';
    case 'quota_exceeded':
      return 'errors.booking.quota_exceeded';
    default: {
      const never: never = conflict;
      throw new Error(`Unhandled conflict: ${JSON.stringify(never)}`);
    }
  }
}

The never branch is the point: when a new conflict kind is added, the compiler finds every place that must handle it. And note that the function returns a message key, not a hardcoded string — hardcoded operator-facing text is a correctness problem, because the message a support agent reads must match the one the guest saw.

Validation belongs at the same boundary. Free-text intake answers that become identity records should be parsed, not trusted:

const IntakeAnswer = z.object({
  email: z.string().trim().toLowerCase().email(),
  fields: z.record(z.string().min(1)),
});
export function parseIntake(input: unknown) {
  const result = IntakeAnswer.safeParse(input);
  if (!result.success) {
    return err({ kind: 'invalid_intake', issues: result.error.issues });
  }
  return ok(result.data);
}

---

Summary

The through-line of this iteration was not a single feature. It was a set of habits:

1. Normalize at the boundary. Compare instants, not strings; parse inputs, don't trust them.
2. Derive lifecycle state in one total function, with precedence written down and pinned by table tests.
3. Guard background writers with safety predicates, mirrored in SQL and in testable code, and lock them with invariant tests that read like policy.
4. Count immutable events, not mutable status columns — history should not change retroactively.
5. Make authorization order explicit and every decision explainable (reason, source), and log conflicts rather than resolving them arbitrarily.
6. Preserve object identity when nothing changed; a state container should represent transitions, not notifications.
7. Recover instead of blanking, with loop guards, idempotent replay, and counter reconciliation.
8. Enforce dangerous-edit rules mechanically — immutable migrations, deny-by-default API registries — instead of relying on review discipline.

None of these are glamorous. Together they are the difference between a system that usually works and a system you are willing to put on a door.