Lease keeper

Spec companion · demo

Every cloud machine now runs under a . No heartbeat, no machine — worst-case zombie cost is one TTL.

TL;DR

7
Agents
running nightly
412 ms
P95 latency
was 1.9 s
2
Open risks
both scoped
Open decision. Lease TTL: 5 min vs 15 min. Owner: Hasan.
Rollout
8/10 regions
Coverage
45/100 paths

Scope

In scope

  • Lease issue, renew, expire

Not in scope

  • Multi-region failover

Launch checklist

  • Shadow mode ran 7 days clean
  • Runbook reviewed by on-call
Lease
A time-limited permit to keep one machine running.
Zombie
A machine still billing after its work ended.

Reclaim runs off a , so a crashed run can't outlive one . Hover or focus a dotted term to read it in the legend panel.

What changes

Join callLease keeperLedger

Engine grabbed machines directly; crashed runs billed for weeks.

Before

Every machine runs under a lease with forced reclaim.

After
  1. Specsigned off
  2. Shadowday 4 of 7
  3. Cutovergated on shadow
Lease world · what runs where
new piece unchanged

Promised vs delivered

Promised (spec)Delivered (debrief)Verdict
Leases expire within one TTLMedian 40 skept
5-minute TTLShipped 15 min — renewal stormschanged
15 min
Trace appendix (not for reading) 
keeper/lease.ts:42

Code exhibit

Click a line to annotate it; shift-click extends the range. The floating button copies every note in agent-ready form.

keeper/lease.tsclick a line to annotate · shift-click extends
async function renewLease(lease: Lease) {
  const deadline = lease.expiresAt - RENEW_MARGIN_MS;
  if (Date.now() < deadline) return;
  await keeper.renew(lease.id, TTL_MS);
  metrics.count("lease.renewed");
}

Structure: stack and shapes

Before/after of who calls whom and what flows through — the abstractions at a glance.

Calls
  • handleRequestserver.ts
    • renewLeasekeeper.tsvalidation, IO, and billing inline
    Shapes — before
    renewLease (id: string) → Promise<boolean>
    Before
    Calls
    • handleRequestserver.ts
      • renewLeasekeeper.ts
        • validateTtlkeeper.ts
          • persistLeasestore.tsall IO now lives here
        Shapes — after
        renewLease (lease: Lease, ttl: Ms) → Renewed | Expired outcome is a value, not a boolean
        persistLease (lease: Lease) → Promise<void>
        After

        Recall what stuck

        The preferred check: flip cards that ask before they tell. Answer in your head, flip, then mark yourself. "Copy missed" emits the cards you missed as agent-ready text.

        Lease keeper recall

        Check yourself

        Lease basics

        What forces a zombie machine off?

        Why did the TTL ship at 15 minutes instead of 5?