ops.lifecycle

ops.lifecycle

Operational lifecycle — the one-command start/stop for the whole mind-palace.

ops/ is the outermost orchestration layer (it may import scheduler + core; nothing imports it back). This package turns the already-built pieces — the supervisor, the durable queue, the vault watcher, the stores — into a single supervised process with:

• a run ledger (runs.py) pinning which git commit each run executed under, and whether it shut down cleanly (the basis for recovery mode);
• preflight (preflight.py) — ensure our own components, verify the external daemons (Vault / Ollama / podman) read-only and fail closed with a clear checklist;
• the launcher (launcher.py) — start (preflight → record run → rebuild-if-empty → supervise) with a graceful shutdown hook (SIGTERM/SIGINT → finish at a job boundary → mark the run clean → optional final snapshot), stop, status, and reset (the surgical fresh-start wipe that guards the production Vault Raft store).

children

Supervised child processes — the thin-master/child runtime model.

palace is the always-on master; components that MUST be separate processes (the network-facing edge monitor — Invariant 2) are spawned, liveness-checked, and gracefully SIGTERM'd by palace, which waits for them to drain before it exits (ASG-style). The core itself stays one process (it is the model-slot arbiter — the ceiling has a single owner); only the genuinely-separate components are children. spawn is injectable so the whole lifecycle is testable without real OS processes.

Spawn = Callable[[list[str]], Proc] module-attribute

Proc

Bases: Protocol

A Popen-like: .pid, .poll() (None = alive), .terminate(), .wait(timeout), .kill(). Structural on purpose — tests inject a bare fake (no subprocess.Popen inheritance) that satisfies this shape, so Spawn stays swappable without real OS processes.

pid instance-attribute

poll()

terminate()

wait(timeout=None)

kill()

Child dataclass

One supervised child process. Idempotent start (won't double-spawn a live child); stop is graceful (SIGTERM → wait → SIGKILL on timeout) and never raises (shutdown must not crash).

name instance-attribute

argv instance-attribute

spawn = _default_spawn class-attribute instance-attribute

stop_timeout_s = 10.0 class-attribute instance-attribute

pid property

start()

alive()

stop()

launcher

The launcher — one supervised process for the whole mind-palace (operational lifecycle).

start → preflight (ensure own, verify externals, fail-closed) → record the run pinned to the git commit → reconcile the corpus (a catch-up vault sync; rebuilds an empty cache) → run the supervisor + watcher with a graceful shutdown hook (SIGTERM/SIGINT → stop claiming new work, let the in-flight job finish at its boundary — the scheduler is already cooperative — then mark the run CLEAN). stop signals the live run's pid. status shows preflight + the last runs. reset is the surgical fresh-start wipe.

Recovery (nervous-system-and-ambassador.md §1): if the previous run never marked itself stopped (crash / kill -9 / power loss), start comes up in recovery mode — scheduler halted, watcher off, read-only — and asks the owner to inspect, then palace stop once the cause is cleared: the recovery run closes CLEAN, so the next start is normal (under KeepAlive the stop IS the restart). --force does not exit a live recovery run (single-instance gate, finding-0186); it only skips recovery at a fresh boot. State itself lives in the stores/files, so a clean restart just resumes; recovery is the cautious response to an unclean exit, not the normal path.

DEFAULT_DRAIN_MAX_TICKS = 64 module-attribute

SupervisorLike

Bases: Protocol

scheduler.supervisor.Supervisor's real surface here — structural so tests inject a bare _FakeSupervisor without subclassing the real Supervisor.

Widened by bp-108 Item 4 from a no-arg run() to the real Supervisor's run(*, max_ticks=...) -> int, because _serve now uses BOTH halves of that signature: it bounds the drain so supervisory ticks reach a job boundary, and it reads the returned dispatch count to decide whether to sleep. A Protocol is only as wide as its actual call sites — the call site grew, so this grew with it.

run(*, max_ticks=None)

WatcherLike

Bases: Protocol

core.ingest.watch.DirectoryWatcher's real surface here (structural, same reasoning). start() narrowed to no-arg (the only call shape here: iterating c.watchers and calling w.start()/w.stop()).

start()

stop()

SweepLike

Bases: Protocol

scheduler.queue.OrphanSweep's surface here — structural, so ops keeps no import-time dependency on scheduler (the QueueLike pattern, bp-101/finding-0177).

render()

QueueLike

Bases: Protocol

scheduler.queue.JobQueue's real surface here (.close() and, since bp-101/bp-103 integration, .sweep_orphans() are called through Components.queue — build_components calls .depth() on its own JobQueue directly).

close()

sweep_orphans(active_run_id)

ChildLike

Bases: Protocol

ops.lifecycle.children.Child's real surface here — structural so tests/integration/test_lifecycle.py's bare _FakeChild satisfies it without subclassing.

name instance-attribute

pid property

start()

alive()

stop()

LaunchDomain dataclass

Which launchd domain a Launcher drives — the dn-plane-principals §3.1/§3.2 axis.

DEFAULT = the per-user GUI LaunchAgent (gui/$UID): today's path, byte-identical — no sudo, plist in ~/Library/LaunchAgents/, control target gui/$UID/<label>. The system-daemon variant runs the palace as the ouroboros core principal under a LaunchDaemon (UserName ouroboros): control targets system/<label>, goes through sudo launchctl, and the plist installs to /Library/LaunchDaemons/. The domain is the ONLY thing that differs between the two — the launchctl runner stays injectable, so tests drive both with a fake and no real launchd domain is touched (the migration itself is owner-run, dn-plane-principals §3.5).

kind = 'gui' class-attribute instance-attribute

needs_sudo property

System-domain control requires sudo launchctl (note §3.2; risk (c)); gui does not.

gui() classmethod

system() classmethod

target(label)

The service target for bootout/print (domain + label): system/<label> or gui/$UID/<label>.

bootstrap_domain()

The DOMAIN argument for bootstrap (no label): system or gui/$UID.

launchctl_argv(args)

The full argv to execute — sudo prepended ONLY for the system domain. The gui form is byte-identical to the historical ["launchctl", *args].

installed_plist()

Where the installed plist lives: /Library/LaunchDaemons/ (system — needs root to write, an owner-run migration step) or ~/Library/LaunchAgents/ (gui). The filename keeps the label (com.mind-palace.palace.plist) either way.

repo_plist(repo_root)

The committed SOURCE plist for this domain: the daemon variant (UserName ouroboros) for system, the LaunchAgent for gui.

Components dataclass

What serve drives. Injectable so tests exercise the lifecycle without models.

supervisor instance-attribute

watchers instance-attribute

queue instance-attribute

enqueue_catchup = lambda: None class-attribute instance-attribute

enqueue_housekeeping = lambda: None class-attribute instance-attribute

health_check = lambda: [] class-attribute instance-attribute

children = field(default_factory=list) class-attribute instance-attribute

snapshot = lambda _run, _flags: None class-attribute instance-attribute

Launcher dataclass

cfg instance-attribute

runs instance-attribute

repo_root instance-attribute

components_factory = build_components class-attribute instance-attribute

preflight_fn = run_preflight class-attribute instance-attribute

tick_seconds = 1.0 class-attribute instance-attribute

drain_max_ticks = DEFAULT_DRAIN_MAX_TICKS class-attribute instance-attribute

housekeeping_interval_s = _HOUSEKEEPING_INTERVAL_S class-attribute instance-attribute

health_interval_s = 60.0 class-attribute instance-attribute

snapshot_interval_s = 5.0 class-attribute instance-attribute

on_shutdown = None class-attribute instance-attribute

gate_cmd = ('uv', 'run', 'pytest', '-q', '-m', 'not live and not podman and not needs_vault and not needs_restic', '--deselect', 'tests/unit/test_core_self_containment.py::test_core_imports_nothing_outside_core') class-attribute instance-attribute

ci_check_cmd = ('uv', 'run', 'scripts/ci_witness.py', 'check') class-attribute instance-attribute

deploy_wait_s = 60.0 class-attribute instance-attribute

deploy_poll_s = 0.5 class-attribute instance-attribute

launchd_label = 'com.mind-palace.palace' class-attribute instance-attribute

domain = field(default_factory=(LaunchDomain.gui)) class-attribute instance-attribute

launchctl = _run_launchctl class-attribute instance-attribute

installed_plist = field(default_factory=_default_installed_agent_plist) class-attribute instance-attribute

stop_verify_s = 5.0 class-attribute instance-attribute

stop_poll_s = 0.25 class-attribute instance-attribute

status_window_minutes = _STATUS_WINDOW_MINUTES class-attribute instance-attribute

start(*, force=False, max_ticks=None)

deploy(*, skip_tests=False)

Apply committed code/infra to the always-on system by a GRACEFUL cycle (owner rule 2026-07-11) — never a kill. Gate, then drain, then verify.

The gate: an active run exists; the working tree is clean; the branch is main; HEAD differs from the live run's commit; the fast ratchet is green (--skip-tests is the emergency hatch). Under launchd (KeepAlive) the graceful stop IS the restart — drain → exit → relaunch on the new code — so deploy just waits for the successor run and verifies its pinned SHA. Infra half: if the repo plist drifted from the installed copy, the cycle is bootout → cp → bootstrap instead, so plist changes deploy the same way code does. (Corollary the owner should know: under KeepAlive, palace stop means RESTART; a true stop is launchctl bootout gui/$UID/com.mind-palace.palace.)

stop()

SIGTERM the live run and report what was verified, not what was requested.

The old line — "it will drain + mark clean" — asserted a future the command cannot see. The drain finishes at the in-flight job's boundary and has no time bound (finding-0171), so a wedged job means the process outlives the signal indefinitely. This now says which of the two happened. NO escalation is added: SIGKILL / job budgets are the owner's open decision (finding-0171 (a)/(b)/(c)); this command only signals, observes, and reports.

ingest_chat()

Build the bp-063 chat sensor and run one idempotent sync(), printing the report.

The scheduled chat_sync job does this in the daemon (startup catch-up + housekeeping); this is the owner's MANUAL trigger — e.g. the very first ingest, before the daemon's first housekeeping tick. Reads local transcripts only (no network, no vault) — safe inside the seal. Idempotent: a session already in the store is skipped.

code_seed()

Enqueue the one-time code SEED onto the running daemon's supervisor queue — every HEAD .py blob embedded once (note §2.7 the deliberate owner-visible run).

Unlike ingest-chat (a lightweight in-process sync()), the code seed is HEAVY, so it must ride the single-writer supervisor queue rather than write the store from this CLI process: we INSERT one code_sync job into the shared on-disk queue (the same queue the daemon drains) and the daemon runs it at BACKGROUND priority under the memory ceiling. sync() is idempotent + blob-sha keyed, so a duplicate seed re-embeds nothing. The queue is durable, so if the daemon is down the job simply waits until it next starts (said, not silent).

code_backfill()

Enqueue the one-time code HISTORY backfill onto the running daemon's supervisor queue — every distinct ledger (path, blob_sha) version embedded (D1) + the first-parent commit diffs captured (D4). Same discipline as code_seed: HEAVY, so it rides the single-writer supervisor queue (a durable job insert, never a store write from this CLI). Idempotent — already-embedded versions re-embed nothing, so a duplicate backfill is safe; the catch-up probe also enqueues one automatically when the store is incomplete. If the daemon is down job waits in the durable queue until palace start.

down()

Maintenance-down that OUTLASTS KeepAlive (finding-0066): launchctl bootout. Plain stop only SIGTERMs and launchd immediately relaunches it — so a true down boots the agent out. Idempotent (already-out reports and returns 0); if the agent isn't installed there is no KeepAlive to outlast, so fall back to a plain stop.

down no longer claims a state it has not observed (finding-0171). Observed 2026-07-25: it printed its success line while pid 96950 kept running at 96% CPU and launchctl print showed active count = 1 pending — the launchd JOB was unloaded, the PROCESS was not. Booting the agent out and the process exiting are two different facts, so this now reports them separately: it verifies the pid for stop_verify_s and, if the process outlives the bootout, says so by pid/run/elapsed and returns non-zero. The escalation policy (SIGTERM→SIGKILL, job budgets) remains the owner's open decision (finding-0171 (a)/(b)/(c)); nothing here kills anything.

up()

Bring the agent back: launchctl bootstrap. Idempotent (already-up reports, returns 0); if the agent isn't installed there is nothing to bootstrap (run palace start).

restart()

A plain down→up cycle. NOT deploy — no HEAD promotion, no test/CI gate; this just cycles the running code as-is (a deploy is the gated ratchet onto HEAD).

Because down is now honest, a down that could not verify the process exited returns non-zero and this refuses to bring the agent back — which is the point: bootstrapping a successor while the predecessor still runs is the double-instance hazard, and the old code would have done exactly that on the strength of a success line it had not earned.

status()

The read-only truth report. Two properties are load-bearing (finding-0172):

(1) Liveness is tested, never assumed. A ledger row marked active whose pid is gone renders DEAD (stale ledger row) and suppresses the running HEAD banner — the exact false green the owner read through a 90-minute incident.

(2) Derivatives, not just levels. _report_snapshot adds rates, budgets, failures.

Read-only in the strict sense: it opens nothing it would have to create (previously it constructed a JobQueue, which CREATES queue.sqlite), enqueues nothing, and is safe to run with the daemon down — which is when it matters most.

reset_targets()

The corpus + its derived/chain layer + the stale queue. Computed from cfg.paths; each is asserted to be under data/ and outside the guard set (never the Vault Raft).

reset(*, confirm)

build_components(cfg)

Wire the full daemon: vault_sync (+watcher), the delegating Ambassador inbox, the delegated-task worker, and the trough dream/curate handlers — all on one supervisor.

build_launcher(config=None, **kw)

lock

The supervisor lock — the supervisor ROLE is kernel-exclusive (dn-supervision-and-liveness §2.6).

An OS-exclusive flock on a lockfile beside the queue, acquired before sweep_orphans and held for the supervisor's lifetime. This is the mechanism that moves "two supervisors, one queue" from tier 5 (remember to check the pid) to tier 3 (a kernel fact) on the note's enforcement ladder, and it is what structurally closes finding-0186's open half: a second claimant fails to acquire whatever entrypoint built it — palace start, start --force, or scripts/watch.py.

Two properties earn the tier, and both were measured on this platform before the module was written (bp-108 Item 1 / V8; commands and output in docs/build-plans/bp-108/journal.md):

• Held-or-not is a kernel fact, so no stale-lock state exists. The kernel drops the lock when the holding process dies, however it dies. Measured: kill -9 on the holder frees it in 17 ms with nothing to clean up. The zero-byte file left behind carries no state — it is a name for the lock, not a record of it. Contrast a pidfile, which is stale state and needs a liveness probe to interpret; that probe is exactly the tier-5 mechanism this replaces.
• Acquisition is atomic — no check-then-act. bp-105's identity gate (launcher.py's _supervisor_alive) probes and then proceeds, a TOCTOU window in which a second start can slip through. flock(LOCK_EX | LOCK_NB) has no window: the kernel either grants or refuses.

Acquire-or-fail, never acquire-or-wait. A supervisor that blocks waiting for the lock is a second supervisor that starts the instant the first dies — precisely the failure the lock exists to deny. Hence LOCK_NB and a raise: no busy-wait, no timeout, no retry loop.

What it does NOT guard: queue writes. The lock covers the supervisor role (sweep + claim), not the queue. CLI enqueues (palace code-seed) legitimately insert concurrently and stay lock-free under WAL; a lock spanning enqueues would break them (§2.6, and bp-108 §9).

Deliberately no pid in the file. Writing the holder's pid would be handy diagnostics and would also reintroduce the thing this mechanism removes: a stored assertion that outlives the actor that made it. The "which run is live, and why" answer stays with bp-105's identity gate — the layer that explains, sitting ahead of the layer that guarantees.

SupervisorLockHeld

Bases: RuntimeError

Another process holds the supervisor role.

SupervisorLock dataclass

An acquire-or-fail exclusive lock on path, held for the process lifetime.

path is cfg.paths.data_dir / "supervisor.lock" — beside the queue it guards, never in the repo (which would scope exclusion to a checkout, so two worktrees over one data dir would both start) and never in /tmp (cleared by the OS, and not co-located with the resource).

path instance-attribute

held property

True while THIS instance holds the lock. Not a probe of the file — a lock held by some other process is not observable here, and deliberately so: the only question this object can answer honestly is whether it is the holder.

acquire()

Take the lock, or raise. Never blocks.

The fd is kept on the instance and never closed except by release() or process death — the kernel drops it either way, so there is no stale-lock state and nothing to clean up after a crash.

Raises:

Type	Description
SupervisorLockHeld	another process holds it.
RuntimeError	this instance already holds it. Re-acquiring is a caller bug (two acquisition sites, or a missing release), and it must not pass silently: flock is per-open-file-description, so a naive re-open would ALSO be refused by the kernel (measured: errno 35 even within one process), and reporting that as "another process holds it" would be a lie about which process is at fault.

release()

Drop the lock. Idempotent — releasing an unheld lock is a no-op, so every exit path can call it unconditionally without first asking whether it got that far.

The lockfile itself is deliberately NOT unlinked. Unlinking races: a successor may already have opened the same path, and removing it out from under them would leave two processes holding flocks on two different inodes with the same name — two supervisors, which is the one outcome this module exists to prevent.

preflight

Preflight — ensure our own components, VERIFY the external daemons (fail-closed).

Owner's chosen scope: the launcher manages the mind-palace's own pieces (data dirs, the queue, the supervisor/watcher loop) and only checks the external daemons it depends on — Vault, Ollama, podman — which have their own lifecycles (LaunchAgents / the Ollama app / a podman VM). A required external being down is a fail-closed refusal with a clear checklist, not an attempt to start it.

ops/ is outside the core import firewall, so a loopback health probe here is fine (Vault and Ollama both listen on 127.0.0.1; this never reaches off-box). The check functions are injectable so tests assert the aggregation/fail-closed logic without any live daemon.

CheckFn = Callable[[object], Check] module-attribute

Check dataclass

name instance-attribute

required instance-attribute

ok instance-attribute

detail instance-attribute

render()

Preflight dataclass

checks instance-attribute

ok property

All REQUIRED checks pass (a failed optional check is a warning, not a blocker).

failures()

render()

check_ollama(cfg)

check_vault(cfg)

check_podman(cfg)

check_own(cfg)

check_constitution(cfg)

Fail-closed integrity check on the fixed point: the live CONSTITUTION.md must match the owner-blessed fingerprint in eval/golden/baseline.json (BUILD-SPEC §15, Invariant 9).

This runs the drift gauge's Constitution-breach comparison AT STARTUP, so a tampered or un-blessed Constitution is caught before any agent is framed with it — the runtime half that was missing: the fingerprint was recorded post-hoc in attestations but never compared in the live loop (a tampered file would otherwise be served to every agent after the next restart).

run_preflight(cfg, *, ollama=check_ollama, vault=check_vault, podman=check_podman, constitution=check_constitution)

Assemble the full preflight. External checks are injectable for tests.

runs

The run ledger — every supervised run pinned to a git commit (operational lifecycle).

The system's state (corpus, vectors, queue, ledgers, secrets) already persists in stores + files, so a normal restart just resumes. What was MISSING is an operational record of the runs themselves: which commit executed, when it started/stopped, and — crucially — whether it shut down cleanly. That last bit is the basis for recovery mode (nervous-system-and-ambassador.md §1): a previous run that never marked itself stopped (killed -9, panic, power loss) means the next start should come up cautious rather than assume consistency.

A small dedicated SQLite (data/runs.sqlite), the same pattern as the other state stores. It is append-mostly: open_run inserts; mark_stopped closes the row. A row with stopped_at IS NULL is either the live run or a crashed one — last_was_clean() distinguishes by recency.

RunRecord dataclass

id instance-attribute

commit_sha instance-attribute

dirty instance-attribute

pid instance-attribute

started_at instance-attribute

stopped_at instance-attribute

clean_shutdown instance-attribute

recovery instance-attribute

note instance-attribute

active property

Still running (or crashed without closing) — stopped_at was never set.

RunLedger dataclass

path instance-attribute

open_run(*, commit_sha, dirty, pid, recovery=False, note='')

mark_stopped(run_id, *, clean, note='')

get(run_id)

last()

The most recently started run (the live one, if any).

last_was_clean()

Did the PRIOR run end cleanly? True if there is no prior run. A prior run still marked active (stopped_at IS NULL) means it crashed — unclean → the caller should recover.

Call this BEFORE open_run (then last() is the prior run) — the launcher does.

recent(n=10)

close()

git_state(repo_root)

(commit_sha, dirty) for repo_root. Best-effort: a non-git checkout reports ('unknown', False) rather than raising — the launcher must still run.

open_run_ledger(config=None)

snapshot

Status snapshot — the core→edge monitoring handoff (Invariant 2) AND the status payload.

The launcher writes a small JSON snapshot of operational METADATA to a file each health tick; the edge monitor process READS only that file to render its dashboard. Same asymmetry as the airlock: the core emits, the networked side never reads a store. The snapshot carries only what the OpsView already narrates — action counts, health, the shape of recent activity, queue depth, memory headroom — plus dream/finding counts. NO note text, NO authored-note titles, NO secrets/tokens.

bp-102 — levels are not enough. status reported LEVELS while every symptom of the 2026-07-25 incident was a RATE or a BUDGET (finding-0172): a queue growing at ~2/min with zero drain, an hour of zero throughput, a job that failed fifteen minutes earlier, and a RUNNING banner over a dead pid. This module therefore also carries:

• run_state — the liveness verdict, PURE, with the liveness primitive INJECTED (there is exactly one _pid_alive, in ops/lifecycle/launcher.py; this module never writes a second one, and taking it as an argument also avoids an import cycle);
• read_queue_stats — windowed throughput / in-rate / out-rate / per-kind oldest age / running elapsed / lease-derived orphanhood / last failure, read from the jobs schema (scheduler/queue.py:62+) over a read-only connection;
• read_store_stats — the METADATA-ONLY store figures.

Cost is a correctness property here (finding-0169 one level up): a diagnostic tool that full-scans data/vectors.lance — or materializes the vector column — is disqualifying even if every number it prints is right. Every read below is an aggregate or a LIMIT-ed row; nothing materializes a payload column, and QueueStats carries rows_read/queries so that bound is asserted mechanically in the tests rather than merely claimed.

STATUS_WINDOW_MINUTES = 20.0 module-attribute

RunningJob dataclass

A job the queue believes is RUNNING, with its elapsed wall clock.

There is deliberately NO budget fraction: no job-level timeout exists anywhere in the system (bp-102 Q4, finding-0174). The 2026-07-25 TimeoutError was the [ollama] request_timeout_s socket timeout on one embed call after 74m50s of elapsed work, not a job budget firing. Printing elapsed / budget would require inventing the denominator.

lease_expired is where "the queue believes it is RUNNING" stops being the end of the story (bp-109 Item 3): a row whose claim deadline has passed is orphaned by definition, and this reader says so with no sweep having run — the sweep stops being a call someone must remember to make (finding-0187's exact failure). Computed at read time from the same now as elapsed_s, because this dataclass is a snapshot, not a live row. It is False whenever the deadline is NULL, which is every row a pre-bp-109 daemon wrote and every kind with no budget configured — so on today's live file every one of these reads exactly as it did before.

id instance-attribute

kind instance-attribute

started_at instance-attribute

elapsed_s instance-attribute

lease_expires_at = None class-attribute instance-attribute

lease_expired = False class-attribute instance-attribute

QueuedKind dataclass

Per-kind backlog: how many are waiting and how long the oldest has waited.

kind instance-attribute

count instance-attribute

oldest_created_at instance-attribute

oldest_age_s instance-attribute

JobFailure dataclass

The most recent FAILED job — the thing status showed six green checkmarks over.

id instance-attribute

kind instance-attribute

error instance-attribute

finished_at instance-attribute

age_s instance-attribute

QueueStats dataclass

Levels AND derivatives over the jobs table, all from bounded aggregates.

rows_read / queries are the cost witnesses: both are independent of table size, which is what the Item-2 falsifier test asserts (a 50-job and a 5,000-job queue must agree).

exists instance-attribute

depth instance-attribute

window_minutes instance-attribute

enqueued_in_window instance-attribute

done_in_window instance-attribute

failed_in_window instance-attribute

lifetime instance-attribute

running instance-attribute

queued_by_kind instance-attribute

last_failure instance-attribute

rows_read instance-attribute

queries instance-attribute

store_idle_s = None class-attribute instance-attribute

in_rate_per_min property

out_rate_per_min property

net_rate_per_min property

d(depth)/dt over W. Positive = the backlog is growing. Both done AND failed leave the queue, so the out term must count both for this to be an honest derivative of depth.

embedding property

Is the RUNNING job demonstrably landing rows? True / False / None = unknown.

The discriminator bp-102 was missing (finding-0188). Every other figure in this class counts job BOUNDARIES, and code_backfill_handler makes one synchronous non-checkpointing call while Supervisor.tick waits (scheduler/supervisor.py:87, no job timeout), so a perfectly healthy multi-hour backfill emits zero terminal transitions — indistinguishable from the wedge, which is exactly what shipped.

The test is threshold-free, which is what makes it trustworthy: was the vector store written after the running job started? If yes, that job has landed rows and is working. If the last write PREDATES the job's own start, the job has landed nothing since it began. No magic constant, no tuned window — the job's own elapsed is the denominator.

Conservative on purpose: with several rows RUNNING it compares against the youngest job's elapsed, so an ambiguous multi-job state reads as NOT progressing. A false alarm costs a second look; a false green is what the incident already cost.

Honest limitation, stated rather than papered over: this senses the embedding lane. A long non-embedding job (dream, curate) legitimately never touches the vector store and so reads as not-progressing — no worse than today, where it also trips both flags.

orphaned_running property

The RUNNING rows whose claim deadline has demonstrably lapsed — the derived view (bp-109 Item 3 / dn-supervision-and-liveness §2.6). No sweep has to have run for this to be populated; orphanhood is read off the row, not off whether anybody remembered to reclaim it.

Empty on every queue file written before the lease column existed, and empty for every kind with no configured budget (which is all of them by default) — so this is strictly NEW information, never a reinterpretation of an old row.

Deliberately NOT folded into wedged / stalled / embedding. Every candidate way of doing that makes an existing flag QUIETER (excluding an orphan from wedged's running test, or from youngest_running_elapsed_s's min, both let a state that flags today go unflagged), and the render that would carry the orphan reason instead (launcher.py:1271-1285, keyed on daemon liveness) belongs to a later plan's write scope. A trade of one loud imprecise flag for one silent precise one is a false green, which is the failure this whole track exists to remove — so the sharpening waits for the render.

youngest_running_elapsed_s property

Elapsed of the most recently STARTED running job — embedding's denominator, exposed so the render prints the same number the predicate decided on rather than recomputing it.

progressing property

Demonstrably making progress — the one state in which an anomaly flag must stay quiet.

stalled property

ZERO DRAIN: work is waiting and nothing completed in the whole window.

Keyed on done, not on done + failed, deliberately. A failure is a terminal transition (it belongs in out_rate, which must stay a true d(depth)/dt term) but it is not progress — and at the exact moment the owner sampled the incident, one job HAD just failed. Counting that as drain would have silenced this flag at 03:45:07, which is precisely when it needed to fire.

Suppressed while progressing (bp-102 §10, discharged by bp-105): a long healthy backfill drains nothing at the job boundary for hours, and an instrument that cries wolf through all of it will be ignored during the next incident.

wedged property

A job is RUNNING and yet nothing has COMPLETED all window — the worker is busy doing the wrong kind of work (99% CPU, 0.3% embedder). Level + derivative, together.

Now genuinely a WEDGE test rather than a long-job test: a running job that is landing rows is working, not wedged (finding-0188).

failure_in_window property

StoreStats dataclass

Metadata-only corpus figures. Exactly ONE, because exactly one is cheap.

Deliberately ABSENT, each for a MEASURED reason rather than an assumed one — bp-102's Item 2 falsifier disqualifies an expensive status even when every number is right:

• distinct code versions embedded and the current=true/false split — VectorStore's only cheap read is count(); all_rows/rows_for_source/relabel_provenance all go through to_arrow().to_pylist(), which materializes the vector column. The metadata-only reader would be count_rows(filter=…) on core/typedshims/lancedb.py plus a method on core/stores/vectorstore.py — both outside bp-102's write scope (core/stores/** is bp-100's).
• the ledger target (COUNT(DISTINCT path, blob_sha) over code_snapshots.sqlite) — measured at 3.5 s: 423,855 rows, SCAN files + USE TEMP B-TREE FOR DISTINCT, over a 2.3 GB table whose rows carry a docstring column. files is keyed (commit_sha, path), so nothing indexes the pair this needs. That is a full scan by any name, and it belongs to the class of read this plan exists to keep out of status.

Both hand-offs are finding-0178. Reporting an expensive figure would fail the falsifier; reporting a fabricated one would be worse; saying nothing and pretending coverage is unknown for a mysterious reason would be worst. The render says which figure is missing and why.

vector_rows instance-attribute

humanize_seconds(seconds)

4490.0 → '1h14m50s'. Compact and exact — an operator reading a status line during an incident needs the magnitude at a glance, and rounding a budget away is how a 74-minute job reads as fine.

run_state(run, *, pid_alive)

(rendered state, is_alive) for one run ledger row.

The defect this closes (finding-0172): status rendered "RUNNING" if r.active straight from the ledger, so a kill -9'd daemon kept reporting RUNNING — the one question an operator trusts status to answer, answered wrongly. deploy already tested _pid_alive(run.pid); the primitive existed and the reporting path simply did not call it.

pid_alive is INJECTED rather than imported: ops.lifecycle.launcher._pid_alive is the single implementation (os.kill(pid, 0)), and injecting it keeps this function pure/testable and snapshot free of an import cycle back into launcher.

Note on the false-alarm falsifier: _pid_alive returns True on PermissionError, so a daemon running as another principal (the ouroboros LaunchDaemon user, dn-plane-principals) is still LIVE here. Only ProcessLookupError reads dead. Residual, documented and unfixed: pid REUSE would read a recycled pid as alive — a false green, no worse than the unconditional RUNNING it replaces.

is_alive is None where liveness is not applicable (no run, or a run already closed out).

read_queue_stats(path, *, now=None, window_minutes=STATUS_WINDOW_MINUTES, max_kinds=6, store_idle_s=None)

Read the rate/budget block off the durable queue — read-only and O(1) in rows returned.

Opened with a file:…?mode=ro URI so this can never create or mutate queue.sqlite: status is the first command anyone runs after an incident and must be safe with the daemon down (it previously CREATED the queue file via JobQueue(...) just to read depth()). A missing file reports exists=False rather than raising.

Why raw SQL rather than JobQueue: JobQueue exposes no windowed or aggregate reads (list() would materialize every one of 300k rows with payloads), and scheduler/queue.py is bp-101's write scope. bp-102 §2.6 pins the jobs schema as the source for exactly these queries. The hand-off — these belong on JobQueue — is finding-0178.

store_idle_seconds(store_dir, *, now=None)

Seconds since anything was last written under the vector store — None if it is absent or unreadable. The prior sample finding-0188 asks for, taken from the filesystem's own clock.

Why a directory walk and not a stored sample. The obvious channel — have the supervisor write a periodic (t, vector_rows) sample that status differences — cannot work here: Supervisor.tick calls handler(job) synchronously with no timeout, so the launcher's serve loop is BLOCKED for the entire duration of the very job being diagnosed. Nothing on that loop can emit while it matters. The filesystem, however, is written by the embed itself, from inside the blocked call. It is the one channel the wedge cannot mute.

Cost is a correctness property (finding-0169, one level up), so this stats DIRECTORIES only, never files: a lance write adds a fragment, a manifest and a transaction record, each of which bumps its parent directory's mtime. Measured against the real 22,621-row store — 8 directories, 0.80 ms, and byte-identical to the full-tree maximum (897 files, 3.9 ms). O(directories), independent of row count, which is the same bound read_queue_stats carries.

Best-effort like every other probe here: status must survive a half-built or corrupt data directory, because an incident is exactly when the store is the thing that is broken.

read_store_stats(*, vector_store=None)

count() on the vector table — LanceDB fragment metadata, measured at 1.4–2.9 ms over the real 22,621-row store, and the vector column is never touched.

Best-effort: a probe failure is reported as None, never raised, because status must survive a half-built or corrupt data directory — an incident is exactly when the store is the thing that is broken.

build_status(*, ops_view, dreams_view, queue_depth, run=None, mem_available_gb=None, flags=(), liveness=None, queue_stats=None, store_stats=None, embedder=None)

Assemble the snapshot dict from the read-only views. Pure — no I/O — so it is unit-testable against in-memory stores. run is the active RunRecord (commit-pinned); flags are the OS watchdog's crossed-threshold flags.

bp-102 extends the payload with liveness / rates / store / embedder. All four are keyword-only with None defaults, so every existing caller (the dormant edge-monitor snapshot, tests/integration/test_monitor_snapshot.py) is unchanged, and every value stays JSON-serializable — write_status still round-trips it. This is the single seam: nothing in the status path may print a datum that did not come through here.

write_status(path, data)

Write the snapshot atomically — the edge reader never sees a partial file (rename swap).