core.ingest
core.ingest
Outer-ring residue of ingest (dn-inner-outer-core §2.7, K1 / bp-090).
The pure text-projection machinery (amend, chunk, logseq, pipeline, verify + the package's inner
init text) moved to core/kernel/ingest/. What remains here is the outer half — the modules whose
closure leaves the admissible base: embed (the embedding path via the model client), watch
(watchdog), and the curated/dialogue/founding/index/mint_ids/purge/run/sync runners.
This init is stdlib-import-free so it stays inner by construction (a pure package marker); the
residue submodules beside it are the outer machinery.
code_corpus
The code embed lane (dn-code-ingest-pipeline §2.1/§2.1b/§2.2/§2.3/§2.7; bp-092/CI-1).
Ouroboros's largest artifact — its own code, carrying the math and the §-warrants — was the one
region outside the semantic self-map (finding-0146). This lane pulls it in under the SAME vector
store, embedder, and group-by-digest machinery the notes use, discriminated by a layer coordinate:
- L0a — the structural (AST) reading (
layer=code_ast): one chunk per symbol, sliced at AST boundaries, header-prefixed# {path}:{qualname}{signature}. Nested defs own their lines, so a parent embeds as its SHELL (own lines minus descendants) and the module shell covers preamble + inter-symbol + trailing — every source line in exactly one L0a chunk (F-CI2 byte-cover). - L0b — the windowed textual reading (
layer=code_text): the note chunker's sliding char-window (chunk_text, the ONE window machinery — NOTderive_chunks, whose Logseq property-strip must not run on code) over the RAW source; bodies and#comments flow together. - L1 — the prose reading (
layer=codedoc): module + symbol docstrings + inline comments in source order, windowed as CANONICAL (header-free) prose and prefixed for retrieval by a single# {path}line — it lives in the note neighbourhood.
Derivation is a PURE function of (path, source): re-running yields bit-identical chunks (F-CI2 re-derivability). All embedding is LOCAL (the core embedder) — zero network egress (non-negotiable #1).
[banner: correction] The three projections WERE joined by line-range coordinates carried ON the
vector rows, and digest (the git blob sha) made group-by-digest yield "file = source object,
chunks = members". Neither is true of a code row any more (dn-vector-membership-store D1,
bp-152). The vector plane holds ONE row per distinct idea-atom (layer, content_hash),
corpus-wide and append-only; ALL occupancy — which (path, blob_sha) version holds which atom, at
which slot and lines, and whether that occupancy is current — moved into the membership relation
(core/stores/memberships.py). A version is a FIBER M(path, blob_sha), the source object is
that fiber, and a code consumer resolves a hit through the membership join (D3), never through
group-by-digest. The measured payoff is the reason: 52,755 duplicated embeds over the full ledger
history become 22,502 atoms (2.34×, D7), and a revert or a git mv costs zero geometry.
[banner: correction] A chunk's IDENTITY (content_hash) hashed its full embed text — coordinate
header included — so a git mv re-hashed every chunk of the file and every (path, slot) lineage
chain through it severed, on an operation this repo performs constantly (dn-vector-membership-store
§0.1 F1/F2). CORRECTED per D0 (owner-ruled 2026-07-27, strip-headers-from-the-atom-hash):
identity is the header-free CANONICAL body; the embed text still carries its header (retrieval
context is untouched, R7). L1 additionally cuts its windows over the canonical prose — stripping at
hash time alone leaves window boundaries computed over header-bearing text, so a path-length change
still recuts them. Measured over all 580 tracked .py files at 45c4a15, renaming every one of them:
11,096 atoms minted under the old identity → 2,373 under strip-at-hash-only → 3 under
this pin (those 3 are the parked oversize-threshold residue, issue #31); a one-line top-of-file
insert in every file: 2,114 → 2,114 → 0. Consequence, named: identity now differs from embed
text, so a shared atom's stored text/vector is its FIRST-LANDED rendering and display coordinates
resolve from memberships, never from the stored text.
[banner: supersession] The incremental sync's delete+replace contract (old §2.7) is REVERSED to
keep-and-link per dn-temporal-code-corpus D2 (warrant finding-0163, bp-099): a superseded code
version is RETAINED with current=false, never deleted, and backfill() embeds the full ledger
history (D1) — so every code version is a semantic node and the causal graph's supersession edge
blob(v)→blob(v+1) has both endpoints resolvable. Default retrieval stays current-view (D3).
CodeChunk
dataclass
One embeddable code chunk with its fiber coordinates. layer discriminates the projection;
(qualname, slot_line_start, slot_line_end) are the §2.4 backpointers, and they travel to the
MEMBERSHIP row now (bp-152 D1), not to the vector row.
TWO renderings, deliberately different (D0): text is the EMBED rendering and KEEPS its
coordinate header (retrieval context, R7); canonical_body is the IDENTITY input and is
header-free. Every chunker passes the canonical body from the site that already holds it — it
is NEVER re-derived by re-parsing text, so a body line that legitimately begins with # can
never be mistaken for a coordinate header (a wrong strip is silent identity corruption).
[banner: correction] line_start / line_end are RENAMED to slot_line_start /
slot_line_end (dn-vector-membership-store Amendment A2, owner-ruled 2026-08-06 on issue #34).
The stored values do not change; the name does, because the old name licensed a wrong reading.
They are the SLOT's declared extent — where the symbol lives — never the atom's text
coverage. L0a partitions by INNERMOST OWNER, so a class's chunk holds the class statement,
its docstring and its attributes but NOT its methods (which became their own chunks) — while
the emitted coordinates are owner.lineno, owner.end_lineno, the owner's full declared span.
They coincide exactly for leaf symbols, which is why every leaf-symbol fixture is blind to the
divergence; the module shell is the maximal case, carrying 1..n (the ENTIRE file) for a few
lines of preamble. The values are also non-contiguous in general: a symbol with nested children
owns lines scattered across its span. A consumer that wants "where is this symbol" reads the
span; a consumer that wants the atom's content reads text. This is the intended behavior, not
a defect to fix — the fix was to stop letting the field name hide the difference.
layer
instance-attribute
qualname
instance-attribute
slot_line_start
instance-attribute
slot_line_end
instance-attribute
text
instance-attribute
canonical_body
instance-attribute
content_hash
property
[banner: correction] Identity = the CANONICAL (header-free) body, never the embed text
(D0, owner-ruled 2026-07-27; this hashed self.text before). A filename is mutable: with
the coordinate header inside the hash, a rename re-hashes every chunk and every
(path, slot) occupancy chain through the file severs — on an operation this repo performs
constantly. Embed text may keep headers; identity may not.
LandReport
dataclass
What one land() actually did. atoms_embedded == 0 on a re-land is the reuse claim; the
CURRENCY numbers are what prove idempotence, because a do-nothing lander also embeds zero.
atoms_embedded = 0
class-attribute
instance-attribute
atoms_reused = 0
class-attribute
instance-attribute
membership_rows = 0
class-attribute
instance-attribute
currency = field(default_factory=CurrencyReport)
class-attribute
instance-attribute
current_any_raised = 0
class-attribute
instance-attribute
current_any_lowered = 0
class-attribute
instance-attribute
CodeLander
dataclass
land(path, blob_sha, chunks) — the D2 write path, in the D8 order.
Vector inserts FIRST (append-only; an unreferenced atom is dormant geometry, harmless), the
membership fiber SECOND (one SQLite transaction — the reference truth), currency reconciliation
and current_any maintenance LAST (both re-derivable, so a crash anywhere is repaired by the
next land or by repair_current_any).
⚑ Re-landing is idempotent BECAUSE reconciliation converges, not because the call
short-circuits. Step 4 runs even when step 3 wrote nothing. The tempting "the fiber already
exists, so return" is the C1 bug in its exact original form: on A → B → A the fiber for blob A
already exists carrying current=false, so a short-circuit leaves B marked HEAD — silent
corruption of every default (current-view) read, with nothing raised and nothing logged. The
repo already learned this once at note grain (core/stores/versions.py:22-27).
vectors
instance-attribute
memberships
instance-attribute
embedder
instance-attribute
embedder_identity
instance-attribute
land(path, blob_sha, chunks, *, head_blob_sha=None)
Land one file version. head_blob_sha names the path's CURRENT HEAD blob — it defaults
to the version being landed (the incremental case) and is passed explicitly by a history
backfill, where the version being landed is usually NOT head.
reconcile(path, head_blob_sha)
Steps 4–5 alone, for a path whose HEAD fiber already stands.
This exists so the incremental sync can honor the C1 rule without re-deriving chunks for every unchanged file on every pass. "Unchanged blob ⇒ skip the path entirely" is the same short-circuit at one level up: after A → B → A the HEAD fiber exists, the file looks unchanged, and B is left current forever. Reconciliation is two counting queries per path, so convergence costs nothing worth trading for that.
supersede_path(path)
A vanished file: every fiber of path goes current=false, nothing is deleted
(keep-and-link, D2). Expressed as reconciliation against a blob no fiber has, so there is
ONE currency mechanism rather than a second, subtly different one.
CodeSyncReport
dataclass
embedded_rows = 0
class-attribute
instance-attribute
changed_files = 0
class-attribute
instance-attribute
unchanged_files = 0
class-attribute
instance-attribute
deleted_files = 0
class-attribute
instance-attribute
superseded_rows = 0
class-attribute
instance-attribute
parse_failures = 0
class-attribute
instance-attribute
membership_rows = 0
class-attribute
instance-attribute
CodeCorpusSync
dataclass
Blob-sha-keyed sync of the tracked .py corpus into the vector store. The store's own set of
CODE (source_path, digest) pairs IS the D-fiber state: a file whose blob is already embedded
costs ZERO embeds. On a changed blob the incremental sync() is now keep-and-link
(dn-temporal-code-corpus D2, bp-099 — reverses the §2.7 delete contract): the superseded version
is RETAINED with current=false (never deleted) and the new version lands current=true; a
vanished file's rows likewise flip current=false rather than being removed. backfill()
embeds every HISTORICAL ledger version (D1) so the whole code history is a set of nodes. The
one-time SEED is sync() against an empty store. The embedder runs locally (no network, #1).
repo
instance-attribute
store
instance-attribute
embedder
instance-attribute
memberships
instance-attribute
embedder_identity
instance-attribute
max_chars = _DEFAULT_MAX_CHARS
class-attribute
instance-attribute
overlap_chars = _DEFAULT_OVERLAP_CHARS
class-attribute
instance-attribute
lander
property
sync()
[banner: correction] The D-fiber state WAS the store's own set of CODE
(source_path, digest) pairs. Atom rows carry neither column (D1), so the state re-homes
to the membership store's (path, blob_sha) fibers — the same number, a sturdier home
(the note's §6 re-home (1), applied here; the daemon's incompleteness probe is bp-153's).
A path whose HEAD fiber already stands is UNCHANGED and re-derives nothing — but it is still reconciled. Skipping it outright is the C1 short-circuit one level up: after A → B → A the HEAD fiber exists, the file reads as unchanged, and B stays current.
seed()
The one-time seed run — sync() on a store with no code rows embeds every HEAD blob
(§2.7-2). Scheduler-gated at the call site (BACKGROUND, pinned tier); the memory ceiling
(#8) is enforced by the loader on each embed call, exactly as for vault_sync.
backfill(versions)
Embed the full code HISTORY (dn-temporal-code-corpus D1, bp-099): every distinct ledger
(path, blob_sha) version in versions (from ops.code_lineage.ledger_versions) becomes a
semantic node. Idempotent by construction — a (path, digest) already in the store is
skipped at zero embeds (digest = blob sha, content-addressed) — so a re-run embeds nothing
and re-running after the seed only adds the non-HEAD versions. Each landed version is
current = (blob is that path's HEAD blob), so backfilling into an un-seeded store also
marks HEAD correctly and every superseded version current=false. A parse-fail blob still
embeds (L0b windows + module shell, derive_code_chunks degrades — never a hard stop) and
is counted. Store writes stay on the caller (the supervisor handler), single-writer kept.
[banner: correction] "Already in the store" is now "already has a FIBER" (D1 — the atom row
carries no (source_path, digest) to test). Idempotence is unchanged in kind and stronger
in fact: a re-run derives nothing for a version whose fiber stands, and a version whose
atoms are all already in the plane costs zero embeds even the FIRST time it is landed —
which is the whole point of the split (D7's 52,755 → 22,502 measured).
derive_code_chunks(path, source, *, max_chars=_DEFAULT_MAX_CHARS, overlap_chars=_DEFAULT_OVERLAP_CHARS)
The PURE derivation: (path, source) -> the file's L0a + L0b + L1 chunks. Deterministic and
bit-identically re-derivable from the blob (F-CI2) — parses ONCE with φ_code's parse_source
(the same interpreter, not a second parser). A parse-error file still yields L0b windows and a
module-shell L0a chunk (it embeds as text even when unparseable).
atom_id(chunk)
The atom's identity (D1): "{layer}:{content_hash}" — PATH-FREE and corpus-wide.
The one place the id shape is spelled, so the vector row and the membership row can never
disagree about what an atom is. Dropping the path from the id is the whole atom model in one
character-level change: it promotes code_rows' old per-path dedup to corpus-wide dedup, so
the same body in two files is ONE point with two memberships (PD-1, owner-ruled in). The layer
stays inside identity because two layers with identical text are different readings, not the
same idea (the D1 stratum fence, carried as a test invariant).
code_rows(chunks, vectors, *, current=False)
Assemble ATOM rows — one per distinct (layer, content_hash) (D1). Provenance is HARDCODED
CODE: there is NO parameter, so a caller physically cannot launder code into an authored class
(F-CI1).
[banner: correction] The old docstring said id is (source_path, layer, chunk_hash) —
"doc+layer-scoped" — and that digest is the git blob sha "so group-by-digest yields file =
source object, its chunks = members". Both clauses stop being true here. Under D1 the id is
(layer, content_hash), corpus-wide; the source object is now the MEMBERSHIP FIBER
M(path, blob_sha) (code_memberships, below), and group-by-digest is not the code lane's
path at all — VectorStore.all_rows structurally keeps shed atom rows out of it, because a
grouping keyed on a column these rows do not have produces one bogus set rather than an error
(bp-152 Item 3; the note's §3 Q5).
The shed, stated exactly. The occupancy columns — source_path, digest, title,
chunk_index, qualname, line_* — leave the ROW, not the schema: note rows still carry
them and no prose-lane consumer changes. (title is shed with them because on a code row it
WAS the path under another name; leaving it would stamp each shared atom with its first-landed
path — a coordinate that lies for every other occupancy, which is exactly what D0's consequence
note forbids relying on. Occupancy resolves from memberships, never from the row.)
provenance STAYS, and that is load-bearing rather than incidental: the mirror firewall is
a row PREFILTER — provenance IN (...) with prefilter=True in VectorStore.search — so
shedding the column would not weaken the firewall, it would REMOVE it, silently and with no
failing call anywhere.
The dedup here is the atom side and ONLY the atom side. by_id.setdefault collapses
duplicates, which is correct for geometry — two identical bodies are one idea — and WRONG for
occupancy: two byte-identical L0b windows in one blob are TWO memberships with distinct
chunk_index (the F5 multiset pin). code_memberships therefore builds its own list and never
reuses this dict.
current is the current_any reading now (D1): does ANY current membership contain this atom?
A freshly landed atom defaults to False — it has no occupancy yet at insert time, since D8
puts the vector insert BEFORE the fiber write — and the lander raises it in step 5 for exactly
the atoms whose current-membership count crossed 0→1.
code_memberships(path, blob_sha, chunks)
The version's FIBER: one membership row per chunk, in derivation order (D1/D2 step 3).
This is the A2 translation point — CodeChunk.slot_line_* becomes Membership.slot_line_*
with the name intact, so the "declared extent, not text coverage" reading survives the trip to
storage instead of being re-lost at the boundary.
One row per CHUNK, never per distinct atom. chunk_index is the position in
derive_code_chunks' output, which is a pure deterministic function of (path, source)
(F-CI2), so the key (path, blob_sha, layer, chunk_index) is stable and re-derivable — and two
identical windows in one blob keep both occupancies instead of colliding. Rows land
current=False; currency is not a property of the fiber's construction but of reconciliation
against the path's HEAD (D2 step 4), which is the only place that decides it.
build_code_corpus_sync(config=None, *, repo=None, embedder=None)
Wire a CodeCorpusSync against the configured vector store, membership store, local embedder and repo root. The membership store and the embedder identity are REQUIRED fields rather than optional ones: a lander without an occupancy record is not a lander, and a reuse decision without an embedder identity is the geometry-mixing bug (D2 step 2's pin) waiting to happen.
curated
Ingest the system's own white papers + design notes as a curated self-knowledge graph
(Track B / B4; nervous-system-and-ambassador.md §4).
The Ambassador can explain its own architecture by reading this graph — "fittingly, the white
papers + design notes ARE the corpus" for self-narration. It is CURATED, not authored: it is
the system's design prose, kept in its OWN graph and never merged into the authored mirror
(curated ∉ MIRROR_READABLE — the same firewall as book dreaming). The Ambassador reaches it
only via a deliberate, non-default provenances={CURATED} query.
Reuses the parametrized ingest pipeline (provenance=CURATED) into the existing multi-provenance VectorStore — no new store. Only PROSE docs are ingested (Constitution, Conventions, the docs/ tree). Config and anything that could hold a secret are never sourced here (there are none in the .md tree; the note's own caveat: explain the design, never expose live keys).
CuratedReport
dataclass
ingested = 0
class-attribute
instance-attribute
chunks = 0
class-attribute
instance-attribute
curated_sources(repo_root)
The self-knowledge corpus: the Constitution, the Conventions, and the whole docs/ tree (white papers + design notes). Prose only — never config or secrets.
ingest_curated(paths, raw, store, embedder, catalog, *, repo_root, attestor=None)
Ingest each doc as CURATED. Titles are repo-relative (e.g. docs/design-notes/...).
Idempotent: delete-then-index per digest, so re-running after a doc edit re-embeds cleanly.
build_and_ingest_curated(config=None)
Wire + run the curated ingest against the configured stores + embedder (needs the live embedder — owner-run, see scripts/ingest_self_knowledge.py).
dialogue
Capture owner↔Ambassador dialogue into the corpus as authored-dialogue (Track B).
Chatting with the Ambassador is itself a form of feeding the system — "your words to it are
more yours than its words to you" (nervous-system-and-ambassador.md §4). So the owner's
messages are captured as a distinct provenance, AUTHORED_DIALOGUE, which IS mirror-readable
(it is the owner's own writing) — closing the capture loop.
It rides the SAME deterministic path as vault ingest — ingest_note (now provenance-parametric)
→ index_records → VaultCatalog.record — never a bespoke writer (the §1 split is exactly what
makes this one-line provenance swap possible). Raw is sacred: the message bytes are stored
content-addressed; identical text dedups. The attestor (optional) stamps a capture action so
the dialogue leaf is part of the same attestation chain as authored notes.
DialogueCapture
dataclass
raw
instance-attribute
store
instance-attribute
embedder
instance-attribute
catalog
instance-attribute
attestor = None
class-attribute
instance-attribute
capture(text, *, conversation='default')
Store one owner message as authored-dialogue; return its content digest.
Idempotent on content: identical text is one raw object + one set of vectors (delete- then-index, the store's own re-index idiom); each call still records a distinct catalog entry keyed by timestamp so the conversation's turns are individually tracked.
build_dialogue_capture(config=None)
Wire a DialogueCapture against the configured stores + embedder.
embed
Embedding adapter (BUILD-SPEC §8 derived layer).
Wraps the local embedding model. Documents are embedded plain; queries are wrapped in the model's instruction format (Qwen3-Embedding is instruction-aware on the query side, which materially improves retrieval). Embeddings are a regenerable derived representation — re-embed from the raw store if the model changes (§8).
Embedder
dataclass
client
instance-attribute
config
instance-attribute
dim
property
embed_documents(texts)
embed_query(text)
build_embedder(config=None)
founding
Founding-corpus ingest — authoring the initial condition (founding-corpus.md; build plan Item 3).
The founding corpus is NOT model training and NOT steady-state ingest (founding-corpus.md §1–§2): it is a hand-selected batch of the owner's musings, authored across a long past, injected at once as the graph's initial condition. It must be a dated, supersession-linked sequence (§2.1 — reconstruct the partial order of thought, not a bag stamped "today"), and it MUST share the steady-state ingest path or the provenance model fractures at the origin (§4 / Q3).
So this driver is a thin batch over the ONE pipeline — ingest_note (provenance AUTHORED_SOLO) →
index_records → VaultCatalog.record, exactly the curated.py shape, never a bespoke writer. Two
founding-specific disciplines it enforces:
- Dated, not a bag (§2.1). Every item carries a reconstructed date, recorded as a
date::property IN the note, so the raw content-addressed blob carries it (permanent provenance) andparse_textlifts it intoproperties['date']. An undated item is refused — the timestamp lie (collapsing years into simultaneous peers) is exactly what founding must avoid. (The temporal layer reading these dates is dormant today; the dates are recorded now regardless.) - Supersession-linked (§2.1). When a later musing revises an earlier one, it is recorded as an
OWNER-DECLARED authored-historical supersession (
core/stores/authored_supersession.py; 8f / PD11) — a K₀↔K₀ RELATION between two authored documents, so the active projection shows the current musing and the earlier lives on in history. It is NOT a claim-supersede(no warrant, no derived alternative) and NOT a note-versionsupersedes(two documents, not one doc's versions). Founding is an owner entry point, so it mints anOwnerDeclaration; the store is owner-declared only and refuses any machine caller at its boundary (the-edge-model.md §4a).
Provenance is AUTHORED_SOLO — the owner's writing, the mirror's ground truth. The founding corpus is deliberately biased-coherent, so it CANNOT be the Track-L control corpus (§2.3): the control is a separate, non-curated CURATED-graph act. This driver writes only AUTHORED_SOLO and never the control — the two acts stay mechanically distinct. Ingest, not fine-tuning; the weights never move.
UndatedFoundingItem
Bases: ValueError
A founding item with no reconstructed date — refused (§2.1: a dated sequence, not a bag stamped 'today'; the timestamp lie is exactly what founding must avoid).
ForwardSupersession
Bases: ValueError
A founding item supersedes one not yet ingested — refused: the sequence is ordered (a musing can only revise an EARLIER one), so a forward reference is a manifest error.
FoundingItem
dataclass
One dated musing in the founding sequence. body is the text; date its reconstructed
original date (a date:: property in the note); supersedes the source_path of an
EARLIER item this one revises, or None.
source_path
instance-attribute
title
instance-attribute
body
instance-attribute
date
instance-attribute
supersedes = None
class-attribute
instance-attribute
FoundingReport
dataclass
ingested = 0
class-attribute
instance-attribute
chunks = 0
class-attribute
instance-attribute
superseded = 0
class-attribute
instance-attribute
ingest_founding(items, raw, store, embedder, catalog, *, supersession_store=None, attestor=None)
Ingest the founding sequence through the STEADY-STATE path (no bespoke writer): each item
rides ingest_note (AUTHORED_SOLO) → index_records → VaultCatalog.record, exactly like
curated/dialogue ingest. Dated (undated refused) and supersession-linked (recorded as an
OWNER-DECLARED authored-historical supersession when a later musing revises an earlier — 8f).
Ingest, not fine-tuning — weights never move (§1); AUTHORED_SOLO, never the control (§2.3).
build_and_ingest_founding(items, config=None)
Wire + run the founding ingest against the configured stores + embedder (owner-run; needs the live embedder — see scripts/ingest_founding.py).
index
Index ingest records into the vector store + provenance-aware semantic search (BUILD-SPEC §8, §9).
Vectors are derived and regenerable: to re-index after a model/strategy change, rebuild the vector store from the raw corpus (§8) rather than mutating in place.
Chunk points are keyed by a DOC-SCOPED content address (source_path, chunk_hash) (§3/§4,
build plan R1): stable across versions of a note (an unchanged chunk keeps its point) and
distinct across documents (two notes sharing a verbatim chunk keep both points — corroboration,
§7). index_amendment uses that to re-embed only the chunks that actually changed.
index_records(records, embedder, store)
Embed each record's chunks and add them to the vector store. Returns rows added. Identical-content notes (same digest) are embedded once; within a note, chunks are deduplicated by content hash — one point per canonical chunk (§3).
index_amendment(record, existing_rows, embedder, store)
Re-index one note as a chunk-level amendment (ingest-identity-and-amendment.md §4).
Reuse the vector of any chunk whose content is unchanged from the note's current projection
(existing_rows) — NO re-embed — embed only genuinely new chunks, dedup this version's chunks
by content, and replace the note's projection under its stable source_path. Returns
(embedded, reused). The stable parts of a frequently-edited note therefore never re-embed and
keep a stable point id; only changed/new chunks cost an embedding call.
rekey_preview(store)
(total rows, count whose id would change) under the doc-scoped re-key — a read that mutates nothing. The dry-run half of the Item-1c migration.
rekey_store(store)
Re-key every stored row to its doc-scoped content id (source_path, chunk_hash) IN PLACE,
preserving vectors (NO re-embed) — the migration off the old {digest}:{chunk_index} scheme
(build plan Item 1c). Identical chunks within a source coalesce to one point (§3). Idempotent:
a row already under the new key re-keys to itself, so re-running is a no-op. Returns
(rows_read, points_written); the raw store and catalog are untouched (this is a derived-layer
re-key, not a re-ingest — so it needs no embedder and cannot be defeated by catalog change-
detection the way a reset + rescan() would be). Regenerable from raw regardless (§8).
semantic_search(query, embedder, store, *, k=5, provenances=MIRROR_READABLE)
Search the thought-graph. Defaults to MIRROR_READABLE (AUTHORED only) — the introspective default that keeps observed exhaust out of the mirror. Pass provenances=None for the assistant tier to search across all classes.
grouped_semantic_search(query, embedder, store, *, k=5, provenances=MIRROR_READABLE)
Semantic search returning results grouped by SOURCE OBJECT instead of flat chunks.
The explicit opt-in to source-grained retrieval: flat semantic_search stays the default
and is untouched (byte-identical), and this is a separate entry point rather than a flag on
it — the recursive-strata I3 floor-zero posture (the grouped mode adds nothing to the flat
path). k is the flat chunk budget; the returned sources are those chunks grouped by
digest, so a query hitting two chunks of one note yields one source with two members. Source
order follows search rank (each source at its best hit; see group_sources). Defaults to
MIRROR_READABLE like semantic_search. To expand a hit to its full membership rather than
only the matched chunks, call source_set(store, hit.digest).
mint_ids
The id-mint migration — durable id:: identity + version re-key (bp-034; oq-0019 B; §11 ruling).
The owner-run, offline, idempotent, reversible migrator that mints a durable Logseq id:: into each
vault note lacking a stable id AND re-keys that note's version history from source_path to the
minted id, so no lineage forks at the identity switch — now or on any future rename (the A6 payoff,
temporal-retrieval-algebra.md §2.4). It mirrors core/ingest/purge.py: a deliberate, owner-gated
(confirm=True, fail-closed), offline act — NEVER the watcher's default, NEVER fired in a build
session. The owner runs it once, corpus-wide, with the daemon DOWN (a live-store migration is
deploy-coupled — finding-0066).
Three primitives + one orchestrator:
* preview() — Item 13: pure read. Enumerate the mint set (no-stable-id notes), the re-key plan,
and a pre-state manifest (per-chain (seq, digest)) for verification. Mutates
nothing.
* mint() — Item 15: byte-preserving id:: <uuid4> insertion (one line added, nothing else).
Idempotent-skip any note that already carries a stable id.
* run() — Item 16: backup → dry-run → [confirm] → PER-NOTE (mint-then-rekey) → rescan →
verify no lineage forked → report. Refuses unless the daemon is down.
Why per-note mint-then-rekey and not "re-key all stores, then mint all notes" (the §6 amendment):
the batch order has a crash window — a chain re-keyed to id₁ never written into its note, so a naive
re-run mints a fresh id₂ and orphans the id₁ chain. Minting a note FIRST, then re-keying that note's
chain from its ACTUAL state (its id:: vs which key holds its chain), makes any interleaving
converge — the re-key old is always the note's source_path (a chain lives under either its
source_path, un-migrated, or its own id::, already migrated → CHECK ORDER (ii) no-op).
Zone A, no network (the seal holds); the script scripts/mint_ids.py is the owner-facing entry.
MintRefusedError
Bases: RuntimeError
The migration was refused by a safety gate — no confirm, a live daemon, or a note whose
structure the byte-preserving minter cannot place an id:: into safely (§10: never guess a
format that could corrupt an authored note). The PurgeRefusedError pattern: fail-closed,
named, never a silent partial write.
NotePlan
dataclass
One note's place in the migration: where its chain lives now, and where it must end up.
source_path
instance-attribute
current_doc_id
instance-attribute
target_id
instance-attribute
action
instance-attribute
ChainSnapshot
dataclass
A per-chain pre-state manifest (§4a): the (version_seq, digest) sequence, so verification
checks CARRIED CONTENT equality (old key emptiness alone is necessary but not sufficient).
doc_id
instance-attribute
rows
instance-attribute
MigrationPlan
dataclass
mint_set
instance-attribute
rekey
instance-attribute
skipped
instance-attribute
pre_state
instance-attribute
pre_state_for(doc_id)
MintReport
dataclass
minted = field(default_factory=dict)
class-attribute
instance-attribute
rekeyed = 0
class-attribute
instance-attribute
rescan = ''
class-attribute
instance-attribute
verified = False
class-attribute
instance-attribute
backup_dir = ''
class-attribute
instance-attribute
logseq_id(parsed)
The note's durable id:: value (what bp-031 resolution reads back), or None if absent.
has_yaml_id(text)
Does the note's YAML front-matter (if any) carry an id: key? Repo docs do (id: dn-…).
has_stable_id(parsed)
True if the note already carries a durable id — a Logseq id:: OR a YAML id: — so the
minter SKIPS it (idempotent). This is what keeps repo design-notes/findings untouched.
preview(sync)
Item 13 — the migration's auditable plan, computed by PURE READ (opens no write handle,
mutates no store or file). Enumerates: (a) the mint set — vault notes lacking a stable id — and
(b) the re-key plan — for every in-scope note, source_path → its durable id:: where the chain
is not already filed there. Emits the §4a pre-state manifest so run() can verify carried
content, not merely old-key emptiness.
mint(sync, source_paths)
Item 15 — insert a durable id:: <uuid4> into each named note, byte-preserving except the
single added line. Idempotent-skip any note that gained a stable id since the preview (checked
here). Writes ONLY under the vault (the caller passes vault-scoped source_paths). Returns the
source_path → minted id map the re-key consumes.
run(sync, *, declaration, confirm=False, backup_dir, run_ledger=None)
Item 16 — the offline, confirm-gated, reversible orchestration. Order (§6, the crash-safe
per-note amendment):
daemon-down + confirm gates → backup(vault + version/catalog stores) → dry-run plan →
PER NOTE { mint id:: (if none) → re-key ITS chain source_path→id:: on both stores } →
rescan → VERIFY no lineage forked (against the §4a manifest) → report.
Fail-closed: refuses unless confirm=True, unless the daemon is down (a live daemon would race
the re-key against the watcher's re-ingest — finding-0066), and unless every backup is readable
+ non-empty before any mutation. Reversible: restore_from_backup(backup_dir, sync) reproduces
the pre-migration state exactly (the integration test rehearses this). The re-key is owner-
authorized (declaration); a machine caller is refused at each store boundary.
restore_from_backup(backup_dir, sync)
Reverse a migration: copy the backed-up stores + vault back over the live ones. The stores must be CLOSED first (SQLite holds the file); the caller re-opens after. The integration test exercises this to prove reversibility (§4b) — it is not merely asserted.
purge
Purge-raw — deliberate, owner-gated TRUE deletion (design-notes/vault-sync-and-capture.md).
The watcher never deletes raw: a vault delete only TOMBSTONES (derived dropped, raw kept) so nothing is lost and a re-add dedups. But for genuine privacy deletion the owner must be able to destroy the source bytes too. That is this action — and it is deliberately NOT the watcher's default, mirroring the propose/approve posture of the self-modification gate (Invariant 4): destroying ground truth is consequential and irreversible, so it requires an explicit owner act and refuses to fire on content still in use.
Two gates, both fail-closed:
1. confirm=True must be passed explicitly (no accidental purge).
2. the digest must have zero active references — an active note still holds this content,
so tombstone/delete it from the vault first. (Purge operates on already-tombstoned data.)
On success it drops derived rows, removes the raw blob, and deletes the tombstoned catalog
rows for that digest. scripts/purge_raw.py is the owner-facing entry.
PurgeRefusedError
Bases: RuntimeError
The purge was refused by a safety gate (no confirm, or content still referenced).
PurgeResult
dataclass
digest
instance-attribute
raw_removed
instance-attribute
paths_removed
instance-attribute
purge_raw(digest, *, raw, store, catalog, confirm=False)
Permanently remove a note's raw bytes + derived rows. Owner-gated; see module docstring.
run
Run a full ingest of the configured vault into the real stores (BUILD-SPEC §8, §9).
Rebuild semantics: the raw store is append-only and content-addressed (immutable, dedup); the vector store is rebuilt from scratch each run, because vectors are a derived layer regenerable from raw. This is the entry the scheduler (Phase 3) will drive as a job.
IngestSummary
dataclass
notes
instance-attribute
new_raw
instance-attribute
chunks_indexed
instance-attribute
vector_rows
instance-attribute
run_ingest(config=None, *, rebuild=True)
sync
Incremental vault sync — re-ingest changed notes (design-notes/vault-sync-and-capture.md).
Core-side, LOCAL filesystem only: it reads vault files and writes the local stores. No
network, no edge, no sockets — the seal holds and the import-lint proves it. This is the
deterministic engine; the watcher (core/ingest/watch.py) only triggers it, and the
scheduler runs it as a background job so all store mutation stays on the single writer.
Idempotency rides on the existing content-addressing plus the vault catalog:
- unchanged (same digest, still active) → no-op: no re-embed, no new rows.
- changed / new → (re)embed the note's chunks; the previous digest's derived rows are dropped iff no other active file still references that content.
- deleted → tombstone: derived rows dropped, the catalog row marked inactive, and the
raw blob kept (raw is sacred) so a later re-add dedups. True deletion is the separate,
owner-gated purge (
core/ingest/purge.py), never done here.
Everything ingested is authored-solo — the existing AUTHORED provenance tag (the spectrum
split is deferred, see PROGRESS.md). The mirror firewall is unaffected: these are the owner's
own notes, the mirror's ground truth.
SyncOutcome
Bases: Enum
UNCHANGED = 'unchanged'
class-attribute
instance-attribute
INDEXED = 'indexed'
class-attribute
instance-attribute
TOMBSTONED = 'tombstoned'
class-attribute
instance-attribute
SyncReport
dataclass
indexed = 0
class-attribute
instance-attribute
unchanged = 0
class-attribute
instance-attribute
tombstoned = 0
class-attribute
instance-attribute
tally(outcome)
VaultSync
dataclass
vault
instance-attribute
raw
instance-attribute
store
instance-attribute
catalog
instance-attribute
embedder
instance-attribute
pattern = '**/*.md'
class-attribute
instance-attribute
exclude_dirs = DEFAULT_EXCLUDE_DIRS
class-attribute
instance-attribute
max_chars = 1200
class-attribute
instance-attribute
overlap_chars = 150
class-attribute
instance-attribute
attestor = None
class-attribute
instance-attribute
version_store = None
class-attribute
instance-attribute
sync_path(path, *, rename_by_digest=None)
Re-ingest one note as a chunk-level amendment; unchanged content is a no-op.
rename_by_digest (passed by rescan) maps a just-vanished path's content digest to its
(source_path, doc_id); a NEW path with an exact-content match adopts that doc_id so a
rename continues the version chain instead of forking it (bp-031 Item 2).
handle_deleted(source_path)
A vault file disappeared: tombstone it and drop its projection (by source_path).
Source-scoped, so an identical-content file elsewhere keeps its own rows. Raw is kept
(sacred); true deletion is the separate, owner-gated purge (core/ingest/purge.py).
rescan()
Full catalog-vs-vault reconciliation. The watcher triggers this; it is the idempotent backbone (an unchanged re-scan does no work) and also the catch-up path for changes that happened while no watcher was running.
build_vault_sync(config=None)
Wire a VaultSync against the configured vault + real stores + embedder.
watch
Local directory watcher — core-side, LOCAL filesystem only, NO network.
Watches a configured directory and signals when its files change, so the system keeps derived
state current as the source is written. It does not mutate the stores itself and does
not import the scheduler: on a change it just calls an injected on_change callback. A
scheduler wiring supplies that callback — scheduler/vault_sync.py enqueues a background
vault_sync (the owner's vault), scheduler/chat_sync.py enqueues a chat_sync (the Claude
Code transcripts, bp-069) — so all store writes stay on the single supervisor writer. One
generic watcher, many watched directories (the class was VaultWatcher; generalizing it to
DirectoryWatcher is a pure rename — a Vault-named class watching chat is a DRY smell).
Seal integrity: this module imports no edge, no sockets, no http — only the local
filesystem (the import-lint proves it). watchdog (FSEvents/inotify) is an OPTIONAL real-time
backend, imported lazily; without it the watcher falls back to polling (a timer that
triggers a periodic rescan). Either way on_change ultimately runs an idempotent re-ingest, so
missed/duplicate events are harmless.
OnChange = Callable[[], None]
module-attribute
ObserverLike
Bases: Protocol
The slice of a watchdog Observer this watcher drives (watchdog is an OPTIONAL dependency, so its own types never appear in signatures here).
stop()
join(timeout=...)
DirectoryWatcher
dataclass
path
instance-attribute
on_change
instance-attribute
debounce_s = 1.0
class-attribute
instance-attribute
poll_interval_s = 5.0
class-attribute
instance-attribute
backend = field(default='', init=False)
class-attribute
instance-attribute
notify()
A change was observed. Arm/re-arm the debounce timer so a save burst fires once.
start(*, prefer='auto')
Begin watching. prefer: 'auto' (watchdog if importable, else poll), 'watchdog',
or 'poll'. Returns the backend actually started.