core.research
core.research
Research airlock — the sealed-core (Zone A) side of the one-way research flow (§16).
The owner asks a health/research question; the core must NOT leak the question or any note content to the network. Instead it emits de-identified research criteria (short topical terms only), which the Zone-B bridge carries to the cloud fetcher; public literature comes back and is ranked inside the walls against the private corpus.
This package is sealed-core code: it has NO network, NO S3, NO boto3, and never imports
edge/cloud. It only reads and writes a filesystem handoff directory (mirroring the §12
interface handoff). The structural firewall (Invariant 11, §16):
criteria.py— only aResearchCriteria(de-identified terms + filters) can be constructed, anddeidentify()rejects anything that looks like PII or free narrative. The type that crosses the airlock has no field that could carry note content.airlock.py— the core writes criteria requests / reads literature results on disk; it never touches S3 or the network. The corpus never crosses.rank.py— public literature is ranked against the private corpus transiently, inside the core. It is NEVER ingested into the AUTHORED mirror (it is public, external, not the owner's own writing).
ResearchAirlock
dataclass
handoff
instance-attribute
emit(criteria)
Write a de-identified criteria request to the handoff. Returns the request id.
The ONLY thing serialized is criteria.to_request() — de-identified terms + filters.
assert_clean() re-validates here so nothing un-scrubbed can leave even if the
criteria object was built by hand (defense in depth at the trust boundary).
collect_one(criteria_id, *, consume=True)
Read (and by default consume) the literature result for one criteria id.
collect(*, consume=True)
Read every available literature result. The core ranks these inside the walls.
ResearchResult
dataclass
A parsed public-literature result file. Public data — never the mirror.
criteria_id
instance-attribute
papers
instance-attribute
sources_queried
instance-attribute
ts
instance-attribute
from_dict(obj)
classmethod
DeidentificationError
Bases: ValueError
A proposed term/topic could not be safely de-identified, so it is refused.
Raised rather than silently dropped at the boundary (assert_clean) so a leak attempt
is loud. deidentify() (the constructor path) drops un-cleanable terms but still raises
if nothing usable remains — conservative either way.
Paper
dataclass
One public-literature record returned by the fetcher. Public data, not the mirror.
source
instance-attribute
id
instance-attribute
title
instance-attribute
abstract
instance-attribute
year
instance-attribute
venue
instance-attribute
type
instance-attribute
doi
instance-attribute
url
instance-attribute
is_preprint
instance-attribute
open_access = False
class-attribute
instance-attribute
full_text = None
class-attribute
instance-attribute
from_dict(d)
classmethod
ResearchCriteria
dataclass
The de-identified search request. The ONLY object that crosses the airlock outbound.
By construction it carries no note content: a short topic label, scrubbed terms, and
coarse filters. to_request() is the exact outbound wire payload.
topic
instance-attribute
terms
instance-attribute
from_year = None
class-attribute
instance-attribute
publication_types = ()
class-attribute
instance-attribute
max_results = 50
class-attribute
instance-attribute
id = field(default_factory=(lambda: uuid4().hex))
class-attribute
instance-attribute
assert_clean()
Re-validate at the trust boundary (defense in depth). Raises on any violation — a criteria object built by hand cannot bypass the scrubber on its way out.
to_request()
The outbound wire payload — de-identified criteria only, no corpus content.
RankedPaper
dataclass
paper
instance-attribute
relevance
instance-attribute
evidence_tier
instance-attribute
score
instance-attribute
flags
instance-attribute
build_airlock(config=None)
Wire the core-side airlock against the configured handoff directory.
deidentify(topic, terms, *, from_year=None, publication_types=(), max_results=50)
Build a ResearchCriteria from proposed (possibly model-suggested) topic + terms.
Conservative: un-cleanable terms are dropped; if nothing usable remains it raises.
Publication types outside the allowlist are dropped silently. max_results is capped.
rank_literature(result_papers, criteria, embedder, store, *, k_notes=5)
Rank public papers against the owner's relevant AUTHORED notes. Pure read; no writes.
Personalization centroid: retrieve the top-k AUTHORED notes for the topic, re-embed their text locally to get vectors, average them. Each paper is scored by cosine to that centroid plus an evidence-quality adjustment. If the corpus has nothing on the topic, relevance falls back to similarity with the topic query embedding (still fully local).
airlock
Core side of the research-airlock handoff (§16, Invariant 2 & 11).
The sealed core never speaks S3, never opens a socket. It writes a de-identified criteria request into a shared handoff directory and later reads a public-literature result back — exactly mirroring the §12 interface handoff. The Zone-B bridge is the only component that moves these files to/from S3; it never reads the vault, and this module never imports it.
airlock/requests/<id>.json = de-identified criteria (core writes → bridge PUTs to S3)
airlock/results/<id>.json = public literature corpus (bridge GETs from S3 → core reads)
The outbound direction is the firewall: emit() accepts a ResearchCriteria and serializes
only to_request() (de-identified terms + filters), after re-asserting it is clean. There
is no code path here that writes note content outbound — the corpus cannot cross.
REQUESTS = 'requests'
module-attribute
RESULTS = 'results'
module-attribute
ResearchResult
dataclass
A parsed public-literature result file. Public data — never the mirror.
criteria_id
instance-attribute
papers
instance-attribute
sources_queried
instance-attribute
ts
instance-attribute
from_dict(obj)
classmethod
ResearchAirlock
dataclass
handoff
instance-attribute
emit(criteria)
Write a de-identified criteria request to the handoff. Returns the request id.
The ONLY thing serialized is criteria.to_request() — de-identified terms + filters.
assert_clean() re-validates here so nothing un-scrubbed can leave even if the
criteria object was built by hand (defense in depth at the trust boundary).
collect_one(criteria_id, *, consume=True)
Read (and by default consume) the literature result for one criteria id.
collect(*, consume=True)
Read every available literature result. The core ranks these inside the walls.
build_airlock(config=None)
Wire the core-side airlock against the configured handoff directory.
criteria
De-identified research criteria — the one thing allowed to cross the airlock (§16).
THIS IS THE PRIVACY BOUNDARY. The owner's question may be deeply personal ("my migraines since the divorce in March, ..."). What leaves the sealed core to reach the public internet must contain only de-identified topical search terms — no names, dates, places, free narrative, or note content (Invariant 11).
Two layers of defense, both structural:
-
Type shape.
ResearchCriteriahas only short, structured fields (a topic label, a tuple of terms, coarse filters). It has NO free-text body and NO field that could hold the raw query or a note.to_request()serializes only these fields, so the payload that crosses the airlock cannot carry corpus content — the same wayDerivedStorehas noprovenanceparameter, the firewall is in what the type cannot represent. -
Validation (
deidentify). Even the terms are scrubbed: each must be short, in a conservative charset, and free of PII patterns (emails, phone numbers, URLs, handles, long digit runs). The policy is conservative — it raises rather than letting anything doubtful through (DeidentificationError).assert_clean()re-runs the check at the emit boundary so a hand-built criteria object cannot bypass it.
The model only advises (proposes candidate terms); this code acts — it is the enforcer (Invariant 2). The richer PII scrubber (corpus proper-noun denylist, NER) is a documented extension point; the conservative baseline ships now.
ALLOWED_PUBLICATION_TYPES = frozenset({'systematic-review', 'meta-analysis', 'review', 'randomized-controlled-trial', 'clinical-trial', 'cohort-study', 'case-control-study', 'guideline'})
module-attribute
DeidentificationError
Bases: ValueError
A proposed term/topic could not be safely de-identified, so it is refused.
Raised rather than silently dropped at the boundary (assert_clean) so a leak attempt
is loud. deidentify() (the constructor path) drops un-cleanable terms but still raises
if nothing usable remains — conservative either way.
ResearchCriteria
dataclass
The de-identified search request. The ONLY object that crosses the airlock outbound.
By construction it carries no note content: a short topic label, scrubbed terms, and
coarse filters. to_request() is the exact outbound wire payload.
topic
instance-attribute
terms
instance-attribute
from_year = None
class-attribute
instance-attribute
publication_types = ()
class-attribute
instance-attribute
max_results = 50
class-attribute
instance-attribute
id = field(default_factory=(lambda: uuid4().hex))
class-attribute
instance-attribute
assert_clean()
Re-validate at the trust boundary (defense in depth). Raises on any violation — a criteria object built by hand cannot bypass the scrubber on its way out.
to_request()
The outbound wire payload — de-identified criteria only, no corpus content.
Paper
dataclass
One public-literature record returned by the fetcher. Public data, not the mirror.
source
instance-attribute
id
instance-attribute
title
instance-attribute
abstract
instance-attribute
year
instance-attribute
venue
instance-attribute
type
instance-attribute
doi
instance-attribute
url
instance-attribute
is_preprint
instance-attribute
open_access = False
class-attribute
instance-attribute
full_text = None
class-attribute
instance-attribute
from_dict(d)
classmethod
clean_term(term)
The conservative outbound-term scrubber, public seam. ONE implementation shared by every
boundary where a term leaves the sealed core (airlock criteria here; sensing requests in
core/sensing.py) — so the de-identification policy cannot drift between surfaces. Raises
DeidentificationError on anything unsafe.
deidentify(topic, terms, *, from_year=None, publication_types=(), max_results=50)
Build a ResearchCriteria from proposed (possibly model-suggested) topic + terms.
Conservative: un-cleanable terms are dropped; if nothing usable remains it raises.
Publication types outside the allowlist are dropped silently. max_results is capped.
curate
Curate a reference manifest: flip DISTILLED → EMBEDDED, and guard the dangling claim (Item 30).
After the persist step (Item 29) embeds a keeper's open-access full text into the curated store,
its docs/reference_material/<slug>/manifest.md records the transition: source_ingestion.state:
embedded with the store_ref join to data/ (the v0 schema, reference_material/README.md).
Two mechanisms, both deliberately format-respecting (the repo has NO YAML dependency — manifests are hand-authored markdown with inline comments, so a naive YAML round-trip would strip them):
set_embedded(text, ...)rewrites ONLY thesource_ingestion:block, leaving every other block (and its comments) byte-identical — the minimal edit a curation-time flip needs.ingestion_errors(...)is the dangling-claim guard — the Item-30 falsifier as a check: a manifest may not readembeddedwithout a non-nullstore_ref/venue/retrieved, and (given the store) without curated vectors actually present for thatstore_ref. A pointer that promises full text the store does not hold is exactly the "verify before trust" failure the whole arc guards against.
This module writes/validates manifest TEXT; the real DISTILLED→EMBEDDED flip of a live keeper rides a live driver run (there is no core-side fetch — Inv 2), so it is a curation-time act, not a build-time one (bp-029 journal; the offline build delivers the mechanism + guard, tested).
parse_frontmatter(text)
Parse a reference manifest's front matter (one level of nesting: dicts + lists).
Enough for the v0 schema — top-level scalars, the nested verification/identifiers/
source_ingestion mappings, and the load_bearing_for/cited_by/docs lists. No external
YAML dependency (the repo hand-parses front matter everywhere).
set_embedded(text, *, venue, store_ref, retrieved)
Return text with its source_ingestion: block rewritten to the EMBEDDED state.
Only that block changes; every other line (and comment) is preserved. Idempotent in shape.
mark_manifest_embedded(path, record, *, retrieved)
Flip a manifest file DISTILLED → EMBEDDED from a persist CuratedRecord (venue + store_ref).
Refuses to write a dangling claim: validates the result against the store the record was
persisted into is the caller's responsibility (ingestion_errors); here we only guarantee the
written block carries the record's real store_ref.
schema_errors(fm)
Validate a parsed manifest against the v0 schema (required keys + enum values).
ingestion_errors(fm, *, curated_store=None)
The dangling-claim guard (Item-30 falsifier): an embedded manifest must be backed by a
real store_ref/venue/retrieved and, when the store is given, by curated vectors that
actually exist for that store_ref. A not_fetched manifest must carry no store_ref.
persist
Persist ranked open-access keepers into the curated store (bp-029 Item 29, §2.4/§2.6).
The flip the whole arc turns on: rank_literature produced a reading list that was ranked and
DISCARDED (transient — there was no curated home). This module gives the keepers a home: for a
keeper whose open-access full text clears the licence gate, chunk_text → embed_documents →
curated_store.add(...) with provenance="curated". The result is the pattern-finding substrate
the design note calls EMBEDDED (§2.2).
Three invariants are load-bearing and all hold STRUCTURALLY here:
- Inv 2 / core never fetches. This step reads the full text ALREADY on the
Paper(fetched by the Zone-C fetcher, carried back through the airlockresults/); it imports no network and performs no fetch. The full text is an input, never something core reaches for. - Never-pollute-the-mirror. Rows are written with
provenance="curated"— a class excluded fromMIRROR_READABLE— into the SEPARATE curated store the caller passes (open_curated_store), never the mirror.source_pathis areference:marker, never a note. - The licence gate (default-DENY, §11). A keeper is embedded ONLY if
open_accessis set AND it carries non-emptyfull_text. Belt-and-suspenders: the explicit OA flag (set at the fetch boundary, Item 27), not mere text-presence, is what admits a paper. Everything else is SKIPPED and stays DISTILLED-only — the plan's Item-29 falsifier.
store_ref is the content-address of the full text (sha256), the manifest's join to data/
(Item 30). The step returns data (curated records), takes no action — Inv 4.
CuratedRecord
dataclass
One persisted keeper — the handle Item 30 writes into its reference_material/ manifest.
paper_source
instance-attribute
paper_id
instance-attribute
title
instance-attribute
store_ref
instance-attribute
source_path
instance-attribute
n_chunks
instance-attribute
persist_keepers(ranked, embedder, curated_store, *, retrieved=None)
Embed every open-access keeper's full text into the curated store; skip the rest.
Pure over its inputs + the passed store: reads RankedPapers (full text already fetched),
never touches the network. Keepers without open-access full text are left DISTILLED-only.
retrieved is accepted for symmetry with the manifest write (Item 30) and is not used here.
rank
Rank public literature against the private corpus — inside the walls (§16).
"Dumb outside / smart inside": the fetcher does broad public aggregation in the cloud; the personalization — which papers actually matter to THIS owner's notes — happens here, on the local embedder, where the corpus never has to move (Invariant 11).
The fetched papers are public, external literature. They are ranked transiently and are NEVER written into the AUTHORED mirror (that pool is the owner's own writing; mixing in external papers would corrupt the mirror and the §15 baselines). Ranking produces a reading list the owner reads; it does not mutate any store.
Two signals, combined deterministically:
- Relevance — cosine similarity of each paper (title + abstract, embedded locally) to the centroid of the owner's most-relevant AUTHORED notes for the topic. This is the private-corpus personalization, computed without the corpus ever leaving.
- Evidence quality (§16 / Constitution §III.1 honesty) — bias toward systematic reviews / meta-analyses / guidelines; flag preprints as not yet vetted; flag any paper lacking a resolvable identifier. We surface evidence quality, not just a conclusion.
rank_literature returns an ordered list; the research advisor informs, and final health
decisions defer to a clinician (Invariant 7) — enforced by the Librarian/Constitution frame,
not here.
RankedPaper
dataclass
paper
instance-attribute
relevance
instance-attribute
evidence_tier
instance-attribute
score
instance-attribute
flags
instance-attribute
rank_literature(result_papers, criteria, embedder, store, *, k_notes=5)
Rank public papers against the owner's relevant AUTHORED notes. Pure read; no writes.
Personalization centroid: retrieve the top-k AUTHORED notes for the topic, re-embed their text locally to get vectors, average them. Each paper is scored by cosine to that centroid plus an evidence-quality adjustment. If the corpus has nothing on the topic, relevance falls back to similarity with the topic query embedding (still fully local).