ADR-004 — AI & agentic layer (E6)

Status: Proposed (design-panel synthesis) · Date: 2026-07-22 · Track: E6 (Pillar 4 — permissioned index + governed agents) Method: adversarial design panel — estate-scout + four thinker-canon lenses (AI-native/moat · ML-systems & eval-economics · retrieval/IR · data & consistency). Dissent preserved below.

Follows the ADR-001 template. Brand-free: engines/products named by capability only. This ADR decides the retrieval + agent-runtime architecture for Pillar 4; ADR-001's log/tier/index spine (I9/I10) is its foundation and is assumed, not re-litigated.

---

1. Context

Pillar 4 is a permissioned index over all content (courses + boards + events + transcripts + the member's own history) and a set of governed agents on top of it — the AI-curated home (the moat), the tutor, the authoring assistant, and the "talk to the platform" conversational layer. ADR-001 already locked the substrate they read from: an append-only, member-keyed, tier-stamped log is the source of truth; boards/DMs/feed are derived views; tier is evaluated at read against the reader's current tier (I9); any borrowed component is a transport/library/derived-view, never the record or the permission authority (I10). E6 must extend that same discipline one layer up, to the index and the agent runtime — which is exactly where it is most likely to break quietly, because a vector index looks like a cache but behaves like a second database that the AI actively republishes.

The estate-scout investigated the local clones and returned a blunt correction to "the moat is largely built": what exists is a mature retrieval + citation + tamper-evident-audit core plus hard-won trust patterns (single outbound policy gate, deterministic model-free promotion, per-query privacy receipt, refuse-over-guess, hybrid keyword/dense rank-fusion) — but every one of those components was built agent-facing, local-first, and single-operator. The agent-runtime repos are coding-agent-shaped (human-in-the-loop-per-tool-call, sandboxed code execution) or unbuilt scaffolds. Net: roughly ~40% adapted-from-estate (the RAG/trust core), ~60% greenfield (multi-tenant learner memory + progress, a web-delivered learner tutor UX, and per-reader/per-tier/mutable-visibility permission the estate brain has never been asked to enforce). The retrieval engine is real. Permissioned, drift-monitored, cost-instrumented, human-traffic retrieval is not — and that gap is the entire security surface of a paid country-club.

2. Decision drivers

• One permission authority: tier is the same attribute as Pillar 1, resolved at retrieval against the reader's current tier — never a value frozen at ingest, never a post-filter.
• The moat is the substrate, not the ranker: born-chunkable + tier-tagged content + a permissioned index + evals — not a clever bespoke ranking agent (the narrow method that loses).
• Small ops team: the index is a second consistency system; every seam it adds (reindex, snapshot-pinning, tombstone GC, lag monitoring) is a 3am liability that must be justified, not assumed.
• Reversibility: the swappable parts (embedder, reranker, retrieval engine, LLM) must be repoints behind an owned contract — never rewrites of the moat.
• Observability from call one: inference cost per flow and retrieval quality must be measurable before an agent ships, not reconstructed from a surprise invoice or a customer complaint.

3. Options considered

• A — Build directly on the estate brain + governance plane. Estate governed knowledge brain is the permissioned index; agent-governance plane is the runtime; wire tier-at-retrieval + the outbox on top.
• B — Adopt a RAG/orchestration framework. Rent chunking + vector store + retriever + agent loop as a library/service; wire permission + governance around it.
• C — Hybrid. Estate governance plane owns permission + index-of-record semantics + audit; the retrieval-quality internals (chunker, embedder, filtered-ANN, reranker) and the agent-loop are a benchmarked, swappable module behind our contract — a thin framework library admitted only as a dumb pipeline component.
• D — Fully custom. Own retrieval + ranking + agent runtime down to the embedding call.

4. The panel (rankings converged on #1; read the core argument and the dissent, not the rank)

Lens	Rank #1	Core argument
AI-native / moat (Karpathy)	C (C>A>D>B)	The moat is born-chunkable, tier-tagged content emitted at write off the log + a deterministic tier pre-filter + evals as source code — not the ranker. Reuse the runtime; do not believe the product-specific index + permission model ship for free. A framework's default is post-filtering + its own chunker — it fights every invariant.
ML-systems / eval-economics (Huyen)	C (C>A>D>B)	The index is a second, silent copy of the permission system that drifts on downgrade/re-tier. Resolve tier at query time against live state — never ingest-frozen metadata. Inference economics are absent: instrument cost-per-flow from call one; an LLM-per-home-load beats a query on nothing at this member count. Drift needs a production-traffic sampler, not just a model-bump gate.
Retrieval / IR	C (C>A>B>D)	Reuse the governance/index-of-record spine; re-earn retrieval quality — the estate brain was tuned on curated, untiered agent-knowledge, not noisy human boards (BM25's turf) + ASR transcripts (dense's turf). Tier must be an exact pre-filter pushed into the ANN. First deliverable is a tier-correct recall@10 eval on real member queries — BM25 as the mandatory baseline arm — not a model pick.
Data / consistency (Kleppmann)	C (C>A>B>D)	The index is a derived, rebuildable, staleness-bounded view of the log — never a source of truth. Idempotent ≠ ordered: needs per-object monotonic version + last-writer-by-seq + durable tombstones a retried insert can't resurrect. Pin the retrieval set into every agent-run event or curation is unreproducible. A framework store as record is the split-brain ADR-001 forbade, one floor up.
Estate-scout	—	Estate supplies ~40% (a real RAG + citation + audit core + trust patterns), all agent-facing/local-first/single-operator; the multi-tenant, web-delivered, learner-centered layer is ~60% greenfield; the coding-agent runtime repos are STRETCH or unbuilt scaffolds. Reuse the engine + patterns; the permissioned learner layer is new.

5. The convergence (what all five agree on, under one shared rank and divergent reasons)

Every lens ranks C first — but they arrive from four different doors into the same room, and each carries a distinct non-negotiable:

1. The estate is the substrate, never the record. Reuse the governed knowledge brain's retrieval/citation/audit mechanics and the agent-governance plane's runtime — but the index is a derived, rebuildable view of the log, and our tier attribute is the sole permission authority (ADR-001 I9/I10, extended up). A framework store or the brain's native store as system-of-record reintroduces the two-permission-systems / chunk-drift failure ADR-001 already ruled fatal — now laundered through an agent that synthesizes and republishes stale content where no human sees the provenance. This is why B ranks last for everyone who keeps the moat.
2. Permission is resolved at query time, exact, pushed-down, fail-closed. The object's tier_visibility is immutable and stamped at write; the reader's tier is live and re-resolved on every retrieval. The filter is reader_current_tier ⊇ object_tier_visibility inside the ANN query (a pre-filter), never a post-hoc top-k drop (which both leaks via ranking cutoffs and costs recall). Retrieval fails closed on unknown or in-flight tier. The AI-native, IR, and data lenses state this independently; the eval lens supplies the fixture that catches it.
3. The moat is the pipeline, not the ranker. Content is born chunkable + tier-tagged off the log's outbox, not scraped from rendered HTML. Given that substrate, ranking is a repoint — so ship the home as "Latest / Top this week" queries first (ADR-001 §6.3) and let the ranking agent earn its place on a modeled cost-per-home-load + a live engagement proxy, not a calendar date.
4. Measure before you buy the stack. Retrieval quality on this corpus is faith until benchmarked; BM25 is the mandatory baseline arm; dense + reranker are added only where a recall@10 eval on real member queries shows lift. Cost-per-flow is instrumented from call one. Drift is monitored from production traffic, not merely gated at model-bump.
5. The evals are the moat's honesty, and they are source code — built before the first agent reads the first board. Fail-closed tier-leak (with the stale-metadata-after-downgrade fixture), staleness/tombstone consistency, and tier-correct retrieval quality. Absent these, "AI is permissioned at retrieval" (Invariant 3) is a wish.

6. Decision — "Own the emit-and-tier contract · rent the retrieval mechanics · re-earn the quality · defer the ranker"

A hybrid (Option C) that takes the winning move from each lens.

6.1 Own (the moat — cheap now, catastrophic to retrofit)

• The index-emission contract. The outbox from the append-only log (ADR-001) produces chunks at write time, tagged {member_sub, tier_visibility (immutable on the object), source_object_id, object_version, content_type, chunk_seq}. Content is born chunkable — never re-derived by scraping a rendering engine's cooked store. Chunking is corpus-specific and ours: course sections chunk cleanly; board posts chunk per-post with a prepended thread header (title/board/tags) so the citation lands on a verifiable post; transcripts are semantic windows with timestamps so the citation is jump-to-time. Every chunk carries the immutable object-tier stamp and the object version.
• The permission pre-filter. One deterministic function — the same tier attribute as Pillar 1 — that re-resolves the reader's current tier and the object's current visibility at query time, pushed exactly into the retrieval query, applied before the vector/keyword search returns, fail-closed on unknown or in-flight tier. Shared by home, tutor, authoring, and the conversational layer — re-implemented in none of them. It never trusts a tier value frozen in a chunk at emit.
• The index-of-record semantics. The index is a derived, rebuildable, staleness-bounded view of the log — never a source of truth. Application is ordered and tombstone-durable: per-object monotonic version, last-writer-by-seq, deletes/redactions/re-tiers land as durable tombstones a retried insert cannot resurrect; a bounded, monitored lag that trips an alert (not a silent stale read) when exceeded.
• Reproducible agent reads. Every governed agent run pins its retrieval set (object ids + versions + index-snapshot marker) into its append-only run event, so any curation/tutor/synthesis output is reproducible, attestable, and eval-able against the exact inputs it saw.
• The eval harness as source code (§8) — the honesty of all of the above.

6.2 Rent (the commodity — behind our contract, never the record)

• Retrieval mechanics from the estate governed knowledge brain: embedding + ANN + citation binding + tamper-evident audit journal. Reused as a module under our contract, exposing an exact filtered-ANN pushdown and accepting our externally-computed, tier-stamped chunks — not as the definer of chunking, permission, or index-of-record.
• The agent runtime from the estate agent-governance plane: governed headless execution + signed, offline-verifiable audit trail + the single-outbound-policy-gate + refuse-on-insufficient-evidence patterns the estate already proved. Prompts are versioned files, eval-gated, reviewed — not transcripts.
• Commodity plumbing (chunk helpers, evaluators, embedding-provider adapter) admitted as dumb libraries behind a provider-adapter seam, so an embedder or LLM swap is a repoint and the cost instrument lives at that seam.

6.3 Re-earn (retrieval quality is measured, not inherited)

• BM25 is the mandatory baseline arm. Dense embedding + a cross-encoder reranker are added only on grounded-QA surfaces (tutor/authoring) and only where the recall@10 eval shows lift — expect dense to earn transcripts + paraphrase queries, BM25 to hold boards (exact terms, names, code, error strings). The home candidate-generation needs only tier-visible latest/top; "talk to the platform" is substantially NL→query over the relational store, not vector retrieval — do not route it through the index by reflex.
• Embedder capability targets (no product name): English-primary, permissive license, ≥8k-token context (board threads + transcript windows are long), top-band retrieval score, sized to run on the platform's iron; multilingual/larger only if an eval on real non-English member content demands it. A reranker is a second model to serve — justified on measured lift, on QA surfaces only.
• An embedding-model change is a full re-index + a shadow-retrieval comparison (old index vs new, same queries, diff the top-k), never a config flip — a prompt-level eval cannot catch an embedding-space regression.

6.4 Defer / sequence (the scoped cut, not a kill)

• No ranking agent for v1 — the home ships as tier-filtered "Latest / Top this week" queries over the permissioned index; the ranker is a fast-follow whose gate is a modeled cost-per-home-load + a live engagement proxy. Because the index + tier-at-read substrate exists day one, the upgrade is a repoint, not a rebuild.
• v1 retrieval may ship as transactional, same-store, tier-filtered relational/keyword queries over the log's derived views (Kleppmann's sequencing) — consistent with the log by construction, in the same transaction, zero separate-index staleness. The estate vector index is introduced as a fast-follow that must obey the index-of-record invariants from its first byte, and only once a recall@10 eval shows semantic retrieval measurably beats keyword on this corpus.
• A pre-flight verification gates all agent work (§8 first task): confirm the estate brain resolves tier at query time and does exact filtered-ANN pushdown. If it bakes visibility at ingest or only post-filters, that is a corrective bead ahead of any agent code — not a discovery made in production.

7. Tradeoff matrix

Dimension	A build-on-estate	B RAG framework	C hybrid (contract + rented mechanics)	D custom	Decision (C)
Index is a derived view of the log (moat)	⚠️ invites treating brain as record	❌ framework store = 2nd record	✅ by contract	✅	✅ by construction
One permission authority, tier resolved at read	⚠️ only if brain already does query-time	❌ default post-filter, no tier model	✅ owned pre-filter	✅	✅
Exact tier pre-filter pushed into ANN	⚠️ substrate-dependent	❌ metadata post-filter default	✅ required of substrate	✅	✅ (pre-flight verified)
Retrieval quality measured on this corpus	❌ inherited on faith	◑ swappable but unowned	✅ BM25 baseline + eval-gated	◑	✅ recall@10 first
Embedder / LLM swap = repoint	⚠️ coupled to brain	✅ (framework seam)	✅ (adapter seam)	❌ own re-embed forever	✅
Cost-per-flow observable from call one	❌ no seam	◑	✅ at the seam	◑	✅
Ops-service / consistency-system count / small team	✅ fewest seams	❌ 2nd orchestrator	◑ owned contract to operate	❌ highest	◑ (defer index to fast-follow)
Reuses estate governance + audit runtime	✅	❌ competes with plane	✅	❌ rebuilds it	✅

8. Consequences

New locked invariants (promote into the foundation spec §7, extending I9/I10 up to the index layer):

• I11 — The permissioned index is a derived, rebuildable, staleness-bounded view of the content/conversation log; it is never a source of truth. Tier is an exact read-time pre-filter pushed into the retrieval query against the reader's current tier and the object's current visibility; it never trusts a value frozen at ingest; retrieval fails closed on unknown or in-flight tier. (AI-native + eval + IR + data lenses, unanimous.)
• I12 — Index application is ordered and tombstone-durable: per-object monotonic version, last-writer-by-seq; deletes/redactions/re-tiers are durable tombstones a retried insert cannot resurrect; the index carries a bounded, monitored lag whose breach trips an alert, not a silent stale read. (Kleppmann.)
• I13 — Every governed agent run pins its retrieval set (object ids + versions + index-snapshot marker) into its append-only run event, so any curation/tutor/synthesis output is reproducible, attestable, and eval-able against the exact inputs it read. (Kleppmann.)

New required evals (built before the first agent reads the first board — the cheapest thing to add now and the single most expensive thing to retrofit):

• P0 — Retrieval-permission tier-leak eval, fail-closed (extends ADR-001 P0). Golden set of (member fixture T0–T3, query, must-return / must-not-return), including an explicit stale-metadata-after-downgrade fixture (member downgraded / board made private / access revoked → assert the retriever does not surface the now-forbidden chunk). Trigger set: every model bump, prompt change, schema migration, embedding-model swap, and estate-runtime version-pin change. Wired fail-closed into the deploy gate — advisory is a wish, not a property.
• P0 — Staleness / tombstone consistency eval. Re-tier an object → assert it leaves retrieval within the lag bound; delete an object → assert no retried insert resurrects it. Guards I11/I12.
• P1 — Tier-correct retrieval-quality eval. recall@10 / nDCG@10 on 30–50 hand-labeled real member queries against the real corpus, per tier fixture, recall measured against the member's tier-visible relevant set, with a hard must-not-return leak check. BM25 as the baseline arm — this turns BM25-vs-hybrid-vs-hybrid+rerank from faith into arithmetic. This is E6's first deliverable, not a model pick.
• P1 — Directional home-ranking regression eval + a production engagement proxy (dwell/CTR on ranked items) so the feed can't rot silently across model versions (ADR-001 P1).
• P1 — Per-flow prompt eval for each governed agent (curation, tutor, thread-synthesis, conversational); fixtures are source code under review (ADR-001 P1).
• Instrument, not gate — but required from call one: a per-flow cost meter (tokens in/out × price, tagged home/tutor/authoring/conversational) at the adapter seam, emitting to the estate telemetry backend; and a production-traffic sampler logging (query, retrieved-set, member-tier, engagement-proxy) that periodically refreshes the eval set from real distribution — so drift is monitored, not just gated.

E6's first task is a pre-flight verification + the retrieval eval, not an agent build:

1. Verify the estate brain resolves tier at query time and does exact filtered-ANN pushdown (not ingest-time baking, not post-filtering). If it fails either, file a corrective bead ahead of any agent work — this is the pre-flight that decides whether A collapses cleanly into C.
2. Stand up the tier-correct recall@10 eval on real member queries with BM25 as the baseline arm — the first number that decides the retrieval stack.
3. Ship v1 retrieval as transactional same-store tier-filtered queries; introduce the vector index as a fast-follow obeying I11–I13 only once the eval shows semantic lift.

Estate seams (what we actually reuse):

• Governed knowledge brain → retrieval + ANN + citation binding + tamper-evident audit (as a module under our contract).
• Agent-governance plane → governed headless runtime + signed audit journal + single-outbound-policy-gate + refuse-over-guess.
• Deterministic model-free promotion + hybrid keyword/dense rank-fusion patterns → the retrieval-quality module's design north-star.
• Signed evidence-bundle + ship/no-ship rollout gate → regression-gating model/prompt/index changes for this platform at its own boundary (never coupled to the estate's release cadence).

9. Preserved dissent (do not resolve away)

• Skip the vector index entirely for v1 (Kleppmann's self-dissent). At this corpus size (thousands of threads, not millions) and with no firehose, the honest move may be tier-filtered keyword/relational retrieval over the log's derived views in the same store as the log — consistent by construction, zero reindex ordering, zero tombstone race, zero separate-index staleness. The embedding index earns its place only when semantic retrieval measurably beats keyword. The hybrid risks standing up the entire split-brain machinery to serve a capability v1 may not need. Honored by sequencing (§6.4): v1 same-store, vector index as an eval-gated fast-follow — but if the recall@10 eval never shows dense lift, the vector index may never ship, and that is an acceptable outcome.
• BM25 might win the tutor outright (retrieval lens). A cross-encoder reranker + a dense embedder for a lift the eval can't find is quality theater. The pick commits to measuring (BM25 baseline first), not to dense — but the honest prior stands: at this corpus size BM25 may hold the boards and dense earns its place only on transcripts + paraphrase. Run the baseline before buying the stack.
• The provider-adapter seam may be over-engineering (eval-economics self-dissent). For 60 founders + a small paid tier, the extra seam is a hedge against the embedder-swap and cost-visibility failures, not a scale argument. If the pre-flight verification comes back clean and Option A is instrumented directly, A collapses into C minus the framework layer, and A wins on simplicity. The seam is ceremony the day the estate brain already resolves permission at query time and the cost meter is wired to A directly.
• C's discipline is unenforceable by a small team (AI-native self-dissent). "Own the contract, rent the mechanics behind our interface" is the boundary that erodes first under deadline pressure — the day someone calls the brain's native chunker "just for the tutor," or lets a borrowed post-filter stand in "because the eval was green that week," C silently decays into A-done-badly or B-done-badly and no test catches it, because the abstraction boundary isn't evaluated, only outputs are. The rebuttal is the P0 eval wired fail-closed into the deploy gate: the tier-leak golden set at the index boundary is the enforcement of the boundary — but only if it gates deploys, not runs advisory. Which is exactly the most-costly decision below.

10. Single most-costly-to-recover-from decision (the seat's shared non-negotiable)

Whether tier-visibility and retractions stay consistent between the log and the index — the fail-closed tier-leak deploy gate + query-time tier resolution + reindex ordering + tombstone durability + a bounded, monitored staleness window — built before the first agent reads the first board. All four thinker lenses land here from different doors. The failure is silent and non-recoverable: a downgraded member, a deleted post, or a newly-private board keeps surfacing through the AI, and the AI synthesizes and republishes it into rendered feeds, cached agent outputs, tutor answers, and screenshots — so the leak is invisible at the point of exposure and can run for months before anyone notices the historical blast radius, which often can't even be reconstructed after the fact. Every other E6 choice — framework vs estate, one box vs fleet, ranker-now vs later, BM25 vs hybrid, embedder pick — is a repoint. This one is a trust-destroying disclosure that no test catches unless the P0 eval is a fail-closed deploy gate on the index boundary, with the stale-metadata-after-downgrade fixture, triggered on estate-runtime version pin and embedding-model swap. Absent that, Invariant 3 ("AI is permissioned at retrieval") is a wish, and it is exactly the wish whose failure destroys a paid country-club's trust.

11. Follow-up (beads to file in Phase B under the E6 epic)

1. Lock invariants I11 + I12 + I13 into the foundation spec §7.
2. P0 retrieval-permission tier-leak eval (fail-closed, deploy-gated), with the stale-metadata-after-downgrade fixture and the estate-runtime-pin / embedder-swap triggers — before any agent reads community content.
3. P0 staleness/tombstone consistency eval (re-tier leaves retrieval within lag bound; delete survives no resurrection).
4. Pre-flight verification: does the estate brain resolve tier at query time + support exact filtered-ANN pushdown? Corrective bead if not — ahead of all agent work.
5. Tier-correct recall@10 eval on real member queries with BM25 as the baseline arm — E6's first deliverable.
6. Specify the log→index outbox emission contract (born-chunkable, per-chunk immutable tier stamp + object version; corpus-specific chunkers for sections/posts/transcripts).
7. Wire the per-flow cost meter (tokens × price, tagged home/tutor/authoring/conversational) at the adapter seam → estate telemetry.
8. Stand up the production-traffic sampler + eval-set refresh loop (drift monitored, not just gated).
9. Ship v1 home + tutor retrieval as transactional same-store tier-filtered queries; introduce the estate vector index as a fast-follow obeying I11–I13, gated on demonstrated semantic lift.
10. Annotate the estate-integration map with per-seam reversibility (tier/permission authority + index-of-record semantics = irreversible; embedder, reranker, retrieval engine, LLM = repoint behind the adapter seam).

---

Panel transcript basis: estate-scout + AI-native/eval-economics/retrieval/data-consistency lenses, 2026-07-22. Builds on ADR-001 (E4+E5) I9/I10; extends the log/tier/index spine into the index and agent-runtime layer.