Field Horizon

A local retrieval, provenance, replay and canonical-synthesis system: heterogeneous corpora are acquired under an evidence-based rights engine, indexed for hybrid retrieval, argued over by adversarial agents, judged, and folded back into a canon that every later cycle can retrieve and every later reader can trace.

Mission profile

• Acquire corpora from official library catalogues through typed source adapters, never by crawling and never past authentication.
• Decide rights deterministically, from recorded evidence, and quarantine everything that cannot be evidenced.
• Retrieve with a plan that names its strategies rather than a similarity score that explains nothing.
• Keep provenance as queryable edges with exact source offsets, so a claim can be walked back to the bytes it came from.
• Record every operation as an append-only event, so a run can be replayed instead of remembered.

Architecture

Verified state

What exists and has been exercised at the time of writing.

• The source is public: 97 Python modules in the engine package, a Rust core, a FastAPI service and a Tauri desktop client, published on 2 September 2026.
• Corpus acquisition runs through 15 source adapters over OPDS, OAI-PMH and SRU catalogues, behind a per-source host allowlist.
• The rights engine is deterministic code, never a model call, and it cannot return accept for a signal that carries no recorded evidence.
• Retrieval is planned across nine declared strategies, and each strategy reports whether a writer in the codebase actually backs it.
• Provenance edges cover six object types and eleven relation types, carry validity intervals, and are queryable rather than narrated.
• The service contract is generated, not hand-written: 49 operations and 87 schemas in the OpenAPI document, with TypeScript types derived from it.
• The suite declares 1,196 test functions across 85 files, including a parity suite that holds the Rust and Python pressure engines to the same result.

Stated limits

What this project is not, written here so nobody has to discover it later.

• This site reports figures counted from the published repository at commit 7a4fd60. It did not run the test suite, and a declared test is not a passing test.
• The engine is local-first by design: it reaches models through a local Ollama instance and has no hosted deployment, no multi-user model and no independent security audit.
• Nothing it produces is wired to distribution. It is a closed instrument, and opening that boundary would be an architectural decision rather than a configuration change.
• Retrieval quality, verdict quality and the value of a synthesized fragment are editorial judgements. No gate in this system signs them.
• Several planner strategies are rule-based and deliberately small. That makes them exhaustively testable; it does not make them clever.

Acquisition is an adapter problem

Corpus material does not arrive as a folder of files. It arrives as catalogue records from institutions that each publish through a different protocol, so the ingestion path is a registry of typed adapters rather than a scraper. Fifteen adapters cover OPDS feeds, OAI-PMH repositories and SRU endpoints, and each one declares the capabilities it actually has instead of pretending to a uniform interface it cannot honour.

Two rules bound the whole path. No URL found inside a downloaded document is ever fetched: only URLs an adapter extracted from a catalogue record, against a per-source host allowlist, are ever requested. And nothing is indexed whose rights are unknown.

Rights are decided by evidence, never by a model

The rights engine is ordinary deterministic code. It classifies a licence signal into accept, quarantine or reject, and its last guard is the one that matters: an accept is impossible without recorded evidence of where the licence assertion was made. A publication date is enriching context, never a public-domain verdict, because copyright term depends on jurisdiction, edition, translation and editorial apparatus — none of which a year can settle.

    # The evidence gate. Everything above could pass and this still
    # refuses: an accept with no recorded proof is exactly what rule 8-10
    # forbids, and "the provider's field said so" only counts when the
    # adapter recorded WHERE it said so.
    if not signal.evidence:
        work.add(ReasonCode.EVIDENCE_MISSING)
        return _refuse(Decision.QUARANTINE, normalized, work, profile, signal, checked_at)

    if lic_facts.accept_reason:
        work.add(lic_facts.accept_reason)

    return RightsDecision(
        decision=Decision.ACCEPT,

Retrieval is planned, and the plan is typed

Retrieval combines SQLite FTS5 over ingested texts, embedding similarity over the same material, domain-biased routing from a declared ontology, and maximal-marginal-relevance selection over accepted canon so that retrieved fragments are relevant to the query and dissimilar to each other. A planner sits above all of it and emits a typed plan: which strategies were selected, which were available, and why.

# The full strategy vocabulary retrieval planner v3 recognizes (Implementation
# Brief III, Phase E). Every one of these has REAL backing in this
# codebase today (see the Phase E dossier's per-strategy grounding) --
# CONTRADICTS (as opposed to OPPOSES) has no writer anywhere in this
# repository and is deliberately absent, not stubbed.
STRATEGY_LEXICAL = "LEXICAL"
STRATEGY_VECTOR = "VECTOR"
STRATEGY_DOMAIN_ROUTING = "DOMAIN_ROUTING"
STRATEGY_ENTITY = "ENTITY"
STRATEGY_MOTIF = "MOTIF"
STRATEGY_CONTRADICTION = "CONTRADICTION"
STRATEGY_CANON_GENEALOGY = "CANON_GENEALOGY"
STRATEGY_SCHOOL = "SCHOOL"
STRATEGY_TEMPORAL = "TEMPORAL"

STRATEGIES = (
    STRATEGY_LEXICAL, STRATEGY_VECTOR, STRATEGY_DOMAIN_ROUTING, STRATEGY_ENTITY, STRATEGY_MOTIF,
    STRATEGY_CONTRADICTION, STRATEGY_CANON_GENEALOGY, STRATEGY_SCHOOL, STRATEGY_TEMPORAL,
)

Provenance is a graph, not a footnote

Where a claim came from is stored as relational edges over rows the system already has — cycles, chunks, structured entries, schools, sources and councils — with eleven relation types covering selection, membership, supersession, retirement, rehabilitation, derivation, extraction, support and opposition. Chunks keep exact source offsets, so a retrieved passage resolves back to a byte range in a specific ingested document rather than to a plausible-looking citation.

@dataclass(frozen=True)
class ProvenanceEdge:
    source_type: str
    source_id: str
    target_type: str
    target_id: str
    relation_type: str
    confidence: float = 1.0
    polarity: float | None = None
    evidence_ref: str | None = None
    provenance_ref: str | None = None
    valid_from: str | None = None
    valid_to: str | None = None
    creation_method: str = CREATION_WRITE_TIME
    review_status: str = "unreviewed"
    id: int | None = None
    recorded_at: str | None = None

Every operation leaves an event

Cycles, councils, dream runs, ingests and retrieval plans all emit append-only domain events carrying correlation and causation identifiers alongside a run identifier. Run manifests bind the inputs, model context and outputs of a run together, and the event log exports to JSONL, so a past run is replayed from its record rather than reconstructed from a summary somebody wrote afterwards.

@dataclass(frozen=True)
class DomainEvent:
    event_id: str
    event_type: str
    event_version: int
    occurred_at: str
    recorded_at: str
    actor: str
    aggregate_type: str
    aggregate_id: str | None
    correlation_id: str
    causation_id: str | None
    run_id: str
    payload: dict
    id: int | None = None
    canon_event_id: int | None = None
    principal_id: str | None = None

Downloaded bytes are treated as hostile

Everything the acquisition path downloads is assumed hostile until proven otherwise. No downloaded byte is executed, rendered, or interpreted as markup that could execute; HTML is reduced to text by a tokenizer with no scripting concept at all; archive members are checked for path traversal and expansion limits before extraction; and metadata never reaches a shell, a SQL string or a filesystem path unsanitized.

def safe_parse_xml(data: bytes | str, *, max_bytes: int = MAX_XML_BYTES) -> ElementTree.Element:
    """
    Parse XML with external entities and DTDs refused outright.

    Rather than configuring the parser to *resist* entity expansion, the
    declarations are rejected before parsing begins. That is a stricter
    contract and a far easier one to verify: there is no expansion budget
    to tune and no parser-version-dependent behaviour to reason about.
    Legitimate OPDS, OAI-PMH, RDF, SRU, and TEI records do not carry
    DOCTYPEs.
    """
    raw = data.encode("utf-8", errors="replace") if isinstance(data, str) else data

    if len(raw) > max_bytes:
        raise UnsafeXMLError(f"XML payload of {len(raw)} bytes exceeds the {max_bytes}-byte limit")

    head = raw[:8192]
    if _DOCTYPE_RE.search(head):
        raise UnsafeXMLError("Refusing XML containing a DOCTYPE declaration")
    if _ENTITY_RE.search(raw):
        raise UnsafeXMLError("Refusing XML containing an ENTITY declaration")

A second implementation that is not allowed to disagree

The ontological pressure graph — the structure that pairs entries whose categories a declared ontology holds to be opposed — exists twice: once in Python inside the engine, once in Rust as an independent core over the same merged corpus and the same ontology file. A parity suite holds the two to the same result, and the Rust core falls back loudly, not silently, when its binary is not built. The Rust boundary exists for exactly this reason: the computation must remain deterministic and comparable, not merely fast.

/// Edges only ever exist between the (~10) category pairs ontology.yaml
/// declares as opposed, so there is no reason to examine all N*(N-1)/2
/// entry pairs: group entries by category first, then cross only the
/// groups an opposition pair actually names (review §4).
pub fn build_pressure_report(
    entries: &[JsonEntry],
    limit: usize,
    oppositions: &OppositionTable,
) -> CoreReport {
    let mut groups: HashMap<String, Vec<&JsonEntry>> = HashMap::new();
    for entry in entries {
        let category = cat(entry);
        if !category.is_empty() {
            groups.entry(category).or_default().push(entry);
        }
    }

    let mut edges: Vec<PressureEdge> = Vec::new();
    let mut seen_pairs: HashSet<(String, String)> = HashSet::new();
    let empty: Vec<&JsonEntry> = Vec::new();

One contract, two clients

A FastAPI service exposes the engine under a generated OpenAPI document — 49 operations and 87 schemas — from which the TypeScript types used by the desktop client are derived rather than hand-maintained. The client itself is Tauri and React, with an end-to-end suite driving the real application. A command-line entry point covers the same operations, and a parity test holds the two entry points to the same surface.