Semantic search

Semantic search orders a corpus against a query and hands back a prefix. It is the read path of AI memory over a knowledge base.

import lexical

cards
| lexical("bearer token authorization")
| take(5)
| expand(depth = 1)
| graph

A retrieval is written out because it is an extension capability rather than the core's business — see hql/extensions.

It has a vocabulary of its own but mints no node type and no edge type, so it is not a peer domain in the sense type-system uses: nothing in it is anyone's parent.

Vocabulary

type Hit<T> {
    value      : T
    score      : Score
    provenance : Retrieval
}

type Retrieval {
    query       : Query
    index       : IndexRef
    model       : ModelRef
    metric      : Metric
    approximate : Bool
}

type Ranking<T> : Seq<Hit<T>>

Ranking narrows Seq the way a Relation narrows an Edge: it adds what makes the order meaningful — the query the order is relative to, and the rule that broke ties. A bare Seq<Hit<T>> is an ordered pile of scored things with no statement about where the order came from. See orderable for the tie rule.

A score is evidence, never relevance

The knowledge domain rests on one rule: interpretation is a step, and the rule used must be retained with the result. A similarity score is an interpretation — produced by a named model over a particular chunking under a particular metric against an index at a particular revision.

Proposition   Relevant(card, query)
Evidence      the Hit — score, model, index revision, metric
Provenance    which index answered, and when

A card scoring 0.91 is not more true than one scoring 0.42; it is more similar to a query under one model. That is why Retrieval is not optional on Hit: a hit that cannot say how it was produced is not evidence.

An index contributes embeddings, never scores

Belongs in card.metadata	Cannot	Why
Embeddings, chunk identities		a property of the card
Index membership, indexed revision		a property of the card
Staleness against the card's content		a property of the card
	A score	exists only relative to a query
	A rank	exists only relative to a query and a corpus

A card has an embedding whether or not anyone ever searches; it has no score until someone asks something. A retrieval extension contributes under a key named after itself — the matcher writes metadata.lexical — because two retrievals cannot both be "the" search, and a key either of them could claim is one they would take turns overwriting. See fields and card-type.

Cards are not concepts

semantic ranks representations. AI memory wants to recall knowledge.

semantic(q) : Set<Card>  -> Ranking<Card>      // over representations
recall(q)   : Knowledge  -> Ranking<Concept>   // over knowledge

A Concept can exist with no card at all — imported, derived, or read from a store — which is why Graph<Concept, Relation> is parameterized by Concept. Those cardless concepts have no text, so text search cannot reach them, and a memory that silently cannot recall part of itself is worse than one that says so. recall is the AI-memory operation and is not implemented.

A RelationCard hit is not a seed. It contributes an edge and is not a node, so treating every hit as a node reintroduces the duplication the card split prevents — see card-type.

Approximation is semantics, not optimization

Approximate nearest-neighbour search does not preserve observable values, so pushing semantic | take into a store is illegal under the pipes optimizer rule. The resolution is not to weaken that rule: semantic declares itself approximate in its own contract, so the difference is part of what the step means rather than a liberty the executor took. approximate is a field on Retrieval for the same reason — a consumer can see which answer it got.

Purity follows the same line. The same query against a changed index gives different answers; pinning an index makes semantic reproducible, and without pinning it is an effect. What is pinned today is the model's revision, carried on every Retrieval, and a card whose text has drifted from its vector says so rather than scoring quietly.

What is built

Two retrievals, each an extension and each honestly named. lexical matches shared spellings with hql.hashbag.v1, a hashed bag of words compared by cosine, offline, deterministic and exact, reporting approximate: false. It models no meaning and its name no longer claims to.

semantic scores against vectors a language model computed ahead of time, stored in an SQLite index a vault names in hql.toml. The model stays out of this binary, so the index carries the queries too and a query it does not hold is a failure naming the command that would add one. Scoring is exact brute force over stored vectors, so approximate is false here as well; an approximate index is step four and is not built.

Replacing one retrieval with another changes the numbers, not the shape — which is the point of Retrieval travelling on every Hit, and why it carries the model's revision as well as its name.

expand and graph stay two stages. expand(depth, direction) traverses and produces a graph carrying why each node is present; graph projects a collection, and projecting a graph yields it unchanged.

Open

• Query construction and whether a bare string literal may stand for one.
• Chunking. A card is not a retrieval unit; a passage is.
• Multiple indexes, hybrid retrieval, and whether a fused score can still carry an honest Retrieval.
• Filtering before or after — cards | typed ConceptCard | semantic(q) versus filtering the ranking.
• The write path — remembering, reconciliation, consolidation of co-retrieval into derived relations, and eviction. A score may propose that two concepts are the same and may never decide it.

See the semantic-search requirements for the staged implementation path and the full set of open questions.