Add a vector store / retrieval provider

Swap Dhaga's embedding model and vector store independently through the retrieval gateway — any external engine, as long as declared dimensions match.

Semantic search has two independent seams: the embedding provider that turns text into vectors, and the vector store that persists and searches them. They are separate plugins so you can swap either without touching the other — as long as their declared dimensions agree. Both live in packages/core/src/retrieval/types.ts. The built-ins are local-huggingface (384-dim, on-device, $0) and pgvector.

The two contracts

// packages/core/src/retrieval/types.ts
export interface EmbeddingProvider {
  id: string;
  dimensions: number;
  isConfigured(): boolean;
  embedDocuments(texts: string[]): Promise<number[][] | null>;
  embedQuery(text: string): Promise<number[] | null>;
}

/** Whose content a vector is about — a contact OR a company, never both. */
export interface VectorSubject {
  contactId?: string | null;
  companyId?: string | null;
}

export interface VectorRecord extends VectorSubject {
  ownerType: string;
  ownerId: string;
  content: string;
  vector: number[];
}

export interface VectorHit extends VectorSubject {
  content: string;
  ownerType: string;
  similarity: number;
}

export interface VectorStore {
  id: string;
  dimensions: number;
  upsert(records: VectorRecord[], options?: VectorWriteOptions): Promise<void>;
  search(vector: number[], options?: VectorSearchOptions): Promise<VectorHit[]>;
  has(ownerType: string, ownerId: string): Promise<boolean>;
  delete(ownerType: string, ownerId: string, options?: VectorWriteOptions): Promise<void>;
  deleteMany(records: Array<Pick<VectorRecord, "ownerType" | "ownerId">>, options?: VectorWriteOptions): Promise<void>;
  deleteByContact(contactId: string, options?: VectorWriteOptions): Promise<void>;
  deleteByCompany(companyId: string, options?: VectorWriteOptions): Promise<void>;
}

An embed* method may return null to signal "embeddings are unavailable right now" (e.g. the model isn't loaded) — callers treat that as a clean degrade to keyword search, not an error. VectorSearchOptions accepts limit and minimumSimilarity.

A vector has exactly one subject

VectorRecord and VectorHit both carry a VectorSubject: the record a piece of content is about. It is a contact or a company, never both and never neither — Dhaga indexes notes and facts written about a company as well as about a person, and each is stored under the one it actually belongs to.

Store both columns, return both on a hit, and never substitute one for the other. A company vector cannot borrow the id of a contact who happens to work there: the id is what an answer cites back to the user, so a borrowed id is a fabricated receipt — an answer pointing at a person's page for something that was only ever written about their employer. Every AI-derived fact keeps a receipt naming the record it came from; that invariant reaches into the vector store too.

Breaking change for custom vector stores

Two changes a store written against the older contract will not satisfy:

deleteByCompany(companyId, options?) is new and required. It sits next to deleteByContact rather than being optional, because a store that cannot forget a deleted company's vectors would keep feeding them into answers about a company that no longer exists.
contactId is now optional (string | null | undefined) and companyId joins it, on both VectorRecord and VectorHit. A store that declared contactId: string and indexed on it must widen the column and handle a row that has a company instead.

The built-in pgvector store already does both: embeddings gained a nullable company_id and contact_id dropped its NOT NULL — an expand-only change, with the primary key still (ownerType, ownerId).

Dimensions must match

An embedding provider and a vector store are only compatible if they agree on dimensions. Dhaga checks this before indexing or searching and reports both ids and dimensions on mismatch — use the exported helper if you compose them yourself:

import { assertCompatibleVectorDimensions, DEFAULT_EMBEDDING_DIMENSIONS } from "@dhaga/core";

// throws a descriptive error if the two disagree
assertCompatibleVectorDimensions(embeddingProvider, vectorStore);
// DEFAULT_EMBEDDING_DIMENSIONS === 384 (the local-huggingface default)

Register and select

import { registerEmbeddingProvider, registerVectorStore } from "@dhaga/core";

registerEmbeddingProvider({
  id: "my-embeddings",
  dimensions: 768,
  isConfigured: () => Boolean(process.env.MY_EMBEDDINGS_URL),
  embedDocuments: async (texts) => embedBatch(texts),
  embedQuery: async (text) => embedOne(text),
});

registerVectorStore({
  id: "my-vectors",
  dimensions: 768,
  upsert: async (records, options) => { /* idempotent by (ownerType, ownerId) */ },
  search: async (vector, options) => { /* return VectorHit[] */ return []; },
  has: async (ownerType, ownerId) => false,
  delete: async (ownerType, ownerId, options) => {},
  deleteMany: async (records, options) => {},
  deleteByContact: async (contactId, options) => {},
  deleteByCompany: async (companyId, options) => {},
});

Select them in code with selectEmbeddingProvider / selectVectorStore, or with environment variables:

DHAGA_EMBEDDING_PROVIDER=my-embeddings
DHAGA_VECTOR_STORE=my-vectors

getEmbeddingProvider() defaults to local-huggingface and getVectorStore() to pgvector (packages/core/src/retrieval/index.ts). Registration validates that dimensions is a positive integer and that id is non-empty.

Idempotency and the transaction handle

Owner pairs (ownerType, ownerId) are stable keys, so upsert and delete must be idempotent — re-running a sync must not duplicate or corrupt rows.

VectorWriteOptions.transaction is an optional store-specific handle supplied by an owning repository. The built-in pgvector store uses it to keep relational tombstones and vector deletion atomic within one Postgres transaction. An external service can't join a Postgres transaction — ignore the handle and make deletion idempotent and retry-safe instead.

Setting DHAGA_VECTOR_STORE to an external store skips pgvector setup

When you point DHAGA_VECTOR_STORE at an external engine, Dhaga skips the pgvector extension and table DDL entirely — so the relational database no longer needs the vector extension at all. Vector data can be moved to an external engine fully independently of the primary Postgres.

See Providers for the provider checklist and packaging guidance.