Ingest & Embed¶
When search feels stale, catch up:
That is the day-to-day loop. Ingest copies harness history into the warehouse; embed turns message text into vectors for meaning-based search; a nightly schedule runs both so you are not babysitting freshness by hand.
Day-to-day search still goes through the agent (sr-search and friends). This page is for when you want to know what those pipelines do — or when you need to re-run them yourself.
Ingest¶
Ingest is ETL from agentic coding harness transcript roots into the warehouse under stockroom home ($STOCKROOM_HOME/warehouse.duckdb — see Installed layout).
stockroom ingest # both harnesses, incremental
stockroom ingest --full # ignore watermarks; re-read everything (still idempotent)
stockroom ingest --verbose # progress lines (quiet by default)
--harness cursor or --harness claude limits the run to one harness. To read from non-default transcript roots, see Harness sources.
It writes harness-labeled rows into shared tables: sessions, messages, and tool_calls. Prompts and responses are stored whole; tool inputs are kept; tool result payloads are dropped. Thinking/reasoning blocks the harness keeps separate are not stored. Rows whose source transcripts later vanish are not pruned — the warehouse is allowed to outlive its sources.
Default is incremental. Stockroom remembers a per-(harness, source_root) watermark in _sync_state and only reads files past that point. Cursor therefore tracks projects and chats roots independently. Re-runs are cheap and safe.
Migrations do not Backfill. Structural migrations do not backfill columns such as entrypoint — use stockroom ingest --full after a database schema upgrade if you want older rows repopulated from sources (this will be infrequent).
sr-initialize runs stockroom ingest --full once so you are not waiting for the first nightly job. On years of history that first pass can take many minutes (varying greatly depending on your machine's CPU and disk speed); it prints per-harness session/message/tool_call counts when done.
Embed¶
Embed turns non-empty message text into local vectors (BAAI/bge-small-en-v1.5, 384-dim, one row per chunk in embeddings). SQL query works without embeddings; meaning-based recall does not.
Embedding needs a working PyTorch install in the engine venv. Ingest does not. If embed or semantic search fails citing torch / the environment, fix torch first — Troubleshooting > Torch.
Default is incremental. Only messages that still lack an embedding for the current model are processed. Re-runs resume cleanly after interruption.
stockroom embed # pending messages only
stockroom embed --full # re-embed all non-empty messages
stockroom embed --verbose
There must already be a warehouse (run ingest first). The first embed is the long pole: a large corpus on modest hardware can take hours, and the first run may also download the embedding model once if smoke testing during initialize did not already warm it. Nightly jobs stay cheap because they only catch up.
Re-run and Check Coverage¶
If you skipped the first full load (or want to force a full re-read), use the same commands initialize used:
Then sanity-check counts:
stockroom query "SELECT (SELECT count(*) FROM sessions) AS sessions, (SELECT count(*) FROM messages) AS messages, (SELECT count(*) FROM embeddings) AS embeddings"
Non-zero in all three columns means the warehouse is populated and searchable.