<!-- rev: 3252200 -->
# stockroom

> A local, faithful, searchable warehouse of your agentic-coding history.

## Home

### Home

# stockroom

A local, faithful, searchable warehouse of your agentic-coding history.

Stockroom captures prompts, responses, and tool inputs from supported agentic coding harnesses into a single-file [DuckDB](https://duckdb.org/) warehouse with local embeddings so you can use SQL queries when you know exactly what you're looking for, and semantic searches when you don't.

## Agents Can Query the Past

Day-to-day use is expected to go through the `/sr-search` skill & its companions in your agent harness, allowing agents to intelligently assemble queries for you to find what you're looking for...

![Stockroom agentic query of session metadata](img/stockroom-agentic-query.png)

... or even to recall past interactions on their own while working on something for you!

![Stockroom recovering a lost file](img/stockroom-recover.png)

## Dashboard Overview for You, Human!

It includes a dashboard that gives an at-a-glance visual overview of your coding habits and history...

![Stockroom Dashboard](img/stockroom-dashboard-top-light.png){ width="400"}

... along with the ability to drill into and reconstruct past conversations:

![Stockroom Conversation Reconstruction](img/stockroom-dashboard-convo-light.png){ width="400"}

## No Need for Internet or AI

It can even be used **fully offline** and **without AI** once it's set up: after installation, nothing else needs to be downloaded from the internet. Dependencies are fully locked so what you see on GitHub is *exactly* what you get, and you can query the warehouse directly with local `stockroom` and `duckdb` CLIs:

![Querying the Warehouse with DuckDB CLI](img/stockroom-duckdb-query.png)

## Where to Go from Here

- Want it? [Quickstart](user-guide/quickstart.md)
- Want the systems mental model? [Architecture](architecture/index.md)
- Contributing or changing the product? [Contributing](contributing/index.md)

## User Guide

### Index

# Stockroom User Guide

Stockroom keeps a local warehouse of your Cursor and Claude Code history so you (or an agent) can search past work later. You install and initialize once; after that the loop is mostly **ask → answer**, with a quiet nightly job keeping the warehouse fresh.

This page is a light mental model only — not the full [Architecture](../architecture/index.md) tour.

Just want to get it working? Head over to the [Quickstart](quickstart.md) page.

## What you do

### Install Once

Install the plugin from the marketplace and run the `sr-initialize` skill.

That does a one-time dependency sync, selects the correct [PyTorch](troubleshooting/torch.md) for your machine, does the initial load of the warehouse and offers to schedule a nightly warehouse refresh.

---

```mermaid
sequenceDiagram
	actor You
	participant Harness as Cursor / Claude
	participant Init as sr-initialize
	participant Eng as stockroom CLI
	participant WH as warehouse.duckdb

	You->>Harness: Install plugin
	You->>Harness: /sr-initialize
	Harness->>Init: orchestrate
	Init-->Eng: install PyTorch
	Init->>Eng: install stockroom CLI
	Init->>Eng: optional nightly schedule
	Init->>Eng: ingest --full && embed
	Eng->>WH: write sessions / messages / embeddings
	Note over You,WH: Setup done — warehouse searchable
```

---

### Use It

Ask your agents to `/sr-search ...` for things, or notice them searching on their own.

Outside your harnesses, you can use the `stockroom query <SQL>` and `stockroom semantic <text>` CLI commands to dig into the warehouse w/out spending any tokens.

The [Dashboard](dashboard.md) will be there for a visual summary of your work, too.

---


```mermaid
sequenceDiagram
	actor You
	participant Harness as Cursor / Claude
	participant Eng as stockroom CLI
	participant WH as warehouse.duckdb

	alt Agent skills
		You->>Harness: /sr-search "..."
		Harness->>Eng: query and/or semantic
		Eng->>WH: read
		WH-->>You: hits / answer
	else CLI directly
		You->>Eng: stockroom query "<SQL>" <br> stockroom semantic "<text>"
		Eng->>WH: read
		WH-->>You: rows / hits
	else Dashboard
		You->>Harness: /sr-dashboard
		Harness->>Eng: stockroom dashboard
		Eng-->>You: http://localhost:58008/
	end
```

---

### Stay Fresh

If you opted into a nightly warehouse refresh, it will ingest new conversations & generate embeddings for them each night.

You can also use the `stockroom ingest` and `stockroom embed` CLI commands to catch up manually.

---


```mermaid
sequenceDiagram
	actor You
	participant Night as Nightly schedule
	participant Eng as stockroom CLI
	participant WH as warehouse.duckdb

	loop Every night
		Night->>Eng: ingest && embed
		Eng->>WH: catch up
	end

	opt Results feel stale
		You->>Eng: ingest then embed
		Eng->>WH: catch up now
	end
```

---

## Where Next?

- Get it working with [Quickstart](quickstart.md)
- Learn more about the [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) process on [Load the Warehouse](load/index.md)
- Recover history from before your harness kept transcripts with [Backfill Legacy History](load/backfill/index.md)
- Troubleshoot PyTorch at [Troubleshooting > Torch](troubleshooting/torch.md)

### Quickstart

# Quickstart

Get Stockroom installed and running in a few minutes.

## Prerequisites

- [Cursor](https://cursor.com/) or [Claude Code](https://code.claude.com/)
- A POSIX shell and network access for the first-time setup (torch wheel + first ingest)

## Install and initialize

1. Add the [`txrk9-agent-plugins`](https://github.com/Texarkanine/txrk9-agent-plugins) marketplace (that README shows the Cursor and Claude Code UI steps), then install the `stockroom` plugin from it.
2. **Cursor only:** ensure **Include third-party Plugins, Skills, and other configs** (Cursor Settings → Rules, Skills, Subagents) is enabled. Plugin hooks do not register without this until [Cursor's plugin-hooks bug](https://forum.cursor.com/t/plugin-hooks-not-loading-into-cursor-ide/156702) is fixed:

   ![Include third-party Plugins, Skills, and other configs — toggle on](../img/3rd-party-configs.png)

3. Run first-time setup:
	- **Cursor:** `/sr-initialize`
	- **Claude Code:** `/stockroom:sr-initialize`
4. Ask the agent something about past work, or slash-invoke `/sr-search`:

		/sr-search "What was the most-recent time I had to correct an agent's behavior?"

`sr-initialize` checks prerequisites, provisions the per-machine torch wheel, puts `stockroom` on your PATH, offers nightly ingest+embed scheduling, and runs the first full ingest + embed. Re-runs are safe & idempotent: it re-probes and only does what is still missing.

## What to try next

- Prefer **`sr-search`** when you are unsure whether the question is structured SQL or meaning-based recall — [Search](search.md).
- Open the local metrics UI with **`sr-dashboard`** (also launched automatically on session start - [click here!](http://localhost:58008) - when hooks are registered) — [Dashboard](dashboard.md).
- Curious what landed on disk? See [Installed layout](installed-layout.md).
- If something fails *and your agent can't figure it out*, see [Troubleshooting](troubleshooting/index.md) (sections follow this guide's order).

### Installed Layout

# Installed layout

Stockroom installs as a Plugin into your chosen harness, but it has some surprises:

1. it carries a whole python app inside the `sr-search` skill
2. The setup (`sr-initialize` skill) will put some things on your machine:
	- a `stockroom` CLI on your PATH
	- a `warehouse.duckdb` file in `$XDG_DATA_HOME/stockroom` (`~/.local/share/stockroom` by default)
	- (optional) a `crontab` or `launchd` schedule entry for nightly ingest + embed

## Plugin payload

This all lands in wherever your chosen harness stores plugin data:

| Path | Role |
| --- | --- |
| `.cursor-plugin/plugin.json` / `.claude-plugin/plugin.json` | Harness manifests; same skills tree underneath |
| `skills/sr-*` | Skill wrappers (`SKILL.md`) agents invoke |
| `skills/sr-search/` | Python engine (`uv` project, warehouse, dashboard, CLI) |
| `hooks/` | Session-start hooks (dashboard + shim rectify — never ingest/migrate) |

## Runtime home

After `sr-initialize`, machine-local state lives under stockroom home — `$XDG_DATA_HOME/stockroom` or `~/.local/share/stockroom`, overridable with `STOCKROOM_HOME`:

| Path | What it is |
| --- | --- |
| `$STOCKROOM_HOME/warehouse.duckdb` | **DuckDB warehouse:** session/message/tool/embedding tables |
| `$STOCKROOM_HOME/torch-requirements.txt` | **Torch freeze:** hashed requirements so heal can reinstall the same wheel ([Torch](troubleshooting/torch.md)) |
| `$STOCKROOM_HOME/torch-index` | **Torch index sidecar:** https wheel index URL used when the freeze was written ([Torch](troubleshooting/torch.md)) |
| `$XDG_CONFIG_HOME/stockroom/config.toml` (or `~/.config/stockroom/config.toml` by default) | **Optional settings:** distinct from data home; today additive Cursor `ai_tracking_dbs` pins for model enrichment ([Harness Sources](load/sources.md#cursor-sessionsmodels-enrichment)) and `state_vscdb`, the legacy store path for [backfill](load/backfill/cursor-vscdb.md#pointing-at-the-store) |
| `~/.local/bin/stockroom` | **On-path shim:** bakes the correct `uv` invocation to run Stockroom + Torch offline, from the plugin payload directory |

Shim / PATH failures: [Troubleshooting · Installed layout](troubleshooting/index.md#installed-layout).

### Index

# Load the Warehouse

Getting your harness history into the warehouse and keeping it fresh. When search feels stale, catch up:

```bash
stockroom ingest
stockroom embed
```

## In This Section

* **[Ingest & Embed](basic.md)** — what those two commands do, their flags, and how to check the warehouse is populated.
* **[Scheduling](schedule.md)** — the nightly job that runs both so you are not doing it by hand.
* **[Harness Sources](sources.md)** — where each harness's history is read from, and how to point stockroom somewhere else.
* **[Backfill Legacy History](backfill/index.md)** — a one-shot excavation of old stores that ordinary ingest never reads.

### Ingest & Embed

# Ingest & Embed

When search feels stale, catch up:

```bash
stockroom ingest
stockroom embed
```

That is the day-to-day loop. **Ingest** copies harness history into the warehouse; **embed** turns message text into vectors for meaning-based search; a **nightly schedule** runs both so you are not babysitting freshness by hand.

Day-to-day search still goes through the agent (`sr-search` and friends). This page is for when you want to know what those pipelines do — or when you need to re-run them yourself.

## Ingest

Ingest is ETL from agentic coding harness transcript roots into the warehouse under stockroom home (`$STOCKROOM_HOME/warehouse.duckdb` — see [Installed layout](../installed-layout.md)).

```bash
stockroom ingest              # both harnesses, incremental
stockroom ingest --full       # ignore watermarks; re-read everything (still idempotent)
stockroom ingest --verbose    # progress lines (quiet by default)
```

`--harness cursor` or `--harness claude` limits the run to one harness. To read from non-default transcript roots, see [Harness sources](sources.md).

It writes harness-labeled rows into shared tables: `sessions`, `messages`, and `tool_calls`. Prompts and responses are stored whole; tool *inputs* are kept; tool *result* payloads are dropped. Thinking/reasoning blocks the harness keeps separate are not stored. Rows whose source transcripts later vanish are **not** pruned — the warehouse is allowed to outlive its sources.

**Default is incremental.** Stockroom remembers a per-`(harness, source_root)` watermark in `_sync_state` and only reads files past that point. Cursor therefore tracks projects and chats roots independently. Re-runs are cheap and safe.


**Migrations do not Backfill.** Structural migrations do not backfill columns such as `entrypoint` — use `stockroom ingest --full` after a database schema upgrade if you want older rows repopulated from sources (this will be infrequent).

`sr-initialize` runs `stockroom ingest --full` once so you are not waiting for the first nightly job. On years of history that first pass can take many minutes (varying greatly depending on your machine's CPU and disk speed); it prints per-harness session/message/tool_call counts when done.

## Embed

Embed turns non-empty message text into local vectors ([BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5), 384-dim, one row per chunk in `embeddings`). SQL query works without embeddings; **meaning-based recall does not.**

Embedding needs a working PyTorch install in the engine venv. Ingest does not. If embed or semantic search fails citing torch / the environment, fix torch first — [Troubleshooting > Torch](../troubleshooting/torch.md).

**Default is incremental.** Only messages that still lack an embedding for the current model are processed. Re-runs resume cleanly after interruption.

```bash
stockroom embed              # pending messages only
stockroom embed --full       # re-embed all non-empty messages
stockroom embed --verbose
```

There must already be a warehouse (run ingest first). The first embed is the long pole: a large corpus on modest hardware can take hours, and the first run may also download the embedding model once if smoke testing during initialize did not already warm it. Nightly jobs stay cheap because they only catch up.

## Re-run and Check Coverage

If you skipped the first full load (or want to force a full re-read), use the same commands initialize used:

```bash
stockroom ingest --full
stockroom embed
```

Then sanity-check counts:

```bash
stockroom query "SELECT (SELECT count(*) FROM sessions) AS sessions, (SELECT count(*) FROM messages) AS messages, (SELECT count(*) FROM embeddings) AS embeddings"
```

Non-zero in all three columns means the warehouse is populated and searchable.

### Scheduling

# Scheduling

Freshness is a nightly `stockroom ingest && stockroom embed` (incremental, not `--full`) on the platform scheduler — cron on Linux/WSL, launchd on macOS. Native Windows is not supported; use WSL. Output goes to `$STOCKROOM_HOME/logs/nightly.log`.

`sr-initialize` asks before installing the job. You can change the time, skip scheduling entirely, or manage it later:

```bash
stockroom schedule status
stockroom schedule install
stockroom schedule install --time 01:15
stockroom schedule remove
```

`install` is idempotent — it replaces Stockroom's own entry, never duplicates it, and on cron it only touches a comment-delimited block. If `status` warns that the cron daemon is not running, the entry is written but will not fire until you start the daemon.

The optional schedule entry is also called out under [Installed layout](../installed-layout.md). Session-start hooks never ingest or embed — they only heal the shim and launch the dashboard.

### Harness Sources

# Harness Sources

Where [ingest](basic.md#ingest) reads each harness's history from, how to point it somewhere else, and the per-harness quirks worth knowing about.

## Cursor

Cursor has two transcript roots, watermarked independently:

| What | Default | Override |
| --- | --- | --- |
| IDE agent transcripts | `~/.cursor/projects` | `STOCKROOM_CURSOR_ROOT` |
| Agent CLI chats | `~/.cursor/chats` | `STOCKROOM_CURSOR_CHATS_ROOT` |

Overrides are environment variables on the ingest command:

```bash
STOCKROOM_CURSOR_ROOT=/path/to/cursor/projects stockroom ingest
STOCKROOM_CURSOR_CHATS_ROOT=/path/to/cursor/chats stockroom ingest
```

### Best-Effort Parsing

Cursor Agent CLI chats (`~/.cursor/chats/**/store.db`) are parsed best-effort: if a store is locked, corrupt, or its internal blob layout drifts, that session is skipped and the rest of the ingest continues (the chats watermark does not advance past a skipped store, so a later run can retry). Empty or meta-only stores still upsert a session with zero messages. Fixture tests in the repo fail loudly when the known layout changes — operators should not expect a hard ingest failure from layout drift alone.

### Cursor `sessions.models` Enrichment

Cursor has no in-band session model grain. When available, ingest fills `sessions.models` from Cursor's optional `ai-code-tracking.db` sidecar(s).

**Default ingest walks and merges every readable candidate:**

* Linux/Mac paths under `~/.cursor/`
* WSL Windows-home mounts under `/mnt/<drive>/Users/*/.cursor/...`

Optional **additive** pins (if you've got a weird setup) live in XDG config — `$XDG_CONFIG_HOME/stockroom/config.toml` or `~/.config/stockroom/config.toml`:

```toml
[cursor]
ai_tracking_dbs = [
  "/some/funky/path/.cursor/ai-tracking/ai-code-tracking.db",
]
```

Pins are unioned with discovery (not a replacement). Missing pins fail soft.

For tests or one-shots, `STOCKROOM_AI_TRACKING_DB` forces a **single** DB and disables the multi-path walk:

```bash
STOCKROOM_AI_TRACKING_DB=/path/to/ai-code-tracking.db stockroom ingest
```

## Claude Code

Claude Code has one transcript root:

| What | Default | Override |
| --- | --- | --- |
| Project transcripts | `~/.claude/projects` | `STOCKROOM_CLAUDE_ROOT` |

```bash
STOCKROOM_CLAUDE_ROOT=/path/to/claude/projects stockroom ingest
```

Claude transcripts carry model and token usage in-band, per message, so there is no sidecar to discover. Model attribution lands in `messages.model`; `sessions.models` stays empty because Claude's model grain is per-message.

### Index

# Backfill Legacy History

`stockroom backfill` excavates finite legacy stores that ordinary [ingest](../basic.md#ingest) never reads. Run it deliberately, once, if you know you need it. Backfill is not and should not be scheduled.

## The Required Sequence

!!! warning "Run these four steps in this order, every time"

1. **Quit the harness completely.** Not just the window — quit every instance of the application.
2. **`stockroom ingest`** — let ordinary ingest finish first.
3. **`stockroom backfill`**
4. **`stockroom embed`** — backfill never embeds; semantic search needs this after.

**Why quit:** a running harness can tear the read or leave recent writes invisible — backfill may exit 0 having silently missed conversations.

**Why ingest first:** backfill skips what the warehouse already holds. Skipping ingest may pull in live sessions that ingest would have done better, wastes embed work when those rows get superseded, and may corrupt the "written" summary as a measure of legacy-only recovery.

## Sources

Each legacy store is a named **source**. Backfill runs every configured source by default.

| Source (type) | Harness | What It Recovers | Page |
| --- | --- | --- | --- |
| `cursor-vscdb` | `cursor` | Cursor IDE "composer" conversations from before agent transcripts existed | [Cursor `state.vscdb`](cursor-vscdb.md) |

A source needs to be told where its store is; there are no discoverable defaults. A source with no configured path is reported and skipped so the others still run — naming it explicitly with `--source` makes it an error instead.

## Running It

Once the harness is closed and ingest has finished:

```bash
stockroom backfill            # the run
stockroom backfill --dry-run  # rehearse: report only, writes nothing
stockroom backfill --verbose  # all sources w/ progress
```

`--dry-run` does everything a real run does — resolves each source, works out what is already present, reconstructs the rest — then reports what it would have written instead of writing it. It opens the warehouse read-only and takes no write lock, so it will not contend with ingest. It still **reads the legacy store**, so quit the harness first (step 1): an open Cursor can hide WAL-backed conversations from the read, and a dry-run exits 0 having silently missed them. It also needs a warehouse to compare against, and will tell you to run `stockroom ingest` first if there is not one.

## What To Expect

* **Legacy stores are read strictly read-only.** Your harness's own state is never modified.
* **Re-running is safe.** Any session already in the warehouse is skipped, not overwritten — so an interrupted run is resumed simply by running it again.
* **Ingest is never disturbed.** Backfill reuses the same writer ordinary ingest uses, but it does not advance ingest's watermarks. Running it does not change what tonight's job will read.
* **Expect a long embed afterwards.** A large legacy store can substantially grow the message corpus. Start `stockroom embed` when you can leave it alone.

## Fixing A Run

Backfilled rows are exactly identifiable: `sessions.source_path` is the store they came from.

`--force` re-parses sessions **this same source** authored — the escape hatch for when a corrected parse needs to replace an earlier one:

```bash
stockroom backfill --source cursor-vscdb --force
```

It is deliberately narrow. Sessions ordinary ingest authored carry a transcript `source_path`, so they are never matched and never re-parsed, even under `--force`. Backfill cannot overwrite higher-fidelity history with its own reconstruction.

!!! warning "Always embed after a `--force` run"
	Message IDs are positional (`{session_id}#{ordinal}`), so a corrected parse that keeps or drops a different set of messages renumbers everything after the change. Those messages' embeddings are dropped as stale, and only `stockroom embed` puts them back — until it runs, the re-parsed conversations are missing from semantic search.

## Undoing A Run

Delete by the same `source_path`. Count first:

```bash
stockroom query "SELECT count(*) FROM sessions WHERE source_path = '/path/to/store'"
```

`stockroom query` opens the warehouse read-only, so the deletion itself needs a [DuckDB client](../../../advanced/duckdb.md) — with nothing else holding the warehouse open:

```sql
BEGIN;
CREATE TEMP TABLE doomed AS
  SELECT harness, session_id FROM sessions WHERE source_path = '/path/to/store';
DELETE FROM tool_calls  WHERE (harness, session_id) IN (SELECT * FROM doomed);
DELETE FROM messages    WHERE (harness, session_id) IN (SELECT * FROM doomed);
DELETE FROM sessions    WHERE (harness, session_id) IN (SELECT * FROM doomed);
COMMIT;
```

All three tables are needed — the warehouse has no foreign keys, so nothing cascades. Any embeddings those messages owned are pruned by the next `stockroom embed`.

### Cursor Vscdb

# Cursor `state.vscdb`

| Source Name         | Harness  |
|---------------------|----------|
| **`cursor-vscdb`**  | `cursor` |

Before Cursor wrote agent transcripts under `~/.cursor/projects`, IDE conversations ("composers") lived only inside Cursor's own key-value store, `globalStorage/state.vscdb`. Ordinary [ingest](../basic.md#ingest) never reads that file. This source recovers those conversations.

## Pointing At The Store

There is no discoverable default. **Put the path in config** if you expect to re-run:

`$XDG_CONFIG_HOME/stockroom/config.toml` (or `~/.config/stockroom/config.toml` by default):

```toml
[cursor]
state_vscdb = "/mnt/c/Users/you/AppData/Roaming/Cursor/User/globalStorage/state.vscdb"
```

Typical locations:

| Platform | Path |
| --- | --- |
| Linux | `~/.config/Cursor/User/globalStorage/state.vscdb` |
| macOS | `~/Library/Application Support/Cursor/User/globalStorage/state.vscdb` |
| Windows | `%APPDATA%\Cursor\User\globalStorage/state.vscdb` |
| WSL → Windows | `/mnt/c/Users/<you>/AppData/Roaming/Cursor/User/globalStorage/state.vscdb` |

Alternatives for one-offs (same path, different injection):

1. Flag: `stockroom backfill --state-vscdb "/path/to/state.vscdb"`
2. Environment: `STOCKROOM_CURSOR_STATE_VSCDB=/path/to/state.vscdb stockroom backfill`

Priority when more than one is set: flag → environment → config.

## How It Reads

The database is opened **strictly read-only** — your Cursor state is never modified.

**If Cursor is open, backfill can exit 0 and silently miss conversations.** An immutable open (needed on some mounts, including WSL→Windows) cannot see the write-ahead log; quitting Cursor checkpoints that log into the database. A torn read from a live writer announces itself and exits nonzero — the silent miss does not.

The database can be several gigabytes. Backfill reads one conversation at a time, so memory stays flat on a slow mount.

## What Lands In The Warehouse

| Column | Value |
| --- | --- |
| `harness` | `cursor` |
| `session_id` | The composer id |
| `entrypoint` | `ide` — these are genuinely IDE conversations |
| `source_path` | The `state.vscdb` path, which is what makes a run identifiable and reversible |
| `project_id` | Cursor's own workspace id |
| `cwd` | The workspace folder, when Cursor's workspace storage still records it |
| `models` | Models Cursor recorded for the conversation (may be empty) |
| `messages.model` | The model that produced that turn, where Cursor recorded it |

Composer ids share a namespace with agent-transcript session ids; this is how backfill can skip existing sessions, and how `ingest` can take over any recent sessions that you accidentally backfill.

Because `cwd` is recovered, backfilled sessions land in the same workspace grouping as ordinary transcript sessions for the same project. They show up alongside the rest of that project's history in the [dashboard](../../dashboard.md) and in search.

### What Is Left Out

* **Empty drafts.** A composer you opened and never used has nothing to reconstruct; it is counted and skipped.
* **Thinking and reasoning blocks**, as everywhere else in stockroom — they are never stored.
* **Tool results.** Tool *inputs* are kept whole; result payloads are dropped, matching ingest.
* **Timestamps Cursor never recorded.** A composer with no recoverable times keeps NULL `started_at`, which means it is honestly absent from time-windowed dashboard metrics rather than being parked on today's date. Its messages are still fully searchable.

## Reference

**Models.** Backfill keeps both session and per-message models — ordinary Cursor ingest only gets session models from the recent `ai-code-tracking` sidecar, so older history is often blank. `sessions.models` lists every model used, in order. `messages.model` is sparse: Cursor stamps it on choose/change, not every turn. Literal `default` is stored as written.

**Tokens.** Per-turn usage lands on that message; the warehouse's [`session_token_usage`](../../../architecture/warehouse.md#dual-grain-token-usage) VIEW rolls it up with `token_grain = 'message'`. Unmetered turns stay NULL (not zero). Pre-usage conversations get `token_grain = 'none'`, same as ordinary ingest. This is usage metadata only — contemporary Cursor **API tokens** (credentials) remain unavailable from `state.vscdb` and cannot be recovered by backfill.

### Search

# Search

Ask the agent about past work, or slash-invoke a search skill. Prefer **`sr-search`** when you are not sure whether the answer is a structured SQL lookup or meaning-based recall — it routes to the right surface(s) and synthesizes one answer.

After [Quickstart](quickstart.md), the warehouse must already have data ([Load the Warehouse](load/index.md)). Empty results are often a freshness or torch problem, not a bad question — see [Troubleshooting](troubleshooting/index.md).

## How to ask

Natural language is enough (“when did we last fight the dashboard port?”). When you want a specific surface:

| Skill | Cursor | Claude Code |
| --- | --- | --- |
| `sr-search` | `/sr-search` | `/stockroom:sr-search` |
| `sr-query` | `/sr-query` | `/stockroom:sr-query` |
| `sr-semantic` | `/sr-semantic` | `/stockroom:sr-semantic` |

Example:

```text
/sr-search "What was the most-recent time I had to correct an agent's behavior?"
```

Operational flags and recovery tables live in each skill's `SKILL.md` — this page does not duplicate them. To run the engine without another agent turn, see [Advanced → CLI](../advanced/cli.md) (`stockroom query` / `stockroom semantic`).

## The three search skills

### `sr-search`

The friendly default. It classifies the ask, delegates to `sr-query` and/or `sr-semantic`, and presents one answer with supporting session/message ids.

| The ask | What it does |
| --- | --- |
| Exact or structured (ids, filters, counts, joins) | Routes to `sr-query` |
| Meaning-based (describe the topic, not the id) | Routes to `sr-semantic` |
| Broad or ambiguous (both a nameable shape and a concept) | Runs both, then synthesizes |

If one surface comes back empty or thin, it should try the other before concluding the content is absent. Scores from semantic search are never blended with SQL rows — different kinds of evidence.

### `sr-query`

Read-only SQL against the warehouse (`sessions`, `messages`, `tool_calls`, `embeddings`, and views such as `session_token_usage`). Reach for it when the question has a **known shape**: a message or session id, `WHERE` filters, counts, `GROUP BY`, joins, date ranges, token sums.

```bash
stockroom query "SELECT DISTINCT harness FROM sessions ORDER BY harness"
```

For per-conversation token rollups, prefer VIEW `session_token_usage` over hand-rolled `SUM` on `messages` — worked examples live in the `sr-query` skill (agents: `/sr-query` or `/stockroom:sr-query`).

Some common but gnarly queries (full tool rankings, richer token rollups, per-harness skill-use SQL, etc.) have already been figured out for you; see the Advanced [Query cookbook](../advanced/cookbook/index.md).

The surface is read-only by construction — you cannot corrupt the warehouse by querying. Do **not** use SQL `ILIKE` as a substitute for meaning-based recall; that is `sr-semantic`.

### `sr-semantic`

Vector (meaning-based) search. Reach for it when you can describe the content but not name an id — “conversations about flaky tests,” “where did we debug the warehouse deadlock.”

```bash
stockroom semantic "how does the warehouse locking work"
```

Phrase the query as a short description of the content you want. Embedding/search needs a working torch install ([Torch](troubleshooting/torch.md)); ingest and `sr-query` do not. Weak results on *recent* work often mean ingest caught up but embed has not — [Load the Warehouse](load/index.md).

## What to try next

- Prefer **`sr-search`** unless you already know you want pure SQL or pure vectors.
- Browse metrics and past conversations in the UI: [Dashboard](dashboard.md).
- Skill index for all `sr-*` surfaces: [Skill index](skills.md).
- Stuck on empty/thin results or read-only SQL errors? [Troubleshooting · Search](troubleshooting/index.md#search) · [Torch](troubleshooting/torch.md).

### Dashboard

# Dashboard

The stockroom dashboard is a **local, read-only, fully offline** metrics UI over your warehouse — an at-a-glance view of cross-harness agentic-coding history. It does not ingest, embed, or migrate; freshness is owned by [Load the Warehouse](load/index.md).

The long-lived dashboard process caches successful API JSON responses in memory until `warehouse.duckdb` changes on disk (ingest, backfill, or any other writer). A browser refresh against an unchanged warehouse reuses that cache instead of reopening and re-querying DuckDB.

Default URL: [http://localhost:58008](http://localhost:58008/) (also `http://127.0.0.1:58008/`). Every front-end asset is vendored — no CDN or external web requests are made at runtime. You can use it w/out an internet connection!

![Stockroom dashboard — aggregate metrics](../img/stockroom-dashboard-top-light.png)

## `sr-dashboard`

The skill launches (or re-prints) the dashboard URL. Use it when you want the UI, not a SQL or semantic answer.

| Harness | Slash form |
| --- | --- |
| Cursor | `/sr-dashboard` |
| Claude Code | `/stockroom:sr-dashboard` |

```bash
~ $ stockroom dashboard
http://127.0.0.1:58008/
~ $
```

The server is idempotent: if something is already listening on the port, the command still prints the URL and exits cleanly.

Session-start hooks also attempt to launch the dashboard automatically when plugin hooks are registered.

## What you see

### Metrics

Harness filters, time ranges, and Aggregate / Compare views over sessions, messages, projects, daily activity, tool distribution, and related rollups. The warehouse is machine-scoped: the UI stays up across harness sessions and is not stopped when one IDE closes.

The date range runs `Default` · `7d` · `30d` · `90d` · `1y` · `All`. **`Default`** is not the widest setting — it lets each panel keep its own natural window (30 days for most, 14 days and 12 weeks for the activity trends). **`All`** starts at your earliest recorded activity rather than at some fixed epoch, so the axis covers your history and nothing more; the KPI cards read `New` under it, since there is no preceding period to compare against.

### Sessions

The metrics **Sessions** panel shows up to 20 matching conversations (10 newest + `… N more` + 10 oldest when there are more). Click a row to open reconstruction, or `… N more` for the paginated sessions-list view. That list has its own harnesses, time range, and per-page control; filter state lives in the URL.

Both the panel table and the full list include a **Tokens** display. Counts use a compact **K**ilo / **M**ega, etc-style total; hover the `?` for an input / output / cache breakdown when usage is known

List deep-link examples:

```text
http://127.0.0.1:58008/?view=sessions&harness=cursor&per_page=50
http://127.0.0.1:58008/?view=sessions&harness=cursor&harness=claude&per_page=100
http://127.0.0.1:58008/?view=sessions&harness=claude&since=2026-07-01T00:00:00Z&until=2026-08-01T00:00:00Z&page=2&per_page=25
http://127.0.0.1:58008/?view=sessions&per_page=all
```

### Session inspection

Open a conversation from Sessions (or a deep link) to see session metrics and tool/skill composition charts, then read through the whole conversation. Copy a deep-link or export markdown/JSON when in-dashboard rendering is not enough.

![Stockroom dashboard — session conversation view](../img/stockroom-dashboard-convo-light.png)

Session deep-link shape (both query params required):

```text
http://127.0.0.1:58008/?view=session&harness={harness}&session={session_id}
```

Appending an optional message hash scrolls to that message after the conversation loads:

```text
http://127.0.0.1:58008/?view=session&harness={harness}&session={session_id}#msg-{ordinal}
```

## Lifecycle notes

- After a plugin update moves the engine path, harness hooks are what rebake the on-path shim (they know the plugin root). Cursor also runs a non-blocking path-only `shim rectify` on each prompt submit as suspenders when `sessionStart` misses (common on some macOS Cursor setups); full ensure + dashboard launch still belong to session start.
- If a document request cannot be served as real UI, the listener may return a short **recovery HTML** page (instead of bare JSON) with a few starting steps and a link into troubleshooting. Many causes are possible — [Dashboard UI will not load](troubleshooting/index.md#dashboard-ui-will-not-load).
- Port conflicts and auto-start misses: [Troubleshooting > Dashboard](troubleshooting/index.md#dashboard).

For search (not browsing), see [Search](search.md). For every skill at a glance, see [Skill index](skills.md).

### Skill Index

# Skill index

Stockroom's agent-facing surfaces are the `sr-*` skills. Ask in natural language, or slash-invoke when you want a specific one.

Invocation forms differ by harness. After setup, engine calls are always `stockroom <subcommand>` on PATH — see [CLI](../advanced/cli.md). Operational flags and recovery tables live in each skill's `SKILL.md`; this page is only an index.

| Skill | Cursor | Claude Code | What it's for |
| --- | --- | --- | --- |
| [`sr-dashboard`](#sr-dashboard) | `/sr-dashboard` | `/stockroom:sr-dashboard` | Open the local metrics UI |
| [`sr-initialize`](#sr-initialize) | `/sr-initialize` | `/stockroom:sr-initialize` | First-time / heal machine setup |
| [`sr-query`](#sr-query) | `/sr-query` | `/stockroom:sr-query` | Read-only SQL (structured lookups) |
| [`sr-search`](#sr-search) | `/sr-search` | `/stockroom:sr-search` | Default search (routes query / semantic) |
| [`sr-semantic`](#sr-semantic) | `/sr-semantic` | `/stockroom:sr-semantic` | Meaning-based (vector) search |

## `sr-dashboard`

Launches (or re-prints) the local read-only dashboard URL — use when you want the metrics UI, not a search answer.

→ [Dashboard](dashboard.md)

## `sr-initialize`

Walks a machine from “plugin installed” to “warehouse ready”: prerequisites, per-machine torch, on-path `stockroom` shim, optional nightly schedule, first ingest + embed. Idempotent — re-run anytime setup looks broken.

→ [Quickstart](quickstart.md) (get running) · [Load the Warehouse](load/index.md) (ingest / embed / schedule) · [Torch](troubleshooting/torch.md)

## `sr-query`

Read-only SQL against the warehouse for exact or structured lookups — ids, filters, counts, joins. Not for meaning-based recall.

→ [Search — `sr-query`](search.md#sr-query)

## `sr-search`

Friendly default when you are unsure whether the ask is structured SQL or meaning-based recall. Routes to `sr-query` and/or `sr-semantic` and synthesizes one answer.

→ [Search — `sr-search`](search.md#sr-search)

## `sr-semantic`

Vector search for content you can describe but not name exactly. Needs torch / embeddings; not for id filters or counts.

→ [Search — `sr-semantic`](search.md#sr-semantic)

### Index

# Troubleshooting

Human-oriented recovery for common failure modes. Agents already carry short recovery tables in each `SKILL.md`; this page is the longer catalog with UI and environment checks.

When in doubt: re-run **`sr-initialize`**. It re-probes and only does what is still missing.

Sections follow the [user guide](../index.md) order. Each symptom is its own heading so you can deep-link it.

## Quickstart

### Skills missing after marketplace install

Reload the window; confirm the plugin is enabled in the harness plugin UI.

### Cursor hooks / auto-dashboard never fire

Enable **Include third-party Plugins, Skills, and other configs** (see the [Quickstart](../quickstart.md) screenshot). Then reload.

### “Add plugins from folder” rejects this repo

Expected — stockroom is a **plugin**, not a marketplace. Install via [`txrk9-agent-plugins`](https://github.com/Texarkanine/txrk9-agent-plugins).

### Local checkout skills do not load

Contributor localdev wires a Cursor skills mirror after you uninstall the marketplace plugin — see [Preparation](../../contributing/preparation.md). Confirm `make localdev-status` shows the skills mirror, reload the window, and use `HARNESS=cursor make localdev` (Claude uses `claude --plugin-dir` instead of a skills mirror). Marketplace sessionStart hooks are gone after uninstall; the dashboard remains reachable via `stockroom dashboard` / `make local-dashboard`.

## Installed layout

### `stockroom: command not found`

Prefer, in order:

1. **`sr-initialize`** when you can spend an agent turn — it re-probes and only does what is still missing ([Quickstart](../quickstart.md)).
2. **New harness session** — session-start hooks run `shim rectify`, which can create a missing on-path shim or rebake an owned one after a plugin path move.
3. **Last resort: bind the shim yourself** — when the marketplace plugin is already installed and you cannot spend an agent turn.

What lands on disk: [Installed layout](../installed-layout.md).

#### Last resort: bind the shim yourself

The shim is **baked** to one engine directory (`…/skills/sr-search`). Bind the install that is already on disk — not a random git clone. The same recipe lives in your installed plugin under `skills/sr-initialize/SKILL.md`, btw.

1. Find the engine dir (pick the marketplace/plugin tree you actually run — not a contributor checkout unless that is intentional):

	```bash
	find ~/.cursor/plugins ~/.claude/plugins -type d -path '*/skills/sr-search' 2>/dev/null
	```

	If a broken on-path shim still exists, its header names the baked directory: `grep '^# STOCKROOM_APP_DIR=' "$(command -v stockroom)"`.

2. Set `APP_DIR` to that absolute path and choose the owner for this harness (`cursor` or `claude`):

	```bash
	APP_DIR=/absolute/path/to/skills/sr-search
	OWNER=cursor   # or: claude

	PYTHONPATH="$APP_DIR/src" uv run --project "$APP_DIR" --no-sync --no-config \
	python -m stockroom shim install --owner "$OWNER"
	```

3. Confirm:

	```bash
	command -v stockroom
	stockroom --version
	```

If the installer warns that `~/.local/bin` is not on `PATH`, add it and retry the check.

**Ownership:** if install refuses because another owner's shim is alive, read the refusal line. Replacing a live foreign shim needs explicit `--takeover` — prefer `sr-initialize` or consent carefully.

### Shim refuses with a one-line remedy

Follow the remedy printed on stderr. Often that is: open a new session so `shim rectify` can heal, or re-run `sr-initialize`. If you cannot use an agent turn and the remedy is effectively “rebind the launcher,” use the [last-resort bind](#last-resort-bind-the-shim-yourself) under `stockroom: command not found`.

### Engine env cannot import locked deps

Let session-start heal run (`shim rectify` includes ensure-env), run `stockroom shim ensure-env` yourself, or re-run `sr-initialize`.

## Ingest

### Empty or sparse results after first install

Confirm the first ingest + embed finished (`sr-initialize`). Wait for the nightly schedule, or run ingest/embed yourself — [Load the Warehouse](../load/index.md) · [CLI](../../advanced/cli.md).

### Weak semantic results for recent work

Silent staleness is possible: ingest may have new messages that are not embedded yet. Catch up with `stockroom ingest` then `stockroom embed` before concluding the content is absent — [Load the Warehouse](../load/index.md).

### Nightly schedule installed but nothing updates

Check `stockroom schedule status`. If the cron daemon is not running, the entry is written but will not fire (WSL: `sudo service cron start`, or enable systemd). See [Scheduling](../load/schedule.md).

## Search

### SQL errors on write-looking statements

Read surfaces open the warehouse read-only by construction — use ingest/embed for writes ([Search](../search.md) · [CLI](../../advanced/cli.md)).

### Truncated-looking cells in output

Truncation is read-time only; use a higher `--detail` (or refetch a targeted row). Full content remains in the warehouse ([CLI](../../advanced/cli.md)).

### Semantic search returns nothing useful

Confirm the warehouse has embeddings (ingest + embed), then decide structured vs meaning-based — [Search](../search.md). If the error cites torch / the environment, see [Torch](torch.md).

## Dashboard

### Dashboard UI will not load

Something is answering on the dashboard port, but you are not getting the real UI — blank page, odd HTML, or the short in-browser recovery page that says the dashboard could not load this page. There is no single root cause; the steps below are common things worth trying, not a diagnosis of your machine.

#### Things that sometimes contribute

- A **stale dashboard process** left over after a plugin update, still bound to the port but no longer able to serve current static assets.
- An on-path **`stockroom` shim** that is missing, or still baked to an old plugin path. Harness hooks are one way that path gets refreshed (they can see `CURSOR_PLUGIN_ROOT` / `CLAUDE_PLUGIN_ROOT`). If `stockroom` itself is missing or refuses, terminal commands that start with `stockroom …` may not be able to find the correct install on their own.
- Less often for *this* symptom: a broken **engine env** (locked deps missing from the engine `.venv`) that blocks starting a *new* listener. The dashboard does not use Torch, so Torch issues are usually a different problem — [Torch](torch.md) · [Engine env cannot import locked deps](#engine-env-cannot-import-locked-deps).

#### Things to try

1. **From the harness.** Open a **new chat** and run **`/sr-dashboard`** (Claude Code: `/stockroom:sr-dashboard`), then reload [http://127.0.0.1:58008/](http://127.0.0.1:58008/). On Cursor, submitting a short prompt can refresh the on-path shim via before-submit suspenders, but that path does **not** validate the engine env or launch/replace the dashboard — still run `/sr-dashboard` (or rely on session-start) before expecting the UI to recover.
2. **Cursor hooks.** If auto-heal never seems to run, check the third-party plugins setting — [Quickstart](#cursor-hooks--auto-dashboard-never-fire).
3. **If `stockroom` is missing or refuses.** In a chat, try **`/sr-initialize`** (Claude Code: `/stockroom:sr-initialize`) and ask it to restore the on-path shim and get the dashboard serving. That skill can see the plugin tree even when the shim cannot.
4. **If `stockroom --version` already works** but the page is still wrong, `stockroom dashboard --replace` can replace a stale listener. If the shim is not healthy yet, `--replace` often does nothing useful.
5. **Without an agent turn.** Last-resort manual bind — [last-resort bind](#last-resort-bind-the-shim-yourself) under [`stockroom: command not found`](#stockroom-command-not-found).

The in-browser recovery page links here for the longer walkthrough. API clients still see JSON 404 for unknown `/api/*` routes.

### Port 58008 already in use

When `stockroom --version` works but the UI looks stale, try `stockroom dashboard --replace` (or stop the old `stockroom.dashboard` process once, then `/sr-dashboard`) — [Dashboard](../dashboard.md). If you are not sure the shim is healthy, the broader checklist above may be a better starting point.

### Auto-start missing on Cursor

Third-party plugins setting ([Quickstart](#cursor-hooks--auto-dashboard-never-fire) above). If that is already on and nothing seems to auto-heal, the [Dashboard UI will not load](#dashboard-ui-will-not-load) checklist is the next place to look.

## Still stuck

- Ask the agent with `/sr-search` (or Claude `/stockroom:sr-search`) and describe the error text — [Skill index](../skills.md).
- Torch / embeddings / heal soft-fails: [Torch](torch.md).
- Contributors debugging from a checkout: [Preparation](../../contributing/preparation.md) · [Iteration](../../contributing/iteration/index.md).

### Torch

# Torch

[PyTorch](https://pytorch.org/) (“torch”) is the machine-learning library Stockroom uses to turn conversation text into embedding vectors for semantic search. Without it, ingest still works and SQL query still works — meaning-based recall does not.

It is also a pain to ship: the install is large, and the right build depends on *your* machine (CPU-only vs a specific CUDA toolkit - and hey, what's your CPU architecture, by the way?). There is no single wheel that fits every box, and a build from the wrong index will fail in confusing ways.

So Stockroom keeps torch **out of** the locked dependency set on purpose; `uv sync` does not install Torch.

Each machine picks a CPU or CUDA or other wheel once at first install, guided by `sr-initialize`. After a plugin-root move (such as when a new `stockroom` version is published), the engine `.venv` is disposable — locked deps come back from the lockfile. But Torch isn't in the lockfile - it must be restored from a **machine-local hashed freeze** written by that first install.

This ensures that Torch *also* doesn't drift to new versions without your explicit involvement, while still giving you a way to get the Torch that's actually going to work on your machine.

Day-to-day, **`sr-initialize`** owns install → smoke → freeze. Prefer re-running it over hand-editing freeze files.

## Contract

1. **Install** the chosen wheel into the engine venv (`uv pip install torch --no-config --directory <engine> --index <url>`).
2. **Smoke** with `stockroom doctor smoke` (or the form used inside `sr-initialize`).
3. **Freeze** only after smoke succeeds: `stockroom torch freeze --index <url>` (same index URL that passed smoke).

Heal (`ensure_engine_env` → `ensure_torch`) never floating-installs from the index alone. If torch is missing, it runs:

```bash
uv pip install --no-config --directory <engine> --require-hashes -r <stockroom_home>/torch-requirements.txt
```

Indexes embedded in the freeze (from `--emit-index-url` at compile time) resolve pytorch + PyPI deps. The `torch-index` sidecar is for debug / re-freeze input — not heal resolve.

## Failure remedies

### Semantic search or embed fails citing torch / environment

Re-run `sr-initialize` (do not retry the query hoping torch appears).

### Heal soft-fails: no freeze / corrupt freeze

Re-run `sr-initialize` (pick → install → smoke → freeze).

### Heal soft-fails: hash mismatch / yanked wheel

Re-pick a working index, reinstall, smoke, freeze again — do not edit hashes by hand.

### Freeze soft-fails: torch not importable

Install torch into the engine venv first, then freeze.

### Freeze soft-fails: compile error / timeout

Check network / index URL; retry; see `uv pip compile` stderr.

### `stockroom semantic` fails with missing torch

Run `stockroom shim ensure-env` to re-install the frozen Torch into the `stockroom` engine.

### `stockroom doctor smoke` fails with missing torch

If a hashed freeze already exists under stockroom home, the diagnosis recommends `stockroom shim ensure-env` (same heal as above). If there is no freeze yet, follow `sr-initialize` (install → smoke → freeze).

## Advanced

### Overview

# Advanced usage

Escape hatches, niche techniques, and complex operations for power users who already finished `sr-initialize`.

## Audience

You have `stockroom` on `PATH`, a warehouse under stockroom home, and a reason to go outside the normal [User Guide](../user-guide/index.md) path. If you are still bootstrapping or healing a broken install, stay on `sr-initialize` and the User Guide — Advanced is not a second onboarding track.

## What is here

- [CLI](cli.md) — on-path `stockroom` without an agent turn
- [DuckDB](duckdb.md) — open the warehouse file directly
- [Cookbook](cookbook/index.md) — starter SQL for token rollups, tools, and skill use (shared with the `sr-query` skill)

### CLI

# CLI

The on-path `stockroom` command is the torch-safe entrypoint to the engine. After initialization it usually lives at `~/.local/bin/stockroom` (or wherever your user bin is on `PATH`). Use it when you want the same engine the skills call — without an agent turn.

## Prerequisites

- A completed `sr-initialize` so the shim is installed and on `PATH`
- Familiarity with the agent path when you want it back: [Skill index](../user-guide/skills.md)

```bash
which stockroom
stockroom --help
```

If `stockroom` is missing or the shim refuses, recover under [Troubleshooting → Installed layout](../user-guide/troubleshooting/index.md#installed-layout) — do not invent a clone-based `uv` bootstrap from this page.

## Invocation

Skills (`sr-query`, `sr-semantic`, …) orchestrate inside a harness. Out-of-band, you call the shim directly:

```bash
stockroom <subcommand> [flags…]
stockroom <subcommand> --help
```

The shim owns the torch-safe run contract and dispatches into the engine. Prefer it over calling the engine with bare `uv` as an end user.

## Read surfaces

```bash
stockroom query "SELECT DISTINCT harness FROM sessions"
stockroom query --format table --detail full "SELECT message_id, role FROM messages LIMIT 5"
stockroom semantic "flaky dashboard tests" -k 10
```

| Subcommand | Role |
| --- | --- |
| `query` | Read-only SQL against the warehouse |
| `semantic` | Vector (semantic) search |

For schema and search mental model, see [Search](../user-guide/search.md) and [Architecture → Warehouse](../architecture/warehouse.md) / [Embeddings](../architecture/embeddings.md). This page does not fork skill flag tables — use `--help` and the skill `SKILL.md` files for operational recovery detail.

## Output shape

`query` and `semantic` share read-time presentation flags:

- `--format {tsv,json,table}` — default `tsv` (stream-friendly)
- `--detail {compact,snippet,full,raw}` — default `snippet`; truncation is display-only. Prefer `--format json --detail raw` when exact whitespace must match storage.

Full flag semantics: `stockroom query --help` / `stockroom semantic --help`.

!!! tip "The Default was for Agents"

	Semantic search just outputs data - nothing that works as a foreign key into other rows or an SQL query, *except* with `--format json`.

	Additionally, its output is truncated.

`stockroom semantic` was designed for agents to cast a wide net and find something promising, which they'd then do a fuller-detail JSON dump on. For you as a human (who doesn't care about the "context window" of your terminal), you probably always want to use

```bash
stockroom semantic --format json --detail raw "my query..."
```

instead of the default.

## See also

- [DuckDB](duckdb.md) — raw DuckDB CLI when you need SQL outside the presentation layer
- [Architecture](../architecture/index.md) — why the shim and read chokepoint exist
- Missing / broken shim: [Troubleshooting → Installed layout](../user-guide/troubleshooting/index.md#installed-layout)

### DuckDB

# DuckDB

Open the warehouse with the [DuckDB CLI](https://duckdb.org/docs/stable/clients/cli/overview) when you need ad-hoc SQL outside the `stockroom query` presentation layer.

## Warehouse path

The warehouse file is `$STOCKROOM_HOME/warehouse.duckdb`. Default home is `$XDG_DATA_HOME/stockroom` or `~/.local/share/stockroom`. Full path topology: [Installed layout](../user-guide/installed-layout.md).

```bash
echo "${STOCKROOM_HOME:-${XDG_DATA_HOME:-$HOME/.local/share}/stockroom}/warehouse.duckdb"
```

## Open read-only

Always open read-only so you cannot accidentally write:

```bash
duckdb -readonly "${STOCKROOM_HOME:-${XDG_DATA_HOME:-$HOME/.local/share}/stockroom}/warehouse.duckdb"
```

Then run SQL interactively, or pass a one-shot statement:

```bash
duckdb -readonly "${STOCKROOM_HOME:-${XDG_DATA_HOME:-$HOME/.local/share}/stockroom}/warehouse.duckdb" -c "SELECT COUNT(*) FROM sessions"
```

(`-readonly` is a DuckDB CLI flag — see `duckdb --help`.)

## Prefer stockroom query

For routine lookups, prefer `stockroom query`. It already opens the warehouse read-only and applies the project's `--format` / `--detail` conventions. Reach for raw DuckDB when those conventions get in the way (exploratory joins, DuckDB-native tooling, or SQL you do not want wrapped by the presentation layer).

See [CLI](cli.md) for out-of-band `stockroom` invocation.

## Caveats

- **Do not write** through the DuckDB CLI. Schema and ETL go through the engine (`ingest` / `embed` / migrations). Opening without `-readonly` risks corrupting a live warehouse.
- **Locks**: ingest, embed, and other writers may hold locks. If open fails or hangs, stop writers or wait — see [Architecture → Warehouse](../architecture/warehouse.md) for the lock model.
- **No presentation layer**: raw DuckDB will not apply Stockroom's detail/format truncation. Large text columns can flood your terminal.
- **Migrations**: the on-disk schema is versioned by the engine. Do not hand-edit schema in the DuckDB CLI; contributor schema work belongs in [Contributing → Iteration](../contributing/iteration/index.md).

## See also

- [CLI](cli.md) — `stockroom query` / `semantic` with format and detail
- [Architecture → Warehouse](../architecture/warehouse.md) — what is stored and how locks/migrations work
- [Search](../user-guide/search.md) — product how-to for asking questions

### Overview

# Query cookbook

Copy-paste starter SQL for the questions that are awkward to reinvent every time — token rollups, full tool rankings, and per-harness skill use — run through the same `stockroom query` surface your agents use.

## Prerequisites

- `stockroom` on `PATH` (after `sr-initialize`)
- A warehouse you can already query — see [CLI](../cli.md) if you need a refresher

```bash
stockroom query --format table "SELECT count(*) FROM sessions"
```

## Recipes

Each recipe page is the same markdown file the `sr-query` skill ships. Edit the SQL under [`skills/sr-query/references/cookbook/`](https://github.com/Texarkanine/stockroom/tree/main/skills/sr-query/references/cookbook) in the repo; Material "Edit this page" on a recipe hits the docs symlink path.

| Recipe | Use it when you want… |
| --- | --- |
| [Token usage](token-usage.md) | Per-session / by-harness / by-day totals from VIEW `session_token_usage` |
| [Tools](tools.md) | Full `tool_name` ranking (activity window), or one conversation |
| [Skills — Claude](skills-claude.md) | Claude skill × invoker counts (warehouse window or one session) |
| [Skills — Cursor](skills-cursor.md) | Cursor skill × invoker counts (warehouse window or one session) |

## How to run a recipe

Copy a statement from a recipe page, then:

```bash
stockroom query --format table "<paste SQL here>"
```

Prefer `--format table` when you are reading the result yourself; use the default `tsv` (or `--format json`) when piping into other tools.

### Skills Claude

# Skill Use (Claude)

**When:** Claude skill × invoker counts from warehouse SQL (user `<command-name>` + agent `Skill` tool).

```sql
WITH activity AS (
  SELECT s.harness, s.session_id
  FROM sessions s
  WHERE s.harness = 'claude'
    AND NOT s.is_subagent
    AND COALESCE(s.started_at, s.source_mtime) IS NOT NULL
),
user_skills AS (
  SELECT
    regexp_extract(m.text, '<command-name>\s*/([^<\s]+)\s*</command-name>', 1) AS skill,
    'user' AS invoker
  FROM messages m
  JOIN activity a
    ON a.harness = m.harness AND a.session_id = m.session_id
  WHERE m.role = 'user'
    AND m.text LIKE '%<command-name>/%'
    AND NOT starts_with(ltrim(m.text), 'Base directory for this skill:')
),
agent_skills AS (
  SELECT
    trim(json_extract_string(t.tool_input, '$.skill')) AS skill,
    'agent' AS invoker
  FROM tool_calls t
  JOIN activity a
    ON a.harness = t.harness AND a.session_id = t.session_id
  WHERE t.tool_name = 'Skill'
),
events AS (
  SELECT skill, invoker FROM user_skills
  WHERE skill <> ''
    AND skill NOT IN (
      'add-dir',
      'agents',
      'bashes',
      'bug',
      'clear',
      'color',
      'compact',
      'config',
      'context',
      'cost',
      'desktop',
      'diff',
      'effort',
      'exit',
      'export',
      'extra-usage',
      'fast',
      'files',
      'help',
      'hooks',
      'ide',
      'init',
      'insights',
      'install-github-app',
      'keybindings',
      'login',
      'logout',
      'mcp',
      'memory',
      'migrate-installer',
      'mobile',
      'model',
      'output-style',
      'permissions',
      'plan',
      'plugin',
      'privacy-settings',
      'release-notes',
      'reload-plugins',
      'rename',
      'review',
      'rewind',
      'security-review',
      'stats',
      'status',
      'terminal-setup',
      'theme',
      'think',
      'todos',
      'upgrade',
      'usage',
      'vim',
      'voice'
    )
  UNION ALL
  SELECT skill, invoker FROM agent_skills
  WHERE skill IS NOT NULL AND skill <> ''
)
SELECT skill, invoker, count(*) AS uses
FROM events
GROUP BY skill, invoker
ORDER BY uses DESC, skill, invoker
```

Only the first `<command-name>` match per message is taken. Builtin `NOT IN` list tracks `_CLAUDE_BUILTIN_COMMANDS` in `skill_usage.py` (test-pinned).

## One session

For one conversation (dashboard session composition), replace the warehouse-window `activity` CTE with:

```sql
WITH activity AS (
  SELECT 'claude' AS harness, 'YOUR_SESSION_ID' AS session_id
),
```

The rest of the query is unchanged (including the builtin denylist).

### Skills Cursor

# Skill Use (Cursor)

**When:** Cursor skill × invoker counts from warehouse SQL (user `Skill Name:` attach lines + agent `Read` of `…/SKILL.md`).

```sql
WITH activity AS (
  SELECT s.harness, s.session_id
  FROM sessions s
  WHERE s.harness = 'cursor'
    AND NOT s.is_subagent
    AND COALESCE(s.started_at, s.source_mtime) IS NOT NULL
),
user_skills AS (
  SELECT
    unnest(regexp_extract_all(m.text, 'Skill Name:\s*(\S+)', 1)) AS skill,
    'user' AS invoker
  FROM messages m
  JOIN activity a
    ON a.harness = m.harness AND a.session_id = m.session_id
  WHERE m.role = 'user'
    AND m.text LIKE '%<manually_attached_skills>%'
),
agent_skills AS (
  SELECT
    regexp_extract(
      replace(
        coalesce(
          json_extract_string(t.tool_input, '$.path'),
          json_extract_string(t.tool_input, '$.file_path'),
          ''
        ),
        chr(92),
        '/'
      ),
      '.*/([^/]+)/SKILL\.md$',
      1
    ) AS skill,
    'agent' AS invoker
  FROM tool_calls t
  JOIN activity a
    ON a.harness = t.harness AND a.session_id = t.session_id
  WHERE t.tool_name = 'Read'
    AND coalesce(
      json_extract_string(t.tool_input, '$.path'),
      json_extract_string(t.tool_input, '$.file_path')
    ) LIKE '%/SKILL.md'
),
events AS (
  SELECT skill, invoker FROM user_skills WHERE skill <> ''
  UNION ALL
  SELECT skill, invoker FROM agent_skills WHERE skill <> ''
)
SELECT skill, invoker, count(*) AS uses
FROM events
GROUP BY skill, invoker
ORDER BY uses DESC, skill, invoker
```

Agent skill name is the parent directory of `SKILL.md`. User regex is slightly looser than the extractor's line-anchored `^Skill Name:`.

## One session

For one conversation (dashboard session composition), replace the warehouse-window `activity` CTE with:

```sql
WITH activity AS (
  SELECT 'cursor' AS harness, 'YOUR_SESSION_ID' AS session_id
),
```

The rest of the query is unchanged.

### Token Usage

# Token Usage VIEW

**When:** top-N sessions, by-harness totals, or a day rollup from `session_token_usage` (prefer the VIEW over hand-rolled `SUM` on `messages`).

## Top sessions

```sql
SELECT harness, session_id, input_tokens_total, output_tokens_total, token_grain
FROM session_token_usage
ORDER BY input_tokens_total DESC NULLS LAST
LIMIT 20
```

## By harness

```sql
SELECT harness,
       sum(input_tokens_total) AS input_tokens,
       sum(output_tokens_total) AS output_tokens,
       count(*) AS sessions
FROM session_token_usage
GROUP BY harness
ORDER BY input_tokens DESC NULLS LAST
```

## By day

Join sessions for activity time (`COALESCE(started_at, source_mtime)`):

```sql
SELECT date_trunc('day', COALESCE(s.started_at, s.source_mtime)) AS day,
       s.harness,
       sum(t.input_tokens_total) AS input_tokens,
       sum(t.output_tokens_total) AS output_tokens
FROM session_token_usage t
JOIN sessions s
  ON s.harness = t.harness AND s.session_id = t.session_id
WHERE COALESCE(s.started_at, s.source_mtime) IS NOT NULL
GROUP BY 1, 2
ORDER BY 1 DESC, 2
```

`*_total` is `COALESCE(native, from_messages)` — do not also `SUM` message tokens on top. `token_grain` is `'session'` | `'message'` | `'none'`.

### Tools

# Tool Use

**When:** full `tool_name` table (beyond dashboard top-10), optionally by harness, with the same activity window as dashboard metrics.

## All tools in a window

Edit the timestamps (or drop them for all-time):

```sql
SELECT t.tool_name,
       count(*) AS calls
FROM tool_calls t
JOIN sessions s
  ON s.harness = t.harness AND s.session_id = t.session_id
WHERE NOT s.is_subagent
  AND COALESCE(s.started_at, s.source_mtime) IS NOT NULL
  AND COALESCE(s.started_at, s.source_mtime) >= TIMESTAMP '2026-01-01'
  AND COALESCE(s.started_at, s.source_mtime) <  TIMESTAMP '2027-01-01'
GROUP BY t.tool_name
ORDER BY calls DESC, t.tool_name
```

## By harness

```sql
SELECT s.harness,
       t.tool_name,
       count(*) AS calls
FROM tool_calls t
JOIN sessions s
  ON s.harness = t.harness AND s.session_id = t.session_id
WHERE NOT s.is_subagent
  AND COALESCE(s.started_at, s.source_mtime) IS NOT NULL
  AND COALESCE(s.started_at, s.source_mtime) >= TIMESTAMP '2026-01-01'
  AND COALESCE(s.started_at, s.source_mtime) <  TIMESTAMP '2027-01-01'
GROUP BY s.harness, t.tool_name
ORDER BY s.harness, calls DESC, t.tool_name
```

Activity clock is session `COALESCE(started_at, source_mtime)`, not `messages.ts`. Subagents stay excluded unless you want that noise.

## One session

Tool distribution for one conversation (dashboard session composition). Replace the harness / session id:

```sql
SELECT t.tool_name,
       count(*) AS calls
FROM tool_calls t
WHERE t.harness = 'cursor'
  AND t.session_id = 'YOUR_SESSION_ID'
GROUP BY t.tool_name
ORDER BY calls DESC, t.tool_name
```

## Architecture

### Overview

# Architecture

Architecture is the systems atlas for Stockroom: how the pieces fit together, and which unusual constraints you must not remove without understanding them. It is not product how-to — that lives in the [User Guide](../user-guide/index.md). It is not day-to-day contributor loops — that lives in [Contributing](../contributing/index.md). It is not escape-hatch CLI recipes — that lives in [Advanced](../advanced/index.md).

If you already know how to operate the product and need the whole design surface in your head before changing it, start here.

```mermaid
flowchart TB
  subgraph actors["Actors"]
    Human
    Agent
    Hook[Session-start hooks]
    Sched[Nightly schedule]
  end

  subgraph code["Code on PATH / plugin"]
    Shim[stockroom shim]
    Eng[Python Engine]
  end

  subgraph sources["Data sources"]
    Logs[(Harness session logs)]
  end

  subgraph store["Warehouse & vectors"]
    WH[(DuckDB warehouse)]
    Emb[Local embeddings / torch]
  end

  subgraph viz["Data visualization"]
    Dash[Dashboard :58008]
  end

  Human --> Shim
  Agent -->|"sr-* skills"| Shim
  Hook -->|"rectify + dashboard"| Shim
  Sched -->|"ingest + embed"| Shim
  Shim -->|"safely calls"| Eng
  Logs -->|"ingest"| Eng
  Eng -->|"ETL write"| WH
  Eng -->|"embed write"| Emb
  Emb -.->|"vectors live in"| WH
  Eng -->|"open_current"| Dash
  Dash -->|"RO read"| WH
```

Everything that runs Stockroom on a machine goes through the on-path `stockroom` shim into the Python engine under `skills/sr-search/`. Skills, session-start hooks, the nightly schedule, and direct human CLI use are different callers of the same contract.

## Pieces

- **Dual-manifest plugin** — Cursor and Claude Code each have a manifest; both point at one shared `skills/` tree. The committed layout is the install layout.
- **Skills** — `sr-*` agent procedures. Sibling skills have no Python of their own; they invoke `stockroom <subcommand>`.
- **Shim** — generated `~/.local/bin/stockroom`. Owns engine-dir resolution, `PYTHONPATH`, and torch-safe uv flags. Baked-only: succeed correctly or refuse with a one-line remedy. See [The stockroom shim](packaging.md#the-stockroom-shim) and [Heal](packaging.md#heal).
- **Engine** — locked uv project under `skills/sr-search/` (`src/stockroom/`, migrations, tests). Run-in-place; not an installed Python package.
- **Warehouse** — single-file DuckDB under stockroom home. Rebuildable ETL from harness session records.
- **Embeddings** — local `sentence-transformers` vectors; torch is provisioned per-machine and held out of the dependency lock.
- **Hooks** — session-start commands that rectify the shim and launch the dashboard. Fire-and-forget; never the ingest path.
- **Schedule** — nightly `stockroom ingest && stockroom embed` on the platform scheduler.
- **Dashboard** — local offline metrics UI on port 58008; opens the warehouse without migrating.

## Change surfaces

| If you change… | Read first |
| --- | --- |
| Plugin manifests, skill layout, engine packaging, uv lock, torch provisioning, the shim, or heal | [Packaging](packaging.md) — especially [shim](packaging.md#the-stockroom-shim) and [heal](packaging.md#heal) |
| Session-start hooks, nightly schedule, or dashboard process lifecycle | [Lifecycle](lifecycle.md) |
| Schema, ingest parsers/writer, warehouse open paths, or identity/provenance | [Warehouse](warehouse.md) |
| One-shot excavation of a harness's legacy store, or adding a backfill source | [Backfill](backfill.md) |
| Embedding model, VSS/HNSW, semantic search, or how skills route over query/semantic | [Embeddings](embeddings.md) |
| Human install/heal *recipes*, torch troubleshooting steps | [User Guide](../user-guide/index.md) |
| Make / localdev / iteration loops | [Contributing](../contributing/index.md) |
| Out-of-band `stockroom` CLI | [Advanced → CLI](../advanced/cli.md) |
| Raw DuckDB CLI against the warehouse | [Advanced → DuckDB](../advanced/duckdb.md) |

## Related surfaces

- **Agents** use skill procedures plus the compact [`system-model.md`](https://github.com/Texarkanine/stockroom/blob/main/skills/sr-search/references/system-model.md) that ships with the plugin.
- **Maintainers** in a checkout also have `memory-bank/systemPatterns.md` — related themes, different audience (implementation briefing). Do not collapse Architecture and that briefing into one SSOT.
- **Licensing** is layered (AGPL base with a PPL-S carveout for prompt-shaped skill payload). Detail lives in [Contributing → Licensing](../contributing/licensing.md).

### Packaging

# Packaging

How Stockroom code arrives on a machine and how it is invoked. Procedures for install, heal steps, and day-to-day Make loops live elsewhere; this page is the shape of the design.

## Entrypoint

The on-path `stockroom` command and the repair surface that keeps it honest. Other docs name these constantly — start here.

### The stockroom shim

The generated on-path command (`~/.local/bin/stockroom`) owns the entire invocation contract: engine-directory resolution, `PYTHONPATH`, and torch-safe uv flags. Every caller that runs Stockroom — skills, session-start hooks, the nightly schedule, humans on the CLI — goes through this entrypoint.

Wrapper skills say only `stockroom <subcommand>`. They carry **no fallback incantation**. If `stockroom` is missing from `PATH`, the machine is not initialized; the correct next action is `sr-initialize` (or the contributor equivalent in [Preparation](../contributing/preparation.md)).

The shim is **baked-only and succeed-or-refuse**. A baked `APP_DIR` is written into the script at install/rectify time. At runtime the shim does not scan, rank, or guess an engine location — it either execs that bake through the torch-safe contract or refuses with a one-line remedy.

Ownership is explicit (`STOCKROOM_OWNER` in the shim header). Only the owner may rewrite an existing shim; foreign takes over require explicit flags (contributor Make paths — not Architecture recipes).

Three writers share one tested surface:

| Writer | Role |
| --- | --- |
| `shim install` | First bake / guarded rewrite (`sr-initialize`, `make shim`) |
| `shim rectify` | Hook-safe heal: create if absent, rebake when owned and drifted, ensure engine env — never touch a foreign owner |
| `shim ensure-env` | Env-only subset when the shim file itself is already correct |

Because the engine is run-in-place (not an installed package), making `stockroom` importable is part of this contract — not something skills invent ad hoc. Layout and lock details: [Engine inside sr-search](#engine-inside-sr-search), [Locked uv project](#locked-uv-project).

### Heal

**Heal** is the repair surface for the packaging/runtime contract: restore a correct on-path shim bake, a usable engine uv environment, and (when needed) torch from the per-machine freeze — without treating every failure as a full re-onboard from scratch.

What heal restores:

- **Shim bake** — missing on-path file, or owned shim whose `APP_DIR` drifted after a plugin path move
- **Engine env** — locked deps present via torch-safe inexact sync (never an exact sync that would strip torch)
- **Torch** — reinstall from the hashed freeze under stockroom home when the env cannot import the accepted stack

Who triggers it:

- **Session-start hooks** — `shim rectify` on every session (fire-and-forget; see [Lifecycle](lifecycle.md#session-start-hooks))
- **`sr-initialize`** — full machine setup *and* the intentional re-run when setup looks broken
- **Explicit `shim ensure-env`** — when the shim file is fine but the env/torch side is not

What heal is **not**:

- Not ingest, embed, or migrate
- Not a second copy of User Guide troubleshooting steps — recipes live in [Quickstart](../user-guide/quickstart.md), [Torch troubleshooting](../user-guide/troubleshooting/torch.md), and [Preparation](../contributing/preparation.md)
- Not “retry the query” — a missing torch or stale bake is an environment problem

Heal and [torch held out of the lock](#torch-held-out-of-the-lock) are one story: the freeze exists so heal can replay a machine-specific wheel that the lockfile cannot name.

## Plugin layout

How the plugin is shaped on disk: dual manifests, where the engine lives, the hermetic lock, and the torch exception to that lock.

### Dual-manifest plugin

Stockroom is one plugin with two manifests: `.cursor-plugin/plugin.json` and `.claude-plugin/plugin.json`, both over a shared `skills/` tree. The committed repository layout **is** the install layout — what the plugin manager copies to disk is exactly what runs. There is no separate build step that produces the plugin payload.

### Engine inside sr-search

The full Python engine lives under `skills/sr-search/`: `pyproject.toml`, `uv.lock`, `src/stockroom/`, migrations, and tests. Sibling `sr-*` skills have no Python of their own. Skill prose and engine behavior must stay in sync because they ship as one unit — there is no separate download of engine code at runtime.

### Locked uv project

The engine is a locked [uv](https://docs.astral.sh/uv/) project with `[tool.uv] package = false`: run-in-place, no console-script entry points, not installed onto `sys.path` by packaging. Dependencies are pinned through a committed `uv.lock`. Lock hermetically (`uv lock --no-config`); what GitHub shows is what the machine is expected to run.

### Torch held out of the lock

Torch is required for embedding (writing vectors and encoding semantic-search queries) but is **held out of the dependency lock**. The correct torch build is a per-machine choice (CPU, CUDA generation, MPS) that no single lockfile can make. It is provisioned out of band, smoke-tested, and frozen under stockroom home; [heal](#heal) reinstalls from that freeze.

After torch is installed, runs must not do an *exact* dependency sync: an exact sync removes anything not in the lock, including the provisioned torch. Torch-safe paths use inexact sync / `--no-sync` as appropriate.

A missing torch is an environment problem, never a query-phrasing problem. Operational steps: [User Guide → Troubleshooting → Torch](../user-guide/troubleshooting/torch.md); contributor sync loops: [Contributing → Iteration](../contributing/iteration/index.md).

## Related procedures

- First-time setup and heal recipes: [User Guide → Quickstart](../user-guide/quickstart.md), [Installed layout](../user-guide/installed-layout.md), [Torch troubleshooting](../user-guide/troubleshooting/torch.md)
- Contributor checkout wiring and Make loops: [Contributing → Preparation](../contributing/preparation.md), [Iteration](../contributing/iteration/index.md)
- Licensing carveouts: [Contributing → Licensing](../contributing/licensing.md)
- Agent-facing compact form of these doctrines: [`system-model.md`](https://github.com/Texarkanine/stockroom/blob/main/skills/sr-search/references/system-model.md)

### Lifecycle

# Lifecycle

When Stockroom work runs on a live machine: hooks, the nightly schedule, and the dashboard process. This is *when* things fire — not how to install or how to write SQL.

## Session-start hooks

Constraints on the hook, and what it actually runs. Heavy ETL does not belong here — see [Scheduled ingest and embed](#scheduled-ingest-and-embed).

### Hook doctrine

Harness session-start hooks are designed to be:

- **Fire-and-forget** — stdout/stderr discarded; failures must not block the session.
- **Idempotent** — safe to run on every session start; repeated runs are no-ops when already healthy.
- **Concurrent** — multiple harness sessions may start at once; hooks must not assume exclusive ownership of the machine.
- **Fault-tolerant** — wrapped so a bad heal or busy port cannot take down session start (`|| true` / equivalent).

Hooks are short-budget work. Session-start hook commands carry an explicit timeout (hundreds of seconds, not “as long as ETL needs”). Anything that can run for minutes does not belong on session start.

### Session start

On session start, Stockroom does two things through the shim:

1. **`shim rectify`** — heal the on-path shim and ensure the engine environment (see [Heal](packaging.md#heal) and [The stockroom shim](packaging.md#the-stockroom-shim)).
2. **`stockroom dashboard`** — launch (or re-print) the local dashboard URL.

Session start does **not** ingest, embed, or migrate as its primary work. Those are heavier, longer, and already owned by the schedule and explicit CLI/skill paths. Putting them on the hook would fight timeout limits and turn every new chat into an inelegant ETL termination race.

### Cursor beforeSubmitPrompt suspenders

On Cursor, `sessionStart` has been observed to miss on some macOS setups. Cursor also registers a trimmed **`beforeSubmitPrompt`** hook that must never block prompt send: it emits `{"continue":true}` immediately, then backgrounds a **path-only** `shim rectify --path-only` (create/rebake the on-path shim; **skip** `ensure-env`). Full ensure + dashboard launch stay on `sessionStart` only. Claude Code does not get this suspenders event.

## Scheduled ingest and embed

Freshness is a nightly `stockroom ingest && stockroom embed` (incremental) on the platform scheduler — cron on Linux/WSL, launchd on macOS. The job invokes the shim by name; it does not embed a raw engine path. Output lands under stockroom home logs.

`sr-initialize` offers to install the job once. Manual catch-up remains available via CLI when results feel stale — see [User Guide → Load the Warehouse](../user-guide/load/index.md).

Backfill of *legacy* stores is deliberately not on this schedule, or any other — see [Backfill](backfill.md).

## Dashboard launch

The dashboard is a **local, read-only, fully offline** metrics UI (default port 58008). Front-end assets are vendored — no CDN or external web requests at runtime. It does not ingest, embed, or migrate; warehouse content freshness is owned by ingest/embed/backfill. The long-lived listener caches API JSON (bounded LRU) until the warehouse file's on-disk fingerprint changes, so a refresh does not re-query when nothing has been written.

Session-start hooks attempt to launch it automatically. The CLI is idempotent: if something already listens on the port, the command still prints the URL and exits cleanly. The process uses a torch-safe engine env (same shim contract as other subcommands) and opens the warehouse through `open_current()` so a UI process never becomes the migrator — see [Warehouse](warehouse.md#concurrency-and-open-paths).

When the listener cannot serve real static UI (missing `index.html`, unknown document path, stale assets after a plugin move), it returns an in-memory **recovery HTML** page: short harness-first rundown (new chat / prompt → `sr-initialize` if the shim stays dead) plus a link to the user-guide troubleshooting section — not bare JSON `{"error":"not found"}`, and not circular `stockroom …` CLI while PATH is broken. API and session-miss responses stay JSON.

## Rendered-out artifacts

Shim, harness hooks (`hooks/*.json`), and scheduler entries are each owned by one module with structural idempotency. No rendered artifact carries a raw engine path — callers invoke `stockroom` by name so plugin moves are healed by rectify rather than by rewriting every consumer.

Harness hooks are not the same JSON shape or event: Cursor uses flat `sessionStart` / `beforeSubmitPrompt` commands; Claude Code uses nested `SessionStart` / `hooks[]` / `type: "command"`. Do not copy one harness's structure into the other.

## Related procedures

- Operating the dashboard: [User Guide → Dashboard](../user-guide/dashboard.md)
- Ingest, embed, and scheduling how-to: [User Guide → Load the Warehouse](../user-guide/load/index.md)
- Contributor schedule / hook iteration: [Contributing → Iteration](../contributing/iteration/index.md)

### Warehouse

# Warehouse

The DuckDB warehouse as rebuildable ETL output, and the doctrines that shape what it stores and how processes open it.

## What the warehouse is

Rebuildable projection of harness history — not the system of record — how rows get in, and read surfaces that cannot write by construction.

### Rebuildable ETL

The warehouse is a single-file DuckDB database under stockroom home (`$XDG_DATA_HOME/stockroom` or `~/.local/share/stockroom`, overridable via `STOCKROOM_HOME`). It is rebuildable ETL from the harnesses' own session records — never the system of record. Ingestion re-derives rows from those sources; the warehouse is the queryable projection.

### Ingest pipeline

Per-harness parsers emit shared dataclasses; the writer is the only SQL touchpoint. Default ingest is incremental (per-harness watermarks in `_sync_state`). The warehouse is allowed to **outlive its sources**: rows whose transcripts later vanish are never pruned. Observation-time fields (for example `messages.first_seen_at`) are not rebuildable from sources alone — that is why “delete and re-ingest” is not a free reset of every column.

Cursor has two discovery roots (IDE `agent-transcripts` and Agent CLI `store.db` chats) with independent watermarks; on `session_id` collision the chats store wins. CLI parsing is fail-soft: a locked/corrupt `store.db` or unrecognized root-blob layout skips that session without aborting the batch (and without advancing the chats watermark), while committed fixture tests fail loudly when the known layout drifts.

### Read-only by construction

The read surfaces (`query`, `semantic`) open the warehouse read-only at the connection level — DuckDB itself rejects writes through them. “You cannot corrupt anything by querying” is a property of the connection mode, not of good manners.

## What we store

Fidelity doctrines: which fields are kept, whole text at rest, uniform identity, honest workspace paths, and UTC timestamps.

### Kept fields

Shared tables are `sessions`, `messages`, `tool_calls`, `embeddings`, and `_sync_state`. Prompts and responses are stored whole; tool *inputs* are kept; tool *result* payloads are dropped. Thinking/reasoning blocks the harness keeps separate are not stored. There is no raw mirror layer beside the typed model.

### No truncation at rest

Kept fields are stored whole. Truncation is a **read-time** display bound so one fat column does not flood a context window. Elision markers report how much was withheld; the full content remains in the warehouse for a targeted re-fetch. Both read surfaces print through one render chokepoint (`--detail` / `--format`); see [Embeddings](embeddings.md#read-time-rendering) for the search-side note and [Advanced → CLI](../advanced/cli.md) for flags.

### Harness-labeled identity

Every row carries a `harness` column. Columns mean one thing independent of harness — extraction may differ; meaning must not. Identity is uniform: `(harness, session_id)` for sessions, `message_id = '{session_id}#{ordinal}'` for messages. Native harness identifiers are demoted to `source_*` provenance — kept for traceability, never used as join keys, because they exist at different grains and formats per harness. A value that only exists for one harness is `NULL` for the other, never fabricated.

`sessions.entrypoint` is nullable surface provenance within a harness, e.g.

* Claude Code text UI vs Claude Code desktop app?
* Cursor IDE vs `agent` CLI?

Values are taken from source data verbatim if present, synthesized based on our knowledge of harness' data provenance if not.

### Workspace identity

`sessions.project_id` is the harness slug verbatim; `sessions.cwd` is best-effort real path, `NULL` when unknown. Path candidates are accepted only when encoding them for that harness reproduces the slug — verify, don't invert. Guessing a workspace from a slug without that check invents false identity.

`sessions.workspace_key` is a nullable cross-harness rollup key derived at ingest (per-harness strategies in `stockroom.ingest.paths.workspace_key_for`). Same machine + same absolute `cwd` ⇒ same key when both sides can derive it; different on-disk paths stay different keys; underivable inputs stay `NULL`. Chart Sessions by Project and SQL `GROUP BY workspace_key` share that key — `project_id` is never rewritten for merge convenience.

### Dual-grain token usage

Token fields follow the same one-meaning-per-column rule as models (`messages.model` vs `sessions.models`):

- **Message grain** — `messages.input_tokens` / `output_tokens` / `cache_creation_tokens` / `cache_read_tokens`. Claude fills these from per-assistant-message usage; Cursor leaves them `NULL`.
- **Session grain** — the same four names on `sessions`, for harnesses that report conversation-level totals only. Claude and Cursor leave them `NULL` today. Ingest never invents session totals from message sums, and never invents per-message splits from session totals.

The read surface for conversation rollups is VIEW `session_token_usage`: `*_from_messages` (SUM of message columns), `*_native` (session columns), `*_total` (`COALESCE(native, from_messages)`), and `token_grain` (`session` | `message` | `none`). Totals are a warehouse rollup of reported fields, not a vendor invoice. Writers target base tables only; query the VIEW for session spend/usage. Message-level detail stays on `messages`.

### UTC timestamps

DuckDB `TIMESTAMP` is timezone-naive; Stockroom's contract is that every persisted value is **UTC wall clock**. Clients that display times own timezone rendering.

## How we open and evolve it

Connection chokepoints, concurrency, and forward-only schema migrations.

### Concurrency and open paths

Every consumer reaches DuckDB through a warehouse chokepoint:

- **`open()`** — path resolution, lazy migration, VSS load. Writers take an exclusive coordination flock for the connection's lifetime; readers open read-only and back off to a typed busy error when the file stays locked.
- **`open_current()`** — the dashboard exception: read-only, **no migrate**, typed stale/busy errors. A UI process must not become the migrator mid-browse.

Coordination uses `fcntl.flock` on a sidecar lock file; data integrity uses DuckDB's own file lock. Those are two layers with different jobs — do not collapse them into “just open the file.”

### Migrations

Migrations are numbered forward-only SQL under the engine's `migrations/` tree; `schema_version` is runner-owned. Schema changes go through the `open()` chokepoint — Architecture does not list DDL here.

## Related procedures

- Operating ingest/embed/schedule: [User Guide → Load the Warehouse](../user-guide/load/index.md)
- Escape-hatch SQL: [Advanced → DuckDB](../advanced/duckdb.md)
- Contributor engine/schema work: [Contributing → Iteration](../contributing/iteration/index.md)

### Backfill

# Backfill

Backfill is one-shot excavation of a harness's *legacy* store — history that predates the transcript roots [ingest](warehouse.md) reads, is finite, and does not grow. It is a sibling of the ingest pipeline, not a mode of it, and the separation is structural rather than conventional.

Operator how-to lives in [User Guide → Backfill Legacy History](../user-guide/load/backfill/index.md). This page is why it is shaped the way it is.

## Invariants

- Never on nightly or hooks; import-edge and schedule guards assert the absence
- Writes only through `ingest.writer`; never touches `_sync_state`
- Skip set = warehouse snapshot; `--force` only matches this adapter's `source_path`
- Tokens at source grain; `source_mtime` stays NULL for a shared store

## Not On Any Automatic Path

`stockroom backfill` is manually run, always. No session-start hook invokes it, the scheduler entry stays exactly `stockroom ingest && stockroom embed`, and nothing reachable from the nightly path imports the `backfill` package.

That last one is the load-bearing guarantee, and it is an *absence*, so it is asserted rather than assumed: guard tests fail if the rendered schedule payload gains a `backfill` token, or if the `stockroom.ingest` package acquires an import edge onto `stockroom.backfill`. Legacy-store reads stay one deliberate command away from the thing that runs unattended every night.

```mermaid
flowchart LR
    subgraph nightly["Nightly (automatic)"]
        Sched[schedule] --> Ing[ingest] --> Emb[embed]
    end
    subgraph oneshot["Backfill (manual)"]
        Operator([Operator]) --> BF[backfill]
    end
    Roots[(transcript roots)] --> Ing
    Legacy[(legacy store)] --> BF
    Ing --> W[[ingest.writer]]
    BF --> W
    W --> WH[(warehouse)]
    Ing --> SS[_sync_state watermarks]
    BF -.->|never| SS
```

## Orchestrator Over Adapters

Backfill is cross-harness by construction, even though exactly one legacy store is known today. The package mirrors how ingest is an orchestrator plus per-harness parsers:

| Module | Role |
| --- | --- |
| `backfill/__init__.py` | Source registry, skip set, write loop, per-source summary |
| `backfill/cursor_vscdb.py` | Today's only adapter |
| `backfill/__main__.py` | CLI |

Adapters own their source format and nothing else — they read their store, resolve their own path from flag/env/config, and yield the same `NormalizedSession` objects ingest parsers yield. **The orchestrator owns every warehouse interaction**; no adapter is handed a connection, and a test asserts a run still writes when the adapter never sees one. A second legacy store is a new file plus a registry entry, not orchestrator surgery.

The contract and how to add one: [Contributing → Backfill Adapters](../contributing/iteration/backfill-adapters.md).

## Reuses The Writer, Never The Watermark

Backfill writes through `ingest.writer.write_session` — the same single SQL touchpoint for session persistence that ingest uses — so backfilled rows are shaped, keyed, and de-duplicated identically. `workspace_key` in particular is derived by the writer, which is what lets a backfilled session converge with a transcript-authored session for the same working directory.

It deliberately never calls `update_watermark`. A run leaves `_sync_state` exactly as it found it, so excavating history does not change what tonight's incremental ingest will read.

That isolation makes the [required operating sequence](../user-guide/load/backfill/index.md#the-required-sequence) merely a cost concern, not correctness issue. The skip set is a snapshot of what the warehouse currently holds, so backfill before ingest reconstructs conversations whose transcripts are already on disk. Nothing is lost — the next ingest still selects them and supersedes the reconstruction — but the overlap is paid twice in embedding work and corrupts the run summary as a measurement. Ingest first; the skip set only grows.

A dry run opens through `warehouse.open_current()` instead: read-only, never migrating, and off the single-writer flock entirely. Rehearsing a backfill must not be able to create a warehouse, move its schema, or delay a running ingest — so a missing or behind-head warehouse is a typed refusal. Get your (ware)house in order before a backfill.

## Never Clobbering What It Did Not Write

The writer persists idempotently by delete-then-insert on `(harness, session_id)`, which makes a wrong skip set actively destructive rather than merely wasteful. Two things contain that:

**Provenance is exact.** Backfilled sessions carry sufficient identifiers to uniquely and exclusively identify their logical content. If some other source has been populating the same session's data... well, that data does, at least, belong in that session.

**The default skip set is everything already present.** Any `session_id` already in the warehouse for the adapter's harness is skipped before parsing, not after — adapters enumerate candidate ids cheaply first, so the expensive parse only runs on what will actually be written. A test asserts a skipped row is byte-identical afterwards.

`--force` narrows that skip set to sessions whose `source_path` is *this adapter's own store*, so a corrected parse can replace its own earlier output without hand-written SQL. `ingest`-authored rows carry a transcript `source_path` and are therefore unmatchable even under force.

Changing the keep predicate under `--force` renumbers positional `message_id`s and invalidates embeddings — see [User Guide → Fixing A Run](../user-guide/load/backfill/index.md#fixing-a-run). You'd only need this if you were changing how an existing backfill worked and needed to repair data in a warehouse that had already been backfilled w/ the older code path.

## Reading Foreign Stores Safely

A legacy store belongs to the harness, which may be running. Adapters open it strictly read-only and are expected to fail soft: an absent, unreadable, or actively-written store yields a one-line message and a nonzero exit. One unparseable record does not abort a run, and one failed source does not stop the others.

## Grain And Honesty

**Tokens are stored at the grain the source reports.** A per-turn count lands on *that* message, and session `*_tokens` stay NULL so `session_token_usage` reports `token_grain = 'message'`. Summing message counts into the session columns is explicitly forbidden by migration `0007`, and would additionally make the rollup view mislabel the grain as `'session'`.

That said, if a source only *has* session-grain token data, it will be written as such.

**`source_mtime` stays NULL for a shared store.** The column means "the mtime of *this conversation's* source transcript." If a legacy store is one file for thousands of conversations, its mtime is approximately the last time it was updated and will be actively incorrect for the majority of sessions. In such cases, `source_mtime` is left `NULL`.

When `source_mtime` is absent, the writer seeds `messages.first_seen_at` from time of the backfill run. That field means "when stockroom first observed this message" and is [not rebuildable from sources](warehouse.md#ingest-pipeline).

### Embeddings

# Embeddings

Local vectors, the index they live in, and how search surfaces split power from judgement.

## Local vectors

The model, the index path, and why semantic recall can lag ingest.

### Model and dimensions

Embeddings use [`sentence-transformers`](https://www.sbert.net/) with [`BAAI/bge-small-en-v1.5`](https://huggingface.co/BAAI/bge-small-en-v1.5) at **384 dimensions**. The model is local after first fetch: once provisioned, semantic search does not need the network.

BGE-small is an *asymmetric* retrieval model: stored passages are embedded with **no** prefix; only incoming queries get the engine's query instruction prefix. Mixing a prefixed query against wrongly-prefixed passages, or thresholding on an absolute cosine score as if it were a universal quality meter, will mislead you — scores are meaningful only **relative to each other within one query**.

Torch is required to encode; it is held out of the lock for machine-specific builds — see [Packaging](packaging.md#torch-held-out-of-the-lock).

### VSS and HNSW

Vectors live in the warehouse and are queried through DuckDB's VSS extension over an HNSW index (migration-owned). Semantic search embeds the query with the same local model, runs cosine KNN with over-fetch, then dedups multi-chunk hits back to one row per owner message (max-sim at owner grain).

Long messages may produce several chunk vectors; that is expected. SQL `query` does not need embeddings; meaning-based recall does.

### Ingest lag and staleness

Ingest and embed are separate passes. Embed is heavier (torch + real compute) and is allowed to lag. Recent sessions may exist in SQL but be invisible to semantic search until embedded — the **silent staleness** failure mode. Weak semantic results for recent work warrant a coverage check before concluding the content is absent.

When ingest rewrite-replaces a session, it invalidates embeddings only for message ids that were removed or whose text changed. Append-only growth and unchanged history keep their vectors, so embed lag after a successful ingest leaves a small hole rather than emptying the session's semantic coverage.

`stockroom embed` encodes pending chunks in **cross-message batches** (throughput only — same model, chunking, and float32-near vectors as single-chunk encode) and, after the normal sweep, deletes **orphaned** `owner_table='messages'` embedding rows whose `(harness, owner_id)` no longer matches a `messages` row (any `embed_model`). That heals warehouses left inconsistent by an interrupted ingest rewrite without a separate operator chore.

Nightly schedule runs both incrementally; manual catch-up is the same pair of commands — see [User Guide → Load the Warehouse](../user-guide/load/index.md).

## Search-surface split

Python modules are raw power surfaces (`query`, `semantic`, and friends). Each `sr-*` skill holds LLM-safe usage guidance. [`sr-search`](https://github.com/Texarkanine/stockroom/blob/main/skills/sr-search/SKILL.md) is a **judgement router** over `sr-query` / `sr-semantic` — there is no `stockroom.search` fusion module that pretends one ranking can replace that judgement.

The same split shows up elsewhere: `stockroom.doctor` facts vs `sr-initialize` judgement; `stockroom.schedule` mechanism vs skill consent. Engine power stays boring and composable; skills own when to call which surface.

## Read-time rendering

Both read surfaces print only through one render chokepoint. `--detail` (compact / snippet / full / raw) is orthogonal to `--format` (tsv default; json / table opt-in). Truncation markers are display bounds, not warehouse mutation — see [Warehouse](warehouse.md#no-truncation-at-rest).

## Related procedures

- Day-to-day search: [User Guide → Search](../user-guide/search.md), [Skill index](../user-guide/skills.md)
- Torch / env heal: [User Guide → Troubleshooting → Torch](../user-guide/troubleshooting/torch.md)
- CLI flags: [Advanced → CLI](../advanced/cli.md)

## Contributing

### Index

# Contributing to Stockroom

If you want to contribute to Stockroom, you've probably used it.

This means that its Rube Goldberg machine of a warehouse ETL process is probably installed on your machine.

You might want to [switch off of that](preparation.md#rip-it-out) to run your local checkout, in order to develop and validate your changes...

... and then be able to [switch back to the released version](preparation.md#done-developing) without data loss, once your changes are incorporated.

That round-trip lives in [Preparation](preparation.md).

Day-to-day once you are wired — how to hack on each part of Stockroom, where it lives, how to validate, etc. — lives in [Iteration](iteration/index.md).

Before changing packaging, hooks, warehouse doctrines, or embeddings, load the systems atlas in [Architecture](../architecture/index.md) — that section owns the design model; these pages own the loops.

### Preparation

# Preparing for Local Development

## Rip it Out

Do this once when you want the harness and on-path CLI to run **only** from your local checkout.

### 1. Stop Writers

Close the dashboard, your harness(es), and any other process that might have `warehouse.duckdb` open. A copy taken while DuckDB still has writers can be an inconsistent recovery image.

### 2. Make a Backup

Always back up your data before switching off of the normal plugin-marketplace install. Forward migrations can make it hard to go back to an older released engine against the same DB.

```bash
cp -r ~/.local/share/stockroom/warehouse.duckdb ~/warehouse.duckdb.backup
```

### 3. Uninstall the Plugin

Un-install the Stockroom plugin from your harness(es).

## Run a Local Checkout

Once you do not have an active Stockroom engine hooked into a harness, you're ready to hook up a local checkout.

### 1. Shim Takeover

```bash
make shim TAKEOVER=1 FORCE=1
```

Now, the `stockroom` CLI points at the engine in your local checkout. Python code changes you make *will* be reflected.

### 2. Harness Wiring

From the **repo root**, set `HARNESS` to the IDE you are entering with:

```bash
make sync                       # lock-faithful engine env (strips torch)
HARNESS=cursor make localdev    # or HARNESS=claude, etc
```

`HARNESS=… make localdev` composes three atoms, in order:

1. **`local-skills`**
	- Cursor: symlink `skills/*` into `.cursor/skills/stockroom-local/` (with a managed pre-commit guard so the mirror never lands in a commit).
	- Claude: no-op with a reminder to use `claude --plugin-dir .` for a session-scoped plugin load.
1. **`local-engine`** — claim the on-path shim as owner `dev` with `--takeover --force`, then `stockroom shim ensure-env` for this checkout's engine dir (locked deps + torch from freeze).
2. **`local-dashboard`** — force-replace `stockroom dashboard` (`--replace`) so this checkout's code is loaded even when identity matches.

### 3. Dashboard Start

Run `stockroom dashboard` to bounce the dashboard process to the new engine path.

### 4. Harness Reload

Re-open your harness.

If using Claude Code, launch with

```bash
claude --plugin-dir .
```

You should now have:

- `stockroom` on PATH with an engine path baked in pointing at this checkout (owner `dev`)
- For Cursor: `/sr-*` skills resolving to the checkout mirror when working in this project

## Verify

```bash
make localdev-status    # read-only status report
stockroom doctor probe  # show shim & system info
stockroom doctor smoke  # ensure Torch actually works
```

Confirm your harness is loading **this** checkout's skills, not a leftover marketplace install.

You are now ready to start developing!

## Done Developing

### 1. No More Local

```bash
HARNESS=cursor make localdev-clean    # or HARNESS=claude
```

That removes the Cursor skills mirror / pre-commit guard (Claude: nothing to mirror) and deletes `~/.local/bin/stockroom` **only if** its header says `owner=dev`. Harness-owned shims are left alone. Warehouse untouched.

### 2. Restore Database

Wherever you backed up your database, restore it now.

### 3. Re-Install Marketplace Plugin

Then restore a normal install:

1. Reinstall / enable the marketplace plugin in the harness UI.
2. Launch the harness — sessionStart `shim rectify` recreates the on-path shim for the plugin (and rebakes after path moves).

Confirm:

```bash
make localdev-status
stockroom doctor probe
stockroom doctor smoke
```

You're back running the released Stockroom!

## Appendix: Modular Atoms

Use these when you do not need the full rip-it-out path. Harness-scoped targets require `HARNESS=cursor` or `HARNESS=claude` and error if unset or invalid.

| Target | `HARNESS`? | Role |
| --- | --- | --- |
| `local-skills` | required | Wire checkout skills for that harness |
| `local-engine` | no | `stockroom` CLI points at local python code |
| `local-dashboard` | no | Force-replace `stockroom dashboard` (`--replace`) |
| `localdev` | required | Invokes the three atoms above |
| `localdev-clean` | required | Undo harness-managed bits + remove `owner=dev` shim (not warehouse) |
| `localdev-status` | optional | Report managed vs shim sections |

### Engine-only Shim Claim

```bash
make shim                      # bake this checkout (owner: dev)
make shim TAKEOVER=1           # replace a *dead* foreign bake
make shim TAKEOVER=1 FORCE=1   # replace a *live* foreign bake (dangerous)
```

`FORCE=1` is for localdev and recovery of a broken install. You run the risk of pointing python code at a different migration level than your DB, at your DB, which may corrupt your data if you don't know what you're doing.

### Status Semantics

`make localdev-status` prints two sections (read-only; no mutations):

1. **localdev-managed** — skills mirror and pre-commit block (when present)
2. **shim** — on-PATH location, default dest (`~/.local/bin/stockroom`), owner + baked `app-dir` from the shim header, whether that engine dir is alive, and torch version in that engine's `.venv` (or “not installed”)

### Clean Semantics

`HARNESS=… make localdev-clean` removes that harness's localdev-managed artifacts and deletes `~/.local/bin/stockroom` **only when** the shim header is `owner=dev`. 

It does **not** touch the warehouse, marketplace installs, or a harness-owned shim. After a `dev` unclaim, reinstall the marketplace plugin and launch — sessionStart `shim rectify` creates the missing on-path shim for the plugin.

### Claude Without Marketplace

For a session-scoped Claude load of the whole plugin tree (skills + committed plugin hooks):

```bash
claude --plugin-dir /path/to/stockroom
```

`HARNESS=claude make local-skills` prints that reminder and does not create a Cursor-style skills mirror.

### Skip Marketplace Uninstall

If you are only going to work with the engine (the python code), you can leave the marketplace plugin installed in your harness: all its skills and hooks will use the `stockroom` CLI shim (which now points at your local checkout's python code).

### Changing the Hooks

Committed plugin hooks under `hooks/*.json` use `CURSOR_PLUGIN_ROOT` / `CLAUDE_PLUGIN_ROOT`. After you uninstall the marketplace plugin those variables are unset, so copying those files into the project does **not** restore sessionStart. Make does not install project hooks. Only edit hooks by hand if you are changing the hook bootstrap surface itself; for day-to-day localdev, use `local-dashboard` / `stockroom dashboard` to launch the dashboard.

### Footguns

- **`make sync` / `make ci` strips torch.** Restore with `stockroom shim ensure-env` (hashed freeze) — not `make torch`, which picks `TORCH_INDEX` and rewrites the freeze. Use `make torch` only when deliberately choosing/changing the accepted stack ([Iteration](iteration/index.md), [Torch](../user-guide/troubleshooting/torch.md)).
- **Shim is succeed-or-refuse.** It never guesses an engine location. `TAKEOVER=1` alone is for dead foreign bakes; live foreign needs `TAKEOVER=1 FORCE=1`.
- **Always backup the warehouse** before local development.

### Engine

# Engine

The Stockroom Engine is the python code that powers [ingestion](../../user-guide/load/index.md), database migration, and serves the data to the [Dashboard](../../user-guide/dashboard.md).

The Python engine lives under [`skills/sr-search/`](https://github.com/Texarkanine/stockroom/tree/main/skills/sr-search) as a locked [uv](https://docs.astral.sh/uv/) project (`[tool.uv] package = false` — run-in-place). Everything is pinned through `uv.lock` **except torch**.

## Development Loop

With a `dev` shim baked to this checkout, edits under `skills/sr-search/src/` are what `stockroom` runs — no separate install step for Python sources.

Just edit the python code and try again!

### Changing Dependencies

If you need to change the dependency specification, `uv sync` via `make sync` will remove torch from the venv - it rebuilds the venv just from the lockfile (which Torch, you may recall, is not in).

When you genuinely need to sync without stripping torch:

```bash
uv sync --project skills/sr-search --inexact --no-config
```

Prefer `make sync` + restore torch via `stockroom shim ensure-env` when you want lock fidelity; use `--inexact` when you must keep an already-installed torch in the venv during dep iteration.

!!! tip "Re-Lock When Done!"
	Be sure you use `make lock` to regenerate the lockfile when done.

## Relevant Make Targets

| Target | Role |
| --- | --- |
| `sync` | Install deps from the committed lock (torch-free; strips torch if already installed — see [Torch](#torch)) |
| `lock` | Regenerate `uv.lock` hermetically |
| `lock-check` | Fail if the lock is stale vs `pyproject.toml` |
| `test` | pytest + dashboard JS tests (runs `sync` first; no coverage) |
| `coverage-engine` | pytest with `pytest-cov` → `skills/sr-search/coverage/lcov.info` (CI upload flag `engine`) |
| `coverage` | Both roots' lcov reports (`coverage-engine` + `coverage-dashboard-js`) |
| `lint` / `format` / `format-check` | ruff check / format / format --check |
| `reuse` | Whole-tree REUSE lint |
| `ci` | Full engine gate (lint/format/test/reuse; Codecov upload is CI-only) |
| `shim` | Bake this checkout onto PATH (owner `dev`; takeover flags in Local workflow) |
| `local-engine` | Claim shim + `ensure-env` for this checkout |

Coverage is opt-in: `make test` does not enable `--cov`. CI runs `make coverage-engine` (and the dashboard JS sibling) and uploads with `codecov/codecov-action`; the repository secret `CODECOV_TOKEN` is required before the README Codecov badge leaves 404 (expected until the first successful upload).

Engine pytest defaults to process workers via [`pytest-xdist`](https://pytest-xdist.readthedocs.io/) (`addopts = ["-n", "auto"]` in [`skills/sr-search/pyproject.toml`](https://github.com/Texarkanine/stockroom/blob/main/skills/sr-search/pyproject.toml)). Make and CI call bare `pytest`, so they inherit that. For serial debugging (or a single flaky case), override with `-n0`:

```bash
cd skills/sr-search && uv run --no-sync --no-config pytest -n0 tests/test_smoke.py -v
```

## Ad-hoc Invocation

The on-path `stockroom` command (`~/.local/bin/stockroom`) owns the torch-safe run contract and forwards to subcommands (`query`, `semantic`, `ingest`, `embed`, `migrate`, `shim`, `torch`, `doctor`, `schedule`, `dashboard`, `backfill`). Use `stockroom --help` / `stockroom <subcommand> --help`.

A correctly-[prepared](../preparation.md) local checkout will have the `stockroom` CLI on your PATH, pointing at your local checkout's python code. You can use it to run the engine's subcommands directly without having to use a long `uv ...` command.

```bash
stockroom ingest --full
stockroom ingest --full --verbose
stockroom embed --verbose
stockroom query "SELECT DISTINCT harness FROM sessions"
stockroom doctor smoke
```

<details>
<summary>Invoking the engine without the shim</summary>

The raw incantation the shim owns (`PYTHONPATH` makes the run-in-place package importable):

```bash
PYTHONPATH=skills/sr-search/src uv run --project skills/sr-search --no-sync --no-config python -m stockroom <subcommand>
```

You should never need to do this - doing this is the on-path stockroom CLI's job.

However, you could use this to run the engine from a project that is not wired up for local development, against your actual warehouse/database.

</details>

## Torch

Torch is held out of the lock on purpose so each machine gets a wheel that actually works - there are too many possibilities to try to ship a lockfile with Torch in it that would actually work.

### Relevant Make Targets

| Target | Role |
| --- | --- |
| `torch` | Install torch out-of-band + freeze under stockroom home |
| `sync` / `test` / `ci` | Lock-faithful installs that **strip** a previously installed torch |

### Restore After Sync

After `make sync`, `make test`, or `make ci`, restore the machine's **accepted** stack from the hashed freeze:

```bash
stockroom shim ensure-env
```

Do **not** run `make torch` for a routine restore — that picks `TORCH_INDEX` and **rewrites** the freeze.

### Try a new Torch

When you deliberately want a different wheel or index:

```bash
make torch                                    # CPU wheels (default)
make torch TORCH_INDEX=https://download.pytorch.org/whl/cu126   # CUDA example
stockroom doctor smoke                        # confirm import / embed path
```

`make torch` installs the wheel and freezes the accepted stack under stockroom home so heal can replay it with `--require-hashes`.

## Manual freeze

If torch is already importable in the engine venv and you only need the durable freeze:

```bash
stockroom torch freeze --index https://download.pytorch.org/whl/cpu
# or, before the shim exists:
PYTHONPATH=skills/sr-search/src python3 -m stockroom torch freeze \
	--app-dir skills/sr-search \
	--index https://download.pytorch.org/whl/cpu
```

The freeze also pins some PyPI transitives of torch that appear in `uv.lock`. Heal installs the freeze **after** the torch-safe inexact deps sync. Minor version drift of those shared deps between lock and freeze is acceptable.

### Docs

# Documentation Site

Human docs live under [`docs/`](https://github.com/Texarkanine/stockroom/tree/main/docs).

The documentation site is built with [properdocs](https://properdocs.org/) (a fork of [mkdocs](https://www.mkdocs.org/)) and Material for MkDocs.

1. Configuration: [`./properdocs.yaml`](https://github.com/Texarkanine/stockroom/blob/main/properdocs.yaml)
2. Content: [`./docs/`](https://github.com/Texarkanine/stockroom/tree/main/docs)
3. Dependencies: [`./pyproject.toml`](https://github.com/Texarkanine/stockroom/blob/main/pyproject.toml)

## Development Loop

1. `make docs` to start the local preview server
	* If you are doing heavy refactoring and causing many broken links, it may be helpful to run in non-strict mode: `uv run properdocs serve --no-strict`. CI will be strict, though.
2. Edit the markdown files in `docs/`

### Changing Dependencies

The root `pyproject.toml` uses the `docs` dependency group to specify the dependencies for the documentation site.

There's nothing special here; just normal [uv](https://docs.astral.sh/uv/) usage. Once you modify the root `pyproject.toml`'s dependency spec, just run `uv sync --group docs && uv lock`.

## Relevant Make Targets

| Target | Role |
| --- | --- |
| `docs` | Local preview (`properdocs serve`) |
| `docs-build` | Strict build — matches docs CI |

Config: [`properdocs.yaml`](https://github.com/Texarkanine/stockroom/blob/main/properdocs.yaml). Contributing nav order is controlled by [`docs/contributing/.pages`](https://github.com/Texarkanine/stockroom/blob/main/docs/contributing/.pages).

## Publishing

CI builds with `properdocs build --strict` on every PR (`.github/workflows/docs.yaml`). Deploy runs on a published GitHub Release or a manual `workflow_dispatch`.

### Dashboard

# Dashboard

Product behavior and URL: [Dashboard](../../user-guide/dashboard.md) (default [http://localhost:58008](http://localhost:58008/)).

| Layer | Path |
| --- | --- |
| Front-end | `skills/sr-search/src/stockroom/dashboard/static/` — native ES modules, vendored Chart.js + markdown-it, no bundler / no npm install |
| JS tests | `skills/sr-search/tests-js/*.test.mjs` |
| Server | `skills/sr-search/src/stockroom/dashboard/` |
| CLI | `skills/sr-search/src/stockroom/dashboard/__main__.py` |
| Python tests | `skills/sr-search/tests/test_dashboard_*.py` |

## Development Loop

Static ESM is read from disk on each request; Python changes only get picked up after the dashboard server process is replaced.

1. Edit server/metrics (and any other Python under `dashboard/`) plus the static modules that consume the API.
2. Bounce so this checkout's Python is what is listening:

	```bash
	make local-dashboard
	```

3. Hard-refresh the browser so cached ESM is not stale.
4. Run the dashboard contract gates:

	```bash
	make test-dashboard-js
	make test-dashboard-py
	```

## Relevant Make targets

| Target | Role |
| --- | --- |
| `test-dashboard-js` | Dashboard ES-module tests (`node --test`; Node 22; no sync; no lcov) |
| `coverage-dashboard-js` | Same tests with Node coverage → `skills/sr-search/coverage-js/lcov.info` (CI upload flag `dashboard-js`) |
| `test-dashboard-py` | `tests/test_dashboard_*.py` only (torch-safe; no sync) |
| `test` | Full pytest (xdist `-n auto`) + JS (runs `sync` first — strips torch) |
| `local-dashboard` | Force-replace `stockroom dashboard` for this checkout (`--replace`) |

`make test-dashboard-js` stays the human-readable default. Use `make coverage-dashboard-js` (or `make coverage`) when you need the Codecov-ready lcov artifact. Uploads happen in GitHub Actions via `CODECOV_TOKEN`; the aggregate README badge 404s until that secret exists and CI has uploaded at least once.

### Skills

# Skills

Wrapper skills are the agent-facing how-to for Stockroom's engine surfaces. They live under [`skills/`](https://github.com/Texarkanine/stockroom/tree/main/skills) — each skill is a directory with a `SKILL.md` (plus optional `references/`). Skills invoke the engine only as `stockroom <subcommand>`; they do not call `uv` or `make`.

| Skill | Path | Role |
| --- | --- | --- |
| `sr-dashboard` | `skills/sr-dashboard/` | Open / print the local dashboard URL |
| `sr-initialize` | `skills/sr-initialize/` | How to onboard a machine to Stockroom and/or heal an existing installation |
| `sr-query` | `skills/sr-query/` | How to use SQL to query the warehouse |
| `sr-search` | `skills/sr-search/` | How to use the engine to search the warehouse. Also contains the Python `engine` code. |
| `sr-semantic` | `skills/sr-semantic/` | How to search the warehouse w/ Semantic (vector) search |

## Development Loop

1. Edit `skills/<name>/SKILL.md` and any `references/` that skill owns.
2. Reload so the harness picks up the text:
	- **Cursor:** with localdev skills wired, the project mirror under `.cursor/skills/stockroom-local/` follows the checkout (`HARNESS=cursor make local-skills`). Close & re-open Cursor once you make Skill text changes. Do not commit the mirror — localdev installs a pre-commit guard so it stays out of git to help you avoid this.
	- **Claude Code:** load the plugin tree with `claude --plugin-dir /path/to/stockroom` (see [Preparation](../preparation.md)); there is no Cursor-style skills mirror.
3. Exercise the skill in a session.

## Relevant Make Targets

| Target | Role |
| --- | --- |
| `local-skills` | Wire checkout skills (`HARNESS` must be `cursor` or `claude`) |
| `localdev` / `localdev-clean` / `localdev-status` | Full enter / clean / status composition — see [Preparation](../preparation.md) |

### Backfill Adapters

# Backfill Adapters

`stockroom backfill` is an orchestrator over a registry of per-source adapters, mirroring how ingest is an orchestrator over per-harness parsers. Teaching it to read another harness's legacy store is a new module plus a registry entry — not orchestrator surgery.

Read [Architecture → Backfill](../../architecture/backfill.md) first if you have not; it owns the *why* (why this is off the nightly path, why the orchestrator holds all the SQL, why provenance is exact). This page is the loop.

## Layout

Paths below are relative to `skills/sr-search/`.

| Path | Role |
| --- | --- |
| `src/stockroom/backfill/__init__.py` | Registry `_SOURCES`, skip set, write loop, per-source summary |
| `src/stockroom/backfill/cursor_vscdb.py` | Today's only adapter — the worked example |
| `src/stockroom/backfill/__main__.py` | CLI |
| `tests/test_backfill.py` | Orchestrator, registry conformance, guard tests |
| `tests/test_backfill_cursor_vscdb.py` | Adapter-level tests |
| `tests/test_backfill_cli.py` | End-to-end subprocess runs |

## The Adapter Contract

An adapter is a module exporting five names, added to `_SOURCES` in `backfill/__init__.py`:

| Name | Contract |
| --- | --- |
| `NAME` | Registry key and `--source` value. Must equal its key in `_SOURCES` (e.g. `cursor-vscdb`) |
| `HARNESS` | Existing harness label. Scopes the skip set and labels the summary |
| `resolve_source(override)` | Returns the store path from flag → env → config. Raises `BackfillError` naming **all three** inputs when unconfigured |
| `candidates(source)` | Cheap id enumeration. Must not parse — the skip set is applied to this list, *before* the expensive work |
| `parse_all(source, ids)` | Yields `NormalizedSession` for those ids — the same contract ingest parsers produce |

Three rules the orchestrator relies on:

1. **Adapters never touch the warehouse.** No connection is passed in, and none should be opened. Every skip decision, write, and summary count belongs to the orchestrator.
2. **`candidates` is cheap and `parse_all` is not.** The split exists so a re-run skips already-present sessions without reading them. Collapsing the two throws that away.
3. **Fail soft, per record and per source.** One unparseable record is skipped, not fatal; one broken source does not stop the others. `parse_all` yields `None`-free results and simply omits what it cannot reconstruct.

A parametrized conformance test in `tests/test_backfill.py` runs over `_SOURCES`, so a new adapter is checked for all of this the day it lands.

## Adding One

1. **Write the adapter tests first**, in `tests/test_backfill_<source>.py`. Synthesize the store in-test rather than committing a binary fixture — see the `build_vscdb` factory in `tests/conftest.py` for the pattern.
2. **Write the adapter**, exporting the five names above. Give it a module docstring recording the store's shape; that store is undocumented by its vendor and the docstring is the only place that knowledge lands.
3. **Register it** in `_SOURCES`. The CLI's `--source` choices come from the registry, so nothing in `__main__.py` needs editing unless the source needs its own path flag (`--state-vscdb` is the precedent).
4. **Add a user-guide page** under `docs/user-guide/load/backfill/`, sibling to `cursor-vscdb.md`, and a row in that section's index table. Per-source read caveats and warehouse-column mappings belong there, not in the shared page.
5. **Run the gate**: `make ci` plus `make docs-build`.

## Trying It Against A Real Store

Backfill writes to the warehouse, so exercise it against a scratch one rather than your own:

```bash
STOCKROOM_HOME=/tmp/backfill-scratch stockroom migrate
STOCKROOM_HOME=/tmp/backfill-scratch stockroom backfill --source <name> --dry-run --verbose
STOCKROOM_HOME=/tmp/backfill-scratch stockroom backfill --source <name> --verbose
STOCKROOM_HOME=/tmp/backfill-scratch stockroom query "SELECT harness, count(*) FROM sessions GROUP BY 1"
```

`--dry-run` does everything but the write, which makes it the fast loop while a parser is still wrong; `--force` re-parses what the same source previously wrote, which is the loop after it is nearly right. A dry run goes through `warehouse.open_current()` — read-only, never migrating, no single-writer flock — so it needs a warehouse that already exists and is at schema head, which is why `stockroom migrate` is the first line above.

## Guard Tests You Must Not Weaken

Two tests in `tests/test_backfill.py` encode the "not nightly" invariant as an absence, which means nothing else will catch a regression:

* `schedule.render_payload()` contains no `backfill` token.
* The `stockroom.ingest` package source contains no import of `stockroom.backfill`.

The second matches the dotted import path rather than the bare word on purpose — the writer's own docstring has to *discuss* the backfill case to justify its run-clock fallback, and a guard that forbids naming what it protects against is one somebody weakens the next time they write a sentence.

### Index

# Development Iteration Cycles

This section is day-to-day work **after** your local checkout is wired up. Don't know what that means? Go through the [Preparation](../preparation.md) process first!

## Prerequisites

- A local checkout already on the [Preparation](../preparation.md) and wired up in your harness of choice.
- [uv](https://docs.astral.sh/uv/) for the engine and docs toolchains.
- **Node 22** for dashboard JS tests and the full `make test` / `make ci` gate

Machine onboarding for a *released* install (torch pick, doctor smoke, schedule, first ingest) is still [`sr-initialize`](https://github.com/Texarkanine/stockroom/blob/main/skills/sr-initialize/SKILL.md) — not `make`. Contributors use Make against a checkout they already own, in order to develop and test changes.

## Mental Models

From the **repo root**, the [`Makefile`](https://github.com/Texarkanine/stockroom/blob/main/Makefile) is the usual entrypoint — it handles the `skills/sr-search/` directory and the `--no-config` / `--no-sync` flags. Run `make help` anytime for the full target list; the sections below only name the targets that matter for that surface.

### Two uv projects

| Project | Path | Purpose |
| --- | --- | --- |
| Engine | `skills/sr-search/` | Runtime + tests; torch held out of lock |
| Docs | repo root | `properdocs` site only (`uv sync --group docs`) |

## Things You can Iterate On

| Surface   | Description |
| --------- | ----------- |
| [Engine](engine.md)    | The Python code that powers Stockroom's engine, including ingesting data and the CLI |
| [Docs](docs.md)      | The documentation site |
| [Dashboard](dashboard.md) | The web interface for Stockroom's data |
| [Skills](skills.md)    | The how-to for Stockroom's agent-facing surfaces |
| [Backfills](backfill-adapters.md) | The code that powers Stockroom's backfill functionality |

### Licensing

# Licensing

Stockroom uses [REUSE](https://reuse.software/) for licensing, allowing multiple licenses to be assigned & attributed throughout the codebase.

## Licensing Intent

| Target | What | License |
| --- | --- | --- |
| Default | Code, docs, memory-bank, and everything else | [GNU Affero General Public License v3](https://www.gnu.org/licenses/agpl-3.0.en.html) |
| Prompt/Skill Text | `skills/**/SKILL.md` and `skills/**/references/**` only | [Public Prompt License (PPL-S)](https://shipfail.github.io/public-prompt-license/) |
| Vendored Chart.js / markdown-it | Exact upstream dashboard artifacts | MIT (reiterated from upstream) |
| `.cursor/**` | Vendored agent tooling | None Specified (check upstream) |

## Checks

Is every file licensed?

```bash
make reuse
```

What license does a file fall under?

```bash
reuse spdx | grep -A 5 <path>
```

e.g. 

```
$ reuse spdx | grep -A 5 skills/sr-search/src/stockroom/dashboard/static/chart-4.5.1.umd.min.js
FileName: ./skills/sr-search/src/stockroom/dashboard/static/chart-4.5.1.umd.min.js
SPDXID: SPDXRef-67d5565acf332d4d6accfe56e67873b1
FileChecksum: SHA1: cb555814104cfb8bf88e4d1b21033b495c3c5a77
LicenseConcluded: NOASSERTION
LicenseInfoInFile: MIT
FileCopyrightText: <text>SPDX-FileCopyrightText: 2014-2025 Chart.js Contributors</text>
```

Prefer path aggregates in `REUSE.toml` over per-file SPDX headers when adding many new files.