Databox

When you need it #

Watch an agent work on a big dataset for a week and a pattern shows up: every day it unpacks the same files, writes a fresh script to decode them, answers one question, and throws the work away. Our own case was a game-car catalogue of several gigabytes, surveyed again for every new question.

Databox breaks that loop. The decoding happens once and is kept; every later question is a query that answers in milliseconds, and every conclusion can cite the exact query that produced it.

• Use Databox for numbers and records: filter by a value, join two tables, count across thousands of items.
• Use docs and facts for prose and rules — what you wrote down and what you decided. Databox never puts rows into recall.

How it works #

A box is a folder. Your data stays plain JSON files you can open, diff and back up; Databox adds a fast index beside them.

• The files are the truth. Delete index/ and nothing is lost: Databox rebuilds it from items/, never from the original raw data.
• No index yet? Still queryable. Until the first build finishes, Databox answers straight from the JSON files: the first query takes about a second, later ones are fast.
• Rebuilds only what changed. Each item is tracked by size and modified time, and a new index is written beside the old one and swapped in, so queries never see a half-built index.

Measured on a catalogue of 2,489 items (1.6 GB of JSON): a full build takes 36 s into a 252 MiB index, and re-indexing after one item changes takes 0.5 s.

Turn it on #

On Windows, open Add-ons in Companion and switch on Databox. Anywhere, the CLI does it: it checks DuckDB, writes the config, and restarts the service so the box loads.

That writes a block like this into config.toml. You can edit it by hand too; restart after.

If DuckDB is missing, GET /health says so plainly in its databox block instead of failing quietly.

Fill a box #

Grounded doesn't decode file formats — you know your data, and a tool built for the format will always do it better. Databox only asks for the result: an extractor, written in any language, that writes one JSON object per item into the box's items/ folder.

• Use arrays of objects for repeated things, not objects keyed by id: arrays become tables; keyed objects become one wide column.
• Write only what the source states. A missing value is a finding; don't fill it with a default.
• Rewrite a file only when its content changes, so re-indexing stays incremental.
• Keep each item under 16 MB. Bigger items are refused; split them.

A reverse-engineering toolkit, a game-format CLI, a CSV export or a ten-line script can all be extractors. The source repo has a worked example that catalogues npm packages (examples/databox).

From files to tables #

• One row per file. Items directly under items/ make a table called items. Items in a subfolder, items/cars/, make cars.items.
• Every row has _item, the file name without .json (CAR_417).
• Each top-level array becomes a table named after its key — parts, or cars.parts in a subfolder — with one row per element and _n, its position (from 1).
• Arrays of objects inside an array table become child tables named <table>_<key>. A child row is keyed by _item, then its parent's position as <parent>_n, then its own _n. This nests up to four levels deep; anything deeper stays a list column you can unnest().

Join a child to its parent on _item plus the position: parts_matis.parts_n = parts._n. GET /databox/:box/schema lists every table and column, so an agent never has to guess.

Query it #

Every surface Grounded already has gets Databox — read-only, so a query can never change your data.

• One statement per query, DuckDB dialect: SELECT, WITH or EXPLAIN. Writes, file access and settings are refused with a 400.
• limit caps the rows returned: 100 by default, up to 5,000. meta.available still counts every match and meta.truncated says when the limit cut it.
• timeoutMs defaults to 15 seconds (up to 120). A query that runs over is cancelled with a 504.

Keeping the index fresh #

Core notices when files in items/ change and rebuilds in the background. Until it finishes, answers carry meta.stale: true; re-ask if it matters.

POST /databox/:box/index rebuilds by hand: only the changed items, in seconds. {"force": true} rebuilds everything, and {"wait": true} answers when it's done. A changed item that adds a key or widens a type, or a large share of changed items, triggers a full build on its own. GET /databox/:box says which kind of build ran and why.

Cite what you learn #

Every answer carries meta.citation (databox:<box>) and the SQL. When a query answers a question worth keeping, save the conclusion as a fact with that citation. Anyone — you next month, or another agent — can re-run the query and check it, instead of trusting a summary.

Limits #

• Read-only. Queries never write. Only Core's own indexer changes the index.
• Not a decoder. Databox stores and queries; turning a binary format into JSON is the extractor's job.
• Not a memory lane. Rows never enter recall or the brief. A box reaches recall only through its schema card.
• Re-indexing is local. Over LAN sharing, a paired computer can query a box but not rebuild its index.