The Five Enrichment Scopes: Taxonomy, Row, Column, Group, Cell

The most common question we get from a technical buyer is: "If I just want to refresh one field on one row, do I pay for the whole pipeline to run?" The answer is no, and the reason is the runner architecture. PYRAMYD's enrichment pipeline serves five distinct scopes from the same code path — taxonomy, row, column, group, and cell. They share the model call, the verification gate, the embedding step, and the provenance write. They differ only in the work unit, and in whether the run takes the bulk-run lock.

Same pipeline · five granularities. A full-taxonomy refresh (2,546 categories) and a single-cell fix run through the same code. The difference is the work unit, and whether the run takes the bulk-run lock.

This is the architecture post — why the runner was designed for five scopes instead of one, how the advisory lock serializes bulk runs, why the economics work, and what each scope is actually optimized for.

Section 1: Why five scopes

Most data tools are built around one unit of work: whole tables, single records, or one field across many rows. None of those units serves the operator who needs "refresh the demand field for these 12 categories before Tuesday's board meeting."

The five scopes map to five distinct operator workflows:

• Taxonomy · the hardening run. Every row in the table, every enrichable column. The only scope that takes the advisory lock, so only one bulk run is in flight at a time.
• Row · the on-demand refresh. One node (e.g. one product, one category), every enrichable column. The shape an analyst hits when they need a category's page to be current before a customer meeting.
• Column · the single-field refresh. One column on one row · e.g. the pricing field on one product. Deterministic fields such as founding year or a stock ticker skip the model entirely and resolve from a free authoritative API (Wikidata, SEC EDGAR).
• Group · the targeted refresh. One of the 10 field groups (e.g. all of "demand" or all of "compliance") on one row. Mid-grain — used when one section of a taxonomy page needs to be current.
• Cell · the surgical fix. One value inside a field on one row. Used when an operator spots a bad value or wants to refresh a single metric.

Section 2: The pipeline · shared across all five scopes

Every scope runs through the same five-stage pipeline. What changes is the work unit and the lock boundary; what doesn't change is the model call, the verification gate, the embedding step, and the provenance write.

Stage 1: Source resolution

For each row in scope, the runner loads the row's prompt and its catalog of data sources, and fills the prompt with the row's anchor fields. Each job's ID is derived from a hash of its taxonomy, row, and scope, an adaptation of the idempotency-key pattern Stripe describes,[1] so the same work is never queued twice in one run.

Stage 2: LLM enrichment

A GPT-5-family model on Azure OpenAI,[2] with web search switched on, reads the sources and returns the new values as structured JSON; every value is written with its sources. The model is configured, not hard-coded (one environment setting swaps it), per the twelve-factor backing-services pattern.[3]

Stage 3: Verification gate

After the write, a rule-based auditor re-reads the row and checks it: 26 rules in the core auditor (required fields filled, links to other nodes resolving, provenance in the right shape, banned filler words, length, and coverage), plus provenance and model checks on the governed taxonomies. This is where bad enrichments are caught. Rows that fail are flagged needs-review with the rule that failed, not silently dropped; rows that pass are stamped verified.

Stage 4: Embedding

Rows that pass the verification gate are embedded with text-embedding-3-large (1536 dimensions) and written to the pgvector store. This is what makes the enriched row retrievable by APEX and by every surface that runs vector search.

Stage 5: Provenance

Every value is written to the canonical Postgres table with its provenance · the source URLs and when they were retrieved, the model, and a cell-quality score. A value that did not change is still recorded, for traceability. This is the audit trail buyers ask for when they review AI transparency, and it's what makes the substrate trustable.

Section 3: Advisory locks and atomicity

The taxonomy scope · the "refresh everything" case — is the expensive one, and two of them at once would fight over the same model quota. So only one bulk run is allowed in flight at a time. A run is not all-or-nothing: each row is written as it finishes, and the run's progress (rows targeted, succeeded, failed, and skipped) is recorded in the enrichment_runs table, so an operator can see exactly where a crashed run stopped.

The runner uses a Postgres advisory lock[4] · an application-level mutex primitive that don't require a table or row to attach to. The bulk-run lock is keyed by hashtext('enrichment_taxonomy_lock'), one lock shared by every taxonomy, taken with pg_try_advisory_lock so it never waits, and held until the run's finalizer completes. If a second bulk run is requested while the lock is held, it returns at once without starting — no queue, no race.

Row, column, group, and cell scopes don't take the lock — they share the same small concurrency budget and run alongside a bulk run. Bulk runs are serialized; smaller runs interleave.

Section 4: Why the economics work

a16z's analysis found that, for a model of a given capability, inference cost has been falling about 10x a year.[5] That curve makes whole-taxonomy refreshes tractable. But the bigger lever is scope.

A cell or column run calls the model for one value, not for a whole row or table. Deterministic fields skip the model entirely and resolve from a free authoritative API. And because the taxonomies are shared, a row enriched once is there for every customer who opens it.

That is the structural reason scoped refreshes stay cheap: you pay for the unit you asked for, not for the pipeline around it.

Section 5: When to use each scope

Operator guidance · the heuristic we recommend. In the app, you ask for the cell, column, and row scopes yourself (billed in tokens to you); group and taxonomy runs are run by the PYRAMYD team:

• Cell when you spotted a bad value or need one number to be current before a meeting.
• Row when you're about to use a category/product/role in a customer deliverable and want every field group fresh.
• Group when one section of a taxonomy page (demand, compliance, economics) needs a targeted refresh.
• Column when one field on a record is stale — e.g. the pricing field on one product.
• Taxonomy after a schema change, or when a whole table needs hardening. Lock-gated: one bulk run at a time.

The reason this matters for buyers: an enrichment platform that only serves one scope is always overpaying for the others. Bulk-only tools force you to refresh a whole taxonomy when you wanted one cell. Per-row tools force you to spin up a job per row when you wanted to batch. The five-scope runner is the "one tool, five workflows" pattern that eliminates that overhead.