Using small models to break down problems
Decompose messy documents with a fast local worker model, verify in software, and escalate only the disagreements to a larger reasoner.
SectionsA large model given an entire spec will invent scope. A small model given one paragraph and an enum will usually stay in its lane.
The four-step workflow
The pattern we use on messy project documents:
- Parse in code: headings, lists, table rows. Give every source unit a stable ID. Do not chunk by token count if the document already has structure.
- Extract, do not summarize: a 3B-class instruct model pulls atomic facts, requirements, and qualifiers (
must,maybe,phase 2). Two independent passes. If they disagree, escalate. - Verify against the source: another constrained call: did we drop a qualifier, invent a field, or change scope? Failures go to a larger model with *only* the disputed evidence.
- Ask a human last: when guessing would change architecture, cost, or acceptance criteria.
What application code owns
Application code owns counts, IDs, retries, and “what stage are we in.” Models do narrow semantic work. That is the opposite of an autonomous agent chatting with itself until the context window fills up.
Related: how we think about eval and tracing once those workers are in production.