Cart Building with Deepgram

OverviewWhat cart building is

A customer speaks. Deepgram builds a live cart in real time as items are added, validated, removed, or changed. The cart is structured data. Each line has a canonical menu ID and a price. It is ready to write into the POS.

Deepgram ingests your menus, modifiers, limited-time offers, and brand vocabulary before launch. Every layer of the stack uses that context:

OverviewTwo integration paths

Cart building works with or without a conversational agent. Most partners start with the first path.

Order understandingThe Menu Extraction Pipeline

The hard part is not transcription. It is turning free-form speech into correct changes to a structured order, turn after turn. A naive approach re-generates the full cart with an LLM each turn. That fails in production: items drop or mutate between turns, and the model can invent item IDs. The Menu Extraction Pipeline closes off these failure modes.

Modifier grounding

The pipeline checks free-text modifiers ("extra", "no", "light") against the menu's own vocabulary before mapping. A modifier that does not match becomes an order note. It is never forced onto the wrong attribute. The same check separates instructions from menu items in one utterance.

Validation before pricing

Deepgram's validator normalizes the final cart before pricing and writeback. It flags unavailable items and invalid combinations. It repairs over-selections. The POS's acceptance response is monitored as telemetry.

Writeback is incremental. Each turn's cart is diffed against the last synced POS state. Only the deltas are sent, as native add, update, and remove calls. A quantity change updates the existing line.

Order understandingThe pipeline on a live turn

Every turn produces a full trace: what the model proposed, what the validator decided, how the cart changed, and what the agent said. The traces below come from live conversations.

{
  "check": {
    "lines": [
      { "posId": "BIC_CLASSIC_3P_WM",  "qty": 1 },
      { "posId": "BISCUIT_1PC",        "qty": 1 },
      { "posId": "RG_MASHED_POT",      "qty": 1 },
      { "posId": "REG_FANTA_STRAW",    "qty": 1 }
    ],
    "subtotal": 11.39,
    "tax": 0.16,
    "total": 11.55,
    "status": "POSTED"
  }
}

Each stage chip is clickable and opens that stage's detail. "Three piece combo" is an exact match (d=0.00) and passes. "Hamburger" lands nearest to Coleslaw at d=0.128, past the 0.120 gate, so the validator rejects it. Modifiers resolve independently under their parent item. The i button defines every element in the trace.

Order understandingSpecial requests and escalation

Not everything a guest says maps to a cart operation. Coupons, loyalty accounts, payment questions, and custom requests need an action, and the right action differs by store. Escalation prompts define these actions in plain language.

Each prompt has a name and a natural-language trigger. The agent evaluates every turn against the active prompts. When one matches, the agent runs the mapped action.

From request to action

Defining a tool call

Built-in actions cover handoff and spoken answers. Custom actions are tools. A tool has a name, a description, and typed parameters. The agent calls it when a prompt matches.

Menu intelligenceAgentic Menu

POS menus are structured for cashiers, not conversation. A real catalog has 18 latte SKUs instead of one latte with options. Combos are not linked to their base items. "No Onions" is a flat modifier ID. To make a menu voice-ready once took weeks of custom code and manual curation per brand. An agentic pipeline now does it automatically.

What the pipeline produces

Deepgram builds the POS connector once per platform. After that, merchant onboarding is sync, pipeline, review, publish. No custom code per merchant, cuisine, or menu structure. Menu changes propagate automatically after launch, by webhook or by change-detection polling, with no human reconciliation.

Speech layerKeyterm Prompting

keyterm string

Boost recognition of up to 100 important terms per request, at decode time. Available on Nova-3 and Flux.

Add a keyterm parameter to the query string. Repeat the parameter for each term. Encode multi-word phrases with %20 or +. Do not use commas, semicolons, or weights; the API treats such a value as one literal term and boosts nothing.

curl \  --request POST \  --header 'Authorization: Token YOUR_DEEPGRAM_API_KEY' \  --header 'Content-Type: audio/wav' \  --data-binary @youraudio.wav \  --url 'https://api.deepgram.com/v1/listen?model=nova-3&keyterm=Cajun%20fries&keyterm=Ghost%20Pepper%20wings'

Menu examples

Menu terms are the words a general model has seen least. Keyterms fix the words that carry the order:

Keyterms keep their formatting. Use capitals for proper nouns (Fanta) and lowercase for common terms (fries). The limit is 500 tokens per request across all keyterms. Focus on the 20-50 terms that matter most.

Dynamic updates on Flux

On Flux, update keyterms mid-stream with a Configure message. No reconnect is needed.

Speech layerAudio conditioning

Restaurant audio is hostile: dining-room noise, kitchen clatter, music, cross-talk, and the device's own speaker. An audio conditioning front end cleans the signal before it reaches the model.

Deepgram's restaurant models are also fine-tuned on real quick-service audio, including drive-thru, the harshest environment in the category.

Speech layerFine-tuned models

Keyterm prompting fixes vocabulary instantly. Fine-tuning goes deeper: the model's weights adapt to your audio.

What it learns. Deepgram trains custom versions of its models on real audio from your environment, paired with corrected transcripts: device and microphone characteristics, room acoustics, regional accents, and ordering phrasing.

How it deploys. A fine-tuned model is a new model name on the same API. The switch is a one-parameter change. Deepgram already ships a drive-thru model this way, next to the general models in the catalog. A partner-tuned model deploys the same way.

The improvement loop. Production audio shows what the model misses: a new device, an underrepresented accent, a cuisine's phrasing. Deepgram annotates the misses and adds them to the next training round. The updated model rolls out as a version bump. Accuracy compounds with volume.

Keyterms and fine-tuning together

The two compose. A per-cuisine template carries a keyterm profile on day one, then graduates to a fine-tuned model as volume grows.