Engineering blog
Real engineering problems we ran into while building a multimodal AI gateway.
Start here
GLM 5.2 Reasoning Effort: the Setting That Cuts Cost 20x (Measured)
Same coding answer: $0.0031 with reasoning effort set right vs $0.062 on GLM 5.2's unbounded default. 20x cheaper, 30x faster. How to set the dial per task.
Claude Fable 5 Won't Run Under ZDR: 30-Day Retention Is Mandatory
ZDR orgs get a 400 error on claude-fable-5: no opt-out on the Claude API, Bedrock, Vertex or Foundry. What it breaks for HIPAA/COPPA, and the routing fix.
LLM Prompt Caching: The Complete 2026 Guide (Cut Input Cost 50-90%)
How prompt caching works across Claude, GPT, Gemini and DeepSeek: cut input cost 50-90% and TTFT 3-10x. Architecture, provider comparison, Python code.
All articles57
-
How Many Tokens Is a PDF? 4 Vendors, 4 Rules, 15 to 2,100 a Page
GPT-6 bills a PDF page as its text. Claude adds 1,558 tokens a page, Qwen 2,098, and Gemini charges a flat 515 with text free. Six measured ways to pay less.
-
Claude Haiku 5.5 Pricing: One Token Past 100K Costs 5x (Measured)
A 100,000-token prompt on Claude Haiku 5.5 billed $0.0100 and a 100,001-token one $0.0500. Cached tokens count toward the line. Three ways to stay under it.
-
GPT-4 and GPT-3.5 Turbo Shut Down Oct 23: 3.5's Swap Costs About 5x
OpenAI retires GPT-4, GPT-3.5 Turbo, o1 and o4-mini on Oct 23. Its GPT-3.5 Turbo swap cost $0.75 per 1,000 calls vs an estimated $0.15; GPT-6 Luna, $0.037.
-
GPT-6 Luna vs GPT-5.6 Luna: Half the Price, Half the Words
GPT-6 Luna wrote 0.55x the words of GPT-5.6 Luna on goal-only prompts, matched it with the parts named (74 vs 69 of 80), and cost $0.81 vs $1.61 per 1,000.
-
GPT-6.1 Sol vs Claude Sonnet 5.5: Same $2/$10, Half the Cost per Task
GPT-6.1 Sol and Sonnet 5.5 both list at $2/$10. Sol cost 0.49x per task in our tests and $0.72 vs $2.74 at a matched index score; Sonnet scores higher at max.
-
Claude Sonnet 5.5 vs Sonnet 5: Same Price, 80% Less per Task
Sonnet 5.5 cost $0.0041 per task vs Sonnet 5's $0.021 at the same $2/$10. Below xhigh effort it wrote its working into 68 of 156 answer-only replies.
-
Over-Refusal Measured: Most of It Is the Benchmark, Not the Model
We ran XSTest and OR-Bench over ten models, then audited the prompts. Only 77 of 200 hard ones were clearly benign, and Opus 5.5 drops from 23% refused to 4%.
-
Claude Opus 5.5 vs Opus 5: Same Answers, Half the Output Tokens
Opus 5.5 lists 20% under Opus 5, so only savings past 20% count. At equal prices it ran 56% cheaper on single-shot tasks and 22% cheaper in a tool loop.
-
Jev vs Flash LLMs: 7x Cheaper on Workflows, Same Cost on One Label
TypeSafe's Jev vs 5 flash LLMs, measured: 5-37x cheaper and 11-27x faster on 12-question workflows, same cost as Qwen or GLM on one label, never most accurate.
-
Best Flash LLM API 2026: GLM Best Value, Gemini 3.8 Strongest
9 current flash LLM APIs on price, benchmarks and a 33-run check: newer models cost 2.5-3.9x less per answer; GLM 5.3 Flash best value, Gemini 3.8 strongest.
-
GLM 5.3 API Cost: Always Thinking, 2.5x Cheaper per Answer Than 5.2
GLM 5.3 rejects thinking off and defaults to max: 33 of 33 right at $0.0047 per answer vs GLM 5.2's $0.0117. low saves 63% but misses 5. Flash costs a tenth.
-
DeepSeek V4.1 Flash API Cost: Matches V4 Pro at 3-6x Less per Answer
Day four: 33 of 33 right like V4 Pro at 3 to 6x less per answer and 2.9x the speed. Thinking off breaks multi-step work; no-answer questions burn 16K tokens.
-
GPT-6 Astra Reasoning Effort: max Costs 2.3x low for the Same Answers
Measured on 11 verified tasks: none fails 17 of 33, disabled does not disable, max costs 2.3x low, and Astra is 1.6x GPT-5.6 Sol per correct answer.
-
LLM API Rate Limits: 13 Providers Compared, 2 SDKs Never Retry a 429
Which tokens count toward RPM and TPM on 13 LLM APIs and clouds, which return headers, the four things a 429 can mean, and the two SDKs that never retry one.
-
Long-Context Pricing Tiers: Up to 6.7x, and Gateways Never Show It
Through one aggregator, nine tiered models billed up to 6.7x past lines their pages never mention, and one endpoint billed 1.25x its listing. How to stay under.
-
AI API Billing Units: 17 Meters from Tokens to PTU-Hours
The same API dollar is metered 17 ways: tokens, characters, minutes, GB-days, token-hours, PTU-hours. A field guide with the gotchas that bite billing code.
-
Best LLM for Translation: 9 Models, 9 Languages, a 400x Cost Spread
9 LLMs, 9 languages, 8,455 blind verdicts: gpt-5.6-sol leads in 7 of 9, gemini-3.7-flash ties it in Korean and Italian; cost spans $0.26 to $104 per 1M chars.
-
LLM Data Retention and ZDR: Every Party That Can Read Your Prompts
Vendors delete API logs in about 30 days. Your tracing SaaS, vector memory, and gateway console keep prompts until you delete them. What ZDR actually covers.
-
LLM Structured Outputs: 4 of 12 APIs Return Valid but Wrong JSON
Structured outputs measured on 12 LLM APIs: 100% schema-valid JSON, wrong values on 4 models with thinking on, same call billed 30 to 4,959 prompt tokens.
-
Which APIs Mark AI Content? 16 Measured Against 10 Marking Laws
16 generative APIs against 10 marking laws: 7 embed C2PA, 4 embed China's label, none both, audio and video nothing, and no C2PA manifest survives a resize.
-
Voice Agent API Cost: a 10-Minute Call Runs $0.04 to $0.57
Every leg measured: STT at $0.002-0.006/min, TTS at $0.004-0.034 per audio-minute, LLM turns, and GPT Realtime. Cascade vs speech-to-speech, priced per call.
-
DeepSeek V4 Pro GA vs Preview, Measured: 18-62% Less Thinking
deepseek-v4-pro-0813 vs the preview build on identical tasks: 18-62% fewer reasoning tokens, a fixed 8,192-token runaway, and a lost 'I don't know'.
-
Gemini 3.7 Flash API Cost, Measured: Tasks Bill 2.5-8x Less Than 3.6
Day-one probes of gemini-3.7-flash: half-price intro until Dec 31, thinking down 26-77% vs 3.6, off-switch gone, prefill 400s. Task bills drop 2.5-8x.
-
How Many Tokens Is an Image? 15 APIs Measured, Same Icon 6 to 1,298
The same 64px icon bills 6 tokens on GPT-5.6 and 1,298 on ByteDance Seed. Three billing schemes, three broken doc rules; dollars follow the rate, not the image.
-
LLM Thinking Controls: What 13 Models Accept, Ignore, or Enforce
reasoning_effort and thinking_budget across 13 LLM APIs: three vendors enforce budgets to the token, others silently ignore them, a cap of 16 can burn 97.
-
Web Search API and Web Fetch API: How They Work and What $0.01 Buys
How server-side web search and web fetch bill: a $0.01 fee, 1,500-3,100 injected result tokens at your model's rate, and a turn two that can re-search.
-
Qwen-Image 3.0 API, Measured: 9 Scenes for One $0.03 Image
Day-one probes of qwen-image-3.0: $0.03 per image with prompts unbilled even at 5,000 tokens, a real nine-in-one trick, and small print that misspells itself.
-
DeepSeek V4 Flash API Cost: Thinking Mode Corrupts Strict JSON
Day-one 0731 measurements: $0.14/$0.28 list, a 1,024-token cache page, and a defect: thinking plus strict json_schema corrupted integers in 8 of 13 runs.
-
Qwen 3.8 Max API Pricing: 16 Thinking Tokens Beat the Off Switch
Day-one measurements of qwen3.8-max at $2/$6: an effort dial that is really a budget cap, a 1/6 accuracy collapse with thinking off, and a 16-token fix.
-
MCP Tool Overhead, Measured: 26 Tools, $0.03 Every Call
Every MCP tool is re-billed as input tokens on every call: one small tool is 401 tokens on Claude, the GitHub server $0.03/call, and families differ 2.7x.
-
Prompt Cache Write Cost: When Does the 1.25x Premium Pay?
One measured agent suite, one premium: a 6% loss on RAG, an 83% saving on batch. The read:write ratio decides, and you can measure yours before the invoice.
-
Claude Opus 5 vs Opus 4.8, Measured: Same Price, 3x Apart
Opus 5 and Opus 4.8 share a $5/$25 rate card, yet Opus 5 billed 3.1x more on identical tasks out of the box. Where it goes, and the switch that closes it.
-
Speech-to-Text APIs: 14 Models, $0.002 to $0.016 a Minute
14 speech-to-text APIs behind one gateway: per-minute rates, who streams, what token-billed gpt-4o converts to, and the China ASR tier most comparisons skip.
-
Gemini 3.6 Flash: the Thinking Dial That Moves Cost 30x (Measured)
Gemini 3.6 Flash bills reasoning you never see, and one request setting swings the same task's cost up to 30x. Measured across five task types, with the catch.
-
Seedance API Pricing, Measured: the Video-Token Formula, Solved
Seedance bills W×H×(24s+1)/1024 video tokens; we solved it to the token. 720p bills as 1248×704, and 4k's cheaper rate costs 2.1x more per second. Measured.
-
GPT-5.6 Prompting Guide: Two Defaults That Bill 1.5x and 10x More
GPT-5.6 defaults are expensive: omitting reasoning_effort bills 1.5x vs 'none'; unmarked prefixes cost 10x cached reads. Request-shape playbook, measured.
-
Kimi K3 API Pricing, Measured: Turn Off the 'Always-On' Reasoning
Kimi K3's docs say reasoning can't be disabled. reasoning_effort:'none' works and cuts simple queries 6x. Measured: effort dial, cache floor, 9-language rates.
-
GPT Live API Pricing: gpt-realtime Speaking Costs 4x
GPT Live is the ChatGPT feature, not an API; behind it is gpt-realtime-2.1: $0.019/min listening, $0.077/min speaking, silence free, cached replay 1/80th.
-
LLM Token Usage: Why a 4-Token Answer Bills 217 Tokens
Measured across GPT-5.6, Claude Fable 5, Qwen3.7-max and five more families: reasoning dominates the output bill. How to read every usage field, and cap them.
-
Prompt Cache Minimums: The Docs Under-State by 1.4-2.4x
Vendors publish a prompt-cache token minimum. Measured across LLM families, auto-cache needs 1.4-2.4x more than the docs say; Claude's explicit cache is exact.
-
GPT-5.6 Cost Guide: Prompt Caching 90% Off, Reasoning Effort
GPT-5.6's two cost levers, measured: explicit breakpoints bill cached input at 10% of the rate, and not sending reasoning_effort bills 1.5x as much as none.
-
Which LLM Is Cheapest for Your Language? Tokenizer Costs Measured
GPT-5.5 bills fewest tokens for European languages, Kimi for Chinese, DeepSeek for Japanese; Claude Fable 5, Opus 4.8 and Sonnet 5 run 1.2-2.3x. Measured.
-
Claude Fable 5 for Agents: Tool-Call Refusals, Cost vs GLM 5.2
Claude Fable 5 across five agent workloads vs glm-5.2, opus-4-8 and sonnet-5: mid-tool-call refusals, adaptive thinking, and cost that shifts 5-15x by shape.
-
LangChain Prompt Caching: Setups That Actually Hit the Cache
LangChain's friendliest syntax silently disables Claude's prompt cache. Measured fixes: cache_control via content blocks, variable placement, usage fields.
-
Claude Sonnet 5's New Tokenizer: 41% More Tokens per Prompt
Claude Sonnet 5's new tokenizer makes the same text about 41% more tokens than Sonnet 4.6, reshaping cost, budgets, and cache eligibility on the gateway.
-
GLM 5.2 Tool Calls in Agent Loops: What 'OpenAI-Compatible' Hides
GLM 5.2 speaks the OpenAI tool-calling API, but text rides with the tool calls and reasoning shows on the turn. How it lines up against OpenAI and Anthropic.
-
Transcription API Cost: 7 Models on the Same Audio
Seven transcription models, one multilingual audio set, one gateway: per-minute cost spans $0.0020 to $0.0164 and accuracy is not the differentiator.
-
Image Generation API Cost: 5 Models Compared ($0.006-$0.039)
Five image models, same prompts, one gateway: $0.006 to $0.039 per image at defaults, plus a quality knob that swings one model's bill 36x. Measured.
-
Open-Weight LLM Caching: Why Yours Is Provider Roulette
For open-weight LLMs, prompt caching is solved in the inference engine and broken by routing. A five-layer map, measured across DeepSeek, Qwen and Kimi.
-
Claude Fable 5 Caching: Same Contract, 2.9x the Bill vs Opus 4.6
Claude Fable 5 is live on Synthorai. Measured prompt caching, TTL, tokenization and cost vs Opus 4.6/4.8: same cache contract, new tokenizer, ~2.9x the bill.
-
Provider Drift: How Default Routing Inflates LLM Cost
On a multi-provider gateway's default routing, identical requests scatter across upstreams with separate caches. Hit rate craters and your bill climbs.
-
Does Your LLM Gateway Lie About Cache? A 5-Min Audit
Gateways can report cache hits while billing full price. One script audits both auto-cache (DeepSeek) and marker-based (Claude) caching in five minutes.
-
Claude Opus 4.8 on Synthorai: Caching & TTL vs 4.7/4.6
Claude Opus 4.8 is live on Synthorai. Measured prompt caching and TTL behavior vs Opus 4.7/4.6, what carries over, plus the tokenizer shift to re-check.
-
Best LLM by Use Case (2026): Chat, RAG & Agents Cost Matrix
Chat, RAG, or agents? Pick the cheapest model that still performs, with measured cost sketches (a 15-step agent task, 100K-query/day RAG) and a decision matrix.
-
LLM Prompt Caching in Python: A Working Code Tutorial
Measured prompt-cache savings across Claude, GPT-5, Gemini 2.5, DeepSeek-v4 and Qwen3 via Synthorai's OpenAI-compatible gateway. Real usage.cost and TTFT.
-
Which LLM Prompt Cache Is Cheapest? 5 Providers Compared (2026)
Claude, GPT-5.x, Gemini, DeepSeek and Qwen cache in five shapes: explicit vs automatic, 5-min vs 1-hour TTLs, reads from 0.1x to 0.5x. Measured side-by-side.
-
How LLM Prompt Caching Works: KV Cache & TTL Explained
How LLM prompt caching actually works: Transformer attention math behind K/V reuse, the memory-compute tradeoff that shapes TTL, and why it cuts cost and TTFT.
No articles under that topic yet.