Open models vs Claude: Kimi K3, GLM 5.3 and DeepSeek V4 Pro against Sonnet 5.5 and Opus 5.5

The usual reason to choose an open model over Claude is price. Now that every Claude model on QuickSilver Pro costs 75% less than Anthropic's list price, that reason needs checking again, and so does the question of what you give up in quality. So we put three large open models from our catalog and two Claude models through the same set of tasks, on 2026-10-20.

How we tested

17 tasks, each with an answer a script can check, in two groups:

• 13 standard tasks: six small Python functions (a duration parser, interval merging, an LRU cache, strict Roman numerals, an expression evaluator with C-style division, semantic-version ordering), three extractions to JSON (an invoice from an email, attendees from meeting notes, log-line counts) and four short maths and logic questions.
• 4 harder tasks: shell-style word splitting with quoting rules, the final state of an order after a chat full of changes and reversals, a counting-out puzzle where the count alternates between 3 and 4, and the number of integers up to 100,000 whose digits sum to 20.

Code passes only if it survives a fixed set of unit tests; JSON passes only if it equals the expected object; maths answers must match exactly. Every model got the same single user message per task through the OpenAI-compatible /v1/chat/completions endpoint, non-streaming, max_tokens 8,000, no system prompt, no temperature setting. Each task ran three times: 51 requests per model.

With these defaults the three open models think before answering and bill those reasoning tokens as output; the two Claude models answered directly, without extended thinking. Cost below is the billed input and output tokens multiplied by QuickSilver Pro prices on 2026-10-20.

Results

Costs are at QuickSilver Pro prices on 2026-10-20, computed from the measured tokens.

Where the open models were good enough

• Routine extraction. All three open models got the invoice, the attendee list and the log counts right in all nine of their runs. If your workload is "turn this text into this JSON", any of them will do it.
• Small, well-specified functions. Kimi K3 passed every coding task in every run. GLM 5.3 missed two of 21 coding runs: once it accepted an empty string as a Roman numeral, once it rejected a valid quoted string in the shell splitter.
• GLM 5.3 on price and speed. It was the only open model that was both cheaper than Sonnet 5.5 (about 30% less per task) and as fast (the same 3.7-second median). On the 13 standard tasks alone it passed 37 of 39 for $0.85 per 1,000 tasks, against 39 of 39 for $1.59 on Sonnet.

Where they were not

• Multi-step reasoning. The counting-out puzzle needs about forty careful steps. Sonnet 5.5 and Opus 5.5 solved it in all six runs. Kimi K3 solved it once in three, GLM 5.3 once (its other answers were 37 and 13), DeepSeek V4 Pro never.
• Getting an answer out at all. DeepSeek V4 Pro returned an empty answer 8 times out of 51: it spent its tokens reasoning and stopped before writing anything, five times by reaching the 8,000-token limit. A ninth request had not finished after ten minutes. Kimi K3's two misses were the same pattern. A larger max_tokens might have let them finish, at a higher cost per task; we did not test that.
• Following the output format. In one run GLM 5.3 wrote the correct JSON object and then appended its reasoning after it, which breaks any parser that expects JSON only.
• Latency. Kimi K3's median was 26.6 seconds and DeepSeek V4 Pro's 110 seconds, with DeepSeek's median ranging from 24 seconds in one run to 233 seconds in another. Fine for a batch job, not for anything a user waits on.
• Price. At 75% off list, Sonnet 5.5 cost less per task than both Kimi K3 (ten times less) and DeepSeek V4 Pro (a third), and it passed every task. Even Opus 5.5 was cheaper per task than both on this test.

So which should you use?

• For extraction and simple functions where cost matters most, GLM 5.3 held up: cheaper than Sonnet and just as fast, with a few more misses. Validate its output, as you would with any model in a pipeline.
• For anything with several reasoning steps, or where a wrong or empty answer is expensive, Claude Sonnet 5.5 was the better buy on this test: every task passed, a few seconds each, at a lower cost per task than two of the three open models.
• Kimi K3 was accurate on the standard tasks but slow and, at its price, by far the most expensive model here per task.
• DeepSeek V4 Pro had the lowest pass rate and the slowest answers on this harness; if you use it, allow a large max_tokens and retry empty responses.

Limits of this test

17 tasks and three runs per model is a small sample: one task moves a pass rate by two points, and the harder group has only four tasks. The tasks are short, single-turn and in English. We did not test long contexts, tool use, agentic coding, other languages, or the open models with reasoning turned down or Claude with extended thinking turned on, and any of those could change the picture. Latency was measured from one client on one day. Treat the table as a starting point and run your own prompts: the request code is the same for every model, so switching is a one-word change.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.quicksilverpro.io/v1",
    api_key="YOUR_QSP_KEY",
)

for model in ["glm-5.3", "kimi-k3", "deepseek-v4-pro", "claude-sonnet-5-5"]:
    response = client.chat.completions.create(
        model=model,
        max_tokens=8000,
        messages=[{"role": "user", "content": "Your task here"}],
    )
    print(model, response.usage.completion_tokens, response.choices[0].message.content)

For the full Claude price list, see Claude API pricing in 2026. For requesting output that follows a JSON schema, see the JSON schema guide.