Here’s a number that should worry anyone building with AI coding assistants: 7-16%.

That’s the success rate of state-of-the-art models on CodeAssistBench, a new benchmark that tests AI on real GitHub issues. Compare that to 70-83% accuracy on Stack Overflow-style questions.

Same models. Same prompts. 5x performance drop.

The Benchmark Illusion

Most coding benchmarks test isolated snippets. “Write a function that reverses a string.” “Implement binary search.” These are the AI equivalent of flashcards — useful for measuring pattern recognition, useless for predicting real-world performance.

CodeAssistBench (CAB) does something different. It automatically extracts issues from GitHub repositories, builds runnable containers, and verifies solutions actually work. Multi-turn conversations. Project context. Real dependencies.

The results are humbling.

Why the Gap?

Three factors compound:

1. Context Degradation Research on “Lost in the Middle” shows LLMs struggle with information scattered across long contexts. In a real project, the relevant code might be split across dozens of files. The model sees everything but understands nothing.

2. Training Data Cutoff CAB specifically tests on issues from post-training-cutoff repositories. The model has never seen these codebases. No pattern matching from training data — just actual reasoning.

3. Multi-Turn Complexity Real assistance isn’t single-shot. Users clarify requirements, provide feedback, iterate. Each turn introduces opportunities for the model to drift, misunderstand, or lose track of the goal.

What This Means for Practice

If you’re building AI coding tools, this has immediate implications:

Don’t trust benchmark scores. A model scoring 85% on HumanEval might be effectively unusable for your use case.

Test on YOUR codebase. The gap between generic benchmarks and your specific project could be enormous.

Invest in context engineering. The problem isn’t model capability — it’s context presentation. Better project understanding might matter more than better models.

Expect iteration. First-shot success rates are low. Design workflows that assume multiple attempts.

The Optimistic Reading

The gap is also an opportunity. If 7-16% is the baseline with naive prompting, there’s enormous headroom for improvement through better:

  • Project context summarization
  • Codebase indexing and retrieval
  • Type-aware code understanding
  • Test-driven verification loops

The models aren’t broken. They’re just being asked to navigate with a flashlight when they need a map.

The Deeper Question

Why do benchmarks diverge so dramatically from reality?

Because benchmarks measure what’s easy to measure, not what matters. Isolated code generation is easy to evaluate. Multi-turn, project-grounded assistance is hard. So we measure the easy thing and pretend it predicts the hard thing.

It doesn’t.

The 70-83% number feels good. The 7-16% number is real. Build for the real number.