Recent models push context windows past 1M tokens: DeepSeek V4 Pro, Claude Opus 4.7, Gemini 2.5 Pro, and more.

Researchers have uncovered common failure patterns in LLMs — primacy effects, silent instruction dropping, quality degradation under multiple constraints.

At the time of publication, none of these had been systematically measured as instruction density scales.

Context windows have grown from thousands to a million tokens. But as the table shows, no existing benchmark tests how instruction-following degrades when instruction count itself becomes the variable — even though omission errors and primacy effects are already known problems.

Models Evaluated (20)

r = reasoning mode

Protocol — Design Rationale

Six Metrics

How to Read This

$$\text{Primacy Ratio} = \dfrac{\text{errors(last }\frac{1}{3})}{\text{errors(first }\frac{1}{3})}\quad \begin{cases} > 1.0 & \text{later violated more} \\ = 1.0 & \text{no bias} \\ < 1.0 & \text{earlier violated more} \end{cases}$$

• Near 1.0 at low density: healthy — no position bias
• Rising above 1.0: bias toward early instructions grows
• Falling to 1.0 at high density: uniform collapse, not recovery

The Universal Pattern

1. Low (10–50): ratios near 1.0
2. Mid (150–200): ratios peak — selective attention under strain
3. Extreme (300+): converge to 1.0–1.5 — uniform failure

Put important instructions first — but at extreme densities, ordering stops helping.

Two Ways to Fail

The Shift

At low densities, errors are balanced. As density increases, omission takes over.

Omission Rate at 500 Instructions

Latency Explosion (Reasoning Models)

General-purpose models maintain near-constant latency.

The Efficiency Inversion

Reasoning models earn their accuracy by spending more time thinking — latency explodes as instruction count grows. General-purpose models stay fast regardless of density.

When you divide accuracy by latency, speed wins:

Reasoning models take minutes. Smaller models take seconds. For latency-sensitive applications, a fast mid-tier model often wins.

The Worry

We have seen that models drop instructions, skip the later ones, and stop trying under load. With all of that happening, you might wonder: does the report still make any sense?

If a model has to include 500 keywords, maybe it just stuffs them in and stops writing real sentences. So the paper also measured something beyond accuracy: basic writing quality.

The Answer

Two Models That Broke the Pattern

o3 and o4-mini were the only models whose reports became noticeably worse as instruction count grew. At high densities, their output stopped reading like a real business report.

Why did this happen? These models produced very few tokens to begin with. o3 wrote only about 1,500 tokens when given 500 instructions. Since 500 of those words had to be keywords, roughly one out of every three words was predetermined — leaving almost no room for normal sentences to form.

Model Selection

1. Use reasoning models for heavy prompts. If you need 100+ instructions followed reliably, pick o3 or gemini-2.5-pro. But expect them to be slow.
2. Use smaller models for light prompts. Below ~50 instructions, small models perform just as well, and they are much faster.
3. grok-3 hits a sweet spot. Nearly as accurate as o3 (61.9% vs. 62.8%), but faster and more consistent.

Prompt Engineering

4. Put your most important instructions first. Every model pays more attention to what comes earlier.
5. But do not rely on ordering alone. Beyond ~300 instructions, even the first ones start getting dropped.

System Design

6. There is a hard ceiling. Even the best model caps at ~68%. If you need more than that, break the task across multiple calls or agents.

What This Paper Did Not Answer

• Why do the three degradation patterns happen? The paper describes them but does not explain the cause.
• Do these patterns repeat across different tasks and topics?

From the Paper

• Investigate degradation mechanisms
• Explore instruction types beyond simple constraints
• Generalize degradation curves across tasks
• Investigate why o3 and o4-mini lose writing quality under high density while grok-3 does not

Personal Directions

• Extend to evaluate instruction-following across multiple general instruction scenarios

Variance Behaviors

• Top models: variance steadily increases — reliability drops, but not yet collapsed at 500.
• Mid-tier: peaks at 150–300 — a "critical zone" of unstable performance.
• Worst models: variance drops immediately — already overwhelmed.

llama-4-maverick is an extreme outlier with abnormally high variance, suggesting fundamentally different instruction-processing mechanisms.