Average pellets, 10 games, seeds 100–109, real-time clock · every model reads the same ~380-token text · a level has 244 pellets
phi4-mini
232pellets
p50 187 ms per decision
Jev 1.13.0
178pellets
p50 115 ms per decision
Average pellets, 10 games, seeds 100–109, real-time clock · paired by seed against Jev
On Pac-Man
0
qwen3.5:4b, gemma4:e4b and phi4-mini, real-time games
Beyond Pac-Man
External results: SemIf comparison; 220 hard JevBench questions
Output side
Four open reproductions within 72 hours of launch SemIf, open-source reimplementation · 777 decisions, total seconds · output side: reading logits vs generating JSON
The price
Average pellets · real time: 10 games, seeds 100–109 · Qwen3.8-27B: lockstep, seeds 100–102, about 2.4 s per decision with thinking · the two clocks are not directly comparable
425pellets
55 ms per decision · real time
Jev, same clock: 178
Distilled 0.8B, trained on labels from Qwen3.8-27B · average of 3 real-time runs (lockstep 417) · reads only the current state
194 public JevBench judgment questions (intent, routing, policy, severity, extraction; no Pac-Man) · Jev’s 0.89 is JevBench’s published result, not measured by us · the student started from the Qwen3.5-0.8B base
Hosted · Jev
Your own model
Setup
Latency
Intelligence
Generality
Cost
Jev: hosted by TypeSafe, ~115 ms p50 over the public internet · our 0.8B: 55 ms per decision, real time · cost: Vercel AI Gateway price