SWE-bench vs preference Elo
Why the best coding model is not always the best chatbot, and how CompareLLM keeps both numbers visible.
They measure different things
Elo is “which anonymous reply did a human prefer?” SWE-bench is “did an agent resolve a real issue under a named harness?” A model can win one and lose the other without anyone lying.
Harnesses matter more than people admit
The same weights with a different agent, more retries, or a different split will move SWE-bench by several points. We keep the source URL and date so you can see which table we ingested.
Use both on a compare page
If you are buying a coding agent, sort the coding leaderboard, then open vs pages against your current model. If you are buying a general assistant, start from Elo and still glance at SWE-bench so you do not pick a charming model that cannot edit a repo.
Ready to evaluate your stack?
Calculate your optimal model weights with Stack Engine or compare top models head-to-head.