CompareLLM Methodology
CompareLLM operates as an automated, non-opinionated benchmark aggregator. We record every metric as a timestamped snapshot with a source name, optional URL, and observed date. Pages never fabricate or extrapolate missing scores.
Automated 4-Step Ingestion Pipeline
Runs Daily at 06:00 UTCIngest Verified Snapshots
Fetches latest dumps from OpenRouter Models API, official LiveBench JSON releases, Princeton SWE-bench Verified logs, and Chatbot Arena Elo tables.
Deterministic Name Resolution
Maps provider-specific IDs (e.g. anthropic/claude-3.5-sonnet:beta) to canonical catalog models using verified alias dictionaries.
Pareto & Percentile Scoring
Computes multi-dimensional percentiles (0–100) and calculates non-dominated Pareto frontier boundaries across quality vs cost axes.
Audit Trail & ISR Updates
If scores drift beyond sensitivity thresholds, changes are appended to /changelog and Next.js revalidates static page caches instantly.
Primary Data Sources
OpenRouter Models API
List prices ($/1M input and output tokens), maximum context window limits, and prompt format specs.
LiveBench (Monthly Releases)
Contamination-resistant evaluation over mathematical reasoning, data analysis, and coding tasks.
Princeton SWE-bench Verified
Real-world software engineering resolution rate across verified GitHub issue test harnesses.
LMSYS Chatbot Arena
Crowdsourced pairwise Bradley-Terry Elo preference rankings across coding and general conversations.
Transparency: What We Do vs What We Do Not Do
What We Do
- • Record timestamped, dated snapshots for every single metric.
- • Link directly to evaluator sources and harness versions.
- • Keep full public audit logs of every score shift in
/changelog. - • Render mathematical non-dominated Pareto convex hulls.
What We Do Not Do
- • We do not scrape closed third-party portals.
- • We do not fabricate subjective 0–100 scores for missing metrics.
- • We do not accept sponsored placements or artificial rank boosting.
- • We do not promote unverified models into indexable sitemaps.
What preference Elo means here
Preference Elo is a crowd vote from LMArena / Arena. People see two hidden answers and pick the one they like more. The model that wins more often gets a higher Elo. That means people preferred it — not that it passed a school test. It is not SWE-bench, not accuracy, and not a number we invent.
Longer explainer: What is Elo on an AI leaderboard? · Live Elo ranking
How the site updates (plain language)
- Every day at 06:00 UTC a cron calls
/api/cron/ingest. - OpenRouter, LiveBench, official SWE-bench JSON, and Arena Elo are fetched.
- Names match catalog aliases. New OpenRouter ids stay preview until a second source agrees or an admin promotes them.
- If a number moved enough, we write
/changelogand refresh the pages that show it. - Public login is only votes, comments, and saved models. Publishing news and promoting models is admin.
Frequently asked questions
Plain-English methodology and leaderboard answers
- Preference Elo is a crowd vote from LMArena / Arena. People see two hidden answers and pick the one they like more. The model that wins more often gets a higher Elo. That means people preferred it — not that it passed a school test. It is not SWE-bench, not accuracy, and not a number we invent.