Optima tackles AI benchmarking's biggest flaw by letting users test models against their own data

Details

Public AI benchmarks measure models on standardized tasks that rarely reflect what a specific business needs. Artificial Analysis, known for independent LLM evaluations including GDPval-AA and AA-Briefcase, has launched Optima to close this gap directly.\n\nOptima accepts three types of input: existing evaluation datasets (from files or Hugging Face), AI agent traces from platforms like Arize, Braintrust, or Langfuse, and plain use-case descriptions with sample inputs and outputs — from which Optima generates suggested test inputs and evaluation criteria that users can refine.\n\nTwo evaluation methods are available: rubric-based scoring against predefined criteria, or pairwise comparison where users rate a sample of response pairs and the platform derives full model rankings from those preferences. Results include quality scores, cost per task, and time per task — metrics that matter especially for agent workloads where a 10% quality gain may not justify a 3x cost increase.\n\nThe platform is live now. Whether a custom benchmark captures real business value still depends entirely on how well users design the test cases — Optima lowers the tooling barrier without changing the underlying methodology challenge.