Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking
| Source: arXiv AI
Tags: scientific discovery, hypothesis generation, LLM evaluation, energy scoring, AI research tools
Logit-based energy scoring reaches 33% Hit@1 on scientific hypothesis ranking across 1,323 papers in 12 disciplines — double the 16.6% achieved by prompted LLM-as-judge — with the best 1B parameter configuration hitting 53.1%, though post-hoc selection limits direct comparison.
Details
Evaluating AI-generated scientific hypotheses is an open problem: LLM-as-judge tends to favor familiar ideas, and semantic similarity penalizes novelty. This paper benchmarks an alternative — logit-based energy scoring — which uses the model's intrinsic confidence (token log-probabilities) rather than comparative prompting. The experimental setup: 1,323 papers across 12 disciplines, each paired with its actual hypothesis and 15 incorrect alternatives. Seven language models were tested with two scoring strategies. Intrinsic scoring reached 33.0% Hit@1 pooled across scorers, versus 16.6% for prompted listwise ranking — a roughly 2× improvement. The best result, 53.1%, came from a 1-billion-parameter model using logit-based energy scoring. The authors acknowledge this is the maximum across 14 model-by-scorer combinations selected post hoc, which limits its reliability as a headline number. For teams building AI-assisted research workflows, the key practical takeaway is that smaller models with intrinsic scoring can outperform prompted judgment from larger models on hypothesis evaluation tasks. The post-hoc selection caveat means the 53.1% should not be taken at face value without independent validation.