# Agent Evaluation 核心论文与项目索引

本索引用于网站“论文与资料”页面。优先列原始论文和官方项目文档。

## 方法论与综述

1. **Evaluation and Benchmarking of LLM Agents: A Survey** — Mahmoud Mohammadi, Yipeng Li, Jane Lo, Wendy Yip, 2025  
   https://arxiv.org/abs/2507.21504  
   用途：建立 “What to evaluate / How to evaluate” 二维 taxonomy。

2. **An Evaluation-Driven Approach to Designing LLM Agents: Process and Architecture** — Boming Xia et al., 2024/2025  
   https://arxiv.org/abs/2411.13768  
   用途：Evaluation-Driven Development，连接 Offline Eval、Online Eval 与持续迭代。

3. **Establishing Best Practices for Building Rigorous Agentic Benchmarks** — Yuxuan Zhu et al., 2025  
   https://arxiv.org/abs/2507.02825  
   用途：Agentic Benchmark Checklist，强调 Task Validity、Outcome Validity 与 Benchmark Reporting。

4. **AlphaEval: Evaluating Agents in Production** — Pengrui Lu et al., 2026  
   https://arxiv.org/abs/2604.12162  
   用途：从真实企业 Requirement 构造 Production-grounded Eval，组合多种 Evaluator。

## 通用 Agent Benchmark

5. **AgentBench: Evaluating LLMs as Agents** — Xiao Liu et al., 2023  
   https://arxiv.org/abs/2308.03688

6. **AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents** — Chang Ma et al., 2024  
   https://arxiv.org/abs/2401.13178

7. **GAIA: a benchmark for General AI Assistants** — Grégoire Mialon et al., 2023  
   https://arxiv.org/abs/2311.12983

## Tool / User / State

8. **τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains** — Shunyu Yao et al., 2024  
   https://arxiv.org/abs/2406.12045

9. **τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment** — Victor Barres et al., 2025  
   https://arxiv.org/abs/2506.07982

## Web Agent

10. **WebArena: A Realistic Web Environment for Building Autonomous Agents** — Shuyan Zhou et al., 2023  
    https://arxiv.org/abs/2307.13854

11. **BrowserGym** — ServiceNow  
    https://github.com/ServiceNow/BrowserGym

## Coding Agent

12. **SWE-bench: Can Language Models Resolve Real-World GitHub Issues?** — Carlos E. Jimenez et al., 2023 / ICLR 2024  
    https://arxiv.org/abs/2310.06770

## Tool / Function Calling

13. **ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs** — Yujia Qin et al., 2023 / ICLR 2024  
    https://arxiv.org/abs/2307.16789

14. **StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models** — Zhicheng Guo et al., 2024  
    https://arxiv.org/abs/2403.07714

15. **The Berkeley Function Calling Leaderboard** — Shishir G. Patil et al., ICML 2025  
    https://proceedings.mlr.press/v267/patil25a.html

## Safety

16. **AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents** — Edoardo Debenedetti et al., 2024  
    https://arxiv.org/abs/2406.13352

17. **Agent-SafetyBench: Evaluating the Safety of LLM Agents** — Zhexin Zhang et al., 2024  
    https://arxiv.org/abs/2412.14470

18. **OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety** — Sanidhya Vijayvargiya et al., 2025  
    https://arxiv.org/abs/2507.06134

## Judge

19. **Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena** — Lianmin Zheng et al., 2023  
    https://arxiv.org/abs/2306.05685

20. **Agent-as-a-Judge: Evaluate Agents with Agents** — 2024  
    https://arxiv.org/abs/2410.10934

## Reliability / pass@k 背景

21. **Evaluating Large Language Models Trained on Code** — Mark Chen et al., 2021  
    https://arxiv.org/abs/2107.03374

## Inspect AI 官方资料

22. Inspect AI  
    https://inspect.aisi.org.uk/

23. Inspect AI — Tasks  
    https://inspect.aisi.org.uk/tasks.html

24. Inspect AI — Agents  
    https://inspect.aisi.org.uk/agents.html

25. Inspect AI — Sandboxing  
    https://inspect.aisi.org.uk/sandboxing.html

26. Inspect AI — Eval Logs  
    https://inspect.aisi.org.uk/eval-logs.html

27. Inspect AI — Hooks  
    https://inspect.aisi.org.uk/extensions-hooks.html

28. Inspect AI — Extensions  
    https://inspect.aisi.org.uk/extensions.html

29. Inspect AI GitHub  
    https://github.com/UKGovernmentBEIS/inspect_ai

## 开源评测与观测项目

这些不是论文排行榜，而是可以直接研究、安装或自托管的工程项目。选型时先确认它属于 Runner、Evaluator、Benchmark、Environment 还是 Observability 层。

30. OpenAI Evals — 模型与 LLM 系统评测框架和公开 Registry  
    https://github.com/openai/evals

31. Promptfoo — Prompt、模型、RAG、Agent 对比与 CI 回归  
    https://www.promptfoo.dev/docs/

32. DeepEval — Python 测试式 LLM 应用、RAG 和 Agent 指标  
    https://deepeval.com/docs/introduction

33. Ragas — RAG 检索与生成质量评估  
    https://docs.ragas.io/

34. Phoenix — OpenTelemetry/OpenInference Trace、评估、数据集与实验  
    https://arize.com/docs/phoenix/

35. Langfuse — 自托管 Trace、Prompt、Dataset、Experiment 与线上评估  
    https://langfuse.com/docs

36. EleutherAI lm-evaluation-harness — 大规模基础模型任务 Harness  
    https://github.com/EleutherAI/lm-evaluation-harness

37. HELM — Stanford CRFM 的多场景、多指标基础模型评估  
    https://crfm.stanford.edu/helm/

38. Harbor — 将任意 Agent 接入沙箱任务并批量运行 Benchmark  
    https://github.com/harbor-framework/harbor

39. Terminal-Bench — 真实终端环境中的执行型 Agent 任务  
    https://github.com/harbor-framework/terminal-bench

40. TUA-Bench — 通用 Terminal-Use Agent 的执行型任务集合  
    https://github.com/facebookresearch/TUA-Bench

## 阅读规则

论文的 Benchmark 数字只代表论文所声明版本、模型、环境、Agent Harness 与运行协议。网站正文应重点解释：
- 为什么这样设计；
- success signal 是否可信；
- environment 是否可复现；
- metric 是否与真实目标一致；
- 哪些设计值得迁移到企业 Eval；
- 哪些结果不能直接外推。