{
  "cells": [
    {"cell_type": "markdown", "metadata": {}, "source": [
      "# 01 — Deterministic State / Tool / Answer Eval\n",
      "本 Notebook 从一个 Case 开始，逐步查看 Fixture、Trace 和五类评分。\n",
      "目标：你应该能解释每一个布尔结果来自哪条证据。"
    ]},
    {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [
      "import sys\n",
      "from pathlib import Path\n",
      "from tempfile import TemporaryDirectory\n",
      "sys.path.insert(0, str(Path.cwd() / 'agent-eval' / 'lab'))\n",
      "from cases import case_by_id\n",
      "from pipeline import run_case\n",
      "case = case_by_id('refund-001')\n",
      "print(case.case_id, case.expected_state)\n"
    ]},
    {"cell_type": "markdown", "metadata": {}, "source": [
      "## 运行一次并查看证据\n",
      "环境会在每个 Case 前 reset，Trace 会写入临时目录。"
    ]},
    {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [
      "with TemporaryDirectory() as directory:\n",
      "    result, score = run_case(case, seed=0, variant='candidate', output_dir=Path(directory))\n",
      "    print(result.final_text)\n",
      "    print([(event.kind, event.name, event.result) for event in result.trace])\n",
      "    print(score)\n",
      "    assert score['state_ok'] and score['tool_ok'] and score['answer_ok']\n"
    ]},
    {"cell_type": "markdown", "metadata": {}, "source": [
      "## 练习\n",
      "把 case_id 改成 `cancel-no-confirm-001`，观察为什么没有调用更新工具。"
    ]}
  ],
  "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"name": "python"}},
  "nbformat": 4,
  "nbformat_minor": 5
}