{
  "cells": [
    {"cell_type": "markdown", "metadata": {}, "source": [
      "# 02 — Reliability / Cost / Failure Injection\n",
      "同一 Case 重复运行，比较普通成功率、pass^k、p95 延迟和故障恢复。"
    ]},
    {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [
      "import sys\n",
      "from pathlib import Path\n",
      "from tempfile import TemporaryDirectory\n",
      "sys.path.insert(0, str(Path.cwd() / 'agent-eval' / 'lab'))\n",
      "from pipeline import run_suite\n",
      "with TemporaryDirectory() as directory:\n",
      "    report = run_suite(runs=5, variant='candidate', output_dir=Path(directory))\n",
      "print({key: report[key] for key in ('case_count', 'pass_rate', 'pass_k', 'p95_latency_ms', 'failure_tags')})\n",
      "assert report['runs'] == 5 and report['pass_k'] == 1.0\n"
    ]},
    {"cell_type": "markdown", "metadata": {}, "source": [
      "## 看 Timeout Recovery\n",
      "Candidate 在第一次超时后重试；把 variant 改为 baseline 可以看到相同 Case 失败。"
    ]},
    {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [
      "from cases import case_by_id\n",
      "from pipeline import run_case\n",
      "with TemporaryDirectory() as directory:\n",
      "    result, score = run_case(case_by_id('timeout-recovery-001'), 0, 'candidate', Path(directory))\n",
      "    print([(event.name, event.result) for event in result.trace])\n",
      "    assert score['success'] and len(result.trace) == 4\n"
    ]},
    {"cell_type": "markdown", "metadata": {}, "source": [
      "## 练习\n",
      "把 runs 改成 1、3、10，比较点估计和可靠性指标的变化。"
    ]}
  ],
  "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"name": "python"}},
  "nbformat": 4,
  "nbformat_minor": 5
}