{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 06｜训练并比较 BPE Tokenizer\n",
    "\n",
    "目标：改变词表大小，测量同一文本的 token 数，并验证编码—解码往返。实验使用临时目录，不污染项目文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from tempfile import TemporaryDirectory\n",
    "from tokenizers import Tokenizer\n",
    "from tokenizers.decoders import ByteLevel as ByteLevelDecoder\n",
    "from tokenizers.models import BPE\n",
    "from tokenizers.pre_tokenizers import ByteLevel\n",
    "from tokenizers.trainers import BpeTrainer\n",
    "\n",
    "candidates = [Path('llm/downloads/data/tiny_corpus.txt'), Path('../data/tiny_corpus.txt')]\n",
    "corpus = next((path for path in candidates if path.exists()), None)\n",
    "assert corpus is not None, '找不到 tiny_corpus.txt；请从实验资料包的 notebooks/ 目录启动 Jupyter'\n",
    "sample = '注意力会复用 KV Cache，tokenizer 也影响序列成本。🙂'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 单变量实验\n",
    "\n",
    "只改变 `vocab_size`，其余训练设置保持一致。小语料能产生的有效词表可能少于请求值，因此同时记录实际词表大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = []\n",
    "with TemporaryDirectory() as temp_dir:\n",
    "    for requested_size in (128, 256, 512):\n",
    "        tokenizer = Tokenizer(BPE(unk_token='[UNK]'))\n",
    "        tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=False)\n",
    "        tokenizer.decoder = ByteLevelDecoder()\n",
    "        trainer = BpeTrainer(vocab_size=requested_size, special_tokens=['[UNK]', '[BOS]', '[EOS]', '[PAD]'])\n",
    "        tokenizer.train([str(corpus)], trainer)\n",
    "        encoded = tokenizer.encode(sample)\n",
    "        decoded = tokenizer.decode(encoded.ids)\n",
    "        rows.append((requested_size, tokenizer.get_vocab_size(), len(encoded.ids), decoded))\n",
    "        assert encoded.ids, '编码结果不能为空'\n",
    "for row in rows:\n",
    "    print(f'requested={row[0]:3d} actual={row[1]:3d} tokens={row[2]:3d} decoded={row[3]!r}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 验收与解释\n",
    "\n",
    "1. 记录三组 token 数，不预设它必须严格单调；小语料、预切分器和未知字符都会影响结果。\n",
    "2. 检查 Emoji、空格和中英文混排的往返结果。\n",
    "3. 解释词表增大为何会减少部分序列长度，同时增大 embedding 与输出头。\n",
    "4. 把 sample 换成代码和纯中文各测试一次，形成对照表。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
  "language_info": {"name": "python", "version": "3.12"}
 },
 "nbformat": 4,
 "nbformat_minor": 5
}