diff --git a/docs/exps/sw_hw_cam_retrieval_benchmark.md b/docs/exps/sw_hw_cam_retrieval_benchmark.md index 93365fd..4994b56 100644 --- a/docs/exps/sw_hw_cam_retrieval_benchmark.md +++ b/docs/exps/sw_hw_cam_retrieval_benchmark.md @@ -41,6 +41,7 @@ just remote "python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retr ```bash just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz 512 +just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz 512 ``` ## 2. 软件检索速度与质量 @@ -133,38 +134,40 @@ benchmark。实现位置： - `hw/sim/tests/top/utils.py::query_topk_once_with_latency` - `hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py::cam_retrieval_benchmark` -周期口径如下： +周期口径如下。本节只报告Cocotb/Verilator仿真周期，不将cycle直接换算为ns或queries/s。 | 指标 | 含义 | |---|---| -| `accept_to_first_result_cycles` | query 被 `query_valid && query_ready` 接受后，到首个 `result_valid` beat 的周期数。 | -| `accept_to_last_result_cycles` | query 被接受后，到 `result_last` 断言的周期数，即完整 Top-K 串行结果输出完成。 | -| `total_query_cycles` | 从拉高 `query_valid` 到 Top-K 输出完成的完整事务周期数。 | -| `cycles_per_query` | 当前报告中的主指标，等于平均 `accept_to_last_result_cycles`。 | -| `queries_per_cycle` | `num_queries / total_query_cycles`，用于观察仿真事务吞吐率。 | +| `query_only_cycles_per_query` | 主指标。每个query从`query_valid && query_ready`握手成功的时钟沿，到对应`result_valid && result_ready && result_last`完成的平均周期数。 | +| `query_only_total_cycles` | 所有query-only事务周期之和，不包含装载、写入、噪声注入和查询间统计代码。 | +| `query_only_queries_per_cycle` | `num_queries / query_only_total_cycles`。 | +| `load_write_noise_cycles` | 写入CAM行以及可选写噪声注入阶段的周期数。无噪声模式下仍包含CAM行写入周期。 | +| `end_to_end_cycles` | 从开始写入数据集到最后一个query完成的完整benchmark硬件仿真周期数。 | +| `end_to_end_queries_per_cycle` | `num_queries / end_to_end_cycles`。 | +| `accept_to_first_result_cycles` | query被接受后，到首个结果beat完成握手的平均周期数。 | +| `accept_to_last_result_cycles` | query被接受后，到`result_last`结果beat完成握手的平均周期数。 | +| `cycles_per_query`、`total_query_cycles`、`queries_per_cycle` | 兼容旧字段，当前均为query-only口径，不代表end-to-end。 | -选择 `accept_to_last_result_cycles` 作为主 `cycles_per_query` 的原因是：Top-K -检索只有在串行结果流输出到 `result_last` 后才算完整完成；仅用首个 -`result_valid` 会低估实际 Top-K 检索事务成本。 +选择query-only的`accept→last`作为主cycles/query，是因为Top-K检索只有在串行结果流输出到`result_last`并被接收后才算完整完成；仅用首个`result_valid`会低估实际Top-K查询事务成本。`load_write_noise_cycles`和`end_to_end_cycles`单独报告，避免把非查询阶段混入query-only性能。 -### CIFAR-10 无噪声硬件周期结果 +### 无噪声硬件周期结果 -配置：512 rows × 128 queries × 512-bit hash，`TOPK_K=5`，`LANES=8`， +配置：512 rows×128 queries×512-bit hash，`TOPK_K=5`，`LANES=8`， `WRITE_NOISE_EN=0`。 -| 数据集 | 模式 | 查询数 | accept→first | accept→last / cycles/query | total query cycles | queries/cycle | 状态 | -|---|---|---:|---:|---:|---:|---:|---| -| CIFAR-10 | hardware no-noise | 128 | 1027.000000 | 1031.000000 | 132096 | 0.000968992 | pass | +| 数据集 | 模式 | 查询数 | query-only cycles/query | query-only total cycles | query-only queries/cycle | load/write/noise cycles | end-to-end cycles | end-to-end cycles/query | end-to-end queries/cycle | accept→first | accept→last | 状态 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---| +| CIFAR-10 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass | +| CIFAR-100 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass | 对应日志标记： ```text -RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=132096 queries_per_cycle=0.000968992 status=pass +RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 query_only_cycles_per_query=1031.000000 query_only_total_cycles=131968 query_only_queries_per_cycle=0.000969932 load_write_noise_cycles=1024 end_to_end_cycles=133628 end_to_end_cycles_per_query=1043.968750 end_to_end_queries_per_cycle=0.000957883 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=131968 queries_per_cycle=0.000969932 status=pass ``` 该结果说明：在当前 `NUM_ROWS=512, LANES=8, TOPK_K=5` 的硬件仿真配置下， -一次完整 Top-K 检索事务约为 **1031 cycles/query**。首个结果 beat 约在 -1027 cycles 后出现，完整 Top-K 输出额外消耗约 4 个周期。 +一次完整Top-K查询事务的query-only成本为**1031cycles/query**。首个结果beat约在1027cycles后出现，完整Top-K输出额外消耗约4cycles。完整benchmark端到端口径为**1043.968750cycles/query**，其中数据写入阶段为1024cycles。 > 注：以上数据来自 Verilator/Cocotb 仿真，不是 FPGA 板上实测。它可用于 > 架构级周期趋势分析，但不能直接等同于板级频率、吞吐或端到端系统延迟。 @@ -180,8 +183,11 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000 | Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 | | `ns_per_query` | 软件 Top-K 匹配阶段平均耗时；不含加载和写盘。 | | `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 | -| `cycles_per_query` | 硬件仿真中一次完整 Top-K 检索事务的平均周期数，当前采用 accept→last 口径。 | -| `queries_per_cycle` | 硬件仿真中完成 query 数除以总 query 事务周期数。 | +| `query_only_cycles_per_query` | 硬件仿真中一次完整Top-K查询事务的平均周期数，采用`query_valid && query_ready`到`result_last`完成握手口径。 | +| `query_only_queries_per_cycle` | 硬件仿真中完成query数除以query-only总事务周期数。 | +| `load_write_noise_cycles` | CAM行装载、写入及可选写噪声注入阶段周期数。 | +| `end_to_end_cycles` | 从开始写入数据到最后一个query完成的完整benchmark硬件仿真周期数。 | +| `end_to_end_queries_per_cycle` | 硬件仿真中完成query数除以end-to-end总周期数。 | ## 7. 当前结论 @@ -191,9 +197,8 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000 2. **当前软件基线速度约为 4.86k queries/s。** 该结果来自 Python integer brute-force Hamming scan，数据规模为 512 rows × 128 queries × 512 bits。 -3. **硬件检索质量基准现在已经报告 cycles/query。** - 在 CIFAR-10 512-row/128-query 无噪声配置下，完整 Top-K 检索为 - **1031 cycles/query**，首个结果为 **1027 cycles/query**。 +3. **硬件检索质量基准现在显式拆分query-only与end-to-end周期。** + 在CIFAR-10和CIFAR-100的512-row/128-query无噪声配置下，完整Top-K查询事务均为**1031cycles/query**，首个结果为**1027cycles/query**；写入阶段为**1024cycles**，端到端为**1043.968750cycles/query**。 4. **写噪声对精确排序影响显著。** 即使 Hit@K 保持较高，Golden Match@K 也会快速下降，说明噪声首先破坏精确排序，再进一步破坏类别命中。 @@ -203,11 +208,10 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000 ## 8. 后续建议 -1. 对 CIFAR-100 也运行同样的硬件周期检测，补齐与软件质量表同尺度的硬件周期表。 -2. 在 `docs/exps` 中继续维护： +1. 在 `docs/exps` 中继续维护： - 软件检索速度表； - 硬件无噪声一致性表； - 硬件噪声鲁棒性表； - - 硬件 `cycles/query` 表。 -3. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan，以区分“朴素 Python baseline”和“优化软件 baseline”。 -4. 增加 `NUM_ROWS` sweep：例如 512、1024、2048、4096 rows，观察软件 brute-force scan 的线性增长趋势。 + - 硬件query-only、load/write/noise、end-to-end周期表。 +2. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan，以区分“朴素 Python baseline”和“优化软件 baseline”。 +3. 增加 `NUM_ROWS` sweep：例如 512、1024、2048、4096 rows，观察软件 brute-force scan 的线性增长趋势。 diff --git a/hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py b/hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py index 40bd454..c7705b2 100644 --- a/hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py +++ b/hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py @@ -10,6 +10,7 @@ from pathlib import Path import cocotb import numpy as np from cocotb.clock import Clock +from cocotb.utils import get_sim_time from model.ref_model import ( match_topk, @@ -32,6 +33,7 @@ DEFAULT_ROW_FLIP_BITS = 16 DEFAULT_QUERY_FLIP_BITS = 16 DEFAULT_SEED = 20260522 BENCHMARK_KS = (1, 5) +SIM_CLOCK_PERIOD_NS = 10 @dataclass(frozen=True) @@ -214,6 +216,11 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]: if not timings: return { "num_queries": 0, + "query_only_total_cycles": 0, + "query_only_cycles_per_query": 0.0, + "query_only_min_cycles": 0, + "query_only_max_cycles": 0, + "query_only_queries_per_cycle": 0.0, "total_query_cycles": 0, "mean_total_query_cycles": 0.0, "min_total_query_cycles": 0, @@ -224,24 +231,57 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]: "queries_per_cycle": 0.0, } - total_cycles = sum(t.total_query_cycles for t in timings) + total_cycles = sum(t.accept_to_last_result_cycles for t in timings) total_first = sum(t.accept_to_first_result_cycles for t in timings) total_last = sum(t.accept_to_last_result_cycles for t in timings) count = len(timings) mean_last = total_last / float(count) + queries_per_cycle = count / float(total_cycles) if total_cycles > 0 else 0.0 return { "num_queries": count, + "query_only_total_cycles": total_cycles, + "query_only_cycles_per_query": mean_last, + "query_only_min_cycles": min(t.accept_to_last_result_cycles for t in timings), + "query_only_max_cycles": max(t.accept_to_last_result_cycles for t in timings), + "query_only_queries_per_cycle": queries_per_cycle, + # Backward-compatible aliases: query-only, not end-to-end. "total_query_cycles": total_cycles, "mean_total_query_cycles": total_cycles / float(count), - "min_total_query_cycles": min(t.total_query_cycles for t in timings), - "max_total_query_cycles": max(t.total_query_cycles for t in timings), + "min_total_query_cycles": min(t.accept_to_last_result_cycles for t in timings), + "max_total_query_cycles": max(t.accept_to_last_result_cycles for t in timings), "mean_accept_to_first_result_cycles": total_first / float(count), "mean_accept_to_last_result_cycles": mean_last, "cycles_per_query": mean_last, - "queries_per_cycle": count / float(total_cycles) if total_cycles > 0 else 0.0, + "queries_per_cycle": queries_per_cycle, } +def build_hardware_performance( + timings: list[QueryTiming], + *, + load_write_noise_cycles: int, + end_to_end_cycles: int, +) -> dict[str, float]: + performance = summarize_query_timings(timings) + num_queries = int(performance["num_queries"]) + performance.update({ + "load_write_noise_cycles": int(load_write_noise_cycles), + "end_to_end_cycles": int(end_to_end_cycles), + "end_to_end_cycles_per_query": ( + float(end_to_end_cycles) / float(num_queries) if num_queries > 0 else 0.0 + ), + "end_to_end_queries_per_cycle": ( + float(num_queries) / float(end_to_end_cycles) if end_to_end_cycles > 0 else 0.0 + ), + }) + return performance + + +def current_sim_cycle() -> int: + """Return the current benchmark clock cycle from simulator time.""" + return int(get_sim_time("ns") // SIM_CLOCK_PERIOD_NS) + + def output_dir_for(mode: str) -> Path: run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") if not run_id: @@ -264,7 +304,11 @@ def write_outputs(out_dir: Path, result: dict) -> None: "write_noise_en", "write_noise_rate_num", "write_noise_rate_den", "num_queries", "k", "macro_precision", "retrieval_recall", "macro_f1", - "recall@k", "exact_match_rate", "cycles_per_query", + "recall@k", "exact_match_rate", + "query_only_cycles_per_query", "query_only_total_cycles", + "query_only_queries_per_cycle", "load_write_noise_cycles", + "end_to_end_cycles", "end_to_end_cycles_per_query", + "end_to_end_queries_per_cycle", "cycles_per_query", "mean_accept_to_first_result_cycles", "mean_accept_to_last_result_cycles", "mean_total_query_cycles", "total_query_cycles", "queries_per_cycle", "status", ] @@ -289,6 +333,13 @@ def write_outputs(out_dir: Path, result: dict) -> None: "macro_f1": metrics["macro_f1"], "recall@k": metrics["recall@k"], "exact_match_rate": metrics["exact_match_rate"], + "query_only_cycles_per_query": result.get("performance", {}).get("query_only_cycles_per_query", ""), + "query_only_total_cycles": result.get("performance", {}).get("query_only_total_cycles", ""), + "query_only_queries_per_cycle": result.get("performance", {}).get("query_only_queries_per_cycle", ""), + "load_write_noise_cycles": result.get("performance", {}).get("load_write_noise_cycles", ""), + "end_to_end_cycles": result.get("performance", {}).get("end_to_end_cycles", ""), + "end_to_end_cycles_per_query": result.get("performance", {}).get("end_to_end_cycles_per_query", ""), + "end_to_end_queries_per_cycle": result.get("performance", {}).get("end_to_end_queries_per_cycle", ""), "cycles_per_query": result.get("performance", {}).get("cycles_per_query", ""), "mean_accept_to_first_result_cycles": result.get("performance", {}).get( "mean_accept_to_first_result_cycles", "", @@ -313,11 +364,18 @@ def write_outputs(out_dir: Path, result: dict) -> None: "", "## Hardware performance", "", - f"- cycles_per_query: `{result.get('performance', {}).get('cycles_per_query', '')}`", + f"- query-only cycles/query: `{result.get('performance', {}).get('query_only_cycles_per_query', '')}`", + f"- query-only total cycles: `{result.get('performance', {}).get('query_only_total_cycles', '')}`", + f"- query-only queries/cycle: `{result.get('performance', {}).get('query_only_queries_per_cycle', '')}`", + f"- load/write/noise cycles: `{result.get('performance', {}).get('load_write_noise_cycles', '')}`", + f"- end-to-end cycles: `{result.get('performance', {}).get('end_to_end_cycles', '')}`", + f"- end-to-end cycles/query: `{result.get('performance', {}).get('end_to_end_cycles_per_query', '')}`", + f"- end-to-end queries/cycle: `{result.get('performance', {}).get('end_to_end_queries_per_cycle', '')}`", + f"- cycles_per_query (compat, query-only): `{result.get('performance', {}).get('cycles_per_query', '')}`", f"- accept_to_first_result_cycles: `{result.get('performance', {}).get('mean_accept_to_first_result_cycles', '')}`", f"- accept_to_last_result_cycles: `{result.get('performance', {}).get('mean_accept_to_last_result_cycles', '')}`", - f"- total_query_cycles: `{result.get('performance', {}).get('total_query_cycles', '')}`", - f"- queries_per_cycle: `{result.get('performance', {}).get('queries_per_cycle', '')}`", + f"- total_query_cycles (compat, query-only): `{result.get('performance', {}).get('total_query_cycles', '')}`", + f"- queries_per_cycle (compat, query-only): `{result.get('performance', {}).get('queries_per_cycle', '')}`", "", "## Retrieval quality", "", @@ -338,7 +396,7 @@ def write_outputs(out_dir: Path, result: dict) -> None: @cocotb.test() async def cam_retrieval_benchmark(dut): - cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start()) + cocotb.start_soon(Clock(dut.clk, SIM_CLOCK_PERIOD_NS, unit="ns").start()) await reset_dut(dut) num_rows = dut_num_rows(dut) @@ -358,7 +416,11 @@ async def cam_retrieval_benchmark(dut): dataset = load_retrieval_dataset_npz(dataset_path) if len(dataset.rows) != num_rows: raise AssertionError(f"artifact row count {len(dataset.rows)} must equal DUT NUM_ROWS {num_rows}") + + benchmark_start_cycle = current_sim_cycle() + load_write_noise_start_cycle = current_sim_cycle() await write_rows(dut, dataset.rows) + load_write_noise_cycles = current_sim_cycle() - load_write_noise_start_cycle accumulators = {k: MetricAccumulator() for k in BENCHMARK_KS} timings: list[QueryTiming] = [] @@ -380,6 +442,8 @@ async def cam_retrieval_benchmark(dut): label_hit = query_label in retrieved_labels accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact) + end_to_end_cycles = current_sim_cycle() - benchmark_start_cycle + run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}" result = { "run_id": run_id, @@ -402,7 +466,11 @@ async def cam_retrieval_benchmark(dut): "seed": dataset.seed, }, "metrics": {str(k): accumulators[k].as_dict() for k in BENCHMARK_KS}, - "performance": summarize_query_timings(timings), + "performance": build_hardware_performance( + timings, + load_write_noise_cycles=load_write_noise_cycles, + end_to_end_cycles=end_to_end_cycles, + ), } out_dir = output_dir_for(mode) @@ -419,11 +487,22 @@ async def cam_retrieval_benchmark(dut): performance = result["performance"] dut._log.info( - "RETRIEVAL_PERF_RESULT mode=%s num_queries=%d cycles_per_query=%.6f " + "RETRIEVAL_PERF_RESULT mode=%s num_queries=%d query_only_cycles_per_query=%.6f " + "query_only_total_cycles=%d query_only_queries_per_cycle=%.9f " + "load_write_noise_cycles=%d end_to_end_cycles=%d " + "end_to_end_cycles_per_query=%.6f end_to_end_queries_per_cycle=%.9f " + "cycles_per_query=%.6f " "accept_to_first_result_cycles=%.6f accept_to_last_result_cycles=%.6f " "total_query_cycles=%d queries_per_cycle=%.9f status=pass output_dir=%s", mode, performance["num_queries"], + performance["query_only_cycles_per_query"], + performance["query_only_total_cycles"], + performance["query_only_queries_per_cycle"], + performance["load_write_noise_cycles"], + performance["end_to_end_cycles"], + performance["end_to_end_cycles_per_query"], + performance["end_to_end_queries_per_cycle"], performance["cycles_per_query"], performance["mean_accept_to_first_result_cycles"], performance["mean_accept_to_last_result_cycles"], diff --git a/hw/sim/tests/top/utils.py b/hw/sim/tests/top/utils.py index 26ce642..258aed3 100644 --- a/hw/sim/tests/top/utils.py +++ b/hw/sim/tests/top/utils.py @@ -216,7 +216,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None): ``timing`` 字段： - accept_to_first_result_cycles: query 接受到首个 result_valid beat - accept_to_last_result_cycles: query 接受到 result_last beat（Top-K 完成） - - total_query_cycles: 从拉高 query_valid 到 Top-K 完成的总上升沿数 + - total_query_cycles: 纯查询事务周期，等于 accept_to_last_result_cycles ``query_ready`` 是组合信号，握手周期在上升沿前采样；结果信号在 ReadOnly settled phase 采样，避免重新引入 query_ready 采样时序问题。 @@ -236,8 +236,10 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None): await RisingEdge(dut.clk) edge_count += 1 + q_valid = int(dut.query_valid.value) q_ready = int(dut.query_ready.value) assert q_ready, "Query accept handshake was missed despite query_ready pre-wait" + assert q_valid, "Query valid deasserted before accept handshake" accept_edge = edge_count dut.query_valid.value = 0 @@ -252,7 +254,9 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None): await RisingEdge(dut.clk) edge_count += 1 await ReadOnly() - if int(dut.result_valid.value): + result_valid = int(dut.result_valid.value) + result_ready = int(dut.result_ready.value) + if result_valid and result_ready: if first_result_edge is None: first_result_edge = edge_count rank = int(dut.result_rank.value) @@ -282,7 +286,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None): timing = { "accept_to_first_result_cycles": int(first_result_edge - accept_edge), "accept_to_last_result_cycles": int(last_result_edge - accept_edge), - "total_query_cycles": int(edge_count), + "total_query_cycles": int(last_result_edge - accept_edge), } return beats, beats[0][1], beats[0][2], score_debug, timing diff --git a/tests/test_retrieval_benchmark_performance.py b/tests/test_retrieval_benchmark_performance.py index 82272f3..5f26f7c 100644 --- a/tests/test_retrieval_benchmark_performance.py +++ b/tests/test_retrieval_benchmark_performance.py @@ -11,38 +11,73 @@ if str(HW_SIM_DIR) not in sys.path: from benchmarks.retrieval.test_retrieval_benchmark import ( # noqa: E402 QueryTiming, + build_hardware_performance, summarize_query_timings, write_outputs, ) -def test_summarize_query_timings_reports_topk_completion_headline() -> None: +def test_summarize_query_timings_uses_query_only_accept_to_last_cycles() -> None: summary = summarize_query_timings([ QueryTiming( accept_to_first_result_cycles=10, accept_to_last_result_cycles=14, - total_query_cycles=16, + total_query_cycles=14, ), QueryTiming( accept_to_first_result_cycles=12, accept_to_last_result_cycles=18, - total_query_cycles=21, + total_query_cycles=18, ), ]) assert summary == { "num_queries": 2, - "total_query_cycles": 37, - "mean_total_query_cycles": 18.5, - "min_total_query_cycles": 16, - "max_total_query_cycles": 21, + "query_only_total_cycles": 32, + "query_only_cycles_per_query": 16.0, + "query_only_min_cycles": 14, + "query_only_max_cycles": 18, + "query_only_queries_per_cycle": 2 / 32, "mean_accept_to_first_result_cycles": 11.0, "mean_accept_to_last_result_cycles": 16.0, + "total_query_cycles": 32, + "mean_total_query_cycles": 16.0, + "min_total_query_cycles": 14, + "max_total_query_cycles": 18, "cycles_per_query": 16.0, - "queries_per_cycle": 2 / 37, + "queries_per_cycle": 2 / 32, } +def test_build_hardware_performance_separates_query_and_end_to_end_cycles() -> None: + performance = build_hardware_performance( + [ + QueryTiming( + accept_to_first_result_cycles=10, + accept_to_last_result_cycles=14, + total_query_cycles=14, + ), + QueryTiming( + accept_to_first_result_cycles=12, + accept_to_last_result_cycles=18, + total_query_cycles=18, + ), + ], + load_write_noise_cycles=100, + end_to_end_cycles=140, + ) + + assert performance["query_only_total_cycles"] == 32 + assert performance["query_only_cycles_per_query"] == 16.0 + assert performance["query_only_queries_per_cycle"] == 2 / 32 + assert performance["load_write_noise_cycles"] == 100 + assert performance["end_to_end_cycles"] == 140 + assert performance["end_to_end_cycles_per_query"] == 70.0 + assert performance["end_to_end_queries_per_cycle"] == 2 / 140 + assert performance["cycles_per_query"] == performance["query_only_cycles_per_query"] + assert performance["queries_per_cycle"] == performance["query_only_queries_per_cycle"] + + def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> None: result = { "run_id": "test-run", @@ -75,14 +110,23 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N }, "performance": { "num_queries": 2, - "total_query_cycles": 37, - "mean_total_query_cycles": 18.5, - "min_total_query_cycles": 16, - "max_total_query_cycles": 21, + "query_only_total_cycles": 32, + "query_only_cycles_per_query": 16.0, + "query_only_min_cycles": 14, + "query_only_max_cycles": 18, + "query_only_queries_per_cycle": 2 / 32, "mean_accept_to_first_result_cycles": 11.0, "mean_accept_to_last_result_cycles": 16.0, + "load_write_noise_cycles": 100, + "end_to_end_cycles": 140, + "end_to_end_cycles_per_query": 70.0, + "end_to_end_queries_per_cycle": 2 / 140, + "total_query_cycles": 32, + "mean_total_query_cycles": 16.0, + "min_total_query_cycles": 14, + "max_total_query_cycles": 18, "cycles_per_query": 16.0, - "queries_per_cycle": 2 / 37, + "queries_per_cycle": 2 / 32, }, } @@ -91,12 +135,20 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N with (tmp_path / "metrics.csv").open(newline="", encoding="utf-8") as f: row = next(csv.DictReader(f)) + assert row["query_only_cycles_per_query"] == "16.0" + assert row["query_only_total_cycles"] == "32" + assert row["query_only_queries_per_cycle"] == str(2 / 32) + assert row["load_write_noise_cycles"] == "100" + assert row["end_to_end_cycles"] == "140" + assert row["end_to_end_queries_per_cycle"] == str(2 / 140) assert row["cycles_per_query"] == "16.0" assert row["mean_accept_to_first_result_cycles"] == "11.0" assert row["mean_accept_to_last_result_cycles"] == "16.0" - assert row["queries_per_cycle"] == str(2 / 37) + assert row["queries_per_cycle"] == str(2 / 32) summary = (tmp_path / "summary.md").read_text(encoding="utf-8") assert "## Hardware performance" in summary - assert "cycles_per_query" in summary + assert "query-only cycles/query" in summary + assert "load/write/noise cycles" in summary + assert "end-to-end cycles/query" in summary assert "accept_to_last_result_cycles" in summary