feat(hw/sim): distinguish query-only and end-to-end performance cycles in retrieval benchmark

Add explicit separation between query-only cycles (accept→last) and end-to-end cycles
(load + write + noise + queries) in hardware retrieval benchmarks.

- Add query_only_cycles_per_query, load_write_noise_cycles, end_to_end_cycles metrics
- Refactor summarize_query_timings() to use accept_to_last_result_cycles as query-only base
- Add build_hardware_performance() to compute end-to-end performance separately
- Add current_sim_cycle() helper using cocotb get_sim_time
- Update CSV/Markdown outputs and RETRIEVAL_PERF_RESULT log format
- Update documentation to clarify cycle-counting methodology
- Update tests to cover new performance measurement logic
This commit is contained in:
2026-05-28 13:46:09 +08:00
parent 42d4a9728d
commit 97e53d44f8
4 changed files with 195 additions and 56 deletions

View File

@@ -41,6 +41,7 @@ just remote "python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retr
```bash ```bash
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz 512 just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz 512
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz 512
``` ```
## 2. 软件检索速度与质量 ## 2. 软件检索速度与质量
@@ -133,38 +134,40 @@ benchmark。实现位置
- `hw/sim/tests/top/utils.py::query_topk_once_with_latency` - `hw/sim/tests/top/utils.py::query_topk_once_with_latency`
- `hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py::cam_retrieval_benchmark` - `hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py::cam_retrieval_benchmark`
周期口径如下 周期口径如下。本节只报告Cocotb/Verilator仿真周期不将cycle直接换算为ns或queries/s。
| 指标 | 含义 | | 指标 | 含义 |
|---|---| |---|---|
| `accept_to_first_result_cycles` | query`query_valid && query_ready` 接受后,到首个 `result_valid` beat 的周期数。 | | `query_only_cycles_per_query` | 主指标。每个query`query_valid && query_ready`握手成功的时钟沿,到对应`result_valid && result_ready && result_last`完成的平均周期数。 |
| `accept_to_last_result_cycles` | query 被接受后,到 `result_last` 断言的周期数,即完整 Top-K 串行结果输出完成。 | | `query_only_total_cycles` | 所有query-only事务周期之和不包含装载、写入、噪声注入和查询间统计代码。 |
| `total_query_cycles` | 从拉高 `query_valid` 到 Top-K 输出完成的完整事务周期数。 | | `query_only_queries_per_cycle` | `num_queries / query_only_total_cycles`。 |
| `cycles_per_query` | 当前报告中的主指标,等于平均 `accept_to_last_result_cycles`。 | | `load_write_noise_cycles` | 写入CAM行以及可选写噪声注入阶段的周期数。无噪声模式下仍包含CAM行写入周期。 |
| `queries_per_cycle` | `num_queries / total_query_cycles`,用于观察仿真事务吞吐率。 | | `end_to_end_cycles` | 从开始写入数据集到最后一个query完成的完整benchmark硬件仿真周期数。 |
| `end_to_end_queries_per_cycle` | `num_queries / end_to_end_cycles`。 |
| `accept_to_first_result_cycles` | query被接受后到首个结果beat完成握手的平均周期数。 |
| `accept_to_last_result_cycles` | query被接受后`result_last`结果beat完成握手的平均周期数。 |
| `cycles_per_query``total_query_cycles``queries_per_cycle` | 兼容旧字段当前均为query-only口径不代表end-to-end。 |
选择 `accept_to_last_result_cycles` 作为主 `cycles_per_query` 的原因是Top-K 选择query-only的`accept→last`作为主cycles/query是因为Top-K检索只有在串行结果流输出到`result_last`并被接收后才算完整完成;仅用首个`result_valid`会低估实际Top-K查询事务成本。`load_write_noise_cycles``end_to_end_cycles`单独报告避免把非查询阶段混入query-only性能。
检索只有在串行结果流输出到 `result_last` 后才算完整完成;仅用首个
`result_valid` 会低估实际 Top-K 检索事务成本。
### CIFAR-10 无噪声硬件周期结果 ### 无噪声硬件周期结果
配置512 rows × 128 queries × 512-bit hash`TOPK_K=5``LANES=8` 配置512 rows×128 queries×512-bit hash`TOPK_K=5``LANES=8`
`WRITE_NOISE_EN=0` `WRITE_NOISE_EN=0`
| 数据集 | 模式 | 查询数 | accept→first | accept→last / cycles/query | total query cycles | queries/cycle | 状态 | | 数据集 | 模式 | 查询数 | query-only cycles/query | query-only total cycles | query-only queries/cycle | load/write/noise cycles | end-to-end cycles | end-to-end cycles/query | end-to-end queries/cycle | accept→first | accept→last | 状态 |
|---|---|---:|---:|---:|---:|---:|---| |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| CIFAR-10 | hardware no-noise | 128 | 1027.000000 | 1031.000000 | 132096 | 0.000968992 | pass | | CIFAR-10 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
| CIFAR-100 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
对应日志标记: 对应日志标记:
```text ```text
RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=132096 queries_per_cycle=0.000968992 status=pass RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 query_only_cycles_per_query=1031.000000 query_only_total_cycles=131968 query_only_queries_per_cycle=0.000969932 load_write_noise_cycles=1024 end_to_end_cycles=133628 end_to_end_cycles_per_query=1043.968750 end_to_end_queries_per_cycle=0.000957883 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=131968 queries_per_cycle=0.000969932 status=pass
``` ```
该结果说明:在当前 `NUM_ROWS=512, LANES=8, TOPK_K=5` 的硬件仿真配置下, 该结果说明:在当前 `NUM_ROWS=512, LANES=8, TOPK_K=5` 的硬件仿真配置下,
一次完整 Top-K 检索事务约为 **1031 cycles/query**。首个结果 beat 约在 一次完整Top-K查询事务的query-only成本为**1031cycles/query**。首个结果beat约在1027cycles后出现完整Top-K输出额外消耗约4cycles。完整benchmark端到端口径为**1043.968750cycles/query**其中数据写入阶段为1024cycles。
1027 cycles 后出现,完整 Top-K 输出额外消耗约 4 个周期。
> 注:以上数据来自 Verilator/Cocotb 仿真,不是 FPGA 板上实测。它可用于 > 注:以上数据来自 Verilator/Cocotb 仿真,不是 FPGA 板上实测。它可用于
> 架构级周期趋势分析,但不能直接等同于板级频率、吞吐或端到端系统延迟。 > 架构级周期趋势分析,但不能直接等同于板级频率、吞吐或端到端系统延迟。
@@ -180,8 +183,11 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
| Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 | | Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 |
| `ns_per_query` | 软件 Top-K 匹配阶段平均耗时;不含加载和写盘。 | | `ns_per_query` | 软件 Top-K 匹配阶段平均耗时;不含加载和写盘。 |
| `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 | | `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 |
| `cycles_per_query` | 硬件仿真中一次完整 Top-K 检索事务的平均周期数,当前采用 accept→last 口径。 | | `query_only_cycles_per_query` | 硬件仿真中一次完整Top-K查询事务的平均周期数,采用`query_valid && query_ready``result_last`完成握手口径。 |
| `queries_per_cycle` | 硬件仿真中完成 query 数除以query 事务周期数。 | | `query_only_queries_per_cycle` | 硬件仿真中完成query数除以query-only总事务周期数。 |
| `load_write_noise_cycles` | CAM行装载、写入及可选写噪声注入阶段周期数。 |
| `end_to_end_cycles` | 从开始写入数据到最后一个query完成的完整benchmark硬件仿真周期数。 |
| `end_to_end_queries_per_cycle` | 硬件仿真中完成query数除以end-to-end总周期数。 |
## 7. 当前结论 ## 7. 当前结论
@@ -191,9 +197,8 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
2. **当前软件基线速度约为 4.86k queries/s。** 2. **当前软件基线速度约为 4.86k queries/s。**
该结果来自 Python integer brute-force Hamming scan数据规模为 512 rows × 128 queries × 512 bits。 该结果来自 Python integer brute-force Hamming scan数据规模为 512 rows × 128 queries × 512 bits。
3. **硬件检索质量基准现在已经报告 cycles/query** 3. **硬件检索质量基准现在显式拆分query-only与end-to-end周期**
CIFAR-10 512-row/128-query 无噪声配置下,完整 Top-K 检索为 在CIFAR-10和CIFAR-100的512-row/128-query无噪声配置下完整Top-K查询事务均为**1031cycles/query**,首个结果为**1027cycles/query**;写入阶段为**1024cycles**,端到端为**1043.968750cycles/query**。
**1031 cycles/query**,首个结果为 **1027 cycles/query**
4. **写噪声对精确排序影响显著。** 4. **写噪声对精确排序影响显著。**
即使 Hit@K 保持较高Golden Match@K 也会快速下降,说明噪声首先破坏精确排序,再进一步破坏类别命中。 即使 Hit@K 保持较高Golden Match@K 也会快速下降,说明噪声首先破坏精确排序,再进一步破坏类别命中。
@@ -203,11 +208,10 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
## 8. 后续建议 ## 8. 后续建议
1. 对 CIFAR-100 也运行同样的硬件周期检测,补齐与软件质量表同尺度的硬件周期表。 1. `docs/exps` 中继续维护:
2.`docs/exps` 中继续维护:
- 软件检索速度表; - 软件检索速度表;
- 硬件无噪声一致性表; - 硬件无噪声一致性表;
- 硬件噪声鲁棒性表; - 硬件噪声鲁棒性表;
- 硬件 `cycles/query` 表。 - 硬件query-only、load/write/noise、end-to-end周期表。
3. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan以区分“朴素 Python baseline”和“优化软件 baseline”。 2. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan以区分“朴素 Python baseline”和“优化软件 baseline”。
4. 增加 `NUM_ROWS` sweep例如 512、1024、2048、4096 rows观察软件 brute-force scan 的线性增长趋势。 3. 增加 `NUM_ROWS` sweep例如 512、1024、2048、4096 rows观察软件 brute-force scan 的线性增长趋势。

View File

@@ -10,6 +10,7 @@ from pathlib import Path
import cocotb import cocotb
import numpy as np import numpy as np
from cocotb.clock import Clock from cocotb.clock import Clock
from cocotb.utils import get_sim_time
from model.ref_model import ( from model.ref_model import (
match_topk, match_topk,
@@ -32,6 +33,7 @@ DEFAULT_ROW_FLIP_BITS = 16
DEFAULT_QUERY_FLIP_BITS = 16 DEFAULT_QUERY_FLIP_BITS = 16
DEFAULT_SEED = 20260522 DEFAULT_SEED = 20260522
BENCHMARK_KS = (1, 5) BENCHMARK_KS = (1, 5)
SIM_CLOCK_PERIOD_NS = 10
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -214,6 +216,11 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]:
if not timings: if not timings:
return { return {
"num_queries": 0, "num_queries": 0,
"query_only_total_cycles": 0,
"query_only_cycles_per_query": 0.0,
"query_only_min_cycles": 0,
"query_only_max_cycles": 0,
"query_only_queries_per_cycle": 0.0,
"total_query_cycles": 0, "total_query_cycles": 0,
"mean_total_query_cycles": 0.0, "mean_total_query_cycles": 0.0,
"min_total_query_cycles": 0, "min_total_query_cycles": 0,
@@ -224,24 +231,57 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]:
"queries_per_cycle": 0.0, "queries_per_cycle": 0.0,
} }
total_cycles = sum(t.total_query_cycles for t in timings) total_cycles = sum(t.accept_to_last_result_cycles for t in timings)
total_first = sum(t.accept_to_first_result_cycles for t in timings) total_first = sum(t.accept_to_first_result_cycles for t in timings)
total_last = sum(t.accept_to_last_result_cycles for t in timings) total_last = sum(t.accept_to_last_result_cycles for t in timings)
count = len(timings) count = len(timings)
mean_last = total_last / float(count) mean_last = total_last / float(count)
queries_per_cycle = count / float(total_cycles) if total_cycles > 0 else 0.0
return { return {
"num_queries": count, "num_queries": count,
"query_only_total_cycles": total_cycles,
"query_only_cycles_per_query": mean_last,
"query_only_min_cycles": min(t.accept_to_last_result_cycles for t in timings),
"query_only_max_cycles": max(t.accept_to_last_result_cycles for t in timings),
"query_only_queries_per_cycle": queries_per_cycle,
# Backward-compatible aliases: query-only, not end-to-end.
"total_query_cycles": total_cycles, "total_query_cycles": total_cycles,
"mean_total_query_cycles": total_cycles / float(count), "mean_total_query_cycles": total_cycles / float(count),
"min_total_query_cycles": min(t.total_query_cycles for t in timings), "min_total_query_cycles": min(t.accept_to_last_result_cycles for t in timings),
"max_total_query_cycles": max(t.total_query_cycles for t in timings), "max_total_query_cycles": max(t.accept_to_last_result_cycles for t in timings),
"mean_accept_to_first_result_cycles": total_first / float(count), "mean_accept_to_first_result_cycles": total_first / float(count),
"mean_accept_to_last_result_cycles": mean_last, "mean_accept_to_last_result_cycles": mean_last,
"cycles_per_query": mean_last, "cycles_per_query": mean_last,
"queries_per_cycle": count / float(total_cycles) if total_cycles > 0 else 0.0, "queries_per_cycle": queries_per_cycle,
} }
def build_hardware_performance(
timings: list[QueryTiming],
*,
load_write_noise_cycles: int,
end_to_end_cycles: int,
) -> dict[str, float]:
performance = summarize_query_timings(timings)
num_queries = int(performance["num_queries"])
performance.update({
"load_write_noise_cycles": int(load_write_noise_cycles),
"end_to_end_cycles": int(end_to_end_cycles),
"end_to_end_cycles_per_query": (
float(end_to_end_cycles) / float(num_queries) if num_queries > 0 else 0.0
),
"end_to_end_queries_per_cycle": (
float(num_queries) / float(end_to_end_cycles) if end_to_end_cycles > 0 else 0.0
),
})
return performance
def current_sim_cycle() -> int:
"""Return the current benchmark clock cycle from simulator time."""
return int(get_sim_time("ns") // SIM_CLOCK_PERIOD_NS)
def output_dir_for(mode: str) -> Path: def output_dir_for(mode: str) -> Path:
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID")
if not run_id: if not run_id:
@@ -264,7 +304,11 @@ def write_outputs(out_dir: Path, result: dict) -> None:
"write_noise_en", "write_noise_rate_num", "write_noise_en", "write_noise_rate_num",
"write_noise_rate_den", "write_noise_rate_den",
"num_queries", "k", "macro_precision", "retrieval_recall", "macro_f1", "num_queries", "k", "macro_precision", "retrieval_recall", "macro_f1",
"recall@k", "exact_match_rate", "cycles_per_query", "recall@k", "exact_match_rate",
"query_only_cycles_per_query", "query_only_total_cycles",
"query_only_queries_per_cycle", "load_write_noise_cycles",
"end_to_end_cycles", "end_to_end_cycles_per_query",
"end_to_end_queries_per_cycle", "cycles_per_query",
"mean_accept_to_first_result_cycles", "mean_accept_to_last_result_cycles", "mean_accept_to_first_result_cycles", "mean_accept_to_last_result_cycles",
"mean_total_query_cycles", "total_query_cycles", "queries_per_cycle", "status", "mean_total_query_cycles", "total_query_cycles", "queries_per_cycle", "status",
] ]
@@ -289,6 +333,13 @@ def write_outputs(out_dir: Path, result: dict) -> None:
"macro_f1": metrics["macro_f1"], "macro_f1": metrics["macro_f1"],
"recall@k": metrics["recall@k"], "recall@k": metrics["recall@k"],
"exact_match_rate": metrics["exact_match_rate"], "exact_match_rate": metrics["exact_match_rate"],
"query_only_cycles_per_query": result.get("performance", {}).get("query_only_cycles_per_query", ""),
"query_only_total_cycles": result.get("performance", {}).get("query_only_total_cycles", ""),
"query_only_queries_per_cycle": result.get("performance", {}).get("query_only_queries_per_cycle", ""),
"load_write_noise_cycles": result.get("performance", {}).get("load_write_noise_cycles", ""),
"end_to_end_cycles": result.get("performance", {}).get("end_to_end_cycles", ""),
"end_to_end_cycles_per_query": result.get("performance", {}).get("end_to_end_cycles_per_query", ""),
"end_to_end_queries_per_cycle": result.get("performance", {}).get("end_to_end_queries_per_cycle", ""),
"cycles_per_query": result.get("performance", {}).get("cycles_per_query", ""), "cycles_per_query": result.get("performance", {}).get("cycles_per_query", ""),
"mean_accept_to_first_result_cycles": result.get("performance", {}).get( "mean_accept_to_first_result_cycles": result.get("performance", {}).get(
"mean_accept_to_first_result_cycles", "", "mean_accept_to_first_result_cycles", "",
@@ -313,11 +364,18 @@ def write_outputs(out_dir: Path, result: dict) -> None:
"", "",
"## Hardware performance", "## Hardware performance",
"", "",
f"- cycles_per_query: `{result.get('performance', {}).get('cycles_per_query', '')}`", f"- query-only cycles/query: `{result.get('performance', {}).get('query_only_cycles_per_query', '')}`",
f"- query-only total cycles: `{result.get('performance', {}).get('query_only_total_cycles', '')}`",
f"- query-only queries/cycle: `{result.get('performance', {}).get('query_only_queries_per_cycle', '')}`",
f"- load/write/noise cycles: `{result.get('performance', {}).get('load_write_noise_cycles', '')}`",
f"- end-to-end cycles: `{result.get('performance', {}).get('end_to_end_cycles', '')}`",
f"- end-to-end cycles/query: `{result.get('performance', {}).get('end_to_end_cycles_per_query', '')}`",
f"- end-to-end queries/cycle: `{result.get('performance', {}).get('end_to_end_queries_per_cycle', '')}`",
f"- cycles_per_query (compat, query-only): `{result.get('performance', {}).get('cycles_per_query', '')}`",
f"- accept_to_first_result_cycles: `{result.get('performance', {}).get('mean_accept_to_first_result_cycles', '')}`", f"- accept_to_first_result_cycles: `{result.get('performance', {}).get('mean_accept_to_first_result_cycles', '')}`",
f"- accept_to_last_result_cycles: `{result.get('performance', {}).get('mean_accept_to_last_result_cycles', '')}`", f"- accept_to_last_result_cycles: `{result.get('performance', {}).get('mean_accept_to_last_result_cycles', '')}`",
f"- total_query_cycles: `{result.get('performance', {}).get('total_query_cycles', '')}`", f"- total_query_cycles (compat, query-only): `{result.get('performance', {}).get('total_query_cycles', '')}`",
f"- queries_per_cycle: `{result.get('performance', {}).get('queries_per_cycle', '')}`", f"- queries_per_cycle (compat, query-only): `{result.get('performance', {}).get('queries_per_cycle', '')}`",
"", "",
"## Retrieval quality", "## Retrieval quality",
"", "",
@@ -338,7 +396,7 @@ def write_outputs(out_dir: Path, result: dict) -> None:
@cocotb.test() @cocotb.test()
async def cam_retrieval_benchmark(dut): async def cam_retrieval_benchmark(dut):
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start()) cocotb.start_soon(Clock(dut.clk, SIM_CLOCK_PERIOD_NS, unit="ns").start())
await reset_dut(dut) await reset_dut(dut)
num_rows = dut_num_rows(dut) num_rows = dut_num_rows(dut)
@@ -358,7 +416,11 @@ async def cam_retrieval_benchmark(dut):
dataset = load_retrieval_dataset_npz(dataset_path) dataset = load_retrieval_dataset_npz(dataset_path)
if len(dataset.rows) != num_rows: if len(dataset.rows) != num_rows:
raise AssertionError(f"artifact row count {len(dataset.rows)} must equal DUT NUM_ROWS {num_rows}") raise AssertionError(f"artifact row count {len(dataset.rows)} must equal DUT NUM_ROWS {num_rows}")
benchmark_start_cycle = current_sim_cycle()
load_write_noise_start_cycle = current_sim_cycle()
await write_rows(dut, dataset.rows) await write_rows(dut, dataset.rows)
load_write_noise_cycles = current_sim_cycle() - load_write_noise_start_cycle
accumulators = {k: MetricAccumulator() for k in BENCHMARK_KS} accumulators = {k: MetricAccumulator() for k in BENCHMARK_KS}
timings: list[QueryTiming] = [] timings: list[QueryTiming] = []
@@ -380,6 +442,8 @@ async def cam_retrieval_benchmark(dut):
label_hit = query_label in retrieved_labels label_hit = query_label in retrieved_labels
accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact) accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact)
end_to_end_cycles = current_sim_cycle() - benchmark_start_cycle
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}" run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}"
result = { result = {
"run_id": run_id, "run_id": run_id,
@@ -402,7 +466,11 @@ async def cam_retrieval_benchmark(dut):
"seed": dataset.seed, "seed": dataset.seed,
}, },
"metrics": {str(k): accumulators[k].as_dict() for k in BENCHMARK_KS}, "metrics": {str(k): accumulators[k].as_dict() for k in BENCHMARK_KS},
"performance": summarize_query_timings(timings), "performance": build_hardware_performance(
timings,
load_write_noise_cycles=load_write_noise_cycles,
end_to_end_cycles=end_to_end_cycles,
),
} }
out_dir = output_dir_for(mode) out_dir = output_dir_for(mode)
@@ -419,11 +487,22 @@ async def cam_retrieval_benchmark(dut):
performance = result["performance"] performance = result["performance"]
dut._log.info( dut._log.info(
"RETRIEVAL_PERF_RESULT mode=%s num_queries=%d cycles_per_query=%.6f " "RETRIEVAL_PERF_RESULT mode=%s num_queries=%d query_only_cycles_per_query=%.6f "
"query_only_total_cycles=%d query_only_queries_per_cycle=%.9f "
"load_write_noise_cycles=%d end_to_end_cycles=%d "
"end_to_end_cycles_per_query=%.6f end_to_end_queries_per_cycle=%.9f "
"cycles_per_query=%.6f "
"accept_to_first_result_cycles=%.6f accept_to_last_result_cycles=%.6f " "accept_to_first_result_cycles=%.6f accept_to_last_result_cycles=%.6f "
"total_query_cycles=%d queries_per_cycle=%.9f status=pass output_dir=%s", "total_query_cycles=%d queries_per_cycle=%.9f status=pass output_dir=%s",
mode, mode,
performance["num_queries"], performance["num_queries"],
performance["query_only_cycles_per_query"],
performance["query_only_total_cycles"],
performance["query_only_queries_per_cycle"],
performance["load_write_noise_cycles"],
performance["end_to_end_cycles"],
performance["end_to_end_cycles_per_query"],
performance["end_to_end_queries_per_cycle"],
performance["cycles_per_query"], performance["cycles_per_query"],
performance["mean_accept_to_first_result_cycles"], performance["mean_accept_to_first_result_cycles"],
performance["mean_accept_to_last_result_cycles"], performance["mean_accept_to_last_result_cycles"],

View File

@@ -216,7 +216,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
``timing`` 字段: ``timing`` 字段:
- accept_to_first_result_cycles: query 接受到首个 result_valid beat - accept_to_first_result_cycles: query 接受到首个 result_valid beat
- accept_to_last_result_cycles: query 接受到 result_last beatTop-K 完成) - accept_to_last_result_cycles: query 接受到 result_last beatTop-K 完成)
- total_query_cycles: 从拉高 query_valid 到 Top-K 完成的总上升沿数 - total_query_cycles: 纯查询事务周期,等于 accept_to_last_result_cycles
``query_ready`` 是组合信号,握手周期在上升沿前采样;结果信号在 ``query_ready`` 是组合信号,握手周期在上升沿前采样;结果信号在
ReadOnly settled phase 采样,避免重新引入 query_ready 采样时序问题。 ReadOnly settled phase 采样,避免重新引入 query_ready 采样时序问题。
@@ -236,8 +236,10 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
await RisingEdge(dut.clk) await RisingEdge(dut.clk)
edge_count += 1 edge_count += 1
q_valid = int(dut.query_valid.value)
q_ready = int(dut.query_ready.value) q_ready = int(dut.query_ready.value)
assert q_ready, "Query accept handshake was missed despite query_ready pre-wait" assert q_ready, "Query accept handshake was missed despite query_ready pre-wait"
assert q_valid, "Query valid deasserted before accept handshake"
accept_edge = edge_count accept_edge = edge_count
dut.query_valid.value = 0 dut.query_valid.value = 0
@@ -252,7 +254,9 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
await RisingEdge(dut.clk) await RisingEdge(dut.clk)
edge_count += 1 edge_count += 1
await ReadOnly() await ReadOnly()
if int(dut.result_valid.value): result_valid = int(dut.result_valid.value)
result_ready = int(dut.result_ready.value)
if result_valid and result_ready:
if first_result_edge is None: if first_result_edge is None:
first_result_edge = edge_count first_result_edge = edge_count
rank = int(dut.result_rank.value) rank = int(dut.result_rank.value)
@@ -282,7 +286,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
timing = { timing = {
"accept_to_first_result_cycles": int(first_result_edge - accept_edge), "accept_to_first_result_cycles": int(first_result_edge - accept_edge),
"accept_to_last_result_cycles": int(last_result_edge - accept_edge), "accept_to_last_result_cycles": int(last_result_edge - accept_edge),
"total_query_cycles": int(edge_count), "total_query_cycles": int(last_result_edge - accept_edge),
} }
return beats, beats[0][1], beats[0][2], score_debug, timing return beats, beats[0][1], beats[0][2], score_debug, timing

View File

@@ -11,38 +11,73 @@ if str(HW_SIM_DIR) not in sys.path:
from benchmarks.retrieval.test_retrieval_benchmark import ( # noqa: E402 from benchmarks.retrieval.test_retrieval_benchmark import ( # noqa: E402
QueryTiming, QueryTiming,
build_hardware_performance,
summarize_query_timings, summarize_query_timings,
write_outputs, write_outputs,
) )
def test_summarize_query_timings_reports_topk_completion_headline() -> None: def test_summarize_query_timings_uses_query_only_accept_to_last_cycles() -> None:
summary = summarize_query_timings([ summary = summarize_query_timings([
QueryTiming( QueryTiming(
accept_to_first_result_cycles=10, accept_to_first_result_cycles=10,
accept_to_last_result_cycles=14, accept_to_last_result_cycles=14,
total_query_cycles=16, total_query_cycles=14,
), ),
QueryTiming( QueryTiming(
accept_to_first_result_cycles=12, accept_to_first_result_cycles=12,
accept_to_last_result_cycles=18, accept_to_last_result_cycles=18,
total_query_cycles=21, total_query_cycles=18,
), ),
]) ])
assert summary == { assert summary == {
"num_queries": 2, "num_queries": 2,
"total_query_cycles": 37, "query_only_total_cycles": 32,
"mean_total_query_cycles": 18.5, "query_only_cycles_per_query": 16.0,
"min_total_query_cycles": 16, "query_only_min_cycles": 14,
"max_total_query_cycles": 21, "query_only_max_cycles": 18,
"query_only_queries_per_cycle": 2 / 32,
"mean_accept_to_first_result_cycles": 11.0, "mean_accept_to_first_result_cycles": 11.0,
"mean_accept_to_last_result_cycles": 16.0, "mean_accept_to_last_result_cycles": 16.0,
"total_query_cycles": 32,
"mean_total_query_cycles": 16.0,
"min_total_query_cycles": 14,
"max_total_query_cycles": 18,
"cycles_per_query": 16.0, "cycles_per_query": 16.0,
"queries_per_cycle": 2 / 37, "queries_per_cycle": 2 / 32,
} }
def test_build_hardware_performance_separates_query_and_end_to_end_cycles() -> None:
performance = build_hardware_performance(
[
QueryTiming(
accept_to_first_result_cycles=10,
accept_to_last_result_cycles=14,
total_query_cycles=14,
),
QueryTiming(
accept_to_first_result_cycles=12,
accept_to_last_result_cycles=18,
total_query_cycles=18,
),
],
load_write_noise_cycles=100,
end_to_end_cycles=140,
)
assert performance["query_only_total_cycles"] == 32
assert performance["query_only_cycles_per_query"] == 16.0
assert performance["query_only_queries_per_cycle"] == 2 / 32
assert performance["load_write_noise_cycles"] == 100
assert performance["end_to_end_cycles"] == 140
assert performance["end_to_end_cycles_per_query"] == 70.0
assert performance["end_to_end_queries_per_cycle"] == 2 / 140
assert performance["cycles_per_query"] == performance["query_only_cycles_per_query"]
assert performance["queries_per_cycle"] == performance["query_only_queries_per_cycle"]
def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> None: def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> None:
result = { result = {
"run_id": "test-run", "run_id": "test-run",
@@ -75,14 +110,23 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N
}, },
"performance": { "performance": {
"num_queries": 2, "num_queries": 2,
"total_query_cycles": 37, "query_only_total_cycles": 32,
"mean_total_query_cycles": 18.5, "query_only_cycles_per_query": 16.0,
"min_total_query_cycles": 16, "query_only_min_cycles": 14,
"max_total_query_cycles": 21, "query_only_max_cycles": 18,
"query_only_queries_per_cycle": 2 / 32,
"mean_accept_to_first_result_cycles": 11.0, "mean_accept_to_first_result_cycles": 11.0,
"mean_accept_to_last_result_cycles": 16.0, "mean_accept_to_last_result_cycles": 16.0,
"load_write_noise_cycles": 100,
"end_to_end_cycles": 140,
"end_to_end_cycles_per_query": 70.0,
"end_to_end_queries_per_cycle": 2 / 140,
"total_query_cycles": 32,
"mean_total_query_cycles": 16.0,
"min_total_query_cycles": 14,
"max_total_query_cycles": 18,
"cycles_per_query": 16.0, "cycles_per_query": 16.0,
"queries_per_cycle": 2 / 37, "queries_per_cycle": 2 / 32,
}, },
} }
@@ -91,12 +135,20 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N
with (tmp_path / "metrics.csv").open(newline="", encoding="utf-8") as f: with (tmp_path / "metrics.csv").open(newline="", encoding="utf-8") as f:
row = next(csv.DictReader(f)) row = next(csv.DictReader(f))
assert row["query_only_cycles_per_query"] == "16.0"
assert row["query_only_total_cycles"] == "32"
assert row["query_only_queries_per_cycle"] == str(2 / 32)
assert row["load_write_noise_cycles"] == "100"
assert row["end_to_end_cycles"] == "140"
assert row["end_to_end_queries_per_cycle"] == str(2 / 140)
assert row["cycles_per_query"] == "16.0" assert row["cycles_per_query"] == "16.0"
assert row["mean_accept_to_first_result_cycles"] == "11.0" assert row["mean_accept_to_first_result_cycles"] == "11.0"
assert row["mean_accept_to_last_result_cycles"] == "16.0" assert row["mean_accept_to_last_result_cycles"] == "16.0"
assert row["queries_per_cycle"] == str(2 / 37) assert row["queries_per_cycle"] == str(2 / 32)
summary = (tmp_path / "summary.md").read_text(encoding="utf-8") summary = (tmp_path / "summary.md").read_text(encoding="utf-8")
assert "## Hardware performance" in summary assert "## Hardware performance" in summary
assert "cycles_per_query" in summary assert "query-only cycles/query" in summary
assert "load/write/noise cycles" in summary
assert "end-to-end cycles/query" in summary
assert "accept_to_last_result_cycles" in summary assert "accept_to_last_result_cycles" in summary