mirror of
https://github.com/SikongJueluo/Mini-Nav.git
synced 2026-07-12 20:15:31 +08:00
feat(hw/sim): distinguish query-only and end-to-end performance cycles in retrieval benchmark
Add explicit separation between query-only cycles (accept→last) and end-to-end cycles (load + write + noise + queries) in hardware retrieval benchmarks. - Add query_only_cycles_per_query, load_write_noise_cycles, end_to_end_cycles metrics - Refactor summarize_query_timings() to use accept_to_last_result_cycles as query-only base - Add build_hardware_performance() to compute end-to-end performance separately - Add current_sim_cycle() helper using cocotb get_sim_time - Update CSV/Markdown outputs and RETRIEVAL_PERF_RESULT log format - Update documentation to clarify cycle-counting methodology - Update tests to cover new performance measurement logic
This commit is contained in:
@@ -41,6 +41,7 @@ just remote "python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retr
|
||||
|
||||
```bash
|
||||
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz 512
|
||||
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz 512
|
||||
```
|
||||
|
||||
## 2. 软件检索速度与质量
|
||||
@@ -133,38 +134,40 @@ benchmark。实现位置：
|
||||
- `hw/sim/tests/top/utils.py::query_topk_once_with_latency`
|
||||
- `hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py::cam_retrieval_benchmark`
|
||||
|
||||
周期口径如下：
|
||||
周期口径如下。本节只报告Cocotb/Verilator仿真周期，不将cycle直接换算为ns或queries/s。
|
||||
|
||||
| 指标 | 含义 |
|
||||
|---|---|
|
||||
| `accept_to_first_result_cycles` | query 被 `query_valid && query_ready` 接受后，到首个 `result_valid` beat 的周期数。 |
|
||||
| `accept_to_last_result_cycles` | query 被接受后，到 `result_last` 断言的周期数，即完整 Top-K 串行结果输出完成。 |
|
||||
| `total_query_cycles` | 从拉高 `query_valid` 到 Top-K 输出完成的完整事务周期数。 |
|
||||
| `cycles_per_query` | 当前报告中的主指标，等于平均 `accept_to_last_result_cycles`。 |
|
||||
| `queries_per_cycle` | `num_queries / total_query_cycles`，用于观察仿真事务吞吐率。 |
|
||||
| `query_only_cycles_per_query` | 主指标。每个query从`query_valid && query_ready`握手成功的时钟沿，到对应`result_valid && result_ready && result_last`完成的平均周期数。 |
|
||||
| `query_only_total_cycles` | 所有query-only事务周期之和，不包含装载、写入、噪声注入和查询间统计代码。 |
|
||||
| `query_only_queries_per_cycle` | `num_queries / query_only_total_cycles`。 |
|
||||
| `load_write_noise_cycles` | 写入CAM行以及可选写噪声注入阶段的周期数。无噪声模式下仍包含CAM行写入周期。 |
|
||||
| `end_to_end_cycles` | 从开始写入数据集到最后一个query完成的完整benchmark硬件仿真周期数。 |
|
||||
| `end_to_end_queries_per_cycle` | `num_queries / end_to_end_cycles`。 |
|
||||
| `accept_to_first_result_cycles` | query被接受后，到首个结果beat完成握手的平均周期数。 |
|
||||
| `accept_to_last_result_cycles` | query被接受后，到`result_last`结果beat完成握手的平均周期数。 |
|
||||
| `cycles_per_query`、`total_query_cycles`、`queries_per_cycle` | 兼容旧字段，当前均为query-only口径，不代表end-to-end。 |
|
||||
|
||||
选择 `accept_to_last_result_cycles` 作为主 `cycles_per_query` 的原因是：Top-K
|
||||
检索只有在串行结果流输出到 `result_last` 后才算完整完成；仅用首个
|
||||
`result_valid` 会低估实际 Top-K 检索事务成本。
|
||||
选择query-only的`accept→last`作为主cycles/query，是因为Top-K检索只有在串行结果流输出到`result_last`并被接收后才算完整完成；仅用首个`result_valid`会低估实际Top-K查询事务成本。`load_write_noise_cycles`和`end_to_end_cycles`单独报告，避免把非查询阶段混入query-only性能。
|
||||
|
||||
### CIFAR-10 无噪声硬件周期结果
|
||||
### 无噪声硬件周期结果
|
||||
|
||||
配置：512 rows × 128 queries × 512-bit hash，`TOPK_K=5`，`LANES=8`，
|
||||
配置：512 rows×128 queries×512-bit hash，`TOPK_K=5`，`LANES=8`，
|
||||
`WRITE_NOISE_EN=0`。
|
||||
|
||||
| 数据集 | 模式 | 查询数 | accept→first | accept→last / cycles/query | total query cycles | queries/cycle | 状态 |
|
||||
|---|---|---:|---:|---:|---:|---:|---|
|
||||
| CIFAR-10 | hardware no-noise | 128 | 1027.000000 | 1031.000000 | 132096 | 0.000968992 | pass |
|
||||
| 数据集 | 模式 | 查询数 | query-only cycles/query | query-only total cycles | query-only queries/cycle | load/write/noise cycles | end-to-end cycles | end-to-end cycles/query | end-to-end queries/cycle | accept→first | accept→last | 状态 |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| CIFAR-10 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
|
||||
| CIFAR-100 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
|
||||
|
||||
对应日志标记：
|
||||
|
||||
```text
|
||||
RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=132096 queries_per_cycle=0.000968992 status=pass
|
||||
RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 query_only_cycles_per_query=1031.000000 query_only_total_cycles=131968 query_only_queries_per_cycle=0.000969932 load_write_noise_cycles=1024 end_to_end_cycles=133628 end_to_end_cycles_per_query=1043.968750 end_to_end_queries_per_cycle=0.000957883 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=131968 queries_per_cycle=0.000969932 status=pass
|
||||
```
|
||||
|
||||
该结果说明：在当前 `NUM_ROWS=512, LANES=8, TOPK_K=5` 的硬件仿真配置下，
|
||||
一次完整 Top-K 检索事务约为 **1031 cycles/query**。首个结果 beat 约在
|
||||
1027 cycles 后出现，完整 Top-K 输出额外消耗约 4 个周期。
|
||||
一次完整Top-K查询事务的query-only成本为**1031cycles/query**。首个结果beat约在1027cycles后出现，完整Top-K输出额外消耗约4cycles。完整benchmark端到端口径为**1043.968750cycles/query**，其中数据写入阶段为1024cycles。
|
||||
|
||||
> 注：以上数据来自 Verilator/Cocotb 仿真，不是 FPGA 板上实测。它可用于
|
||||
> 架构级周期趋势分析，但不能直接等同于板级频率、吞吐或端到端系统延迟。
|
||||
@@ -180,8 +183,11 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
|
||||
| Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 |
|
||||
| `ns_per_query` | 软件 Top-K 匹配阶段平均耗时；不含加载和写盘。 |
|
||||
| `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 |
|
||||
| `cycles_per_query` | 硬件仿真中一次完整 Top-K 检索事务的平均周期数，当前采用 accept→last 口径。 |
|
||||
| `queries_per_cycle` | 硬件仿真中完成 query 数除以总 query 事务周期数。 |
|
||||
| `query_only_cycles_per_query` | 硬件仿真中一次完整Top-K查询事务的平均周期数，采用`query_valid && query_ready`到`result_last`完成握手口径。 |
|
||||
| `query_only_queries_per_cycle` | 硬件仿真中完成query数除以query-only总事务周期数。 |
|
||||
| `load_write_noise_cycles` | CAM行装载、写入及可选写噪声注入阶段周期数。 |
|
||||
| `end_to_end_cycles` | 从开始写入数据到最后一个query完成的完整benchmark硬件仿真周期数。 |
|
||||
| `end_to_end_queries_per_cycle` | 硬件仿真中完成query数除以end-to-end总周期数。 |
|
||||
|
||||
## 7. 当前结论
|
||||
|
||||
@@ -191,9 +197,8 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
|
||||
2. **当前软件基线速度约为 4.86k queries/s。**
|
||||
该结果来自 Python integer brute-force Hamming scan，数据规模为 512 rows × 128 queries × 512 bits。
|
||||
|
||||
3. **硬件检索质量基准现在已经报告 cycles/query。**
|
||||
在 CIFAR-10 512-row/128-query 无噪声配置下，完整 Top-K 检索为
|
||||
**1031 cycles/query**，首个结果为 **1027 cycles/query**。
|
||||
3. **硬件检索质量基准现在显式拆分query-only与end-to-end周期。**
|
||||
在CIFAR-10和CIFAR-100的512-row/128-query无噪声配置下，完整Top-K查询事务均为**1031cycles/query**，首个结果为**1027cycles/query**；写入阶段为**1024cycles**，端到端为**1043.968750cycles/query**。
|
||||
|
||||
4. **写噪声对精确排序影响显著。**
|
||||
即使 Hit@K 保持较高，Golden Match@K 也会快速下降，说明噪声首先破坏精确排序，再进一步破坏类别命中。
|
||||
@@ -203,11 +208,10 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
|
||||
|
||||
## 8. 后续建议
|
||||
|
||||
1. 对 CIFAR-100 也运行同样的硬件周期检测，补齐与软件质量表同尺度的硬件周期表。
|
||||
2. 在 `docs/exps` 中继续维护：
|
||||
1. 在 `docs/exps` 中继续维护：
|
||||
- 软件检索速度表；
|
||||
- 硬件无噪声一致性表；
|
||||
- 硬件噪声鲁棒性表；
|
||||
- 硬件 `cycles/query` 表。
|
||||
3. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan，以区分“朴素 Python baseline”和“优化软件 baseline”。
|
||||
4. 增加 `NUM_ROWS` sweep：例如 512、1024、2048、4096 rows，观察软件 brute-force scan 的线性增长趋势。
|
||||
- 硬件query-only、load/write/noise、end-to-end周期表。
|
||||
2. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan，以区分“朴素 Python baseline”和“优化软件 baseline”。
|
||||
3. 增加 `NUM_ROWS` sweep：例如 512、1024、2048、4096 rows，观察软件 brute-force scan 的线性增长趋势。
|
||||
|
||||
@@ -10,6 +10,7 @@ from pathlib import Path
|
||||
import cocotb
|
||||
import numpy as np
|
||||
from cocotb.clock import Clock
|
||||
from cocotb.utils import get_sim_time
|
||||
|
||||
from model.ref_model import (
|
||||
match_topk,
|
||||
@@ -32,6 +33,7 @@ DEFAULT_ROW_FLIP_BITS = 16
|
||||
DEFAULT_QUERY_FLIP_BITS = 16
|
||||
DEFAULT_SEED = 20260522
|
||||
BENCHMARK_KS = (1, 5)
|
||||
SIM_CLOCK_PERIOD_NS = 10
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -214,6 +216,11 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]:
|
||||
if not timings:
|
||||
return {
|
||||
"num_queries": 0,
|
||||
"query_only_total_cycles": 0,
|
||||
"query_only_cycles_per_query": 0.0,
|
||||
"query_only_min_cycles": 0,
|
||||
"query_only_max_cycles": 0,
|
||||
"query_only_queries_per_cycle": 0.0,
|
||||
"total_query_cycles": 0,
|
||||
"mean_total_query_cycles": 0.0,
|
||||
"min_total_query_cycles": 0,
|
||||
@@ -224,24 +231,57 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]:
|
||||
"queries_per_cycle": 0.0,
|
||||
}
|
||||
|
||||
total_cycles = sum(t.total_query_cycles for t in timings)
|
||||
total_cycles = sum(t.accept_to_last_result_cycles for t in timings)
|
||||
total_first = sum(t.accept_to_first_result_cycles for t in timings)
|
||||
total_last = sum(t.accept_to_last_result_cycles for t in timings)
|
||||
count = len(timings)
|
||||
mean_last = total_last / float(count)
|
||||
queries_per_cycle = count / float(total_cycles) if total_cycles > 0 else 0.0
|
||||
return {
|
||||
"num_queries": count,
|
||||
"query_only_total_cycles": total_cycles,
|
||||
"query_only_cycles_per_query": mean_last,
|
||||
"query_only_min_cycles": min(t.accept_to_last_result_cycles for t in timings),
|
||||
"query_only_max_cycles": max(t.accept_to_last_result_cycles for t in timings),
|
||||
"query_only_queries_per_cycle": queries_per_cycle,
|
||||
# Backward-compatible aliases: query-only, not end-to-end.
|
||||
"total_query_cycles": total_cycles,
|
||||
"mean_total_query_cycles": total_cycles / float(count),
|
||||
"min_total_query_cycles": min(t.total_query_cycles for t in timings),
|
||||
"max_total_query_cycles": max(t.total_query_cycles for t in timings),
|
||||
"min_total_query_cycles": min(t.accept_to_last_result_cycles for t in timings),
|
||||
"max_total_query_cycles": max(t.accept_to_last_result_cycles for t in timings),
|
||||
"mean_accept_to_first_result_cycles": total_first / float(count),
|
||||
"mean_accept_to_last_result_cycles": mean_last,
|
||||
"cycles_per_query": mean_last,
|
||||
"queries_per_cycle": count / float(total_cycles) if total_cycles > 0 else 0.0,
|
||||
"queries_per_cycle": queries_per_cycle,
|
||||
}
|
||||
|
||||
|
||||
def build_hardware_performance(
|
||||
timings: list[QueryTiming],
|
||||
*,
|
||||
load_write_noise_cycles: int,
|
||||
end_to_end_cycles: int,
|
||||
) -> dict[str, float]:
|
||||
performance = summarize_query_timings(timings)
|
||||
num_queries = int(performance["num_queries"])
|
||||
performance.update({
|
||||
"load_write_noise_cycles": int(load_write_noise_cycles),
|
||||
"end_to_end_cycles": int(end_to_end_cycles),
|
||||
"end_to_end_cycles_per_query": (
|
||||
float(end_to_end_cycles) / float(num_queries) if num_queries > 0 else 0.0
|
||||
),
|
||||
"end_to_end_queries_per_cycle": (
|
||||
float(num_queries) / float(end_to_end_cycles) if end_to_end_cycles > 0 else 0.0
|
||||
),
|
||||
})
|
||||
return performance
|
||||
|
||||
|
||||
def current_sim_cycle() -> int:
|
||||
"""Return the current benchmark clock cycle from simulator time."""
|
||||
return int(get_sim_time("ns") // SIM_CLOCK_PERIOD_NS)
|
||||
|
||||
|
||||
def output_dir_for(mode: str) -> Path:
|
||||
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID")
|
||||
if not run_id:
|
||||
@@ -264,7 +304,11 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
||||
"write_noise_en", "write_noise_rate_num",
|
||||
"write_noise_rate_den",
|
||||
"num_queries", "k", "macro_precision", "retrieval_recall", "macro_f1",
|
||||
"recall@k", "exact_match_rate", "cycles_per_query",
|
||||
"recall@k", "exact_match_rate",
|
||||
"query_only_cycles_per_query", "query_only_total_cycles",
|
||||
"query_only_queries_per_cycle", "load_write_noise_cycles",
|
||||
"end_to_end_cycles", "end_to_end_cycles_per_query",
|
||||
"end_to_end_queries_per_cycle", "cycles_per_query",
|
||||
"mean_accept_to_first_result_cycles", "mean_accept_to_last_result_cycles",
|
||||
"mean_total_query_cycles", "total_query_cycles", "queries_per_cycle", "status",
|
||||
]
|
||||
@@ -289,6 +333,13 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
||||
"macro_f1": metrics["macro_f1"],
|
||||
"recall@k": metrics["recall@k"],
|
||||
"exact_match_rate": metrics["exact_match_rate"],
|
||||
"query_only_cycles_per_query": result.get("performance", {}).get("query_only_cycles_per_query", ""),
|
||||
"query_only_total_cycles": result.get("performance", {}).get("query_only_total_cycles", ""),
|
||||
"query_only_queries_per_cycle": result.get("performance", {}).get("query_only_queries_per_cycle", ""),
|
||||
"load_write_noise_cycles": result.get("performance", {}).get("load_write_noise_cycles", ""),
|
||||
"end_to_end_cycles": result.get("performance", {}).get("end_to_end_cycles", ""),
|
||||
"end_to_end_cycles_per_query": result.get("performance", {}).get("end_to_end_cycles_per_query", ""),
|
||||
"end_to_end_queries_per_cycle": result.get("performance", {}).get("end_to_end_queries_per_cycle", ""),
|
||||
"cycles_per_query": result.get("performance", {}).get("cycles_per_query", ""),
|
||||
"mean_accept_to_first_result_cycles": result.get("performance", {}).get(
|
||||
"mean_accept_to_first_result_cycles", "",
|
||||
@@ -313,11 +364,18 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
||||
"",
|
||||
"## Hardware performance",
|
||||
"",
|
||||
f"- cycles_per_query: `{result.get('performance', {}).get('cycles_per_query', '')}`",
|
||||
f"- query-only cycles/query: `{result.get('performance', {}).get('query_only_cycles_per_query', '')}`",
|
||||
f"- query-only total cycles: `{result.get('performance', {}).get('query_only_total_cycles', '')}`",
|
||||
f"- query-only queries/cycle: `{result.get('performance', {}).get('query_only_queries_per_cycle', '')}`",
|
||||
f"- load/write/noise cycles: `{result.get('performance', {}).get('load_write_noise_cycles', '')}`",
|
||||
f"- end-to-end cycles: `{result.get('performance', {}).get('end_to_end_cycles', '')}`",
|
||||
f"- end-to-end cycles/query: `{result.get('performance', {}).get('end_to_end_cycles_per_query', '')}`",
|
||||
f"- end-to-end queries/cycle: `{result.get('performance', {}).get('end_to_end_queries_per_cycle', '')}`",
|
||||
f"- cycles_per_query (compat, query-only): `{result.get('performance', {}).get('cycles_per_query', '')}`",
|
||||
f"- accept_to_first_result_cycles: `{result.get('performance', {}).get('mean_accept_to_first_result_cycles', '')}`",
|
||||
f"- accept_to_last_result_cycles: `{result.get('performance', {}).get('mean_accept_to_last_result_cycles', '')}`",
|
||||
f"- total_query_cycles: `{result.get('performance', {}).get('total_query_cycles', '')}`",
|
||||
f"- queries_per_cycle: `{result.get('performance', {}).get('queries_per_cycle', '')}`",
|
||||
f"- total_query_cycles (compat, query-only): `{result.get('performance', {}).get('total_query_cycles', '')}`",
|
||||
f"- queries_per_cycle (compat, query-only): `{result.get('performance', {}).get('queries_per_cycle', '')}`",
|
||||
"",
|
||||
"## Retrieval quality",
|
||||
"",
|
||||
@@ -338,7 +396,7 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
||||
|
||||
@cocotb.test()
|
||||
async def cam_retrieval_benchmark(dut):
|
||||
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
|
||||
cocotb.start_soon(Clock(dut.clk, SIM_CLOCK_PERIOD_NS, unit="ns").start())
|
||||
await reset_dut(dut)
|
||||
|
||||
num_rows = dut_num_rows(dut)
|
||||
@@ -358,7 +416,11 @@ async def cam_retrieval_benchmark(dut):
|
||||
dataset = load_retrieval_dataset_npz(dataset_path)
|
||||
if len(dataset.rows) != num_rows:
|
||||
raise AssertionError(f"artifact row count {len(dataset.rows)} must equal DUT NUM_ROWS {num_rows}")
|
||||
|
||||
benchmark_start_cycle = current_sim_cycle()
|
||||
load_write_noise_start_cycle = current_sim_cycle()
|
||||
await write_rows(dut, dataset.rows)
|
||||
load_write_noise_cycles = current_sim_cycle() - load_write_noise_start_cycle
|
||||
|
||||
accumulators = {k: MetricAccumulator() for k in BENCHMARK_KS}
|
||||
timings: list[QueryTiming] = []
|
||||
@@ -380,6 +442,8 @@ async def cam_retrieval_benchmark(dut):
|
||||
label_hit = query_label in retrieved_labels
|
||||
accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact)
|
||||
|
||||
end_to_end_cycles = current_sim_cycle() - benchmark_start_cycle
|
||||
|
||||
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}"
|
||||
result = {
|
||||
"run_id": run_id,
|
||||
@@ -402,7 +466,11 @@ async def cam_retrieval_benchmark(dut):
|
||||
"seed": dataset.seed,
|
||||
},
|
||||
"metrics": {str(k): accumulators[k].as_dict() for k in BENCHMARK_KS},
|
||||
"performance": summarize_query_timings(timings),
|
||||
"performance": build_hardware_performance(
|
||||
timings,
|
||||
load_write_noise_cycles=load_write_noise_cycles,
|
||||
end_to_end_cycles=end_to_end_cycles,
|
||||
),
|
||||
}
|
||||
|
||||
out_dir = output_dir_for(mode)
|
||||
@@ -419,11 +487,22 @@ async def cam_retrieval_benchmark(dut):
|
||||
|
||||
performance = result["performance"]
|
||||
dut._log.info(
|
||||
"RETRIEVAL_PERF_RESULT mode=%s num_queries=%d cycles_per_query=%.6f "
|
||||
"RETRIEVAL_PERF_RESULT mode=%s num_queries=%d query_only_cycles_per_query=%.6f "
|
||||
"query_only_total_cycles=%d query_only_queries_per_cycle=%.9f "
|
||||
"load_write_noise_cycles=%d end_to_end_cycles=%d "
|
||||
"end_to_end_cycles_per_query=%.6f end_to_end_queries_per_cycle=%.9f "
|
||||
"cycles_per_query=%.6f "
|
||||
"accept_to_first_result_cycles=%.6f accept_to_last_result_cycles=%.6f "
|
||||
"total_query_cycles=%d queries_per_cycle=%.9f status=pass output_dir=%s",
|
||||
mode,
|
||||
performance["num_queries"],
|
||||
performance["query_only_cycles_per_query"],
|
||||
performance["query_only_total_cycles"],
|
||||
performance["query_only_queries_per_cycle"],
|
||||
performance["load_write_noise_cycles"],
|
||||
performance["end_to_end_cycles"],
|
||||
performance["end_to_end_cycles_per_query"],
|
||||
performance["end_to_end_queries_per_cycle"],
|
||||
performance["cycles_per_query"],
|
||||
performance["mean_accept_to_first_result_cycles"],
|
||||
performance["mean_accept_to_last_result_cycles"],
|
||||
|
||||
@@ -216,7 +216,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
||||
``timing`` 字段：
|
||||
- accept_to_first_result_cycles: query 接受到首个 result_valid beat
|
||||
- accept_to_last_result_cycles: query 接受到 result_last beat（Top-K 完成）
|
||||
- total_query_cycles: 从拉高 query_valid 到 Top-K 完成的总上升沿数
|
||||
- total_query_cycles: 纯查询事务周期，等于 accept_to_last_result_cycles
|
||||
|
||||
``query_ready`` 是组合信号，握手周期在上升沿前采样；结果信号在
|
||||
ReadOnly settled phase 采样，避免重新引入 query_ready 采样时序问题。
|
||||
@@ -236,8 +236,10 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
||||
|
||||
await RisingEdge(dut.clk)
|
||||
edge_count += 1
|
||||
q_valid = int(dut.query_valid.value)
|
||||
q_ready = int(dut.query_ready.value)
|
||||
assert q_ready, "Query accept handshake was missed despite query_ready pre-wait"
|
||||
assert q_valid, "Query valid deasserted before accept handshake"
|
||||
accept_edge = edge_count
|
||||
dut.query_valid.value = 0
|
||||
|
||||
@@ -252,7 +254,9 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
||||
await RisingEdge(dut.clk)
|
||||
edge_count += 1
|
||||
await ReadOnly()
|
||||
if int(dut.result_valid.value):
|
||||
result_valid = int(dut.result_valid.value)
|
||||
result_ready = int(dut.result_ready.value)
|
||||
if result_valid and result_ready:
|
||||
if first_result_edge is None:
|
||||
first_result_edge = edge_count
|
||||
rank = int(dut.result_rank.value)
|
||||
@@ -282,7 +286,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
||||
timing = {
|
||||
"accept_to_first_result_cycles": int(first_result_edge - accept_edge),
|
||||
"accept_to_last_result_cycles": int(last_result_edge - accept_edge),
|
||||
"total_query_cycles": int(edge_count),
|
||||
"total_query_cycles": int(last_result_edge - accept_edge),
|
||||
}
|
||||
return beats, beats[0][1], beats[0][2], score_debug, timing
|
||||
|
||||
|
||||
@@ -11,38 +11,73 @@ if str(HW_SIM_DIR) not in sys.path:
|
||||
|
||||
from benchmarks.retrieval.test_retrieval_benchmark import ( # noqa: E402
|
||||
QueryTiming,
|
||||
build_hardware_performance,
|
||||
summarize_query_timings,
|
||||
write_outputs,
|
||||
)
|
||||
|
||||
|
||||
def test_summarize_query_timings_reports_topk_completion_headline() -> None:
|
||||
def test_summarize_query_timings_uses_query_only_accept_to_last_cycles() -> None:
|
||||
summary = summarize_query_timings([
|
||||
QueryTiming(
|
||||
accept_to_first_result_cycles=10,
|
||||
accept_to_last_result_cycles=14,
|
||||
total_query_cycles=16,
|
||||
total_query_cycles=14,
|
||||
),
|
||||
QueryTiming(
|
||||
accept_to_first_result_cycles=12,
|
||||
accept_to_last_result_cycles=18,
|
||||
total_query_cycles=21,
|
||||
total_query_cycles=18,
|
||||
),
|
||||
])
|
||||
|
||||
assert summary == {
|
||||
"num_queries": 2,
|
||||
"total_query_cycles": 37,
|
||||
"mean_total_query_cycles": 18.5,
|
||||
"min_total_query_cycles": 16,
|
||||
"max_total_query_cycles": 21,
|
||||
"query_only_total_cycles": 32,
|
||||
"query_only_cycles_per_query": 16.0,
|
||||
"query_only_min_cycles": 14,
|
||||
"query_only_max_cycles": 18,
|
||||
"query_only_queries_per_cycle": 2 / 32,
|
||||
"mean_accept_to_first_result_cycles": 11.0,
|
||||
"mean_accept_to_last_result_cycles": 16.0,
|
||||
"total_query_cycles": 32,
|
||||
"mean_total_query_cycles": 16.0,
|
||||
"min_total_query_cycles": 14,
|
||||
"max_total_query_cycles": 18,
|
||||
"cycles_per_query": 16.0,
|
||||
"queries_per_cycle": 2 / 37,
|
||||
"queries_per_cycle": 2 / 32,
|
||||
}
|
||||
|
||||
|
||||
def test_build_hardware_performance_separates_query_and_end_to_end_cycles() -> None:
|
||||
performance = build_hardware_performance(
|
||||
[
|
||||
QueryTiming(
|
||||
accept_to_first_result_cycles=10,
|
||||
accept_to_last_result_cycles=14,
|
||||
total_query_cycles=14,
|
||||
),
|
||||
QueryTiming(
|
||||
accept_to_first_result_cycles=12,
|
||||
accept_to_last_result_cycles=18,
|
||||
total_query_cycles=18,
|
||||
),
|
||||
],
|
||||
load_write_noise_cycles=100,
|
||||
end_to_end_cycles=140,
|
||||
)
|
||||
|
||||
assert performance["query_only_total_cycles"] == 32
|
||||
assert performance["query_only_cycles_per_query"] == 16.0
|
||||
assert performance["query_only_queries_per_cycle"] == 2 / 32
|
||||
assert performance["load_write_noise_cycles"] == 100
|
||||
assert performance["end_to_end_cycles"] == 140
|
||||
assert performance["end_to_end_cycles_per_query"] == 70.0
|
||||
assert performance["end_to_end_queries_per_cycle"] == 2 / 140
|
||||
assert performance["cycles_per_query"] == performance["query_only_cycles_per_query"]
|
||||
assert performance["queries_per_cycle"] == performance["query_only_queries_per_cycle"]
|
||||
|
||||
|
||||
def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> None:
|
||||
result = {
|
||||
"run_id": "test-run",
|
||||
@@ -75,14 +110,23 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N
|
||||
},
|
||||
"performance": {
|
||||
"num_queries": 2,
|
||||
"total_query_cycles": 37,
|
||||
"mean_total_query_cycles": 18.5,
|
||||
"min_total_query_cycles": 16,
|
||||
"max_total_query_cycles": 21,
|
||||
"query_only_total_cycles": 32,
|
||||
"query_only_cycles_per_query": 16.0,
|
||||
"query_only_min_cycles": 14,
|
||||
"query_only_max_cycles": 18,
|
||||
"query_only_queries_per_cycle": 2 / 32,
|
||||
"mean_accept_to_first_result_cycles": 11.0,
|
||||
"mean_accept_to_last_result_cycles": 16.0,
|
||||
"load_write_noise_cycles": 100,
|
||||
"end_to_end_cycles": 140,
|
||||
"end_to_end_cycles_per_query": 70.0,
|
||||
"end_to_end_queries_per_cycle": 2 / 140,
|
||||
"total_query_cycles": 32,
|
||||
"mean_total_query_cycles": 16.0,
|
||||
"min_total_query_cycles": 14,
|
||||
"max_total_query_cycles": 18,
|
||||
"cycles_per_query": 16.0,
|
||||
"queries_per_cycle": 2 / 37,
|
||||
"queries_per_cycle": 2 / 32,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -91,12 +135,20 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N
|
||||
with (tmp_path / "metrics.csv").open(newline="", encoding="utf-8") as f:
|
||||
row = next(csv.DictReader(f))
|
||||
|
||||
assert row["query_only_cycles_per_query"] == "16.0"
|
||||
assert row["query_only_total_cycles"] == "32"
|
||||
assert row["query_only_queries_per_cycle"] == str(2 / 32)
|
||||
assert row["load_write_noise_cycles"] == "100"
|
||||
assert row["end_to_end_cycles"] == "140"
|
||||
assert row["end_to_end_queries_per_cycle"] == str(2 / 140)
|
||||
assert row["cycles_per_query"] == "16.0"
|
||||
assert row["mean_accept_to_first_result_cycles"] == "11.0"
|
||||
assert row["mean_accept_to_last_result_cycles"] == "16.0"
|
||||
assert row["queries_per_cycle"] == str(2 / 37)
|
||||
assert row["queries_per_cycle"] == str(2 / 32)
|
||||
|
||||
summary = (tmp_path / "summary.md").read_text(encoding="utf-8")
|
||||
assert "## Hardware performance" in summary
|
||||
assert "cycles_per_query" in summary
|
||||
assert "query-only cycles/query" in summary
|
||||
assert "load/write/noise cycles" in summary
|
||||
assert "end-to-end cycles/query" in summary
|
||||
assert "accept_to_last_result_cycles" in summary
|
||||
|
||||
Reference in New Issue
Block a user