mirror of
https://github.com/SikongJueluo/Mini-Nav.git
synced 2026-07-12 20:15:31 +08:00
feat(hw/sim): distinguish query-only and end-to-end performance cycles in retrieval benchmark
Add explicit separation between query-only cycles (accept→last) and end-to-end cycles (load + write + noise + queries) in hardware retrieval benchmarks. - Add query_only_cycles_per_query, load_write_noise_cycles, end_to_end_cycles metrics - Refactor summarize_query_timings() to use accept_to_last_result_cycles as query-only base - Add build_hardware_performance() to compute end-to-end performance separately - Add current_sim_cycle() helper using cocotb get_sim_time - Update CSV/Markdown outputs and RETRIEVAL_PERF_RESULT log format - Update documentation to clarify cycle-counting methodology - Update tests to cover new performance measurement logic
This commit is contained in:
@@ -41,6 +41,7 @@ just remote "python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retr
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz 512
|
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz 512
|
||||||
|
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz 512
|
||||||
```
|
```
|
||||||
|
|
||||||
## 2. 软件检索速度与质量
|
## 2. 软件检索速度与质量
|
||||||
@@ -133,38 +134,40 @@ benchmark。实现位置:
|
|||||||
- `hw/sim/tests/top/utils.py::query_topk_once_with_latency`
|
- `hw/sim/tests/top/utils.py::query_topk_once_with_latency`
|
||||||
- `hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py::cam_retrieval_benchmark`
|
- `hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py::cam_retrieval_benchmark`
|
||||||
|
|
||||||
周期口径如下:
|
周期口径如下。本节只报告Cocotb/Verilator仿真周期,不将cycle直接换算为ns或queries/s。
|
||||||
|
|
||||||
| 指标 | 含义 |
|
| 指标 | 含义 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `accept_to_first_result_cycles` | query 被 `query_valid && query_ready` 接受后,到首个 `result_valid` beat 的周期数。 |
|
| `query_only_cycles_per_query` | 主指标。每个query从`query_valid && query_ready`握手成功的时钟沿,到对应`result_valid && result_ready && result_last`完成的平均周期数。 |
|
||||||
| `accept_to_last_result_cycles` | query 被接受后,到 `result_last` 断言的周期数,即完整 Top-K 串行结果输出完成。 |
|
| `query_only_total_cycles` | 所有query-only事务周期之和,不包含装载、写入、噪声注入和查询间统计代码。 |
|
||||||
| `total_query_cycles` | 从拉高 `query_valid` 到 Top-K 输出完成的完整事务周期数。 |
|
| `query_only_queries_per_cycle` | `num_queries / query_only_total_cycles`。 |
|
||||||
| `cycles_per_query` | 当前报告中的主指标,等于平均 `accept_to_last_result_cycles`。 |
|
| `load_write_noise_cycles` | 写入CAM行以及可选写噪声注入阶段的周期数。无噪声模式下仍包含CAM行写入周期。 |
|
||||||
| `queries_per_cycle` | `num_queries / total_query_cycles`,用于观察仿真事务吞吐率。 |
|
| `end_to_end_cycles` | 从开始写入数据集到最后一个query完成的完整benchmark硬件仿真周期数。 |
|
||||||
|
| `end_to_end_queries_per_cycle` | `num_queries / end_to_end_cycles`。 |
|
||||||
|
| `accept_to_first_result_cycles` | query被接受后,到首个结果beat完成握手的平均周期数。 |
|
||||||
|
| `accept_to_last_result_cycles` | query被接受后,到`result_last`结果beat完成握手的平均周期数。 |
|
||||||
|
| `cycles_per_query`、`total_query_cycles`、`queries_per_cycle` | 兼容旧字段,当前均为query-only口径,不代表end-to-end。 |
|
||||||
|
|
||||||
选择 `accept_to_last_result_cycles` 作为主 `cycles_per_query` 的原因是:Top-K
|
选择query-only的`accept→last`作为主cycles/query,是因为Top-K检索只有在串行结果流输出到`result_last`并被接收后才算完整完成;仅用首个`result_valid`会低估实际Top-K查询事务成本。`load_write_noise_cycles`和`end_to_end_cycles`单独报告,避免把非查询阶段混入query-only性能。
|
||||||
检索只有在串行结果流输出到 `result_last` 后才算完整完成;仅用首个
|
|
||||||
`result_valid` 会低估实际 Top-K 检索事务成本。
|
|
||||||
|
|
||||||
### CIFAR-10 无噪声硬件周期结果
|
### 无噪声硬件周期结果
|
||||||
|
|
||||||
配置:512 rows × 128 queries × 512-bit hash,`TOPK_K=5`,`LANES=8`,
|
配置:512 rows×128 queries×512-bit hash,`TOPK_K=5`,`LANES=8`,
|
||||||
`WRITE_NOISE_EN=0`。
|
`WRITE_NOISE_EN=0`。
|
||||||
|
|
||||||
| 数据集 | 模式 | 查询数 | accept→first | accept→last / cycles/query | total query cycles | queries/cycle | 状态 |
|
| 数据集 | 模式 | 查询数 | query-only cycles/query | query-only total cycles | query-only queries/cycle | load/write/noise cycles | end-to-end cycles | end-to-end cycles/query | end-to-end queries/cycle | accept→first | accept→last | 状态 |
|
||||||
|---|---|---:|---:|---:|---:|---:|---|
|
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||||
| CIFAR-10 | hardware no-noise | 128 | 1027.000000 | 1031.000000 | 132096 | 0.000968992 | pass |
|
| CIFAR-10 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
|
||||||
|
| CIFAR-100 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
|
||||||
|
|
||||||
对应日志标记:
|
对应日志标记:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=132096 queries_per_cycle=0.000968992 status=pass
|
RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 query_only_cycles_per_query=1031.000000 query_only_total_cycles=131968 query_only_queries_per_cycle=0.000969932 load_write_noise_cycles=1024 end_to_end_cycles=133628 end_to_end_cycles_per_query=1043.968750 end_to_end_queries_per_cycle=0.000957883 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=131968 queries_per_cycle=0.000969932 status=pass
|
||||||
```
|
```
|
||||||
|
|
||||||
该结果说明:在当前 `NUM_ROWS=512, LANES=8, TOPK_K=5` 的硬件仿真配置下,
|
该结果说明:在当前 `NUM_ROWS=512, LANES=8, TOPK_K=5` 的硬件仿真配置下,
|
||||||
一次完整 Top-K 检索事务约为 **1031 cycles/query**。首个结果 beat 约在
|
一次完整Top-K查询事务的query-only成本为**1031cycles/query**。首个结果beat约在1027cycles后出现,完整Top-K输出额外消耗约4cycles。完整benchmark端到端口径为**1043.968750cycles/query**,其中数据写入阶段为1024cycles。
|
||||||
1027 cycles 后出现,完整 Top-K 输出额外消耗约 4 个周期。
|
|
||||||
|
|
||||||
> 注:以上数据来自 Verilator/Cocotb 仿真,不是 FPGA 板上实测。它可用于
|
> 注:以上数据来自 Verilator/Cocotb 仿真,不是 FPGA 板上实测。它可用于
|
||||||
> 架构级周期趋势分析,但不能直接等同于板级频率、吞吐或端到端系统延迟。
|
> 架构级周期趋势分析,但不能直接等同于板级频率、吞吐或端到端系统延迟。
|
||||||
@@ -180,8 +183,11 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
|
|||||||
| Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 |
|
| Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 |
|
||||||
| `ns_per_query` | 软件 Top-K 匹配阶段平均耗时;不含加载和写盘。 |
|
| `ns_per_query` | 软件 Top-K 匹配阶段平均耗时;不含加载和写盘。 |
|
||||||
| `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 |
|
| `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 |
|
||||||
| `cycles_per_query` | 硬件仿真中一次完整 Top-K 检索事务的平均周期数,当前采用 accept→last 口径。 |
|
| `query_only_cycles_per_query` | 硬件仿真中一次完整Top-K查询事务的平均周期数,采用`query_valid && query_ready`到`result_last`完成握手口径。 |
|
||||||
| `queries_per_cycle` | 硬件仿真中完成 query 数除以总 query 事务周期数。 |
|
| `query_only_queries_per_cycle` | 硬件仿真中完成query数除以query-only总事务周期数。 |
|
||||||
|
| `load_write_noise_cycles` | CAM行装载、写入及可选写噪声注入阶段周期数。 |
|
||||||
|
| `end_to_end_cycles` | 从开始写入数据到最后一个query完成的完整benchmark硬件仿真周期数。 |
|
||||||
|
| `end_to_end_queries_per_cycle` | 硬件仿真中完成query数除以end-to-end总周期数。 |
|
||||||
|
|
||||||
## 7. 当前结论
|
## 7. 当前结论
|
||||||
|
|
||||||
@@ -191,9 +197,8 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
|
|||||||
2. **当前软件基线速度约为 4.86k queries/s。**
|
2. **当前软件基线速度约为 4.86k queries/s。**
|
||||||
该结果来自 Python integer brute-force Hamming scan,数据规模为 512 rows × 128 queries × 512 bits。
|
该结果来自 Python integer brute-force Hamming scan,数据规模为 512 rows × 128 queries × 512 bits。
|
||||||
|
|
||||||
3. **硬件检索质量基准现在已经报告 cycles/query。**
|
3. **硬件检索质量基准现在显式拆分query-only与end-to-end周期。**
|
||||||
在 CIFAR-10 512-row/128-query 无噪声配置下,完整 Top-K 检索为
|
在CIFAR-10和CIFAR-100的512-row/128-query无噪声配置下,完整Top-K查询事务均为**1031cycles/query**,首个结果为**1027cycles/query**;写入阶段为**1024cycles**,端到端为**1043.968750cycles/query**。
|
||||||
**1031 cycles/query**,首个结果为 **1027 cycles/query**。
|
|
||||||
|
|
||||||
4. **写噪声对精确排序影响显著。**
|
4. **写噪声对精确排序影响显著。**
|
||||||
即使 Hit@K 保持较高,Golden Match@K 也会快速下降,说明噪声首先破坏精确排序,再进一步破坏类别命中。
|
即使 Hit@K 保持较高,Golden Match@K 也会快速下降,说明噪声首先破坏精确排序,再进一步破坏类别命中。
|
||||||
@@ -203,11 +208,10 @@ RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 cycles_per_query=1031.000000
|
|||||||
|
|
||||||
## 8. 后续建议
|
## 8. 后续建议
|
||||||
|
|
||||||
1. 对 CIFAR-100 也运行同样的硬件周期检测,补齐与软件质量表同尺度的硬件周期表。
|
1. 在 `docs/exps` 中继续维护:
|
||||||
2. 在 `docs/exps` 中继续维护:
|
|
||||||
- 软件检索速度表;
|
- 软件检索速度表;
|
||||||
- 硬件无噪声一致性表;
|
- 硬件无噪声一致性表;
|
||||||
- 硬件噪声鲁棒性表;
|
- 硬件噪声鲁棒性表;
|
||||||
- 硬件 `cycles/query` 表。
|
- 硬件query-only、load/write/noise、end-to-end周期表。
|
||||||
3. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan,以区分“朴素 Python baseline”和“优化软件 baseline”。
|
2. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan,以区分“朴素 Python baseline”和“优化软件 baseline”。
|
||||||
4. 增加 `NUM_ROWS` sweep:例如 512、1024、2048、4096 rows,观察软件 brute-force scan 的线性增长趋势。
|
3. 增加 `NUM_ROWS` sweep:例如 512、1024、2048、4096 rows,观察软件 brute-force scan 的线性增长趋势。
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ from pathlib import Path
|
|||||||
import cocotb
|
import cocotb
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from cocotb.clock import Clock
|
from cocotb.clock import Clock
|
||||||
|
from cocotb.utils import get_sim_time
|
||||||
|
|
||||||
from model.ref_model import (
|
from model.ref_model import (
|
||||||
match_topk,
|
match_topk,
|
||||||
@@ -32,6 +33,7 @@ DEFAULT_ROW_FLIP_BITS = 16
|
|||||||
DEFAULT_QUERY_FLIP_BITS = 16
|
DEFAULT_QUERY_FLIP_BITS = 16
|
||||||
DEFAULT_SEED = 20260522
|
DEFAULT_SEED = 20260522
|
||||||
BENCHMARK_KS = (1, 5)
|
BENCHMARK_KS = (1, 5)
|
||||||
|
SIM_CLOCK_PERIOD_NS = 10
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -214,6 +216,11 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]:
|
|||||||
if not timings:
|
if not timings:
|
||||||
return {
|
return {
|
||||||
"num_queries": 0,
|
"num_queries": 0,
|
||||||
|
"query_only_total_cycles": 0,
|
||||||
|
"query_only_cycles_per_query": 0.0,
|
||||||
|
"query_only_min_cycles": 0,
|
||||||
|
"query_only_max_cycles": 0,
|
||||||
|
"query_only_queries_per_cycle": 0.0,
|
||||||
"total_query_cycles": 0,
|
"total_query_cycles": 0,
|
||||||
"mean_total_query_cycles": 0.0,
|
"mean_total_query_cycles": 0.0,
|
||||||
"min_total_query_cycles": 0,
|
"min_total_query_cycles": 0,
|
||||||
@@ -224,24 +231,57 @@ def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]:
|
|||||||
"queries_per_cycle": 0.0,
|
"queries_per_cycle": 0.0,
|
||||||
}
|
}
|
||||||
|
|
||||||
total_cycles = sum(t.total_query_cycles for t in timings)
|
total_cycles = sum(t.accept_to_last_result_cycles for t in timings)
|
||||||
total_first = sum(t.accept_to_first_result_cycles for t in timings)
|
total_first = sum(t.accept_to_first_result_cycles for t in timings)
|
||||||
total_last = sum(t.accept_to_last_result_cycles for t in timings)
|
total_last = sum(t.accept_to_last_result_cycles for t in timings)
|
||||||
count = len(timings)
|
count = len(timings)
|
||||||
mean_last = total_last / float(count)
|
mean_last = total_last / float(count)
|
||||||
|
queries_per_cycle = count / float(total_cycles) if total_cycles > 0 else 0.0
|
||||||
return {
|
return {
|
||||||
"num_queries": count,
|
"num_queries": count,
|
||||||
|
"query_only_total_cycles": total_cycles,
|
||||||
|
"query_only_cycles_per_query": mean_last,
|
||||||
|
"query_only_min_cycles": min(t.accept_to_last_result_cycles for t in timings),
|
||||||
|
"query_only_max_cycles": max(t.accept_to_last_result_cycles for t in timings),
|
||||||
|
"query_only_queries_per_cycle": queries_per_cycle,
|
||||||
|
# Backward-compatible aliases: query-only, not end-to-end.
|
||||||
"total_query_cycles": total_cycles,
|
"total_query_cycles": total_cycles,
|
||||||
"mean_total_query_cycles": total_cycles / float(count),
|
"mean_total_query_cycles": total_cycles / float(count),
|
||||||
"min_total_query_cycles": min(t.total_query_cycles for t in timings),
|
"min_total_query_cycles": min(t.accept_to_last_result_cycles for t in timings),
|
||||||
"max_total_query_cycles": max(t.total_query_cycles for t in timings),
|
"max_total_query_cycles": max(t.accept_to_last_result_cycles for t in timings),
|
||||||
"mean_accept_to_first_result_cycles": total_first / float(count),
|
"mean_accept_to_first_result_cycles": total_first / float(count),
|
||||||
"mean_accept_to_last_result_cycles": mean_last,
|
"mean_accept_to_last_result_cycles": mean_last,
|
||||||
"cycles_per_query": mean_last,
|
"cycles_per_query": mean_last,
|
||||||
"queries_per_cycle": count / float(total_cycles) if total_cycles > 0 else 0.0,
|
"queries_per_cycle": queries_per_cycle,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def build_hardware_performance(
|
||||||
|
timings: list[QueryTiming],
|
||||||
|
*,
|
||||||
|
load_write_noise_cycles: int,
|
||||||
|
end_to_end_cycles: int,
|
||||||
|
) -> dict[str, float]:
|
||||||
|
performance = summarize_query_timings(timings)
|
||||||
|
num_queries = int(performance["num_queries"])
|
||||||
|
performance.update({
|
||||||
|
"load_write_noise_cycles": int(load_write_noise_cycles),
|
||||||
|
"end_to_end_cycles": int(end_to_end_cycles),
|
||||||
|
"end_to_end_cycles_per_query": (
|
||||||
|
float(end_to_end_cycles) / float(num_queries) if num_queries > 0 else 0.0
|
||||||
|
),
|
||||||
|
"end_to_end_queries_per_cycle": (
|
||||||
|
float(num_queries) / float(end_to_end_cycles) if end_to_end_cycles > 0 else 0.0
|
||||||
|
),
|
||||||
|
})
|
||||||
|
return performance
|
||||||
|
|
||||||
|
|
||||||
|
def current_sim_cycle() -> int:
|
||||||
|
"""Return the current benchmark clock cycle from simulator time."""
|
||||||
|
return int(get_sim_time("ns") // SIM_CLOCK_PERIOD_NS)
|
||||||
|
|
||||||
|
|
||||||
def output_dir_for(mode: str) -> Path:
|
def output_dir_for(mode: str) -> Path:
|
||||||
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID")
|
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID")
|
||||||
if not run_id:
|
if not run_id:
|
||||||
@@ -264,7 +304,11 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
|||||||
"write_noise_en", "write_noise_rate_num",
|
"write_noise_en", "write_noise_rate_num",
|
||||||
"write_noise_rate_den",
|
"write_noise_rate_den",
|
||||||
"num_queries", "k", "macro_precision", "retrieval_recall", "macro_f1",
|
"num_queries", "k", "macro_precision", "retrieval_recall", "macro_f1",
|
||||||
"recall@k", "exact_match_rate", "cycles_per_query",
|
"recall@k", "exact_match_rate",
|
||||||
|
"query_only_cycles_per_query", "query_only_total_cycles",
|
||||||
|
"query_only_queries_per_cycle", "load_write_noise_cycles",
|
||||||
|
"end_to_end_cycles", "end_to_end_cycles_per_query",
|
||||||
|
"end_to_end_queries_per_cycle", "cycles_per_query",
|
||||||
"mean_accept_to_first_result_cycles", "mean_accept_to_last_result_cycles",
|
"mean_accept_to_first_result_cycles", "mean_accept_to_last_result_cycles",
|
||||||
"mean_total_query_cycles", "total_query_cycles", "queries_per_cycle", "status",
|
"mean_total_query_cycles", "total_query_cycles", "queries_per_cycle", "status",
|
||||||
]
|
]
|
||||||
@@ -289,6 +333,13 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
|||||||
"macro_f1": metrics["macro_f1"],
|
"macro_f1": metrics["macro_f1"],
|
||||||
"recall@k": metrics["recall@k"],
|
"recall@k": metrics["recall@k"],
|
||||||
"exact_match_rate": metrics["exact_match_rate"],
|
"exact_match_rate": metrics["exact_match_rate"],
|
||||||
|
"query_only_cycles_per_query": result.get("performance", {}).get("query_only_cycles_per_query", ""),
|
||||||
|
"query_only_total_cycles": result.get("performance", {}).get("query_only_total_cycles", ""),
|
||||||
|
"query_only_queries_per_cycle": result.get("performance", {}).get("query_only_queries_per_cycle", ""),
|
||||||
|
"load_write_noise_cycles": result.get("performance", {}).get("load_write_noise_cycles", ""),
|
||||||
|
"end_to_end_cycles": result.get("performance", {}).get("end_to_end_cycles", ""),
|
||||||
|
"end_to_end_cycles_per_query": result.get("performance", {}).get("end_to_end_cycles_per_query", ""),
|
||||||
|
"end_to_end_queries_per_cycle": result.get("performance", {}).get("end_to_end_queries_per_cycle", ""),
|
||||||
"cycles_per_query": result.get("performance", {}).get("cycles_per_query", ""),
|
"cycles_per_query": result.get("performance", {}).get("cycles_per_query", ""),
|
||||||
"mean_accept_to_first_result_cycles": result.get("performance", {}).get(
|
"mean_accept_to_first_result_cycles": result.get("performance", {}).get(
|
||||||
"mean_accept_to_first_result_cycles", "",
|
"mean_accept_to_first_result_cycles", "",
|
||||||
@@ -313,11 +364,18 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
|||||||
"",
|
"",
|
||||||
"## Hardware performance",
|
"## Hardware performance",
|
||||||
"",
|
"",
|
||||||
f"- cycles_per_query: `{result.get('performance', {}).get('cycles_per_query', '')}`",
|
f"- query-only cycles/query: `{result.get('performance', {}).get('query_only_cycles_per_query', '')}`",
|
||||||
|
f"- query-only total cycles: `{result.get('performance', {}).get('query_only_total_cycles', '')}`",
|
||||||
|
f"- query-only queries/cycle: `{result.get('performance', {}).get('query_only_queries_per_cycle', '')}`",
|
||||||
|
f"- load/write/noise cycles: `{result.get('performance', {}).get('load_write_noise_cycles', '')}`",
|
||||||
|
f"- end-to-end cycles: `{result.get('performance', {}).get('end_to_end_cycles', '')}`",
|
||||||
|
f"- end-to-end cycles/query: `{result.get('performance', {}).get('end_to_end_cycles_per_query', '')}`",
|
||||||
|
f"- end-to-end queries/cycle: `{result.get('performance', {}).get('end_to_end_queries_per_cycle', '')}`",
|
||||||
|
f"- cycles_per_query (compat, query-only): `{result.get('performance', {}).get('cycles_per_query', '')}`",
|
||||||
f"- accept_to_first_result_cycles: `{result.get('performance', {}).get('mean_accept_to_first_result_cycles', '')}`",
|
f"- accept_to_first_result_cycles: `{result.get('performance', {}).get('mean_accept_to_first_result_cycles', '')}`",
|
||||||
f"- accept_to_last_result_cycles: `{result.get('performance', {}).get('mean_accept_to_last_result_cycles', '')}`",
|
f"- accept_to_last_result_cycles: `{result.get('performance', {}).get('mean_accept_to_last_result_cycles', '')}`",
|
||||||
f"- total_query_cycles: `{result.get('performance', {}).get('total_query_cycles', '')}`",
|
f"- total_query_cycles (compat, query-only): `{result.get('performance', {}).get('total_query_cycles', '')}`",
|
||||||
f"- queries_per_cycle: `{result.get('performance', {}).get('queries_per_cycle', '')}`",
|
f"- queries_per_cycle (compat, query-only): `{result.get('performance', {}).get('queries_per_cycle', '')}`",
|
||||||
"",
|
"",
|
||||||
"## Retrieval quality",
|
"## Retrieval quality",
|
||||||
"",
|
"",
|
||||||
@@ -338,7 +396,7 @@ def write_outputs(out_dir: Path, result: dict) -> None:
|
|||||||
|
|
||||||
@cocotb.test()
|
@cocotb.test()
|
||||||
async def cam_retrieval_benchmark(dut):
|
async def cam_retrieval_benchmark(dut):
|
||||||
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
|
cocotb.start_soon(Clock(dut.clk, SIM_CLOCK_PERIOD_NS, unit="ns").start())
|
||||||
await reset_dut(dut)
|
await reset_dut(dut)
|
||||||
|
|
||||||
num_rows = dut_num_rows(dut)
|
num_rows = dut_num_rows(dut)
|
||||||
@@ -358,7 +416,11 @@ async def cam_retrieval_benchmark(dut):
|
|||||||
dataset = load_retrieval_dataset_npz(dataset_path)
|
dataset = load_retrieval_dataset_npz(dataset_path)
|
||||||
if len(dataset.rows) != num_rows:
|
if len(dataset.rows) != num_rows:
|
||||||
raise AssertionError(f"artifact row count {len(dataset.rows)} must equal DUT NUM_ROWS {num_rows}")
|
raise AssertionError(f"artifact row count {len(dataset.rows)} must equal DUT NUM_ROWS {num_rows}")
|
||||||
|
|
||||||
|
benchmark_start_cycle = current_sim_cycle()
|
||||||
|
load_write_noise_start_cycle = current_sim_cycle()
|
||||||
await write_rows(dut, dataset.rows)
|
await write_rows(dut, dataset.rows)
|
||||||
|
load_write_noise_cycles = current_sim_cycle() - load_write_noise_start_cycle
|
||||||
|
|
||||||
accumulators = {k: MetricAccumulator() for k in BENCHMARK_KS}
|
accumulators = {k: MetricAccumulator() for k in BENCHMARK_KS}
|
||||||
timings: list[QueryTiming] = []
|
timings: list[QueryTiming] = []
|
||||||
@@ -380,6 +442,8 @@ async def cam_retrieval_benchmark(dut):
|
|||||||
label_hit = query_label in retrieved_labels
|
label_hit = query_label in retrieved_labels
|
||||||
accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact)
|
accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact)
|
||||||
|
|
||||||
|
end_to_end_cycles = current_sim_cycle() - benchmark_start_cycle
|
||||||
|
|
||||||
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}"
|
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}"
|
||||||
result = {
|
result = {
|
||||||
"run_id": run_id,
|
"run_id": run_id,
|
||||||
@@ -402,7 +466,11 @@ async def cam_retrieval_benchmark(dut):
|
|||||||
"seed": dataset.seed,
|
"seed": dataset.seed,
|
||||||
},
|
},
|
||||||
"metrics": {str(k): accumulators[k].as_dict() for k in BENCHMARK_KS},
|
"metrics": {str(k): accumulators[k].as_dict() for k in BENCHMARK_KS},
|
||||||
"performance": summarize_query_timings(timings),
|
"performance": build_hardware_performance(
|
||||||
|
timings,
|
||||||
|
load_write_noise_cycles=load_write_noise_cycles,
|
||||||
|
end_to_end_cycles=end_to_end_cycles,
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
out_dir = output_dir_for(mode)
|
out_dir = output_dir_for(mode)
|
||||||
@@ -419,11 +487,22 @@ async def cam_retrieval_benchmark(dut):
|
|||||||
|
|
||||||
performance = result["performance"]
|
performance = result["performance"]
|
||||||
dut._log.info(
|
dut._log.info(
|
||||||
"RETRIEVAL_PERF_RESULT mode=%s num_queries=%d cycles_per_query=%.6f "
|
"RETRIEVAL_PERF_RESULT mode=%s num_queries=%d query_only_cycles_per_query=%.6f "
|
||||||
|
"query_only_total_cycles=%d query_only_queries_per_cycle=%.9f "
|
||||||
|
"load_write_noise_cycles=%d end_to_end_cycles=%d "
|
||||||
|
"end_to_end_cycles_per_query=%.6f end_to_end_queries_per_cycle=%.9f "
|
||||||
|
"cycles_per_query=%.6f "
|
||||||
"accept_to_first_result_cycles=%.6f accept_to_last_result_cycles=%.6f "
|
"accept_to_first_result_cycles=%.6f accept_to_last_result_cycles=%.6f "
|
||||||
"total_query_cycles=%d queries_per_cycle=%.9f status=pass output_dir=%s",
|
"total_query_cycles=%d queries_per_cycle=%.9f status=pass output_dir=%s",
|
||||||
mode,
|
mode,
|
||||||
performance["num_queries"],
|
performance["num_queries"],
|
||||||
|
performance["query_only_cycles_per_query"],
|
||||||
|
performance["query_only_total_cycles"],
|
||||||
|
performance["query_only_queries_per_cycle"],
|
||||||
|
performance["load_write_noise_cycles"],
|
||||||
|
performance["end_to_end_cycles"],
|
||||||
|
performance["end_to_end_cycles_per_query"],
|
||||||
|
performance["end_to_end_queries_per_cycle"],
|
||||||
performance["cycles_per_query"],
|
performance["cycles_per_query"],
|
||||||
performance["mean_accept_to_first_result_cycles"],
|
performance["mean_accept_to_first_result_cycles"],
|
||||||
performance["mean_accept_to_last_result_cycles"],
|
performance["mean_accept_to_last_result_cycles"],
|
||||||
|
|||||||
@@ -216,7 +216,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
|||||||
``timing`` 字段:
|
``timing`` 字段:
|
||||||
- accept_to_first_result_cycles: query 接受到首个 result_valid beat
|
- accept_to_first_result_cycles: query 接受到首个 result_valid beat
|
||||||
- accept_to_last_result_cycles: query 接受到 result_last beat(Top-K 完成)
|
- accept_to_last_result_cycles: query 接受到 result_last beat(Top-K 完成)
|
||||||
- total_query_cycles: 从拉高 query_valid 到 Top-K 完成的总上升沿数
|
- total_query_cycles: 纯查询事务周期,等于 accept_to_last_result_cycles
|
||||||
|
|
||||||
``query_ready`` 是组合信号,握手周期在上升沿前采样;结果信号在
|
``query_ready`` 是组合信号,握手周期在上升沿前采样;结果信号在
|
||||||
ReadOnly settled phase 采样,避免重新引入 query_ready 采样时序问题。
|
ReadOnly settled phase 采样,避免重新引入 query_ready 采样时序问题。
|
||||||
@@ -236,8 +236,10 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
|||||||
|
|
||||||
await RisingEdge(dut.clk)
|
await RisingEdge(dut.clk)
|
||||||
edge_count += 1
|
edge_count += 1
|
||||||
|
q_valid = int(dut.query_valid.value)
|
||||||
q_ready = int(dut.query_ready.value)
|
q_ready = int(dut.query_ready.value)
|
||||||
assert q_ready, "Query accept handshake was missed despite query_ready pre-wait"
|
assert q_ready, "Query accept handshake was missed despite query_ready pre-wait"
|
||||||
|
assert q_valid, "Query valid deasserted before accept handshake"
|
||||||
accept_edge = edge_count
|
accept_edge = edge_count
|
||||||
dut.query_valid.value = 0
|
dut.query_valid.value = 0
|
||||||
|
|
||||||
@@ -252,7 +254,9 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
|||||||
await RisingEdge(dut.clk)
|
await RisingEdge(dut.clk)
|
||||||
edge_count += 1
|
edge_count += 1
|
||||||
await ReadOnly()
|
await ReadOnly()
|
||||||
if int(dut.result_valid.value):
|
result_valid = int(dut.result_valid.value)
|
||||||
|
result_ready = int(dut.result_ready.value)
|
||||||
|
if result_valid and result_ready:
|
||||||
if first_result_edge is None:
|
if first_result_edge is None:
|
||||||
first_result_edge = edge_count
|
first_result_edge = edge_count
|
||||||
rank = int(dut.result_rank.value)
|
rank = int(dut.result_rank.value)
|
||||||
@@ -282,7 +286,7 @@ async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
|
|||||||
timing = {
|
timing = {
|
||||||
"accept_to_first_result_cycles": int(first_result_edge - accept_edge),
|
"accept_to_first_result_cycles": int(first_result_edge - accept_edge),
|
||||||
"accept_to_last_result_cycles": int(last_result_edge - accept_edge),
|
"accept_to_last_result_cycles": int(last_result_edge - accept_edge),
|
||||||
"total_query_cycles": int(edge_count),
|
"total_query_cycles": int(last_result_edge - accept_edge),
|
||||||
}
|
}
|
||||||
return beats, beats[0][1], beats[0][2], score_debug, timing
|
return beats, beats[0][1], beats[0][2], score_debug, timing
|
||||||
|
|
||||||
|
|||||||
@@ -11,38 +11,73 @@ if str(HW_SIM_DIR) not in sys.path:
|
|||||||
|
|
||||||
from benchmarks.retrieval.test_retrieval_benchmark import ( # noqa: E402
|
from benchmarks.retrieval.test_retrieval_benchmark import ( # noqa: E402
|
||||||
QueryTiming,
|
QueryTiming,
|
||||||
|
build_hardware_performance,
|
||||||
summarize_query_timings,
|
summarize_query_timings,
|
||||||
write_outputs,
|
write_outputs,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_summarize_query_timings_reports_topk_completion_headline() -> None:
|
def test_summarize_query_timings_uses_query_only_accept_to_last_cycles() -> None:
|
||||||
summary = summarize_query_timings([
|
summary = summarize_query_timings([
|
||||||
QueryTiming(
|
QueryTiming(
|
||||||
accept_to_first_result_cycles=10,
|
accept_to_first_result_cycles=10,
|
||||||
accept_to_last_result_cycles=14,
|
accept_to_last_result_cycles=14,
|
||||||
total_query_cycles=16,
|
total_query_cycles=14,
|
||||||
),
|
),
|
||||||
QueryTiming(
|
QueryTiming(
|
||||||
accept_to_first_result_cycles=12,
|
accept_to_first_result_cycles=12,
|
||||||
accept_to_last_result_cycles=18,
|
accept_to_last_result_cycles=18,
|
||||||
total_query_cycles=21,
|
total_query_cycles=18,
|
||||||
),
|
),
|
||||||
])
|
])
|
||||||
|
|
||||||
assert summary == {
|
assert summary == {
|
||||||
"num_queries": 2,
|
"num_queries": 2,
|
||||||
"total_query_cycles": 37,
|
"query_only_total_cycles": 32,
|
||||||
"mean_total_query_cycles": 18.5,
|
"query_only_cycles_per_query": 16.0,
|
||||||
"min_total_query_cycles": 16,
|
"query_only_min_cycles": 14,
|
||||||
"max_total_query_cycles": 21,
|
"query_only_max_cycles": 18,
|
||||||
|
"query_only_queries_per_cycle": 2 / 32,
|
||||||
"mean_accept_to_first_result_cycles": 11.0,
|
"mean_accept_to_first_result_cycles": 11.0,
|
||||||
"mean_accept_to_last_result_cycles": 16.0,
|
"mean_accept_to_last_result_cycles": 16.0,
|
||||||
|
"total_query_cycles": 32,
|
||||||
|
"mean_total_query_cycles": 16.0,
|
||||||
|
"min_total_query_cycles": 14,
|
||||||
|
"max_total_query_cycles": 18,
|
||||||
"cycles_per_query": 16.0,
|
"cycles_per_query": 16.0,
|
||||||
"queries_per_cycle": 2 / 37,
|
"queries_per_cycle": 2 / 32,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_hardware_performance_separates_query_and_end_to_end_cycles() -> None:
|
||||||
|
performance = build_hardware_performance(
|
||||||
|
[
|
||||||
|
QueryTiming(
|
||||||
|
accept_to_first_result_cycles=10,
|
||||||
|
accept_to_last_result_cycles=14,
|
||||||
|
total_query_cycles=14,
|
||||||
|
),
|
||||||
|
QueryTiming(
|
||||||
|
accept_to_first_result_cycles=12,
|
||||||
|
accept_to_last_result_cycles=18,
|
||||||
|
total_query_cycles=18,
|
||||||
|
),
|
||||||
|
],
|
||||||
|
load_write_noise_cycles=100,
|
||||||
|
end_to_end_cycles=140,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert performance["query_only_total_cycles"] == 32
|
||||||
|
assert performance["query_only_cycles_per_query"] == 16.0
|
||||||
|
assert performance["query_only_queries_per_cycle"] == 2 / 32
|
||||||
|
assert performance["load_write_noise_cycles"] == 100
|
||||||
|
assert performance["end_to_end_cycles"] == 140
|
||||||
|
assert performance["end_to_end_cycles_per_query"] == 70.0
|
||||||
|
assert performance["end_to_end_queries_per_cycle"] == 2 / 140
|
||||||
|
assert performance["cycles_per_query"] == performance["query_only_cycles_per_query"]
|
||||||
|
assert performance["queries_per_cycle"] == performance["query_only_queries_per_cycle"]
|
||||||
|
|
||||||
|
|
||||||
def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> None:
|
def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> None:
|
||||||
result = {
|
result = {
|
||||||
"run_id": "test-run",
|
"run_id": "test-run",
|
||||||
@@ -75,14 +110,23 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N
|
|||||||
},
|
},
|
||||||
"performance": {
|
"performance": {
|
||||||
"num_queries": 2,
|
"num_queries": 2,
|
||||||
"total_query_cycles": 37,
|
"query_only_total_cycles": 32,
|
||||||
"mean_total_query_cycles": 18.5,
|
"query_only_cycles_per_query": 16.0,
|
||||||
"min_total_query_cycles": 16,
|
"query_only_min_cycles": 14,
|
||||||
"max_total_query_cycles": 21,
|
"query_only_max_cycles": 18,
|
||||||
|
"query_only_queries_per_cycle": 2 / 32,
|
||||||
"mean_accept_to_first_result_cycles": 11.0,
|
"mean_accept_to_first_result_cycles": 11.0,
|
||||||
"mean_accept_to_last_result_cycles": 16.0,
|
"mean_accept_to_last_result_cycles": 16.0,
|
||||||
|
"load_write_noise_cycles": 100,
|
||||||
|
"end_to_end_cycles": 140,
|
||||||
|
"end_to_end_cycles_per_query": 70.0,
|
||||||
|
"end_to_end_queries_per_cycle": 2 / 140,
|
||||||
|
"total_query_cycles": 32,
|
||||||
|
"mean_total_query_cycles": 16.0,
|
||||||
|
"min_total_query_cycles": 14,
|
||||||
|
"max_total_query_cycles": 18,
|
||||||
"cycles_per_query": 16.0,
|
"cycles_per_query": 16.0,
|
||||||
"queries_per_cycle": 2 / 37,
|
"queries_per_cycle": 2 / 32,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -91,12 +135,20 @@ def test_write_outputs_includes_hardware_performance_fields(tmp_path: Path) -> N
|
|||||||
with (tmp_path / "metrics.csv").open(newline="", encoding="utf-8") as f:
|
with (tmp_path / "metrics.csv").open(newline="", encoding="utf-8") as f:
|
||||||
row = next(csv.DictReader(f))
|
row = next(csv.DictReader(f))
|
||||||
|
|
||||||
|
assert row["query_only_cycles_per_query"] == "16.0"
|
||||||
|
assert row["query_only_total_cycles"] == "32"
|
||||||
|
assert row["query_only_queries_per_cycle"] == str(2 / 32)
|
||||||
|
assert row["load_write_noise_cycles"] == "100"
|
||||||
|
assert row["end_to_end_cycles"] == "140"
|
||||||
|
assert row["end_to_end_queries_per_cycle"] == str(2 / 140)
|
||||||
assert row["cycles_per_query"] == "16.0"
|
assert row["cycles_per_query"] == "16.0"
|
||||||
assert row["mean_accept_to_first_result_cycles"] == "11.0"
|
assert row["mean_accept_to_first_result_cycles"] == "11.0"
|
||||||
assert row["mean_accept_to_last_result_cycles"] == "16.0"
|
assert row["mean_accept_to_last_result_cycles"] == "16.0"
|
||||||
assert row["queries_per_cycle"] == str(2 / 37)
|
assert row["queries_per_cycle"] == str(2 / 32)
|
||||||
|
|
||||||
summary = (tmp_path / "summary.md").read_text(encoding="utf-8")
|
summary = (tmp_path / "summary.md").read_text(encoding="utf-8")
|
||||||
assert "## Hardware performance" in summary
|
assert "## Hardware performance" in summary
|
||||||
assert "cycles_per_query" in summary
|
assert "query-only cycles/query" in summary
|
||||||
|
assert "load/write/noise cycles" in summary
|
||||||
|
assert "end-to-end cycles/query" in summary
|
||||||
assert "accept_to_last_result_cycles" in summary
|
assert "accept_to_last_result_cycles" in summary
|
||||||
|
|||||||
Reference in New Issue
Block a user