diff --git a/docs/exps/sw_hw_cam_retrieval_benchmark.md b/docs/exps/sw_hw_cam_retrieval_benchmark.md new file mode 100644 index 0000000..06ac2ba --- /dev/null +++ b/docs/exps/sw_hw_cam_retrieval_benchmark.md @@ -0,0 +1,160 @@ +# 软件/硬件 CAM 检索基准实验总结 + +**日期**：2026-05-27 +**工作区**：`.workspace/feat_sw_retrieval_benchmark` +**目标**：对比同一组 CAM 检索数据在硬件仿真与软件汉明距离检索下的检索质量，并记录软件检索速度基线。 + +## 1. 实验配置 + +### 数据集与哈希配置 + +| 数据集 | 行数 | 查询数 | 类别数 | 哈希宽度 | Top-K | +|---|---:|---:|---:|---:|---:| +| CIFAR-10 | 512 | 128 | 10 | 512 bit | 5 | +| CIFAR-100 | 512 | 128 | 100 | 512 bit | 5 | + +数据文件来自： + +- `outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz` +- `outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz` + +`.npz` 内部字段为： + +- `rows_words` +- `row_labels` +- `queries_words` +- `query_labels` + +软件基准直接复用硬件基准数据格式，将 little-endian `uint64` words 转为 Python `int` 后执行汉明距离 / XNOR-popcount Top-K 检索，避免通过软件 CAM 时序模拟带来额外开销。 + +### 运行命令 + +软件检索速度基准： + +```bash +just remote "python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz --hash-bits 512 --topk-k 5 --run-id sw_cifar10_hash512_rows512_queries128 && python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz --hash-bits 512 --topk-k 5 --run-id sw_cifar100_hash512_rows512_queries128" +``` + +硬件噪声扫描数据来自远端已有 Cocotb/Verilator 输出与 `docs/exps/cam_retrieval_noise_sweep_*.md`。 + +## 2. 软件检索速度与质量 + +软件路径只计时 Top-K 匹配阶段，不包含 `.npz` 加载、指标聚合和结果写盘。 + +| 数据集 | Hit@1 | Precision@1 | Std-Recall@1 | Hit@5 | Precision@5 | Std-Recall@5 | Golden Match@5 | ns/query | queries/s | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| CIFAR-10 | 1.000000 | 1.000000 | 0.019531 | 1.000000 | 1.000000 | 0.097656 | 1.000000 | 205835.281 | 4858.254 | +| CIFAR-100 | 0.695312 | 0.695312 | 0.134635 | 0.867188 | 0.462500 | 0.445052 | 1.000000 | 205868.953 | 4857.459 | + +观察： + +- 两个数据集的软件吞吐都约为 **4.86k queries/s**。 +- CIFAR-10 的 Hit@1/Hit@5 均为 1.0，说明在该 512-row 子集上 Top-K 中总能命中同类样本。 +- CIFAR-100 的类别更多、每类样本更少，Hit@1 降至 0.695312，Hit@5 为 0.867188。 +- `Golden Match@K = 1.0` 表示软件实现与 `hw/sim/model/ref_model.py::match_topk` 的排序结果一致。 + +## 3. 无噪声硬件仿真与软件基线一致性 + +硬件无噪声结果来自 `WRITE_NOISE_EN=0` 的 Cocotb/Verilator 检索基准。该结果用于确认硬件 Top-K 输出与参考模型一致。 + +| 数据集 | 模式 | Hit@1 | Hit@5 | Precision@5 | Std-Recall@5 | Golden Match@1 | Golden Match@5 | +|---|---|---:|---:|---:|---:|---:|---:| +| CIFAR-10 | software-hamming | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 1.000000 | 1.000000 | +| CIFAR-10 | hardware no-noise | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 1.000000 | 1.000000 | +| CIFAR-100 | software-hamming | 0.695312 | 0.867188 | 0.462500 | 0.445052 | 1.000000 | 1.000000 | +| CIFAR-100 | hardware no-noise | 0.695312 | 0.867188 | 0.462500 | 0.445052 | 1.000000 | 1.000000 | + +结论： + +- 无噪声硬件仿真与软件汉明距离基线在两个数据集上质量指标一致。 +- 这说明 `.npz` 数据加载、little-endian word 转换、硬件 CAM 匹配和软件参考模型在当前配置下是对齐的。 + +## 4. 写噪声对硬件检索质量的影响 + +硬件噪声实验使用 `WRITE_NOISE_EN=1`，按 10% 步长扫描 `WRITE_NOISE_RATE_NUM / 100`。以下表格保留主要指标 Hit@1、Hit@5 与 Golden Match@K。 + +### CIFAR-10 噪声扫描 + +| 写噪声率 | Hit@1 | Hit@5 | Golden Match@1 | Golden Match@5 | +|---:|---:|---:|---:|---:| +| 0% | 1.000000 | 1.000000 | 1.000000 | 1.000000 | +| 10% | 1.000000 | 1.000000 | 0.507812 | 0.000000 | +| 20% | 1.000000 | 1.000000 | 0.234375 | 0.000000 | +| 30% | 0.992188 | 1.000000 | 0.164062 | 0.000000 | +| 40% | 0.984375 | 1.000000 | 0.093750 | 0.000000 | +| 50% | 0.257812 | 0.750000 | 0.023438 | 0.000000 | +| 60% | 0.000000 | 0.015625 | 0.000000 | 0.000000 | +| 70% | 0.000000 | 0.000000 | 0.000000 | 0.000000 | +| 80% | 0.000000 | 0.000000 | 0.000000 | 0.000000 | +| 90% | 0.000000 | 0.000000 | 0.000000 | 0.000000 | +| 100% | 0.000000 | 0.000000 | 0.000000 | 0.000000 | + +观察： + +- CIFAR-10 在 0%–40% 写噪声下 Hit@5 仍保持 1.0。 +- Golden Match@5 从 10% 噪声开始降为 0，说明 Top-5 的精确排序对噪声非常敏感。 +- 50% 噪声是明显拐点：Hit@1 降至 0.257812，Hit@5 降至 0.75。 +- 60% 以后检索基本失效。 + +### CIFAR-100 噪声扫描 + +| 写噪声率 | Hit@1 | Hit@5 | Golden Match@1 | Golden Match@5 | +|---:|---:|---:|---:|---:| +| 0% | 0.695312 | 0.867188 | 1.000000 | 1.000000 | +| 10% | 0.585938 | 0.812500 | 0.593750 | 0.023438 | +| 20% | 0.562500 | 0.742188 | 0.460938 | 0.000000 | +| 30% | 0.460938 | 0.640625 | 0.304688 | 0.000000 | +| 40% | 0.234375 | 0.460938 | 0.101562 | 0.000000 | +| 50% | 0.000000 | 0.062500 | 0.000000 | 0.000000 | +| 60% | 0.000000 | 0.007812 | 0.000000 | 0.000000 | +| 70% | 0.000000 | 0.007812 | 0.000000 | 0.000000 | +| 80% | 0.000000 | 0.000000 | 0.000000 | 0.000000 | +| 90% | 0.000000 | 0.000000 | 0.000000 | 0.000000 | +| 100% | 0.000000 | 0.000000 | 0.000000 | 0.000000 | + +观察： + +- CIFAR-100 对噪声更敏感：10% 噪声时 Hit@1 从 0.695312 降至 0.585938，Hit@5 从 0.867188 降至 0.812500。 +- 40% 噪声时 Hit@5 已降至 0.460938。 +- 50% 噪声后检索基本不可用。 +- 与 CIFAR-10 相同，Golden Match@5 在低噪声下也迅速接近 0，说明精确排序比类别命中率更脆弱。 + +## 5. 指标解释 + +| 指标 | 含义 | +|---|---| +| Hit@K / `recall@k` | 每个 query 的 Top-K 中是否至少出现一个同类样本，然后对 query 取平均。 | +| Precision@K / `macro_precision` | 每个 query 的 Top-K 中同类样本比例，即 `tp / k`，再对 query 取平均。 | +| Std-Recall@K / `retrieval_recall` | 每个 query 检出的同类样本数占数据库中所有同类样本数的比例，即 `tp / |relevant|`，再取平均。 | +| Std-F1@K / `macro_f1` | 使用 Precision@K 与 Std-Recall@K 计算的 F1。 | +| Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 | +| `ns_per_query` | 软件 Top-K 匹配阶段平均耗时；不含加载和写盘。 | +| `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 | + +## 6. 当前结论 + +1. **软件汉明距离基线已可作为硬件 CAM 检索的功能参考。** + 在无噪声条件下，硬件仿真和软件基线的质量指标及 Golden Match 均一致。 + +2. **当前软件基线速度约为 4.86k queries/s。** + 该结果来自 Python integer brute-force Hamming scan，数据规模为 512 rows × 128 queries × 512 bits。 + +3. **硬件检索质量基准目前主要报告质量指标，不报告 cycles/query。** + 现有硬件性能基准中已有 cycle 统计逻辑，但尚未和真实 retrieval dataset 的 Top-K benchmark 合并。因此本文不报告硬件 query/s 或 cycles/query。 + +4. **写噪声对精确排序影响显著。** + 即使 Hit@K 保持较高，Golden Match@K 也会快速下降，说明噪声首先破坏精确排序，再进一步破坏类别命中。 + +5. **CIFAR-100 比 CIFAR-10 更能体现检索难度。** + 在无噪声下 CIFAR-100 的 Hit@1/Hit@5 分别为 0.695312/0.867188，明显低于 CIFAR-10 的 1.0/1.0，更适合作为后续检索质量对比主数据集。 + +## 7. 后续建议 + +1. 将 `hw/sim/tests/perf/test_cam_perf.py::query_once_with_latency` 的周期测量逻辑合并到 retrieval benchmark，记录真实数据集上的 `cycles/query`。 +2. 在 `docs/exps` 中继续维护： + - 软件检索速度表； + - 硬件无噪声一致性表； + - 硬件噪声鲁棒性表； + - 后续硬件 `cycles/query` 表。 +3. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan，以区分“朴素 Python baseline”和“优化软件 baseline”。 +4. 增加 `NUM_ROWS` sweep：例如 512、1024、2048、4096 rows，观察软件 brute-force scan 的线性增长趋势。 diff --git a/scripts/sw_retrieval_benchmark.py b/scripts/sw_retrieval_benchmark.py new file mode 100644 index 0000000..8666ab0 --- /dev/null +++ b/scripts/sw_retrieval_benchmark.py @@ -0,0 +1,329 @@ +from __future__ import annotations + +import argparse +import csv +import json +import sys +import time +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +from typing import Callable, Iterable, Sequence + +import numpy as np + + +BENCHMARK_KS = (1, 5) +PROJECT_ROOT = Path(__file__).resolve().parents[1] +HW_SIM_DIR = PROJECT_ROOT / "hw" / "sim" +if str(HW_SIM_DIR) not in sys.path: + sys.path.insert(0, str(HW_SIM_DIR)) + +from model.ref_model import match_topk as ref_match_topk # noqa: E402 + + +@dataclass(frozen=True) +class RetrievalDataset: + rows: list[int] + row_labels: list[int] + queries: list[int] + query_labels: list[int] + hash_bits: int + num_classes: int + positives_per_class: int = 0 + queries_per_class: int = 0 + seed: int = 0 + + +@dataclass(frozen=True) +class MetricAccumulator: + precision_sum: float = 0.0 + recall_sum: float = 0.0 + f1_sum: float = 0.0 + exact_matches: int = 0 + label_hits: int = 0 + count: int = 0 + + def add(self, precision: float, recall: float, f1: float, label_hit: bool, exact: bool) -> "MetricAccumulator": + return MetricAccumulator( + precision_sum=self.precision_sum + precision, + recall_sum=self.recall_sum + recall, + f1_sum=self.f1_sum + f1, + exact_matches=self.exact_matches + int(exact), + label_hits=self.label_hits + int(label_hit), + count=self.count + 1, + ) + + def as_dict(self) -> dict[str, float]: + if self.count == 0: + return { + "macro_precision": 0.0, + "retrieval_recall": 0.0, + "macro_f1": 0.0, + "exact_match_rate": 0.0, + "recall@k": 0.0, + } + return { + "macro_precision": self.precision_sum / self.count, + "retrieval_recall": self.recall_sum / self.count, + "macro_f1": self.f1_sum / self.count, + "exact_match_rate": self.exact_matches / self.count, + "recall@k": self.label_hits / self.count, + } + + +def project_root() -> Path: + return PROJECT_ROOT + + +def words_le_to_int(words: np.ndarray) -> int: + value = 0 + for idx, word in enumerate(words.tolist()): + value |= int(word) << (64 * idx) + return value + + +def load_retrieval_dataset_npz(path: str | Path) -> RetrievalDataset: + dataset_path = Path(path) + if not dataset_path.is_absolute(): + dataset_path = project_root() / dataset_path + if not dataset_path.exists(): + raise FileNotFoundError(f"retrieval dataset not found: {dataset_path}") + + loaded = np.load(dataset_path) + rows_words = loaded["rows_words"] + queries_words = loaded["queries_words"] + if rows_words.ndim != 2 or queries_words.ndim != 2: + raise ValueError("rows_words and queries_words must have shape [N, words]") + if rows_words.shape[1] != queries_words.shape[1]: + raise ValueError("rows_words and queries_words must use the same word width") + + rows = [words_le_to_int(words) for words in rows_words] + queries = [words_le_to_int(words) for words in queries_words] + row_labels = [int(x) for x in loaded["row_labels"].tolist()] + query_labels = [int(x) for x in loaded["query_labels"].tolist()] + return RetrievalDataset( + rows=rows, + row_labels=row_labels, + queries=queries, + query_labels=query_labels, + hash_bits=int(rows_words.shape[1] * 64), + num_classes=len(set(row_labels)), + ) + + +def hamming_similarity_score(query_row: int, stored_row: int, *, width: int) -> int: + if width <= 0: + raise ValueError("width must be greater than 0") + mask = (1 << width) - 1 + distance = ((int(query_row) ^ int(stored_row)) & mask).bit_count() + return int(width - distance) + + +def match_topk_hamming(query: int, rows: Sequence[int], *, width: int, k: int) -> list[int]: + if k <= 0: + raise ValueError("k must be greater than 0") + scored = [ + (-hamming_similarity_score(query, row, width=width), row_index) + for row_index, row in enumerate(rows) + ] + scored.sort() + return [row_index for _, row_index in scored[: min(k, len(scored))]] + + +def compute_metrics(topk_indices: list[int], row_labels: list[int], query_label: int, k: int) -> tuple[float, float, float]: + retrieved = topk_indices[:k] + relevant = {idx for idx, label in enumerate(row_labels) if label == query_label} + tp = len(set(retrieved) & relevant) + precision = tp / float(k) + recall = tp / float(len(relevant)) if relevant else 0.0 + f1 = 0.0 if precision + recall == 0 else (2.0 * precision * recall) / (precision + recall) + return precision, recall, f1 + + +def _normalized_topk_values(topk_values: Iterable[int]) -> tuple[int, ...]: + values = tuple(sorted({int(k) for k in topk_values})) + if not values or values[0] <= 0: + raise ValueError("topk_values must contain positive integers") + return values + + +def run_benchmark( + dataset_path: str | Path, + *, + hash_bits: int = 512, + topk_values: Iterable[int] = BENCHMARK_KS, + run_id: str | None = None, + timer_ns: Callable[[], int] = time.perf_counter_ns, +) -> dict: + dataset = load_retrieval_dataset_npz(dataset_path) + if not dataset.rows: + raise ValueError("cannot benchmark an empty row set") + if hash_bits != dataset.hash_bits: + raise ValueError( + f"hash_bits={hash_bits} does not match dataset width {dataset.hash_bits}" + ) + if len(dataset.queries) != len(dataset.query_labels): + raise ValueError("queries and query_labels must have the same length") + + ks = _normalized_topk_values(topk_values) + max_k = max(ks) + if max_k > len(dataset.rows): + raise ValueError("topk_values cannot exceed the number of dataset rows") + + start_ns = timer_ns() + all_topk = [ + match_topk_hamming(query, dataset.rows, width=hash_bits, k=max_k) + for query in dataset.queries + ] + end_ns = timer_ns() + + golden_topk = [ + ref_match_topk(query, dataset.rows, width=hash_bits, k=max_k)[0] + for query in dataset.queries + ] + + accumulators = {k: MetricAccumulator() for k in ks} + for topk_indices, golden_indices, query_label in zip(all_topk, golden_topk, dataset.query_labels): + for k in ks: + precision, recall, f1 = compute_metrics(topk_indices, dataset.row_labels, query_label, k) + retrieved_labels = [dataset.row_labels[idx] for idx in topk_indices[:k]] + label_hit = query_label in retrieved_labels + exact = topk_indices[:k] == golden_indices[:k] + accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact) + + metrics = {str(k): accumulators[k].as_dict() for k in ks} + + elapsed_ns = max(0, int(end_ns - start_ns)) + num_queries = len(dataset.queries) + ns_per_query = (elapsed_ns / float(num_queries)) if num_queries else 0.0 + qps = (1_000_000_000.0 / ns_per_query) if ns_per_query > 0 else 0.0 + resolved_run_id = run_id or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-software-hamming" + + return { + "run_id": resolved_run_id, + "mode": "software-hamming", + "status": "pass", + "params": { + "num_rows": len(dataset.rows), + "hash_bits": int(hash_bits), + "topk_k": max_k, + }, + "dataset": { + "num_classes": dataset.num_classes, + "positives_per_class": dataset.positives_per_class, + "queries_per_class": dataset.queries_per_class, + "num_queries": num_queries, + "seed": dataset.seed, + }, + "metrics": metrics, + "performance": { + "total_elapsed_ns": elapsed_ns, + "total_elapsed_sec": elapsed_ns / 1_000_000_000.0, + "ns_per_query": ns_per_query, + "queries_per_second": qps, + }, + } + + +def write_outputs(out_dir: Path, result: dict) -> None: + out_dir.mkdir(parents=True, exist_ok=True) + (out_dir / "metrics.json").write_text(json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8") + + fieldnames = [ + "run_id", "mode", "num_rows", "hash_bits", "topk_k", "num_queries", "k", + "macro_precision", "retrieval_recall", "macro_f1", "recall@k", "exact_match_rate", + "total_elapsed_ns", "ns_per_query", "queries_per_second", "status", + ] + with (out_dir / "metrics.csv").open("w", newline="", encoding="utf-8") as f: + writer = csv.DictWriter(f, fieldnames=fieldnames) + writer.writeheader() + for k, metrics in result["metrics"].items(): + writer.writerow({ + "run_id": result["run_id"], + "mode": result["mode"], + "num_rows": result["params"]["num_rows"], + "hash_bits": result["params"]["hash_bits"], + "topk_k": result["params"]["topk_k"], + "num_queries": result["dataset"]["num_queries"], + "k": int(k), + "macro_precision": metrics["macro_precision"], + "retrieval_recall": metrics["retrieval_recall"], + "macro_f1": metrics["macro_f1"], + "recall@k": metrics["recall@k"], + "exact_match_rate": metrics["exact_match_rate"], + "total_elapsed_ns": result["performance"]["total_elapsed_ns"], + "ns_per_query": result["performance"]["ns_per_query"], + "queries_per_second": result["performance"]["queries_per_second"], + "status": result["status"], + }) + + lines = [ + "# Software CAM Retrieval Benchmark Summary", + "", + f"- run_id: `{result['run_id']}`", + f"- mode: `{result['mode']}`", + f"- status: `{result['status']}`", + f"- num_queries: `{result['dataset']['num_queries']}`", + f"- ns_per_query: `{result['performance']['ns_per_query']}`", + f"- queries_per_second: `{result['performance']['queries_per_second']}`", + "", + "| k | macro_precision | retrieval_recall | macro_f1 | recall@k | exact_match_rate |", + "|---:|---:|---:|---:|---:|---:|", + ] + for k, metrics in result["metrics"].items(): + lines.append( + f"| {k} | {metrics['macro_precision']:.6f} | {metrics['retrieval_recall']:.6f} | " + f"{metrics['macro_f1']:.6f} | {metrics['recall@k']:.6f} | {metrics['exact_match_rate']:.6f} |" + ) + lines.extend([ + "", + "说明：软件路径直接对 `.npz` 中的 CAM 行整数执行汉明距离 / XNOR-popcount 检索，不使用软件 CAM 时序仿真。", + ]) + (out_dir / "summary.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def output_dir_for(run_id: str, output_root: Path) -> Path: + return output_root / run_id + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Run software Hamming CAM retrieval benchmark.") + parser.add_argument("--dataset", required=True, help="Prepared CAM retrieval .npz artifact") + parser.add_argument("--hash-bits", type=int, default=512, help="Hash width in bits") + parser.add_argument("--topk-k", type=int, default=5, help="Maximum Top-K to report; reports k=1 and this value") + parser.add_argument("--run-id", default=None, help="Output run id") + parser.add_argument( + "--output-root", + type=Path, + default=Path("outputs/sw_retrieval_benchmark"), + help="Directory under which the run directory is written", + ) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + topk_values = (1,) if args.topk_k == 1 else (1, args.topk_k) + result = run_benchmark( + args.dataset, + hash_bits=args.hash_bits, + topk_values=topk_values, + run_id=args.run_id, + ) + out_dir = output_dir_for(result["run_id"], args.output_root) + write_outputs(out_dir, result) + print( + "SW_RETRIEVAL_RESULT " + f"run_id={result['run_id']} " + f"num_rows={result['params']['num_rows']} " + f"hash_bits={result['params']['hash_bits']} " + f"num_queries={result['dataset']['num_queries']} " + f"ns_per_query={result['performance']['ns_per_query']:.3f} " + f"queries_per_second={result['performance']['queries_per_second']:.3f} " + f"output_dir={out_dir}" + ) + + +if __name__ == "__main__": + main() diff --git a/tests/test_sw_retrieval_benchmark.py b/tests/test_sw_retrieval_benchmark.py new file mode 100644 index 0000000..2983f8b --- /dev/null +++ b/tests/test_sw_retrieval_benchmark.py @@ -0,0 +1,152 @@ +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path + +import numpy as np +import pytest + + +def load_sw_benchmark(): + path = Path(__file__).resolve().parent.parent / "scripts" / "sw_retrieval_benchmark.py" + spec = importlib.util.spec_from_file_location("sw_retrieval_benchmark", path) + assert spec is not None, f"Could not find spec for {path}" + assert spec.loader is not None, f"No loader available for {path}" + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def _write_dataset(path: Path) -> None: + np.savez_compressed( + path, + rows_words=np.array( + [ + [0b11110000], + [0b11100000], + [0b00001111], + ], + dtype=np.uint64, + ), + row_labels=np.array([0, 0, 1], dtype=np.int64), + queries_words=np.array( + [ + [0b11110000], + [0b00001111], + ], + dtype=np.uint64, + ), + query_labels=np.array([0, 1], dtype=np.int64), + ) + + +def test_load_dataset_preserves_little_endian_words(tmp_path): + bench = load_sw_benchmark() + dataset_path = tmp_path / "dataset.npz" + np.savez_compressed( + dataset_path, + rows_words=np.array([[0x2, 0x1]], dtype=np.uint64), + row_labels=np.array([7], dtype=np.int64), + queries_words=np.array([[0x4, 0x3]], dtype=np.uint64), + query_labels=np.array([8], dtype=np.int64), + ) + + dataset = bench.load_retrieval_dataset_npz(dataset_path) + + assert dataset.rows == [0x1_0000_0000_0000_0002] + assert dataset.queries == [0x3_0000_0000_0000_0004] + assert dataset.row_labels == [7] + assert dataset.query_labels == [8] + + +def test_run_benchmark_reports_quality_and_query_speed(tmp_path): + bench = load_sw_benchmark() + dataset_path = tmp_path / "dataset.npz" + _write_dataset(dataset_path) + timer_values = iter([1_000, 3_000]) + + result = bench.run_benchmark( + dataset_path, + hash_bits=64, + topk_values=(1, 2), + run_id="unit-test", + timer_ns=lambda: next(timer_values), + ) + + assert result["mode"] == "software-hamming" + assert result["status"] == "pass" + assert result["dataset"]["num_queries"] == 2 + assert result["params"]["num_rows"] == 3 + assert result["params"]["topk_k"] == 2 + assert result["metrics"]["1"]["exact_match_rate"] == 1.0 + assert result["metrics"]["1"]["recall@k"] == 1.0 + assert result["performance"]["total_elapsed_ns"] == 2_000 + assert result["performance"]["ns_per_query"] == 1_000.0 + assert result["performance"]["queries_per_second"] == 1_000_000.0 + + +def test_run_benchmark_exact_match_compares_against_reference(tmp_path, monkeypatch): + bench = load_sw_benchmark() + dataset_path = tmp_path / "dataset.npz" + _write_dataset(dataset_path) + + monkeypatch.setattr(bench, "ref_match_topk", lambda query, rows, width, k: ([1, 0], [])) + + result = bench.run_benchmark( + dataset_path, + hash_bits=64, + topk_values=(1,), + timer_ns=lambda: 0, + ) + + assert result["metrics"]["1"]["exact_match_rate"] == 0.0 + + +def test_run_benchmark_rejects_hash_bits_that_do_not_match_npz_width(tmp_path): + bench = load_sw_benchmark() + dataset_path = tmp_path / "dataset.npz" + _write_dataset(dataset_path) + + with pytest.raises(ValueError, match="hash_bits"): + bench.run_benchmark(dataset_path, hash_bits=8) + + +def test_write_outputs_includes_quality_and_performance_csv(tmp_path): + bench = load_sw_benchmark() + result = { + "run_id": "unit-test", + "mode": "software-hamming", + "status": "pass", + "params": {"num_rows": 3, "hash_bits": 64, "topk_k": 2}, + "dataset": {"num_classes": 2, "num_queries": 2}, + "metrics": { + "1": { + "macro_precision": 1.0, + "retrieval_recall": 0.75, + "macro_f1": 0.85, + "exact_match_rate": 1.0, + "recall@k": 1.0, + } + }, + "performance": { + "total_elapsed_ns": 2_000, + "total_elapsed_sec": 0.000002, + "ns_per_query": 1_000.0, + "queries_per_second": 1_000_000.0, + }, + } + + bench.write_outputs(tmp_path, result) + + metrics_json = json.loads((tmp_path / "metrics.json").read_text(encoding="utf-8")) + metrics_csv = (tmp_path / "metrics.csv").read_text(encoding="utf-8") + summary_md = (tmp_path / "summary.md").read_text(encoding="utf-8") + + assert metrics_json["performance"]["queries_per_second"] == 1_000_000.0 + assert "queries_per_second" in metrics_csv + assert "1000000.0" in metrics_csv + assert "Software CAM Retrieval Benchmark Summary" in summary_md + assert "queries_per_second" in summary_md