mirror of https://github.com/SikongJueluo/Mini-Nav.git synced 2026-07-12 20:15:31 +08:00

Files

feat(benchmarks): add noise injection experiment support to CAM retrieval benchmark

- Remove hard assertion blocking WRITE_NOISE_EN=1 in retrieval benchmark tests
- Add conditional exact_match assertion: enforces 100% when noise=off, skips when noise=on
- New script run_retrieval_noise_sweep.py: sweeps noise 0–100% (step 10%) and produces markdown summary
- Add just recipes: cam-benchmark-retrieval-sweep, cam-benchmark-sweep-cifar10, cam-benchmark-sweep-cifar100
- Add rsync-based remote sync commands for outputs and docs

2026-05-27 19:04:25 +08:00

4.6 KiB

Raw Blame History

CAM Retrieval Benchmark — Noise Sweep Summary

Generated: 2026-05-27 19:00:42

Configuration

Parameter	Value
Dataset path	`outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz`
NUM_ROWS	512
TOPK_K	5
HASH_BITS	512
Noise rates	0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%
Total runs	11
Passed	11
Failed	0

Dataset Details

Field	Value
num_queries	128
num_classes	10
seed	0

Results by Noise Rate

k=1

Noise (%)	WRITE_NOISE_EN	NUM/DEN	Hit@K	Precision@K	Hit-F1@K	Std-Recall@K	Std-F1@K	Golden Match@K	Status
0%	0	—	1.000000	1.000000	1.000000	0.019531	0.038314	1.000000	✓
10%	1	10/100	1.000000	1.000000	1.000000	0.019531	0.038314	0.507812	✓
20%	1	20/100	1.000000	1.000000	1.000000	0.019531	0.038314	0.234375	✓
30%	1	30/100	0.992188	0.992188	0.992188	0.019378	0.038014	0.164062	✓
40%	1	40/100	0.984375	0.984375	0.984375	0.019228	0.037719	0.093750	✓
50%	1	50/100	0.257812	0.257812	0.257812	0.005043	0.009893	0.023438	✓
60%	1	60/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
70%	1	70/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
80%	1	80/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
90%	1	90/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
100%	1	100/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓

k=5

Noise (%)	WRITE_NOISE_EN	NUM/DEN	Hit@K	Precision@K	Hit-F1@K	Std-Recall@K	Std-F1@K	Golden Match@K	Status
0%	0	—	1.000000	1.000000	1.000000	0.097656	0.177936	1.000000	✓
10%	1	10/100	1.000000	1.000000	1.000000	0.097656	0.177936	0.000000	✓
20%	1	20/100	1.000000	1.000000	1.000000	0.097656	0.177936	0.000000	✓
30%	1	30/100	1.000000	0.995313	0.997651	0.097197	0.177099	0.000000	✓
40%	1	40/100	1.000000	0.939062	0.968574	0.091729	0.167132	0.000000	✓
50%	1	50/100	0.750000	0.234375	0.357143	0.022913	0.041745	0.000000	✓
60%	1	60/100	0.015625	0.003125	0.005208	0.000306	0.000558	0.000000	✓
70%	1	70/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
80%	1	80/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
90%	1	90/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
100%	1	100/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓

Cross-Noise Comparison (primary: Hit@K)

Noise (%)	Hit@1	Hit@5	Δ(Hit@1 vs 0%)	Δ(Hit@5 vs 0%)
0%	1.000000	1.000000	+0.000000	+0.000000
10%	1.000000	1.000000	+0.000000	+0.000000
20%	1.000000	1.000000	+0.000000	+0.000000
30%	0.992188	1.000000	-0.007812	+0.000000
40%	0.984375	1.000000	-0.015625	+0.000000
50%	0.257812	0.750000	-0.742188	-0.250000
60%	0.000000	0.015625	-1.000000	-0.984375
70%	0.000000	0.000000	-1.000000	-1.000000
80%	0.000000	0.000000	-1.000000	-1.000000
90%	0.000000	0.000000	-1.000000	-1.000000
100%	0.000000	0.000000	-1.000000	-1.000000

Metric Definitions

Hit@K: fraction of queries where at least one relevant item appears in Top-K results (primary metric).
Precision@K: mean per-query precision — averaged tp/k across all queries.
Hit-F1@K: 2 × Hit@K × Precision@K / (Hit@K + Precision@K) — F1 using hit-rate recall.
Std-Recall@K: mean per-query standard retrieval recall — tp / |relevant| averaged across queries (supplementary).
Std-F1@K: 2 × Precision@K × Std-Recall@K / (Precision@K + Std-Recall@K) — F1 using standard recall (supplementary).
Golden Match@K: fraction of queries where DUT Top-K exactly matches the reference golden Top-K.

The paper uses Hit@K and Precision@K as primary metrics. Std-Recall@K and Std-F1@K are supplementary, included to show Top-K coverage against all relevant items in the database.

Results from Verilator/Cocotb simulation. Not measured on physical FPGA hardware.

4.6 KiB Raw Blame History Unescape Escape