mirror of https://github.com/SikongJueluo/Mini-Nav.git synced 2026-07-13 04:25:32 +08:00

Files

feat(benchmarks): add noise injection experiment support to CAM retrieval benchmark

- Remove hard assertion blocking WRITE_NOISE_EN=1 in retrieval benchmark tests
- Add conditional exact_match assertion: enforces 100% when noise=off, skips when noise=on
- New script run_retrieval_noise_sweep.py: sweeps noise 0–100% (step 10%) and produces markdown summary
- Add just recipes: cam-benchmark-retrieval-sweep, cam-benchmark-sweep-cifar10, cam-benchmark-sweep-cifar100
- Add rsync-based remote sync commands for outputs and docs

2026-05-27 19:04:25 +08:00

4.6 KiB

Raw Blame History

CAM Retrieval Benchmark — Noise Sweep Summary

Generated: 2026-05-27 19:00:46

Configuration

Parameter	Value
Dataset path	`outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz`
NUM_ROWS	512
TOPK_K	5
HASH_BITS	512
Noise rates	0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%
Total runs	11
Passed	11
Failed	0

Dataset Details

Field	Value
num_queries	128
num_classes	100
seed	0

Results by Noise Rate

k=1

Noise (%)	WRITE_NOISE_EN	NUM/DEN	Hit@K	Precision@K	Hit-F1@K	Std-Recall@K	Std-F1@K	Golden Match@K	Status
0%	0	—	0.695312	0.695312	0.695312	0.134635	0.225589	1.000000	✓
10%	1	10/100	0.585938	0.585938	0.585938	0.113281	0.189857	0.593750	✓
20%	1	20/100	0.562500	0.562500	0.562500	0.109115	0.182774	0.460938	✓
30%	1	30/100	0.460938	0.460938	0.460938	0.088802	0.148915	0.304688	✓
40%	1	40/100	0.234375	0.234375	0.234375	0.044792	0.075210	0.101562	✓
50%	1	50/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
60%	1	60/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
70%	1	70/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
80%	1	80/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
90%	1	90/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
100%	1	100/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓

k=5

Noise (%)	WRITE_NOISE_EN	NUM/DEN	Hit@K	Precision@K	Hit-F1@K	Std-Recall@K	Std-F1@K	Golden Match@K	Status
0%	0	—	0.867188	0.462500	0.603261	0.445052	0.453608	1.000000	✓
10%	1	10/100	0.812500	0.421875	0.555380	0.405990	0.413780	0.023438	✓
20%	1	20/100	0.742188	0.364062	0.488502	0.350000	0.356893	0.000000	✓
30%	1	30/100	0.640625	0.248437	0.358029	0.238802	0.243525	0.000000	✓
40%	1	40/100	0.460938	0.117187	0.186867	0.113021	0.115066	0.000000	✓
50%	1	50/100	0.062500	0.015625	0.025000	0.014844	0.015224	0.000000	✓
60%	1	60/100	0.007812	0.001563	0.002604	0.001563	0.001563	0.000000	✓
70%	1	70/100	0.007812	0.001563	0.002604	0.001563	0.001563	0.000000	✓
80%	1	80/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
90%	1	90/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓
100%	1	100/100	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	✓

Cross-Noise Comparison (primary: Hit@K)

Noise (%)	Hit@1	Hit@5	Δ(Hit@1 vs 0%)	Δ(Hit@5 vs 0%)
0%	0.695312	0.867188	+0.000000	+0.000000
10%	0.585938	0.812500	-0.109375	-0.054688
20%	0.562500	0.742188	-0.132812	-0.125000
30%	0.460938	0.640625	-0.234375	-0.226562
40%	0.234375	0.460938	-0.460938	-0.406250
50%	0.000000	0.062500	-0.695312	-0.804688
60%	0.000000	0.007812	-0.695312	-0.859375
70%	0.000000	0.007812	-0.695312	-0.859375
80%	0.000000	0.000000	-0.695312	-0.867188
90%	0.000000	0.000000	-0.695312	-0.867188
100%	0.000000	0.000000	-0.695312	-0.867188

Metric Definitions

Hit@K: fraction of queries where at least one relevant item appears in Top-K results (primary metric).
Precision@K: mean per-query precision — averaged tp/k across all queries.
Hit-F1@K: 2 × Hit@K × Precision@K / (Hit@K + Precision@K) — F1 using hit-rate recall.
Std-Recall@K: mean per-query standard retrieval recall — tp / |relevant| averaged across queries (supplementary).
Std-F1@K: 2 × Precision@K × Std-Recall@K / (Precision@K + Std-Recall@K) — F1 using standard recall (supplementary).
Golden Match@K: fraction of queries where DUT Top-K exactly matches the reference golden Top-K.

The paper uses Hit@K and Precision@K as primary metrics. Std-Recall@K and Std-F1@K are supplementary, included to show Top-K coverage against all relevant items in the database.

Results from Verilator/Cocotb simulation. Not measured on physical FPGA hardware.

4.6 KiB Raw Blame History Unescape Escape