mirror of
https://github.com/SikongJueluo/Mini-Nav.git
synced 2026-07-12 20:15:31 +08:00
feat(benchmarks): add noise injection experiment support to CAM retrieval benchmark
- Remove hard assertion blocking WRITE_NOISE_EN=1 in retrieval benchmark tests - Add conditional exact_match assertion: enforces 100% when noise=off, skips when noise=on - New script run_retrieval_noise_sweep.py: sweeps noise 0–100% (step 10%) and produces markdown summary - Add just recipes: cam-benchmark-retrieval-sweep, cam-benchmark-sweep-cifar10, cam-benchmark-sweep-cifar100 - Add rsync-based remote sync commands for outputs and docs
This commit is contained in:
95
docs/exps/cam_retrieval_noise_sweep_cifar100.md
Normal file
95
docs/exps/cam_retrieval_noise_sweep_cifar100.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# CAM Retrieval Benchmark — Noise Sweep Summary
|
||||
|
||||
**Generated:** 2026-05-27 19:00:46
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Dataset path | `outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz` |
|
||||
| NUM_ROWS | 512 |
|
||||
| TOPK_K | 5 |
|
||||
| HASH_BITS | 512 |
|
||||
| Noise rates | 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100% |
|
||||
| Total runs | 11 |
|
||||
| Passed | 11 |
|
||||
| Failed | 0 |
|
||||
|
||||
### Dataset Details
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| num_queries | 128 |
|
||||
| num_classes | 100 |
|
||||
| seed | 0 |
|
||||
|
||||
---
|
||||
|
||||
## Results by Noise Rate
|
||||
|
||||
### k=1
|
||||
|
||||
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|
||||
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
|
||||
| 0% | 0 | — | 0.695312 | 0.695312 | 0.695312 | 0.134635 | 0.225589 | 1.000000 | ✓ |
|
||||
| 10% | 1 | 10/100 | 0.585938 | 0.585938 | 0.585938 | 0.113281 | 0.189857 | 0.593750 | ✓ |
|
||||
| 20% | 1 | 20/100 | 0.562500 | 0.562500 | 0.562500 | 0.109115 | 0.182774 | 0.460938 | ✓ |
|
||||
| 30% | 1 | 30/100 | 0.460938 | 0.460938 | 0.460938 | 0.088802 | 0.148915 | 0.304688 | ✓ |
|
||||
| 40% | 1 | 40/100 | 0.234375 | 0.234375 | 0.234375 | 0.044792 | 0.075210 | 0.101562 | ✓ |
|
||||
| 50% | 1 | 50/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
| 60% | 1 | 60/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
| 70% | 1 | 70/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
|
||||
### k=5
|
||||
|
||||
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|
||||
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
|
||||
| 0% | 0 | — | 0.867188 | 0.462500 | 0.603261 | 0.445052 | 0.453608 | 1.000000 | ✓ |
|
||||
| 10% | 1 | 10/100 | 0.812500 | 0.421875 | 0.555380 | 0.405990 | 0.413780 | 0.023438 | ✓ |
|
||||
| 20% | 1 | 20/100 | 0.742188 | 0.364062 | 0.488502 | 0.350000 | 0.356893 | 0.000000 | ✓ |
|
||||
| 30% | 1 | 30/100 | 0.640625 | 0.248437 | 0.358029 | 0.238802 | 0.243525 | 0.000000 | ✓ |
|
||||
| 40% | 1 | 40/100 | 0.460938 | 0.117187 | 0.186867 | 0.113021 | 0.115066 | 0.000000 | ✓ |
|
||||
| 50% | 1 | 50/100 | 0.062500 | 0.015625 | 0.025000 | 0.014844 | 0.015224 | 0.000000 | ✓ |
|
||||
| 60% | 1 | 60/100 | 0.007812 | 0.001563 | 0.002604 | 0.001563 | 0.001563 | 0.000000 | ✓ |
|
||||
| 70% | 1 | 70/100 | 0.007812 | 0.001563 | 0.002604 | 0.001563 | 0.001563 | 0.000000 | ✓ |
|
||||
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||||
|
||||
---
|
||||
|
||||
## Cross-Noise Comparison (primary: Hit@K)
|
||||
|
||||
| Noise (%) | Hit@1 | Hit@5 | Δ(Hit@1 vs 0%) | Δ(Hit@5 vs 0%) |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 0% | 0.695312 | 0.867188 | +0.000000 | +0.000000 |
|
||||
| 10% | 0.585938 | 0.812500 | -0.109375 | -0.054688 |
|
||||
| 20% | 0.562500 | 0.742188 | -0.132812 | -0.125000 |
|
||||
| 30% | 0.460938 | 0.640625 | -0.234375 | -0.226562 |
|
||||
| 40% | 0.234375 | 0.460938 | -0.460938 | -0.406250 |
|
||||
| 50% | 0.000000 | 0.062500 | -0.695312 | -0.804688 |
|
||||
| 60% | 0.000000 | 0.007812 | -0.695312 | -0.859375 |
|
||||
| 70% | 0.000000 | 0.007812 | -0.695312 | -0.859375 |
|
||||
| 80% | 0.000000 | 0.000000 | -0.695312 | -0.867188 |
|
||||
| 90% | 0.000000 | 0.000000 | -0.695312 | -0.867188 |
|
||||
| 100% | 0.000000 | 0.000000 | -0.695312 | -0.867188 |
|
||||
|
||||
---
|
||||
|
||||
## Metric Definitions
|
||||
|
||||
- **Hit@K**: fraction of queries where at least one relevant item appears in Top-K results (primary metric).
|
||||
- **Precision@K**: mean per-query precision — averaged `tp/k` across all queries.
|
||||
- **Hit-F1@K**: `2 × Hit@K × Precision@K / (Hit@K + Precision@K)` — F1 using hit-rate recall.
|
||||
- **Std-Recall@K**: mean per-query standard retrieval recall — `tp / |relevant|` averaged across queries (supplementary).
|
||||
- **Std-F1@K**: `2 × Precision@K × Std-Recall@K / (Precision@K + Std-Recall@K)` — F1 using standard recall (supplementary).
|
||||
- **Golden Match@K**: fraction of queries where DUT Top-K exactly matches the reference golden Top-K.
|
||||
|
||||
The paper uses Hit@K and Precision@K as primary metrics. Std-Recall@K and Std-F1@K are supplementary,
|
||||
included to show Top-K coverage against all relevant items in the database.
|
||||
|
||||
*Results from Verilator/Cocotb simulation. Not measured on physical FPGA hardware.*
|
||||
|
||||
Reference in New Issue
Block a user