mirror of
https://github.com/SikongJueluo/Mini-Nav.git
synced 2026-07-12 20:15:31 +08:00
SikongJueluo
7cb6257531
- Remove hard assertion blocking WRITE_NOISE_EN=1 in retrieval benchmark tests - Add conditional exact_match assertion: enforces 100% when noise=off, skips when noise=on - New script run_retrieval_noise_sweep.py: sweeps noise 0–100% (step 10%) and produces markdown summary - Add just recipes: cam-benchmark-retrieval-sweep, cam-benchmark-sweep-cifar10, cam-benchmark-sweep-cifar100 - Add rsync-based remote sync commands for outputs and docs
96 lines
4.6 KiB
Markdown
96 lines
4.6 KiB
Markdown
# CAM Retrieval Benchmark — Noise Sweep Summary
|
||
|
||
**Generated:** 2026-05-27 19:00:42
|
||
|
||
## Configuration
|
||
|
||
| Parameter | Value |
|
||
|---|---|
|
||
| Dataset path | `outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz` |
|
||
| NUM_ROWS | 512 |
|
||
| TOPK_K | 5 |
|
||
| HASH_BITS | 512 |
|
||
| Noise rates | 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100% |
|
||
| Total runs | 11 |
|
||
| Passed | 11 |
|
||
| Failed | 0 |
|
||
|
||
### Dataset Details
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| num_queries | 128 |
|
||
| num_classes | 10 |
|
||
| seed | 0 |
|
||
|
||
---
|
||
|
||
## Results by Noise Rate
|
||
|
||
### k=1
|
||
|
||
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|
||
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
|
||
| 0% | 0 | — | 1.000000 | 1.000000 | 1.000000 | 0.019531 | 0.038314 | 1.000000 | ✓ |
|
||
| 10% | 1 | 10/100 | 1.000000 | 1.000000 | 1.000000 | 0.019531 | 0.038314 | 0.507812 | ✓ |
|
||
| 20% | 1 | 20/100 | 1.000000 | 1.000000 | 1.000000 | 0.019531 | 0.038314 | 0.234375 | ✓ |
|
||
| 30% | 1 | 30/100 | 0.992188 | 0.992188 | 0.992188 | 0.019378 | 0.038014 | 0.164062 | ✓ |
|
||
| 40% | 1 | 40/100 | 0.984375 | 0.984375 | 0.984375 | 0.019228 | 0.037719 | 0.093750 | ✓ |
|
||
| 50% | 1 | 50/100 | 0.257812 | 0.257812 | 0.257812 | 0.005043 | 0.009893 | 0.023438 | ✓ |
|
||
| 60% | 1 | 60/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
| 70% | 1 | 70/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
|
||
### k=5
|
||
|
||
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|
||
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
|
||
| 0% | 0 | — | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 0.177936 | 1.000000 | ✓ |
|
||
| 10% | 1 | 10/100 | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 0.177936 | 0.000000 | ✓ |
|
||
| 20% | 1 | 20/100 | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 0.177936 | 0.000000 | ✓ |
|
||
| 30% | 1 | 30/100 | 1.000000 | 0.995313 | 0.997651 | 0.097197 | 0.177099 | 0.000000 | ✓ |
|
||
| 40% | 1 | 40/100 | 1.000000 | 0.939062 | 0.968574 | 0.091729 | 0.167132 | 0.000000 | ✓ |
|
||
| 50% | 1 | 50/100 | 0.750000 | 0.234375 | 0.357143 | 0.022913 | 0.041745 | 0.000000 | ✓ |
|
||
| 60% | 1 | 60/100 | 0.015625 | 0.003125 | 0.005208 | 0.000306 | 0.000558 | 0.000000 | ✓ |
|
||
| 70% | 1 | 70/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
|
||
|
||
---
|
||
|
||
## Cross-Noise Comparison (primary: Hit@K)
|
||
|
||
| Noise (%) | Hit@1 | Hit@5 | Δ(Hit@1 vs 0%) | Δ(Hit@5 vs 0%) |
|
||
|---|---:|---:|---:|---:|
|
||
| 0% | 1.000000 | 1.000000 | +0.000000 | +0.000000 |
|
||
| 10% | 1.000000 | 1.000000 | +0.000000 | +0.000000 |
|
||
| 20% | 1.000000 | 1.000000 | +0.000000 | +0.000000 |
|
||
| 30% | 0.992188 | 1.000000 | -0.007812 | +0.000000 |
|
||
| 40% | 0.984375 | 1.000000 | -0.015625 | +0.000000 |
|
||
| 50% | 0.257812 | 0.750000 | -0.742188 | -0.250000 |
|
||
| 60% | 0.000000 | 0.015625 | -1.000000 | -0.984375 |
|
||
| 70% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
|
||
| 80% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
|
||
| 90% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
|
||
| 100% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
|
||
|
||
---
|
||
|
||
## Metric Definitions
|
||
|
||
- **Hit@K**: fraction of queries where at least one relevant item appears in Top-K results (primary metric).
|
||
- **Precision@K**: mean per-query precision — averaged `tp/k` across all queries.
|
||
- **Hit-F1@K**: `2 × Hit@K × Precision@K / (Hit@K + Precision@K)` — F1 using hit-rate recall.
|
||
- **Std-Recall@K**: mean per-query standard retrieval recall — `tp / |relevant|` averaged across queries (supplementary).
|
||
- **Std-F1@K**: `2 × Precision@K × Std-Recall@K / (Precision@K + Std-Recall@K)` — F1 using standard recall (supplementary).
|
||
- **Golden Match@K**: fraction of queries where DUT Top-K exactly matches the reference golden Top-K.
|
||
|
||
The paper uses Hit@K and Precision@K as primary metrics. Std-Recall@K and Std-F1@K are supplementary,
|
||
included to show Top-K coverage against all relevant items in the database.
|
||
|
||
*Results from Verilator/Cocotb simulation. Not measured on physical FPGA hardware.*
|
||
|