Compare commits

...

83 Commits

Author SHA1 Message Date
b5a40819cc feat: vectorize CAM retrieval with NumPy and add multi-worker support
- Replace scalar hamming distance with NumPy bitwise_count for batch retrieval
- Add ThreadPoolExecutor-based multi-worker query parallelism
- Improve missing dataset error message with generation command hint
- Increase DEFAULT_MAX_QUERIES from 128 to 8192 for meaningful throughput tests
2026-06-07 20:45:20 +08:00
e5b764520c feat(scenegraph): add JSON serialization and implement top-down object/edge rendering
- Add save_scene_graph() and load_scene_graph() for persisting scene graphs to JSON
- Implement object node overlay (colored by label) in render_topdown_scene_map
- Implement edge arrows (room → object) in render_topdown_scene_map
- Add TopDownRenderStyle fields for object/edge visual configuration
- Add scene graph caching to verification notebook to skip rebuilds
- Add render_scene_graph_birdseye cell for full scene graph visualization
- Add display_objects_by_room and display_room_summary cells
2026-05-31 19:48:33 +08:00
9eb52f8cef feat(benchmark): support multi-dataset evaluation with configurable top-k list
- Evaluate multiple datasets in a single run (CIFAR10 + CIFAR100)
- Report Recall@K for a list of K values from one underlying search
- Save results to disk: summary.md, metrics.csv, per-dataset predictions.csv, confusion_matrix.csv
- Richer evaluation output: query_labels, topk_ids, topk_labels for downstream analysis
- Add --dataset and --top-k CLI overrides for benchmark command
- Update config schema: dataset→datasets, top_k→top_k_list
2026-05-31 18:58:01 +08:00
7a1e1ccf3f feat(scenegraph): add depth-based 3D positioning via pinhole projection
- Add bbox_depth_center position strategy in SceneGraphBuilder using depth
  at bbox centre and configurable camera_hfov_degrees for pinhole projection.
- Add optional depth_sensor_uuid to HabitatSimulatorConfig; create depth
  sensor spec alongside RGB sensor.
- Add camera_position/camera_rotation fields to RoomView; capture pose from
  sensor_states when depth sensor is available.
- Update flatten_room_views for backward compatibility with legacy tuple
  format.
- Wired in depth sensor and bbox_depth_center strategy in verification
  notebook.
- Add tests for depth sensor support and new position strategies.
2026-05-31 14:37:46 +08:00
a127032e18 feat(scenegraph): add SceneGraphBuilder for pipeline-driven graph construction
Introduce SceneGraphBuilder + SceneGraphBuildConfig to decouple scene graph
construction from the verification notebook. The builder handles batch
inference, hash encoding, and object node creation internally.

- Add SceneGraphBuilder.build_from_room_views() as the main entry point
- Add SceneGraphBuildConfig for inference_batch_size and position strategy
- Add SceneGraphBuildArtifacts to carry cropped images and debug metadata
- Extend ObjectNode with optional detection metadata (label, confidence,
  bbox_xyxy, source_view_id, position_confidence)
- Add RoomView frozen dataclass as a structured view container
- Add flatten_room_views() utility to replace inline list comprehensions
- Refactor notebooks/verification.py to use the new builder API

BREAKING CHANGE: ObjectNode now accepts additional optional fields; direct
scene_graph.objects[obj_id] = ObjectNode(...) construction in the notebook
is replaced by builder.build_from_room_views(...).
2026-05-30 16:57:38 +08:00
97e53d44f8 feat(hw/sim): distinguish query-only and end-to-end performance cycles in retrieval benchmark
Add explicit separation between query-only cycles (accept→last) and end-to-end cycles
(load + write + noise + queries) in hardware retrieval benchmarks.

- Add query_only_cycles_per_query, load_write_noise_cycles, end_to_end_cycles metrics
- Refactor summarize_query_timings() to use accept_to_last_result_cycles as query-only base
- Add build_hardware_performance() to compute end-to-end performance separately
- Add current_sim_cycle() helper using cocotb get_sim_time
- Update CSV/Markdown outputs and RETRIEVAL_PERF_RESULT log format
- Update documentation to clarify cycle-counting methodology
- Update tests to cover new performance measurement logic
2026-05-29 18:49:05 +08:00
42d4a9728d feat: add hardware retrieval cycle performance measurement
Add cycle-level performance measurement for hardware CAM retrieval benchmarks
to complement existing quality metrics.

- Add query_topk_once_with_latency with accept→first/last cycle timing
- Add QueryTiming dataclass and summarize_query_timings helper
- Integrate cycle performance into benchmark outputs (CSV + Markdown)
- Log RETRIEVAL_PERF_RESULT with cycles/query and queries/cycle
- Update experiment docs with hardware cycle performance section
- Add unit tests for summarize_query_timings and output writers
2026-05-28 13:05:34 +08:00
715a2c2ed6 feat(architecture): add CAM hardware block design diagram
- Add docs/architecture/cam_hardware_block_design.drawio
- Diagram illustrates CAM top-level architecture including:
  - Host write path with write-priority half-duplex arbiter
  - Write noise injector with Bernoulli flip mask
  - Banked CAM storage (8 BRAM banks × 512 depth)
  - 8-lane match engine pipeline with popcount
  - Top-K tracker and result serializer
- Supports documentation for 4096-row × 512-bit hash CAM design
2026-05-28 12:09:26 +08:00
acf0c75132 feat(benchmark): add software CAM retrieval benchmark
Add software-based CAM retrieval benchmark to compare retrieval quality
and speed against hardware simulation. Includes experiment documentation
with noise sweep analysis on CIFAR-10/100 datasets.

- Add sw_retrieval_benchmark.py for software Hamming distance Top-K retrieval
- Add test_sw_retrieval_benchmark.py with unit tests for dataset loading and metrics
- Add experiment doc (sw_hw_cam_retrieval_benchmark.md) comparing software vs hardware
- Document noise sweep impact on retrieval quality at various WRITE_NOISE_RATE values
2026-05-27 19:52:14 +08:00
7cb6257531 feat(benchmarks): add noise injection experiment support to CAM retrieval benchmark
- Remove hard assertion blocking WRITE_NOISE_EN=1 in retrieval benchmark tests
- Add conditional exact_match assertion: enforces 100% when noise=off, skips when noise=on
- New script run_retrieval_noise_sweep.py: sweeps noise 0–100% (step 10%) and produces markdown summary
- Add just recipes: cam-benchmark-retrieval-sweep, cam-benchmark-sweep-cifar10, cam-benchmark-sweep-cifar100
- Add rsync-based remote sync commands for outputs and docs
2026-05-27 19:04:25 +08:00
d6e1b9d8ba refactor(cam): rename read-noise path to read-pass-through and reorganize test module structure
- Rename `read_noise` scenarios and noise_mode to `read_pass_through` across
  run_cam_correctness.py, test_run_cam_correctness.py, and Makefiles
- Update RTL comment in match_engine_pipeline.sv to reflect pass-through behavior
- Move unit-level cocotb tests from `tests.test_*` flat namespace to
  `tests.modules.*` and `tests.top.*` subdirectory layout, matching actual
  Makefile paths (hw/sim/tests/modules/..., hw/sim/tests/top/...)
- Remove redundant dual-noise subtarget from read_noise/Makefile
- Update help text and docs to reflect read-path pass-through semantics
- Add .codegraph to .gitignore
2026-05-27 16:34:34 +08:00
8b4d4c1b57 refactor(cam): remove read noise from noise architecture (Phase 2)
- Make cam_read_noise a pass-through module, removing all noise injection logic
- Switch write noise to use noise_mask_bernoulli instead of noise_mask_grouped
- Add state machine to cam_write_noise for mask generation timing
- Remove noise_mask_grouped.sv (no longer needed)
- Remove read noise parameters from cam_noisy and cam_top
- Update simulation and benchmark code to reflect read noise removal
- Sync documentation to reflect Phase 2 architecture
2026-05-26 23:45:52 +08:00
e5d13917b2 feat(cam): add Bernoulli noise mask module and cocotb tests
- Add noise_mask_bernoulli.sv RTL module for probabilistic masking
- Add cocotb test suite with reset, threshold, determinism, and distribution checks  
- Update rtl-sources.mk to include new module
- Fix PYTHONPATH in devenv.nix shell hook
2026-05-26 19:41:17 +08:00
1ff9a5f18b feat(retrieval-benchmark): add support for external pre-prepared CAM retrieval datasets with recall@k metric
- Add just recipes for preparing CIFAR10/100 hash artifacts and running benchmarks
- Add CAM_RETRIEVAL_DATASET env var support in Makefile
- Add load_retrieval_dataset_npz() to load pre-prepared retrieval datasets
- Add label_hits counter and recall@k metric for retrieval evaluation
- Rename macro_recall to retrieval_recall to clarify semantics
2026-05-22 21:07:10 +08:00
e1bed00cc4 feat(retrieval): add CAM retrieval benchmark with topk scoring and read noise support
- Add cocotb benchmark infrastructure under hw/sim/benchmarks/retrieval/ with Makefile
- Implement test_retrieval_benchmark.py supporting configurable topk-k, read/write noise
- Add cluster-based synthetic dataset generator with configurable bit-flip rates
- Add reference model functions: match_topk, match_topk_from_scores, score_rows_with_read_noise
- Add .justfile shortcuts: cam-test-retrieval-no-noise, cam-test-retrieval-read-noise
- Add TOPK_K to Verilator EXTRA_ARGS via cocotb-common.mk
- Add unit tests for topk sorting logic and stateful read-noise scoring
2026-05-22 19:04:54 +08:00
29f4cc91f6 fix(pipeline): add S_WAIT_READ_RESP state to fix read-noise PRNG timing
- Emit rd_valid_o for exactly one combinational cycle before waiting for
  read response, ensuring the read-noise PRNG advances once per batch issue
- Fix query handshake: wait for query_ready before asserting query_valid
  to avoid valid&&ready handshake drops on clock edges
- Add dynamic timeout estimation in test utilities based on DUT parameters
- Update test-top Makefile to run all noise configurations by default
- Remove uv run prefix from cocotb-config Makefile invocation
2026-05-22 19:04:50 +08:00
424cf6e1d3 refactor(hw/sim): reorganize CAM top-level tests into per-noise-config suites
Split the monolithic test_cam_basic.py into separate test suites
organized by noise configuration (no_noise, write_noise, read_noise),
with shared utilities extracted to tests/top/utils.py.

- Remove test_cam_basic.py; add no_noise/, write_noise/, read_noise/
  test suites with Makefiles that set noise parameters statically
- Extract helpers (reset_dut, write_rows, query_once, collect_topk,
  etc.) into tests/top/utils.py
- Update hw/sim/Makefile with per-config test targets and a
  test-top-all meta-target
- Update scripts/run_cam_correctness.py to build per-directory
  instead of centrally, removing inline parameter overrides
- Add __init__.py for result_serializer and topk_tracker module tests
- Expand docstrings in test_ref_model_noise.py with architectural
  context and test rationale
2026-05-21 21:22:12 +08:00
5a1d3ea977 refactor(verification): extract shared text_labels cell and hash codec
- Move inline _text_labels lists into a shared text_labels cell
- Replace inline np.packbits hash encoding with bits_tensor_to_hash_bytes
- Add static tests to verify notebook API usage patterns
2026-05-21 16:41:39 +08:00
101a675ece feat(scenegraph): add ImageHashPipeline protocol and update query API
- Introduce ImageHashPipeline Protocol for extensible hash computation
- Rename query_crop_index to query_index for clarity
- Make query_crop nullable to handle missing crop edge case
- Add keyword-only arguments for better API clarity
- Handle empty scene graph objects gracefully
- Add comprehensive test coverage for query_image_against_scene_graph
2026-05-21 15:33:10 +08:00
ba96cec406 feat(scenegraph): refactor image scene graph query into reusable function
- Export ImageSceneGraphQueryResult and query_image_against_scene_graph from scenegraph module
- Replace inline hamming-distance-based image matching with dedicated query_image_against_scene_graph function
- Improve top_matches structure by extracting similarity scores and hash_bytes from matches
- Add .codegraph/ to gitignore (machine-local data, should not be committed)
- Add CodeGraph configuration for multi-language indexing
2026-05-21 14:25:50 +08:00
e4cbb5e30d feat(hw/rtl): implement full Top-K CAM search pipeline with serial result output
- add TOPK_K, FIFO_DEPTH, RESULT_SERIAL parameters to cam_params
- add candidate_fifo: synchronous ready/valid FIFO for (row, score) candidates
- add topk_tracker: tracks top-K candidates with clear/ready handshake
- add result_serializer: serializes packed Top-K array into rank-ordered stream
- refactor match_engine_pipeline from 4-state to 8-state Top-K pipeline
- extend cam_top with serial Top-K interface (result_rank/row/score/last)
- add backward-compatible top1_index/top1_score aliases from rank-0 beat
- add comprehensive tests for all new modules
2026-05-19 22:43:21 +08:00
8bcad1f23f refactor(core/cam_core_banked): extract per-bank modules for improved timing isolation
- Extract cam_bank as a parameterized submodule with independent read/write ports
- Replace flat 2D memory array with generate loop of bank instances
- Derive bank selection from address bit slicing instead of modulo arithmetic
- Align rd_base_row_i check with new bank addressing scheme
- Add test verifying bank address isolation across multiple banks
2026-05-19 16:17:08 +08:00
5d09f13a08 refactor(env): remove torch from conda env and add yosys for conda 2026-05-18 15:58:11 +08:00
706d148a0b feat(hw): add CAM top-level synthesis infrastructure and fix RTL synthesis compatibility
- Add hw/syn/Makefile with hier/flat/full synth targets and artifact mirroring
- Add synth_cam_top_hier.ys for hierarchy-preserving resource estimation on Xilinx 7-series
- Add synth_cam_top_flat.ys for flattened Xilinx 7-series synthesis
- Add cam-synth just target for convenient invocation
- Guard runtime assertions (NUM_ROWS/LANES checks, noise seed checks, NOISE_BITS checks)
  behind SYNTHESIS guard in cam_core_banked, cam_read_noise, cam_write_noise, and noise_mask_grouped
- Fix shadowed 'return' variable in random128 xorshift128 function
2026-05-18 15:40:04 +08:00
b9b5684718 test(popcount_pipeline): add cocotb test for popcount pipeline module
- Add Makefile with cocotb configuration for popcount_pipeline RTL simulation
- Add Python test module with reset, drive/expect helpers
- Test row metadata preservation and bit-count correctness across 7 vectors
2026-05-18 15:37:26 +08:00
583b2156ea feat(scenegraph): add SoftwareCamIndex for visual hash similarity queries
- Add SoftwareCamIndex class with xnor_popcount_score for CAM-style matching
- Add CamMatch and SceneGraphMatch dataclasses for query results
- Add query_by_visual_hash method to SimpleSceneGraph
- Add comprehensive tests for SoftwareCamIndex and xnor_popcount_score
2026-05-17 21:57:01 +08:00
ddb8cff6a9 feat(scenegraph): add hash codec for bits/tensor/bytes/cam_row conversion
Introduce hash_codec module providing bidirectional encoding/decoding:
- bits_tensor_to_hash_bytes / hash_bytes_to_bits_array
- bits_tensor_to_cam_row
- hash_bytes_to_cam_row / cam_row_to_hash_bytes
2026-05-17 19:41:44 +08:00
4ea567adba feat(compressors): add JSONL training metrics logging with CLI controls
- Add write_training_metrics() in new compressors/training_metrics.py
  for appending epoch/step/lr/component rows as JSON Lines
- Wire --metrics-path and --log-every CLI options into train.py, passing
  them to the training loop so metrics rows are written every N steps
- Accept absolute metrics paths or paths relative to output directory
- Add quantization component to loss log alongside existing distill/contrastive
- Replace inline torch.device() with get_device() utility
- Add test_hash_training_metrics.py covering multi-row JSONL append

Infrastructure:
- Pin torch 2.7.1 + CUDA 12.8 index for Linux/Windows in pyproject.toml
- Add .justfile rsync upload recipe with .stignore exclusion
- Exclude **/__marimo__ from rsync in .stignore

Dependencies updated: numpy 2.4.5, pandas 3.0.3, black 26.5.0,
click 8.4.0, contourpy, etc.
2026-05-17 14:57:10 +08:00
e8c890a69f feat(sim): enable read noise by default and improve test infrastructure
- Set READ_NOISE_EN=1 as default in cam_top.sv
- Wire READ_NOISE_* parameters to NOISE_* makefile variables in cocotb-common.mk
- Add PYTHON env variable support in Makefile for test invocation
- Update .gitignore with glob patterns (**/sim_build/, **/results.xml, *.fst)
- Add justfile recipe for remote cam-test-top execution
2026-05-17 14:11:59 +08:00
e7765cdb76 refactor(build): consolidate CAM test commands under new make targets
- .envrc: add conda compiler flags (CFLAGS, CXXFLAGS, LDFLAGS) for zlib support
- .justfile: replace inline make commands with reusable targets (test-top, test-all, test-model, test-module)
- environment.yml: reformat YAML, add zlib/gtkwave, rename withbullet → with_bullet
2026-05-16 19:57:34 +08:00
ca167e79c6 refactor(hw/sim): extract common cocotb make infrastructure into shared mk/ directory
- Split monolithic hw/sim/Makefile into modular include files (mk/cocotb-common.mk, mk/rtl-sources.mk)
- Add per-module test Makefiles (cam_core_banked, cam_read_noise, cam_write_noise, match_engine_pipeline, perf, top)
- Add missing simulation tools (yosys, graphviz, xdot) to devenv.nix
- Fix path handling in sweep_noise.py and test modules to be HASH_BITS-aware
2026-05-16 19:24:17 +08:00
2e0e36eea5 feat(sim): add CAM performance sweep test infrastructure
- hw/sim/tests/test_cam_perf.py: new cocotb perf test with bounded wait helpers
- scripts/sweep_cam_perf.py: sweep data model, matrix, and make command builders  
- tests/test_sweep_cam_perf.py: unit tests for sweep helpers
- tests/conftest.py: pytest path configuration for scripts package
- hw/sim/Makefile: deterministic noise params override for perf test compatibility
2026-05-16 15:42:29 +08:00
24b8750f0f feat(sim): add CAM correctness runner with scenario matrix and contract tests
Introduce scripts/run_cam_correctness.py — a data-driven CAM correctness
checker that runs cocotb and pytest scenarios across multiple noise modes
and CAM submodules (cam_top, cam_core_banked, match_engine_pipeline).

Outputs structured CSV results and a paper-ready Chinese summary.
Add tests/test_run_cam_correctness.py with contract tests for scenario
spec validation, command building, subprocess error handling, output file
generation, CLI defaults, and environment preservation.
2026-05-15 14:03:40 +08:00
3203b2d9af chore(project): adjust safety rules 2026-05-15 14:03:40 +08:00
ab616528b4 refactor(benchmark): delegate model loading to tasks and support CIFAR-100
- Extract model loading logic from benchmark CLI into task-owned prepare_benchmark
- Add RetrievalEncoder class wrapping DINO with optional hash compression
- Add accelerate dependency for device management  
- Switch dataset from CIFAR-10 to CIFAR-100 with fine_label column
2026-05-15 09:59:40 +08:00
0fbcd915bd refactor: reorganize RTL files into core/noise subdirectories
- Move CAM core modules (cam_core_banked, match_engine_pipeline, popcount_pipeline) to hw/rtl/core/
- Move noise modules (cam_read_noise, cam_write_noise, noise_mask_grouped) to hw/rtl/noise/
- Update Makefile include paths and VERILOG_SOURCES to reflect new layout
- Update docs/experiments.md file path references
- Add sim/results.xml to .gitignore
- Bump devenv.lock dependencies
2026-05-14 20:59:46 +08:00
443edbfa25 docs: add experiments feasibility survey for paper sections 6.2–6.8 2026-05-13 20:03:58 +08:00
e1a34d6540 refactor!: remove legacy match_engine and related CAM modules
- Delete popcount.sv, argmax_update.sv, cam_core.sv, match_engine.sv
- Remove obsolete VERILOG_SOURCES entries from Makefile
- Update comment in cam_top.sv to reference match_engine_pipeline
- Add verilator to devenv.nix for simulation support
2026-05-13 19:11:33 +08:00
8f59a287c4 feat(hw): add banked CAM pipeline with grouped read/write noise
- Add cam_core_banked.sv with 8-lane banked CAM core
- Add cam_write_noise.sv and cam_read_noise.sv for grouped noise injection
- Add noise_mask_grouped.sv generating grouped flip masks from 128-bit PRNG
- Add match_engine_pipeline.sv with multi-stage pipelined top-1 selection
- Add popcount_pipeline.sv for pipelined popcount operations
- Refactor test_cam_basic.py with parametrized DUT introspection helpers
- Add Python ref_model match_top1_with_read_noise() for read noise verification
- Update Makefile with separate WRITE_NOISE_* and READ_NOISE_* parameter groups
- Add new testbenches: test_cam_core_banked, test_cam_read_noise,
  test_cam_write_noise, test_match_engine_pipeline, test_ref_model_noise
  
breaking change hint: NUM_ROWS default changed from 512→4096, LANES from 16→8
2026-05-13 18:22:28 +08:00
c41e64d1c6 feat(dev): add CAM test recipes and update ignore files
- Add `.opencode/` and `**/sim_build` to ignore lists
- Fix justfile syntax: `set dotenv-required := true` (explicit assignment)
- Update SSH tunnel port 8384 → 42705 for local port forwarding
- Add CAM verification recipes: cam-test-all, cam-test-py, cam-test-module, cam-test
2026-05-13 18:22:14 +08:00
cbafc4524e feat(cam): migrate noise generation from xorshift64 to xorshift128
- Replace NOISE_GEN_BITS/NOISE_SAMPLE_BITS parameters with unified NOISE_BITS
- Use xorshift128 (random128) instead of xorshift64 for PRNG
- Add flip_mask_next combinational helper for single-cycle mask computation
- Add random_enable signal to advance PRNG only on accepted noisy writes
- Simplify FSM by removing mask_group_idx counter
- Update parameter validation: GROUP_BITS (= HASH_BITS/NOISE_BITS) must equal 64
- Update ref_model.py and tests to match new seed convention: {seed, seed}
- Update Makefile and sweep_noise.py with renamed parameters
2026-05-05 20:19:22 +08:00
0dd01fb1b7 feat(hw/rtl): add xorshift PRNG modules and refactor cam_noisy FSM
- Add random32, random64 and random128 xorshift PRNG modules
- Refactor cam_noisy FSM: split state register, next-state logic, and datapath into distinct blocks
- Rename state_q/state_d to curr_state/next_state for clarity
- Add MASK_GROUPS localparam and fix type casting in noise generation
- Update .gitignore to exclude docs/superpowers
2026-05-05 19:30:50 +08:00
2da17e101b feat(rtl): migrate CAM interface to handshake protocol with integrated noise generation
BREAKING CHANGE: CAM write and query interface replaced with standard valid/ready
handshake. wr_en/wr_row/wr_hash → wr_valid/wr_ready/wr_addr/write_hash.
External noise_mask_lanes_flat removed; noise generation now handled internally
by cam_noisy module with configurable rate via parameters.

- cam_top: add parameters (NOISE_EN, NOISE_RATE_NUM/DEN, NOISE_GEN/SAMPLE_BITS, NOISE_SEED)
- cam_top: replace cam_core with cam_noisy (integrated noise generation)
- match_engine: remove external noise_mask_lanes_flat input
- hw/sim: update Makefile with noise parameters and compile args
- hw/sim/model: add generate_write_flip_mask() and xorshift64() matching RTL behavior
- hw/sim/tests: adapt testbench to new handshake protocol
2026-05-04 18:03:06 +08:00
0ae6d757dc refactor(scripts): extract shared utilities into common module
- Move load_env_file, parse_timeout_seconds, build_remote_script, and
  build_ssh_command to scripts/common.py
- Update remote_docker_run.py to import from common module
- Improves code organization and reusability
2026-05-03 19:49:45 +08:00
8b8e4d3118 build(simulation): improve verilator simulation infrastructure
- Add verilator to dependencies
- Add configurable logging via QUIET/VERBOSE/COCOTB_LOG_LEVEL env vars
- Add optional warning suppression (SUPPRESS_VERILATOR_WARNINGS)
- Clean up and restructure Makefile
2026-05-03 15:33:17 +08:00
7450505a86 feat(remote): unify remote execution workflow with env-based config
- Replace hardcoded SSH/docker targets with environment variables
- Add remote-check, remote-dry, remote-test, remote-test-one recipes
- Add remote cmd for arbitrary command execution in docker container
2026-05-03 15:32:32 +08:00
f5daaa2667 refactor(cam): extract match engine into separate module and centralize parameters
- Split cam_core into pure memory (cam_core.sv) and match engine (match_engine.sv)
- Add cam_params.svh with centralized parameter definitions (NUM_ROWS, HASH_BITS, LANES, etc.)
- Update cam_top.sv to use shared parameters and compose match_engine
- Update Makefile to include new match_engine module and correct Verilator define syntax
2026-05-02 23:28:32 +08:00
f71bf06484 feat(hw): add XNOR-popcount CAM design with cocotb verification
Implement a multi-lane Content Addressable Memory (CAM) that scores
rows by XNOR popcount against a query hash and returns the top-1 match.

RTL modules:
- popcount: parallel group-based population count
- argmax_update: iterative best-match tracking with tie-break
- cam_core: parameterized scanning engine (NUM_ROWS/HASH_BITS/LANES)
  with optional SIM_NOISE and SIM_DEBUG ifdef guards
- cam_top: thin wrapper exposing cam_core ports

Verification:
- Python reference model (ref_model.py) for score-level golden comparison
- cocotb testbench (test_cam_basic.py) covering write/query/reset and
  external noise mask scenarios with score debug verification
- Noise sweep script (sweep_noise.py) measuring top-1 stability under
  configurable bit-flip rates
- Verilator-oriented Makefile with parameterizable compile options
2026-05-02 23:26:16 +08:00
ad45123022 chore: migrate habitat dependencies to conda and add cocotb for hardware verification
- Move habitat-baselines and habitat-lab from pip to conda environment
- Add cocotb and cocotb-tools to pyproject.toml dependencies
- Update ty environment roots to include hw/sim directory
- Add sim/sim_build to gitignore
- Create CLAUDE.md with project spec, coding guidelines, and directory structure
- Update uv.lock to reflect dependency changes
2026-05-02 23:24:59 +08:00
d9745f45dc refactor(visualization): enhance proposal filtering visualization 2026-04-18 23:24:30 +08:00
e3cdc1c6e7 chore(project): update .envrc and justfile for server, and update marimo 2026-04-18 21:10:22 +08:00
8a56c9649d refactor(verification): batch pipeline inference with progress tracking
- Increase verification params: image_size 512→768, rooms 4→5, views 6→12
- Refactor single-batch inference to chunked batch processing with mo.progress_bar
- Extract debug_meta and hash_batches from output for clearer variable flow
- Add progress bars to room-view snapshot saving and scene graph building
- Add .ruff_cache/, .pytest_cache/, .sisyphus/ to .justfile upload excludes
2026-04-11 17:09:39 +08:00
79b49f122a feat(test): add collect test images notebook and replace BitImageProcessorFast 2026-04-11 15:22:22 +08:00
01017277c3 feat(project): update marimo and env autoload 2026-04-11 15:21:11 +08:00
eb016385f8 refactor(sync): improve topdown rendering and make justfile adapt to windows 2026-04-06 19:55:52 +08:00
2c78f4b662 refactor(verification): improve object creation / matching and image saving 2026-04-06 13:27:02 +08:00
37b4a08398 refactor(pipeline): support multi-object selection in frame processing 2026-04-06 10:27:23 +08:00
3638ffdb8d feat(compressors): refactor pipeline with FramePacket dataclass and unified process_batch
- Add FramePacket dataclass to encapsulate per-image pipeline state
- Rename internal methods with underscore prefix convention
- Replace separate filter_batch/crop_batch with unified process_batch method
- Update notebook to use new HashPipeline API
2026-04-04 20:09:05 +08:00
94ed05a039 feat(compressors): add OWLv2 bbox crop to HashPipeline and refactor image utilities
- Add Owlv2ForObjectDetection and Owlv2Processor imports to model_loader
- Refactor load_dino_model to return tuple of processor and model
- Rewrite generate_proposals_batch to group images by bbox count for efficient batching
- Add _normalize_single_bbox_list helper for bbox normalization
- Update verification.py to use new pipeline architecture with detect/segment/filter/crop steps
2026-04-04 15:27:47 +08:00
5f41cf5794 feat(compressors): add OWLv2 bbox crop to HashPipeline and refactor image utilities
- Add crop_batch method to HashPipeline for cropping images using OWLv2 detection boxes
- Integrate crop_batch into pipeline forward pass (extract_hash and extract_features)
- Move extract_masked_region from compressors/proposal/utils.py to utils/image.py
- Add crop_image_by_bbox utility function in utils/image.py
- Update type annotations to use dict[str, Any] instead of bare dict
- Update .justfile to add memory server command
- Update marimo dependency to >=0.22.0
- Update nvidia CUDA package markers for platform compatibility
2026-04-03 19:43:43 +08:00
4e16e38f32 refactor(compressors): migrate pipeline to OWLv2-based detection with text labels
- Replace bbox-prompted segmentation with OWLv2 text-guided object detection
- Refactor HashPipeline from nn.Module to plain class with modular stage methods
- Add detect_batch, segment_batch, filter_batch for explicit pipeline stages
- Rename forward to forward_batch with text_labels API instead of bboxes
- Add mask_scoring_config, score_threshold, postprocess_threshold configuration
- Update model_loader to expose Dinov2Model type annotation
2026-04-03 18:07:17 +08:00
4918b654e7 refactor(compressors): migrate to centralized model loaders
- Refactor model_loader.py: improve return type annotations from tuple[Any, Any] to tuple[AutoImageProcessor, AutoModel]
- Refactor proposal/core.py: add input validation for mask array dimensionality, handle 2D masks and batch dimensions gracefully
- Refactor proposal_segament.ipynb: replace inline model loading with centralized load_owlv2_model() and load_sam_model() functions, use batched detect_objects_batch() and generate_proposals_batch() APIs
2026-04-03 15:00:14 +08:00
af0531a5eb feat(compressors/proposal): add OWLv2 object detection pipeline and typed results
- switch OWLv2 loader return type to Owlv2ForObjectDetection
- add detect_objects and detect_objects_batch with two-stage score filtering
- add DetectionResult typed dict and conversion helper for post-processed outputs
- export new detection APIs from proposal module
2026-04-02 20:26:13 +08:00
42acb3ee1b refactor(compressors): switch SAM from automatic mask generation to bbox-prompted segmentation
- Replace SAM2AutomaticMaskGenerator pipeline with Sam2Processor+Sam2Model
- Freeze SAM model parameters at load time, removing torch.no_grad() at call sites
- Rewrite proposal/core.py to use bbox prompts instead of automatic point sampling
- Add bboxes parameter to all HashPipeline public methods (forward, forward_dataset, extract_features, extract_features_dataset)
- Extract mask filtering logic (_filter_masks) from proposal into pipeline
- Rename object_score/ to filter/
- Add load_owlv2_model to model_loader
- Rename notebooks/test.py to habitat_sim_setup.py
2026-04-02 16:47:11 +08:00
eaf02cc97a chore: update dev environment and sync workflow
- Add nil LSP package to devenv.nix
- Add datasets and rich.progress imports to feature_retrieval.py
- Update pyproject.toml ty environment config
- Expand .stignore with cache directories
- Update uv.lock dependency lock
- Update AGENTS.md, adding the content of development environment11
2026-04-02 15:02:51 +08:00
aedcb25610 feat(notebooks): add proposal segmentation notebook and enhance verification 2026-04-01 11:54:23 +08:00
f06b0625b4 refactor(verification): reorganize imports and function structure 2026-03-30 22:07:35 +08:00
cb93d83868 feat(simulator): add image saving utilities for verification 2026-03-30 21:49:51 +08:00
26b00e556a refactor(compressors): reorganize SAM utilities into proposal module 2026-03-30 21:31:47 +08:00
f421b0c56b chore(tests): remove all test files from mini-nav 2026-03-30 21:31:47 +08:00
e544a7e84f docs(agents): update agent guidelines and remove memorix references 2026-03-30 21:31:46 +08:00
a809803979 feat(compressors): add object scoring and selection for SAM masks 2026-03-30 16:50:35 +08:00
f6c1a67e88 feat(verification): add batch segmentation and image saving 2026-03-28 21:30:13 +08:00
f604c85a79 feat(pipeline): add batch processing for scene graph construction 2026-03-28 19:37:52 +08:00
3c9a6f6eaf refactor(simulator): extract habitat simulator and visualization modules 2026-03-28 17:05:29 +08:00
817f45b935 feat(scenegraph): add image storage and verification UI 2026-03-28 16:02:31 +08:00
1c9752cf9e refactor(verification): add progress bars and simplify device handling 2026-03-26 20:08:16 +08:00
968819e113 refactor(compressors): consolidate pipeline and improve mask handling 2026-03-26 19:44:48 +08:00
90d5a8f08a refactor(pipeline): integrate SAM segmentation and modularize model loading 2026-03-26 19:44:47 +08:00
9e6339e580 chore(deps): add sam2 dependency and update docker config 2026-03-26 19:44:47 +08:00
1af9a9caa1 feat(notebooks): add habitat simulation and room visualization 2026-03-24 21:03:02 +08:00
d2481684ac feat(scenegraph): add basic scene graph data structures 2026-03-24 19:49:24 +08:00
55750f1087 chore(project): update configuration files and dependencies 2026-03-24 19:49:06 +08:00
166 changed files with 23606 additions and 5727 deletions

16
.codegraph/.gitignore vendored Normal file
View File

@@ -0,0 +1,16 @@
# CodeGraph data files
# These are local to each machine and should not be committed
# Database
*.db
*.db-wal
*.db-shm
# Cache
cache/
# Logs
*.log
# Hook markers
.dirty

143
.codegraph/config.json Normal file
View File

@@ -0,0 +1,143 @@
{
"version": 1,
"include": [
"**/*.ts",
"**/*.tsx",
"**/*.js",
"**/*.jsx",
"**/*.py",
"**/*.go",
"**/*.rs",
"**/*.java",
"**/*.c",
"**/*.h",
"**/*.cpp",
"**/*.hpp",
"**/*.cc",
"**/*.cxx",
"**/*.cs",
"**/*.php",
"**/*.rb",
"**/*.swift",
"**/*.kt",
"**/*.kts",
"**/*.dart",
"**/*.svelte",
"**/*.vue",
"**/*.liquid",
"**/*.pas",
"**/*.dpr",
"**/*.dpk",
"**/*.lpr",
"**/*.dfm",
"**/*.fmx",
"**/*.scala",
"**/*.sc"
],
"exclude": [
"**/.git/**",
"**/node_modules/**",
"**/vendor/**",
"**/Pods/**",
"**/dist/**",
"**/build/**",
"**/out/**",
"**/bin/**",
"**/obj/**",
"**/target/**",
"**/*.min.js",
"**/*.bundle.js",
"**/.next/**",
"**/.nuxt/**",
"**/.svelte-kit/**",
"**/.output/**",
"**/.turbo/**",
"**/.cache/**",
"**/.parcel-cache/**",
"**/.vite/**",
"**/.astro/**",
"**/.docusaurus/**",
"**/.gatsby/**",
"**/.webpack/**",
"**/.nx/**",
"**/.yarn/cache/**",
"**/.pnpm-store/**",
"**/storybook-static/**",
"**/.expo/**",
"**/web-build/**",
"**/ios/Pods/**",
"**/ios/build/**",
"**/android/build/**",
"**/android/.gradle/**",
"**/__pycache__/**",
"**/.venv/**",
"**/venv/**",
"**/site-packages/**",
"**/dist-packages/**",
"**/.pytest_cache/**",
"**/.mypy_cache/**",
"**/.ruff_cache/**",
"**/.tox/**",
"**/.nox/**",
"**/*.egg-info/**",
"**/.eggs/**",
"**/go/pkg/mod/**",
"**/target/debug/**",
"**/target/release/**",
"**/.gradle/**",
"**/.m2/**",
"**/generated-sources/**",
"**/.kotlin/**",
"**/.dart_tool/**",
"**/.vs/**",
"**/.nuget/**",
"**/artifacts/**",
"**/publish/**",
"**/cmake-build-*/**",
"**/CMakeFiles/**",
"**/bazel-*/**",
"**/vcpkg_installed/**",
"**/.conan/**",
"**/Debug/**",
"**/Release/**",
"**/x64/**",
"**/.pio/**",
"**/release/**",
"**/*.app/**",
"**/*.asar",
"**/DerivedData/**",
"**/.build/**",
"**/.swiftpm/**",
"**/xcuserdata/**",
"**/Carthage/Build/**",
"**/SourcePackages/**",
"**/__history/**",
"**/__recovery/**",
"**/*.dcu",
"**/.composer/**",
"**/storage/framework/**",
"**/bootstrap/cache/**",
"**/.bundle/**",
"**/tmp/cache/**",
"**/public/assets/**",
"**/public/packs/**",
"**/.yardoc/**",
"**/coverage/**",
"**/htmlcov/**",
"**/.nyc_output/**",
"**/test-results/**",
"**/.coverage/**",
"**/.idea/**",
"**/logs/**",
"**/tmp/**",
"**/temp/**",
"**/_build/**",
"**/docs/_build/**",
"**/site/**"
],
"languages": [],
"frameworks": [],
"maxFileSize": 1048576,
"extractDocstrings": true,
"trackCallSites": true
}

6
.editorconfig Normal file
View File

@@ -0,0 +1,6 @@
root = true
[*]
end_of_line = lf
insert_final_newline = true
charset = utf-8

28
.envrc
View File

@@ -14,6 +14,32 @@ else
# 如果不存在 devenv则加载自己的 fallback 配置
echo "devenv not found, loading fallback configuration..."
export UV_PROJECT_ENVIRONMENT="/workspace/envs/mini-nav"
export UV_PROJECT_ENVIRONMENT="/home/ial-pangyg/workspace/envs/mini-nav"
export MY_DEV_ENV=1
layout_mamba() {
# initialize micromamba
eval "$(micromamba shell hook --shell=bash)"
# check if environment name is specified
if [ -n "$1" ]; then
# Explicit environment name from layout command.
local env_name="$1"
# activate environment
micromamba activate ${env_name}
elif (grep -q name: environment.yml 2> /dev/null); then
# Detect environment name from `environment.yml` file in `.envrc` directory
micromamba activate `grep name: environment.yml | sed -e 's/name: //'`
else
# if all else fails, activate base environment
micromamba activate base;
fi;
}
layout mamba
# Static Link Libraries
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
export CFLAGS="-I$CONDA_PREFIX/include"
export CXXFLAGS="-I$CONDA_PREFIX/include"
export LDFLAGS="-L$CONDA_PREFIX/lib -Wl,-rpath,$CONDA_PREFIX/lib -lz"
fi

6
.gitignore vendored
View File

@@ -136,6 +136,7 @@ celerybeat.pid
# Environments
.env
.env.remote
.venv
env/
venv/
@@ -211,11 +212,16 @@ datasets/
data/
deps/
outputs/
.vscode/
# Vibe Coding
.sisyphus
.claude/settings.local.json
openspec/changes/
.codegraph/
.logs/
docs/superpowers
.workspace
# Devenv
.devenv*

137
.justfile
View File

@@ -1,21 +1,18 @@
upload:
rsync -avLh --progress --stats --itemize-changes \
--exclude='.jj/' \
--exclude='.git/' \
--exclude='.devenv/' \
--exclude='.direnv/' \
--exclude='deps/' \
--exclude='outputs/' \
--exclude='data/versioned_data/' \
. ial-jumper-ial-pangyg:/home/ial-pangyg/docker-workspace/projects/mini-nav/
set dotenv-required := true
download:
rsync -avLh --progress --stats --itemize-changes \
ial-jumper-ial-pangyg:/home/ial-pangyg/docker-workspace/projects/mini-nav/outputs .
export MSYS2_ARG_CONV_EXCL := "*"
export MSYS2_ENV_CONV_EXCL := "*"
remote_ssh_target := env("REMOTE_SSH_TARGET")
remote_docker_container := env("REMOTE_DOCKER_CONTAINER")
remote_root := "$REMOTE_SSH_TARGET:$REMOTE_WORKDIR"
rsync_flags := "-avLh --progress --stats --itemize-changes"
upload_excludes := "--exclude-from=.stignore"
upload:
rsync {{ rsync_flags }} {{ upload_excludes }} . {{ remote_root }}/; \
sync-pkgs:
# micromamba env create -f ./environment.yml
export UV_PROJECT_ENVIRONMENT="/workspace/envs/mini-nav/" && uv sync --inexact
uv sync --inexact
sync-data:
python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/
@@ -23,18 +20,20 @@ sync-data:
ssh:
ssh \
-L 127.0.0.1:22001:127.0.0.1:22000 \
-R 127.0.0.1:22001:127.0.0.1:22000 \
-L 127.0.0.1:8385:127.0.0.1:8384 \
-L 127.0.0.1:8385:127.0.0.1:42705 \
-L 127.0.0.1:9098:127.0.0.1:9098 \
-L 127.0.0.1:2718:172.30.0.2:2718 \
ial-gpu_workstation_1
{{ remote_ssh_target }}
docker:
docker exec -it docker-ial-pangyg bash
docker cmd="":
@if [ -z "{{ cmd }}" ]; then \
docker exec -it -w {{ env("REMOTE_WORKDIR") }} {{ remote_docker_container }} bash; \
else \
docker exec -i -w {{ env("REMOTE_WORKDIR") }} {{ remote_docker_container }} bash -lc {{ quote(cmd) }}; \
fi
marimo:
export UV_PROJECT_ENVIRONMENT="/workspace/envs/mini-nav/" \
&& uv run marimo edit ./mini-nav/habitat/test.py --host 0.0.0.0 --port 2718
marimo +notebook:
uv run marimo edit {{ notebook }} --host 0.0.0.0 --port 2718 --no-token
add +packages:
uv add {{ packages }} --no-sync
@@ -44,10 +43,88 @@ remove +packages:
uv remove {{ packages }} --no-sync
just sync-pkgs
add-dev +packages:
uv add {{ packages }} --group dev --no-sync
just sync-pkgs
remote cmd:
uv run python scripts/remote_docker_run.py run -- {{ quote(cmd) }}
remove-dev +packages:
uv remove {{ packages }} --group dev --no-sync
just sync-pkgs
remote-check:
uv run python scripts/remote_docker_run.py check
remote-dry cmd:
uv run python scripts/remote_docker_run.py run --dry-run -- {{ quote(cmd) }}
# ── CAM verification ──────────────────────────────────────────────────────────
# Run all CAM tests in one pass (model + top + all modules)
cam-test-full:
just remote "make -C hw/sim clean && make -C hw/sim test-all"
# Run top CAM tests
cam-test-top:
just remote "make -C hw/sim clean && make -C hw/sim test-top"
# Run full-scale CAM Yosys synthesis and resource reporting
cam-synth:
make -C hw/syn synth
# Run Python reference model tests
cam-test-py:
make -C hw/sim test-model
# Run a specific CAM module test (e.g., cam-test-module MODULE=cam_core_banked)
cam-test-module MODULE:
just remote "make -C hw/sim clean && make -C hw/sim test-module MODULE={{ MODULE }}"
# Run a single test case within a module (e.g., cam-test MODULE=cam_core_banked TESTCASE=banked_core_reads_aligned_eight_row_batch_after_one_cycle)
cam-test MODULE TESTCASE:
just remote "make -C hw/sim clean && make -C hw/sim test-module MODULE={{ MODULE }} COCOTB_TESTCASE={{ TESTCASE }}"
# Run CAM retrieval benchmark without hardware noise
cam-test-retrieval-no-noise:
just remote "make -C hw/sim clean && make -C hw/sim test-benchmark-retrieval TOPK_K=5 WRITE_NOISE_EN=0"
# Run CAM retrieval benchmark with write noise enabled (Phase 2: read noise removed)
cam-test-retrieval-write-noise:
just remote "make -C hw/sim clean && make -C hw/sim test-benchmark-retrieval TOPK_K=5 WRITE_NOISE_EN=1 WRITE_NOISE_RATE_NUM=1 WRITE_NOISE_RATE_DEN=100"
# Prepare CIFAR10 hash artifact for CAM retrieval smoke benchmark
cam-prepare-retrieval-cifar10 ROWS="512" QUERIES="128":
just remote "python scripts/prepare_cam_retrieval_dataset.py --dataset cifar10 --num-rows {{ ROWS }} --max-queries {{ QUERIES }} --compressor-path outputs/hash_compressor.pt"
# Prepare CIFAR100 hash artifact for CAM retrieval benchmark
cam-prepare-retrieval-cifar100 ROWS="512" QUERIES="128":
just remote "python scripts/prepare_cam_retrieval_dataset.py --dataset cifar100 --num-rows {{ ROWS }} --max-queries {{ QUERIES }} --compressor-path outputs/hash_compressor.pt"
# Run CAM retrieval benchmark on a prepared artifact without hardware noise
cam-test-retrieval-artifact DATASET_PATH NUM_ROWS="4096":
just remote "make -C hw/sim clean && make -C hw/sim test-benchmark-retrieval TOPK_K=5 NUM_ROWS={{ NUM_ROWS }} WRITE_NOISE_EN=0 CAM_RETRIEVAL_DATASET={{ DATASET_PATH }}"
# Run CAM retrieval benchmark on a prepared artifact with write noise enabled (Phase 2: read noise removed)
cam-test-retrieval-artifact-write-noise DATASET_PATH NUM_ROWS="4096":
just remote "make -C hw/sim clean && make -C hw/sim test-benchmark-retrieval TOPK_K=5 NUM_ROWS={{ NUM_ROWS }} WRITE_NOISE_EN=1 WRITE_NOISE_RATE_NUM=1 WRITE_NOISE_RATE_DEN=100 CAM_RETRIEVAL_DATASET={{ DATASET_PATH }}"
# ── CAM retrieval benchmark noise sweep ────────────────────────────────────────
# Run noise sweep on a prepared dataset (0%100%, step 10%)
# Usage: just cam-benchmark-retrieval-sweep DATASET=outputs/.../cifar10_hash512_rows512_queries128.npz NUM_ROWS=512
cam-benchmark-retrieval-sweep DATASET NUM_ROWS="512":
just remote "python scripts/run_retrieval_noise_sweep.py --dataset {{ DATASET }} --num-rows {{ NUM_ROWS }} --output docs/cam_retrieval_noise_sweep.md"
# Prepare CIFAR10 dataset + run full noise sweep (all-in-one)
cam-benchmark-sweep-cifar10 ROWS="512" QUERIES="128":
just cam-prepare-retrieval-cifar10 {{ ROWS }} {{ QUERIES }}
just remote "python scripts/run_retrieval_noise_sweep.py --dataset outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows{{ ROWS }}_queries{{ QUERIES }}.npz --num-rows {{ ROWS }} --output docs/cam_retrieval_noise_sweep_cifar10.md"
# Prepare CIFAR100 dataset + run full noise sweep (all-in-one)
cam-benchmark-sweep-cifar100 ROWS="512" QUERIES="128":
just cam-prepare-retrieval-cifar100 {{ ROWS }} {{ QUERIES }}
just remote "python scripts/run_retrieval_noise_sweep.py --dataset outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows{{ ROWS }}_queries{{ QUERIES }}.npz --num-rows {{ ROWS }} --output docs/cam_retrieval_noise_sweep_cifar100.md"
# ── Remote ↔ local sync ────────────────────────────────────────────────────────
# Download benchmark outputs from remote
download-outputs:
rsync {{ rsync_flags }} {{ remote_root }}/outputs/cam_retrieval_benchmark/ outputs/cam_retrieval_benchmark/
# Download docs from remote
download-docs:
rsync {{ rsync_flags }} {{ remote_root }}/docs/ docs/

5
.opencode/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
plugins/
node_modules
package.json
package-lock.json
bun.lock

16
.opencode/opencode.json Normal file
View File

@@ -0,0 +1,16 @@
{
"$schema": "https://opencode.ai/config.json",
"permission": {
"bash": {
"python": "deny",
"python *": "deny",
"just remote *": "ask"
},
"read": {
".env": "deny"
},
"edit": {
".env": "deny"
}
}
}

12
.safety-net.json Normal file
View File

@@ -0,0 +1,12 @@
{
"$schema": "https://raw.githubusercontent.com/kenryu42/claude-code-safety-net/main/assets/cc-safety-net.schema.json",
"version": 1,
"rules": [
{
"name": "block-uv-sync",
"command": "uv",
"block_args": ["sync"],
"reason": "The project is managed by micromamba and uv. Do not sync packages directly by uv. Use \"just sync-pkgs\" or \"uv sync --inexact\" instead."
}
]
}

View File

@@ -1,5 +1,19 @@
.git
.jj
.venv
.devenv
.direnv
.pytest_cache
.ruff_cache
.opencode
.backup
.codegraph
data
datasets
deps
outputs
.sisyphus
**/__pycache__
**/sim_build
**/__marimo__
*.fst

126
AGENTS.md Normal file
View File

@@ -0,0 +1,126 @@
# Memorix — Automatic Memory Rules
You have access to Memorix memory tools. Follow these rules to maintain persistent context across sessions.
## RULE 1: Session Start — Bind Project, Then Load Context
At the **beginning of every conversation**, BEFORE responding to the user:
1. Call `memorix_session_start` with parameters:
- `agent`: your agent identifier (e.g. "windsurf", "codex", "antigravity")
- `projectRoot`: the **absolute path** of the current workspace or repo root
This binds the session to the correct project. Without `projectRoot`, memories may go to the wrong bucket.
2. Then call `memorix_search` with a query related to the user's first message for additional context
3. If search results are found, use `memorix_detail` to fetch the most relevant ones
4. Reference relevant memories naturally — the user should feel you "remember" them
**Important:** `projectRoot` is a detection anchor only; Git remains the source of truth for project identity.
In HTTP control-plane mode (`memorix serve-http` / `memorix background start`), explicit `projectRoot` binding is required for correct multi-project isolation.
`memorix_session_start` is lightweight by default: it starts memory/session context only. Do not set `joinTeam` unless the user explicitly needs autonomous Agent Team tasks, messages, file locks, or orchestrated CLI-agent workflows.
## RULE 2: Store Important Context
**Proactively** call `memorix_store` when any of the following happen:
### What MUST be recorded:
- Architecture/design decisions → type: `decision`
- Bug identified and fixed → type: `problem-solution`
- Unexpected behavior or gotcha → type: `gotcha`
- Config changed (env vars, ports, deps) → type: `what-changed`
- Feature completed or milestone → type: `what-changed`
- Trade-off discussed with conclusion → type: `trade-off`
### What should NOT be recorded:
- Simple file reads, greetings, trivial commands (ls, pwd, git status)
### Use topicKey for evolving topics:
For decisions, architecture docs, or any topic that evolves over time, ALWAYS use `topicKey` parameter.
This ensures the memory is UPDATED instead of creating duplicates.
Use `memorix_suggest_topic_key` to generate a stable key.
Example: `topicKey: "architecture/auth-model"` — subsequent stores with the same key update the existing memory.
### Track progress with the progress parameter:
When working on features or tasks, include the `progress` parameter:
```json
{
"progress": {
"feature": "user authentication",
"status": "in-progress",
"completion": 60
}
}
```
Status values: `in-progress`, `completed`, `blocked`
## RULE 3: Resolve Completed Memories
When a task is completed, a bug is fixed, or information becomes outdated:
1. Call `memorix_resolve` with the observation IDs to mark them as resolved
2. Resolved memories are hidden from default search, preventing context pollution
This is critical — without resolving, old bug reports and completed tasks will keep appearing in future searches.
## RULE 4: Session End — Store Decision Chain Summary
When the conversation is ending, create a **decision chain summary** (not just a checklist):
1. Call `memorix_store` with type `session-request` and `topicKey: "session/latest-summary"`:
**Required structure:**
```
## Goal
[What we were working on — specific, not vague]
## Key Decisions & Reasoning
- Chose X because Y. Rejected Z because [reason].
- [Every architectural/design decision with WHY]
## What Changed
- [File path] — [what changed and why]
## Current State
- [What works now, what's pending]
- [Any blockers or risks]
## Next Steps
- [Concrete next actions, in priority order]
```
**Critical: Include the "Key Decisions & Reasoning" section.** Without it, the next AI session will lack the context to understand WHY things were done a certain way and may suggest conflicting approaches.
2. Call `memorix_resolve` on any memories for tasks completed in this session
## RULE 5: Compact Awareness
Memorix automatically compacts memories on store:
- **With LLM API configured:** Smart dedup — extracts facts, compares with existing, merges or skips duplicates
- **Without LLM (free mode):** Heuristic dedup — uses similarity scores to detect and merge duplicate memories
- **You don't need to manually deduplicate.** Just store naturally and compact handles the rest.
- If you notice excessive duplicate memories, call `memorix_deduplicate` for batch cleanup.
## Guidelines
- **Use concise titles** (~5-10 words) and structured facts
- **Include file paths** in filesModified when relevant
- **Include related concepts** for better searchability
- **Always use topicKey** for recurring topics to prevent duplicates
- **Always resolve** completed tasks and fixed bugs
- **Always include reasoning** — "chose X because Y" is 10x more valuable than "did X"
- Search defaults to `status="active"` — use `status="all"` to include resolved memories
## Beyond These Rules
This file contains the **minimum operating rules** for Memorix memory tools. It is NOT the complete truth about runtime behavior, support tiers, or team semantics.
For authoritative, up-to-date details on:
- **Support tiers** (core / extended / community) and what "installed" vs "runtime-ready" means
- **HTTP control-plane binding** and `projectRoot` isolation rules
- **Opt-in team semantics** (`joinTeam`, `team_manage join`, roles, task claim, handoff validation)
- **Install vs runtime-ready distinction** — hook config written ≠ agent will execute it
- **Agent-specific caveats** (Copilot project-level only, OpenCode plugin lifecycle, etc.)
→ **Read `docs/AGENT_OPERATOR_PLAYBOOK.md`** in the Memorix source or npm package.
If this file and the playbook conflict, the playbook is authoritative.

View File

@@ -6,13 +6,11 @@
详细参阅https://raw.githubusercontent.com/shendeguize/GooglePythonStyleGuideCN/refs/heads/master/README.md
### 代码编写原则
- 简洁,清晰易懂,最小实现
- 简洁,清晰易懂,最小闭环实现
- 条件或循环分支不能超过三层提前Return以减少分支的出现
- 变量说明注释、条件或循环分支注释完全
- 无需向后兼容,避免添加过多功能
- 先编写测试集,再实现代码
- 实现测试集后,先询问用户意见,用户确认后才能继续
- 如非用户要求,无需编写基准测试代码
- 非用户要求,不能编写测试代码
- 英文注释,中文文档
- 完成代码编写后在文档的框架不变的情况下更新文档如CLAUDE.md
@@ -20,14 +18,6 @@
- 精简、干净、快速
- 核心关键逻辑或算法必须测试
- 需要加载transformer模型进行验证的测试与无需加载模型的测试分离
- 无需编写测试集的情况
- UI界面相关的代码
- 过于复杂或耗时的逻辑
- 基准测试相关
### 关键词说明
- 确认:用户认同当前的实现方案或测试集实现,即可以开始工作
- 继续:用户需要你重读上下文,继续未完成的工作
### 文档更新说明
仅在工程目录变化时,更新此文档的目录说明部分。
@@ -35,6 +25,8 @@
## 工程说明
使用UV管理整个工程pytest用于测试justfile用于快捷命令jujutsu用于版本管理。
本地开发环境欠缺无法验证需要调用GPU相关的测试因此请勿在本地跑验证。
请在远程开发环境中进行相关验证(相关脚本暂时为编写完毕)。
### 目录说明
@@ -71,8 +63,8 @@
详细可查询pyproject.toml或使用`uv pip list`获取详细的库信息,请基于目前的库实现功能。
如需添加新库,请先询问,用户确认后才能使用`uv add <package>`新增库。
## 版本管理 (Jujutsu 特有)
本项目使用 Jujutsu (jj) 进行版本控制,并配套 Memorix MCP 作为架构决策与思维轨迹的持久化中心
## 版本管理
本项目使用 Jujutsu (jj) 进行版本控制。
- 技能调用: 必须使用 jujutsu 相关工具技能来执行分支、提交、修改describe等操作禁止直接通过 Shell 执行冗长的 Git 兼容指令。
- 描述规范 (jj desc):
@@ -80,13 +72,6 @@
- 空一行后,仅记录改动的核心业务点。
- 语言使用英文进行描述
- 禁忌: 禁止在 jj 描述中堆砌复杂的算法逻辑或长篇的设计决策。
- 记忆联动 (Memorix 优先):
- 凡涉及架构变更、算法决策或重构逻辑,在执行 jj desc 之前,必须先调用 memorix_store (或对应的添加方法)。
- 关联标记: 在 Memorix 的存储记录中,必须强制包含当前变更的 jj change ID以便实现从代码变更到思维链的完美映射。
- 检索逻辑: 在处理需要深入理解上下文的任务时,应主动调用 memorix_search 检索相关的历史 change_id 决策。
- 无感记录原则:
- 严禁在工程目录下生成任何独立的 change_log.md 或 AI 自动化文档。
- 所有关于“为什么这样改”的知识,应当流向 jj 的原子化提交描述或 Memorix 的知识图谱库。
### 描述示例
```text
@@ -104,22 +89,3 @@ refactor(compressors): Simplify module by removing SAM/DINO separation code
- 调用记忆功能如Memorix记忆先前总结的内容
- 遵循描述规范使用jj进行更改的描述
- 执行`jj new`开启一个新的更改
## 记忆管理 (Memorix MCP)
本项目使用 Memorix 作为核心上下文引擎,用于存储架构决策、复杂逻辑关联和历史重构原因。
### 记忆写入准则
- 主动记录: 在完成以下操作后,必须调用 `memorix.store`
- 用户确认后的核心架构变更例如LanceDB 的索引策略)。
- 复杂的 bug 修复逻辑(记录“为什么”这么修,防止回滚)。
- 用户在对话中表达的明确偏好(例如:对特定 Python 库的厌恶)。
- 代码的修改及其决策逻辑(例如:对于用户特定需求导致的更改)。
- 结构化存储: 存储时请使用 `[Category: Topic] Description` 的格式,确保检索效率。
### 记忆检索准则
- 冷启动检索: 每一轮新对话开始或切换到新任务时,优先调用 `memorix.search` 关键词(如 "project_architecture", "database_schema"),以确保不偏离既有设计。
- 防止幻觉: 如果对某个旧功能的实现细节不确定,先检索记忆,禁止凭空猜测。
### 内存与冗余控制
- 精简描述: 存入 Memorix 的信息必须精简,严禁存入整段代码块,仅存储“逻辑描述”和“决策依据”。
- 清理逻辑: 发现记忆库中存在与当前代码事实冲突的旧信息时,应主动提示用户进行更新或覆盖。

View File

@@ -3,11 +3,11 @@
"devenv": {
"locked": {
"dir": "src/modules",
"lastModified": 1773767086,
"narHash": "sha256-3eKyl4LXswf6P17/u59x8oQXDgCfYX+UX0uh7YHFwJc=",
"lastModified": 1778705971,
"narHash": "sha256-n0LjnKBAjJ5/mgNzOCeVvAeHUrNMUZ3fBQx/UDCkHtQ=",
"owner": "cachix",
"repo": "devenv",
"rev": "a5e5a7f8b1c9a7f33f9d82192b692768b39ec710",
"rev": "64d4353a3628c4138c84d8ba10987da2ba27fddd",
"type": "github"
},
"original": {
@@ -61,11 +61,11 @@
"nixpkgs-src": "nixpkgs-src"
},
"locked": {
"lastModified": 1773704619,
"narHash": "sha256-LKtmit8Sr81z8+N2vpIaN/fyiQJ8f7XJ6tMSKyDVQ9s=",
"lastModified": 1778507786,
"narHash": "sha256-HzSQCKMsMr8r55LwM1JuzIOB+8bzk0FEv6sItKvsfoY=",
"owner": "cachix",
"repo": "devenv-nixpkgs",
"rev": "906534d75b0e2fe74a719559dfb1ad3563485f43",
"rev": "8f24a228a782e24576b155d1e39f0d914b380691",
"type": "github"
},
"original": {
@@ -78,11 +78,11 @@
"nixpkgs-src": {
"flake": false,
"locked": {
"lastModified": 1773597492,
"narHash": "sha256-hQ284SkIeNaeyud+LS0WVLX+WL2rxcVZLFEaK0e03zg=",
"lastModified": 1778274207,
"narHash": "sha256-I4puXmX1iovcCHZlRmztO3vW0mAbbRvq4F8wgIMQ1MM=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "a07d4ce6bee67d7c838a8a5796e75dff9caa21ef",
"rev": "b3da656039dc7a6240f27b2ef8cc6a3ef3bccae7",
"type": "github"
},
"original": {

View File

@@ -4,22 +4,31 @@
config,
inputs,
...
}: let
}:
let
pkgs-nixgl =
(import inputs.nixpkgs {
system = pkgs.stdenv.hostPlatform.system;
config.allowUnfree = true;
overlays = [inputs.nixgl.overlay];
}).nixgl.override {
nvidiaVersionFile = "/proc/driver/nvidia/version";
nvidiaVersion = "580.126.09";
};
in {
packages = [
overlays = [ inputs.nixgl.overlay ];
}).nixgl.override
{
nvidiaVersionFile = "/proc/driver/nvidia/version";
nvidiaVersion = "580.126.09";
};
in
{
packages = with pkgs; [
nil
verilator
yosys
graphviz
xdot
];
enterShell = ''
export UV_PROJECT_ENVIRONMENT=$HOME/.local/share/mamba/envs/mini-nav/
export UV_PROJECT_ENVIRONMENT=$MAMBA_ROOT_PREFIX/envs/mini-nav
unset PYTHONPATH
eval "$(micromamba shell hook --shell bash)"
micromamba activate mini-nav

View File

@@ -0,0 +1,95 @@
<mxfile host="Electron" agent="opencode">
<diagram id="cam-hardware-block-design" name="CAM Hardware Block Design">
<mxGraphModel dx="1405" dy="906" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1280" pageHeight="720" background="none" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="title" parent="1" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=28;fontStyle=1;fontColor=#1B4332;" value="CAM Hardware Architecture" vertex="1">
<mxGeometry height="40" width="480" x="400" y="180" as="geometry" />
</mxCell>
<mxCell id="subtitle" parent="1" style="text;html=1;strokeColor=none;fillColor=none;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=14;fontColor=#52796F;" value="4096 rows · 512-bit hashes · 8 lanes · Top-K serial result · optional 1% write noise" vertex="1">
<mxGeometry height="28" width="650" x="315" y="220" as="geometry" />
</mxCell>
<mxCell id="cam_top" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#D8F3DC;strokeColor=#2D6A4F;fontColor=#1B4332;fontStyle=1;" value="cam_top&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;write-priority half-duplex arbiter&lt;/font&gt;" vertex="1">
<mxGeometry height="270" width="200" x="50" y="260" as="geometry" />
</mxCell>
<mxCell id="host_write" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#E9F5DB;strokeColor=#74A57F;fontColor=#1B4332;" value="Host Write&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;wr_valid / wr_ready&lt;/font&gt;" vertex="1">
<mxGeometry height="62" width="180" x="60" y="285" as="geometry" />
</mxCell>
<mxCell id="query_in" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#E0FBFC;strokeColor=#3D8C95;fontColor=#1B4965;" value="Query Input&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;query_hash&lt;/font&gt;" vertex="1">
<mxGeometry height="62" width="180" x="60" y="445" as="geometry" />
</mxCell>
<mxCell id="write_noise" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#F7E7C6;strokeColor=#B08968;fontColor=#5C4033;" value="Write Noise Injector&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;cam_write_noise&lt;br&gt;Bernoulli flip mask&lt;/font&gt;" vertex="1">
<mxGeometry height="76" width="190" x="310" y="278" as="geometry" />
</mxCell>
<mxCell id="storage" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#B7E4C7;strokeColor=#2D6A4F;fontColor=#1B4332;fontStyle=1;" value="Banked CAM Storage&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;cam_core_banked&lt;br&gt;8 BRAM banks × 512 depth&lt;/font&gt;" vertex="1">
<mxGeometry height="76" width="210" x="570" y="278" as="geometry" />
</mxCell>
<mxCell id="read_noise" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#D8F3DC;strokeColor=#40916C;fontColor=#1B4332;" value="Read Path / Noise Stage&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;cam_read_noise&lt;br&gt;registered lane response&lt;/font&gt;" vertex="1">
<mxGeometry height="76" width="200" x="850" y="278" as="geometry" />
</mxCell>
<mxCell id="match_engine" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#A9D6E5;strokeColor=#2A6F97;fontColor=#123047;fontStyle=1;" value="Match Engine Pipeline&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;batch scan control&lt;br&gt;rd_req base rows&lt;/font&gt;" vertex="1">
<mxGeometry height="82" width="190" x="310" y="435" as="geometry" />
</mxCell>
<mxCell id="popcount" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#CAE9FF;strokeColor=#468FAF;fontColor=#123047;" value="8× XNOR + Popcount&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;512-bit similarity score&lt;/font&gt;" vertex="1">
<mxGeometry height="70" width="210" x="570" y="441" as="geometry" />
</mxCell>
<mxCell id="fifo_topk" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#D9ED92;strokeColor=#76A21E;fontColor=#344E1F;" value="Candidate FIFO + Top-K Tracker&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;rank candidates, keep best K&lt;/font&gt;" vertex="1">
<mxGeometry height="70" width="220" x="840" y="441" as="geometry" />
</mxCell>
<mxCell id="serializer" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#E9F5DB;strokeColor=#74A57F;fontColor=#1B4332;" value="Result Serializer&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;rank / row / score / last&lt;/font&gt;" vertex="1">
<mxGeometry height="62" width="220" x="840" y="555" as="geometry" />
</mxCell>
<mxCell id="result_out" parent="1" style="rounded=1;whiteSpace=wrap;html=1;arcSize=14;strokeWidth=2;fillColor=#FFFFFF;strokeColor=#52796F;fontColor=#1B4332;fontStyle=1;" value="Top-K Results" vertex="1">
<mxGeometry height="62" width="130" x="1100" y="555" as="geometry" />
</mxCell>
<mxCell id="note_write" parent="1" style="shape=note;whiteSpace=wrap;html=1;backgroundOutline=1;darkOpacity=0.05;size=18;strokeWidth=1;fillColor=#FFF7E6;strokeColor=#B08968;fontColor=#5C4033;fontSize=12;" value="Write-time noise is baked into stored rows.&lt;br&gt;Non-zero rate runs full mask generation per write." vertex="1">
<mxGeometry height="74" width="250" x="280" y="549" as="geometry" />
</mxCell>
<mxCell id="note_query" parent="1" style="shape=note;whiteSpace=wrap;html=1;backgroundOutline=1;darkOpacity=0.05;size=18;strokeWidth=1;fillColor=#EFFAF6;strokeColor=#40916C;fontColor=#1B4332;fontSize=12;" value="Query scans banked rows in 8-lane batches,&lt;br&gt;then serializes the Top-K stream." vertex="1">
<mxGeometry height="62" width="250" x="550" y="555" as="geometry" />
</mxCell>
<mxCell id="e_write_noise" edge="1" parent="1" source="host_write" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#74A57F;fontColor=#52796F;" target="write_noise" value="write hash">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="e_noise_storage" edge="1" parent="1" source="write_noise" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#74A57F;fontColor=#52796F;" target="storage" value="noisy hash">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="e_query_match" edge="1" parent="1" source="query_in" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#2A6F97;fontColor=#2A6F97;" target="match_engine" value="query">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="e_match_storage_req" edge="1" parent="1" source="match_engine" style="endArrow=block;html=1;rounded=1;strokeWidth=2;strokeColor=#2A6F97;fontColor=#2A6F97;dashed=1;edgeStyle=orthogonalEdgeStyle;curved=0;" target="storage" value="rd_req">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="405" y="380" />
<mxPoint x="675" y="380" />
</Array>
</mxGeometry>
</mxCell>
<mxCell id="e_storage_read" edge="1" parent="1" source="storage" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#40916C;fontColor=#40916C;" target="read_noise" value="8 lane rows">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="e_read_pop" edge="1" parent="1" source="read_noise" style="endArrow=block;html=1;rounded=1;strokeWidth=2;strokeColor=#40916C;fontColor=#40916C;edgeStyle=orthogonalEdgeStyle;curved=0;" target="popcount" value="hashes + row ids">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="950" y="400" />
<mxPoint x="675" y="400" />
</Array>
</mxGeometry>
</mxCell>
<mxCell id="e_match_pop" edge="1" parent="1" source="match_engine" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#2A6F97;fontColor=#2A6F97;" target="popcount" value="query hash">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="e_pop_fifo" edge="1" parent="1" source="popcount" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#468FAF;fontColor=#2A6F97;" target="fifo_topk" value="scores">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="e_fifo_serializer" edge="1" parent="1" source="fifo_topk" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#76A21E;fontColor=#344E1F;" target="serializer" value="top K">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="e_serializer_out" edge="1" parent="1" source="serializer" style="endArrow=block;html=1;rounded=0;strokeWidth=2;strokeColor=#52796F;" target="result_out" value="">
<mxGeometry relative="1" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>

392
docs/experiments.md Normal file
View File

@@ -0,0 +1,392 @@
# Mini-Nav 实验可行性调研报告
本文档调研当前 Mini-Nav 项目对论文/报告第 6.26.8 节实验的支持程度,判断哪些实验可以直接开展,哪些实验只能部分开展,以及还需要补齐哪些代码、脚本或评估流程。
> 调研范围:当前仓库中的 Python 代码、CAM RTL、Cocotb 仿真、benchmark 框架、Scene Graph 原型、Habitat simulator 辅助代码与现有配置。
## 总体结论
| 实验章节 | 当前状态 | 结论 |
|---|---:|---|
| 6.2 CAM 硬件功能正确性验证 | 较完整 | 可以做 |
| 6.3 CAM 硬件性能基准测试 | 有参数化仿真基础 | 部分可做,性能/资源脚本缺失 |
| 6.4 Hash Compressor 检索性能实验 | 有模型和训练代码 | 部分可做,缺系统化评估脚本 |
| 6.5 对象级 Top-K 检索实验 | 有雏形 | 部分可做,但集成问题较多 |
| 6.6 Scene Graph 构建与路径推理实验 | 很弱 | 大多还不能做 |
| 6.7 CAM 加速后的整体框架实验 | 缺软件-硬件桥接与端到端闭环 | 只能做局部模拟 |
| 6.8 消融实验与误差分析 | 依赖前面模块 | 只能做少量离线消融 |
简短判断:**6.2 最成熟6.3、6.4 可较快补齐6.5 有雏形但需要修集成6.6、6.7、6.8 中涉及 Scene Graph 和端到端导航的部分目前还不具备完整实验条件。**
## 项目中已有的关键能力
### CAM 硬件与仿真
相关路径:
- `hw/rtl/cam_top.sv`
- `hw/rtl/cam_noisy.sv`
- `hw/rtl/core/cam_core_banked.sv`
- `hw/rtl/core/match_engine_pipeline.sv`
- `hw/rtl/core/popcount_pipeline.sv`
- `hw/rtl/noise/cam_write_noise.sv`
- `hw/rtl/noise/cam_read_noise.sv`
- `hw/rtl/noise/noise_mask_bernoulli.sv`Phase 2 后 read noise 改为 pass-through该模块仅用于 write noise mask 生成)
- `hw/rtl/cam_params.svh`
- `hw/sim/model/ref_model.py`
- `hw/sim/sweep_noise.py`
- `hw/sim/tests/*.py`
- `hw/sim/Makefile`
- `.justfile`
已有能力:
- CAM 顶层模块支持写入、查询、匹配输出。
- 参数支持 `NUM_ROWS``HASH_BITS``LANES`
- 有写噪声和读噪声模块。
- 有 Python 参考模型用于验证噪声和匹配行为。
- 有 Cocotb/Verilator 测试命令:
- `just cam-test-py`
- `just cam-test-all`
- `just cam-test-module MODULE=...`
### Hash Compressor 与检索
相关路径:
- `mini-nav/compressors/hash_compressor.py`
- `mini-nav/compressors/common.py`
- `mini-nav/compressors/train.py`
- `mini-nav/compressors/pipeline.py`
- `mini-nav/commands/benchmark.py`
- `mini-nav/benchmarks/tasks/retrieval.py`
已有能力:
- `HashCompressor` 支持 DINO token 到二值 hash bits 的映射。
- 默认目标是 512-bit CAM-compatible hash code。
- `HashLoss` 包含 contrastive、distillation、quantization 三类损失。
- `train.py` 可以在 CIFAR-10 上训练 HashCompressor。
- `common.py` 有 Hamming distance / similarity 工具。
### 对象级检索雏形
相关路径:
- `mini-nav/benchmarks/tasks/multi_object_retrieval.py`
- `mini-nav/data_loading/insdet_scenes.py`
- `mini-nav/data_loading/loader.py`
- `mini-nav/compressors/pipeline.py`
已有能力:
- 有多对象检索 benchmark task 雏形。
- 有 object-level vector 建库、query、scene-level aggregation 的基本逻辑。
- `HashPipeline` 支持 OWLv2 文本检测、SAM 分割、DINO 特征、Hash 压缩的流水线设计。
主要问题:
- `multi_object_retrieval.py` 引入 `utils.sam`,但源码中没有 `mini-nav/utils/sam.py`,只有历史 `__pycache__`
- 普通 benchmark 使用字段 `img`,多对象检索使用字段 `image`,存在数据接口不一致。
- 当前多对象检索主要走 LanceDB 连续向量检索,不是真正的 CAM/hash 检索。
### Scene Graph 与 Habitat 辅助代码
相关路径:
- `mini-nav/scenegraph/objectnode.py`
- `mini-nav/scenegraph/roomnode.py`
- `mini-nav/scenegraph/scenegraph.py`
- `mini-nav/simulator/habitat.py`
- `mini-nav/simulator/views.py`
- `mini-nav/simulator/topdown.py`
已有能力:
- `ObjectNode` 数据结构中已有 `visual_hash``semantic_hash` 字段。
- `SimpleSceneGraph` 目前仅保存 `rooms``objects` 两个字典。
- Habitat 侧有 simulator 创建、随机 navigable point 采图、topdown map 可视化辅助函数。
主要问题:
- 没有 Scene Graph 构建器。
- 没有 room-waypoint-object 拓扑关系。
- 没有检索结果到导航目标点的转换。
- 没有 episode、policy、SPL/success 等导航评估指标。
## 6.2 CAM 硬件功能正确性验证
结论:**可以做。**
| 小节 | 状态 | 说明 |
|---|---:|---|
| 6.2.1 写入与读取功能验证 | 可以做 | `cam_core_banked``cam_top` 有 Cocotb 测试 |
| 6.2.2 汉明距离计算验证 | 可以做 | `match_engine_pipeline` 和 ref model 覆盖 bit match / popcount |
| 6.2.3 Top-K 输出正确性验证 | 部分可做 | 当前硬件更接近 Top-1 输出,不是完整 Top-K 列表 |
| 6.2.4 噪声注入功能验证 | 可以做 | 写噪声、读噪声、组合噪声都有测试和参考模型 |
### 已具备内容
- 写入/读取:`hw/rtl/core/cam_core_banked.sv``hw/sim/tests/test_cam_core_banked.py`
- 匹配与 popcount`hw/rtl/core/match_engine_pipeline.sv``hw/rtl/core/popcount_pipeline.sv`
- 噪声模块:`hw/rtl/noise/cam_write_noise.sv``hw/rtl/noise/cam_read_noise.sv`Phase 2 后 read noise 改为 pass-through实际翻转仅由 write noise 模块产生)
- 参考模型:`hw/sim/model/ref_model.py`
- 集成测试:`hw/sim/tests/test_cam_basic.py`
### 还需补齐
1. 如果论文中写的是严格的 **Top-K 硬件输出**,当前 RTL 需要补 Top-K 选择/排序模块,或者明确实验是 Top-1 / Top-K 由软件侧重排实现。
2. 需要统一整理仿真日志、表格和截图,形成可放入论文的实验结果。
3. 建议补一个 `scripts/run_cam_correctness.py` 或类似脚本,一键跑 6.2 所需配置。
## 6.3 CAM 硬件性能基准测试
结论:**部分可做。**
| 小节 | 状态 | 说明 |
|---|---:|---|
| 6.3.1 不同行数 `NUM_ROWS` 下的延迟 | 部分可做 | Makefile 支持参数化,但缺自动 latency 统计 |
| 6.3.2 不同 `HASH_BITS` 下的资源占用 | 暂不能充分做 | 缺 Yosys/Vivado/Quartus 综合脚本 |
| 6.3.3 不同 `LANES` 下的吞吐率 | 部分可做 | RTL 参数支持,但缺吞吐率测量脚本 |
| 6.3.4 与软件检索方法的性能对比 | 还不能完整做 | 缺统一的硬件/软件 benchmark 脚本 |
### 已具备内容
- `hw/sim/Makefile` 支持:
- `NUM_ROWS`
- `HASH_BITS`
- `LANES`
- `WRITE_NOISE_*`
- 读取路径 pass-through 验证(读阶段不再注入噪声)
- `.justfile` 已封装远程 Verilator/Cocotb 执行命令。
### 还需补齐
1. CAM 性能 sweep 脚本:
- 输入:`NUM_ROWS``HASH_BITS``LANES`
- 输出latency cycles、throughput、仿真耗时
2. Cocotb latency 统计:
- 记录 `query_valid``result_valid` 的周期数。
3. 综合/资源统计流程:
- Yosys 或 Vivado TCL。
- 输出 LUT、FF、BRAM、频率估计。
4. 软件 baseline
- Python/NumPy Hamming scan。
- PyTorch vectorized Hamming。
- LanceDB/DINO continuous vector retrieval。
## 6.4 Hash Compressor 检索性能实验
结论:**部分可做。**
| 小节 | 状态 | 说明 |
|---|---:|---|
| 6.4.1 原始 DINOv2 特征与二值哈希特征对比 | 可补后做 | DINO 和 Hash 都有,但缺统一评估脚本 |
| 6.4.2 不同哈希长度消融实验 | 可补后做 | `HashCompressor(hash_bits=...)` 支持,但训练脚本固定 512 |
| 6.4.3 不同压缩方法对比实验 | 暂不能做 | 目前只有一种 MLP HashCompressor无 PCA/随机投影/ITQ/LSH baseline |
| 6.4.4 相似度保持能力分析 | 可补后做 | 有 DINO embedding 和 hash similarity缺相关性分析脚本 |
### 已具备内容
- `HashCompressor.encode()` 可以生成 binary bits。
- `HashCompressor.compute_similarity()` 可以计算 Hamming similarity。
- `HashLoss` 支持相似度蒸馏,有利于做 similarity preservation 分析。
- `train.py` 可训练基础 512-bit compressor。
### 还需补齐
建议新增 `scripts/eval_hash_retrieval.py`,支持:
- DINO baseline vs Hash baseline。
- `hash_bits = 64 / 128 / 256 / 512`
- Recall@K、mAP、query latency。
- DINO cosine similarity 与 Hash Hamming similarity 的 Spearman/Pearson 相关性。
- storage compression ratio。
如果要做 6.4.3,需要新增压缩 baseline
- random projection + sign。
- PCA + sign。
- LSH。
- ITQ 或简化二值量化方法。
## 6.5 对象级 Top-K 检索实验
结论:**部分可做,但当前不能顺畅跑完整实验。**
| 小节 | 状态 | 说明 |
|---|---:|---|
| 6.5.1 图像查询实验 | 部分可做 | 有 object retrieval 框架,但需修 `utils.sam` 和字段兼容 |
| 6.5.2 文本类别查询实验 | 部分可做 | `HashPipeline` 支持 OWLv2 text labels但 benchmark 未集成 |
| 6.5.3 多视角对象特征融合实验 | 暂不能做 | 没有 object tracking / multi-view merge / feature aggregation |
| 6.5.4 类别过滤与重排序实验 | 只有基础 | 有 mask filtering类别级过滤和重排序逻辑还需补 |
### 已具备内容
- `multi_object_retrieval.py` 有对象级检索任务结构。
- `data_loading/loader.py` 能加载带 bbox/category 的 InsDet scene 数据。
- `HashPipeline` 支持文本标签驱动的 OWLv2 检测。
### 主要问题
1. `utils.sam` 缺失。
2. `img` / `image` 字段不一致。
3. 数据库 schema 与 object-level retrieval 的 schema 需要和 runner 对齐。
4. 当前 object retrieval 不是 hash/CAM 检索。
5. 缺类别过滤、重排序、多视角融合的正式评估接口。
### 还需补齐
1.`mini-nav/utils/sam.py`,或改为复用 `compressors/model_loader.py``compressors/proposal`
2. 统一 benchmark dataset item 字段。
3. 给 object retrieval 增加三种模式:
- DINO continuous vector。
- Hash/Hamming。
- CAM/ref_model Hamming。
4. 增加 category filter 与 rerank 开关。
5. 多视角融合需要新增 object association 和 feature aggregation。
## 6.6 Scene Graph 构建与路径推理实验
结论:**大部分还不能做。**
| 小节 | 状态 | 说明 |
|---|---:|---|
| 6.6.1 ObjectNode 构建质量验证 | 只能做静态/伪数据 | 有数据结构,无自动构建流程 |
| 6.6.2 房间层与 waypoint 层关系验证 | 不能做 | 没有 waypoint node / room-waypoint graph |
| 6.6.3 检索结果到导航目标点转换实验 | 不能做 | 没有 object → navigable point / goal pose 映射 |
| 6.6.4 基于 Scene Graph 的路径生成实验 | 不能做 | 没有 graph search / Habitat path planner 封装 |
### 已具备内容
- `ObjectNode` 能存储 object id、room id、position、visual hash、semantic hash。
- `RoomNode``SimpleSceneGraph` 是极简数据结构。
- Habitat simulator helper 可以采集场景图像和 topdown map。
### 还需补齐
1. SceneGraph builder
- 输入多视角图像、检测结果、相机位姿。
- 输出 ObjectNode、RoomNode、WaypointNode。
2. 多视角 object association
- 判断不同视角中同一物体。
- 合并位置和 hash 特征。
3. Waypoint graph
- 房间节点、waypoint 节点、object 节点关系。
4. Path planning
- 检索结果转 target pose。
- target pose 转 Habitat shortest path 或自定义 graph path。
## 6.7 CAM 加速后的整体框架实验
结论:**端到端还不能做;局部对比可以做。**
| 小节 | 状态 | 说明 |
|---|---:|---|
| 6.7.1 无 CAM 与加入 CAM 的检索延迟对比 | 可补脚本后做局部实验 | 可用 Python CAM ref model 模拟,但无硬件级整体接入 |
| 6.7.2 无 Scene Graph 与加入 Scene Graph 的目标选择对比 | 不能做 | Scene Graph 目标选择逻辑不足 |
| 6.7.3 Top-K 候选切换与失败恢复实验 | 不能做 | 没有导航策略/失败恢复机制 |
| 6.7.4 端到端离线闭环实验 | 不能做 | 缺 query → retrieve → goal → plan → evaluate 闭环 |
### 可以先做的局部实验
- DINO/LanceDB 检索延迟 vs Hash/Hamming 检索延迟。
- Python CAM ref model 检索延迟 vs PyTorch Hamming scan。
- 噪声 CAM 对 Top-1 / Top-K 稳定性的影响。
### 还需补齐
1. Hash output 到 CAM/ref_model 的桥接脚本。
2. CAM 检索 API给定 query hash 和 database hash返回 Top-K candidate。
3. 整体 pipeline
- image/text query
- object retrieval
- scene graph rerank
- target waypoint selection
- path planning
- offline evaluation
## 6.8 消融实验与误差分析
结论:**只能做部分离线消融。**
| 小节 | 状态 | 说明 |
|---|---:|---|
| 6.8.1 去除类别过滤 | 可补后做 | 类别过滤还没有完整 pipeline |
| 6.8.2 去除多视角融合 | 不能做 | 多视角融合本身未实现 |
| 6.8.3 去除 Scene Graph 重排序 | 不能做 | Scene Graph rerank 未实现 |
| 6.8.4 CAM 噪声对最终结果的影响 | 可做局部 | 可测对检索结果影响,不能测完整导航最终结果 |
| 6.8.5 失败案例分析 | 可做检索失败分析 | 导航失败案例暂不可做 |
### 可先做的误差分析
- Hash 检索失败案例query 与 top candidates 可视化。
- CAM 噪声导致 Top-1 改变的案例。
- DINO 与 Hash 排序不一致的案例。
- SAM/OWLv2 分割或检测失败案例。
### 暂不能做的误差分析
- 多视角融合失败原因。
- Scene Graph 重排序失败原因。
- 端到端导航失败恢复分析。
## 推荐实施顺序
### 阶段 1快速产出可用实验结果
优先完成:
1. 6.2 CAM correctness。
2. 6.3 CAM 仿真性能 sweep。
3. 6.4 DINO vs Hash 检索性能对比。
建议新增脚本:
- `scripts/run_cam_correctness.py`
- `scripts/sweep_cam_perf.py`
- `scripts/eval_hash_retrieval.py`
### 阶段 2解锁对象级实验
优先修复:
1. `mini-nav/utils/sam.py` 缺失问题。
2. benchmark dataset 字段不一致问题。
3. object-level schema 与 runner 对齐问题。
4. object retrieval 的 hash/CAM 模式。
然后开展:
- 6.5.1 图像查询实验。
- 6.5.2 文本类别查询实验。
- 6.5.4 类别过滤与重排序实验。
### 阶段 3实现 Scene Graph 与端到端闭环
需要新增核心模块:
1. SceneGraph builder。
2. Waypoint graph。
3. Object-to-goal mapping。
4. Navigation evaluator。
5. Candidate switching / failure recovery。
完成后才能正式开展:
- 6.6 Scene Graph 构建与路径推理。
- 6.7 CAM 加速后的整体框架实验。
- 6.8 中涉及 Scene Graph、导航闭环、多视角融合的消融实验。
## 最小可行实验组合
如果目标是尽快形成一组完整、可信的实验章节,建议先采用以下组合:
1. **CAM 功能正确性**:覆盖 6.2。
2. **CAM 参数化仿真性能**:覆盖 6.3 的一部分。
3. **DINO vs Hash 检索性能**:覆盖 6.4.1。
4. **Hash length ablation**:覆盖 6.4.2。
5. **CAM noise 对检索稳定性的影响**:覆盖 6.2.4、6.8.4 的离线部分。
6. **对象级检索 demo**:修复 `utils.sam` 后覆盖 6.5.1 的基础版本。
这组实验避开了目前尚未实现的 Scene Graph 端到端导航闭环,同时能充分利用当前仓库中已经完成度较高的 CAM 和 HashCompressor 模块。

View File

@@ -0,0 +1,95 @@
# CAM Retrieval Benchmark — Noise Sweep Summary
**Generated:** 2026-05-27 19:00:42
## Configuration
| Parameter | Value |
|---|---|
| Dataset path | `outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz` |
| NUM_ROWS | 512 |
| TOPK_K | 5 |
| HASH_BITS | 512 |
| Noise rates | 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100% |
| Total runs | 11 |
| Passed | 11 |
| Failed | 0 |
### Dataset Details
| Field | Value |
|---|---|
| num_queries | 128 |
| num_classes | 10 |
| seed | 0 |
---
## Results by Noise Rate
### k=1
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
| 0% | 0 | — | 1.000000 | 1.000000 | 1.000000 | 0.019531 | 0.038314 | 1.000000 | ✓ |
| 10% | 1 | 10/100 | 1.000000 | 1.000000 | 1.000000 | 0.019531 | 0.038314 | 0.507812 | ✓ |
| 20% | 1 | 20/100 | 1.000000 | 1.000000 | 1.000000 | 0.019531 | 0.038314 | 0.234375 | ✓ |
| 30% | 1 | 30/100 | 0.992188 | 0.992188 | 0.992188 | 0.019378 | 0.038014 | 0.164062 | ✓ |
| 40% | 1 | 40/100 | 0.984375 | 0.984375 | 0.984375 | 0.019228 | 0.037719 | 0.093750 | ✓ |
| 50% | 1 | 50/100 | 0.257812 | 0.257812 | 0.257812 | 0.005043 | 0.009893 | 0.023438 | ✓ |
| 60% | 1 | 60/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 70% | 1 | 70/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
### k=5
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
| 0% | 0 | — | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 0.177936 | 1.000000 | ✓ |
| 10% | 1 | 10/100 | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 0.177936 | 0.000000 | ✓ |
| 20% | 1 | 20/100 | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 0.177936 | 0.000000 | ✓ |
| 30% | 1 | 30/100 | 1.000000 | 0.995313 | 0.997651 | 0.097197 | 0.177099 | 0.000000 | ✓ |
| 40% | 1 | 40/100 | 1.000000 | 0.939062 | 0.968574 | 0.091729 | 0.167132 | 0.000000 | ✓ |
| 50% | 1 | 50/100 | 0.750000 | 0.234375 | 0.357143 | 0.022913 | 0.041745 | 0.000000 | ✓ |
| 60% | 1 | 60/100 | 0.015625 | 0.003125 | 0.005208 | 0.000306 | 0.000558 | 0.000000 | ✓ |
| 70% | 1 | 70/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
---
## Cross-Noise Comparison (primary: Hit@K)
| Noise (%) | Hit@1 | Hit@5 | Δ(Hit@1 vs 0%) | Δ(Hit@5 vs 0%) |
|---|---:|---:|---:|---:|
| 0% | 1.000000 | 1.000000 | +0.000000 | +0.000000 |
| 10% | 1.000000 | 1.000000 | +0.000000 | +0.000000 |
| 20% | 1.000000 | 1.000000 | +0.000000 | +0.000000 |
| 30% | 0.992188 | 1.000000 | -0.007812 | +0.000000 |
| 40% | 0.984375 | 1.000000 | -0.015625 | +0.000000 |
| 50% | 0.257812 | 0.750000 | -0.742188 | -0.250000 |
| 60% | 0.000000 | 0.015625 | -1.000000 | -0.984375 |
| 70% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
| 80% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
| 90% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
| 100% | 0.000000 | 0.000000 | -1.000000 | -1.000000 |
---
## Metric Definitions
- **Hit@K**: fraction of queries where at least one relevant item appears in Top-K results (primary metric).
- **Precision@K**: mean per-query precision — averaged `tp/k` across all queries.
- **Hit-F1@K**: `2 × Hit@K × Precision@K / (Hit@K + Precision@K)` — F1 using hit-rate recall.
- **Std-Recall@K**: mean per-query standard retrieval recall — `tp / |relevant|` averaged across queries (supplementary).
- **Std-F1@K**: `2 × Precision@K × Std-Recall@K / (Precision@K + Std-Recall@K)` — F1 using standard recall (supplementary).
- **Golden Match@K**: fraction of queries where DUT Top-K exactly matches the reference golden Top-K.
The paper uses Hit@K and Precision@K as primary metrics. Std-Recall@K and Std-F1@K are supplementary,
included to show Top-K coverage against all relevant items in the database.
*Results from Verilator/Cocotb simulation. Not measured on physical FPGA hardware.*

View File

@@ -0,0 +1,95 @@
# CAM Retrieval Benchmark — Noise Sweep Summary
**Generated:** 2026-05-27 19:00:46
## Configuration
| Parameter | Value |
|---|---|
| Dataset path | `outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz` |
| NUM_ROWS | 512 |
| TOPK_K | 5 |
| HASH_BITS | 512 |
| Noise rates | 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100% |
| Total runs | 11 |
| Passed | 11 |
| Failed | 0 |
### Dataset Details
| Field | Value |
|---|---|
| num_queries | 128 |
| num_classes | 100 |
| seed | 0 |
---
## Results by Noise Rate
### k=1
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
| 0% | 0 | — | 0.695312 | 0.695312 | 0.695312 | 0.134635 | 0.225589 | 1.000000 | ✓ |
| 10% | 1 | 10/100 | 0.585938 | 0.585938 | 0.585938 | 0.113281 | 0.189857 | 0.593750 | ✓ |
| 20% | 1 | 20/100 | 0.562500 | 0.562500 | 0.562500 | 0.109115 | 0.182774 | 0.460938 | ✓ |
| 30% | 1 | 30/100 | 0.460938 | 0.460938 | 0.460938 | 0.088802 | 0.148915 | 0.304688 | ✓ |
| 40% | 1 | 40/100 | 0.234375 | 0.234375 | 0.234375 | 0.044792 | 0.075210 | 0.101562 | ✓ |
| 50% | 1 | 50/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 60% | 1 | 60/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 70% | 1 | 70/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
### k=5
| Noise (%) | WRITE_NOISE_EN | NUM/DEN | Hit@K | Precision@K | Hit-F1@K | Std-Recall@K | Std-F1@K | Golden Match@K | Status |
|---|---:|---|---:|---|---:|---|---:|---|---:|---|
| 0% | 0 | — | 0.867188 | 0.462500 | 0.603261 | 0.445052 | 0.453608 | 1.000000 | ✓ |
| 10% | 1 | 10/100 | 0.812500 | 0.421875 | 0.555380 | 0.405990 | 0.413780 | 0.023438 | ✓ |
| 20% | 1 | 20/100 | 0.742188 | 0.364062 | 0.488502 | 0.350000 | 0.356893 | 0.000000 | ✓ |
| 30% | 1 | 30/100 | 0.640625 | 0.248437 | 0.358029 | 0.238802 | 0.243525 | 0.000000 | ✓ |
| 40% | 1 | 40/100 | 0.460938 | 0.117187 | 0.186867 | 0.113021 | 0.115066 | 0.000000 | ✓ |
| 50% | 1 | 50/100 | 0.062500 | 0.015625 | 0.025000 | 0.014844 | 0.015224 | 0.000000 | ✓ |
| 60% | 1 | 60/100 | 0.007812 | 0.001563 | 0.002604 | 0.001563 | 0.001563 | 0.000000 | ✓ |
| 70% | 1 | 70/100 | 0.007812 | 0.001563 | 0.002604 | 0.001563 | 0.001563 | 0.000000 | ✓ |
| 80% | 1 | 80/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 90% | 1 | 90/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
| 100% | 1 | 100/100 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | ✓ |
---
## Cross-Noise Comparison (primary: Hit@K)
| Noise (%) | Hit@1 | Hit@5 | Δ(Hit@1 vs 0%) | Δ(Hit@5 vs 0%) |
|---|---:|---:|---:|---:|
| 0% | 0.695312 | 0.867188 | +0.000000 | +0.000000 |
| 10% | 0.585938 | 0.812500 | -0.109375 | -0.054688 |
| 20% | 0.562500 | 0.742188 | -0.132812 | -0.125000 |
| 30% | 0.460938 | 0.640625 | -0.234375 | -0.226562 |
| 40% | 0.234375 | 0.460938 | -0.460938 | -0.406250 |
| 50% | 0.000000 | 0.062500 | -0.695312 | -0.804688 |
| 60% | 0.000000 | 0.007812 | -0.695312 | -0.859375 |
| 70% | 0.000000 | 0.007812 | -0.695312 | -0.859375 |
| 80% | 0.000000 | 0.000000 | -0.695312 | -0.867188 |
| 90% | 0.000000 | 0.000000 | -0.695312 | -0.867188 |
| 100% | 0.000000 | 0.000000 | -0.695312 | -0.867188 |
---
## Metric Definitions
- **Hit@K**: fraction of queries where at least one relevant item appears in Top-K results (primary metric).
- **Precision@K**: mean per-query precision — averaged `tp/k` across all queries.
- **Hit-F1@K**: `2 × Hit@K × Precision@K / (Hit@K + Precision@K)` — F1 using hit-rate recall.
- **Std-Recall@K**: mean per-query standard retrieval recall — `tp / |relevant|` averaged across queries (supplementary).
- **Std-F1@K**: `2 × Precision@K × Std-Recall@K / (Precision@K + Std-Recall@K)` — F1 using standard recall (supplementary).
- **Golden Match@K**: fraction of queries where DUT Top-K exactly matches the reference golden Top-K.
The paper uses Hit@K and Precision@K as primary metrics. Std-Recall@K and Std-F1@K are supplementary,
included to show Top-K coverage against all relevant items in the database.
*Results from Verilator/Cocotb simulation. Not measured on physical FPGA hardware.*

View File

@@ -0,0 +1,217 @@
# 软件/硬件 CAM 检索基准实验总结
**日期**2026-05-27
**工作区**`/home/sikongjueluo/Projects/Mini-Nav`
**目标**:对比同一组 CAM 检索数据在硬件仿真与软件汉明距离检索下的检索质量,并记录软件检索速度基线与硬件仿真周期性能。
## 1. 实验配置
### 数据集与哈希配置
| 数据集 | 行数 | 查询数 | 类别数 | 哈希宽度 | Top-K |
|---|---:|---:|---:|---:|---:|
| CIFAR-10 | 512 | 128 | 10 | 512 bit | 5 |
| CIFAR-100 | 512 | 128 | 100 | 512 bit | 5 |
数据文件来自:
- `outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz`
- `outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz`
`.npz` 内部字段为:
- `rows_words`
- `row_labels`
- `queries_words`
- `query_labels`
软件基准直接复用硬件基准数据格式,将 little-endian `uint64` words 转为 Python `int` 后执行汉明距离 / XNOR-popcount Top-K 检索,避免通过软件 CAM 时序模拟带来额外开销。
### 运行命令
软件检索速度基准:
```bash
just remote "python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz --hash-bits 512 --topk-k 5 --run-id sw_cifar10_hash512_rows512_queries128 && python scripts/sw_retrieval_benchmark.py --dataset outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz --hash-bits 512 --topk-k 5 --run-id sw_cifar100_hash512_rows512_queries128"
```
硬件噪声扫描数据来自远端已有 Cocotb/Verilator 输出与 `docs/exps/cam_retrieval_noise_sweep_*.md`
硬件检索周期检测命令:
```bash
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar10_hash512_rows512_queries128.npz 512
just cam-test-retrieval-artifact outputs/cam_retrieval_benchmark/datasets/cifar100_hash512_rows512_queries128.npz 512
```
## 2. 软件检索速度与质量
软件路径只计时 Top-K 匹配阶段,不包含 `.npz` 加载、指标聚合和结果写盘。
| 数据集 | Hit@1 | Precision@1 | Std-Recall@1 | Hit@5 | Precision@5 | Std-Recall@5 | Golden Match@5 | ns/query | queries/s |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| CIFAR-10 | 1.000000 | 1.000000 | 0.019531 | 1.000000 | 1.000000 | 0.097656 | 1.000000 | 205835.281 | 4858.254 |
| CIFAR-100 | 0.695312 | 0.695312 | 0.134635 | 0.867188 | 0.462500 | 0.445052 | 1.000000 | 205868.953 | 4857.459 |
观察:
- 两个数据集的软件吞吐都约为 **4.86k queries/s**
- CIFAR-10 的 Hit@1/Hit@5 均为 1.0,说明在该 512-row 子集上 Top-K 中总能命中同类样本。
- CIFAR-100 的类别更多、每类样本更少Hit@1 降至 0.695312Hit@5 为 0.867188。
- `Golden Match@K = 1.0` 表示软件实现与 `hw/sim/model/ref_model.py::match_topk` 的排序结果一致。
## 3. 无噪声硬件仿真与软件基线一致性
硬件无噪声结果来自 `WRITE_NOISE_EN=0` 的 Cocotb/Verilator 检索基准。该结果用于确认硬件 Top-K 输出与参考模型一致。
| 数据集 | 模式 | Hit@1 | Hit@5 | Precision@5 | Std-Recall@5 | Golden Match@1 | Golden Match@5 |
|---|---|---:|---:|---:|---:|---:|---:|
| CIFAR-10 | software-hamming | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 1.000000 | 1.000000 |
| CIFAR-10 | hardware no-noise | 1.000000 | 1.000000 | 1.000000 | 0.097656 | 1.000000 | 1.000000 |
| CIFAR-100 | software-hamming | 0.695312 | 0.867188 | 0.462500 | 0.445052 | 1.000000 | 1.000000 |
| CIFAR-100 | hardware no-noise | 0.695312 | 0.867188 | 0.462500 | 0.445052 | 1.000000 | 1.000000 |
结论:
- 无噪声硬件仿真与软件汉明距离基线在两个数据集上质量指标一致。
- 这说明 `.npz` 数据加载、little-endian word 转换、硬件 CAM 匹配和软件参考模型在当前配置下是对齐的。
## 4. 写噪声对硬件检索质量的影响
硬件噪声实验使用 `WRITE_NOISE_EN=1`,按 10% 步长扫描 `WRITE_NOISE_RATE_NUM / 100`。以下表格保留主要指标 Hit@1、Hit@5 与 Golden Match@K
### CIFAR-10 噪声扫描
| 写噪声率 | Hit@1 | Hit@5 | Golden Match@1 | Golden Match@5 |
|---:|---:|---:|---:|---:|
| 0% | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
| 10% | 1.000000 | 1.000000 | 0.507812 | 0.000000 |
| 20% | 1.000000 | 1.000000 | 0.234375 | 0.000000 |
| 30% | 0.992188 | 1.000000 | 0.164062 | 0.000000 |
| 40% | 0.984375 | 1.000000 | 0.093750 | 0.000000 |
| 50% | 0.257812 | 0.750000 | 0.023438 | 0.000000 |
| 60% | 0.000000 | 0.015625 | 0.000000 | 0.000000 |
| 70% | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 80% | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 90% | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 100% | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
观察:
- CIFAR-10 在 0%40% 写噪声下 Hit@5 仍保持 1.0。
- Golden Match@5 从 10% 噪声开始降为 0说明 Top-5 的精确排序对噪声非常敏感。
- 50% 噪声是明显拐点Hit@1 降至 0.257812Hit@5 降至 0.75。
- 60% 以后检索基本失效。
### CIFAR-100 噪声扫描
| 写噪声率 | Hit@1 | Hit@5 | Golden Match@1 | Golden Match@5 |
|---:|---:|---:|---:|---:|
| 0% | 0.695312 | 0.867188 | 1.000000 | 1.000000 |
| 10% | 0.585938 | 0.812500 | 0.593750 | 0.023438 |
| 20% | 0.562500 | 0.742188 | 0.460938 | 0.000000 |
| 30% | 0.460938 | 0.640625 | 0.304688 | 0.000000 |
| 40% | 0.234375 | 0.460938 | 0.101562 | 0.000000 |
| 50% | 0.000000 | 0.062500 | 0.000000 | 0.000000 |
| 60% | 0.000000 | 0.007812 | 0.000000 | 0.000000 |
| 70% | 0.000000 | 0.007812 | 0.000000 | 0.000000 |
| 80% | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 90% | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 100% | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
观察:
- CIFAR-100 对噪声更敏感10% 噪声时 Hit@1 从 0.695312 降至 0.585938Hit@5 从 0.867188 降至 0.812500。
- 40% 噪声时 Hit@5 已降至 0.460938。
- 50% 噪声后检索基本不可用。
- 与 CIFAR-10 相同Golden Match@5 在低噪声下也迅速接近 0说明精确排序比类别命中率更脆弱。
## 5. 硬件检索周期性能
本次已将硬件周期检测合并进真实 `.npz` 检索数据集的 Cocotb/Verilator
benchmark。实现位置
- `hw/sim/tests/top/utils.py::query_topk_once_with_latency`
- `hw/sim/benchmarks/retrieval/test_retrieval_benchmark.py::cam_retrieval_benchmark`
周期口径如下。本节只报告Cocotb/Verilator仿真周期不将cycle直接换算为ns或queries/s。
| 指标 | 含义 |
|---|---|
| `query_only_cycles_per_query` | 主指标。每个query从`query_valid && query_ready`握手成功的时钟沿,到对应`result_valid && result_ready && result_last`完成的平均周期数。 |
| `query_only_total_cycles` | 所有query-only事务周期之和不包含装载、写入、噪声注入和查询间统计代码。 |
| `query_only_queries_per_cycle` | `num_queries / query_only_total_cycles`。 |
| `load_write_noise_cycles` | 写入CAM行以及可选写噪声注入阶段的周期数。无噪声模式下仍包含CAM行写入周期。 |
| `end_to_end_cycles` | 从开始写入数据集到最后一个query完成的完整benchmark硬件仿真周期数。 |
| `end_to_end_queries_per_cycle` | `num_queries / end_to_end_cycles`。 |
| `accept_to_first_result_cycles` | query被接受后到首个结果beat完成握手的平均周期数。 |
| `accept_to_last_result_cycles` | query被接受后`result_last`结果beat完成握手的平均周期数。 |
| `cycles_per_query``total_query_cycles``queries_per_cycle` | 兼容旧字段当前均为query-only口径不代表end-to-end。 |
选择query-only的`accept→last`作为主cycles/query是因为Top-K检索只有在串行结果流输出到`result_last`并被接收后才算完整完成;仅用首个`result_valid`会低估实际Top-K查询事务成本。`load_write_noise_cycles``end_to_end_cycles`单独报告避免把非查询阶段混入query-only性能。
### 无噪声硬件周期结果
配置512 rows×128 queries×512-bit hash`TOPK_K=5``LANES=8`
`WRITE_NOISE_EN=0`
| 数据集 | 模式 | 查询数 | query-only cycles/query | query-only total cycles | query-only queries/cycle | load/write/noise cycles | end-to-end cycles | end-to-end cycles/query | end-to-end queries/cycle | accept→first | accept→last | 状态 |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| CIFAR-10 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
| CIFAR-100 | hardware no-noise | 128 | 1031.000000 | 131968 | 0.000969932 | 1024 | 133628 | 1043.968750 | 0.000957883 | 1027.000000 | 1031.000000 | pass |
对应日志标记:
```text
RETRIEVAL_PERF_RESULT mode=no_noise num_queries=128 query_only_cycles_per_query=1031.000000 query_only_total_cycles=131968 query_only_queries_per_cycle=0.000969932 load_write_noise_cycles=1024 end_to_end_cycles=133628 end_to_end_cycles_per_query=1043.968750 end_to_end_queries_per_cycle=0.000957883 cycles_per_query=1031.000000 accept_to_first_result_cycles=1027.000000 accept_to_last_result_cycles=1031.000000 total_query_cycles=131968 queries_per_cycle=0.000969932 status=pass
```
该结果说明:在当前 `NUM_ROWS=512, LANES=8, TOPK_K=5` 的硬件仿真配置下,
一次完整Top-K查询事务的query-only成本为**1031cycles/query**。首个结果beat约在1027cycles后出现完整Top-K输出额外消耗约4cycles。完整benchmark端到端口径为**1043.968750cycles/query**其中数据写入阶段为1024cycles。
> 注:以上数据来自 Verilator/Cocotb 仿真,不是 FPGA 板上实测。它可用于
> 架构级周期趋势分析,但不能直接等同于板级频率、吞吐或端到端系统延迟。
## 6. 指标解释
| 指标 | 含义 |
|---|---|
| Hit@K / `recall@k` | 每个 query 的 Top-K 中是否至少出现一个同类样本,然后对 query 取平均。 |
| Precision@K / `macro_precision` | 每个 query 的 Top-K 中同类样本比例,即 `tp / k`,再对 query 取平均。 |
| Std-Recall@K / `retrieval_recall` | 每个 query 检出的同类样本数占数据库中所有同类样本数的比例,即 `tp / |relevant|`,再取平均。 |
| Std-F1@K / `macro_f1` | 使用 Precision@K 与 Std-Recall@K 计算的 F1。 |
| Golden Match@K / `exact_match_rate` | Top-K row index 列表是否与参考模型完全一致。该指标比 Hit@K 更严格。 |
| `ns_per_query` | 软件 Top-K 匹配阶段平均耗时;不含加载和写盘。 |
| `queries_per_second` | 软件 Top-K 匹配阶段吞吐率。 |
| `query_only_cycles_per_query` | 硬件仿真中一次完整Top-K查询事务的平均周期数采用`query_valid && query_ready``result_last`完成握手口径。 |
| `query_only_queries_per_cycle` | 硬件仿真中完成query数除以query-only总事务周期数。 |
| `load_write_noise_cycles` | CAM行装载、写入及可选写噪声注入阶段周期数。 |
| `end_to_end_cycles` | 从开始写入数据到最后一个query完成的完整benchmark硬件仿真周期数。 |
| `end_to_end_queries_per_cycle` | 硬件仿真中完成query数除以end-to-end总周期数。 |
## 7. 当前结论
1. **软件汉明距离基线已可作为硬件 CAM 检索的功能参考。**
在无噪声条件下,硬件仿真和软件基线的质量指标及 Golden Match 均一致。
2. **当前软件基线速度约为 4.86k queries/s。**
该结果来自 Python integer brute-force Hamming scan数据规模为 512 rows × 128 queries × 512 bits。
3. **硬件检索质量基准现在显式拆分query-only与end-to-end周期。**
在CIFAR-10和CIFAR-100的512-row/128-query无噪声配置下完整Top-K查询事务均为**1031cycles/query**,首个结果为**1027cycles/query**;写入阶段为**1024cycles**,端到端为**1043.968750cycles/query**。
4. **写噪声对精确排序影响显著。**
即使 Hit@K 保持较高Golden Match@K 也会快速下降,说明噪声首先破坏精确排序,再进一步破坏类别命中。
5. **CIFAR-100 比 CIFAR-10 更能体现检索难度。**
在无噪声下 CIFAR-100 的 Hit@1/Hit@5 分别为 0.695312/0.867188,明显低于 CIFAR-10 的 1.0/1.0,更适合作为后续检索质量对比主数据集。
## 8. 后续建议
1.`docs/exps` 中继续维护:
- 软件检索速度表;
- 硬件无噪声一致性表;
- 硬件噪声鲁棒性表;
- 硬件query-only、load/write/noise、end-to-end周期表。
2. 对软件基线补充 NumPy/PyTorch vectorized Hamming scan以区分“朴素 Python baseline”和“优化软件 baseline”。
3. 增加 `NUM_ROWS` sweep例如 512、1024、2048、4096 rows观察软件 brute-force scan 的线性增长趋势。

View File

@@ -1,6 +1,5 @@
name: mini-nav
channels:
- pytorch
- nvidia
- aihabitat-nightly
- conda-forge
@@ -8,11 +7,15 @@ dependencies:
- python=3.10
- pip
- habitat-sim=0.3.3
- habitat-baselines=0.3.320250127
- habitat-lab=0.3.320250127
- withbullet
- pytorch
- torchvision
- pytorch-cuda=12.1
# Lib
- zlib
# Toolsets
- uv
- yosys
- verilator
- gtkwave
- just
- nodejs
- nodejs>=20.0.0

3
hw/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
**/sim_build/
**/results.xml
*.fst

82
hw/rtl/cam_noisy.sv Normal file
View File

@@ -0,0 +1,82 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
module cam_noisy #(
parameter bit WRITE_NOISE_EN = 1'b1,
parameter int WRITE_NOISE_RATE_NUM = 1,
parameter int WRITE_NOISE_RATE_DEN = 100,
parameter logic [63:0] WRITE_NOISE_SEED = 64'hB504_F32D_B504_F32D
) (
input logic clk,
input logic rst_n,
input logic wr_valid,
output logic wr_ready,
input logic [(`ROW_BITS)-1:0] wr_addr,
input logic [(`HASH_BITS)-1:0] write_hash,
input logic rd_valid_i,
input logic [(`ROW_BITS)-1:0] rd_base_row_i,
output logic rd_valid_o,
output logic [(`LANES)*(`ROW_BITS)-1:0] rd_row_ids_o,
output logic [(`LANES)*(`HASH_BITS)-1:0] rd_hashes_o,
output logic [(`LANES)-1:0] rd_lane_valid_o
);
// ── Intermediate wires between pipeline stages ──
logic core_wr_valid;
logic [(`ROW_BITS)-1:0] core_wr_row;
logic [(`HASH_BITS)-1:0] core_wr_hash;
logic core_rd_valid;
logic [(`LANES)*(`ROW_BITS)-1:0] core_rd_row_ids;
logic [(`LANES)*(`HASH_BITS)-1:0] core_rd_hashes;
logic [(`LANES)-1:0] core_rd_lane_valid;
// ── Write noise pipeline ──
cam_write_noise #(
.WRITE_NOISE_EN (WRITE_NOISE_EN),
.WRITE_NOISE_RATE_NUM (WRITE_NOISE_RATE_NUM),
.WRITE_NOISE_RATE_DEN (WRITE_NOISE_RATE_DEN),
.WRITE_NOISE_SEED (WRITE_NOISE_SEED)
) u_write_noise (
.clk (clk),
.rst_n (rst_n),
.wr_valid (wr_valid),
.wr_ready (wr_ready),
.wr_row (wr_addr),
.wr_hash (write_hash),
.core_wr_valid (core_wr_valid),
.core_wr_row (core_wr_row),
.core_wr_hash (core_wr_hash)
);
// ── Banked synchronous BRAM storage ──
cam_core_banked u_core_banked (
.clk (clk),
.rst_n (rst_n),
.wr_valid (core_wr_valid),
.wr_ready (),
.wr_row (core_wr_row),
.wr_hash (core_wr_hash),
.rd_valid_i (rd_valid_i),
.rd_base_row_i (rd_base_row_i),
.rd_valid_o (core_rd_valid),
.rd_row_ids_o (core_rd_row_ids),
.rd_hashes_o (core_rd_hashes),
.rd_lane_valid_o (core_rd_lane_valid)
);
// ── Read noise pipeline ──
cam_read_noise u_read_noise (
.clk (clk),
.rst_n (rst_n),
.valid_i (core_rd_valid),
.row_ids_i (core_rd_row_ids),
.hashes_i (core_rd_hashes),
.lane_valid_i (core_rd_lane_valid),
.valid_o (rd_valid_o),
.row_ids_o (rd_row_ids_o),
.hashes_noisy_o (rd_hashes_o),
.lane_valid_o (rd_lane_valid_o)
);
endmodule

72
hw/rtl/cam_params.svh Normal file
View File

@@ -0,0 +1,72 @@
//==============================================================================
// CAM Parameter Definitions
//==============================================================================
// This file defines shared compile-time parameters for the Content-Addressable
// Memory (CAM) subsystem. All `define macros use `ifndef guards to allow
// command-line override via +define+.
//
// Macros:
// NUM_ROWS — Number of rows in the CAM array
// HASH_BITS — Width of the hash value in bits
// LANES — Number of parallel comparison lanes
// TOPK_K — Number of best matches retained per query
// FIFO_DEPTH — Candidate FIFO entries between match engine and tracker
// RESULT_SERIAL — Enables rank-ordered serial result output
// ROW_BITS — Bits needed to index NUM_ROWS rows ($clog2(NUM_ROWS))
// SCORE_BITS — Bits needed to represent HASH_BITS-bit scores ($clog2(HASH_BITS+1))
//
// The TIE_BREAK_SENTINEL localparam is a bitmask with all bits set to 1'b1,
// used as the initial / tie-breaking value in the row-priority arbiter.
//==============================================================================
`ifndef CAM_PARAMS_SVH
`define CAM_PARAMS_SVH
// Number of CAM rows.
`ifndef NUM_ROWS
`define NUM_ROWS 4096
`endif
// Width of the hash value stored per row.
`ifndef HASH_BITS
`define HASH_BITS 512
`endif
// Number of parallel comparison lanes.
`ifndef LANES
`define LANES 8
`endif
// Number of Top-K results to retain and serialize.
`ifndef TOPK_K
`define TOPK_K 4
`endif
// Candidate FIFO depth between score generation and Top-K tracking.
`ifndef FIFO_DEPTH
`define FIFO_DEPTH 16
`endif
// Result interface mode. First implementation uses serial Top-K output.
`ifndef RESULT_SERIAL
`define RESULT_SERIAL 1
`endif
// Bits required to represent a row index.
`ifndef ROW_BITS
`define ROW_BITS $clog2(`NUM_ROWS)
`endif
// Bits required to represent a full-score count.
`ifndef SCORE_BITS
`define SCORE_BITS $clog2(`HASH_BITS + 1)
`endif
// Tie-break sentinel: all ones in the row-index width.
// Used as the initial best-index value before any match is found.
localparam logic [(`ROW_BITS)-1:0] TIE_BREAK_SENTINEL = {(`ROW_BITS){1'b1}};
// First banked-pipeline implementation requires NUM_ROWS divisible by LANES.
// Tail lanes and unaligned read batches are intentionally out of scope.
`endif // CAM_PARAMS_SVH

127
hw/rtl/cam_top.sv Normal file
View File

@@ -0,0 +1,127 @@
`timescale 1ns/1ps
`include "cam_params.svh"
module cam_top #(
parameter bit WRITE_NOISE_EN = 1'b1,
parameter int WRITE_NOISE_RATE_NUM = 1,
parameter int WRITE_NOISE_RATE_DEN = 100,
parameter logic [63:0] WRITE_NOISE_SEED = 64'hB504_F32D_B504_F32D
) (
input logic clk,
input logic rst_n,
// Write interface (handshake)
input logic wr_valid,
output logic wr_ready,
input logic [(`ROW_BITS)-1:0] wr_addr,
input logic [(`HASH_BITS)-1:0] write_hash,
// Query interface
input logic query_valid,
output logic query_ready,
input logic [(`HASH_BITS)-1:0] query_hash,
// Result interface (serial Top-K)
output logic result_valid,
input logic result_ready,
output logic [((`TOPK_K <= 1) ? 1 : $clog2(`TOPK_K))-1:0] result_rank,
output logic [(`ROW_BITS)-1:0] result_row,
output logic [(`SCORE_BITS)-1:0] result_score,
output logic result_last,
// Legacy Top-1 aliases (captured from rank-0 serial beat)
output logic [(`ROW_BITS)-1:0] top1_index,
output logic [(`SCORE_BITS)-1:0] top1_score,
`ifdef SIM_DEBUG
output logic [(`NUM_ROWS)*(`SCORE_BITS)-1:0] score_debug_flat,
output logic [$clog2(`NUM_ROWS+1)-1:0] consumed_candidates_debug,
`endif
output logic busy
);
// ── Internal signals ──
logic storage_wr_ready; // cam_noisy write-side ready
logic match_query_ready; // match_engine_pipeline idle
logic match_busy; // match_engine_pipeline scanning/result pending
// ── Internal valid forwarding ──
logic storage_wr_valid;
logic match_query_valid;
// ── Half-duplex arbitration (write-priority) ──
// Active query scan blocks new writes.
assign wr_ready = storage_wr_ready && match_query_ready && !match_busy;
assign query_ready = storage_wr_ready && match_query_ready && !wr_valid;
assign busy = (!storage_wr_ready) || match_busy || (!match_query_ready);
// ── Internal valid forwarding (only assert to sub-modules when top-level accepts) ──
assign storage_wr_valid = wr_valid && wr_ready;
assign match_query_valid = query_valid && query_ready;
// ── Read request/response bus between cam_noisy and match_engine_pipeline ──
wire rd_req_valid;
wire [(`ROW_BITS)-1:0] rd_req_base_row;
wire rd_resp_valid;
wire [(`LANES)*(`ROW_BITS)-1:0] rd_resp_row_ids;
wire [(`LANES)*(`HASH_BITS)-1:0] rd_resp_hashes;
wire [(`LANES)-1:0] rd_resp_lane_valid;
cam_noisy #(
.WRITE_NOISE_EN (WRITE_NOISE_EN),
.WRITE_NOISE_RATE_NUM (WRITE_NOISE_RATE_NUM),
.WRITE_NOISE_RATE_DEN (WRITE_NOISE_RATE_DEN),
.WRITE_NOISE_SEED (WRITE_NOISE_SEED)
) u_noisy (
.clk (clk),
.rst_n (rst_n),
.wr_valid (storage_wr_valid),
.wr_ready (storage_wr_ready),
.wr_addr (wr_addr),
.write_hash (write_hash),
.rd_valid_i (rd_req_valid),
.rd_base_row_i (rd_req_base_row),
.rd_valid_o (rd_resp_valid),
.rd_row_ids_o (rd_resp_row_ids),
.rd_hashes_o (rd_resp_hashes),
.rd_lane_valid_o (rd_resp_lane_valid)
);
match_engine_pipeline u_match (
.clk (clk),
.rst_n (rst_n),
.query_valid (match_query_valid),
.query_ready (match_query_ready),
.query_hash (query_hash),
.result_valid (result_valid),
.result_ready (result_ready),
.result_rank (result_rank),
.result_row (result_row),
.result_score (result_score),
.result_last (result_last),
.busy (match_busy),
.rd_valid_o (rd_req_valid),
.rd_base_row_o (rd_req_base_row),
.rd_valid_i (rd_resp_valid),
.rd_row_ids_i (rd_resp_row_ids),
.rd_hashes_i (rd_resp_hashes),
.rd_lane_valid_i (rd_resp_lane_valid)
`ifdef SIM_DEBUG
,.score_debug_flat (score_debug_flat)
,.consumed_candidates_debug (consumed_candidates_debug)
`endif
);
// ── Rank-0 (Top-1) aliases ──
// Capture first serial beat (rank==0) into legacy top1_index/top1_score outputs.
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
top1_index <= '0;
top1_score <= '0;
end else if (result_valid && result_ready && (result_rank == '0)) begin
top1_index <= result_row;
top1_score <= result_score;
end
end
endmodule

View File

@@ -0,0 +1,117 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
module cam_bank #(
parameter int HASH_BITS = 512,
parameter int BANK_DEPTH = 512,
parameter int BANK_ADDR_BITS = 9
) (
input logic clk,
input logic wr_en,
input logic [BANK_ADDR_BITS-1:0] wr_addr,
input logic [HASH_BITS-1:0] wr_data,
input logic rd_en,
input logic [BANK_ADDR_BITS-1:0] rd_addr,
output logic [HASH_BITS-1:0] rd_data
);
(* ram_style = "block" *) logic [HASH_BITS-1:0] mem [0:BANK_DEPTH-1];
always_ff @(posedge clk) begin
if (wr_en) begin
mem[wr_addr] <= wr_data;
end
if (rd_en) begin
rd_data <= mem[rd_addr];
end
end
endmodule
module cam_core_banked (
input logic clk,
input logic rst_n,
input logic wr_valid,
output logic wr_ready,
input logic [(`ROW_BITS)-1:0] wr_row,
input logic [(`HASH_BITS)-1:0] wr_hash,
input logic rd_valid_i,
input logic [(`ROW_BITS)-1:0] rd_base_row_i,
output logic rd_valid_o,
output logic [(`LANES)*(`ROW_BITS)-1:0] rd_row_ids_o,
output logic [(`LANES)*(`HASH_BITS)-1:0] rd_hashes_o,
output logic [(`LANES)-1:0] rd_lane_valid_o
);
localparam int BANKS = `LANES;
localparam int BANK_DEPTH = `NUM_ROWS / BANKS;
localparam int BANK_SEL_BITS = $clog2(BANKS);
localparam int BANK_ADDR_BITS = $clog2(BANK_DEPTH);
logic [BANK_SEL_BITS-1:0] wr_bank_id;
logic [BANK_ADDR_BITS-1:0] wr_bank_addr;
logic [BANK_ADDR_BITS-1:0] rd_bank_addr;
logic [(`HASH_BITS)-1:0] bank_rd_data [0:BANKS-1];
assign wr_ready = 1'b1;
assign wr_bank_id = wr_row[BANK_SEL_BITS-1:0];
assign wr_bank_addr = wr_row[(`ROW_BITS)-1:BANK_SEL_BITS];
assign rd_bank_addr = rd_base_row_i[(`ROW_BITS)-1:BANK_SEL_BITS];
`ifndef SYNTHESIS
initial begin
if (`NUM_ROWS != (BANK_DEPTH * BANKS)) $fatal(1, "NUM_ROWS must be divisible by LANES");
if (`LANES != (1 << BANK_SEL_BITS)) $fatal(1, "LANES must be a power of two");
end
`endif
generate
for (genvar b = 0; b < BANKS; b++) begin : g_bank
cam_bank #(
.HASH_BITS(`HASH_BITS),
.BANK_DEPTH(BANK_DEPTH),
.BANK_ADDR_BITS(BANK_ADDR_BITS)
) u_bank (
.clk(clk),
.wr_en(wr_valid && (wr_bank_id == b[BANK_SEL_BITS-1:0])),
.wr_addr(wr_bank_addr),
.wr_data(wr_hash),
.rd_en(rd_valid_i),
.rd_addr(rd_bank_addr),
.rd_data(bank_rd_data[b])
);
end
endgenerate
always_comb begin
rd_hashes_o = '0;
if (rd_valid_o) begin
for (int lane = 0; lane < `LANES; lane++) begin
rd_hashes_o[lane*`HASH_BITS +: `HASH_BITS] = bank_rd_data[lane];
end
end
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
rd_valid_o <= 1'b0;
rd_row_ids_o <= '0;
rd_lane_valid_o <= '0;
end else begin
rd_valid_o <= rd_valid_i;
rd_lane_valid_o <= rd_valid_i ? {`LANES{1'b1}} : '0;
`ifndef SYNTHESIS
if (rd_valid_i && ((rd_base_row_i[BANK_SEL_BITS-1:0]) != '0)) begin
$fatal(1, "rd_base_row_i must be LANES-aligned");
end
`endif
for (int lane = 0; lane < `LANES; lane++) begin
rd_row_ids_o[lane*`ROW_BITS +: `ROW_BITS] <= rd_base_row_i + lane[(`ROW_BITS)-1:0];
end
end
end
endmodule

View File

@@ -0,0 +1,124 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
//==============================================================================
// candidate_fifo — Synchronous ready/valid FIFO for (row, score) candidates
//
// Implements a simple circular-buffer FIFO that accepts writes only when
// wr_valid_i && wr_ready_o and reads only when rd_valid_o && rd_ready_i.
// Provides empty/full flags. Default depth is `FIFO_DEPTH (16).
//
// Quality:
// - Pointers always wrap at DEPTH-1 (safe for non-power-of-two depths).
// - wr_ready_o goes high when full if a read is accepted in the same cycle,
// allowing a simultaneous write (throughput-friendly backpressure).
// - PTR_BITS is at least 1 for DEPTH=1.
//==============================================================================
module candidate_fifo #(
parameter int DEPTH = `FIFO_DEPTH,
parameter int ROW_BITS = `ROW_BITS,
parameter int SCORE_BITS = `SCORE_BITS,
// Ensure at least 1 bit even when DEPTH=1.
parameter int PTR_BITS = (DEPTH <= 1) ? 1 : $clog2(DEPTH)
) (
input logic clk,
input logic rst_n,
// Write interface (producer side)
input logic wr_valid_i,
output logic wr_ready_o,
input logic [ROW_BITS -1:0] wr_row_i,
input logic [SCORE_BITS -1:0] wr_score_i,
// Read interface (consumer side)
output logic rd_valid_o,
input logic rd_ready_i,
output logic [ROW_BITS -1:0] rd_row_o,
output logic [SCORE_BITS -1:0] rd_score_o,
// Status flags
output logic empty_o,
output logic full_o
);
//--------------------------------------------------------------------------
// Storage — dualport array.
//--------------------------------------------------------------------------
logic [ROW_BITS -1:0] mem_row [0:DEPTH-1];
logic [SCORE_BITS -1:0] mem_score [0:DEPTH-1];
//--------------------------------------------------------------------------
// Pointers and fill counter
//--------------------------------------------------------------------------
logic [PTR_BITS -1:0] wr_ptr, rd_ptr;
logic [PTR_BITS :0] count; // extra bit for full detect
// Wrap-at value (DEPTH-1), sized to match pointer width.
// Part-select DEPTH and use a 1'b1 literal so the RHS width equals PTR_BITS.
localparam logic [PTR_BITS-1:0] WRAP_AT = DEPTH[PTR_BITS-1:0] - 1'b1;
//--------------------------------------------------------------------------
// Status flag outputs (combinational)
//--------------------------------------------------------------------------
assign empty_o = (count == 0);
assign full_o = (count == DEPTH[PTR_BITS:0]);
//--------------------------------------------------------------------------
// Handshake outputs
//
// wr_ready_o is asserted when not full, OR when a read is accepted in
// the same cycle (full+pop → slot freed → write can proceed).
//--------------------------------------------------------------------------
assign wr_ready_o = ~full_o | (rd_valid_o & rd_ready_i);
assign rd_valid_o = ~empty_o;
//--------------------------------------------------------------------------
// Read data (combinational — value at current rd_ptr)
//--------------------------------------------------------------------------
assign rd_row_o = mem_row [rd_ptr];
assign rd_score_o = mem_score[rd_ptr];
//--------------------------------------------------------------------------
// Push / Pop decode
//--------------------------------------------------------------------------
logic push, pop;
assign push = wr_valid_i & wr_ready_o;
assign pop = rd_valid_o & rd_ready_i;
//--------------------------------------------------------------------------
// Sequential — pointer and count updates
//
// Pointers wrap at DEPTH-1, safe for any DEPTH (not just powers of two).
// Simultaneous push+pop leaves count unchanged.
//--------------------------------------------------------------------------
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
wr_ptr <= 0;
rd_ptr <= 0;
count <= 0;
end else begin
if (push) begin
mem_row [wr_ptr] <= wr_row_i;
mem_score[wr_ptr] <= wr_score_i;
if (wr_ptr == WRAP_AT)
wr_ptr <= 0;
else
wr_ptr <= wr_ptr + 1;
end
if (pop) begin
if (rd_ptr == WRAP_AT)
rd_ptr <= 0;
else
rd_ptr <= rd_ptr + 1;
end
// Update fill count
if (push & ~pop) count <= count + 1;
else if (~push & pop) count <= count - 1;
// else: no change (neither, or both)
end
end
endmodule

View File

@@ -0,0 +1,476 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
module match_engine_pipeline (
input logic clk,
input logic rst_n,
input logic query_valid,
output logic query_ready,
input logic [(`HASH_BITS)-1:0] query_hash,
// Serial Top-K result interface
output logic result_valid,
input logic result_ready,
output logic [((`TOPK_K <= 1) ? 1 : $clog2(`TOPK_K))-1:0] result_rank,
output logic [(`ROW_BITS)-1:0] result_row,
output logic [(`SCORE_BITS)-1:0] result_score,
output logic result_last,
output logic busy,
output logic rd_valid_o,
output logic [(`ROW_BITS)-1:0] rd_base_row_o,
input logic rd_valid_i,
input logic [(`LANES)*(`ROW_BITS)-1:0] rd_row_ids_i,
input logic [(`LANES)*(`HASH_BITS)-1:0] rd_hashes_i,
input logic [(`LANES)-1:0] rd_lane_valid_i
`ifdef SIM_DEBUG
,output logic [(`NUM_ROWS)*(`SCORE_BITS)-1:0] score_debug_flat
,output logic [($clog2(`NUM_ROWS+1))-1:0] consumed_candidates_debug
`endif
);
//==========================================================================
// State encoding
//==========================================================================
typedef enum logic [3:0] {
S_IDLE,
S_ISSUE_READ,
S_WAIT_READ_RESP,
S_WAIT_SCORE,
S_STAGE_CANDIDATES,
S_PUSH_CANDIDATES,
S_DRAIN,
S_SERIALIZE_RESULT,
S_DONE
} state_t;
state_t state_q;
//==========================================================================
// Query / read tracking
//==========================================================================
logic [(`HASH_BITS)-1:0] query_q;
logic [(`ROW_BITS)-1:0] issue_base_q;
logic [$clog2(`NUM_ROWS/`LANES+1)-1:0] issued_batches_q;
logic [$clog2(`NUM_ROWS/`LANES+1)-1:0] returned_batches_q;
logic [$clog2(`NUM_ROWS+1)-1:0] pushed_candidates_q;
logic [$clog2(`NUM_ROWS+1)-1:0] consumed_candidates_q;
// Total expected valid candidates across all batches
logic [$clog2(`NUM_ROWS+1)-1:0] expected_candidates_q;
// Batch-level valid-lane mask for return tracking
logic [(`LANES)-1:0] batch_lane_mask_q;
//--------------------------------------------------------------------------
// Read/noise staging registers (feed popcount pipeline)
//--------------------------------------------------------------------------
logic rd_stage_valid_q;
logic [(`LANES)*(`ROW_BITS)-1:0] rd_stage_row_ids_q;
logic [(`LANES)*(`HASH_BITS)-1:0] rd_stage_hashes_q;
logic [(`LANES)-1:0] rd_stage_lane_valid_q;
//--------------------------------------------------------------------------
// Candidate staging registers (scored output from popcount, before FIFO)
//--------------------------------------------------------------------------
logic [(`LANES)-1:0] staged_valid_q;
logic [(`LANES)*(`ROW_BITS)-1:0] staged_rows_q;
logic [(`LANES)*(`SCORE_BITS)-1:0] staged_scores_q;
logic [$clog2(`LANES)-1:0] push_lane_idx_q;
//==========================================================================
// Popcount per-lane wires
//==========================================================================
logic [(`HASH_BITS)-1:0] match_bits [0:`LANES-1];
logic score_valid [0:`LANES-1];
logic [(`ROW_BITS)-1:0] score_row [0:`LANES-1];
logic [(`SCORE_BITS)-1:0] lane_score [0:`LANES-1];
//==========================================================================
// Candidate FIFO interface
//==========================================================================
logic cand_valid;
logic cand_ready;
logic [(`ROW_BITS)-1:0] cand_row;
logic [(`SCORE_BITS)-1:0] cand_score;
wire cand_fire = cand_valid && cand_ready;
//==========================================================================
// FIFO wires
//==========================================================================
logic fifo_wr_valid;
logic fifo_wr_ready;
logic [(`ROW_BITS)-1:0] fifo_wr_row;
logic [(`SCORE_BITS)-1:0] fifo_wr_score;
logic fifo_rd_valid;
logic fifo_rd_ready;
logic [(`ROW_BITS)-1:0] fifo_rd_row;
logic [(`SCORE_BITS)-1:0] fifo_rd_score;
logic fifo_empty;
//==========================================================================
// Tracker wires
//==========================================================================
logic tracker_clear;
logic tracker_candidate_ready;
logic [(`TOPK_K)*(`ROW_BITS)-1:0] tracker_topk_rows;
logic [(`TOPK_K)*(`SCORE_BITS)-1:0] tracker_topk_scores;
logic tracker_update_pending;
//==========================================================================
// Serializer wires
//==========================================================================
logic serializer_start;
logic serializer_busy;
logic serializer_done;
logic serializer_result_valid;
logic [((`TOPK_K <= 1) ? 1 : $clog2(`TOPK_K))-1:0] serializer_result_rank;
logic [(`ROW_BITS)-1:0] serializer_result_row;
logic [(`SCORE_BITS)-1:0] serializer_result_score;
logic serializer_result_last;
//==========================================================================
// Completion condition wires
// Width-matched comparison targets to suppress Verilator WIDTHEXPAND.
// The assigned constants fit exactly — WIDTHTRUNC is benign.
/* verilator lint_off WIDTHTRUNC */
wire [$clog2(`NUM_ROWS/`LANES+1)-1:0] num_batches_w = `NUM_ROWS / `LANES;
wire [$clog2(`LANES)-1:0] last_lane_w = `LANES - 1;
/* verilator lint_on WIDTHTRUNC */
wire all_reads_issued = (issued_batches_q == num_batches_w);
wire all_scores_returned = (returned_batches_q == num_batches_w);
wire all_candidates_pushed = (pushed_candidates_q == expected_candidates_q);
wire all_candidates_consumed = (consumed_candidates_q == expected_candidates_q);
wire scan_drained = all_reads_issued && all_scores_returned
&& all_candidates_pushed
&& all_candidates_consumed
&& fifo_empty
&& !tracker_update_pending;
//==========================================================================
// Combinational output assignments
//==========================================================================
assign query_ready = (state_q == S_IDLE);
assign rd_valid_o = (state_q == S_ISSUE_READ);
assign rd_base_row_o = issue_base_q;
assign busy = (state_q != S_IDLE) || serializer_busy;
`ifdef SIM_DEBUG
assign consumed_candidates_debug = consumed_candidates_q;
`endif
// Result outputs come directly from the serializer
assign result_valid = serializer_result_valid;
assign result_rank = serializer_result_rank;
assign result_row = serializer_result_row;
assign result_score = serializer_result_score;
assign result_last = serializer_result_last;
//==========================================================================
// Generate: popcount pipeline per lane
//==========================================================================
generate
for (genvar lane = 0; lane < `LANES; lane++) begin : gen_scores
assign match_bits[lane] = ~(query_q ^ rd_stage_hashes_q[lane*`HASH_BITS +: `HASH_BITS]);
popcount_pipeline #(
.WIDTH(`HASH_BITS),
.ROW_BITS(`ROW_BITS),
.OUT_WIDTH(`SCORE_BITS)
) u_popcount_pipeline (
.clk(clk),
.rst_n(rst_n),
.valid_i(rd_stage_valid_q && rd_stage_lane_valid_q[lane]),
.row_i(rd_stage_row_ids_q[lane*`ROW_BITS +: `ROW_BITS]),
.bits_i(match_bits[lane]),
.valid_o(score_valid[lane]),
.row_o(score_row[lane]),
.count_o(lane_score[lane])
);
end
endgenerate
//==========================================================================
// Batch scores done: true when all valid lanes in the current batch
// have their score_valid asserted, or vacuously true for an all-invalid batch.
wire batch_scores_done;
generate
if (`LANES == 1) begin : gen_batch_done_1
assign batch_scores_done = rd_stage_lane_valid_q[0]
? score_valid[0] : 1'b1;
end else begin : gen_batch_done_n
logic all_valid_ready;
always_comb begin
all_valid_ready = 1'b1;
for (int l = 0; l < `LANES; l++) begin
if (rd_stage_lane_valid_q[l] && !score_valid[l]) begin
all_valid_ready = 1'b0;
end
end
end
assign batch_scores_done = all_valid_ready;
end
endgenerate
//==========================================================================
// Candidate FIFO write interface
//==========================================================================
assign cand_valid = (state_q == S_PUSH_CANDIDATES)
&& staged_valid_q[push_lane_idx_q];
assign cand_row = staged_rows_q[push_lane_idx_q * `ROW_BITS +: `ROW_BITS];
assign cand_score = staged_scores_q[push_lane_idx_q * `SCORE_BITS +: `SCORE_BITS];
assign cand_ready = fifo_wr_ready;
assign fifo_wr_valid = cand_valid;
assign fifo_wr_row = cand_row;
assign fifo_wr_score = cand_score;
//==========================================================================
// FIFO output
//==========================================================================
assign fifo_rd_ready = tracker_candidate_ready;
wire fifo_rd_fire = fifo_rd_valid && fifo_rd_ready;
//==========================================================================
// Serializer start edge detection
//==========================================================================
logic start_ser_q1, start_ser_q2;
wire start_ser_raw = (state_q == S_DRAIN) && scan_drained;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
start_ser_q1 <= 1'b0;
start_ser_q2 <= 1'b0;
end else begin
start_ser_q1 <= start_ser_raw;
start_ser_q2 <= start_ser_q1;
end
end
assign serializer_start = start_ser_q1 && !start_ser_q2;
//==========================================================================
// Tracker clear: pulse on new query
//==========================================================================
assign tracker_clear = (state_q == S_IDLE) && query_valid;
//==========================================================================
// Submodule instantiations
//==========================================================================
// --- candidate_fifo (full_o unused) ---
/* verilator lint_off PINCONNECTEMPTY */
candidate_fifo #(
.DEPTH(`FIFO_DEPTH),
.ROW_BITS(`ROW_BITS),
.SCORE_BITS(`SCORE_BITS)
) u_candidate_fifo (
.clk(clk),
.rst_n(rst_n),
.wr_valid_i(fifo_wr_valid),
.wr_ready_o(fifo_wr_ready),
.wr_row_i(fifo_wr_row),
.wr_score_i(fifo_wr_score),
.rd_valid_o(fifo_rd_valid),
.rd_ready_i(fifo_rd_ready),
.rd_row_o(fifo_rd_row),
.rd_score_o(fifo_rd_score),
.empty_o(fifo_empty),
.full_o() // intentionally unused
);
/* verilator lint_on PINCONNECTEMPTY */
// --- topk_tracker ---
topk_tracker #(
.K(`TOPK_K),
.ROW_BITS(`ROW_BITS),
.SCORE_BITS(`SCORE_BITS)
) u_topk_tracker (
.clk(clk),
.rst_n(rst_n),
.clear_i(tracker_clear),
.candidate_valid_i(fifo_rd_valid),
.candidate_ready_o(tracker_candidate_ready),
.candidate_row_i(fifo_rd_row),
.candidate_score_i(fifo_rd_score),
.topk_rows_o(tracker_topk_rows),
.topk_scores_o(tracker_topk_scores),
.update_pending_o(tracker_update_pending)
);
// --- result_serializer ---
result_serializer #(
.K(`TOPK_K),
.ROW_BITS(`ROW_BITS),
.SCORE_BITS(`SCORE_BITS)
) u_result_serializer (
.clk(clk),
.rst_n(rst_n),
.start_i(serializer_start),
.busy_o(serializer_busy),
.done_o(serializer_done),
.topk_rows_i(tracker_topk_rows),
.topk_scores_i(tracker_topk_scores),
.result_valid_o(serializer_result_valid),
.result_ready_i(result_ready),
.result_rank_o(serializer_result_rank),
.result_row_o(serializer_result_row),
.result_score_o(serializer_result_score),
.result_last_o(serializer_result_last)
);
//==========================================================================
// Sequential: main FSM + registers
//==========================================================================
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
state_q <= S_IDLE;
query_q <= '0;
issue_base_q <= '0;
issued_batches_q <= 0;
returned_batches_q <= 0;
pushed_candidates_q <= 0;
consumed_candidates_q <= 0;
expected_candidates_q <= 0;
batch_lane_mask_q <= '0;
rd_stage_valid_q <= 1'b0;
rd_stage_row_ids_q <= '0;
rd_stage_hashes_q <= '0;
rd_stage_lane_valid_q <= '0;
staged_valid_q <= '0;
staged_rows_q <= '0;
staged_scores_q <= '0;
push_lane_idx_q <= 0;
`ifdef SIM_DEBUG
score_debug_flat <= '0;
`endif
end else begin
// Defaults for one-cycle flags
rd_stage_valid_q <= 1'b0;
// Consumed candidates counter (FIFO → Tracker handshake)
if (fifo_rd_fire) begin
consumed_candidates_q <= consumed_candidates_q + 1;
end
// State machine
unique case (state_q)
S_IDLE: begin
if (query_valid) begin
query_q <= query_hash;
issue_base_q <= '0;
issued_batches_q <= 0;
returned_batches_q <= 0;
pushed_candidates_q <= 0;
consumed_candidates_q <= 0;
expected_candidates_q <= 0;
batch_lane_mask_q <= '0;
push_lane_idx_q <= 0;
staged_valid_q <= '0;
`ifdef SIM_DEBUG
score_debug_flat <= '0;
`endif
state_q <= S_ISSUE_READ;
end
end
S_ISSUE_READ: begin
// Emit rd_valid_o for exactly one cycle (combinational), then
// transition to S_WAIT_READ_RESP. This keeps one logical read
// request per batch issue before waiting for the response path.
state_q <= S_WAIT_READ_RESP;
end
S_WAIT_READ_RESP: begin
if (rd_valid_i) begin
// Capture staging data for popcount pipeline
rd_stage_valid_q <= 1'b1;
rd_stage_row_ids_q <= rd_row_ids_i;
rd_stage_hashes_q <= rd_hashes_i;
rd_stage_lane_valid_q <= rd_lane_valid_i;
// Remember the lane mask for this batch (used in S_WAIT_SCORE
// to determine batch_scores_done).
batch_lane_mask_q <= rd_lane_valid_i;
issue_base_q <= issue_base_q + `LANES;
issued_batches_q <= issued_batches_q + 1;
// Accumulate total expected valid candidates
expected_candidates_q <= expected_candidates_q
+ $countones(rd_lane_valid_i);
state_q <= S_WAIT_SCORE;
end
end
S_WAIT_SCORE: begin
// rd_stage_valid_q already deasserted by default above
if (batch_scores_done) begin
returned_batches_q <= returned_batches_q + 1;
// Capture per-lane scores and rows into staging registers
for (int lane = 0; lane < `LANES; lane++) begin
if (batch_lane_mask_q[lane]) begin
staged_valid_q[lane] <= 1'b1;
staged_rows_q[lane*`ROW_BITS +: `ROW_BITS] <= score_row[lane];
staged_scores_q[lane*`SCORE_BITS +: `SCORE_BITS] <= lane_score[lane];
end else begin
staged_valid_q[lane] <= 1'b0;
end
end
`ifdef SIM_DEBUG
for (int lane = 0; lane < `LANES; lane++) begin
if (batch_lane_mask_q[lane]) begin
score_debug_flat[score_row[lane]*`SCORE_BITS +: `SCORE_BITS]
<= lane_score[lane];
end
end
`endif
state_q <= S_STAGE_CANDIDATES;
end
end
S_STAGE_CANDIDATES: begin
state_q <= S_PUSH_CANDIDATES;
end
S_PUSH_CANDIDATES: begin
// Advance if the current lane fired or wasn't valid
if (cand_fire || !staged_valid_q[push_lane_idx_q]) begin
if (cand_fire) begin
pushed_candidates_q <= pushed_candidates_q + 1;
end
if (push_lane_idx_q == last_lane_w) begin
// Last lane processed — decide next state
push_lane_idx_q <= 0;
if (all_reads_issued) begin
state_q <= S_DRAIN;
end else begin
state_q <= S_ISSUE_READ;
end
end else begin
push_lane_idx_q <= push_lane_idx_q + 1;
end
end
end
S_DRAIN: begin
if (scan_drained) begin
state_q <= S_SERIALIZE_RESULT;
end
end
S_SERIALIZE_RESULT: begin
if (serializer_done) begin
state_q <= S_DONE;
end
end
S_DONE: begin
state_q <= S_IDLE;
end
default: state_q <= S_IDLE;
endcase
end
end
endmodule

View File

@@ -0,0 +1,72 @@
`timescale 1ns / 1ps
module popcount_pipeline #(
parameter int WIDTH = 512,
parameter int ROW_BITS = 12,
parameter int OUT_WIDTH = $clog2(WIDTH + 1)
) (
input logic clk,
input logic rst_n,
input logic valid_i,
input logic [ROW_BITS-1:0] row_i,
input logic [WIDTH-1:0] bits_i,
output logic valid_o,
output logic [ROW_BITS-1:0] row_o,
output logic [OUT_WIDTH-1:0] count_o
);
localparam int GROUP = 8;
localparam int NUM_GROUPS = WIDTH / GROUP;
localparam int GROUP_COUNT_WIDTH = $clog2(GROUP + 1);
localparam int STAGE1_GROUPS = NUM_GROUPS / 4;
logic [GROUP_COUNT_WIDTH-1:0] group_counts [0:NUM_GROUPS-1];
logic [OUT_WIDTH-1:0] partial_q [0:3];
logic [OUT_WIDTH-1:0] sum_q;
logic [ROW_BITS-1:0] row_s1_q, row_s2_q;
logic valid_s1_q, valid_s2_q;
logic [OUT_WIDTH-1:0] partial_comb [0:3];
always_comb begin
for (int g = 0; g < NUM_GROUPS; g++) begin
group_counts[g] = '0;
for (int b = 0; b < GROUP; b++) begin
group_counts[g] = group_counts[g] + bits_i[g*GROUP + b];
end
end
for (int p = 0; p < 4; p++) begin
partial_comb[p] = '0;
for (int g = 0; g < STAGE1_GROUPS; g++) begin
partial_comb[p] = partial_comb[p] + group_counts[p*STAGE1_GROUPS + g];
end
end
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
valid_s1_q <= 1'b0;
valid_s2_q <= 1'b0;
valid_o <= 1'b0;
row_s1_q <= '0;
row_s2_q <= '0;
row_o <= '0;
sum_q <= '0;
count_o <= '0;
for (int p = 0; p < 4; p++) partial_q[p] <= '0;
end else begin
valid_s1_q <= valid_i;
row_s1_q <= row_i;
for (int p = 0; p < 4; p++) begin
partial_q[p] <= partial_comb[p];
end
valid_s2_q <= valid_s1_q;
row_s2_q <= row_s1_q;
sum_q <= partial_q[0] + partial_q[1] + partial_q[2] + partial_q[3];
valid_o <= valid_s2_q;
row_o <= row_s2_q;
count_o <= sum_q;
end
end
endmodule

View File

@@ -0,0 +1,126 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
//==============================================================================
// Result Serializer
//==============================================================================
// Serializes the packed Top-K array (rows and scores) into a rank-ordered
// handshake interface, one result per fire.
//
// Start is edge-triggered (sampled on the cycle start_i is high while not
// already active). Once started, the module presents outputs for the
// current rank (starting at 0) and advances on each valid-ready fire.
// After the last rank (K-1) fires, done_o pulses for one cycle and the
// module returns to idle with rank reset to 0.
//
// Back-pressure: holds current output while result_ready_i is low.
//
// Supports K=1 cleanly (last rank = rank 0, done on first fire).
//==============================================================================
module result_serializer #(
parameter int K = `TOPK_K,
parameter int ROW_BITS = `ROW_BITS,
parameter int SCORE_BITS = `SCORE_BITS,
parameter int RANK_BITS = (K <= 1) ? 1 : $clog2(K)
) (
input logic clk,
input logic rst_n,
input logic start_i,
output logic busy_o,
output logic done_o,
input logic [K*ROW_BITS-1:0] topk_rows_i,
input logic [K*SCORE_BITS-1:0] topk_scores_i,
output logic result_valid_o,
input logic result_ready_i,
output logic [RANK_BITS-1:0] result_rank_o,
output logic [ROW_BITS-1:0] result_row_o,
output logic [SCORE_BITS-1:0] result_score_o,
output logic result_last_o
);
// ── State encoding ─────────────────────────────────────────────────────
typedef enum logic {
IDLE,
ACTIVE
} state_t;
// ── Localparam for last-rank comparison (width-matched to avoid warnings) ─
localparam [RANK_BITS-1:0] LAST_RANK = RANK_BITS'(K - 1);
state_t state_q, state_next;
logic [RANK_BITS-1:0] rank_q, rank_next;
logic done_q, done_next;
// Snapshot of packed inputs captured at start (holds during back-pressure)
logic [K*ROW_BITS-1:0] rows_snapshot_q, rows_snapshot_next;
logic [K*SCORE_BITS-1:0] scores_snapshot_q, scores_snapshot_next;
// ── Sequential: state, rank, done, and snapshot registers ─────────────
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
state_q <= IDLE;
rank_q <= '0;
done_q <= 1'b0;
rows_snapshot_q <= '0;
scores_snapshot_q <= '0;
end else begin
state_q <= state_next;
rank_q <= rank_next;
done_q <= done_next;
rows_snapshot_q <= rows_snapshot_next;
scores_snapshot_q <= scores_snapshot_next;
end
end
// ── Combinational: next-state, snapshot, and output logic ──────────────
always_comb begin
// Defaults: stay in current state, retain rank, no done pulse,
// hold snapshot (outputs stable under back-pressure)
state_next = state_q;
rank_next = rank_q;
done_next = 1'b0;
rows_snapshot_next = rows_snapshot_q;
scores_snapshot_next = scores_snapshot_q;
if (state_q == IDLE) begin
// Edge-triggered start — capture packed inputs into snapshot
if (start_i) begin
state_next = ACTIVE;
rank_next = '0; // start from rank 0
rows_snapshot_next = topk_rows_i;
scores_snapshot_next = topk_scores_i;
end
end else begin
// ACTIVE: advance on fire
if (result_valid_o && result_ready_i) begin
// Is this the last rank?
if (rank_q == LAST_RANK) begin
state_next = IDLE;
rank_next = '0;
done_next = 1'b1;
end else begin
rank_next = rank_q + 1'b1;
end
end
end
end
// ── Output assignments ─────────────────────────────────────────────────
assign busy_o = (state_q == ACTIVE);
assign done_o = done_q; // registered — pulses after deactivation
assign result_valid_o = (state_q == ACTIVE);
// Slice the snapshot registers at the current rank
assign result_row_o = rows_snapshot_q[rank_q * ROW_BITS +: ROW_BITS];
assign result_score_o = scores_snapshot_q[rank_q * SCORE_BITS +: SCORE_BITS];
assign result_rank_o = rank_q;
// result_last_o: high only when active and on the last rank
assign result_last_o = (state_q == ACTIVE) && (rank_q == LAST_RANK);
endmodule

126
hw/rtl/core/topk_tracker.sv Normal file
View File

@@ -0,0 +1,126 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
//==============================================================================
// Top-K Tracker
//==============================================================================
// Maintains a sorted Top-K array of (row, score) candidates.
// Insertion sort on every candidate handshake:
// - Higher score wins.
// - If equal score, lower row wins (tie-break).
// - Entries that are worse than all K current slots are dropped.
//
// candidate_ready_o is always 1 — the tracker accepts a candidate every cycle.
// update_pending_o is always 0 — insertion is fully combinational (one cycle).
//
// Packed outputs:
// topk_rows_o [i*ROW_BITS +: ROW_BITS] = rank i row
// topk_scores_o [i*SCORE_BITS +: SCORE_BITS] = rank i score
//==============================================================================
module topk_tracker #(
parameter int K = `TOPK_K,
parameter int ROW_BITS = `ROW_BITS,
parameter int SCORE_BITS = `SCORE_BITS
) (
input logic clk,
input logic rst_n,
input logic clear_i,
input logic candidate_valid_i,
output logic candidate_ready_o,
input logic [ROW_BITS-1:0] candidate_row_i,
input logic [SCORE_BITS-1:0] candidate_score_i,
output logic [K*ROW_BITS-1:0] topk_rows_o,
output logic [K*SCORE_BITS-1:0] topk_scores_o,
output logic update_pending_o
);
// ── Sequential state: K-entry sorted arrays ────────────────────────────
logic [ROW_BITS-1:0] rows_q [0:K-1];
logic [SCORE_BITS-1:0] scores_q [0:K-1];
// ── Next-state signals ─────────────────────────────────────────────────
logic [ROW_BITS-1:0] rows_next [0:K-1];
logic [SCORE_BITS-1:0] scores_next [0:K-1];
// ── Static output assignments ──────────────────────────────────────────
assign candidate_ready_o = 1'b1;
assign update_pending_o = 1'b0;
// ── Combinational: insertion logic ─────────────────────────────────────
always_comb begin
int insert_idx;
insert_idx = K; // always assigned — default: no insertion
// Default: retain current state
for (int i = 0; i < K; i++) begin
rows_next[i] = rows_q[i];
scores_next[i] = scores_q[i];
end
// Only evaluate insertion when a valid candidate is presented
if (candidate_valid_i) begin
// Find the first slot where candidate is strictly better than the
// current occupant. "Better" means higher score, or equal score
// with lower row (tie-break).
// Note: guard on insert_idx == K avoids loop-local break which
// Yosys rejects; the effect is the same — only the first match
// is captured because subsequent iterations see insert_idx != K.
for (int i = 0; i < K; i++) begin
if (insert_idx == K &&
(candidate_score_i > scores_q[i] ||
(candidate_score_i == scores_q[i] && candidate_row_i < rows_q[i]))) begin
insert_idx = i;
end
end
// Shift slots down from the insertion point and insert.
// Static loop bounds with guard for Yosys compatibility.
if (insert_idx < K) begin
for (int i = 0; i < K; i++) begin
if (i == insert_idx) begin
rows_next[i] = candidate_row_i;
scores_next[i] = candidate_score_i;
end else if (i > insert_idx) begin
rows_next[i] = rows_q[i - 1];
scores_next[i] = scores_q[i - 1];
end
// i < insert_idx: keep default from copy loop above
end
end
end
end
// ── Sequential: state update ───────────────────────────────────────────
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < K; i++) begin
rows_q[i] <= {ROW_BITS{1'b1}}; // all-ones sentinel
scores_q[i] <= '0;
end
end else if (clear_i) begin
for (int i = 0; i < K; i++) begin
rows_q[i] <= {ROW_BITS{1'b1}};
scores_q[i] <= '0;
end
end else if (candidate_valid_i) begin
for (int i = 0; i < K; i++) begin
rows_q[i] <= rows_next[i];
scores_q[i] <= scores_next[i];
end
end
end
// ── Combinational: pack arrays into flat outputs ───────────────────────
always_comb begin
topk_rows_o = '0;
topk_scores_o = '0;
for (int i = 0; i < K; i++) begin
topk_rows_o[i*ROW_BITS +: ROW_BITS] = rows_q[i];
topk_scores_o[i*SCORE_BITS +: SCORE_BITS] = scores_q[i];
end
end
endmodule

View File

@@ -0,0 +1,31 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
module cam_read_noise (
input logic clk,
input logic rst_n,
input logic valid_i,
input logic [(`LANES)*(`ROW_BITS)-1:0] row_ids_i,
input logic [(`LANES)*(`HASH_BITS)-1:0] hashes_i,
input logic [(`LANES)-1:0] lane_valid_i,
output logic valid_o,
output logic [(`LANES)*(`ROW_BITS)-1:0] row_ids_o,
output logic [(`LANES)*(`HASH_BITS)-1:0] hashes_noisy_o,
output logic [(`LANES)-1:0] lane_valid_o
);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
valid_o <= 1'b0;
row_ids_o <= '0;
hashes_noisy_o <= '0;
lane_valid_o <= '0;
end else begin
valid_o <= valid_i;
row_ids_o <= row_ids_i;
hashes_noisy_o <= hashes_i;
lane_valid_o <= lane_valid_i;
end
end
endmodule

View File

@@ -0,0 +1,113 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
module cam_write_noise #(
parameter bit WRITE_NOISE_EN = 1'b1,
parameter int WRITE_NOISE_RATE_NUM = 1,
parameter int WRITE_NOISE_RATE_DEN = 100,
parameter logic [63:0] WRITE_NOISE_SEED = 64'hB504_F32D_B504_F32D
) (
input logic clk,
input logic rst_n,
input logic wr_valid,
output logic wr_ready,
input logic [(`ROW_BITS)-1:0] wr_row,
input logic [(`HASH_BITS)-1:0] wr_hash,
output logic core_wr_valid,
output logic [(`ROW_BITS)-1:0] core_wr_row,
output logic [(`HASH_BITS)-1:0] core_wr_hash
);
localparam int PROB_BITS = 8;
localparam int SAMPLE_RANGE = 1 << PROB_BITS;
localparam int WRITE_NOISE_THRESHOLD_RAW =
(WRITE_NOISE_RATE_NUM * SAMPLE_RANGE) / WRITE_NOISE_RATE_DEN;
localparam int WRITE_NOISE_THRESHOLD =
(WRITE_NOISE_THRESHOLD_RAW > (SAMPLE_RANGE - 1)) ?
(SAMPLE_RANGE - 1) : WRITE_NOISE_THRESHOLD_RAW;
wire noise_active = WRITE_NOISE_EN && (WRITE_NOISE_RATE_NUM > 0) && (WRITE_NOISE_THRESHOLD > 0);
typedef enum logic [1:0] {
STATE_IDLE,
STATE_WAIT_MASK
} state_t;
state_t state_q;
logic mask_start_q;
logic mask_busy;
logic mask_done;
logic [(`HASH_BITS)-1:0] flip_mask;
logic [(`ROW_BITS)-1:0] row_q;
logic [(`HASH_BITS)-1:0] hash_q;
assign wr_ready = (state_q == STATE_IDLE);
noise_mask_bernoulli #(
.HASH_BITS (`HASH_BITS),
.PROB_BITS (PROB_BITS),
.PRNG_WORDS (2),
.BITS_PER_CYCLE (32),
.SEED (WRITE_NOISE_SEED)
) u_bernoulli_mask (
.clk (clk),
.rst_n (rst_n),
.start_i (mask_start_q),
.threshold_i (PROB_BITS'(WRITE_NOISE_THRESHOLD)),
.busy_o (mask_busy),
.done_o (mask_done),
.mask_o (flip_mask)
);
`ifndef SYNTHESIS
initial begin
if (WRITE_NOISE_SEED == 64'd0) $fatal(1, "WRITE_NOISE_SEED must be nonzero");
end
`endif
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
state_q <= STATE_IDLE;
mask_start_q <= 1'b0;
row_q <= '0;
hash_q <= '0;
core_wr_valid <= 1'b0;
core_wr_row <= '0;
core_wr_hash <= '0;
end else begin
core_wr_valid <= 1'b0;
mask_start_q <= 1'b0;
unique case (state_q)
STATE_IDLE: begin
if (wr_valid && wr_ready) begin
row_q <= wr_row;
hash_q <= wr_hash;
if (noise_active) begin
mask_start_q <= 1'b1;
state_q <= STATE_WAIT_MASK;
end else begin
// Noise inactive: pass through immediately (one-cycle)
core_wr_valid <= 1'b1;
core_wr_row <= wr_row;
core_wr_hash <= wr_hash;
end
end
end
STATE_WAIT_MASK: begin
if (mask_done) begin
core_wr_valid <= 1'b1;
core_wr_row <= row_q;
core_wr_hash <= hash_q ^ flip_mask;
state_q <= STATE_IDLE;
end
end
default: begin
state_q <= STATE_IDLE;
end
endcase
end
end
endmodule

View File

@@ -0,0 +1,126 @@
`timescale 1ns / 1ps
`include "cam_params.svh"
module noise_mask_bernoulli #(
parameter int HASH_BITS = `HASH_BITS,
parameter int PROB_BITS = 8,
parameter int PRNG_WORDS = 2,
parameter int BITS_PER_CYCLE = 32,
parameter logic [63:0] SEED = 64'hD1B5_4A32_9E37_79B9
) (
input logic clk,
input logic rst_n,
input logic start_i,
input logic [PROB_BITS-1:0] threshold_i,
output logic busy_o,
output logic done_o,
output logic [HASH_BITS-1:0] mask_o
);
localparam int RANDOM_BITS = PRNG_WORDS * 128;
localparam int BLOCKS = HASH_BITS / BITS_PER_CYCLE;
localparam int BLOCK_INDEX_BITS = (BLOCKS <= 1) ? 1 : $clog2(BLOCKS);
localparam logic [BLOCK_INDEX_BITS-1:0] LAST_BLOCK_IDX = BLOCK_INDEX_BITS'(BLOCKS - 1);
typedef enum logic [1:0] {
STATE_IDLE,
STATE_PRIME,
STATE_RUN
} state_t;
state_t state_q;
logic [BLOCK_INDEX_BITS-1:0] block_idx_q;
logic [PROB_BITS-1:0] threshold_q;
logic [HASH_BITS-1:0] mask_q;
logic prng_en;
logic [RANDOM_BITS-1:0] random_bits;
logic [BITS_PER_CYCLE-1:0] mask_slice;
assign busy_o = (state_q != STATE_IDLE);
assign mask_o = mask_q;
assign prng_en = (state_q == STATE_PRIME) ||
((state_q == STATE_RUN) && (block_idx_q != LAST_BLOCK_IDX));
`ifndef SYNTHESIS
initial begin
if (HASH_BITS <= 0) $fatal(1, "HASH_BITS must be > 0");
if (PROB_BITS <= 0) $fatal(1, "PROB_BITS must be > 0");
if (PRNG_WORDS <= 0) $fatal(1, "PRNG_WORDS must be > 0");
if (BITS_PER_CYCLE <= 0) $fatal(1, "BITS_PER_CYCLE must be > 0");
if (HASH_BITS % BITS_PER_CYCLE != 0) $fatal(1, "HASH_BITS must be divisible by BITS_PER_CYCLE");
if (RANDOM_BITS != BITS_PER_CYCLE * PROB_BITS) $fatal(1, "PRNG_WORDS*128 must equal BITS_PER_CYCLE*PROB_BITS");
if (SEED == 64'h0) $fatal(1, "SEED must be non-zero");
for (int w = 0; w < PRNG_WORDS; w++) begin
if ({SEED, SEED ^ (64'(w+1))} == 128'h0) $fatal(1, "PRNG seed must be non-zero");
for (int w2 = w+1; w2 < PRNG_WORDS; w2++) begin
if ({SEED, SEED ^ (64'(w+1))} == {SEED, SEED ^ (64'(w2+1))})
$fatal(1, "PRNG seeds must differ");
end
end
end
`endif
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
state_q <= STATE_IDLE;
block_idx_q <= '0;
threshold_q <= '0;
mask_q <= '0;
done_o <= 1'b0;
end else begin
done_o <= 1'b0;
unique case (state_q)
STATE_IDLE: begin
block_idx_q <= '0;
if (start_i) begin
threshold_q <= threshold_i;
mask_q <= '0;
state_q <= STATE_PRIME;
end
end
STATE_PRIME: begin
state_q <= STATE_RUN;
end
STATE_RUN: begin
mask_q[block_idx_q * BITS_PER_CYCLE +: BITS_PER_CYCLE] <= mask_slice;
if (block_idx_q == LAST_BLOCK_IDX) begin
state_q <= STATE_IDLE;
block_idx_q <= '0;
done_o <= 1'b1;
end else begin
block_idx_q <= block_idx_q + 1'b1;
end
end
default: begin
state_q <= STATE_IDLE;
block_idx_q <= '0;
end
endcase
end
end
generate
for (genvar w = 0; w < PRNG_WORDS; w++) begin : gen_prng
localparam logic [127:0] PRNG_SEED = {SEED, SEED ^ (64'(w+1))};
random128 prng_inst (
.clk (clk),
.rst_n (rst_n),
.enable (prng_en),
.seed (PRNG_SEED),
.out (random_bits[w*128 +: 128])
);
end
endgenerate
generate
for (genvar i = 0; i < BITS_PER_CYCLE; i++) begin : gen_bernoulli
assign mask_slice[i] = (random_bits[i*PROB_BITS +: PROB_BITS] < threshold_q);
end
endgenerate
endmodule

View File

@@ -0,0 +1,57 @@
`timescale 1ns / 1ps
module random128 (
input logic clk,
input logic rst_n,
input logic enable,
input logic [127:0] seed,
output logic [127:0] out
);
// xorshift128:
// state = {x, y, z, w}
//
// t = x ^ (x << 11)
// x = y
// y = z
// z = w
// w = w ^ (w >> 19) ^ t ^ (t >> 8)
//
// 注意seed 不能为全 0否则会永久输出 0。
function automatic logic [127:0] xorshift128(input logic [127:0] state);
logic [31:0] x;
logic [31:0] y;
logic [31:0] z;
logic [31:0] w;
logic [31:0] t;
logic [31:0] next_x;
logic [31:0] next_y;
logic [31:0] next_z;
logic [31:0] next_w;
begin
x = state[127:96];
y = state[95:64];
z = state[63:32];
w = state[31:0];
t = x ^ (x << 11);
next_x = y;
next_y = z;
next_z = w;
next_w = w ^ (w >> 19) ^ t ^ (t >> 8);
xorshift128 = {next_x, next_y, next_z, next_w};
end
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
out <= seed;
end else if (enable) begin
out <= xorshift128(out);
end
end
endmodule

28
hw/rtl/random/random32.sv Normal file
View File

@@ -0,0 +1,28 @@
`timescale 1ns / 1ps
module random32 (
input logic clk,
input logic rst_n,
input logic enable,
input logic [31:0] seed,
output logic [31:0] out
);
function automatic logic [31:0] xorshift32(input logic [31:0] x);
logic [31:0] s;
s = x;
s ^= s << 13;
s ^= s >> 17;
s ^= s << 5;
return s;
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
out <= seed;
end else if (enable) begin
out <= xorshift32(out);
end
end
endmodule

28
hw/rtl/random/random64.sv Normal file
View File

@@ -0,0 +1,28 @@
`timescale 1ns / 1ps
module random64 (
input logic clk,
input logic rst_n,
input logic enable,
input logic [63:0] seed,
output logic [63:0] out
);
function automatic logic [63:0] xorshift64(input logic [63:0] x);
logic [63:0] s;
s = x;
s ^= s << 13;
s ^= s >> 7;
s ^= s << 17;
return s;
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
out <= seed;
end else if (enable) begin
out <= xorshift64(out);
end
end
endmodule

58
hw/sim/Makefile Normal file
View File

@@ -0,0 +1,58 @@
PYTHON ?= python
MODULE_TESTS := cam_core_banked candidate_fifo match_engine_pipeline cam_write_noise cam_read_noise popcount_pipeline topk_tracker result_serializer
TOP_CONFIGS := no_noise write_noise read_noise
.PHONY: help test-all test-top test-top-all test-modules test-module test-model test-perf clean $(MODULE_TESTS:%=test-module-%) $(TOP_CONFIGS:%=test-top-%) test-benchmark-retrieval
help:
@echo "Available hw/sim targets:"
@echo " make test-model"
@echo " make test-top # 默认运行所有顶层配置"
@echo " make test-top-all # 运行所有顶层配置"
@echo " make test-top-no_noise # 无噪声集成测试"
@echo " make test-top-write_noise # 写入噪声集成测试"
@echo " make test-top-read_noise # 读取路径 pass-through 集成测试"
@echo " make test-module MODULE=cam_core_banked"
@echo " make test-modules"
@echo " make test-perf"
@echo " make test-benchmark-retrieval # 检索质量 benchmark非默认"
@echo " make test-all"
@echo " make clean"
test-all: test-model test-top-all test-modules
test-top: test-top-all
test-top-all: $(TOP_CONFIGS:%=test-top-%)
$(TOP_CONFIGS:%=test-top-%):
$(MAKE) -C tests/top/$(@:test-top-%=%)
test-modules: $(MODULE_TESTS:%=test-module-%)
test-module:
@test -n "$(MODULE)" || (echo "Usage: make test-module MODULE=<name>"; exit 2)
$(MAKE) -C tests/modules/$(MODULE)
$(MODULE_TESTS:%=test-module-%):
$(MAKE) -C tests/modules/$(@:test-module-%=%)
test-model:
env -u PYTHONPATH -u PYTHONHOME $(PYTHON) -m pytest tests/model -q
test-perf:
$(MAKE) -C tests/perf
test-benchmark-retrieval:
$(MAKE) -C benchmarks/retrieval
clean:
@for config in $(TOP_CONFIGS); do \
$(MAKE) -C tests/top/$$config clean || exit $$?; \
done
@for module in $(MODULE_TESTS); do \
$(MAKE) -C tests/modules/$$module clean || exit $$?; \
done
$(MAKE) -C benchmarks/retrieval clean
$(MAKE) -C tests/perf clean
rm -rf .pytest_cache tests/model/.pytest_cache

View File

@@ -0,0 +1,16 @@
SIM_ROOT := $(abspath ../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_top
COCOTB_TEST_MODULES := benchmarks.retrieval.test_retrieval_benchmark
VERILOG_SOURCES := $(RTL_CAM_TOP)
TOPK_K ?= 5
NUM_ROWS ?= 4096
WRITE_NOISE_EN ?= 0
CAM_RETRIEVAL_DATASET ?=
export CAM_RETRIEVAL_DATASET
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

View File

@@ -0,0 +1,523 @@
from __future__ import annotations
import csv
import json
import os
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
import cocotb
import numpy as np
from cocotb.clock import Clock
from cocotb.utils import get_sim_time
from model.ref_model import (
match_topk,
match_topk_from_scores,
)
from tests.top.utils import (
dut_hash_bits,
dut_lanes,
dut_num_rows,
get_param,
query_topk_once_with_latency,
reset_dut,
write_rows,
)
MAX_BENCHMARK_QUERIES = 128
DEFAULT_POSITIVES_PER_CLASS = 8
DEFAULT_QUERIES_PER_CLASS = 2
DEFAULT_ROW_FLIP_BITS = 16
DEFAULT_QUERY_FLIP_BITS = 16
DEFAULT_SEED = 20260522
BENCHMARK_KS = (1, 5)
SIM_CLOCK_PERIOD_NS = 10
@dataclass(frozen=True)
class RetrievalDataset:
rows: list[int]
row_labels: list[int]
queries: list[int]
query_labels: list[int]
num_classes: int
positives_per_class: int
queries_per_class: int
seed: int
@dataclass(frozen=True)
class QueryTiming:
accept_to_first_result_cycles: int
accept_to_last_result_cycles: int
total_query_cycles: int
@dataclass(frozen=True)
class MetricAccumulator:
precision_sum: float = 0.0
recall_sum: float = 0.0
f1_sum: float = 0.0
exact_matches: int = 0
label_hits: int = 0
count: int = 0
def add(self, precision: float, recall: float, f1: float, label_hit: bool, exact: bool) -> "MetricAccumulator":
return MetricAccumulator(
precision_sum=self.precision_sum + precision,
recall_sum=self.recall_sum + recall,
f1_sum=self.f1_sum + f1,
exact_matches=self.exact_matches + int(exact),
label_hits=self.label_hits + int(label_hit),
count=self.count + 1,
)
def as_dict(self) -> dict[str, float]:
if self.count == 0:
return {
"macro_precision": 0.0,
"retrieval_recall": 0.0,
"macro_f1": 0.0,
"exact_match_rate": 0.0,
"recall@k": 0.0,
}
return {
"macro_precision": self.precision_sum / self.count,
"retrieval_recall": self.recall_sum / self.count,
"macro_f1": self.f1_sum / self.count,
"exact_match_rate": self.exact_matches / self.count,
"recall@k": self.label_hits / self.count,
}
def _project_root() -> Path:
return Path(__file__).resolve().parents[4]
def _flip_exact_bits(rng: np.random.Generator, width: int, n_bits: int) -> int:
n_bits = max(0, min(int(n_bits), int(width)))
if n_bits == 0:
return 0
positions = rng.choice(width, size=n_bits, replace=False)
mask = 0
for pos in positions:
mask |= 1 << int(pos)
return mask
def words_le_to_int(words: np.ndarray) -> int:
value = 0
for idx, word in enumerate(words.tolist()):
value |= int(word) << (64 * idx)
return value
def load_retrieval_dataset_npz(path: str | os.PathLike[str]) -> RetrievalDataset:
dataset_path = Path(path)
if not dataset_path.is_absolute():
dataset_path = _project_root() / dataset_path
if not dataset_path.exists():
raise AssertionError(f"CAM_RETRIEVAL_DATASET not found: {dataset_path}")
loaded = np.load(dataset_path)
rows = [words_le_to_int(words) for words in loaded["rows_words"]]
queries = [words_le_to_int(words) for words in loaded["queries_words"]]
row_labels = [int(x) for x in loaded["row_labels"].tolist()]
query_labels = [int(x) for x in loaded["query_labels"].tolist()]
return RetrievalDataset(
rows=rows,
row_labels=row_labels,
queries=queries,
query_labels=query_labels,
num_classes=len(set(row_labels)),
positives_per_class=0,
queries_per_class=0,
seed=0,
)
def make_clustered_dataset(
*,
num_rows: int,
hash_bits: int,
positives_per_class: int = DEFAULT_POSITIVES_PER_CLASS,
queries_per_class: int = DEFAULT_QUERIES_PER_CLASS,
row_flip_bits: int = DEFAULT_ROW_FLIP_BITS,
query_flip_bits: int = DEFAULT_QUERY_FLIP_BITS,
seed: int = DEFAULT_SEED,
) -> RetrievalDataset:
usable_rows = int(num_rows)
if usable_rows < 5:
raise AssertionError("Retrieval benchmark requires at least 5 CAM rows")
positives_per_class = min(positives_per_class, usable_rows)
num_classes = max(1, usable_rows // positives_per_class)
usable_rows = num_classes * positives_per_class
# Cap total queries to keep simulation runtime bounded
max_queries = min(MAX_BENCHMARK_QUERIES, num_classes * queries_per_class)
if max_queries < num_classes * queries_per_class:
queries_per_class = max(1, max_queries // num_classes)
rng = np.random.default_rng(seed)
mask = (1 << hash_bits) - 1
words = (hash_bits + 63) // 64
rows: list[int] = []
row_labels: list[int] = []
queries: list[int] = []
query_labels: list[int] = []
for class_id in range(num_classes):
center = 0
for word in range(words):
center |= int(rng.integers(0, 1 << 64, dtype=np.uint64)) << (64 * word)
center &= mask
for _ in range(positives_per_class):
rows.append((center ^ _flip_exact_bits(rng, hash_bits, row_flip_bits)) & mask)
row_labels.append(class_id)
for _ in range(queries_per_class):
queries.append((center ^ _flip_exact_bits(rng, hash_bits, query_flip_bits)) & mask)
query_labels.append(class_id)
return RetrievalDataset(
rows=rows,
row_labels=row_labels,
queries=queries,
query_labels=query_labels,
num_classes=num_classes,
positives_per_class=positives_per_class,
queries_per_class=queries_per_class,
seed=seed,
)
def compute_metrics(topk_indices: list[int], row_labels: list[int], query_label: int, k: int) -> tuple[float, float, float]:
retrieved = topk_indices[:k]
relevant = {idx for idx, label in enumerate(row_labels) if label == query_label}
tp = len(set(retrieved) & relevant)
precision = tp / float(k)
recall = tp / float(len(relevant)) if relevant else 0.0
f1 = 0.0 if precision + recall == 0 else (2.0 * precision * recall) / (precision + recall)
return precision, recall, f1
def mode_from_params(write_noise_en: int) -> str:
if write_noise_en:
return "write_noise"
return "no_noise"
def summarize_query_timings(timings: list[QueryTiming]) -> dict[str, float]:
if not timings:
return {
"num_queries": 0,
"query_only_total_cycles": 0,
"query_only_cycles_per_query": 0.0,
"query_only_min_cycles": 0,
"query_only_max_cycles": 0,
"query_only_queries_per_cycle": 0.0,
"total_query_cycles": 0,
"mean_total_query_cycles": 0.0,
"min_total_query_cycles": 0,
"max_total_query_cycles": 0,
"mean_accept_to_first_result_cycles": 0.0,
"mean_accept_to_last_result_cycles": 0.0,
"cycles_per_query": 0.0,
"queries_per_cycle": 0.0,
}
total_cycles = sum(t.accept_to_last_result_cycles for t in timings)
total_first = sum(t.accept_to_first_result_cycles for t in timings)
total_last = sum(t.accept_to_last_result_cycles for t in timings)
count = len(timings)
mean_last = total_last / float(count)
queries_per_cycle = count / float(total_cycles) if total_cycles > 0 else 0.0
return {
"num_queries": count,
"query_only_total_cycles": total_cycles,
"query_only_cycles_per_query": mean_last,
"query_only_min_cycles": min(t.accept_to_last_result_cycles for t in timings),
"query_only_max_cycles": max(t.accept_to_last_result_cycles for t in timings),
"query_only_queries_per_cycle": queries_per_cycle,
# Backward-compatible aliases: query-only, not end-to-end.
"total_query_cycles": total_cycles,
"mean_total_query_cycles": total_cycles / float(count),
"min_total_query_cycles": min(t.accept_to_last_result_cycles for t in timings),
"max_total_query_cycles": max(t.accept_to_last_result_cycles for t in timings),
"mean_accept_to_first_result_cycles": total_first / float(count),
"mean_accept_to_last_result_cycles": mean_last,
"cycles_per_query": mean_last,
"queries_per_cycle": queries_per_cycle,
}
def build_hardware_performance(
timings: list[QueryTiming],
*,
load_write_noise_cycles: int,
end_to_end_cycles: int,
) -> dict[str, float]:
performance = summarize_query_timings(timings)
num_queries = int(performance["num_queries"])
performance.update({
"load_write_noise_cycles": int(load_write_noise_cycles),
"end_to_end_cycles": int(end_to_end_cycles),
"end_to_end_cycles_per_query": (
float(end_to_end_cycles) / float(num_queries) if num_queries > 0 else 0.0
),
"end_to_end_queries_per_cycle": (
float(num_queries) / float(end_to_end_cycles) if end_to_end_cycles > 0 else 0.0
),
})
return performance
def current_sim_cycle() -> int:
"""Return the current benchmark clock cycle from simulator time."""
return int(get_sim_time("ns") // SIM_CLOCK_PERIOD_NS)
def output_dir_for(mode: str) -> Path:
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID")
if not run_id:
run_id = f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}"
out_dir = _project_root() / "outputs" / "cam_retrieval_benchmark" / run_id
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "logs").mkdir(exist_ok=True)
return out_dir
def write_outputs(out_dir: Path, result: dict) -> None:
metrics_json = out_dir / "metrics.json"
metrics_csv = out_dir / "metrics.csv"
summary_md = out_dir / "summary.md"
metrics_json.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8")
fieldnames = [
"run_id", "mode", "num_rows", "hash_bits", "lanes", "topk_k",
"write_noise_en", "write_noise_rate_num",
"write_noise_rate_den",
"num_queries", "k", "macro_precision", "retrieval_recall", "macro_f1",
"recall@k", "exact_match_rate",
"query_only_cycles_per_query", "query_only_total_cycles",
"query_only_queries_per_cycle", "load_write_noise_cycles",
"end_to_end_cycles", "end_to_end_cycles_per_query",
"end_to_end_queries_per_cycle", "cycles_per_query",
"mean_accept_to_first_result_cycles", "mean_accept_to_last_result_cycles",
"mean_total_query_cycles", "total_query_cycles", "queries_per_cycle", "status",
]
with metrics_csv.open("w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for k, metrics in result["metrics"].items():
row = {
"run_id": result["run_id"],
"mode": result["mode"],
"num_rows": result["params"]["num_rows"],
"hash_bits": result["params"]["hash_bits"],
"lanes": result["params"]["lanes"],
"topk_k": result["params"]["topk_k"],
"write_noise_en": result["params"]["write_noise_en"],
"write_noise_rate_num": result["params"]["write_noise_rate_num"],
"write_noise_rate_den": result["params"]["write_noise_rate_den"],
"num_queries": result["dataset"]["num_queries"],
"k": int(k),
"macro_precision": metrics["macro_precision"],
"retrieval_recall": metrics["retrieval_recall"],
"macro_f1": metrics["macro_f1"],
"recall@k": metrics["recall@k"],
"exact_match_rate": metrics["exact_match_rate"],
"query_only_cycles_per_query": result.get("performance", {}).get("query_only_cycles_per_query", ""),
"query_only_total_cycles": result.get("performance", {}).get("query_only_total_cycles", ""),
"query_only_queries_per_cycle": result.get("performance", {}).get("query_only_queries_per_cycle", ""),
"load_write_noise_cycles": result.get("performance", {}).get("load_write_noise_cycles", ""),
"end_to_end_cycles": result.get("performance", {}).get("end_to_end_cycles", ""),
"end_to_end_cycles_per_query": result.get("performance", {}).get("end_to_end_cycles_per_query", ""),
"end_to_end_queries_per_cycle": result.get("performance", {}).get("end_to_end_queries_per_cycle", ""),
"cycles_per_query": result.get("performance", {}).get("cycles_per_query", ""),
"mean_accept_to_first_result_cycles": result.get("performance", {}).get(
"mean_accept_to_first_result_cycles", "",
),
"mean_accept_to_last_result_cycles": result.get("performance", {}).get(
"mean_accept_to_last_result_cycles", "",
),
"mean_total_query_cycles": result.get("performance", {}).get("mean_total_query_cycles", ""),
"total_query_cycles": result.get("performance", {}).get("total_query_cycles", ""),
"queries_per_cycle": result.get("performance", {}).get("queries_per_cycle", ""),
"status": result["status"],
}
writer.writerow(row)
lines = [
"# CAM Retrieval Benchmark Summary",
"",
f"- run_id: `{result['run_id']}`",
f"- mode: `{result['mode']}`",
f"- status: `{result['status']}`",
f"- num_queries: `{result['dataset']['num_queries']}`",
"",
"## Hardware performance",
"",
f"- query-only cycles/query: `{result.get('performance', {}).get('query_only_cycles_per_query', '')}`",
f"- query-only total cycles: `{result.get('performance', {}).get('query_only_total_cycles', '')}`",
f"- query-only queries/cycle: `{result.get('performance', {}).get('query_only_queries_per_cycle', '')}`",
f"- load/write/noise cycles: `{result.get('performance', {}).get('load_write_noise_cycles', '')}`",
f"- end-to-end cycles: `{result.get('performance', {}).get('end_to_end_cycles', '')}`",
f"- end-to-end cycles/query: `{result.get('performance', {}).get('end_to_end_cycles_per_query', '')}`",
f"- end-to-end queries/cycle: `{result.get('performance', {}).get('end_to_end_queries_per_cycle', '')}`",
f"- cycles_per_query (compat, query-only): `{result.get('performance', {}).get('cycles_per_query', '')}`",
f"- accept_to_first_result_cycles: `{result.get('performance', {}).get('mean_accept_to_first_result_cycles', '')}`",
f"- accept_to_last_result_cycles: `{result.get('performance', {}).get('mean_accept_to_last_result_cycles', '')}`",
f"- total_query_cycles (compat, query-only): `{result.get('performance', {}).get('total_query_cycles', '')}`",
f"- queries_per_cycle (compat, query-only): `{result.get('performance', {}).get('queries_per_cycle', '')}`",
"",
"## Retrieval quality",
"",
"| k | macro_precision | retrieval_recall | macro_f1 | recall@k | exact_match_rate |",
"|---:|---:|---:|---:|---:|---:|",
]
for k, metrics in result["metrics"].items():
lines.append(
f"| {k} | {metrics['macro_precision']:.6f} | {metrics['retrieval_recall']:.6f} | "
f"{metrics['macro_f1']:.6f} | {metrics['recall@k']:.6f} | {metrics['exact_match_rate']:.6f} |"
)
lines.extend([
"",
"说明:结果来自 Verilator/Cocotb 仿真,不是 FPGA 板上实测。",
])
summary_md.write_text("\n".join(lines) + "\n", encoding="utf-8")
@cocotb.test()
async def cam_retrieval_benchmark(dut):
cocotb.start_soon(Clock(dut.clk, SIM_CLOCK_PERIOD_NS, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
lanes = dut_lanes(dut)
write_noise_en = int(get_param(dut, "WRITE_NOISE_EN", 0) or 0)
write_noise_rate_num = int(get_param(dut, "WRITE_NOISE_RATE_NUM", 0) or 0)
write_noise_rate_den = int(get_param(dut, "WRITE_NOISE_RATE_DEN", 100) or 100)
mode = mode_from_params(write_noise_en)
if num_rows % lanes != 0:
raise AssertionError("Retrieval benchmark requires NUM_ROWS divisible by LANES")
dataset_path = os.environ.get("CAM_RETRIEVAL_DATASET")
if not dataset_path:
raise AssertionError("CAM_RETRIEVAL_DATASET is required; run scripts/prepare_cam_retrieval_dataset.py first")
dataset = load_retrieval_dataset_npz(dataset_path)
if len(dataset.rows) != num_rows:
raise AssertionError(f"artifact row count {len(dataset.rows)} must equal DUT NUM_ROWS {num_rows}")
benchmark_start_cycle = current_sim_cycle()
load_write_noise_start_cycle = current_sim_cycle()
await write_rows(dut, dataset.rows)
load_write_noise_cycles = current_sim_cycle() - load_write_noise_start_cycle
accumulators = {k: MetricAccumulator() for k in BENCHMARK_KS}
timings: list[QueryTiming] = []
for query, query_label in zip(dataset.queries, dataset.query_labels):
beats, _, _, _, timing = await query_topk_once_with_latency(dut, query)
timings.append(QueryTiming(**timing))
if len(beats) < max(BENCHMARK_KS):
raise AssertionError(f"Expected at least {max(BENCHMARK_KS)} Top-K beats, got {len(beats)}")
dut_topk = [int(beat[1]) for beat in beats[: max(BENCHMARK_KS)]]
golden_topk, _ = match_topk(query, dataset.rows, width=hash_bits, k=max(BENCHMARK_KS))
for k in BENCHMARK_KS:
precision, recall, f1 = compute_metrics(dut_topk, dataset.row_labels, query_label, k)
exact = dut_topk[:k] == golden_topk[:k]
retrieved_labels = [dataset.row_labels[idx] for idx in dut_topk[:k]]
label_hit = query_label in retrieved_labels
accumulators[k] = accumulators[k].add(precision, recall, f1, label_hit, exact)
end_to_end_cycles = current_sim_cycle() - benchmark_start_cycle
run_id = os.environ.get("CAM_RETRIEVAL_RUN_ID") or f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{mode}"
result = {
"run_id": run_id,
"mode": mode,
"status": "pass",
"params": {
"num_rows": len(dataset.rows),
"hash_bits": hash_bits,
"lanes": lanes,
"topk_k": max(BENCHMARK_KS),
"write_noise_en": write_noise_en,
"write_noise_rate_num": write_noise_rate_num,
"write_noise_rate_den": write_noise_rate_den,
},
"dataset": {
"num_classes": dataset.num_classes,
"positives_per_class": dataset.positives_per_class,
"queries_per_class": dataset.queries_per_class,
"num_queries": len(dataset.queries),
"seed": dataset.seed,
},
"metrics": {str(k): accumulators[k].as_dict() for k in BENCHMARK_KS},
"performance": build_hardware_performance(
timings,
load_write_noise_cycles=load_write_noise_cycles,
end_to_end_cycles=end_to_end_cycles,
),
}
out_dir = output_dir_for(mode)
write_outputs(out_dir, result)
for k in BENCHMARK_KS:
metrics = result["metrics"][str(k)]
dut._log.info(
"RETRIEVAL_RESULT mode=%s k=%d precision=%.6f retrieval_recall=%.6f f1=%.6f recall_at_k=%.6f exact_match=%.6f output_dir=%s",
mode, k, metrics["macro_precision"], metrics["retrieval_recall"],
metrics["macro_f1"], metrics["recall@k"], metrics["exact_match_rate"],
str(out_dir.relative_to(_project_root())),
)
performance = result["performance"]
dut._log.info(
"RETRIEVAL_PERF_RESULT mode=%s num_queries=%d query_only_cycles_per_query=%.6f "
"query_only_total_cycles=%d query_only_queries_per_cycle=%.9f "
"load_write_noise_cycles=%d end_to_end_cycles=%d "
"end_to_end_cycles_per_query=%.6f end_to_end_queries_per_cycle=%.9f "
"cycles_per_query=%.6f "
"accept_to_first_result_cycles=%.6f accept_to_last_result_cycles=%.6f "
"total_query_cycles=%d queries_per_cycle=%.9f status=pass output_dir=%s",
mode,
performance["num_queries"],
performance["query_only_cycles_per_query"],
performance["query_only_total_cycles"],
performance["query_only_queries_per_cycle"],
performance["load_write_noise_cycles"],
performance["end_to_end_cycles"],
performance["end_to_end_cycles_per_query"],
performance["end_to_end_queries_per_cycle"],
performance["cycles_per_query"],
performance["mean_accept_to_first_result_cycles"],
performance["mean_accept_to_last_result_cycles"],
performance["total_query_cycles"],
performance["queries_per_cycle"],
str(out_dir.relative_to(_project_root())),
)
if write_noise_en == 0:
assert result["metrics"]["5"]["exact_match_rate"] == 1.0, (
f"Expected perfect exact match with no noise, got "
f"{result['metrics']['5']['exact_match_rate']}"
)
else:
dut._log.info(
"Noise enabled (WRITE_NOISE_RATE=%d/%d) — exact_match assertion skipped",
write_noise_rate_num, write_noise_rate_den,
)

View File

@@ -0,0 +1,63 @@
ifndef SIM_ROOT
$(error SIM_ROOT must be set before including mk/cocotb-common.mk)
endif
ifndef RTL_ROOT
$(error RTL_ROOT must be set before including mk/cocotb-common.mk)
endif
ifndef TOPLEVEL
$(error TOPLEVEL must be set before including mk/cocotb-common.mk)
endif
ifndef COCOTB_TEST_MODULES
$(error COCOTB_TEST_MODULES must be set before including mk/cocotb-common.mk)
endif
ifndef VERILOG_SOURCES
$(error VERILOG_SOURCES must be set before including mk/cocotb-common.mk)
endif
SIM ?= verilator
TOPLEVEL_LANG ?= verilog
NUM_ROWS ?= 4096
HASH_BITS ?= 512
LANES ?= 8
TOPK_K ?= 4
EXTRA_ARGS += +define+NUM_ROWS=$(NUM_ROWS) +define+HASH_BITS=$(HASH_BITS) +define+LANES=$(LANES) +define+TOPK_K=$(TOPK_K)
EXTRA_ARGS += --trace --trace-fst --trace-structs
COMPILE_ARGS += -Wall -Wno-fatal
COMPILE_ARGS += -I$(RTL_ROOT) -I$(RTL_ROOT)/core -I$(RTL_ROOT)/noise -I$(RTL_ROOT)/random
COMPILE_ARGS += +define+SIM_DEBUG
COMPILE_ARGS += $(EXTRA_DEFINES)
WRITE_NOISE_EN ?= $(NOISE_EN)
WRITE_NOISE_RATE_NUM ?= $(NOISE_RATE_NUM)
WRITE_NOISE_RATE_DEN ?= $(NOISE_RATE_DEN)
ifneq ($(strip $(WRITE_NOISE_EN)),)
COMPILE_ARGS += -GWRITE_NOISE_EN=$(WRITE_NOISE_EN)
endif
ifneq ($(strip $(WRITE_NOISE_RATE_NUM)),)
COMPILE_ARGS += -GWRITE_NOISE_RATE_NUM=$(WRITE_NOISE_RATE_NUM)
endif
ifneq ($(strip $(WRITE_NOISE_RATE_DEN)),)
COMPILE_ARGS += -GWRITE_NOISE_RATE_DEN=$(WRITE_NOISE_RATE_DEN)
endif
export PYTHONPATH := $(SIM_ROOT):$(PYTHONPATH)
export QUIET ?= 1
export VERBOSE ?= 0
export COCOTB_LOG_LEVEL ?= INFO
export PYTHONWARNINGS ?= ignore::pytest.PytestDeprecationWarning
SUPPRESS_VERILATOR_WARNINGS ?= 0
ifeq ($(SUPPRESS_VERILATOR_WARNINGS),1)
COMPILE_ARGS += -Wno-WIDTHEXPAND
COMPILE_ARGS += -Wno-UNOPTFLAT
endif
include $(shell cocotb-config --makefiles)/Makefile.sim

26
hw/sim/mk/rtl-sources.mk Normal file
View File

@@ -0,0 +1,26 @@
ifndef RTL_ROOT
$(error RTL_ROOT must be set before including mk/rtl-sources.mk)
endif
RTL_RANDOM := $(RTL_ROOT)/random/random128.sv
RTL_BERNOULLI_NOISE_MASK := $(RTL_ROOT)/noise/noise_mask_bernoulli.sv $(RTL_RANDOM)
RTL_WRITE_NOISE := $(RTL_BERNOULLI_NOISE_MASK) $(RTL_ROOT)/noise/cam_write_noise.sv
RTL_READ_NOISE := $(RTL_ROOT)/noise/cam_read_noise.sv
RTL_CAM_CORE_BANKED := $(RTL_ROOT)/core/cam_core_banked.sv
RTL_MATCH_ENGINE := \
$(RTL_ROOT)/core/popcount_pipeline.sv \
$(RTL_ROOT)/core/candidate_fifo.sv \
$(RTL_ROOT)/core/topk_tracker.sv \
$(RTL_ROOT)/core/result_serializer.sv \
$(RTL_ROOT)/core/match_engine_pipeline.sv
RTL_CAM_NOISY := \
$(RTL_CAM_CORE_BANKED) \
$(RTL_WRITE_NOISE) \
$(RTL_READ_NOISE) \
$(RTL_MATCH_ENGINE) \
$(RTL_ROOT)/cam_noisy.sv
RTL_CAM_TOP := $(RTL_CAM_NOISY) $(RTL_ROOT)/cam_top.sv

134
hw/sim/model/ref_model.py Normal file
View File

@@ -0,0 +1,134 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Sequence
import numpy as np
@dataclass(frozen=True)
class MatchResult:
top1_index: int
top1_score: int
scores: np.ndarray
def popcount_int(x: int) -> int:
return int(x.bit_count())
def mask_width(width: int) -> int:
return (1 << width) - 1
def xnor_popcount_score(query: int, stored: int, width: int = 512) -> int:
same_bits = ~(query ^ stored) & mask_width(width)
return popcount_int(same_bits)
def match_top1(
query: int,
rows: Sequence[int],
*,
width: int = 512,
) -> MatchResult:
"""Pure matching — noise is already baked into rows at write time."""
scores = np.zeros(len(rows), dtype=np.int32)
best_index = 0
best_score = -1
for idx, row in enumerate(rows):
score = xnor_popcount_score(int(query), int(row), width)
scores[idx] = score
# Tie-break: choose the smallest row index.
if score > best_score:
best_score = score
best_index = idx
return MatchResult(
top1_index=int(best_index),
top1_score=int(best_score),
scores=scores,
)
def match_topk_from_scores(scores: Sequence[int], k: int) -> list[int]:
"""Return row indices sorted by score desc, row index asc (HW tie-break)."""
if k <= 0:
raise ValueError("k must be greater than 0")
return sorted(range(len(scores)), key=lambda idx: (-int(scores[idx]), idx))[: min(k, len(scores))]
def match_topk(
query: int,
rows: Sequence[int],
*,
width: int = 512,
k: int = 5,
) -> tuple[list[int], np.ndarray]:
"""Pure Top-K matching — noise is already baked into rows if needed.
Returns (list of row indices in rank order, NumPy score array).
"""
scores = np.zeros(len(rows), dtype=np.int32)
for idx, row in enumerate(rows):
scores[idx] = xnor_popcount_score(int(query), int(row), width)
return match_topk_from_scores(scores, k), scores
def xorshift128(state: int) -> int:
"""128-bit xorshift PRNG, single step. Matches random128.sv."""
mask32 = (1 << 32) - 1
mask128 = (1 << 128) - 1
s = state & mask128
x = (s >> 96) & mask32
y = (s >> 64) & mask32
z = (s >> 32) & mask32
w = s & mask32
t = (x ^ ((x << 11) & mask32)) & mask32
next_x = y
next_y = z
next_z = w
next_w = (w ^ (w >> 19) ^ t ^ (t >> 8)) & mask32
return ((next_x << 96) | (next_y << 64) | (next_z << 32) | next_w) & mask128
def random_hashes(
rng: np.random.Generator,
n: int,
*,
width: int = 512,
) -> list[int]:
words = (width + 63) // 64
out: list[int] = []
for _ in range(n):
value = 0
for w in range(words):
value |= int(rng.integers(0, 1 << 64, dtype=np.uint64)) << (64 * w)
out.append(value & mask_width(width))
return out
def unpack_score_debug_flat(flat: int, num_rows: int, score_bits: int) -> np.ndarray:
mask = (1 << score_bits) - 1
return np.array(
[(int(flat) >> (row * score_bits)) & mask for row in range(num_rows)],
dtype=np.int32,
)
def split_hash_to_words_le(value: int, *, width: int = 512, word_bits: int = 32) -> list[int]:
n_words = width // word_bits
word_mask = (1 << word_bits) - 1
return [(int(value) >> (word_bits * i)) & word_mask for i in range(n_words)]
def join_hash_words_le(words: Sequence[int], *, word_bits: int = 32) -> int:
value = 0
word_mask = (1 << word_bits) - 1
for i, word in enumerate(words):
value |= (int(word) & word_mask) << (word_bits * i)
return value

View File

@@ -0,0 +1,100 @@
"""Sweep write-noise rates and measure top-1 stability.
Applies write-noise flip masks to stored rows (simulating noisy writes),
then queries the noisy rows and compares top-1 results against clean rows.
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
SIM_ROOT = Path(__file__).resolve().parents[1]
if str(SIM_ROOT) not in sys.path:
sys.path.insert(0, str(SIM_ROOT))
import numpy as np
from model.ref_model import (
match_top1,
random_hashes,
)
def apply_write_noise(
rows: list[int],
*,
width: int,
rate_num: int,
rate_den: int,
noise_bits: int = 8,
seed: int = 0,
) -> list[int]:
"""No-op: write-noise flip masks are now generated by Bernoulli RTL only.
The sweep now measures top-1 stability of pure matching over queries,
since noise is applied at RTL write time, not in the Python model.
"""
return list(rows)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--rows", type=int, default=512)
parser.add_argument("--queries", type=int, default=128)
parser.add_argument("--width", type=int, default=512)
parser.add_argument("--seed", type=int, default=1234)
parser.add_argument("--noise-bits", type=int, default=8)
parser.add_argument(
"--rates",
type=float,
nargs="+",
default=[0.0, 0.001, 0.005, 0.01, 0.02, 0.05],
)
args = parser.parse_args()
rng = np.random.default_rng(args.seed)
rows = random_hashes(rng, args.rows, width=args.width)
# Construct simple positive queries by selecting existing rows.
query_indices = rng.integers(0, args.rows, size=args.queries)
queries = [rows[int(i)] for i in query_indices]
clean_results = [match_top1(q, rows, width=args.width) for q in queries]
# Use a fixed denominator matching the 8-bit sample space (2^8 = 256).
# Note: floor() is used, matching RTL threshold = (rate_num * 256) // rate_den.
# Rates below 1/256 (≈0.39%) collapse to zero under this scheme.
rate_den = 256
print("rate,rate_num,effective_prob,top1_stability,avg_clean_margin")
for rate in args.rates:
rate_num = int(rate * rate_den)
effective = rate_num / rate_den if rate_den > 0 else 0.0
stable = 0
margins = []
noisy_rows = apply_write_noise(
rows,
width=args.width,
rate_num=rate_num,
rate_den=rate_den,
noise_bits=args.noise_bits,
seed=args.seed,
)
for q, clean in zip(queries, clean_results):
noisy = match_top1(q, noisy_rows, width=args.width)
if noisy.top1_index == clean.top1_index:
stable += 1
sorted_scores = np.sort(clean.scores)
margin = int(sorted_scores[-1] - sorted_scores[-2])
margins.append(margin)
print(f"{rate},{rate_num},{effective:.6f},{stable / args.queries:.6f},{np.mean(margins):.3f}")
if __name__ == "__main__":
main()

0
hw/sim/tests/__init__.py Normal file
View File

View File

View File

@@ -0,0 +1,102 @@
# -*- coding: utf-8 -*-
"""
参考模型ref_model的纯 Python 单元测试 — Phase 2 cleaned.
本文件不涉及任何 RTL / Verilator 仿真,仅验证 Python 参考模型的正确性。
Phase 2 后只保留 pure matching 函数;所有 grouped/read-noise helpers 已删除。
测试覆盖:
1. XNOR 评分语义 — 确认是「匹配位数」而非「汉明距离」
2. Top-1 matching — 纯匹配,正确选出最高分索引
3. Top-K matching — 返回排序后的行索引列表
4. Top-K 排序规则 — 分数降序、平局行号升序
5. Top-K k 超过行数时 clamp
"""
from __future__ import annotations
from model.ref_model import (
match_top1,
match_topk,
match_topk_from_scores,
xnor_popcount_score,
)
import numpy as np
# ==============================================================================
# 测试 1评分函数语义 — 确认是「XNOR 匹配位数」而非「汉明距离」
# ==============================================================================
def test_score_is_bit_match_popcount_not_hamming_distance():
"""
关键语义验证xnor_popcount_score 计算的是匹配位的数量,不是汉明距离。
示例:
query = 0b1010
stored = 0b1000
XNOR = 0b1101 → popcount = 3有 3 个位匹配)
汉明距离 = 1 → 只有一个位不同
为什么这个区分很重要:
- 如果 RTL 或模型错误地使用了汉明距离作为分数,则:
完全匹配的分数会是 0 而非 hash_bits512
Top-K 排序会颠倒(分数低的反而排前面)
- 这会导致整个 CAM 检索系统返回错误结果
"""
query = 0b1010
stored = 0b1000
assert xnor_popcount_score(query, stored, width=4) == 3
# ==============================================================================
# 测试 2Top-1 matching
# ==============================================================================
def test_match_top1_selects_highest_xnor_score_with_row_index_tiebreak():
"""Top-1 应选出 XNOR 分最高的行;平局时选最小行号。"""
rows = [0b0000, 0b1111, 0b0011, 0b0101]
query = 0b0000
result = match_top1(query, rows, width=4)
assert result.top1_index == 0
assert result.top1_score == 4
assert result.scores.tolist() == [4, 0, 2, 2]
# ==============================================================================
# 测试 3Top-K matching
# ==============================================================================
def test_match_topk_scores_rows_by_xnor_popcount():
"""match_topk 通过 xnor_popcount 计算分数,返回排序后的行索引和分数数组。"""
rows = [0b0000, 0b1111, 0b0011, 0b0101]
query = 0b0000
indices, scores = match_topk(query, rows, width=4, k=3)
assert scores.tolist() == [4, 0, 2, 2]
assert indices == [0, 2, 3]
# ==============================================================================
# 测试 4Top-K 排序 — 分数降序、平局行号升序
# ==============================================================================
def test_match_topk_from_scores_uses_score_desc_then_row_asc():
"""Top-K 排序规则:分数越大越优先;分数相同时行号越小越优先。"""
scores = np.array([7, 9, 9, 2, 7], dtype=np.int32)
assert match_topk_from_scores(scores, 4) == [1, 2, 0, 4]
# ==============================================================================
# 测试 5Top-K k 超过行数时 clamp
# ==============================================================================
def test_match_topk_clamps_k_to_row_count():
"""当 k 超过实际行数时,返回所有行(按排序)。"""
indices, scores = match_topk(0, [0, 1], width=1, k=5)
assert scores.tolist() == [1, 0]
assert indices == [0, 1]

View File

View File

@@ -0,0 +1,9 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_core_banked
COCOTB_TEST_MODULES := tests.modules.cam_core_banked.test_cam_core_banked
VERILOG_SOURCES := $(RTL_CAM_CORE_BANKED)
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

@@ -0,0 +1,101 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
async def reset_core(dut):
dut.rst_n.value = 0
dut.wr_valid.value = 0
dut.wr_row.value = 0
dut.wr_hash.value = 0
dut.rd_valid_i.value = 0
dut.rd_base_row_i.value = 0
for _ in range(3):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
def lane_word(value: int, lane: int, width: int) -> int:
return (value >> (lane * width)) & ((1 << width) - 1)
async def write_row(dut, row: int, hash_value: int):
dut.wr_valid.value = 1
dut.wr_row.value = row
dut.wr_hash.value = hash_value
await RisingEdge(dut.clk)
dut.wr_valid.value = 0
async def request_batch(dut, base_row: int):
dut.rd_base_row_i.value = base_row
dut.rd_valid_i.value = 1
await RisingEdge(dut.clk)
dut.rd_valid_i.value = 0
await RisingEdge(dut.clk)
@cocotb.test()
async def banked_core_reads_aligned_eight_row_batch_after_one_cycle(dut):
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_core(dut)
LANES = len(dut.rd_lane_valid_o)
ROW_BITS = len(dut.rd_row_ids_o) // LANES
HASH_BITS = len(dut.rd_hashes_o) // LANES
for row in range(LANES):
dut.wr_valid.value = 1
dut.wr_row.value = row
dut.wr_hash.value = row + 0x100
await RisingEdge(dut.clk)
dut.wr_valid.value = 0
dut.rd_base_row_i.value = 0
dut.rd_valid_i.value = 1
await RisingEdge(dut.clk)
dut.rd_valid_i.value = 0
await RisingEdge(dut.clk)
assert int(dut.rd_valid_o.value) == 1
for lane in range(LANES):
got_row = (int(dut.rd_row_ids_o.value) >> (lane * ROW_BITS)) & ((1 << ROW_BITS) - 1)
got_hash = (int(dut.rd_hashes_o.value) >> (lane * HASH_BITS)) & ((1 << HASH_BITS) - 1)
assert got_row == lane
assert got_hash == lane + 0x100
@cocotb.test()
async def banked_core_keeps_internal_bank_addresses_isolated(dut):
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_core(dut)
LANES = len(dut.rd_lane_valid_o)
ROW_BITS = len(dut.rd_row_ids_o) // LANES
HASH_BITS = len(dut.rd_hashes_o) // LANES
expected_by_row = {}
for bank_addr in range(3):
base_row = bank_addr * LANES
for lane in range(LANES):
row = base_row + lane
hash_value = 0xABC000 + (bank_addr << 8) + lane
expected_by_row[row] = hash_value
await write_row(dut, row, hash_value)
for bank_addr in range(3):
base_row = bank_addr * LANES
await request_batch(dut, base_row)
assert int(dut.rd_valid_o.value) == 1
assert int(dut.rd_lane_valid_o.value) == (1 << LANES) - 1
for lane in range(LANES):
row = base_row + lane
got_row = lane_word(int(dut.rd_row_ids_o.value), lane, ROW_BITS)
got_hash = lane_word(int(dut.rd_hashes_o.value), lane, HASH_BITS)
assert got_row == row
assert got_hash == expected_by_row[row]

View File

@@ -0,0 +1,11 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_read_noise
COCOTB_TEST_MODULES := tests.modules.cam_read_noise.test_cam_read_noise
VERILOG_SOURCES := $(RTL_READ_NOISE)
HASH_BITS ?= 512
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

@@ -0,0 +1,83 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge, Timer
async def reset_read_noise(dut):
dut.rst_n.value = 0
dut.valid_i.value = 0
dut.row_ids_i.value = 0
dut.hashes_i.value = 0
dut.lane_valid_i.value = 0
for _ in range(3):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
@cocotb.test()
async def read_noise_disabled_forwards_hashes_after_one_stage(dut):
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_read_noise(dut)
LANES = len(dut.lane_valid_i)
ROW_BITS = len(dut.row_ids_i) // LANES
HASH_BITS_PER_LANE = len(dut.hashes_i) // LANES
all_lanes_valid = (1 << LANES) - 1
hashes = 0
rows = 0
for lane in range(LANES):
hashes |= (lane + 0x55) << (lane * HASH_BITS_PER_LANE)
rows |= lane << (lane * ROW_BITS)
dut.hashes_i.value = hashes
dut.row_ids_i.value = rows
dut.lane_valid_i.value = all_lanes_valid
dut.valid_i.value = 1
await Timer(1, unit="step")
await RisingEdge(dut.clk) # valid_o ← valid_i=1 internally
await Timer(1, unit="step")
dut.valid_i.value = 0
# One-stage pass-through: valid_o holds the latched value for this cycle
assert int(dut.valid_o.value) == 1
assert int(dut.hashes_noisy_o.value) == hashes
assert int(dut.row_ids_o.value) == rows
assert int(dut.lane_valid_o.value) == all_lanes_valid
@cocotb.test()
async def read_noise_enabled_still_forwards_hashes_unmodified(dut):
"""The read path pass-through forwards hashes unmodified."""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_read_noise(dut)
LANES = len(dut.lane_valid_i)
ROW_BITS = len(dut.row_ids_i) // LANES
HASH_BITS_PER_LANE = len(dut.hashes_i) // LANES
all_lanes_valid = (1 << LANES) - 1
hashes = 0
rows = 0
for lane in range(LANES):
hashes |= (lane + 0x55) << (lane * HASH_BITS_PER_LANE)
rows |= lane << (lane * ROW_BITS)
dut.hashes_i.value = hashes
dut.row_ids_i.value = rows
dut.lane_valid_i.value = all_lanes_valid
dut.valid_i.value = 1
await Timer(1, unit="step")
await RisingEdge(dut.clk) # valid_o ← valid_i=1 internally
await Timer(1, unit="step")
dut.valid_i.value = 0
# One-stage pass-through: valid_o holds latched value from previous cycle
assert int(dut.valid_o.value) == 1
assert int(dut.hashes_noisy_o.value) == hashes
assert int(dut.row_ids_o.value) == rows
assert int(dut.lane_valid_o.value) == all_lanes_valid

View File

@@ -0,0 +1,13 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_write_noise
COCOTB_TEST_MODULES := tests.modules.cam_write_noise.test_cam_write_noise
VERILOG_SOURCES := $(RTL_WRITE_NOISE)
HASH_BITS ?= 512
WRITE_NOISE_EN ?= 1
WRITE_NOISE_RATE_NUM ?= 1
WRITE_NOISE_RATE_DEN ?= 100
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

@@ -0,0 +1,94 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge, Timer
# Bernoulli: 1 PRIME + 16 RUN = 17 cycles internal
# + 1 cycle for mask_start propagation + 1 cycle for core_wr_valid output = 19
DEFAULT_WRITE_NOISE_LATENCY = 19
async def pulse_write(dut, row: int, value: int):
dut.wr_row.value = row
dut.wr_hash.value = value
dut.wr_valid.value = 1
await Timer(1, unit="step")
assert int(dut.wr_ready.value) == 1
await RisingEdge(dut.clk)
await Timer(1, unit="step")
dut.wr_valid.value = 0
async def wait_core_write(dut, max_cycles: int = 128) -> int:
cycles = 0
while int(dut.core_wr_valid.value) == 0:
assert cycles < max_cycles, "timed out waiting for core_wr_valid"
await RisingEdge(dut.clk)
await Timer(1, unit="step")
cycles += 1
return cycles
async def reset_write_noise(dut):
dut.rst_n.value = 0
dut.wr_valid.value = 0
dut.wr_row.value = 0
dut.wr_hash.value = 0
for _ in range(3):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
@cocotb.test()
async def write_noise_enabled_applies_bernoulli_mask_after_generation(dut):
"""Noise active: FSM enters WAIT_MASK, core_wr_hash deterministic across reset."""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_write_noise(dut)
value = (1 << 512) - 1 # all-ones: even low-rate Bernoulli may flip some bits
await pulse_write(dut, row=3, value=value)
await Timer(1, unit="step")
assert int(dut.wr_ready.value) == 0
cycles = await wait_core_write(dut)
assert cycles == DEFAULT_WRITE_NOISE_LATENCY
assert int(dut.core_wr_row.value) == 3
hash_after_first = int(dut.core_wr_hash.value)
await RisingEdge(dut.clk)
await Timer(1, unit="step")
assert int(dut.core_wr_valid.value) == 0
assert int(dut.wr_ready.value) == 1
# Deterministic across reset: same seed → same mask → same noisy hash
await reset_write_noise(dut)
await pulse_write(dut, row=3, value=value)
await wait_core_write(dut)
assert int(dut.core_wr_hash.value) == hash_after_first
@cocotb.test()
async def write_noise_backpressures_second_write_until_done(dut):
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_write_noise(dut)
await pulse_write(dut, row=1, value=0xAA55)
dut.wr_row.value = 2
dut.wr_hash.value = 0x55AA
dut.wr_valid.value = 1
await Timer(1, unit="step")
for _ in range(4):
assert int(dut.wr_ready.value) == 0
assert int(dut.core_wr_valid.value) == 0
await RisingEdge(dut.clk)
await Timer(1, unit="step")
dut.wr_valid.value = 0
cycles = await wait_core_write(dut)
assert cycles == DEFAULT_WRITE_NOISE_LATENCY - 4 # 19 - 4 = 15
assert int(dut.core_wr_row.value) == 1

View File

@@ -0,0 +1,32 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := candidate_fifo
COCOTB_TEST_MODULES := tests.modules.candidate_fifo.test_candidate_fifo
VERILOG_SOURCES := $(RTL_ROOT)/core/candidate_fifo.sv
# Allow override from command line (e.g. make FIFO_DEPTH=10)
FIFO_DEPTH ?= 16
EXTRA_ARGS += +define+FIFO_DEPTH=$(FIFO_DEPTH)
# Parameter-specific build directory so depth variants never collide.
SIM_BUILD := sim_build/fifo_depth_$(FIFO_DEPTH)
include $(SIM_ROOT)/mk/cocotb-common.mk
# ── Depth-specific test targets ──────────────────────────────────────────
# Recompile with a non-default DEPTH and filter to the relevant test(s).
.PHONY: test-depth-10 test-depth-1
test-depth-10:
$(MAKE) -B -f Makefile results.xml FIFO_DEPTH=10 \
COCOTB_TEST_FILTER=non_power_of_two
test-depth-1:
$(MAKE) -B -f Makefile results.xml FIFO_DEPTH=1 \
COCOTB_TEST_FILTER=depth_one
# Also remove variant build directories on clean.
clean::
rm -rf sim_build/fifo_depth_10 sim_build/fifo_depth_1

View File

@@ -0,0 +1,306 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
# ── Helpers ──────────────────────────────────────────────────────────────────
async def reset_fifo(dut):
"""Assert reset for 5 cycles, release, then wait 2 cycles."""
dut.rst_n.value = 0
dut.wr_valid_i.value = 0
dut.rd_ready_i.value = 0
dut.wr_row_i.value = 0
dut.wr_score_i.value = 0
for _ in range(5):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
async def push(dut, row: int, score: int):
"""Push a (row, score) entry, waiting until wr_ready_o is asserted."""
while not dut.wr_ready_o.value:
await RisingEdge(dut.clk)
dut.wr_row_i.value = row
dut.wr_score_i.value = score
dut.wr_valid_i.value = 1
await RisingEdge(dut.clk)
dut.wr_valid_i.value = 0
async def pop(dut) -> tuple[int, int]:
"""Pop an entry, waiting until rd_valid_o is asserted. Return (row, score)."""
while not dut.rd_valid_o.value:
await RisingEdge(dut.clk)
# Sample data in the cycle rd_valid_o is high, before accepting
row = int(dut.rd_row_o.value)
score = int(dut.rd_score_o.value)
dut.rd_ready_i.value = 1
await RisingEdge(dut.clk)
dut.rd_ready_i.value = 0
return (row, score)
async def push_pop_simultaneous(dut, row: int, score: int) -> tuple[int, int]:
"""Push and pop in the same cycle. Return the popped (row, score)."""
# Capture read data before accepting (combinational read)
row_out = int(dut.rd_row_o.value)
score_out = int(dut.rd_score_o.value)
# Assert both handshakes
dut.wr_row_i.value = row
dut.wr_score_i.value = score
dut.wr_valid_i.value = 1
dut.rd_ready_i.value = 1
await RisingEdge(dut.clk)
dut.wr_valid_i.value = 0
dut.rd_ready_i.value = 0
return (row_out, score_out)
# ── Test 1: Reset ───────────────────────────────────────────────────────────
@cocotb.test()
async def reset_asserts_empty_and_defaults(dut):
"""
After reset:
empty_o == 1
full_o == 0
rd_valid_o == 0
wr_ready_o == 1
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_fifo(dut)
assert dut.empty_o.value == 1, "empty_o should be 1 after reset"
assert dut.full_o.value == 0, "full_o should be 0 after reset"
assert dut.rd_valid_o.value == 0, "rd_valid_o should be 0 after reset"
assert dut.wr_ready_o.value == 1, "wr_ready_o should be 1 after reset"
# ── Test 2: Order preservation ──────────────────────────────────────────────
@cocotb.test()
async def fifo_preserves_order(dut):
"""
Push [(3,44),(1,55),(7,22),(2,55)] then pop and verify each entry appears
in the same order. After draining, empty_o == 1.
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_fifo(dut)
entries = [(3, 44), (1, 55), (7, 22), (2, 55)]
for row, score in entries:
await push(dut, row, score)
for expected_row, expected_score in entries:
row, score = await pop(dut)
assert row == expected_row, (
f"row mismatch: got {row}, expected {expected_row}"
)
assert score == expected_score, (
f"score mismatch: got {score}, expected {expected_score}"
)
assert dut.empty_o.value == 1, "fifo should be empty after draining all entries"
# ── Test 3: Full / back-pressure / drain ────────────────────────────────────
@cocotb.test()
async def fifo_full_backpressure_and_drain(dut):
"""
Fill the FIFO until full (discover depth), confirm backpressure,
pop one, push a new item, drain all, verify final inserted item
(99, 999) is preserved.
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_fifo(dut)
# ── Fill the FIFO until full ──
depth = 0
while not dut.full_o.value:
await push(dut, depth, depth * 10)
depth += 1
assert depth > 0, "should have pushed at least one item"
assert dut.full_o.value == 1, "fifo should report full after DEPTH pushes"
assert dut.wr_ready_o.value == 0, "wr_ready should be 0 when full"
# ── Pop one ──
row, score = await pop(dut)
assert row == 0 and score == 0, (
f"first popped entry should be (0,0), got ({row},{score})"
)
assert dut.full_o.value == 0, "fifo should no longer be full after one pop"
assert dut.wr_ready_o.value == 1, "wr_ready should be 1 after one pop"
# ── Push a new item ──
await push(dut, 99, 999)
# ── Drain all entries ──
# Expected order: (1,10), (2,20), … (depth-1, (depth-1)*10), (99,999)
for i in range(1, depth):
row, score = await pop(dut)
assert row == i, (
f"row mismatch during drain: got {row}, expected {i}"
)
assert score == i * 10, (
f"score mismatch during drain: got {score}, expected {i * 10}"
)
row, score = await pop(dut)
assert row == 99 and score == 999, (
f"final inserted item wrong: got ({row},{score}), expected (99,999)"
)
assert dut.empty_o.value == 1, "fifo should be empty after draining everything"
# ── Test 4: Simultaneous push+pop at full ───────────────────────────────────
@cocotb.test()
async def test_simultaneous_push_pop_when_full(dut):
"""
When the FIFO is full, a simultaneous read should allow a write to pass
through in the same cycle (wr_ready_o goes high combinationally because
of the read handshake). Verify the pop returns the oldest item, the
push inserts a new item, and the fill level stays at depth.
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_fifo(dut)
# ── Fill to full ──
depth = 0
while not dut.full_o.value:
await push(dut, depth, depth * 10)
depth += 1
assert dut.full_o.value == 1
assert dut.wr_ready_o.value == 0 # backpressure active
# ── Simultaneous push+pop ──
# wr_ready_o should be 1 this cycle because rd_valid_o && rd_ready_i
old_row, old_score = await push_pop_simultaneous(dut, 99, 999)
assert old_row == 0 and old_score == 0, (
f"simultaneous pop should return the oldest item, got ({old_row},{old_score})"
)
# Fill level unchanged (pop frees a slot, push immediately refills it)
assert dut.full_o.value == 1, "should remain full after simultaneous push+pop"
# ── Drain — expected: (1,10) … (depth-1,…), (99,999) ──
for i in range(1, depth):
row, score = await pop(dut)
assert row == i, (
f"row mismatch: got {row}, expected {i}"
)
row, score = await pop(dut)
assert row == 99 and score == 999, (
f"final item wrong: got ({row},{score})"
)
assert dut.empty_o.value == 1
# ── Test 5: Non-power-of-two depth ──────────────────────────────────────────
@cocotb.test()
async def test_non_power_of_two_depth(dut):
"""
Validate pointer wrap at a non-power-of-two DEPTH boundary (e.g. 10).
Fill the FIFO, then perform multiple pop+push cycles to force the write
pointer past a binary-wrap boundary, then drain and verify data integrity.
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_fifo(dut)
# ── Fill to full ──
depth = 0
while not dut.full_o.value:
await push(dut, depth, depth)
depth += 1
# ── Force several pointer wraps by pop+push ──
# For a broken RTL that wraps at 2**clog2(DEPTH) instead of DEPTH,
# enough iterations will hit out-of-bounds storage or corrupt data.
wraps = depth * 3 # well past the binary wrap point
for i in range(wraps):
row, score = await pop(dut)
await push(dut, 500 + i, 600 + i)
# ── Drain and verify count ──
items = []
while dut.rd_valid_o.value:
row, score = await pop(dut)
items.append((row, score))
assert len(items) == depth, (
f"expected {depth} items after {wraps} push/pop cycles, got {len(items)}"
)
# All items should be from the latest wrap cycle (order preserved)
# Final batch started at iteration wraps - depth:
for idx, (row, _score) in enumerate(items):
expected_row = 500 + (wraps - depth + idx)
assert row == expected_row, (
f"item {idx}: expected row {expected_row}, got {row}"
)
assert dut.empty_o.value == 1
# ── Test 6: Depth = 1 ───────────────────────────────────────────────────────
@cocotb.test()
async def test_depth_one(dut):
"""
Validate DEPTH=1 FIFO: push one item → full, pop → empty,
push another → full, pop → empty.
Gated: pushes one item; if the FIFO does *not* become full, we are
compiled with DEPTH > 1 and the test silently passes (skips).
(Use ``make test-depth-1`` to run with FIFO_DEPTH=1.)
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_fifo(dut)
# ── Probe: is DEPTH == 1? ────────────────────────────────────────────
await push(dut, 42, 100)
if not dut.full_o.value:
# Not compiled at DEPTH=1; pop the item back and return.
await pop(dut)
return
# ── Full DEPTH=1 logic follows ───────────────────────────────────────
assert dut.wr_ready_o.value == 0, "wr_ready should be 0 when full"
row, score = await pop(dut)
assert row == 42 and score == 100, (
f"popped ({row},{score}), expected (42,100)"
)
assert dut.empty_o.value == 1, "depth-1 FIFO should be empty after one pop"
assert dut.full_o.value == 0
await push(dut, 7, 55)
assert dut.full_o.value == 1
assert dut.empty_o.value == 0
row, score = await pop(dut)
assert row == 7 and score == 55
assert dut.empty_o.value == 1
assert dut.full_o.value == 0

View File

@@ -0,0 +1,29 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := match_engine_pipeline
COCOTB_TEST_MODULES := tests.modules.match_engine_pipeline.test_match_engine_pipeline
VERILOG_SOURCES := $(RTL_MATCH_ENGINE)
# Allow override from command line (e.g. make TOPK_K=1)
TOPK_K ?= 4
EXTRA_ARGS += +define+TOPK_K=$(TOPK_K)
export TOPK_K
# Parameter-specific build directory so K variants never collide.
SIM_BUILD := sim_build/k_$(TOPK_K)
include $(SIM_ROOT)/mk/cocotb-common.mk
# ── K-specific test targets ──────────────────────────────────────────────
# Recompile with K=1 and filter to the K=1 test.
.PHONY: test-k1
test-k1:
$(MAKE) -B -f Makefile results.xml TOPK_K=1 \
COCOTB_TEST_FILTER=match_engine_topk_k1
# Also remove all module-local build artifacts, including legacy and variant dirs.
clean::
rm -rf sim_build

View File

@@ -0,0 +1,561 @@
from __future__ import annotations
import os
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge, Timer
# Read TOPK_K from environment (set by Makefile); default 4 for K=4-only tests.
TOPK_K = int(os.environ.get("TOPK_K", "4"))
async def reset_match(dut):
dut.rst_n.value = 0
dut.query_valid.value = 0
dut.query_hash.value = 0
dut.result_ready.value = 1
dut.rd_valid_i.value = 0
dut.rd_row_ids_i.value = 0
dut.rd_hashes_i.value = 0
dut.rd_lane_valid_i.value = 0
for _ in range(3):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
def score_for(query: int, value: int, width: int) -> int:
"""Compute popcount of ~(query ^ value), i.e. number of matching bits."""
return width - ((query ^ value) & ((1 << width) - 1)).bit_count()
def golden_topk(
rows_and_hashes: list[tuple[int, int]], query: int, width: int, k: int
) -> list[tuple[int, int]]:
"""Return the Top-K (row, score) pairs sorted by score desc, row asc."""
scored = [(row, score_for(query, value, width))
for row, value in rows_and_hashes]
return sorted(scored, key=lambda item: (-item[1], item[0]))[:k]
async def collect_serial_results(
dut, k: int, timeout_cycles: int = 200
) -> list[tuple[int, int, int, int]]:
"""Collect serialised Top-K results from the pipeline.
Returns list of (rank, row, score, last) tuples.
Asserts timeout if the stream does not complete.
"""
dut.result_ready.value = 1
observed: list[tuple[int, int, int, int]] = []
for _ in range(timeout_cycles):
if int(dut.result_valid.value):
rank = int(dut.result_rank.value) if hasattr(
dut, "result_rank") else len(observed)
row = int(dut.result_row.value)
score = int(dut.result_score.value)
last = int(dut.result_last.value) if hasattr(
dut, "result_last") else int(len(observed) == k - 1)
observed.append((rank, row, score, last))
if last:
return observed
await RisingEdge(dut.clk)
raise AssertionError(
f"result stream did not complete after {timeout_cycles} cycles"
)
async def feed_all_batches(
dut,
rows_and_hashes: list[tuple[int, int]] | None = None,
lane_mask: int | None = None,
):
"""Feed all batches to the pipeline.
If *rows_and_hashes* is provided, the test uses those values for each row
(row_id -> hash) instead of the default hash=query (with special rows).
If *lane_mask* is provided, only those lanes are marked valid in each batch.
"""
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
for base in range(0, NUM_ROWS, LANES):
while int(dut.rd_valid_o.value) == 0:
await RisingEdge(dut.clk)
assert int(dut.rd_base_row_o.value) == base
rows = 0
hashes = 0
for lane in range(LANES):
row_id = base + lane
rows |= row_id << (lane * ROW_BITS)
if rows_and_hashes is not None and row_id in rows_and_hashes:
row_hash = rows_and_hashes[row_id]
else:
row_hash = 0
hashes |= row_hash << (lane * HASH_BITS)
dut.rd_row_ids_i.value = rows
dut.rd_hashes_i.value = hashes
if lane_mask is not None:
dut.rd_lane_valid_i.value = lane_mask
else:
dut.rd_lane_valid_i.value = (1 << LANES) - 1
dut.rd_valid_i.value = 1
await RisingEdge(dut.clk)
dut.rd_valid_i.value = 0
# ==============================================================================
# Test: Top-1 smoke
# ==============================================================================
@cocotb.test()
async def match_engine_returns_top1_after_pipeline_drain(dut):
"""Top-1 smoke test — verify basic result_valid/result_row/result_score.
Works with both old Top-1 and new Top-K serial interfaces.
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_match(dut)
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
TARGET_ROW = 9
query = (1 << HASH_BITS) - 1
dut.query_hash.value = query
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
rows_and_hashes: dict[int, int] = {9: query}
await feed_all_batches(dut, rows_and_hashes)
for _ in range(20):
if int(dut.result_valid.value):
break
await RisingEdge(dut.clk)
assert int(dut.result_valid.value) == 1
assert int(dut.result_row.value) == TARGET_ROW
assert int(dut.result_score.value) == HASH_BITS
# ==============================================================================
# Test: Top-K all-valid
# ==============================================================================
@cocotb.test()
async def match_engine_keeps_multiple_topk_candidates_from_same_batch(dut):
"""Verify Top-K pipeline returns K best matches in rank order.
Feed all rows with known hash values so the golden Top-K is predictable.
"""
if TOPK_K != 4:
dut._log.info(f"Skipping: hardcoded K=4 test (TOPK_K={TOPK_K})")
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_match(dut)
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
K = 4
query = (1 << HASH_BITS) - 1
special = {0: query, 1: query ^ 1, 2: query ^ 3, 3: query ^ 7}
rows_and_hashes: list[tuple[int, int]] = []
dut.query_hash.value = query
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
for base in range(0, NUM_ROWS, LANES):
while int(dut.rd_valid_o.value) == 0:
await RisingEdge(dut.clk)
rows = 0
hashes = 0
for lane in range(LANES):
row_id = base + lane
row_hash = special.get(row_id, 0)
rows_and_hashes.append((row_id, row_hash))
rows |= row_id << (lane * ROW_BITS)
hashes |= row_hash << (lane * HASH_BITS)
dut.rd_row_ids_i.value = rows
dut.rd_hashes_i.value = hashes
dut.rd_lane_valid_i.value = (1 << LANES) - 1
dut.rd_valid_i.value = 1
await RisingEdge(dut.clk)
dut.rd_valid_i.value = 0
expected = golden_topk(rows_and_hashes, query, HASH_BITS, K)
observed = await collect_serial_results(dut, K, timeout_cycles=800)
assert [(row, score) for _, row, score, _ in observed] == expected, (
f"observed={[(r, s) for _, r, s, _ in observed]} != expected={expected}"
)
assert [rank for rank, _, _, _ in observed] == list(range(K)), (
"ranks are not sequential"
)
assert observed[-1][3] == 1, "last flag not set on final result"
# ==============================================================================
# Test: Invalid-mask coverage — lane 0 invalid, all-invalid batch
# ==============================================================================
@cocotb.test()
async def match_engine_handles_invalid_lane_masks(dut):
"""Pipeline completes correctly with invalid lanes and all-invalid batches.
Coverage:
- Batch 0 is driven with rd_lane_valid_i=0 (all-invalid)
→ batch_scores_done vacuously true, zero candidates pushed.
- All other batches use lane_mask = 0b0110 (lanes 1 and 2 only).
- Invalid lanes (0, 3-7) carry high-score trap hashes that WOULD enter
the Top-K if the pipeline wrongly pushed them.
- Golden Top-K is computed from valid-lane rows only and compared
against the full serialised output — not just a subset check.
"""
if TOPK_K != 4:
dut._log.info(f"Skipping: hardcoded K=4 test (TOPK_K={TOPK_K})")
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_match(dut)
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
K = 4
query = (1 << HASH_BITS) - 1
trap_hash = query # perfect score; would be Top-1 if wrongly pushed
# Only rows on valid lanes (1, 2) from batches AFTER the all-invalid one
# contribute candidates. Use distinct scores so the golden ordering
# is unambiguous.
valid_rows: dict[int, int] = {9: query, 10: query ^ 1}
# Track hash values passed per row (including trap rows on invalid lanes)
# for golden-topk computation.
valid_rows_and_hashes: list[tuple[int, int]] = []
dut.query_hash.value = query
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
for base in range(0, NUM_ROWS, LANES):
while int(dut.rd_valid_o.value) == 0:
await RisingEdge(dut.clk)
# Batch 0 is all-invalid; all others use lanes 1,2 only.
cur_mask = 0x00 if base == 0 else 0b0110
rows = 0
hashes = 0
for lane in range(LANES):
row_id = base + lane
rows |= row_id << (lane * ROW_BITS)
# On invalid lanes drive a trap hash (perfect score).
# On valid lanes drive the designated hash (or 0 if not special).
if cur_mask & (1 << lane):
row_hash = valid_rows.get(row_id, 0)
valid_rows_and_hashes.append((row_id, row_hash))
else:
row_hash = trap_hash # would be Top-1 if wrongly pushed
hashes |= row_hash << (lane * HASH_BITS)
dut.rd_row_ids_i.value = rows
dut.rd_hashes_i.value = hashes
dut.rd_lane_valid_i.value = cur_mask
dut.rd_valid_i.value = 1
await RisingEdge(dut.clk)
dut.rd_valid_i.value = 0
expected = golden_topk(valid_rows_and_hashes, query, HASH_BITS, K)
observed = await collect_serial_results(dut, K, timeout_cycles=800)
assert [(row, score) for _, row, score, _ in observed] == expected, (
f"observed={[(r, s) for _, r, s, _ in observed]} != expected={expected}"
)
assert [rank for rank, _, _, _ in observed] == list(range(K)), (
"ranks are not sequential"
)
assert observed[-1][3] == 1, "last flag not set on final result"
# ==============================================================================
# Test: result_ready stall stability
# ==============================================================================
@cocotb.test()
async def match_engine_result_ready_stall_preserves_output(dut):
"""When result_ready is deasserted while result_valid is high, the output
signals (result_rank, result_row, result_score, result_last) must remain
stable until result_ready is re-asserted.
"""
if TOPK_K != 4:
dut._log.info(f"Skipping: hardcoded K=4 test (TOPK_K={TOPK_K})")
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_match(dut)
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
K = 4
query = (1 << HASH_BITS) - 1
special = {0: query, 1: query ^ 1, 2: query ^ 3, 3: query ^ 7}
rows_and_hashes: list[tuple[int, int]] = []
dut.query_hash.value = query
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
await feed_all_batches(dut)
# Wait for first result
for _ in range(200):
await RisingEdge(dut.clk)
if int(dut.result_valid.value):
break
assert int(dut.result_valid.value) == 1, "result_valid never went high"
# Capture the first-beat values
rank0 = int(dut.result_rank.value)
row0 = int(dut.result_row.value)
score0 = int(dut.result_score.value)
last0 = int(dut.result_last.value)
# Stall — deassert result_ready for several cycles
dut.result_ready.value = 0
for _ in range(5):
await RisingEdge(dut.clk)
assert int(dut.result_valid.value) == 1, (
"result_valid dropped during stall"
)
assert int(dut.result_rank.value) == rank0
assert int(dut.result_row.value) == row0
assert int(dut.result_score.value) == score0
assert int(dut.result_last.value) == last0
# Release stall — should advance to next rank
dut.result_ready.value = 1
await RisingEdge(dut.clk)
# Collect the rest
observed: list[tuple[int, int, int, int]] = [(rank0, row0, score0, last0)]
for _ in range(200):
if int(dut.result_valid.value):
rank = int(dut.result_rank.value)
row = int(dut.result_row.value)
score = int(dut.result_score.value)
last = int(dut.result_last.value)
observed.append((rank, row, score, last))
if last:
break
await RisingEdge(dut.clk)
assert len(observed) == K, (
f"expected {K} results, got {len(observed)}"
)
assert observed[-1][3] == 1, "last flag not set on final result"
# ==============================================================================
# Test: Toggled ready with golden comparison
# ==============================================================================
@cocotb.test()
async def match_engine_holds_serial_result_under_ready_stalls(dut):
"""Toggle result_ready during serialisation and verify golden Top-K matches.
Uses query=(1<<HASH_BITS)-1 and row_hash=query^row_id for deterministic
per-row scoring. result_ready is deasserted every 3rd cycle to exercise
backpressure. Observed beats are captured only on valid+ready handshakes
and compared against golden_topk.
"""
if TOPK_K != 4:
dut._log.info(f"Skipping: hardcoded K=4 test (TOPK_K={TOPK_K})")
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_match(dut)
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
K = TOPK_K
query = (1 << HASH_BITS) - 1
dut.query_hash.value = query
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
# Feed all rows with deterministic hash = query ^ row_id
rows_and_hashes: list[tuple[int, int]] = []
for base in range(0, NUM_ROWS, LANES):
while int(dut.rd_valid_o.value) == 0:
await RisingEdge(dut.clk)
rows = 0
hashes = 0
for lane in range(LANES):
row_id = base + lane
row_hash = query ^ row_id
rows_and_hashes.append((row_id, row_hash))
rows |= row_id << (lane * ROW_BITS)
hashes |= row_hash << (lane * HASH_BITS)
dut.rd_row_ids_i.value = rows
dut.rd_hashes_i.value = hashes
dut.rd_lane_valid_i.value = (1 << LANES) - 1
dut.rd_valid_i.value = 1
await RisingEdge(dut.clk)
dut.rd_valid_i.value = 0
expected = golden_topk(rows_and_hashes, query, HASH_BITS, K)
# Collect with toggling result_ready: stall on every 3rd cycle (cycle % 3 == 1)
observed: list[tuple[int, int]] = []
for cycle in range(800):
dut.result_ready.value = 0 if cycle % 3 == 1 else 1
await RisingEdge(dut.clk)
if int(dut.result_valid.value) and int(dut.result_ready.value):
row = int(dut.result_row.value)
score = int(dut.result_score.value)
observed.append((row, score))
if int(dut.result_last.value):
break
assert observed == expected, (
f"observed={observed} != expected={expected}"
)
# ==============================================================================
# Test: consumed_candidates_debug (SIM_DEBUG only)
# ==============================================================================
@cocotb.test()
async def match_engine_debug_consumed_candidates(dut):
"""Under SIM_DEBUG, consumed_candidates_debug must equal NUM_ROWS after
an all-lane-valid query completes (every row produces one candidate).
"""
if not hasattr(dut, "consumed_candidates_debug"):
dut._log.info("Skipping: consumed_candidates_debug not present (no SIM_DEBUG)")
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_match(dut)
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
K = TOPK_K
query = (1 << HASH_BITS) - 1
dut.query_hash.value = query
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
await feed_all_batches(dut)
# Wait for result completion
await collect_serial_results(dut, K, timeout_cycles=800)
# After completion, consumed_candidates_debug should show all rows consumed
consumed = int(dut.consumed_candidates_debug.value)
assert consumed == NUM_ROWS, (
f"consumed_candidates_debug={consumed} != NUM_ROWS={NUM_ROWS}"
)
# ==============================================================================
# Test: K=1 regression
# ==============================================================================
@cocotb.test()
async def match_engine_topk_k1(dut):
"""K=1: verify single beat with rank=0, last=1, row/score matches golden.
Feeds all rows all-lane-valid with a single high-score row (row 9).
Only one result beat is expected; it must carry rank=0, last=1, and
match the golden Top-1 (row, score).
"""
if TOPK_K != 1:
dut._log.info(f"Skipping: K=1 test requires TOPK_K=1 (got {TOPK_K})")
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_match(dut)
LANES = len(dut.rd_lane_valid_i)
ROW_BITS = len(dut.rd_row_ids_i) // LANES
HASH_BITS = len(dut.rd_hashes_i) // LANES
NUM_ROWS = 1 << len(dut.rd_base_row_o)
query = (1 << HASH_BITS) - 1
# Row 9 gets the best hash; all others get 0
special: dict[int, int] = {9: query}
rows_and_hashes: list[tuple[int, int]] = []
dut.query_hash.value = query
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
for base in range(0, NUM_ROWS, LANES):
while int(dut.rd_valid_o.value) == 0:
await RisingEdge(dut.clk)
rows = 0
hashes = 0
for lane in range(LANES):
row_id = base + lane
row_hash = special.get(row_id, 0)
rows_and_hashes.append((row_id, row_hash))
rows |= row_id << (lane * ROW_BITS)
hashes |= row_hash << (lane * HASH_BITS)
dut.rd_row_ids_i.value = rows
dut.rd_hashes_i.value = hashes
dut.rd_lane_valid_i.value = (1 << LANES) - 1
dut.rd_valid_i.value = 1
await RisingEdge(dut.clk)
dut.rd_valid_i.value = 0
expected = golden_topk(rows_and_hashes, query, HASH_BITS, 1)
assert len(expected) == 1
exp_row, exp_score = expected[0]
# Collect single result
dut.result_ready.value = 1
for _ in range(200):
if int(dut.result_valid.value):
rank = int(dut.result_rank.value)
row = int(dut.result_row.value)
score = int(dut.result_score.value)
last = int(dut.result_last.value)
assert rank == 0, f"rank={rank} != 0"
assert last == 1, f"last={last} != 1"
assert row == exp_row, f"row={row} != {exp_row}"
assert score == exp_score, f"score={score} != {exp_score}"
return
await RisingEdge(dut.clk)
raise AssertionError("result_valid never went high")

View File

@@ -0,0 +1,11 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := noise_mask_bernoulli
COCOTB_TEST_MODULES := tests.modules.noise_mask_bernoulli.test_noise_mask_bernoulli
VERILOG_SOURCES := $(RTL_BERNOULLI_NOISE_MASK)
HASH_BITS ?= 512
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

@@ -0,0 +1 @@
"""Cocotb tests for noise_mask_bernoulli."""

View File

@@ -0,0 +1,242 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge, Timer
HASH_BITS = 512
DEFAULT_BLOCKS = 16
DEFAULT_PRIME_CYCLES = 1
DEFAULT_START_TO_DONE_CYCLES = DEFAULT_PRIME_CYCLES + DEFAULT_BLOCKS
def popcount(value: int) -> int:
return int(value).bit_count()
async def reset_dut(dut):
"""Reset the DUT and drive inactive defaults."""
dut.rst_n.value = 0
dut.start_i.value = 0
dut.threshold_i.value = 0
for _ in range(5):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
async def start_and_wait_done(dut, threshold: int) -> tuple[int, int]:
"""Start one generation and return (mask, cycles from accepted start to done)."""
dut.threshold_i.value = threshold
dut.start_i.value = 1
await Timer(1, unit="step")
await RisingEdge(dut.clk)
await Timer(1, unit="step")
dut.start_i.value = 0
cycles = 0
while True:
assert cycles < 128, "timed out waiting for done_o"
if bool(dut.done_o.value):
break
assert bool(dut.busy_o.value), "busy_o should stay high until done_o"
await RisingEdge(dut.clk)
await Timer(1, unit="step")
cycles += 1
mask = int(dut.mask_o.value)
await RisingEdge(dut.clk)
await Timer(1, unit="step")
assert not bool(dut.done_o.value), "done_o should be a single-cycle pulse"
assert not bool(dut.busy_o.value), "busy_o should be low after done_o"
assert int(dut.mask_o.value) == mask, "mask_o should remain stable after done_o"
return mask, cycles
@cocotb.test()
async def reset_outputs_idle_and_clear_mask(dut):
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
assert not bool(dut.busy_o.value), "busy_o should be low after reset"
assert not bool(dut.done_o.value), "done_o should be low after reset"
assert int(dut.mask_o.value) == 0, "mask_o should reset to zero"
@cocotb.test()
async def threshold_zero_generates_empty_mask(dut):
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
mask, cycles = await start_and_wait_done(dut, threshold=0)
assert mask == 0, "threshold=0 should generate all-zero mask"
assert popcount(mask) == 0, "threshold=0 popcount should be zero"
assert cycles == DEFAULT_START_TO_DONE_CYCLES, (
f"default latency should be {DEFAULT_START_TO_DONE_CYCLES} cycles, got {cycles}"
)
@cocotb.test()
async def test_deterministic_across_reset(dut):
"""Same threshold after reset should produce identical mask sequence."""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
masks_a = []
for _ in range(3):
mask, _ = await start_and_wait_done(dut, threshold=128)
masks_a.append(mask)
await reset_dut(dut)
masks_b = []
for _ in range(3):
mask, _ = await start_and_wait_done(dut, threshold=128)
masks_b.append(mask)
assert masks_a == masks_b, "Mask sequence should be deterministic across reset"
@cocotb.test()
async def test_threshold_monotonicity(dut):
"""Higher threshold should yield higher average popcount."""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
N = 32
masks_64 = []
for _ in range(N):
mask, _ = await start_and_wait_done(dut, threshold=64)
masks_64.append(mask)
await reset_dut(dut)
masks_192 = []
for _ in range(N):
mask, _ = await start_and_wait_done(dut, threshold=192)
masks_192.append(mask)
avg_64 = sum(popcount(m) for m in masks_64) / N
avg_192 = sum(popcount(m) for m in masks_192) / N
# threshold=64: expected popcount ~512*64/256 = 128
# threshold=192: expected popcount ~512*192/256 = 384
assert 60 < avg_64 < 135, f"avg popcount at threshold=64 out of range: {avg_64}"
assert 365 < avg_192 < 405, f"avg popcount at threshold=192 out of range: {avg_192}"
assert avg_192 > avg_64 + 200, (
f"threshold=192 avg ({avg_192}) should exceed threshold=64 avg ({avg_64}) by margin"
)
@cocotb.test()
async def test_threshold_255_not_all_ones(dut):
"""threshold=255 should produce dense masks but not force all ones."""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
masks = []
for _ in range(8):
mask, _ = await start_and_wait_done(dut, threshold=255)
masks.append(mask)
popcounts = [popcount(m) for m in masks]
avg = sum(popcounts) / len(popcounts)
# At least one mask should have some zero bits
assert any(pc < HASH_BITS for pc in popcounts), (
"threshold=255 should produce at least one mask with some zero bits"
)
# Average density should be very high (p=255/256)
assert avg > HASH_BITS * 0.9, f"threshold=255 average popcount too low: {avg}"
@cocotb.test()
async def test_busy_start_ignored(dut):
"""Pulsing start_i while busy_o is high must be ignored.
Start one generation, then pulse start_i with a different threshold_i while
the DUT is busy, and prove the busy pulse is ignored (no second done_o,
same latency, same mask as an uninterrupted run).
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
# --- Baseline: single uninterrupted generation with threshold=64 ---
mask_ref, cycles_ref = await start_and_wait_done(dut, threshold=64)
assert cycles_ref == DEFAULT_START_TO_DONE_CYCLES, (
f"Baseline latency should be {DEFAULT_START_TO_DONE_CYCLES}, got {cycles_ref}"
)
# --- Test: pulse spurious start with threshold=255 while busy ---
await reset_dut(dut)
# Start legitimate operation with threshold=64
dut.threshold_i.value = 64
dut.start_i.value = 1
await Timer(1, unit="step")
await RisingEdge(dut.clk) # sampled: FSM transitions to PRIME
await Timer(1, unit="step")
dut.start_i.value = 0
assert bool(dut.busy_o.value), "busy_o should be high after start"
# Let FSM reach RUN (PRIME→RUN at next posedge, then one RUN cycle)
await RisingEdge(dut.clk) # PRIME → RUN
await Timer(1, unit="step")
await RisingEdge(dut.clk) # RUN, first block processed
await Timer(1, unit="step")
assert bool(dut.busy_o.value), "DUT must be busy during RUN"
# Pulse start_i with threshold=255 while DUT is busy (must be ignored)
dut.threshold_i.value = 255
dut.start_i.value = 1
await Timer(1, unit="step")
await RisingEdge(dut.clk) # sampled but FSM in RUN → ignored
await Timer(1, unit="step")
dut.start_i.value = 0
# Count remaining cycles until done_o
# Cycles already consumed from the legitimate start:
# 1 for PRIME + 1 for RUN before spurious + 1 for spurious-start cycle = 3
elapsed = 3
cycles_from_spurious = 0
while True:
if bool(dut.done_o.value):
break
assert elapsed + cycles_from_spurious < 128, "timed out waiting for done_o"
assert bool(dut.busy_o.value), "busy_o should stay high until done_o"
await RisingEdge(dut.clk)
await Timer(1, unit="step")
cycles_from_spurious += 1
total_cycles = elapsed + cycles_from_spurious
assert total_cycles == DEFAULT_START_TO_DONE_CYCLES, (
f"Latency should be {DEFAULT_START_TO_DONE_CYCLES} despite spurious start, "
f"got {total_cycles} ({elapsed}+{cycles_from_spurious})"
)
mask_test = int(dut.mask_o.value)
await RisingEdge(dut.clk)
await Timer(1, unit="step")
# Verify done_o single-cycle and busy_o falls low
assert not bool(dut.done_o.value), "done_o should be a single-cycle pulse"
assert not bool(dut.busy_o.value), "busy_o should be low after done_o"
assert int(dut.mask_o.value) == mask_test, (
"mask_o should remain stable after done_o"
)
# Compare mask to baseline (same seed + same threshold → same mask)
assert mask_test == mask_ref, (
"Spurious start must not corrupt in-flight mask generation"
)
# Verify no second done_o from the ignored start
for _ in range(32):
await RisingEdge(dut.clk)
await Timer(1, unit="step")
assert not bool(dut.done_o.value), (
"Spurious start must not queue a second operation"
)

View File

@@ -0,0 +1,9 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := popcount_pipeline
COCOTB_TEST_MODULES := tests.modules.popcount_pipeline.test_popcount_pipeline
VERILOG_SOURCES := $(RTL_MATCH_ENGINE)
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

@@ -0,0 +1,85 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
# ── Helpers ──────────────────────────────────────────────────────────────────
async def reset_popcount(dut):
"""Assert reset for 5 cycles, then release and wait 2 cycles."""
dut.rst_n.value = 0
dut.valid_i.value = 0
dut.row_i.value = 0
dut.bits_i.value = 0
for _ in range(5):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
async def drive_and_expect(dut, row: int, bits: int, expected_count: int):
"""Drive valid_i for one cycle, wait for valid_o, assert row_o / count_o."""
dut.row_i.value = row
dut.bits_i.value = bits
dut.valid_i.value = 1
await RisingEdge(dut.clk)
dut.valid_i.value = 0
# Pipeline depth = 3 (valid_s1_q -> valid_s2_q -> valid_o); wait with timeout
for _ in range(20):
await RisingEdge(dut.clk)
if int(dut.valid_o.value):
break
assert int(dut.valid_o.value) == 1, "valid_o never asserted"
assert int(dut.row_o.value) == row, (
f"row_o mismatch: got {int(dut.row_o.value)}, expected {row}"
)
assert int(dut.count_o.value) == expected_count, (
f"count_o mismatch: got {int(dut.count_o.value)}, expected {expected_count}"
)
def _width(dut) -> int:
"""Return the WIDTH of bits_i in the DUT."""
return len(dut.bits_i)
# ── Test ─────────────────────────────────────────────────────────────────────
@cocotb.test()
async def popcount_pipeline_counts_bits_and_preserves_row_metadata(dut):
"""
Popcount pipeline: for each deterministic vector, assert row_o matches row_i
and count_o equals the bit-count of bits_i.
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_popcount(dut)
w = _width(dut)
all_ones = (1 << w) - 1
# Fixed literal: "0123456789ABCDEF" repeated enough for width, masked.
hex_repeat = (w + 63) // 64
fixed_literal = int("0123456789ABCDEF" * hex_repeat, 16) & all_ones
# Alternating 0xAA pattern
alternating_aa = int.from_bytes(b"\xAA" * (w // 8), "little")
vectors = [
(0, 0, 0),
(1, all_ones, w),
(2, 1 << 0, 1),
(3, 1 << min(127, w - 1), 1),
(4, 1 << min(511, w - 1), 1),
(5, alternating_aa, alternating_aa.bit_count()),
(6, fixed_literal, fixed_literal.bit_count()),
]
for row, bits, expected in vectors:
await drive_and_expect(dut, row, bits, expected)

View File

@@ -0,0 +1,29 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := result_serializer
COCOTB_TEST_MODULES := tests.modules.result_serializer.test_result_serializer
VERILOG_SOURCES := $(RTL_ROOT)/core/result_serializer.sv
# Allow override from command line (e.g. make TOPK_K=1)
TOPK_K ?= 4
EXTRA_ARGS += +define+TOPK_K=$(TOPK_K)
export TOPK_K
# Parameter-specific build directory so K variants never collide.
SIM_BUILD := sim_build/k_$(TOPK_K)
include $(SIM_ROOT)/mk/cocotb-common.mk
# ── K-specific test targets ──────────────────────────────────────────────
# Recompile with K=1 and filter to the K=1 test.
.PHONY: test-k1
test-k1:
$(MAKE) -B -f Makefile results.xml TOPK_K=1 \
COCOTB_TEST_FILTER=serializer_handles_k1
# Also remove variant build directories on clean.
clean::
rm -rf sim_build/k_1

View File

@@ -0,0 +1 @@
# Result serializer module tests

View File

@@ -0,0 +1,212 @@
from __future__ import annotations
import os
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
# ── Parameter defaults (matching cam_params.svh: NUM_ROWS=4096, HASH_BITS=512) ──
# K is read from env so Makefile override (e.g. make TOPK_K=1) propagates to tests.
K = int(os.environ.get('TOPK_K', '4'))
ROW_BITS = 12
SCORE_BITS = 10 # $clog2(512 + 1)
# Mirror the RTL's RANK_BITS formula: (K <= 1) ? 1 : $clog2(K)
RANK_BITS = 1 if K <= 1 else (K - 1).bit_length()
def pack(values: list[int], width: int) -> int:
"""Pack a list of values into a flattened integer (LSB = index 0)."""
result = 0
for idx, value in enumerate(values):
result |= value << (idx * width)
return result
# ── Helpers ────────────────────────────────────────────────────────────────────
async def reset_serializer(dut):
"""Drive reset active-low for 5 cycles, then release; zero all inputs."""
dut.rst_n.value = 0
dut.start_i.value = 0
dut.result_ready_i.value = 0
dut.topk_rows_i.value = 0
dut.topk_scores_i.value = 0
for _ in range(5):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
async def assert_output_stable(dut, rank: int, row: int, score: int,
last: bool, valid: bool):
"""Assert output signals match expected values."""
assert int(dut.result_rank_o.value) == rank, \
f"rank: expected {rank}, got {int(dut.result_rank_o.value)}"
assert int(dut.result_row_o.value) == row, \
f"row: expected {row}, got {int(dut.result_row_o.value)}"
assert int(dut.result_score_o.value) == score, \
f"score: expected {score}, got {int(dut.result_score_o.value)}"
assert bool(dut.result_last_o.value) == last, \
f"last: expected {last}, got {bool(dut.result_last_o.value)}"
assert bool(dut.result_valid_o.value) == valid, \
f"valid: expected {valid}, got {bool(dut.result_valid_o.value)}"
# ── Test 1: Serializer outputs rank order and last ─────────────────────────────
@cocotb.test()
async def serializer_outputs_rank_order_and_last(dut):
"""
Start serialization with K=4 results. Clock through all 4 ranks, verifying
row/score/last at each step. Assert done_o pulses after last rank.
Gated: requires TOPK_K=4 (the default). Use ``make test-k1`` for K=1.
"""
if K != 4:
cocotb.log.info("Skipping (designed for TOPK_K=4, got %d)", K)
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_serializer(dut)
# Pre-defined top-4 results
rows = [42, 17, 99, 5]
scores = [10, 9, 8, 7]
# Drive packed inputs and assert start
dut.topk_rows_i.value = pack(rows, ROW_BITS)
dut.topk_scores_i.value = pack(scores, SCORE_BITS)
dut.start_i.value = 1
await RisingEdge(dut.clk)
dut.start_i.value = 0
# DUT should be busy, valid, showing rank 0
assert bool(dut.busy_o.value), "busy_o should be high after start"
await assert_output_stable(dut, rank=0, row=rows[0], score=scores[0],
last=False, valid=True)
# Cycle through remaining ranks with ready held high
for r in range(1, K):
dut.result_ready_i.value = 1
await RisingEdge(dut.clk)
is_last = (r == K - 1)
await assert_output_stable(dut, rank=r, row=rows[r], score=scores[r],
last=is_last, valid=True)
# After last rank fire, next cycle should show done_o and deassert busy
await RisingEdge(dut.clk)
assert bool(dut.done_o.value), "done_o should be high after last rank fire"
assert not bool(dut.busy_o.value), "busy_o should be low after serialization done"
assert not bool(dut.result_valid_o.value), \
"result_valid_o should be low after serialization done"
# done_o should be single-cycle pulse
await RisingEdge(dut.clk)
assert not bool(dut.done_o.value), "done_o should be single-cycle pulse"
assert int(dut.result_rank_o.value) == 0, \
"rank should reset to 0 after done"
# ── Test 2: Holds output when not ready ────────────────────────────────────────
@cocotb.test()
async def serializer_holds_output_when_not_ready(dut):
"""
After start, with result_ready_i=0, output remains stable.
After ready is asserted, advances to next rank.
Gated: requires TOPK_K=4 (the default). Use ``make test-k1`` for K=1.
"""
if K != 4:
cocotb.log.info("Skipping (designed for TOPK_K=4, got %d)", K)
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_serializer(dut)
rows = [10, 20, 30, 40]
scores = [5, 4, 3, 2]
dut.topk_rows_i.value = pack(rows, ROW_BITS)
dut.topk_scores_i.value = pack(scores, SCORE_BITS)
dut.result_ready_i.value = 0 # not ready
dut.start_i.value = 1
await RisingEdge(dut.clk)
dut.start_i.value = 0
# Output should present rank 0 but hold because ready is low
await assert_output_stable(dut, rank=0, row=rows[0], score=scores[0],
last=False, valid=True)
# Stall for 3 cycles — mutating inputs to verify they don't leak through
# (snapshot must hold original values captured at start)
for cycle in range(3):
# Corrupt the live inputs — a buggy combinational slice would change
dut.topk_rows_i.value = pack([cycle + 100, 0, 0, 0], ROW_BITS)
dut.topk_scores_i.value = pack([cycle + 50, 0, 0, 0], SCORE_BITS)
await RisingEdge(dut.clk)
await assert_output_stable(dut, rank=0, row=rows[0], score=scores[0],
last=False, valid=True)
# Restore original inputs and assert ready for 1 cycle; rank advances to 1
dut.topk_rows_i.value = pack(rows, ROW_BITS)
dut.topk_scores_i.value = pack(scores, SCORE_BITS)
dut.result_ready_i.value = 1
await RisingEdge(dut.clk)
await assert_output_stable(dut, rank=1, row=rows[1], score=scores[1],
last=False, valid=True)
# Stall again at rank 1 for 2 cycles — mutate inputs again
dut.result_ready_i.value = 0
for cycle in range(2):
dut.topk_rows_i.value = pack([cycle + 200, 0, 0, 0], ROW_BITS)
dut.topk_scores_i.value = pack([cycle + 100, 0, 0, 0], SCORE_BITS)
await RisingEdge(dut.clk)
await assert_output_stable(dut, rank=1, row=rows[1], score=scores[1],
last=False, valid=True)
# ── Test 3: K=1 edge case ──────────────────────────────────────────────────────
@cocotb.test()
async def serializer_handles_k1(dut):
"""
When K=1 (overridden via generics), start and fire produce done_o and
deassert busy on the next cycle. Run via ``make test-k1``.
"""
if K != 1:
cocotb.log.info("Skipping (designed for TOPK_K=1, got %d)", K)
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_serializer(dut)
rows = [100]
scores = [50]
dut.topk_rows_i.value = pack(rows, ROW_BITS)
dut.topk_scores_i.value = pack(scores, SCORE_BITS)
dut.start_i.value = 1
await RisingEdge(dut.clk)
dut.start_i.value = 0
# K=1: rank 0 is also the last; valid asserted
assert bool(dut.busy_o.value), "busy_o should be high after start"
await assert_output_stable(dut, rank=0, row=rows[0], score=scores[0],
last=True, valid=True)
# Fire the one and only result
dut.result_ready_i.value = 1
await RisingEdge(dut.clk)
# done_o should pulse, busy deasserted
assert bool(dut.done_o.value), "done_o should pulse after K=1 fire"
assert not bool(dut.busy_o.value), "busy_o should be low after done"
assert not bool(dut.result_valid_o.value), "valid should be low after done"
assert int(dut.result_rank_o.value) == 0, "rank should reset to 0"

View File

@@ -0,0 +1,9 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := topk_tracker
COCOTB_TEST_MODULES := tests.modules.topk_tracker.test_topk_tracker
VERILOG_SOURCES := $(RTL_ROOT)/core/topk_tracker.sv
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

@@ -0,0 +1 @@
# Top-K tracker module tests

View File

@@ -0,0 +1,133 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
# ── Parameter defaults (matching cam_params.svh with NUM_ROWS=4096, HASH_BITS=512) ──
K = 4
ROW_BITS = 12
SCORE_BITS = 10 # $clog2(512 + 1)
ROW_SENTINEL = (1 << ROW_BITS) - 1 # all-ones
def golden(candidates, k=K):
"""Return top-k candidates sorted by score desc, then row asc (higher score wins; lower row breaks ties)."""
return sorted(candidates, key=lambda item: (-item[1], item[0]))[:k]
def unpack(dut, k=K):
"""Read packed topk_rows_o and topk_scores_o; return list of (row, score) tuples ordered by rank."""
rows_packed = int(dut.topk_rows_o.value)
scores_packed = int(dut.topk_scores_o.value)
result = []
for i in range(k):
row = (rows_packed >> (i * ROW_BITS)) & ((1 << ROW_BITS) - 1)
score = (scores_packed >> (i * SCORE_BITS)) & ((1 << SCORE_BITS) - 1)
result.append((row, score))
return result
# ── Helpers ──────────────────────────────────────────────────────────────────────
async def reset_tracker(dut):
"""Assert reset for 5 cycles, release, then wait 2 cycles."""
dut.rst_n.value = 0
dut.clear_i.value = 0
dut.candidate_valid_i.value = 0
for _ in range(5):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
async def push_candidate(dut, row: int, score: int):
"""Drive one candidate handshake (candidate_ready_o is always 1)."""
dut.candidate_valid_i.value = 1
dut.candidate_row_i.value = row
dut.candidate_score_i.value = score
await RisingEdge(dut.clk)
dut.candidate_valid_i.value = 0
# ── Test 1: Basic ordering ───────────────────────────────────────────────────────
@cocotb.test()
async def tracker_orders_by_score_then_lower_row(dut):
"""
Push candidates [(5,10), (3,11), (2,11), (9,7), (1,11), (0,3)] and verify
the tracker's Top-4 matches golden: [(1,11), (2,11), (3,11), (5,10)].
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_tracker(dut)
candidates = [(5, 10), (3, 11), (2, 11), (9, 7), (1, 11), (0, 3)]
for row, score in candidates:
await push_candidate(dut, row, score)
result = unpack(dut)
expected = golden(candidates)
assert result == expected, f"Expected {expected}, got {result}"
assert dut.update_pending_o.value == 0, "update_pending_o should be 0"
# ── Test 2: Clear restarts query ─────────────────────────────────────────────────
@cocotb.test()
async def tracker_clear_starts_new_query(dut):
"""
After candidate (7,60), pulse clear_i, send candidate (4,12).
Expect rank 0 = (4,12), remaining slots = sentinel rows, score 0.
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_tracker(dut)
# Push (7, 60)
await push_candidate(dut, 7, 60)
# Pulse clear
dut.clear_i.value = 1
await RisingEdge(dut.clk)
dut.clear_i.value = 0
# Push (4, 12)
await push_candidate(dut, 4, 12)
result = unpack(dut)
expected = [(4, 12)] + [(ROW_SENTINEL, 0)] * (K - 1)
assert result == expected, f"Expected {expected}, got {result}"
# ── Test 3: Random validation against golden ─────────────────────────────────────
@cocotb.test()
async def tracker_matches_random_python_golden(dut):
"""
Push 100 candidates (row in [0,63], score in [0,127]) seeded at 42.
Verify final Top-4 matches Python golden model.
"""
import random
random.seed(42)
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_tracker(dut)
max_row = 64
max_score = 128
candidates = []
for _ in range(100):
row = random.randint(0, max_row - 1)
score = random.randint(0, max_score - 1)
candidates.append((row, score))
await push_candidate(dut, row, score)
result = unpack(dut)
expected = golden(candidates)
assert result == expected, f"Expected {expected}, got {result}"

View File

@@ -0,0 +1,12 @@
SIM_ROOT := $(abspath ../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_top
COCOTB_TEST_MODULES := tests.perf.test_cam_perf
VERILOG_SOURCES := $(RTL_CAM_TOP)
HASH_BITS ?= 512
WRITE_NOISE_EN := 0
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

View File

@@ -0,0 +1,305 @@
from __future__ import annotations
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge, ReadOnly, Timer
# ── Helpers (local reimplementation — no imports from test_cam_basic) ─────────
def _get_param(dut, name, default=None):
"""Read a Verilator-exposed parameter from the DUT."""
try:
val = getattr(dut, name, None)
if val is not None:
return int(val.value)
except Exception:
pass
return default
def dut_num_rows(dut):
val = _get_param(dut, "NUM_ROWS", None)
if val is not None:
return val
return 1 << len(dut.wr_addr)
def dut_hash_bits(dut):
val = _get_param(dut, "HASH_BITS", None)
if val is not None:
return val
return len(dut.write_hash)
def dut_lanes(dut):
val = _get_param(dut, "LANES", None)
if val is not None:
return val
return len(dut.rd_resp_row_ids) // len(dut.wr_addr)
def _pipeline_depth(num_rows, lanes):
"""Number of match-engine pipeline stages (rows processed per query)."""
return num_rows // lanes
# ---------------------------------------------------------------------------
# Bounded wait helper (RisingEdge only — keep the caller writeable)
# ---------------------------------------------------------------------------
async def _wait_bounded(dut, condition_fn, max_cycles, label):
"""Wait for *condition_fn* to become ``True``, sampling at each RisingEdge.
Raises ``AssertionError`` with *label* if *max_cycles* is exceeded.
The caller is left in a writeable phase after return.
"""
for _ in range(max_cycles):
await RisingEdge(dut.clk)
if condition_fn():
return
raise AssertionError(
f"{label}: timed out after {max_cycles} clock cycles"
)
def _condition_signal_high(signal):
"""Return a nullary callable that checks *signal* is high (integer 1)."""
def _check():
return int(signal.value) == 1
return _check
# ---------------------------------------------------------------------------
# DUT helpers
# ---------------------------------------------------------------------------
async def reset_dut(dut):
"""Reset the DUT with new handshake interface.
``result_ready`` is held at 0 during measurement; the performance test
pulses it high for exactly one cycle after capturing ``result_valid``.
"""
dut.rst_n.value = 0
dut.wr_valid.value = 0
dut.wr_addr.value = 0
dut.write_hash.value = 0
dut.query_valid.value = 0
dut.query_hash.value = 0
dut.result_ready.value = 0
for _ in range(5):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
async def wait_idle(dut, max_cycles=10):
"""Wait until both wr_ready=1 and query_ready=1 (system fully idle)."""
await _wait_bounded(
dut,
lambda: int(dut.wr_ready.value) and int(dut.query_ready.value),
max_cycles,
"wait_idle",
)
async def write_row(dut, addr, value, max_cycles=10):
"""Write a single row using wr_valid/wr_ready handshake."""
await wait_idle(dut)
dut.wr_addr.value = addr
dut.write_hash.value = int(value)
dut.wr_valid.value = 1
await _wait_bounded(
dut,
_condition_signal_high(dut.wr_ready),
max_cycles,
f"write_row(addr={addr}) handshake",
)
dut.wr_valid.value = 0
await wait_idle(dut)
async def write_rows(dut, rows):
"""Write all rows sequentially."""
for idx, value in enumerate(rows):
await write_row(dut, idx, value)
async def query_once_with_latency(dut, query, max_result_cycles):
"""Issue a query and return (top1_index, top1_score, latency_cycles, total_cycles).
Parameters
----------
max_result_cycles : int
Hard bound on cycles from query acceptance to *result_valid*.
Derived from ``NUM_ROWS / LANES + pipeline_slack`` by the caller.
Cycle counting
--------------
``query_ready`` is a combinational signal that goes low *immediately* after
the RisingEdge at which the query is accepted (the state machine
transitions from S_IDLE to S_SCAN). We therefore sample ``query_ready``
**before** ``ReadOnly`` (i.e. at the RisingEdge time-point itself) to
capture the handshake.
``result_valid`` is a registered output that stays high until consumed, so
we sample it in the **settled phase** after ``ReadOnly``.
``latency_cycles`` is the number of RisingEdge events between the cycle
where the query is accepted and the cycle where ``result_valid`` is
observed.
"""
await wait_idle(dut)
edge_count = 0
dut.query_hash.value = int(query)
dut.query_valid.value = 1
# ── Phase 1: Accept handshake ───────────────────────────────────────
# query_ready is combinational — sample before ReadOnly.
accept_edge = None
for _ in range(10):
await RisingEdge(dut.clk)
edge_count += 1
q_ready = int(dut.query_ready.value) # sample before state transition
await ReadOnly()
await Timer(1, units="step") # exit ReadOnly for driving
if q_ready:
accept_edge = edge_count
break
assert accept_edge is not None, (
"Query accept handshake timed out after 10 cycles"
)
dut.query_valid.value = 0
# ── Phase 2: Wait for result_valid (measurement window) ─────────────
# result_valid is registered — sample after ReadOnly is fine.
result_edge = None
for _ in range(max_result_cycles):
await RisingEdge(dut.clk)
edge_count += 1
await ReadOnly()
if int(dut.result_valid.value):
result_edge = edge_count
break
assert result_edge is not None, (
f"Query result_valid timed out after {max_result_cycles} cycles "
f"(accepted at edge {accept_edge})"
)
await Timer(1, units="step")
latency_cycles = result_edge - accept_edge
top1_index = int(dut.top1_index.value)
top1_score = int(dut.top1_score.value)
# ── Phase 3: Consume result (pulse result_ready for one cycle) ──────
dut.result_ready.value = 1
await RisingEdge(dut.clk)
edge_count += 1
await ReadOnly()
await Timer(1, units="step")
dut.result_ready.value = 0
return top1_index, top1_score, latency_cycles, edge_count
def deterministic_rows(num_rows, hash_bits, query_hash):
"""Create deterministic rows where only row 0 stores *query_hash*."""
mask = (1 << hash_bits) - 1
rows = [0] * num_rows
rows[0] = query_hash
for i in range(1, num_rows):
# Deterministic non-matching value; golden-ratio-like spread
val = ((i + 1) * 0x9E3779B97F4A7C15) & mask
if val == query_hash or val == 0:
val = ((val ^ query_hash) ^ 0xA5A5A5A5A5A5A5A5) & mask
if val == query_hash or val == 0:
val = (val ^ 1) & mask
rows[i] = val
return rows
# ── Performance Test ──────────────────────────────────────────────────────────
@cocotb.test()
async def cam_perf_benchmark(dut):
"""Performance benchmark: measure query latency in cycles."""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
lanes = dut_lanes(dut)
write_noise_en = _get_param(dut, "WRITE_NOISE_EN", 1)
# ── Deterministic query ─────────────────────────────────────────────
query_hash = 0xAA55_AA55_AA55_AA55_AA55_AA55_AA55_AA55
query_hash &= (1 << hash_bits) - 1
if query_hash == 0:
query_hash = 1
rows = deterministic_rows(num_rows, hash_bits, query_hash)
await write_rows(dut, rows)
# Bound: pipeline depth plus generous slack for read + drain stages
pipeline = _pipeline_depth(num_rows, lanes)
max_result_cycles = pipeline + 30
top1_index, top1_score, latency_cycles, total_cycles = (
await query_once_with_latency(dut, query_hash, max_result_cycles)
)
# ── Performance assertions ──────────────────────────────────────────
assert latency_cycles > 0, (
f"latency_cycles must be positive, got {latency_cycles}"
)
assert total_cycles > 0, (
f"total_cycles must be positive, got {total_cycles}"
)
# ── Correctness assertions (conditional on noise state) ─────────────
if not write_noise_en:
# Without noise: stored hash at row 0 == query_hash → exact match.
assert top1_index == 0, (
f"Noise disabled: expected top1_index=0 (exact match), got "
f"{top1_index}"
)
assert top1_score == hash_bits, (
f"Noise disabled: expected top1_score={hash_bits}, got "
f"{top1_score}"
)
else:
# With noise: write flip masks may corrupt stored values, so
# we cannot reliably assert the exact match. Instead, confirm a
# valid non-zero score was produced (the match engine ran).
assert top1_score > 0, (
f"Noise enabled: expected top1_score > 0, got {top1_score}. "
"Match engine returned invalid result."
)
dut._log.info(
"Noise enabled (WRITE_NOISE_EN=%s) — "
"skipping exact top1_index/top1_score assertion. "
"top1_index=%d top1_score=%d",
write_noise_en, top1_index, top1_score,
)
# ── Machine-readable performance marker ─────────────────────────────
queries_per_cycle = 1.0 / total_cycles
dut._log.info(
"PERF_RESULT latency_cycles=%d total_cycles=%d "
"accepted_queries=1 completed_queries=1 "
"queries_per_cycle=%.6f status=pass",
latency_cycles, total_cycles, queries_per_cycle,
)

View File

View File

@@ -0,0 +1,12 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_top
COCOTB_TEST_MODULES := tests.top.no_noise.test_no_noise
VERILOG_SOURCES := $(RTL_CAM_TOP)
# 禁用所有噪声模块
WRITE_NOISE_EN := 0
include $(SIM_ROOT)/mk/cocotb-common.mk

View File

@@ -0,0 +1 @@
# CAM top-level no-noise integration tests

View File

@@ -0,0 +1,350 @@
# -*- coding: utf-8 -*-
"""
CAM 顶层cam_topno_noise 配置集成测试WRITE_NOISE_EN=0
所有噪声模块禁用,验证 CAM 在无噪声下的标准行为。
=== 测试清单 ===
- compile_includes_grouped_noise_helper — 编译冒烟
- baseline_no_noise — 基线检索正确性
- known_hamming_distance — 汉明距离验证
- tie_break_policy — 平局决胜
- all_zero_all_one_boundary — 全 0 / 全 1 边界
- half_duplex_write_priority — 半双工写入优先
- banked_pipeline_no_noise_top1 — 分块流水线 Top-1
- query_scan_blocks_writes_until_result_consumed — 查询阻塞写入
=== 配置背景 ===
本目录固定使用 WRITE_NOISE_EN=0 编译,
因此所有测试无需运行时参数门控——Makefile 保证配置正确。
"""
from __future__ import annotations
import cocotb
import numpy as np
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
from model.ref_model import (
match_top1,
random_hashes,
)
from tests.top.utils import (
collect_topk,
dut_hash_bits,
dut_lanes,
dut_num_rows,
query_once,
query_topk_once,
reset_dut,
wait_idle,
write_row,
write_rows,
)
# ═══════════════════════════════════════════════════════════════════════════════
# 编译冒烟测试 — 验证 cam_top 能正确 elaborate
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def compile_includes_grouped_noise_helper(dut):
"""编译测试:验证群组噪声辅助模块被正确包含在 cam_top 中。
不验证功能,只确保 Verilator elaboration 不会报错。
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
assert int(dut.wr_ready.value) in (0, 1)
# ═══════════════════════════════════════════════════════════════════════════════
# 测试 A基线WRITE_NOISE_EN=0
# ── 验证写+查在噪声关闭时与旧 CAM 行为完全一致
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def baseline_no_noise(dut):
"""基线测试:噪声全部关闭时,检索结果必须与 Python 参考模型完全一致。
验证内容:
1. Top-1 索引和分数与 match_top1 模型一致
2. 查询自身所在行 → 分数必须等于 HASH_BITS完全匹配
3. 串行 Top-K 流的第一个 beat rank=0
4. 最后一个 beat 的 result_last=1流终止
5. top1_index/top1_score 别名与第一个 beat 的值一致
6. score_debug如果存在与模型分数数组逐元素一致
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rng = np.random.default_rng(1)
rows = random_hashes(rng, num_rows, width=hash_bits)
query_index = min(123, num_rows - 1)
query = rows[query_index]
await write_rows(dut, rows)
beats, top1_index, top1_score, score_debug = await query_topk_once(dut, query)
expected = match_top1(query, rows, width=hash_bits)
assert top1_index == expected.top1_index
assert top1_score == expected.top1_score
assert top1_index == query_index
assert top1_score == hash_bits
# Serial Top-K stream verification: beats from the single query above
assert beats[0][0] == 0, "First beat must have rank 0"
assert beats[-1][3] == 1, "Last beat must assert result_last"
# Verify top1 aliases match first beat after stream fully consumed
assert int(dut.top1_index.value) == beats[0][1]
assert int(dut.top1_score.value) == beats[0][2]
# Verify returned top1 matches first beat rank0
assert top1_index == beats[0][1]
assert top1_score == beats[0][2]
if score_debug is not None:
assert np.array_equal(score_debug, expected.scores)
# ═══════════════════════════════════════════════════════════════════════════════
# 遗留测试 — 仅在噪声关闭时有意义(精确分数才有效)
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def known_hamming_distance(dut):
"""汉明距离验证:写入已知模式的哈希,验证分数计算正确。
测试数据:
- Row 0: 全 0
- Row 10: 低 7 位为 1 → score = hash_bits - 7
- Row 11: 低 31 位为 1 → score = hash_bits - 31
- Row 12: 低 128 位为 1→ score = hash_bits - 128
- query = 0
验证Top-1 为 row 0完全匹配各行的 score_debug
精确等于理论汉明距离对应的匹配位数。
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
query = 0
rows = [0] * num_rows
rows[min(10, num_rows - 1)] = (1 << 7) - 1
rows[min(11, num_rows - 1)] = (1 << 31) - 1
rows[min(12, num_rows - 1)] = (1 << 128) - 1
await write_rows(dut, rows)
top1_index, top1_score, score_debug = await query_once(dut, query)
assert top1_index == 0
assert top1_score == hash_bits
if score_debug is not None:
assert int(score_debug[min(10, num_rows - 1)]) == hash_bits - 7
assert int(score_debug[min(11, num_rows - 1)]) == hash_bits - 31
assert int(score_debug[min(12, num_rows - 1)]) == hash_bits - 128
@cocotb.test()
async def tie_break_policy(dut):
"""平局决胜策略:分数相同时,行号最小的获胜。
设置:
- row 10, 20, 200 都存储了 query 的值(满分匹配)
- 其余行随机填充
预期top1_index = 10不是 20 或 200
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rng = np.random.default_rng(2)
rows = random_hashes(rng, num_rows, width=hash_bits)
query = rows[min(200, num_rows - 1)]
rows[10] = query
rows[20] = query
rows[min(200, num_rows - 1)] = query
await write_rows(dut, rows)
top1_index, top1_score, _ = await query_once(dut, query)
assert top1_index == 10
assert top1_score == hash_bits
@cocotb.test()
async def all_zero_all_one_boundary(dut):
"""全 0 / 全 1 边界测试:验证极端哈希值的检索正确性。
存储:
- row 0: 全 0 (0x000...000)
- row 1: 全 1 (0xFFF...FFF)
- 查询 = 全 0
预期Top-1 = row 0, score = hash_bits
row 1 的 score = 0全 0 与全 1 无任何匹配位)
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rows = [0] * num_rows
rows[0] = 0
rows[1] = (1 << hash_bits) - 1
query = 0
await write_rows(dut, rows)
top1_index, top1_score, score_debug = await query_once(dut, query)
assert top1_score == hash_bits
assert top1_index == 0
if score_debug is not None:
assert int(score_debug[0]) == hash_bits
assert int(score_debug[1]) == 0
# ═══════════════════════════════════════════════════════════════════════════════
# 测试 F半双工写入优先级仲裁
# ── wr_valid 和 query_valid 同时有效 → 写入优先,查询被暂缓
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def half_duplex_write_priority(dut):
"""半双工仲裁:同时发起写入和查询 → 写入必须胜出。
流程:
1. 预先写入 row 0值为 test_val
2. 同时驱动 wr_valid 和 query_valid写入 row 1查询 test_val
3. 断言写入被接受row 1 被正确写入
4. 随后查询 test_val → 应返回 row 0 和 row 1 都是满分
如果仲裁逻辑错误(查询胜出或同时处理),两个行中至少有一个会丢失。
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
hash_bits = dut_hash_bits(dut)
test_val = (1 << hash_bits) - 1
await write_row(dut, 0, test_val)
await wait_idle(dut)
assert int(dut.wr_ready.value) == 1
assert int(dut.query_ready.value) == 1
dut.wr_valid.value = 1
dut.wr_addr.value = 1
dut.write_hash.value = 0
dut.query_valid.value = 1
dut.query_hash.value = test_val
await RisingEdge(dut.clk)
dut.wr_valid.value = 0
dut.query_valid.value = 0
await wait_idle(dut)
top1_index, top1_score, _ = await query_once(dut, test_val)
assert top1_index == 0
assert top1_score == hash_bits
top1_index, top1_score, _ = await query_once(dut, 0)
assert top1_index == 1
assert top1_score == hash_bits
# ═══════════════════════════════════════════════════════════════════════════════
# 测试 G分块流水线无噪声 Top-1
# ── 验证分块存储架构在无噪声时返回正确的 Top-1
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def banked_pipeline_no_noise_top1(dut):
"""分块流水线 Top-1无噪声时分块架构的结果必须与纯模型一致。
这是 banked_pipeline 的冒烟测试——验证分块存储核心、
通道合并逻辑、以及匹配引擎流水线在端到端场景中协同工作。
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rng = np.random.default_rng(7)
rows = random_hashes(rng, num_rows, width=hash_bits)
query_index = min(17, num_rows - 1)
query = rows[query_index]
await write_rows(dut, rows)
top1_index, top1_score, score_debug = await query_once(dut, query)
expected = match_top1(query, rows, width=hash_bits)
assert top1_index == expected.top1_index
assert top1_score == expected.top1_score
assert top1_index == query_index
# ═══════════════════════════════════════════════════════════════════════════════
# 测试 H查询扫描期间阻塞写入
# ── 活跃的查询扫描会撤销 wr_ready直到结果被消费完毕
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def query_scan_blocks_writes_until_result_consumed(dut):
"""半双工阻塞查询扫描活跃期间wr_ready 必须保持低电平。
流程:
1. 写入全部行
2. 发起查询query_valid=1
3. 在结果被消费之前,尝试写入 → 断言 wr_ready=0
4. 消费完整结果流 → DUT 回到空闲
这验证了半双工协议中「读期间禁止写」的约束。
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rows = [0] * num_rows
rows[0] = (1 << hash_bits) - 1
await write_rows(dut, rows)
await wait_idle(dut)
dut.query_hash.value = rows[0]
dut.query_valid.value = 1
await RisingEdge(dut.clk)
dut.query_valid.value = 0
dut.wr_valid.value = 1
dut.wr_addr.value = 1
dut.write_hash.value = 0
await RisingEdge(dut.clk)
assert int(dut.wr_ready.value) == 0
dut.wr_valid.value = 0
# Consume full serial stream so the DUT returns idle
beats = await collect_topk(
dut, timeout_cycles=max(2000, (dut_num_rows(dut) // dut_lanes(dut)) * 24 + 100)
)
assert len(beats) > 0
assert beats[-1][3] == 1 # last asserted

View File

@@ -0,0 +1,15 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_top
COCOTB_TEST_MODULES := tests.top.read_noise.test_read_noise
VERILOG_SOURCES := $(RTL_CAM_TOP)
# 读取路径 pass-through 配置,写入噪声默认关闭
WRITE_NOISE_EN := 0
include $(SIM_ROOT)/mk/cocotb-common.mk
clean::
rm -rf sim_build

View File

@@ -0,0 +1 @@
# CAM top-level read-noise integration tests

View File

@@ -0,0 +1,54 @@
# -*- coding: utf-8 -*-
"""
CAM 读取路径 pass-through 集成测试 — Phase 2 cleaned.
Read noise 已退休cam_read_noise 是纯 pass-through。
本测试验证查询返回的 scores 与 pure matching 一致。
"""
from __future__ import annotations
import cocotb
import numpy as np
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
from model.ref_model import (
match_top1,
random_hashes,
unpack_score_debug_flat,
)
from tests.top.utils import (
collect_topk,
dut_hash_bits,
dut_lanes,
dut_num_rows,
get_param,
query_once,
query_topk_once,
reset_dut,
write_row,
write_rows,
)
@cocotb.test()
async def read_path_pass_through_produces_pure_matching(dut):
"""写 4 行,查询存过的行,验证 Top-1/Top-K 与 pure matching 一致。"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rng = np.random.default_rng(42)
rows = random_hashes(rng, num_rows, width=hash_bits)
await write_rows(dut, rows)
query = rows[min(50, num_rows - 1)]
top1_index, top1_score, score_debug = await query_once(dut, query)
expected = match_top1(query, rows, width=hash_bits)
assert top1_index == expected.top1_index
assert top1_score == expected.top1_score
if score_debug is not None:
assert np.array_equal(score_debug, expected.scores)

341
hw/sim/tests/top/utils.py Normal file
View File

@@ -0,0 +1,341 @@
# -*- coding: utf-8 -*-
"""
CAM 顶层测试的共享辅助函数。
被 no_noise/、write_noise/、read_noise/ 等配置目录的测试文件共同引用。
提供:
- Verilator 参数读取(带回退推断)
- DUT 复位、空闲等待
- 行写入 / 批量写入(握手协议)
- 查询发起 / Top-K 结果流收集
- score_debug 解包SIM_DEBUG 模式)
所有函数都是 async调用者需要处于 cocotb 协程上下文中。
"""
from __future__ import annotations
import numpy as np
from cocotb.triggers import ReadOnly, RisingEdge, Timer
from model.ref_model import ( # noqa: E402
match_top1,
unpack_score_debug_flat,
)
# ── 默认拓扑参数(当 Verilator 参数不可用时使用) ───────────────────────────
DEFAULT_NUM_ROWS = 4096
DEFAULT_HASH_BITS = 512
DEFAULT_LANES = 8
DEFAULT_SCORE_BITS = 10
# ═══════════════════════════════════════════════════════════════════════════════
# 参数读取(从 Verilator 参数或信号宽度推断)
# ═══════════════════════════════════════════════════════════════════════════════
def get_param(dut, name, default=None):
"""从 DUT 读取 Verilator 暴露的参数值,失败则返回 default。
某些参数仅在特定编译配置下暴露(如 SIM_DEBUG 下的 score_debug_flat
此时回退到 default 是预期行为而非错误。
"""
try:
val = getattr(dut, name, None)
if val is not None:
return int(val.value)
except Exception:
pass
return default
def dut_num_rows(dut):
"""获取 NUM_ROWS优先读参数否则从 wr_addr 位宽推断 (NUM_ROWS = 2^ROW_BITS)。"""
val = get_param(dut, "NUM_ROWS", None)
if val is not None:
return val
return 1 << len(dut.wr_addr)
def dut_hash_bits(dut):
"""获取 HASH_BITS优先读参数否则从 write_hash 信号位宽推断。"""
val = get_param(dut, "HASH_BITS", None)
if val is not None:
return val
return len(dut.write_hash)
def dut_lanes(dut):
"""获取 LANES优先读参数否则从 rd_resp_row_ids / wr_addr 位宽推断。"""
val = get_param(dut, "LANES", None)
if val is not None:
return val
return len(dut.rd_resp_row_ids) // len(dut.wr_addr)
def dut_score_bits(dut):
"""获取 SCORE_BITS优先读参数否则从 top1_score 信号位宽推断。"""
val = get_param(dut, "SCORE_BITS", None)
if val is not None:
return val
return len(dut.top1_score)
# ═══════════════════════════════════════════════════════════════════════════════
# 协议层辅助函数(复位 / 空闲等待 / 行写入 / 查询)
# ═══════════════════════════════════════════════════════════════════════════════
async def reset_dut(dut):
"""复位 DUTrst_n 拉低 5 周期,释放后再等 2 周期。
所有控制信号在复位期间保持无效电平。
result_ready 初始化为 1准备接收结果
"""
dut.rst_n.value = 0
dut.wr_valid.value = 0
dut.wr_addr.value = 0
dut.write_hash.value = 0
dut.query_valid.value = 0
dut.query_hash.value = 0
dut.result_ready.value = 1
for _ in range(5):
await RisingEdge(dut.clk)
dut.rst_n.value = 1
for _ in range(2):
await RisingEdge(dut.clk)
async def wait_idle(dut):
"""等待 DUT 完全空闲wr_ready=1 且 query_ready=1。
这是发起新操作前的前置条件——CAM 是半双工的,
同一时刻只能进行写入或查询。
"""
while not (int(dut.wr_ready.value) and int(dut.query_ready.value)):
await RisingEdge(dut.clk)
async def write_row(dut, addr, value):
"""写入单行:使用 wr_valid/wr_ready 握手协议。
流程:等待空闲 → 驱动地址和数据 → 等待握手完成 → 等待写流水线排空。
"""
await wait_idle(dut)
dut.wr_addr.value = addr
dut.write_hash.value = int(value)
dut.wr_valid.value = 1
while True:
await RisingEdge(dut.clk)
if int(dut.wr_ready.value):
break
dut.wr_valid.value = 0
await wait_idle(dut)
async def write_rows(dut, rows):
"""按顺序写入所有行(行索引 = 数组下标)。"""
for idx, value in enumerate(rows):
await write_row(dut, idx, value)
async def collect_topk(dut, timeout_cycles: int = 2000):
"""收集串行 Top-K 结果流的所有 beat。
保持 result_ready=1逐个时钟周期采样 result_valid
直到 result_last 被断言。
返回:[(rank, row, score, last), ...] 列表
超时则抛出 AssertionError。
注意:此函数会「消耗」整个结果流,调用后 DUT 回到空闲状态。
"""
dut.result_ready.value = 1
beats = []
for _ in range(timeout_cycles):
if int(dut.result_valid.value):
rank = int(dut.result_rank.value)
row = int(dut.result_row.value)
score = int(dut.result_score.value)
last = int(dut.result_last.value)
beats.append((rank, row, score, last))
if last:
return beats
await RisingEdge(dut.clk)
raise AssertionError("Top-K result stream did not finish")
# ── 默认超时估算 ──────────────────────────────────────────────────
def dut_query_timeout_cycles(dut):
"""基于 DUT 参数估算完整查询(扫描 + 串行结果输出)的超时周期数。
各通道串行输出一个 beat 需要约 24 个流水线周期;
总超时 = ceil(全部行数 / 通道数) * 24 + 固定裕量 2000 周期。
至少返回 2000 周期以防止极小配置下的不合理值。
Example: 4096 rows / 8 lanes → ceil(4096/8) * 24 + 2000 = 14288 cycles.
"""
num_rows = dut_num_rows(dut)
lanes = dut_lanes(dut)
batches = (num_rows + lanes - 1) // lanes # ceil division, no math import
return max(2000, batches * 24 + 2000)
async def query_topk_once(dut, query, timeout_cycles=None):
"""发起一次查询并收集完整的串行 Top-K 结果流。
完整流程:
1. 等待 DUT 空闲
2. 等待 query_ready 为高
3. 通过 query_valid/query_ready 握手发送查询
4. 消费完整的结果流
5. 读取 score_debug_flat如果存在
返回:(beats, top1_index, top1_score, score_debug)
- beats: [(rank, row, score, last), ...]
- score_debug: np.ndarray 或 NoneSIM_DEBUG 模式)
"""
beats, top1_index, top1_score, score_debug, _ = await query_topk_once_with_latency(
dut, query, timeout_cycles=timeout_cycles,
)
return beats, top1_index, top1_score, score_debug
async def query_topk_once_with_latency(dut, query, timeout_cycles=None):
"""发起一次查询、收集完整 Top-K 结果流,并返回周期计数。
返回:(beats, top1_index, top1_score, score_debug, timing)
``timing`` 字段:
- accept_to_first_result_cycles: query 接受到首个 result_valid beat
- accept_to_last_result_cycles: query 接受到 result_last beatTop-K 完成)
- total_query_cycles: 纯查询事务周期,等于 accept_to_last_result_cycles
``query_ready`` 是组合信号,握手周期在上升沿前采样;结果信号在
ReadOnly settled phase 采样,避免重新引入 query_ready 采样时序问题。
"""
await wait_idle(dut)
dut.query_hash.value = int(query)
# 等待 query_ready 为高DUT 已就绪),避免组合逻辑下降沿导致的
# valid&&ready 握手丢失问题。
while not int(dut.query_ready.value):
await RisingEdge(dut.clk)
edge_count = 0
dut.query_valid.value = 1
dut.result_ready.value = 1
await RisingEdge(dut.clk)
edge_count += 1
q_valid = int(dut.query_valid.value)
q_ready = int(dut.query_ready.value)
assert q_ready, "Query accept handshake was missed despite query_ready pre-wait"
assert q_valid, "Query valid deasserted before accept handshake"
accept_edge = edge_count
dut.query_valid.value = 0
if timeout_cycles is None:
timeout_cycles = dut_query_timeout_cycles(dut)
beats = []
first_result_edge = None
last_result_edge = None
for _ in range(timeout_cycles):
await RisingEdge(dut.clk)
edge_count += 1
await ReadOnly()
result_valid = int(dut.result_valid.value)
result_ready = int(dut.result_ready.value)
if result_valid and result_ready:
if first_result_edge is None:
first_result_edge = edge_count
rank = int(dut.result_rank.value)
row = int(dut.result_row.value)
score = int(dut.result_score.value)
last = int(dut.result_last.value)
beats.append((rank, row, score, last))
if last:
last_result_edge = edge_count
await Timer(1, units="step")
break
await Timer(1, units="step")
if first_result_edge is None or last_result_edge is None:
raise AssertionError("Top-K result stream did not finish")
num_rows = dut_num_rows(dut)
score_bits = dut_score_bits(dut)
score_debug = None
if hasattr(dut, "score_debug_flat"):
score_debug = unpack_score_debug_flat(
int(dut.score_debug_flat.value),
num_rows,
score_bits,
)
timing = {
"accept_to_first_result_cycles": int(first_result_edge - accept_edge),
"accept_to_last_result_cycles": int(last_result_edge - accept_edge),
"total_query_cycles": int(last_result_edge - accept_edge),
}
return beats, beats[0][1], beats[0][2], score_debug, timing
async def query_once(dut, query, timeout_cycles=None):
"""发起查询,返回 (top1_index, top1_score, score_debug)。
内部调用 query_topk_once 并消费完整结果流,仅保留 rank-0 数据。
Parameters
----------
timeout_cycles : int or None
传递给 query_topk_once 的超时周期数。None 表示根据 DUT 参数动态估算。
"""
_, top1_index, top1_score, score_debug = await query_topk_once(
dut, query, timeout_cycles=timeout_cycles,
)
return top1_index, top1_score, score_debug
# ═══════════════════════════════════════════════════════════════════════════════
# 便捷验证函数
# ═══════════════════════════════════════════════════════════════════════════════
def assert_baseline_top1(query, rows, top1_index, top1_score, hash_bits,
query_index, score_debug=None):
"""验证无噪声场景下的基线 Top-1 结果。
检查项:
1. Top-1 与 match_top1 参考模型一致
2. 查询自身所在行 → score == hash_bits完全匹配
3. score_debug 数组(如果存在)与模型一致
"""
expected = match_top1(query, rows, width=hash_bits)
assert top1_index == expected.top1_index, (
f"top1_index mismatch: {top1_index} != {expected.top1_index}"
)
assert top1_score == expected.top1_score, (
f"top1_score mismatch: {top1_score} != {expected.top1_score}"
)
assert top1_index == query_index, (
f"Expected query_index={query_index} to match self, got top1_index={top1_index}"
)
assert top1_score == hash_bits, (
f"Self-query should score {hash_bits}, got {top1_score}"
)
if score_debug is not None:
assert np.array_equal(score_debug, expected.scores), (
"score_debug does not match model scores"
)

View File

@@ -0,0 +1,28 @@
SIM_ROOT := $(abspath ../../..)
RTL_ROOT := $(abspath $(SIM_ROOT)/../rtl)
include $(SIM_ROOT)/mk/rtl-sources.mk
TOPLEVEL := cam_top
COCOTB_TEST_MODULES := tests.top.write_noise.test_write_noise
VERILOG_SOURCES := $(RTL_CAM_TOP)
# 写入噪声 ~1% 默认速率
WRITE_NOISE_EN := 1
WRITE_NOISE_RATE_NUM := 1
WRITE_NOISE_RATE_DEN := 100
include $(SIM_ROOT)/mk/cocotb-common.mk
# ── 速率变体子目标 ─────────────────────────────────────────────────
.PHONY: test-zero-rate test-full-rate
test-zero-rate:
$(MAKE) -B -f Makefile results.xml WRITE_NOISE_RATE_NUM=0 \
COCOTB_TEST_FILTER=zero_rate_noise
test-full-rate:
$(MAKE) -B -f Makefile results.xml WRITE_NOISE_RATE_NUM=1 WRITE_NOISE_RATE_DEN=1 \
COCOTB_TEST_FILTER=full_rate_noise
clean::
rm -rf sim_build

View File

@@ -0,0 +1 @@
# CAM top-level write-noise integration tests

View File

@@ -0,0 +1,133 @@
# -*- coding: utf-8 -*-
"""
CAM 写入噪声Write Noise集成测试 —— 专用配置。
本文件测试 WRITE_NOISE_EN=1 配置下,
写入噪声模块的正确性。默认噪声率约 1%NUM=1, DEN=100
=== 测试列表 ===
1. default_noise_reproducible — 固定种子 = 确定性噪声,两次运行结果一致
2. exact_noise_model_match — RTL 存储的哈希与 ref_model.py 的 PRNG 掩码逐位匹配
3. zero_rate_noise — 写入噪声模块连接但 RATE_NUM=0 → 无翻转
4. full_rate_noise — 100% 写入噪声率,与 Python 模型对比
=== 架构背景 ===
写入噪声流水线位置Write Noise → Banked Core Storage → Match Engine
本测试覆盖完整的 cam_top 链路,写入噪声为唯一活跃噪声源。
=== Makefile 子目标 ===
test-zero-rate : make test-zero-rate (WRITE_NOISE_RATE_NUM=0)
test-full-rate : make test-full-rate (WRITE_NOISE_RATE_NUM=1, RATE_DEN=1, SIM_DEBUG)
"""
from __future__ import annotations
import cocotb
import numpy as np
from cocotb.clock import Clock
from cocotb.triggers import RisingEdge
from model.ref_model import (
match_top1,
random_hashes,
)
from tests.top.utils import (
collect_topk,
dut_hash_bits,
dut_num_rows,
get_param,
query_once,
query_topk_once,
reset_dut,
wait_idle,
write_row,
write_rows,
)
# ═══════════════════════════════════════════════════════════════════════════════
# 测试 1默认噪声 ~1%、可复现
# ── 固定种子 → 两次相同写入产生相同结果
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def default_noise_reproducible(dut):
"""可复现性测试:相同种子、相同数据 → 两次独立运行结果一致。
流程:
1. 写入全部行,查询 row 50 → 记录 top1_index 和 top1_score
2. 复位
3. 再次写入相同数据,查询相同行 → 记录结果
4. 断言两次结果完全一致
如果结果不一致,说明 RTL 的 PRNG 状态没有正确复位,
或存在跨运行的状态残留。
"""
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rng = np.random.default_rng(42)
rows = random_hashes(rng, num_rows, width=hash_bits)
await write_rows(dut, rows)
query = rows[min(50, num_rows - 1)]
top1_index_1, top1_score_1, _ = await query_once(dut, query)
await reset_dut(dut)
await write_rows(dut, rows)
top1_index_2, top1_score_2, _ = await query_once(dut, query)
assert top1_index_1 == top1_index_2
assert top1_score_1 == top1_score_2
# ═══════════════════════════════════════════════════════════════════════════════
# 测试 2零噪声率WRITE_NOISE_EN=1, RATE_NUM=0
# ── 噪声模块已连接但翻转概率为 0 → 行为应与无噪声一致
# ═══════════════════════════════════════════════════════════════════════════════
@cocotb.test()
async def zero_rate_noise(dut):
"""零速率噪声WRITE_NOISE_RATE_NUM=0 时不应有任何位被翻转。
这是噪声模块的边界测试——验证 RATE_NUM=0 确实禁用了翻转,
而非产生「默认速率」的噪声。
"""
rate_num = get_param(dut, "WRITE_NOISE_RATE_NUM", 1)
if rate_num != 0:
dut._log.info(
"Skipping zero_rate_noise: requires WRITE_NOISE_RATE_NUM=0."
)
return
cocotb.start_soon(Clock(dut.clk, 10, unit="ns").start())
await reset_dut(dut)
num_rows = dut_num_rows(dut)
hash_bits = dut_hash_bits(dut)
rng = np.random.default_rng(1)
rows = random_hashes(rng, num_rows, width=hash_bits)
query_index = min(123, num_rows - 1)
query = rows[query_index]
await write_rows(dut, rows)
top1_index, top1_score, score_debug = await query_once(dut, query)
expected = match_top1(query, rows, width=hash_bits)
assert top1_index == expected.top1_index
assert top1_score == expected.top1_score
assert top1_index == query_index
assert top1_score == hash_bits
if score_debug is not None:
assert np.array_equal(score_debug, expected.scores)

78
hw/syn/Makefile Normal file
View File

@@ -0,0 +1,78 @@
#===============================================================================
# CAM Top Synthesis Makefile
# Targets: synth, hier, flat, clean
# Usage:
# make -C hw/syn synth (defaults, output in build/)
# make -C hw/syn synth OUT_DIR=/some/path (mirror artifacts)
# make -C hw/syn synth NUM_ROWS=8192 (override parameter)
# make -C hw/syn hier (hierarchy-only pass)
# make -C hw/syn flat (flattened-synthesis pass)
# make -C hw/syn clean (remove build/)
#===============================================================================
# ── Configurable variables (defaults) ─────────────────────────────────────────
YOSYS ?= yosys
TOP ?= cam_top
NUM_ROWS ?= 4096
HASH_BITS ?= 512
LANES ?= 8
TOPK_K ?= 4
FIFO_DEPTH ?= 16
OUT_DIR ?= build
# ── TOP enforcement (only cam_top is supported) ───────────────────────────────
ifneq ($(TOP),cam_top)
$(error error: only TOP=cam_top is supported)
endif
# ── Internal build directory ──────────────────────────────────────────────────
BUILD_DIR := build
# ── Fixed script and artifact names ───────────────────────────────────────────
HIER_SCRIPT := synth_cam_top_hier.ys
FLAT_SCRIPT := synth_cam_top_flat.ys
HIER_LOG := yosys_hier.log
FLAT_LOG := yosys_flat.log
HIER_JSON := cam_top_hier_resources.json
FLAT_JSON := cam_top_flat_resources.json
SYNTH_V := cam_top_synth.v
ARTIFACTS := $(HIER_JSON) $(FLAT_JSON) $(SYNTH_V) $(HIER_LOG) $(FLAT_LOG)
# ── Yosys command-line defines (always passed) ────────────────────────────────
# cam_params.svh uses `ifndef guards, so -D on the Yosys CLI overrides defaults.
# Use -D NAME=value syntax (Yosys 0.62); do NOT use -define.
YOSYS_DEFINES := -D NUM_ROWS=$(NUM_ROWS) -D HASH_BITS=$(HASH_BITS) -D LANES=$(LANES) -D TOPK_K=$(TOPK_K) -D FIFO_DEPTH=$(FIFO_DEPTH)
# ── Targets ───────────────────────────────────────────────────────────────────
.PHONY: synth hier flat clean
# ── Hierarchy resource reference pass ─────────────────────────────────────────
hier:
mkdir -p $(BUILD_DIR)
$(YOSYS) $(YOSYS_DEFINES) -l $(BUILD_DIR)/$(HIER_LOG) -s $(HIER_SCRIPT)
# ── Flattened Xilinx synthesis pass ───────────────────────────────────────────
flat:
mkdir -p $(BUILD_DIR)
$(YOSYS) $(YOSYS_DEFINES) -l $(BUILD_DIR)/$(FLAT_LOG) -s $(FLAT_SCRIPT)
# ── Full synthesis: hier + flat, then mirror artifacts to OUT_DIR if different ─
synth: hier flat
@if [ "$(abspath $(OUT_DIR))" != "$(abspath $(BUILD_DIR))" ]; then \
set -e; \
echo "Mirroring artifacts to $(OUT_DIR) ..."; \
mkdir -p "$(OUT_DIR)"; \
for f in $(ARTIFACTS); do \
if [ ! -f "$(BUILD_DIR)/$$f" ]; then \
echo "error: missing artifact $(BUILD_DIR)/$$f" >&2; \
exit 1; \
fi; \
cp "$(BUILD_DIR)/$$f" "$(OUT_DIR)/$$f"; \
done; \
fi
# ── Clean ─────────────────────────────────────────────────────────────────────
clean:
rm -rf $(BUILD_DIR)

View File

@@ -0,0 +1,41 @@
# synth_cam_top_flat.ys — Flattened CAM top synthesis for Xilinx Zynq-7020 / 7-series
# Run from hw/syn: yosys -s synth_cam_top_flat.ys
# Set include directories before reading RTL
verilog_defaults -push
verilog_defaults -add -I../rtl
verilog_defaults -add -I../rtl/core
verilog_defaults -add -I../rtl/noise
verilog_defaults -add -I../rtl/random
# Read RTL sources in canonical order
read_verilog -sv -D SYNTHESIS ../rtl/random/random128.sv
read_verilog -sv -D SYNTHESIS ../rtl/noise/noise_mask_bernoulli.sv
read_verilog -sv -D SYNTHESIS ../rtl/noise/cam_write_noise.sv
read_verilog -sv -D SYNTHESIS ../rtl/noise/cam_read_noise.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/cam_core_banked.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/popcount_pipeline.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/candidate_fifo.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/topk_tracker.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/result_serializer.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/match_engine_pipeline.sv
read_verilog -sv -D SYNTHESIS ../rtl/cam_noisy.sv
read_verilog -sv -D SYNTHESIS ../rtl/cam_top.sv
# Restore verilog defaults
verilog_defaults -pop
# Flattened Xilinx synthesis (includes proc, opt, fsm, memory, etc.)
synth_xilinx -family xc7 -top cam_top -flatten
# Ensure build directory exists for output files
exec -- mkdir -p build
# Write flattened Verilog netlist
write_verilog -attr2comment build/cam_top_synth.v
# Resource statistics (JSON to file via tee, not mixed into logs)
tee -q -o build/cam_top_flat_resources.json stat -tech xilinx -json
# Final checks
check -noinit

View File

@@ -0,0 +1,43 @@
# synth_cam_top_hier.ys — Hierarchy-preserving CAM top resource reference flow
# Run from hw/syn: yosys -s synth_cam_top_hier.ys
# Set include directories before reading RTL
verilog_defaults -push
verilog_defaults -add -I../rtl
verilog_defaults -add -I../rtl/core
verilog_defaults -add -I../rtl/noise
verilog_defaults -add -I../rtl/random
# Read RTL sources in canonical order
read_verilog -sv -D SYNTHESIS ../rtl/random/random128.sv
read_verilog -sv -D SYNTHESIS ../rtl/noise/noise_mask_bernoulli.sv
read_verilog -sv -D SYNTHESIS ../rtl/noise/cam_write_noise.sv
read_verilog -sv -D SYNTHESIS ../rtl/noise/cam_read_noise.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/cam_core_banked.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/popcount_pipeline.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/candidate_fifo.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/topk_tracker.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/result_serializer.sv
read_verilog -sv -D SYNTHESIS ../rtl/core/match_engine_pipeline.sv
read_verilog -sv -D SYNTHESIS ../rtl/cam_noisy.sv
read_verilog -sv -D SYNTHESIS ../rtl/cam_top.sv
# Restore verilog defaults
verilog_defaults -pop
# Hierarchy check and elaboration
hierarchy -top cam_top -check
# Process and optimize
proc
opt
memory -nomap
# Ensure build directory exists for output files
exec -- mkdir -p build
# Resource statistics (JSON to file via tee, not mixed into logs)
tee -q -o build/cam_top_hier_resources.json stat -tech xilinx -json
# Final checks
check -noinit

View File

@@ -1,18 +1,45 @@
"""Benchmark runner for executing evaluations."""
import csv
import json
from datetime import datetime
from pathlib import Path
from typing import Any
from typing import Any, Callable, cast
import lancedb
from benchmarks.datasets import HuggingFaceDataset, LocalDataset
from benchmarks.tasks import get_task
from configs.models import BenchmarkConfig, DatasetSourceConfig
from configs.models import BenchmarkConfig, DatasetSourceConfig, ModelConfig
from rich.console import Console
from rich.table import Table
console = Console()
def _create_task(config: BenchmarkConfig, model_config: ModelConfig | None) -> Any:
"""Create benchmark task with task-specific model settings.
Args:
config: Benchmark configuration.
model_config: Optional model configuration for task-owned loading.
Returns:
Benchmark task instance.
"""
task_kwargs: dict[str, Any] = {"top_k_list": config.task.top_k_list}
if config.task.type == "retrieval" and model_config is not None:
task_kwargs.update(
{
"dino_model": model_config.dino_model,
"compression_dim": model_config.compression_dim,
"compressor_path": model_config.compressor_path,
}
)
return get_task(config.task.type, **task_kwargs)
def create_dataset(config: DatasetSourceConfig) -> Any:
"""Create a dataset instance from configuration.
@@ -44,19 +71,23 @@ def create_dataset(config: DatasetSourceConfig) -> Any:
)
def _get_table_name(config: BenchmarkConfig, model_name: str) -> str:
"""Generate database table name from config and model name.
def _get_table_name(
config: BenchmarkConfig,
model_name: str,
dataset_path: str,
) -> str:
"""Generate database table name from config, model, and dataset.
Args:
config: Benchmark configuration.
model_name: Model name for table naming.
dataset_path: Dataset identifier or path (e.g. "uoft-cs/cifar100").
Returns:
Formatted table name.
"""
prefix = config.model_table_prefix
# Use dataset path as part of table name (sanitize)
dataset_name = Path(config.dataset.path).name.lower().replace("-", "_")
dataset_name = Path(dataset_path).name.lower().replace("-", "_")
return f"{prefix}_{dataset_name}_{model_name}"
@@ -64,6 +95,7 @@ def _ensure_table(
config: BenchmarkConfig,
model_name: str,
vector_dim: int,
dataset_path: str,
) -> lancedb.table.Table:
"""Ensure the LanceDB table exists with correct schema.
@@ -71,6 +103,7 @@ def _ensure_table(
config: Benchmark configuration.
model_name: Model name for table naming.
vector_dim: Feature vector dimension.
dataset_path: Dataset identifier or path.
Returns:
LanceDB table instance.
@@ -78,7 +111,7 @@ def _ensure_table(
import pyarrow as pa
from database import db_manager
table_name = _get_table_name(config, model_name)
table_name = _get_table_name(config, model_name, dataset_path)
# Build expected schema
schema = pa.schema(
@@ -108,7 +141,11 @@ def _ensure_table(
def _print_benchmark_info(
config: BenchmarkConfig, vector_dim: int, table_name: str, table_count: int
config: BenchmarkConfig,
vector_dim: int,
table_name: str,
table_count: int,
dataset_path: str = "",
) -> None:
"""Print benchmark configuration info using Rich table.
@@ -117,12 +154,13 @@ def _print_benchmark_info(
vector_dim: Feature vector dimension.
table_name: Database table name.
table_count: Number of entries in the table.
dataset_path: Current dataset identifier or path.
"""
table = Table(title="Benchmark Configuration", show_header=False)
table.add_column("Key", style="cyan", no_wrap=True)
table.add_column("Value", style="magenta")
table.add_row("Dataset", f"{config.dataset.source_type} - {config.dataset.path}")
table.add_row("Dataset", dataset_path)
table.add_row("Model Output Dimension", str(vector_dim))
table.add_row("Table Name", table_name)
table.add_row("Table Entries", str(table_count))
@@ -130,49 +168,210 @@ def _print_benchmark_info(
console.print(table)
def _print_benchmark_results(results: dict[str, Any]) -> None:
"""Print benchmark results using Rich table.
Args:
results: Final benchmark metrics for a single dataset.
"""
table = Table(title="Benchmark Results", show_header=False)
table.add_column("Metric", style="cyan", no_wrap=True)
table.add_column("Value", style="green")
# Print recalls first
recalls = results.get("recalls", {})
for k_str, v in recalls.items():
if isinstance(v, float):
table.add_row(k_str, f"{v:.4f}")
for key, value in results.items():
if key == "recalls":
continue
if isinstance(value, (list, dict)):
continue
if isinstance(value, float):
table.add_row(key, f"{value:.4f}")
continue
table.add_row(key, str(value))
console.print(table)
def _save_benchmark_outputs(
all_results: dict[str, dict[str, Any]],
output_root: Path,
model_name: str,
) -> Path:
"""Save multi-dataset benchmark results to disk.
Writes to ``output_root/{run_id}/``:
- summary.md — overall summary with per-dataset recall table
- metrics.csv — one row per (dataset, K) combination
- {dataset_name}/predictions.csv — per-sample predictions
- {dataset_name}/confusion_matrix.csv — row-normalized (Top-1)
Args:
all_results: Dict mapping dataset name to evaluate() result dict.
output_root: Parent directory (e.g. ``outputs/benchmark``).
model_name: Model identifier for the run_id.
Returns:
Path to the created run directory.
"""
run_id = f"{datetime.now().strftime('%Y-%m-%d-%H%M%S')}-{model_name}"
out_dir = output_root / run_id
out_dir.mkdir(parents=True, exist_ok=True)
# --- metrics.csv: one row per (dataset, k) ---
csv_keys = ["model", "run_id", "dataset", "k", "recall@k", "total", "num_classes"]
all_csv_rows: list[dict[str, Any]] = []
# --- summary.md lines ---
md_lines = [
"# Benchmark Results",
"",
f"- **run_id**: `{run_id}`",
f"- **model**: `{model_name}`",
"",
"## Per-Dataset Recall",
"",
]
for dataset_name, results in all_results.items():
recalls = results.get("recalls", {})
total = results.get("total", 0)
num_classes = results.get("num_classes", 0)
md_lines.append(f"### {dataset_name}")
md_lines.append("")
md_lines.append(f"- total queries: `{total}`")
md_lines.append(f"- num_classes: `{num_classes}`")
md_lines.append("")
md_lines.append("| K | Recall@K |")
md_lines.append("|---:|---:|")
for k_str, recall_val in sorted(recalls.items()):
k = int(k_str.replace("recall@", ""))
md_lines.append(f"| {k} | {recall_val:.4f} |")
all_csv_rows.append({
"model": model_name,
"run_id": run_id,
"dataset": dataset_name,
"k": k,
"recall@k": recall_val,
"total": total,
"num_classes": num_classes,
})
md_lines.append("")
# Per-dataset subdirectory
dataset_dir = out_dir / dataset_name
dataset_dir.mkdir(parents=True, exist_ok=True)
# --- predictions.csv (per-sample) ---
query_labels = results.get("query_labels", [])
topk_labels = results.get("topk_labels", [])
if query_labels and topk_labels:
max_k = max(len(preds) for preds in topk_labels) if topk_labels else 1
fieldnames = ["idx", "true_label"] + [
f"top{i+1}_label" for i in range(max_k)
]
with (dataset_dir / "predictions.csv").open(
"w", newline="", encoding="utf-8"
) as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for idx, (true_label, preds) in enumerate(
zip(query_labels, topk_labels)
):
row_data: dict[str, Any] = {
"idx": idx,
"true_label": true_label,
}
for i in range(max_k):
row_data[f"top{i+1}_label"] = (
preds[i] if i < len(preds) else ""
)
writer.writerow(row_data)
# --- confusion_matrix.csv ---
cm = results.get("confusion_matrix")
if cm is not None:
with (dataset_dir / "confusion_matrix.csv").open(
"w", newline="", encoding="utf-8"
) as fh:
writer = csv.writer(fh)
for row in cm:
writer.writerow([f"{v:.6f}" for v in row])
# Write metrics.csv
if all_csv_rows:
with (out_dir / "metrics.csv").open("w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=csv_keys)
writer.writeheader()
for row in all_csv_rows:
writer.writerow(row)
# Write summary.md
(out_dir / "summary.md").write_text("\n".join(md_lines) + "\n", encoding="utf-8")
console.print(f"[green]Results saved to:[/green] {out_dir}")
return out_dir
def run_benchmark(
model: Any,
processor: Any,
config: BenchmarkConfig,
model_config: ModelConfig | None = None,
model_name: str = "model",
output_root: Path | None = None,
) -> dict[str, Any]:
"""Run benchmark evaluation.
"""Run benchmark evaluation across one or more datasets.
Workflow:
1. Create dataset from configuration
2. Create benchmark task from configuration
3. Build evaluation database from training set
4. Evaluate on test set
1. Load model / processor once
2. For each dataset in config.datasets:
a. Create dataset, get splits
b. Build evaluation database
c. Evaluate with all K values in top_k_list
3. Print per-dataset results
4. Save all results to disk (if ``output_root`` provided)
Args:
model: Feature extraction model.
processor: Image preprocessor.
config: Benchmark configuration.
model_config: Optional model configuration for task-owned loading.
model_name: Model name for table naming.
output_root: Optional root directory for saving results.
Returns:
Dictionary containing evaluation results.
Dict mapping dataset name to evaluation result dict.
Raises:
ValueError: If benchmark is not enabled in config.
"""
# Create dataset
console.print(
f"[cyan]Loading dataset:[/cyan] {config.dataset.source_type} - {config.dataset.path}"
)
dataset = create_dataset(config.dataset)
# ── Resolve model / processor once ──────────────────────────
task = _create_task(config, model_config)
# Get train and test splits
train_dataset = dataset.get_train_split()
test_dataset = dataset.get_test_split()
if train_dataset is None or test_dataset is None:
raise ValueError(
f"Dataset {config.dataset.path} does not have train/test splits"
resolver = getattr(task, "prepare_benchmark", None)
if callable(resolver):
prepare_benchmark = cast(
Callable[[Any, Any, str], tuple[Any, Any, str]],
resolver,
)
model, processor, model_name = prepare_benchmark(model, processor, model_name)
# Infer vector dimension from a sample
sample = train_dataset[0]
if model is None or processor is None:
raise ValueError("Benchmark task did not provide a valid model and processor")
# ── Infer vector dimension from first dataset ─────────────────
first_cfg = config.datasets[0]
first_dataset = create_dataset(first_cfg)
first_train = first_dataset.get_train_split()
sample = first_train[0]
sample_image = sample["img"]
from utils.feature_extractor import infer_vector_dim
@@ -180,27 +379,77 @@ def run_benchmark(
vector_dim = infer_vector_dim(processor, model, sample_image)
console.print(f"[cyan]Model output dimension:[/cyan] {vector_dim}")
# Ensure table exists with correct schema
table = _ensure_table(config, model_name, vector_dim)
table_name = _get_table_name(config, model_name)
# ── Evaluate each dataset ────────────────────────────────────
all_results: dict[str, dict[str, Any]] = {}
# Check if database is already built
table_count = table.count_rows()
if table_count > 0:
for dataset_cfg in config.datasets:
console.print(
f"[yellow]Table '{table_name}' already has {table_count} entries, skipping database build.[/yellow]"
f"\n[bold cyan]Dataset:[/bold cyan] "
f"{dataset_cfg.source_type} - {dataset_cfg.path}"
)
else:
# Create and run benchmark task
task = get_task(config.task.type, top_k=config.task.top_k)
dataset = create_dataset(dataset_cfg)
train_dataset = dataset.get_train_split()
test_dataset = dataset.get_test_split()
if train_dataset is None or test_dataset is None:
raise ValueError(
f"Dataset {dataset_cfg.path} does not have train/test splits"
)
dataset_name = Path(dataset_cfg.path).name.lower()
table = _ensure_table(config, model_name, vector_dim, dataset_cfg.path)
table_name = _get_table_name(config, model_name, dataset_cfg.path)
# Build database (skip if cached)
table_count = table.count_rows()
expected_count = len(train_dataset)
if table_count == expected_count:
console.print(
f"[yellow]Table '{table_name}' already has {table_count} entries, "
f"skipping database build.[/yellow]"
)
else:
if table_count > 0:
console.print(
f"[yellow]Table '{table_name}' has {table_count} entries "
f"(expected {expected_count}), rebuilding.[/yellow]"
)
# Rebuild: drop and recreate
from database import db_manager
db_manager.db.drop_table(table_name)
table = _ensure_table(config, model_name, vector_dim, dataset_cfg.path)
console.print(
f"[cyan]Building database[/cyan] with {expected_count} training samples..."
)
task.build_database(
model, processor, train_dataset, table, config.batch_size,
label_column=dataset_cfg.label_column,
)
table_count = table.count_rows()
_print_benchmark_info(
config, vector_dim, table_name, table_count,
dataset_path=dataset_cfg.path,
)
# Evaluate
console.print(
f"[cyan]Building database[/cyan] with {len(train_dataset)} training samples..."
f"[cyan]Evaluating[/cyan] on {len(test_dataset)} test samples..."
)
task.build_database(model, processor, train_dataset, table, config.batch_size)
results = task.evaluate(
model, processor, test_dataset, table, config.batch_size,
label_column=dataset_cfg.label_column,
)
all_results[dataset_name] = results
# Run evaluation (results with Rich table will be printed by the task)
task = get_task(config.task.type, top_k=config.task.top_k)
console.print(f"[cyan]Evaluating[/cyan] on {len(test_dataset)} test samples...")
results = task.evaluate(model, processor, test_dataset, table, config.batch_size)
# Print per-dataset results
_print_benchmark_results(results)
return results
# ── Save outputs ─────────────────────────────────────────────
if output_root is not None:
_save_benchmark_outputs(all_results, output_root, model_name)
return all_results

View File

@@ -17,7 +17,7 @@ from configs.models import BenchmarkTaskConfig
from rich.progress import track
from torch import nn
from torch.utils.data import DataLoader
from transformers import BitImageProcessorFast
from transformers import BitImageProcessor
from utils.feature_extractor import extract_single_image_feature
from utils.sam import load_sam_model, segment_image
from utils.common import get_device
@@ -85,7 +85,7 @@ def _compute_scene_score(
hit_rate = matched_count / len(query_object_ids)
# Final score: sum_similarity * (hit_rate)^gamma
score = sum_similarity * (hit_rate ** gamma)
score = sum_similarity * (hit_rate**gamma)
scene_scores[image_id] = score
return scene_scores
@@ -143,7 +143,7 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
def build_database(
self,
model: nn.Module,
processor: BitImageProcessorFast,
processor: BitImageProcessor,
train_dataset: Any,
table: lancedb.table.Table,
batch_size: int,
@@ -176,7 +176,9 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
record_id = 0
records = []
for idx in track(range(len(train_dataset)), description="Building object database"):
for idx in track(
range(len(train_dataset)), description="Building object database"
):
item = train_dataset[idx]
image = item["image"]
image_id = item.get("image_id", f"image_{idx}")
@@ -204,13 +206,15 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
object_id = f"{image_id}_obj_{mask_idx}"
category = mask_info.get("category", "unknown")
records.append({
"id": record_id,
"image_id": image_id,
"object_id": object_id,
"category": category,
"vector": vector,
})
records.append(
{
"id": record_id,
"image_id": image_id,
"object_id": object_id,
"category": category,
"vector": vector,
}
)
record_id += 1
# Add all records to table
@@ -220,7 +224,7 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
def evaluate(
self,
model: nn.Module,
processor: BitImageProcessorFast,
processor: BitImageProcessor,
test_dataset: Any,
table: lancedb.table.Table,
batch_size: int,
@@ -246,7 +250,9 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
correct = 0
total = 0
for idx in track(range(len(test_dataset)), description=f"Evaluating Recall@{top_k}"):
for idx in track(
range(len(test_dataset)), description=f"Evaluating Recall@{top_k}"
):
item = test_dataset[idx]
image = item["image"]
target_image_id = item.get("image_id", f"image_{idx}")
@@ -295,7 +301,9 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
retrieved_results[image_id].append((distance, object_id))
# Compute scene scores
query_object_ids = [m.get("object_id", f"query_obj_{i}") for i, m in enumerate(query_masks)]
query_object_ids = [
m.get("object_id", f"query_obj_{i}") for i, m in enumerate(query_masks)
]
scene_scores = _compute_scene_score(
query_object_ids,
retrieved_results,
@@ -303,7 +311,9 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
)
# Rank scenes by score
ranked_scenes = sorted(scene_scores.items(), key=lambda x: x[1], reverse=True)
ranked_scenes = sorted(
scene_scores.items(), key=lambda x: x[1], reverse=True
)
# Check if target is in top-K
top_k_scenes = [scene_id for scene_id, _ in ranked_scenes[:top_k]]
@@ -322,7 +332,7 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
def _infer_vector_dim(
self,
processor: BitImageProcessorFast,
processor: BitImageProcessor,
model: nn.Module,
sample_image: Any,
) -> int:
@@ -347,7 +357,10 @@ class MultiObjectRetrievalTask(BaseBenchmarkTask):
# Ensure mask is the right shape
if mask.shape != image_np.shape[:2]:
from skimage.transform import resize
mask_resized = resize(mask, image_np.shape[:2], order=0, anti_aliasing=False)
mask_resized = resize(
mask, image_np.shape[:2], order=0, anti_aliasing=False
)
else:
mask_resized = mask

View File

@@ -1,17 +1,63 @@
"""Retrieval task for benchmark evaluation (Recall@K)."""
from typing import Any
from typing import TYPE_CHECKING, Any, cast
import lancedb
import numpy as np
import pyarrow as pa
import torch
import torch.nn.functional as F
from benchmarks.base import BaseBenchmarkTask
from benchmarks.tasks.registry import RegisterTask
from compressors.model_loader import get_dino_dim, load_dino_model, load_hash_compressor
from configs import cfg_manager
from rich.progress import track
from torch import nn
from torch.utils.data import DataLoader
from transformers import BitImageProcessorFast
from transformers import BitImageProcessor
from utils.feature_extractor import extract_batch_features, infer_vector_dim
if TYPE_CHECKING:
from compressors.hash_compressor import HashCompressor
class RetrievalEncoder(nn.Module):
"""Benchmark encoder for DINO and optional hash compression."""
def __init__(
self,
dino: nn.Module,
compressor: "HashCompressor | None" = None,
) -> None:
"""Initialize retrieval encoder.
Args:
dino: DINO backbone used for feature extraction.
compressor: Optional hash compressor for recall evaluation.
"""
super().__init__()
self.dino: nn.Module = dino
self.compressor: HashCompressor | None = compressor
def forward(self, inputs: Any) -> torch.Tensor:
"""Encode processor inputs into benchmark vectors.
Args:
inputs: Batched processor outputs.
Returns:
Float tensor used for LanceDB insertion and retrieval.
"""
outputs = self.dino(**inputs)
tokens = outputs.last_hidden_state
if self.compressor is None:
features = tokens.mean(dim=1)
return F.normalize(features, dim=-1)
bits = self.compressor.encode(tokens)
return bits.to(dtype=torch.float32)
def _build_eval_schema(vector_dim: int) -> pa.Schema:
"""Build PyArrow schema for evaluation database table.
@@ -32,10 +78,11 @@ def _build_eval_schema(vector_dim: int) -> pa.Schema:
def _establish_eval_database(
processor: BitImageProcessorFast,
processor: BitImageProcessor,
model: nn.Module,
table: lancedb.table.Table,
dataloader: DataLoader,
dataloader: DataLoader[Any],
label_column: str = "label",
) -> None:
"""Extract features from training images and store them in a database table.
@@ -44,14 +91,16 @@ def _establish_eval_database(
model: Feature extraction model.
table: LanceDB table to store features.
dataloader: DataLoader for the training dataset.
label_column: Column name for labels in the batch dict.
"""
# Extract all features using the utility function
all_features = extract_batch_features(processor, model, dataloader)
config = cfg_manager.get()
# Store features to database
global_idx = 0
for batch in track(dataloader, description="Storing eval database"):
labels = batch["label"]
labels = batch[label_column]
labels_list = labels.tolist()
batch_size = len(labels_list)
@@ -60,7 +109,7 @@ def _establish_eval_database(
{
"id": global_idx + j,
"label": labels_list[j],
"vector": all_features[global_idx + j].numpy(),
"vector": all_features[global_idx + j].detach().cpu().numpy(),
}
for j in range(batch_size)
]
@@ -69,12 +118,13 @@ def _establish_eval_database(
def _evaluate_recall(
processor: BitImageProcessorFast,
processor: BitImageProcessor,
model: nn.Module,
table: lancedb.table.Table,
dataloader: DataLoader,
dataloader: DataLoader[Any],
top_k: int,
) -> tuple[int, int]:
label_column: str = "label",
) -> dict[str, Any]:
"""Evaluate Recall@K by searching the database for each test image.
Args:
@@ -83,19 +133,29 @@ def _evaluate_recall(
table: LanceDB table to search against.
dataloader: DataLoader for the test dataset.
top_k: Number of top results to retrieve.
label_column: Column name for labels in the batch dict.
Returns:
A tuple of (correct_count, total_count).
A dict with keys:
- correct: Number of correct predictions
- total: Total number of test samples
- query_labels: True labels for each query (length N)
- topk_ids: IDs of top-K retrieved items (N x K)
- topk_labels: Labels of top-K retrieved items (N x K)
"""
# Extract all features using the utility function
all_features = extract_batch_features(processor, model, dataloader)
config = cfg_manager.get()
correct = 0
total = 0
feature_idx = 0
query_labels: list[int] = []
topk_ids: list[list[int]] = []
topk_labels: list[list[int]] = []
for batch in track(dataloader, description=f"Evaluating Recall@{top_k}"):
labels = batch["label"]
labels = batch[label_column]
labels_list = labels.tolist()
for j in range(len(labels_list)):
@@ -104,33 +164,156 @@ def _evaluate_recall(
results = (
table.search(feature)
.select(["label", "_distance"])
.select(["id", "label", "_distance"])
.limit(top_k)
.to_polars()
)
retrieved_ids = results["id"].to_list()
retrieved_labels = results["label"].to_list()
if true_label in retrieved_labels:
correct += 1
total += 1
query_labels.append(true_label)
topk_ids.append(retrieved_ids)
topk_labels.append(retrieved_labels)
feature_idx += len(labels_list)
return correct, total
return {
"correct": correct,
"total": total,
"query_labels": query_labels,
"topk_ids": topk_ids,
"topk_labels": topk_labels,
}
def _build_confusion_matrix(
query_labels: list[int],
topk_labels: list[list[int]],
num_classes: int,
) -> np.ndarray:
"""Build row-normalized confusion matrix from Top-1 predictions.
For each query, the true label is compared against the first
retrieved label (Top-1). The resulting matrix is row-normalized
so that each row sums to 1.0, representing "given true class X,
what fraction of queries were predicted as each class Y".
Args:
query_labels: True labels for each query (length N).
topk_labels: Top-K retrieved labels for each query (N x K).
num_classes: Total number of label classes.
Returns:
Row-normalized confusion matrix as (num_classes, num_classes)
float64 array. Rows with zero queries remain all-zero.
"""
cm = np.zeros((num_classes, num_classes), dtype=np.int64)
for y_true, top_labels in zip(query_labels, topk_labels):
y_pred = top_labels[0] # Top-1
cm[int(y_true), int(y_pred)] += 1
# Row normalize: each row sums to 1.0
row_sums = cm.sum(axis=1, keepdims=True)
cm_norm = np.divide(
cm.astype(np.float64),
row_sums,
out=np.zeros_like(cm, dtype=np.float64),
where=row_sums > 0,
)
return cm_norm
@RegisterTask("retrieval")
class RetrievalTask(BaseBenchmarkTask):
"""Retrieval evaluation task (Recall@K)."""
def __init__(self, top_k: int = 10):
def __init__(
self,
top_k: int = 10,
top_k_list: list[int] | None = None,
dino_model: str = "facebook/dinov2-large",
compression_dim: int = 512,
compressor_path: str | None = None,
):
"""Initialize retrieval task.
Args:
top_k: Number of top results to retrieve for recall calculation.
top_k: Maximum K for retrieval search.
Deprecated — prefer ``top_k_list``.
top_k_list: List of K values to evaluate (all derived from
a single max-K search). When not provided, ``[top_k]``
is used as fallback.
dino_model: DINO model name used for feature extraction.
compression_dim: Output dimension of the hash compressor.
compressor_path: Optional path to trained hash compressor weights.
"""
super().__init__(top_k=top_k)
self.top_k = top_k
if top_k_list is not None:
top_k_list = sorted(set(top_k_list))
else:
top_k_list = [top_k]
super().__init__(top_k_list=top_k_list)
self.top_k_list: list[int] = top_k_list
self.max_k: int = max(self.top_k_list)
self.dino_model = dino_model
self.compression_dim = compression_dim
self.compressor_path = compressor_path
self._processor: BitImageProcessor | None = None
self._model: nn.Module | None = None
self._model_name = "hash_compressor" if compressor_path else "dinov2"
def prepare_benchmark(
self,
model: Any,
processor: Any,
model_name: str = "model",
) -> tuple[nn.Module, BitImageProcessor, str]:
"""Resolve benchmark resources for this task.
Args:
model: Optional pre-built model from the caller.
processor: Optional pre-built processor from the caller.
model_name: Fallback table model name.
Returns:
Tuple of benchmark model, processor, and resolved model name.
"""
if model is not None and processor is not None:
return (
cast(nn.Module, model),
cast(BitImageProcessor, processor),
model_name,
)
self._ensure_resources_loaded()
return (
cast(nn.Module, self._model),
cast(BitImageProcessor, self._processor),
self._model_name,
)
def _ensure_resources_loaded(self) -> None:
"""Lazy-load retrieval benchmark resources."""
if self._processor is not None and self._model is not None:
return
processor, dino = load_dino_model(self.dino_model)
compressor = None
if self.compressor_path is not None:
compressor = load_hash_compressor(
input_dim=get_dino_dim(self.dino_model),
hash_bits=self.compression_dim,
compressor_path=self.compressor_path,
)
compressor.eval()
self._processor = processor
self._model = RetrievalEncoder(dino=dino, compressor=compressor)
self._model.eval()
def build_database(
self,
@@ -139,6 +322,7 @@ class RetrievalTask(BaseBenchmarkTask):
train_dataset: Any,
table: lancedb.table.Table,
batch_size: int,
label_column: str = "label",
) -> None:
"""Build the evaluation database from training data.
@@ -148,6 +332,7 @@ class RetrievalTask(BaseBenchmarkTask):
train_dataset: Training dataset.
table: LanceDB table to store features.
batch_size: Batch size for DataLoader.
label_column: Column name for labels in the batch dict.
"""
# Get a sample image to infer vector dimension
sample = train_dataset[0]
@@ -170,7 +355,7 @@ class RetrievalTask(BaseBenchmarkTask):
shuffle=False,
num_workers=4,
)
_establish_eval_database(processor, model, table, train_loader)
_establish_eval_database(processor, model, table, train_loader, label_column)
def evaluate(
self,
@@ -179,6 +364,7 @@ class RetrievalTask(BaseBenchmarkTask):
test_dataset: Any,
table: lancedb.table.Table,
batch_size: int,
label_column: str = "label",
) -> dict[str, Any]:
"""Evaluate the model on the test dataset.
@@ -188,13 +374,17 @@ class RetrievalTask(BaseBenchmarkTask):
test_dataset: Test dataset.
table: LanceDB table to search against.
batch_size: Batch size for DataLoader.
label_column: Column name for labels in the batch dict.
Returns:
Dictionary containing evaluation results with keys:
- accuracy: Recall@K accuracy (0.0 ~ 1.0)
- correct: Number of correct predictions
- recalls: Dict of ``{"recall@K": value}`` for each K
- total: Total number of test samples
- top_k: The K value used
- top_k_list: List of K values evaluated
- num_classes: Number of label classes
- confusion_matrix: Row-normalized confusion matrix (Top-1)
- query_labels: True labels for each query sample
- topk_labels: Top-K retrieved labels per query sample
"""
test_loader = DataLoader(
test_dataset.with_format("torch"),
@@ -202,15 +392,39 @@ class RetrievalTask(BaseBenchmarkTask):
shuffle=False,
num_workers=4,
)
correct, total = _evaluate_recall(
processor, model, table, test_loader, self.top_k
eval_data = _evaluate_recall(
processor, model, table, test_loader, self.max_k, label_column
)
accuracy = correct / total if total > 0 else 0.0
total = eval_data["total"]
query_labels = eval_data["query_labels"]
topk_labels = eval_data["topk_labels"]
# Compute Recall@k for each k in top_k_list
recalls: dict[str, float] = {}
for k in self.top_k_list:
hits = sum(
1
for true, preds in zip(query_labels, topk_labels)
if true in preds[:k]
)
recalls[f"recall@{k}"] = hits / total if total > 0 else 0.0
# Infer number of classes from the data
all_labels = query_labels + [
label for labels in topk_labels for label in labels
]
num_classes = max(all_labels) + 1 if all_labels else 1
# Confusion matrix from Top-1 only
cm = _build_confusion_matrix(query_labels, topk_labels, num_classes)
return {
"accuracy": accuracy,
"correct": correct,
"recalls": recalls,
"total": total,
"top_k": self.top_k,
"top_k_list": self.top_k_list,
"num_classes": num_classes,
"confusion_matrix": cm.tolist(),
"query_labels": query_labels,
"topk_labels": topk_labels,
}

View File

@@ -1,4 +1,4 @@
from typing import Any, Optional, cast
"""Benchmark CLI command."""
import typer
from commands import app
@@ -6,72 +6,63 @@ from commands import app
@app.command()
def benchmark(
ctx: typer.Context,
model_path: Optional[str] = typer.Option(
_ctx: typer.Context,
model_path: str | None = typer.Option(
None, "--model", "-m", help="Path to compressor model weights"
),
dataset: list[str] | None = typer.Option(
None,
"--dataset",
"-d",
help=(
"Override datasets (HuggingFace path, repeatable). "
"Each value creates a single-entry dataset list."
),
),
top_k: list[int] | None = typer.Option(
None,
"--top-k",
"-k",
help="Override top-K values (repeatable). E.g. --top-k 1 --top-k 5 --top-k 10",
),
):
import torch
import torch.nn.functional as F
"""Run benchmark evaluation across configured datasets.
Without flags, runs all datasets and K values from config.yaml.
Use --dataset to override datasets, --top-k to override K values.
"""
from benchmarks import run_benchmark
from configs import cfg_manager
from transformers import AutoImageProcessor, AutoModel, BitImageProcessorFast
from utils import get_device
from configs.models import DatasetSourceConfig
config = cfg_manager.get()
benchmark_cfg = config.benchmark
device = get_device()
model_cfg = config.model
processor = cast(
BitImageProcessorFast,
AutoImageProcessor.from_pretrained(model_cfg.dino_model, device_map=device),
)
# Load DINO model for feature extraction
dino = AutoModel.from_pretrained(model_cfg.dino_model, device_map=device)
dino.eval()
# Optional hash compressor
compressor = None
if model_path:
from compressors import HashCompressor
config.model.compressor_path = model_path
compressor = HashCompressor(
input_dim=model_cfg.compression_dim,
hash_bits=model_cfg.compression_dim,
)
compressor.load_state_dict(torch.load(model_path))
compressor.to(device)
compressor.eval()
# CLI overrides
if dataset:
benchmark_cfg.datasets = [
DatasetSourceConfig(
source_type="huggingface",
path=d,
)
for d in dataset
]
# Create wrapper with extract_features method
class DinoFeatureExtractor:
def __init__(self, dino, compressor=None):
self.dino = dino
self.compressor = compressor
if top_k:
benchmark_cfg.task.top_k_list = sorted(set(top_k))
def extract_features(self, images: list) -> torch.Tensor:
inputs = processor(images, return_tensors="pt").to(device)
with torch.no_grad():
outputs = self.dino(**inputs)
features = outputs.last_hidden_state.mean(dim=1)
features = F.normalize(features, dim=-1)
return features
model_name = "hash_compressor" if config.model.compressor_path else "dinov2"
output_root = config.output.directory / "benchmark"
def encode(self, images: list) -> torch.Tensor:
if self.compressor is None:
return self.extract_features(images)
tokens = self.dino(**processor(images, return_tensors="pt").to(device)).last_hidden_state
_, _, bits = self.compressor(tokens)
return bits
model = DinoFeatureExtractor(dino, compressor)
run_benchmark(
model=model,
processor=processor,
results = run_benchmark(
model=None,
processor=None,
config=benchmark_cfg,
model_name="dinov2",
model_config=config.model,
model_name=model_name,
output_root=output_root,
)
return results

View File

@@ -11,6 +11,17 @@ def train(
checkpoint_path: str = typer.Option(
"hash_checkpoint.pt", "--checkpoint", "-c", help="Checkpoint path"
),
metrics_path: str = typer.Option(
"hash_training_metrics.jsonl",
"--metrics-path",
"-m",
help="JSONL metrics path for loss curves; relative to output directory unless absolute",
),
log_every: int = typer.Option(
1,
"--log-every",
help="Write one metrics row every N global steps; <=0 disables metrics logging",
),
):
from compressors import train as train_module
@@ -19,4 +30,6 @@ def train(
batch_size=batch_size,
lr=lr,
checkpoint_path=checkpoint_path,
metrics_path=metrics_path,
log_every=log_every,
)

View File

@@ -1,6 +1,20 @@
from .common import BinarySign, bits_to_hash, hamming_distance, hamming_similarity, hash_to_bits
from .common import (
BinarySign,
bits_to_hash,
hamming_distance,
hamming_similarity,
hash_to_bits,
)
from .hash_compressor import HashCompressor, HashLoss, VideoPositiveMask
from .pipeline import HashPipeline, SAMHashPipeline, create_pipeline_from_config
from .filter import (
MaskFeatures,
MaskScoringConfig,
compute_mask_features,
rank_masks,
score_mask,
select_best_mask,
)
from .pipeline import HashPipeline, create_pipeline_from_config
from .train import train
__all__ = [
@@ -9,8 +23,13 @@ __all__ = [
"HashLoss",
"VideoPositiveMask",
"HashPipeline",
"SAMHashPipeline", # Backward compatibility alias
"create_pipeline_from_config",
"MaskFeatures",
"MaskScoringConfig",
"compute_mask_features",
"score_mask",
"rank_masks",
"select_best_mask",
"BinarySign",
"hamming_distance",
"hamming_similarity",

View File

@@ -0,0 +1,14 @@
from .config import MaskScoringConfig
from .features import MaskFeatures, compute_mask_features
from .scorer import score_mask, should_reject_mask
from .selector import rank_masks, select_best_mask
__all__ = [
"MaskFeatures",
"MaskScoringConfig",
"compute_mask_features",
"should_reject_mask",
"score_mask",
"rank_masks",
"select_best_mask",
]

View File

@@ -0,0 +1,27 @@
from dataclasses import dataclass
@dataclass(frozen=True)
class MaskScoringConfig:
min_area_ratio: float = 0.003
max_area_ratio: float = 0.70
max_aspect_ratio: float = 6.0
min_fill_ratio_hard: float = 0.08
max_components: int = 6
min_largest_component_ratio: float = 0.60
reject_edge_touch_count: int = 4
reject_large_edge_touch_count: int = 3
reject_large_edge_area_ratio: float = 0.25
area_score_low: float = 0.02
area_score_high: float = 0.35
fill_score_low: float = 0.25
fill_score_high: float = 0.90
soft_aspect_ratio: float = 2.5
hole_penalty_step: float = 0.05
max_hole_penalty_count: int = 5
weight_area: float = 0.35
weight_shape: float = 0.25
weight_fragment: float = 0.25
weight_boundary: float = 0.15

Some files were not shown because too many files have changed in this diff Show More