Commit Graph

119 Commits

Author SHA1 Message Date
e5d13917b2 feat(cam): add Bernoulli noise mask module and cocotb tests
- Add noise_mask_bernoulli.sv RTL module for probabilistic masking
- Add cocotb test suite with reset, threshold, determinism, and distribution checks  
- Update rtl-sources.mk to include new module
- Fix PYTHONPATH in devenv.nix shell hook
2026-05-26 19:41:17 +08:00
1ff9a5f18b feat(retrieval-benchmark): add support for external pre-prepared CAM retrieval datasets with recall@k metric
- Add just recipes for preparing CIFAR10/100 hash artifacts and running benchmarks
- Add CAM_RETRIEVAL_DATASET env var support in Makefile
- Add load_retrieval_dataset_npz() to load pre-prepared retrieval datasets
- Add label_hits counter and recall@k metric for retrieval evaluation
- Rename macro_recall to retrieval_recall to clarify semantics
2026-05-22 21:07:10 +08:00
e1bed00cc4 feat(retrieval): add CAM retrieval benchmark with topk scoring and read noise support
- Add cocotb benchmark infrastructure under hw/sim/benchmarks/retrieval/ with Makefile
- Implement test_retrieval_benchmark.py supporting configurable topk-k, read/write noise
- Add cluster-based synthetic dataset generator with configurable bit-flip rates
- Add reference model functions: match_topk, match_topk_from_scores, score_rows_with_read_noise
- Add .justfile shortcuts: cam-test-retrieval-no-noise, cam-test-retrieval-read-noise
- Add TOPK_K to Verilator EXTRA_ARGS via cocotb-common.mk
- Add unit tests for topk sorting logic and stateful read-noise scoring
2026-05-22 19:04:54 +08:00
29f4cc91f6 fix(pipeline): add S_WAIT_READ_RESP state to fix read-noise PRNG timing
- Emit rd_valid_o for exactly one combinational cycle before waiting for
  read response, ensuring the read-noise PRNG advances once per batch issue
- Fix query handshake: wait for query_ready before asserting query_valid
  to avoid valid&&ready handshake drops on clock edges
- Add dynamic timeout estimation in test utilities based on DUT parameters
- Update test-top Makefile to run all noise configurations by default
- Remove uv run prefix from cocotb-config Makefile invocation
2026-05-22 19:04:50 +08:00
424cf6e1d3 refactor(hw/sim): reorganize CAM top-level tests into per-noise-config suites
Split the monolithic test_cam_basic.py into separate test suites
organized by noise configuration (no_noise, write_noise, read_noise),
with shared utilities extracted to tests/top/utils.py.

- Remove test_cam_basic.py; add no_noise/, write_noise/, read_noise/
  test suites with Makefiles that set noise parameters statically
- Extract helpers (reset_dut, write_rows, query_once, collect_topk,
  etc.) into tests/top/utils.py
- Update hw/sim/Makefile with per-config test targets and a
  test-top-all meta-target
- Update scripts/run_cam_correctness.py to build per-directory
  instead of centrally, removing inline parameter overrides
- Add __init__.py for result_serializer and topk_tracker module tests
- Expand docstrings in test_ref_model_noise.py with architectural
  context and test rationale
2026-05-21 21:22:12 +08:00
5a1d3ea977 refactor(verification): extract shared text_labels cell and hash codec
- Move inline _text_labels lists into a shared text_labels cell
- Replace inline np.packbits hash encoding with bits_tensor_to_hash_bytes
- Add static tests to verify notebook API usage patterns
2026-05-21 16:41:39 +08:00
101a675ece feat(scenegraph): add ImageHashPipeline protocol and update query API
- Introduce ImageHashPipeline Protocol for extensible hash computation
- Rename query_crop_index to query_index for clarity
- Make query_crop nullable to handle missing crop edge case
- Add keyword-only arguments for better API clarity
- Handle empty scene graph objects gracefully
- Add comprehensive test coverage for query_image_against_scene_graph
2026-05-21 15:33:10 +08:00
ba96cec406 feat(scenegraph): refactor image scene graph query into reusable function
- Export ImageSceneGraphQueryResult and query_image_against_scene_graph from scenegraph module
- Replace inline hamming-distance-based image matching with dedicated query_image_against_scene_graph function
- Improve top_matches structure by extracting similarity scores and hash_bytes from matches
- Add .codegraph/ to gitignore (machine-local data, should not be committed)
- Add CodeGraph configuration for multi-language indexing
2026-05-21 14:25:50 +08:00
e4cbb5e30d feat(hw/rtl): implement full Top-K CAM search pipeline with serial result output
- add TOPK_K, FIFO_DEPTH, RESULT_SERIAL parameters to cam_params
- add candidate_fifo: synchronous ready/valid FIFO for (row, score) candidates
- add topk_tracker: tracks top-K candidates with clear/ready handshake
- add result_serializer: serializes packed Top-K array into rank-ordered stream
- refactor match_engine_pipeline from 4-state to 8-state Top-K pipeline
- extend cam_top with serial Top-K interface (result_rank/row/score/last)
- add backward-compatible top1_index/top1_score aliases from rank-0 beat
- add comprehensive tests for all new modules
2026-05-19 22:43:21 +08:00
8bcad1f23f refactor(core/cam_core_banked): extract per-bank modules for improved timing isolation
- Extract cam_bank as a parameterized submodule with independent read/write ports
- Replace flat 2D memory array with generate loop of bank instances
- Derive bank selection from address bit slicing instead of modulo arithmetic
- Align rd_base_row_i check with new bank addressing scheme
- Add test verifying bank address isolation across multiple banks
2026-05-19 16:17:08 +08:00
5d09f13a08 refactor(env): remove torch from conda env and add yosys for conda 2026-05-18 15:58:11 +08:00
706d148a0b feat(hw): add CAM top-level synthesis infrastructure and fix RTL synthesis compatibility
- Add hw/syn/Makefile with hier/flat/full synth targets and artifact mirroring
- Add synth_cam_top_hier.ys for hierarchy-preserving resource estimation on Xilinx 7-series
- Add synth_cam_top_flat.ys for flattened Xilinx 7-series synthesis
- Add cam-synth just target for convenient invocation
- Guard runtime assertions (NUM_ROWS/LANES checks, noise seed checks, NOISE_BITS checks)
  behind SYNTHESIS guard in cam_core_banked, cam_read_noise, cam_write_noise, and noise_mask_grouped
- Fix shadowed 'return' variable in random128 xorshift128 function
2026-05-18 15:40:04 +08:00
b9b5684718 test(popcount_pipeline): add cocotb test for popcount pipeline module
- Add Makefile with cocotb configuration for popcount_pipeline RTL simulation
- Add Python test module with reset, drive/expect helpers
- Test row metadata preservation and bit-count correctness across 7 vectors
2026-05-18 15:37:26 +08:00
583b2156ea feat(scenegraph): add SoftwareCamIndex for visual hash similarity queries
- Add SoftwareCamIndex class with xnor_popcount_score for CAM-style matching
- Add CamMatch and SceneGraphMatch dataclasses for query results
- Add query_by_visual_hash method to SimpleSceneGraph
- Add comprehensive tests for SoftwareCamIndex and xnor_popcount_score
2026-05-17 21:57:01 +08:00
ddb8cff6a9 feat(scenegraph): add hash codec for bits/tensor/bytes/cam_row conversion
Introduce hash_codec module providing bidirectional encoding/decoding:
- bits_tensor_to_hash_bytes / hash_bytes_to_bits_array
- bits_tensor_to_cam_row
- hash_bytes_to_cam_row / cam_row_to_hash_bytes
2026-05-17 19:41:44 +08:00
4ea567adba feat(compressors): add JSONL training metrics logging with CLI controls
- Add write_training_metrics() in new compressors/training_metrics.py
  for appending epoch/step/lr/component rows as JSON Lines
- Wire --metrics-path and --log-every CLI options into train.py, passing
  them to the training loop so metrics rows are written every N steps
- Accept absolute metrics paths or paths relative to output directory
- Add quantization component to loss log alongside existing distill/contrastive
- Replace inline torch.device() with get_device() utility
- Add test_hash_training_metrics.py covering multi-row JSONL append

Infrastructure:
- Pin torch 2.7.1 + CUDA 12.8 index for Linux/Windows in pyproject.toml
- Add .justfile rsync upload recipe with .stignore exclusion
- Exclude **/__marimo__ from rsync in .stignore

Dependencies updated: numpy 2.4.5, pandas 3.0.3, black 26.5.0,
click 8.4.0, contourpy, etc.
2026-05-17 14:57:10 +08:00
e8c890a69f feat(sim): enable read noise by default and improve test infrastructure
- Set READ_NOISE_EN=1 as default in cam_top.sv
- Wire READ_NOISE_* parameters to NOISE_* makefile variables in cocotb-common.mk
- Add PYTHON env variable support in Makefile for test invocation
- Update .gitignore with glob patterns (**/sim_build/, **/results.xml, *.fst)
- Add justfile recipe for remote cam-test-top execution
2026-05-17 14:11:59 +08:00
e7765cdb76 refactor(build): consolidate CAM test commands under new make targets
- .envrc: add conda compiler flags (CFLAGS, CXXFLAGS, LDFLAGS) for zlib support
- .justfile: replace inline make commands with reusable targets (test-top, test-all, test-model, test-module)
- environment.yml: reformat YAML, add zlib/gtkwave, rename withbullet → with_bullet
2026-05-16 19:57:34 +08:00
ca167e79c6 refactor(hw/sim): extract common cocotb make infrastructure into shared mk/ directory
- Split monolithic hw/sim/Makefile into modular include files (mk/cocotb-common.mk, mk/rtl-sources.mk)
- Add per-module test Makefiles (cam_core_banked, cam_read_noise, cam_write_noise, match_engine_pipeline, perf, top)
- Add missing simulation tools (yosys, graphviz, xdot) to devenv.nix
- Fix path handling in sweep_noise.py and test modules to be HASH_BITS-aware
2026-05-16 19:24:17 +08:00
2e0e36eea5 feat(sim): add CAM performance sweep test infrastructure
- hw/sim/tests/test_cam_perf.py: new cocotb perf test with bounded wait helpers
- scripts/sweep_cam_perf.py: sweep data model, matrix, and make command builders  
- tests/test_sweep_cam_perf.py: unit tests for sweep helpers
- tests/conftest.py: pytest path configuration for scripts package
- hw/sim/Makefile: deterministic noise params override for perf test compatibility
2026-05-16 15:42:29 +08:00
24b8750f0f feat(sim): add CAM correctness runner with scenario matrix and contract tests
Introduce scripts/run_cam_correctness.py — a data-driven CAM correctness
checker that runs cocotb and pytest scenarios across multiple noise modes
and CAM submodules (cam_top, cam_core_banked, match_engine_pipeline).

Outputs structured CSV results and a paper-ready Chinese summary.
Add tests/test_run_cam_correctness.py with contract tests for scenario
spec validation, command building, subprocess error handling, output file
generation, CLI defaults, and environment preservation.
2026-05-15 14:03:40 +08:00
3203b2d9af chore(project): adjust safety rules 2026-05-15 14:03:40 +08:00
ab616528b4 refactor(benchmark): delegate model loading to tasks and support CIFAR-100
- Extract model loading logic from benchmark CLI into task-owned prepare_benchmark
- Add RetrievalEncoder class wrapping DINO with optional hash compression
- Add accelerate dependency for device management  
- Switch dataset from CIFAR-10 to CIFAR-100 with fine_label column
2026-05-15 09:59:40 +08:00
0fbcd915bd refactor: reorganize RTL files into core/noise subdirectories
- Move CAM core modules (cam_core_banked, match_engine_pipeline, popcount_pipeline) to hw/rtl/core/
- Move noise modules (cam_read_noise, cam_write_noise, noise_mask_grouped) to hw/rtl/noise/
- Update Makefile include paths and VERILOG_SOURCES to reflect new layout
- Update docs/experiments.md file path references
- Add sim/results.xml to .gitignore
- Bump devenv.lock dependencies
2026-05-14 20:59:46 +08:00
443edbfa25 docs: add experiments feasibility survey for paper sections 6.2–6.8 2026-05-13 20:03:58 +08:00
e1a34d6540 refactor!: remove legacy match_engine and related CAM modules
- Delete popcount.sv, argmax_update.sv, cam_core.sv, match_engine.sv
- Remove obsolete VERILOG_SOURCES entries from Makefile
- Update comment in cam_top.sv to reference match_engine_pipeline
- Add verilator to devenv.nix for simulation support
2026-05-13 19:11:33 +08:00
8f59a287c4 feat(hw): add banked CAM pipeline with grouped read/write noise
- Add cam_core_banked.sv with 8-lane banked CAM core
- Add cam_write_noise.sv and cam_read_noise.sv for grouped noise injection
- Add noise_mask_grouped.sv generating grouped flip masks from 128-bit PRNG
- Add match_engine_pipeline.sv with multi-stage pipelined top-1 selection
- Add popcount_pipeline.sv for pipelined popcount operations
- Refactor test_cam_basic.py with parametrized DUT introspection helpers
- Add Python ref_model match_top1_with_read_noise() for read noise verification
- Update Makefile with separate WRITE_NOISE_* and READ_NOISE_* parameter groups
- Add new testbenches: test_cam_core_banked, test_cam_read_noise,
  test_cam_write_noise, test_match_engine_pipeline, test_ref_model_noise
  
breaking change hint: NUM_ROWS default changed from 512→4096, LANES from 16→8
2026-05-13 18:22:28 +08:00
c41e64d1c6 feat(dev): add CAM test recipes and update ignore files
- Add `.opencode/` and `**/sim_build` to ignore lists
- Fix justfile syntax: `set dotenv-required := true` (explicit assignment)
- Update SSH tunnel port 8384 → 42705 for local port forwarding
- Add CAM verification recipes: cam-test-all, cam-test-py, cam-test-module, cam-test
2026-05-13 18:22:14 +08:00
cbafc4524e feat(cam): migrate noise generation from xorshift64 to xorshift128
- Replace NOISE_GEN_BITS/NOISE_SAMPLE_BITS parameters with unified NOISE_BITS
- Use xorshift128 (random128) instead of xorshift64 for PRNG
- Add flip_mask_next combinational helper for single-cycle mask computation
- Add random_enable signal to advance PRNG only on accepted noisy writes
- Simplify FSM by removing mask_group_idx counter
- Update parameter validation: GROUP_BITS (= HASH_BITS/NOISE_BITS) must equal 64
- Update ref_model.py and tests to match new seed convention: {seed, seed}
- Update Makefile and sweep_noise.py with renamed parameters
2026-05-05 20:19:22 +08:00
0dd01fb1b7 feat(hw/rtl): add xorshift PRNG modules and refactor cam_noisy FSM
- Add random32, random64 and random128 xorshift PRNG modules
- Refactor cam_noisy FSM: split state register, next-state logic, and datapath into distinct blocks
- Rename state_q/state_d to curr_state/next_state for clarity
- Add MASK_GROUPS localparam and fix type casting in noise generation
- Update .gitignore to exclude docs/superpowers
2026-05-05 19:30:50 +08:00
2da17e101b feat(rtl): migrate CAM interface to handshake protocol with integrated noise generation
BREAKING CHANGE: CAM write and query interface replaced with standard valid/ready
handshake. wr_en/wr_row/wr_hash → wr_valid/wr_ready/wr_addr/write_hash.
External noise_mask_lanes_flat removed; noise generation now handled internally
by cam_noisy module with configurable rate via parameters.

- cam_top: add parameters (NOISE_EN, NOISE_RATE_NUM/DEN, NOISE_GEN/SAMPLE_BITS, NOISE_SEED)
- cam_top: replace cam_core with cam_noisy (integrated noise generation)
- match_engine: remove external noise_mask_lanes_flat input
- hw/sim: update Makefile with noise parameters and compile args
- hw/sim/model: add generate_write_flip_mask() and xorshift64() matching RTL behavior
- hw/sim/tests: adapt testbench to new handshake protocol
2026-05-04 18:03:06 +08:00
0ae6d757dc refactor(scripts): extract shared utilities into common module
- Move load_env_file, parse_timeout_seconds, build_remote_script, and
  build_ssh_command to scripts/common.py
- Update remote_docker_run.py to import from common module
- Improves code organization and reusability
2026-05-03 19:49:45 +08:00
8b8e4d3118 build(simulation): improve verilator simulation infrastructure
- Add verilator to dependencies
- Add configurable logging via QUIET/VERBOSE/COCOTB_LOG_LEVEL env vars
- Add optional warning suppression (SUPPRESS_VERILATOR_WARNINGS)
- Clean up and restructure Makefile
2026-05-03 15:33:17 +08:00
7450505a86 feat(remote): unify remote execution workflow with env-based config
- Replace hardcoded SSH/docker targets with environment variables
- Add remote-check, remote-dry, remote-test, remote-test-one recipes
- Add remote cmd for arbitrary command execution in docker container
2026-05-03 15:32:32 +08:00
f5daaa2667 refactor(cam): extract match engine into separate module and centralize parameters
- Split cam_core into pure memory (cam_core.sv) and match engine (match_engine.sv)
- Add cam_params.svh with centralized parameter definitions (NUM_ROWS, HASH_BITS, LANES, etc.)
- Update cam_top.sv to use shared parameters and compose match_engine
- Update Makefile to include new match_engine module and correct Verilator define syntax
2026-05-02 23:28:32 +08:00
f71bf06484 feat(hw): add XNOR-popcount CAM design with cocotb verification
Implement a multi-lane Content Addressable Memory (CAM) that scores
rows by XNOR popcount against a query hash and returns the top-1 match.

RTL modules:
- popcount: parallel group-based population count
- argmax_update: iterative best-match tracking with tie-break
- cam_core: parameterized scanning engine (NUM_ROWS/HASH_BITS/LANES)
  with optional SIM_NOISE and SIM_DEBUG ifdef guards
- cam_top: thin wrapper exposing cam_core ports

Verification:
- Python reference model (ref_model.py) for score-level golden comparison
- cocotb testbench (test_cam_basic.py) covering write/query/reset and
  external noise mask scenarios with score debug verification
- Noise sweep script (sweep_noise.py) measuring top-1 stability under
  configurable bit-flip rates
- Verilator-oriented Makefile with parameterizable compile options
2026-05-02 23:26:16 +08:00
ad45123022 chore: migrate habitat dependencies to conda and add cocotb for hardware verification
- Move habitat-baselines and habitat-lab from pip to conda environment
- Add cocotb and cocotb-tools to pyproject.toml dependencies
- Update ty environment roots to include hw/sim directory
- Add sim/sim_build to gitignore
- Create CLAUDE.md with project spec, coding guidelines, and directory structure
- Update uv.lock to reflect dependency changes
2026-05-02 23:24:59 +08:00
d9745f45dc refactor(visualization): enhance proposal filtering visualization 2026-04-18 23:24:30 +08:00
e3cdc1c6e7 chore(project): update .envrc and justfile for server, and update marimo 2026-04-18 21:10:22 +08:00
8a56c9649d refactor(verification): batch pipeline inference with progress tracking
- Increase verification params: image_size 512→768, rooms 4→5, views 6→12
- Refactor single-batch inference to chunked batch processing with mo.progress_bar
- Extract debug_meta and hash_batches from output for clearer variable flow
- Add progress bars to room-view snapshot saving and scene graph building
- Add .ruff_cache/, .pytest_cache/, .sisyphus/ to .justfile upload excludes
2026-04-11 17:09:39 +08:00
79b49f122a feat(test): add collect test images notebook and replace BitImageProcessorFast 2026-04-11 15:22:22 +08:00
01017277c3 feat(project): update marimo and env autoload 2026-04-11 15:21:11 +08:00
eb016385f8 refactor(sync): improve topdown rendering and make justfile adapt to windows 2026-04-06 19:55:52 +08:00
2c78f4b662 refactor(verification): improve object creation / matching and image saving 2026-04-06 13:27:02 +08:00
37b4a08398 refactor(pipeline): support multi-object selection in frame processing 2026-04-06 10:27:23 +08:00
3638ffdb8d feat(compressors): refactor pipeline with FramePacket dataclass and unified process_batch
- Add FramePacket dataclass to encapsulate per-image pipeline state
- Rename internal methods with underscore prefix convention
- Replace separate filter_batch/crop_batch with unified process_batch method
- Update notebook to use new HashPipeline API
2026-04-04 20:09:05 +08:00
94ed05a039 feat(compressors): add OWLv2 bbox crop to HashPipeline and refactor image utilities
- Add Owlv2ForObjectDetection and Owlv2Processor imports to model_loader
- Refactor load_dino_model to return tuple of processor and model
- Rewrite generate_proposals_batch to group images by bbox count for efficient batching
- Add _normalize_single_bbox_list helper for bbox normalization
- Update verification.py to use new pipeline architecture with detect/segment/filter/crop steps
2026-04-04 15:27:47 +08:00
5f41cf5794 feat(compressors): add OWLv2 bbox crop to HashPipeline and refactor image utilities
- Add crop_batch method to HashPipeline for cropping images using OWLv2 detection boxes
- Integrate crop_batch into pipeline forward pass (extract_hash and extract_features)
- Move extract_masked_region from compressors/proposal/utils.py to utils/image.py
- Add crop_image_by_bbox utility function in utils/image.py
- Update type annotations to use dict[str, Any] instead of bare dict
- Update .justfile to add memory server command
- Update marimo dependency to >=0.22.0
- Update nvidia CUDA package markers for platform compatibility
2026-04-03 19:43:43 +08:00
4e16e38f32 refactor(compressors): migrate pipeline to OWLv2-based detection with text labels
- Replace bbox-prompted segmentation with OWLv2 text-guided object detection
- Refactor HashPipeline from nn.Module to plain class with modular stage methods
- Add detect_batch, segment_batch, filter_batch for explicit pipeline stages
- Rename forward to forward_batch with text_labels API instead of bboxes
- Add mask_scoring_config, score_threshold, postprocess_threshold configuration
- Update model_loader to expose Dinov2Model type annotation
2026-04-03 18:07:17 +08:00
4918b654e7 refactor(compressors): migrate to centralized model loaders
- Refactor model_loader.py: improve return type annotations from tuple[Any, Any] to tuple[AutoImageProcessor, AutoModel]
- Refactor proposal/core.py: add input validation for mask array dimensionality, handle 2D masks and batch dimensions gracefully
- Refactor proposal_segament.ipynb: replace inline model loading with centralized load_owlv2_model() and load_sam_model() functions, use batched detect_objects_batch() and generate_proposals_batch() APIs
2026-04-03 15:00:14 +08:00