diff --git a/AGENTS.md b/AGENTS.md index 1e11b77..144596b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,7 @@ The root `Makefile` is a lean delegator: base commands (`run`, `build`, `test`, - Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path). - Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly. - Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together. +- **Diagrams in this repo's documentation are Mermaid flowcharts.** Gitea renders ` ```mermaid ` fenced blocks natively. Prefer a Mermaid flowchart over ASCII art / box-drawing diagrams; if a diagram genuinely can't be expressed as a flowchart, fall back to a plain text-based markdown diagram (e.g. a code block or table) rather than hand-rawn ASCII boxes. ## Gotchas - Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release. diff --git a/resources/ai/research/data-streaming/01-architecture.md b/resources/ai/research/data-streaming/01-architecture.md index b649458..3504594 100644 --- a/resources/ai/research/data-streaming/01-architecture.md +++ b/resources/ai/research/data-streaming/01-architecture.md @@ -59,11 +59,13 @@ get_catalog_objects :: proc(url: string, catalog_name: string) // nil ## Data flow gap -``` -[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI] - ^ ^ - | serde JSON models | hand-mirrored structs - | | (stubs, never used) +```mermaid +flowchart LR + A["DESI catalog store"] + B["Rust / axum API
serde JSON models"] + C["Odin + raylib GUI
hand-mirrored structs
stubs, never used
"] + A -. "future ingestion" .-> B + B --x|"nothing today"| C ``` There is **no live data flow**. The API currently returns JSON placeholders; the @@ -116,17 +118,16 @@ response protocol with a cheap binary payload would fit this well. ## High-level target architecture -``` - catalog.fbs (single source of truth, checked into repo) - | - +--------+---------+ - | | -flatc --rust flatcc --c (or hand-rolled Odin reader) - | | -api/ (Rust) gui/ (Odin + raylib) - | ^ - | HTTP / WebSocket (framed FlatBuffer binary stream) - +------------------+ +```mermaid +flowchart TB + S["catalog.fbs
single source of truth
checked into repo
"] + R["flatc --rust"] + C["flatcc --c
or hand-rolled Odin reader"] + API["api/ · Rust"] + GUI["gui/ · Odin + raylib"] + S --> R --> API + S --> C --> GUI + API <-->|"HTTP / WebSocket
framed FlatBuffer binary stream"| GUI ``` - One schema file. Two generators. Byte-for-byte identical wire format. diff --git a/resources/ai/research/data-streaming/02-flatbuffers-overview.md b/resources/ai/research/data-streaming/02-flatbuffers-overview.md index e3549c3..916b423 100644 --- a/resources/ai/research/data-streaming/02-flatbuffers-overview.md +++ b/resources/ai/research/data-streaming/02-flatbuffers-overview.md @@ -105,19 +105,28 @@ These are the "thou shalt" rules for keeping buffers compatible: ## Reading a buffer (conceptual) +Buffer layout: + +```mermaid +flowchart LR + subgraph BUF["bytes: &[u8]"] + O["uoffset
root table offset"] + FI["file_identifier
(optional)"] + D["tables · vtables · data"] + end + O --> FI --> D ``` -bytes: &[u8] - ┌─────────────────────────────┐ - │ uoffset (root table offset) │ - │ file_identifier (optional) │ - │ ... tables, vtables, data ...│ - └─────────────────────────────┘ +Access sequence — each field is a few offset dereferences and a read: -root = follow(bytes) // jump to root table via uoffset -vtable = root - root.vtable_off // locate vtable for this table -field_ra = vtable.slot_ra != 0 // present? -if present: ra = read_f64(bytes, root + slot_ra) +```mermaid +flowchart TD + R["root = follow(bytes)
jump to root table via uoffset"] + V["vtable = root − root.vtable_off
locate vtable for this table"] + Q{"field slot present?"} + R --> V --> Q + Q -- "no" --> DEF["use schema default"] + Q -- "yes" --> RD["ra = read_f64(bytes, root + slot_ra)"] ``` There is **no parsing loop**. Each accessor is a few offset dereferences and a diff --git a/resources/ai/research/data-streaming/03-rust-integration.md b/resources/ai/research/data-streaming/03-rust-integration.md index 4629d58..acc0b9d 100644 --- a/resources/ai/research/data-streaming/03-rust-integration.md +++ b/resources/ai/research/data-streaming/03-rust-integration.md @@ -196,6 +196,18 @@ Notes: ## Building buffers efficiently (Rust specifics) +Build order is **back-to-front** (children before parents): + +```mermaid +flowchart TB + A["create_string / create_vector
children first"] + B["create nested child tables"] + C["create parent table
ObjectBatch::create(&args)"] + D["builder.finish(root, Some("DESI"))"] + E["Bytes::copy_from_slice(fbb.finished_data())
→ HTTP / WebSocket response"] + A --> B --> C --> D --> E +``` + - `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the buffer across messages. In a loop streaming batches, create one builder, reuse it — avoid repeated reallocation. diff --git a/resources/ai/research/data-streaming/04-odin-client-integration.md b/resources/ai/research/data-streaming/04-odin-client-integration.md index 52f9c98..c406bd6 100644 --- a/resources/ai/research/data-streaming/04-odin-client-integration.md +++ b/resources/ai/research/data-streaming/04-odin-client-integration.md @@ -17,6 +17,21 @@ has two C-related access points: schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's `foreign import` can consume it. +Choosing a path: + +```mermaid +flowchart TD + NAT{"primary target is native desktop?"} + NAT -- "yes" --> CGO{"want to avoid C in the build?"} + CGO -- "yes" --> PATHA["Path B · pure-Odin reader
hand-rolled, no C dependency"] + CGO -- "no" --> PATHA + CGO -- "prefer proven lib / less maintenance" --> PATHC["Path A · FFI to FlatCC
bind generated C headers"] + PATHC --> REUSE["Path C · OdinArrow reuse
or borrow its decode patterns"] + NAT -- "no · browser/WASM" --> PATHD["Path D · TS/JS interop
official JS lib → typed arrays into WASM"] +``` + +Index of paths: + | Path | Effort | Zero-copy on reads | Notes | |---|---|---|---| | A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` | @@ -175,12 +190,14 @@ typed Odin slices. The rendering win only materializes if data stays zero-copy **into the frame loop**: -1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime - of the frame). -2. `verify` the buffer once. -3. Get `ra_slice := ObjectBatch.ra(&buf)` → `[]f64` view. -4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight - from that slice; build `rl.Vector3`; `DrawPoint3D`. +```mermaid +flowchart LR + A["fetch frame bytes → [dynamic]u8
or a slice pinned for the frame"] + B["verify the buffer once"] + C["ObjectBatch.ra(&buf) → []f64 view"] + D["per object in update()/draw()
ra[i] · dec[i] · z[i] → rl.Vector3 → DrawPoint3D"] + A --> B --> C --> D +``` No per-object allocation. The current `Galaxy { position, color }` dynamic array in `main.odin` is the data structure you'd replace with *slices into the diff --git a/resources/ai/research/data-streaming/05-streaming-protocols.md b/resources/ai/research/data-streaming/05-streaming-protocols.md index 89a5798..76b8f48 100644 --- a/resources/ai/research/data-streaming/05-streaming-protocols.md +++ b/resources/ai/research/data-streaming/05-streaming-protocols.md @@ -24,14 +24,17 @@ offers two built-in mechanisms plus the community pattern: ### Option 1: Size-prefixed FlatBuffers (built-in) +```mermaid +flowchart LR + A["u32 LE
total buffer len
size prefix"] + B["u32 LE
root table offset"] + C["file identifier
(4 bytes)"] + D["tables · vtables · data"] + A --> B --> C --> D +``` + ```rust builder.finish_size_prefixed(root, Some("DESI")); -// +---------------------------+ -// | u32 LE: total buffer len | <-- size prefix -// | u32 LE: root table offset | -// | file identifier (4 bytes) | -// | ... data ... | -// +---------------------------+ ``` Reader side: @@ -53,8 +56,12 @@ message kinds). ### Option 2: Custom length-prefix framing (like `flatstream`) -``` -[ u32 LE: message_len ] [ optional checksum (e.g. u32 crc/xxhash) ] [ flatbuffer payload ] +```mermaid +flowchart LR + A["u32 LE
message_len"] + B["optional checksum
(u32 crc / xxhash)"] + C["FlatBuffer payload"] + A --> B --> C ``` - `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation @@ -260,6 +267,16 @@ FlatBuffers long-term (mmap-friendly, page-in-what-you-touch). ## Decision summary for this repo +```mermaid +flowchart TD + A["HTTP GET → one FlatBuffer body per batch
validate Rust builder + Odin reader"] + B["WebSocket → one Binary message per batch
interactive path · no custom framing"] + C["Self-identifying messages
file_identifier "DESI""] + D["size-prefixed / flatstream-style framing
or HTTP-range + mmap for static catalogs"] + A --> B --> C + C -. "later, if needed" .-> D +``` + 1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust builder + Odin reader (no protocol work at all). 2. Then add **WebSocket** with one `Binary` message per batch (no custom framing) diff --git a/resources/ai/research/data-streaming/07-alternatives.md b/resources/ai/research/data-streaming/07-alternatives.md index cf16a6e..d6761ff 100644 --- a/resources/ai/research/data-streaming/07-alternatives.md +++ b/resources/ai/research/data-streaming/07-alternatives.md @@ -108,6 +108,15 @@ message shape — see `10-performance-benchmarks.md`.) ## Bottom line +```mermaid +flowchart TD + Q1{"zero-copy reads
in the per-frame render loop?"} + Q1 -- "no" --> PB["Protobuf / gRPC
decode once into draw buffers"] + Q1 -- "yes" --> Q2{"truly columnar?
millions of rows"} + Q2 -- "yes" --> ARR["Apache Arrow IPC
via OdinArrow"] + Q2 -- "no · batched vectors" --> FB["FlatBuffers · this proposal"] +``` + - **FlatBuffers is the best default** for this project: the zero-copy read model matches the render loop, the wire format is compact for numeric vectors, schema evolution fits DESI's release cadence, and the Rust + WASM/JS official story diff --git a/resources/ai/research/data-streaming/08-testing-strategies.md b/resources/ai/research/data-streaming/08-testing-strategies.md index 2aaf26b..67bb3c0 100644 --- a/resources/ai/research/data-streaming/08-testing-strategies.md +++ b/resources/ai/research/data-streaming/08-testing-strategies.md @@ -139,7 +139,23 @@ side / flatc* — see the "cross-language fixture" section below. ### 4. Cross-language conformance suite (THE key integration test) -This is the test that actually catches incompatibility. Design: +This is the test that actually catches incompatibility. Pipeline: + +```mermaid +flowchart LR + S["schema/catalog.fbs"] + J["testdata/catalog_sample.json"] + S --> F["flatc --binary"] + J --> F + F --> BIN["committed .bin fixtures
repo-checked-in"] + BIN --> OT["Odin tests
assert identical values"] + BIN --> RT["Rust tests
assert expected values"] + RT -. "deterministic builder" .-> PAR["byte-for-byte parity"] + OT -. "reads it" .-> PAR + PAR -. "catch drift" .-> F +``` + +Design: 1. **Static fixtures, committed to the repo** (`testdata/*.bin`): - Built once by `flatc --binary .fbs .json` (deterministic, diff --git a/resources/ai/research/data-streaming/README.md b/resources/ai/research/data-streaming/README.md index 79848d7..f308b34 100644 --- a/resources/ai/research/data-streaming/README.md +++ b/resources/ai/research/data-streaming/README.md @@ -70,6 +70,16 @@ directly from the buffer each frame without allocations. ## Immediate Next Steps (when you're ready to implement) +```mermaid +flowchart TD + A["Prototype schema
catalog.fbs: Catalog · CatalogObject · ServerMessage union"] + B["Generate Rust code
flatc --rust → api/build.rs · serve WS via axum"] + C["Prototype the Odin reader
flatcc FFI · pure-Odin · OdinArrow"] + D["Static fixture files
flatc --binary → committed .bin"] + E["Cross-language tests
Rust + Odin read the same fixtures identically"] + A --> B --> C --> D --> E +``` + 1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`, and a `ServerMessage` union (handshake / catalog list / chunk of objects / end). 2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see