diff --git a/AGENTS.md b/AGENTS.md
index 1e11b77..144596b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -25,6 +25,7 @@ The root `Makefile` is a lean delegator: base commands (`run`, `build`, `test`,
- Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path).
- Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly.
- Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together.
+- **Diagrams in this repo's documentation are Mermaid flowcharts.** Gitea renders ` ```mermaid ` fenced blocks natively. Prefer a Mermaid flowchart over ASCII art / box-drawing diagrams; if a diagram genuinely can't be expressed as a flowchart, fall back to a plain text-based markdown diagram (e.g. a code block or table) rather than hand-rawn ASCII boxes.
## Gotchas
- Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release.
diff --git a/resources/ai/research/data-streaming/01-architecture.md b/resources/ai/research/data-streaming/01-architecture.md
index b649458..3504594 100644
--- a/resources/ai/research/data-streaming/01-architecture.md
+++ b/resources/ai/research/data-streaming/01-architecture.md
@@ -59,11 +59,13 @@ get_catalog_objects :: proc(url: string, catalog_name: string) // nil
## Data flow gap
-```
-[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI]
- ^ ^
- | serde JSON models | hand-mirrored structs
- | | (stubs, never used)
+```mermaid
+flowchart LR
+ A["DESI catalog store"]
+ B["Rust / axum API
serde JSON models"]
+ C["Odin + raylib GUI
hand-mirrored structs
stubs, never used"]
+ A -. "future ingestion" .-> B
+ B --x|"nothing today"| C
```
There is **no live data flow**. The API currently returns JSON placeholders; the
@@ -116,17 +118,16 @@ response protocol with a cheap binary payload would fit this well.
## High-level target architecture
-```
- catalog.fbs (single source of truth, checked into repo)
- |
- +--------+---------+
- | |
-flatc --rust flatcc --c (or hand-rolled Odin reader)
- | |
-api/ (Rust) gui/ (Odin + raylib)
- | ^
- | HTTP / WebSocket (framed FlatBuffer binary stream)
- +------------------+
+```mermaid
+flowchart TB
+ S["catalog.fbs
single source of truth
checked into repo"]
+ R["flatc --rust"]
+ C["flatcc --c
or hand-rolled Odin reader"]
+ API["api/ · Rust"]
+ GUI["gui/ · Odin + raylib"]
+ S --> R --> API
+ S --> C --> GUI
+ API <-->|"HTTP / WebSocket
framed FlatBuffer binary stream"| GUI
```
- One schema file. Two generators. Byte-for-byte identical wire format.
diff --git a/resources/ai/research/data-streaming/02-flatbuffers-overview.md b/resources/ai/research/data-streaming/02-flatbuffers-overview.md
index e3549c3..916b423 100644
--- a/resources/ai/research/data-streaming/02-flatbuffers-overview.md
+++ b/resources/ai/research/data-streaming/02-flatbuffers-overview.md
@@ -105,19 +105,28 @@ These are the "thou shalt" rules for keeping buffers compatible:
## Reading a buffer (conceptual)
+Buffer layout:
+
+```mermaid
+flowchart LR
+ subgraph BUF["bytes: &[u8]"]
+ O["uoffset
root table offset"]
+ FI["file_identifier
(optional)"]
+ D["tables · vtables · data"]
+ end
+ O --> FI --> D
```
-bytes: &[u8]
- ┌─────────────────────────────┐
- │ uoffset (root table offset) │
- │ file_identifier (optional) │
- │ ... tables, vtables, data ...│
- └─────────────────────────────┘
+Access sequence — each field is a few offset dereferences and a read:
-root = follow(bytes) // jump to root table via uoffset
-vtable = root - root.vtable_off // locate vtable for this table
-field_ra = vtable.slot_ra != 0 // present?
-if present: ra = read_f64(bytes, root + slot_ra)
+```mermaid
+flowchart TD
+ R["root = follow(bytes)
jump to root table via uoffset"]
+ V["vtable = root − root.vtable_off
locate vtable for this table"]
+ Q{"field slot present?"}
+ R --> V --> Q
+ Q -- "no" --> DEF["use schema default"]
+ Q -- "yes" --> RD["ra = read_f64(bytes, root + slot_ra)"]
```
There is **no parsing loop**. Each accessor is a few offset dereferences and a
diff --git a/resources/ai/research/data-streaming/03-rust-integration.md b/resources/ai/research/data-streaming/03-rust-integration.md
index 4629d58..acc0b9d 100644
--- a/resources/ai/research/data-streaming/03-rust-integration.md
+++ b/resources/ai/research/data-streaming/03-rust-integration.md
@@ -196,6 +196,18 @@ Notes:
## Building buffers efficiently (Rust specifics)
+Build order is **back-to-front** (children before parents):
+
+```mermaid
+flowchart TB
+ A["create_string / create_vector
children first"]
+ B["create nested child tables"]
+ C["create parent table
ObjectBatch::create(&args)"]
+ D["builder.finish(root, Some("DESI"))"]
+ E["Bytes::copy_from_slice(fbb.finished_data())
→ HTTP / WebSocket response"]
+ A --> B --> C --> D --> E
+```
+
- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
buffer across messages. In a loop streaming batches, create one builder, reuse
it — avoid repeated reallocation.
diff --git a/resources/ai/research/data-streaming/04-odin-client-integration.md b/resources/ai/research/data-streaming/04-odin-client-integration.md
index 52f9c98..c406bd6 100644
--- a/resources/ai/research/data-streaming/04-odin-client-integration.md
+++ b/resources/ai/research/data-streaming/04-odin-client-integration.md
@@ -17,6 +17,21 @@ has two C-related access points:
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
`foreign import` can consume it.
+Choosing a path:
+
+```mermaid
+flowchart TD
+ NAT{"primary target is native desktop?"}
+ NAT -- "yes" --> CGO{"want to avoid C in the build?"}
+ CGO -- "yes" --> PATHA["Path B · pure-Odin reader
hand-rolled, no C dependency"]
+ CGO -- "no" --> PATHA
+ CGO -- "prefer proven lib / less maintenance" --> PATHC["Path A · FFI to FlatCC
bind generated C headers"]
+ PATHC --> REUSE["Path C · OdinArrow reuse
or borrow its decode patterns"]
+ NAT -- "no · browser/WASM" --> PATHD["Path D · TS/JS interop
official JS lib → typed arrays into WASM"]
+```
+
+Index of paths:
+
| Path | Effort | Zero-copy on reads | Notes |
|---|---|---|---|
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
@@ -175,12 +190,14 @@ typed Odin slices.
The rendering win only materializes if data stays zero-copy **into the frame
loop**:
-1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime
- of the frame).
-2. `verify` the buffer once.
-3. Get `ra_slice := ObjectBatch.ra(&buf)` → `[]f64` view.
-4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight
- from that slice; build `rl.Vector3`; `DrawPoint3D`.
+```mermaid
+flowchart LR
+ A["fetch frame bytes → [dynamic]u8
or a slice pinned for the frame"]
+ B["verify the buffer once"]
+ C["ObjectBatch.ra(&buf) → []f64 view"]
+ D["per object in update()/draw()
ra[i] · dec[i] · z[i] → rl.Vector3 → DrawPoint3D"]
+ A --> B --> C --> D
+```
No per-object allocation. The current `Galaxy { position, color }` dynamic array
in `main.odin` is the data structure you'd replace with *slices into the
diff --git a/resources/ai/research/data-streaming/05-streaming-protocols.md b/resources/ai/research/data-streaming/05-streaming-protocols.md
index 89a5798..76b8f48 100644
--- a/resources/ai/research/data-streaming/05-streaming-protocols.md
+++ b/resources/ai/research/data-streaming/05-streaming-protocols.md
@@ -24,14 +24,17 @@ offers two built-in mechanisms plus the community pattern:
### Option 1: Size-prefixed FlatBuffers (built-in)
+```mermaid
+flowchart LR
+ A["u32 LE
total buffer len
size prefix"]
+ B["u32 LE
root table offset"]
+ C["file identifier
(4 bytes)"]
+ D["tables · vtables · data"]
+ A --> B --> C --> D
+```
+
```rust
builder.finish_size_prefixed(root, Some("DESI"));
-// +---------------------------+
-// | u32 LE: total buffer len | <-- size prefix
-// | u32 LE: root table offset |
-// | file identifier (4 bytes) |
-// | ... data ... |
-// +---------------------------+
```
Reader side:
@@ -53,8 +56,12 @@ message kinds).
### Option 2: Custom length-prefix framing (like `flatstream`)
-```
-[ u32 LE: message_len ] [ optional checksum (e.g. u32 crc/xxhash) ] [ flatbuffer payload ]
+```mermaid
+flowchart LR
+ A["u32 LE
message_len"]
+ B["optional checksum
(u32 crc / xxhash)"]
+ C["FlatBuffer payload"]
+ A --> B --> C
```
- `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation
@@ -260,6 +267,16 @@ FlatBuffers long-term (mmap-friendly, page-in-what-you-touch).
## Decision summary for this repo
+```mermaid
+flowchart TD
+ A["HTTP GET → one FlatBuffer body per batch
validate Rust builder + Odin reader"]
+ B["WebSocket → one Binary message per batch
interactive path · no custom framing"]
+ C["Self-identifying messages
file_identifier "DESI""]
+ D["size-prefixed / flatstream-style framing
or HTTP-range + mmap for static catalogs"]
+ A --> B --> C
+ C -. "later, if needed" .-> D
+```
+
1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust
builder + Odin reader (no protocol work at all).
2. Then add **WebSocket** with one `Binary` message per batch (no custom framing)
diff --git a/resources/ai/research/data-streaming/07-alternatives.md b/resources/ai/research/data-streaming/07-alternatives.md
index cf16a6e..d6761ff 100644
--- a/resources/ai/research/data-streaming/07-alternatives.md
+++ b/resources/ai/research/data-streaming/07-alternatives.md
@@ -108,6 +108,15 @@ message shape — see `10-performance-benchmarks.md`.)
## Bottom line
+```mermaid
+flowchart TD
+ Q1{"zero-copy reads
in the per-frame render loop?"}
+ Q1 -- "no" --> PB["Protobuf / gRPC
decode once into draw buffers"]
+ Q1 -- "yes" --> Q2{"truly columnar?
millions of rows"}
+ Q2 -- "yes" --> ARR["Apache Arrow IPC
via OdinArrow"]
+ Q2 -- "no · batched vectors" --> FB["FlatBuffers · this proposal"]
+```
+
- **FlatBuffers is the best default** for this project: the zero-copy read model
matches the render loop, the wire format is compact for numeric vectors, schema
evolution fits DESI's release cadence, and the Rust + WASM/JS official story
diff --git a/resources/ai/research/data-streaming/08-testing-strategies.md b/resources/ai/research/data-streaming/08-testing-strategies.md
index 2aaf26b..67bb3c0 100644
--- a/resources/ai/research/data-streaming/08-testing-strategies.md
+++ b/resources/ai/research/data-streaming/08-testing-strategies.md
@@ -139,7 +139,23 @@ side / flatc* — see the "cross-language fixture" section below.
### 4. Cross-language conformance suite (THE key integration test)
-This is the test that actually catches incompatibility. Design:
+This is the test that actually catches incompatibility. Pipeline:
+
+```mermaid
+flowchart LR
+ S["schema/catalog.fbs"]
+ J["testdata/catalog_sample.json"]
+ S --> F["flatc --binary"]
+ J --> F
+ F --> BIN["committed .bin fixtures
repo-checked-in"]
+ BIN --> OT["Odin tests
assert identical values"]
+ BIN --> RT["Rust tests
assert expected values"]
+ RT -. "deterministic builder" .-> PAR["byte-for-byte parity"]
+ OT -. "reads it" .-> PAR
+ PAR -. "catch drift" .-> F
+```
+
+Design:
1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
- Built once by `flatc --binary .fbs .json` (deterministic,
diff --git a/resources/ai/research/data-streaming/README.md b/resources/ai/research/data-streaming/README.md
index 79848d7..f308b34 100644
--- a/resources/ai/research/data-streaming/README.md
+++ b/resources/ai/research/data-streaming/README.md
@@ -70,6 +70,16 @@ directly from the buffer each frame without allocations.
## Immediate Next Steps (when you're ready to implement)
+```mermaid
+flowchart TD
+ A["Prototype schema
catalog.fbs: Catalog · CatalogObject · ServerMessage union"]
+ B["Generate Rust code
flatc --rust → api/build.rs · serve WS via axum"]
+ C["Prototype the Odin reader
flatcc FFI · pure-Odin · OdinArrow"]
+ D["Static fixture files
flatc --binary → committed .bin"]
+ E["Cross-language tests
Rust + Odin read the same fixtures identically"]
+ A --> B --> C --> D --> E
+```
+
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see