updated documentation to have mermaid diagrams; updated AGENTS.md to note that all future diagrams should be mermaid diagrams first with text-based diagrams as fallback where not applicable
CI / Detect changed paths (pull_request) Successful in 6s
CI / Odin unit tests and build (pull_request) Successful in 1m23s
CI / API unit tests and lint (pull_request) Has been skipped
CI / Infra unit tests, vet, and preview (pull_request) Successful in 1m22s

This commit is contained in:
2026-09-06 14:28:48 -06:00
parent 8c74efcedb
commit 3803787fe8
9 changed files with 133 additions and 41 deletions
@@ -59,11 +59,13 @@ get_catalog_objects :: proc(url: string, catalog_name: string) // nil
## Data flow gap
```
[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI]
^ ^
| serde JSON models | hand-mirrored structs
| | (stubs, never used)
```mermaid
flowchart LR
A["DESI catalog store"]
B["Rust / axum API<br/><i>serde JSON models</i>"]
C["Odin + raylib GUI<br/><i>hand-mirrored structs<br/>stubs, never used</i>"]
A -. "future ingestion" .-> B
B --x|"nothing today"| C
```
There is **no live data flow**. The API currently returns JSON placeholders; the
@@ -116,17 +118,16 @@ response protocol with a cheap binary payload would fit this well.
## High-level target architecture
```
catalog.fbs (single source of truth, checked into repo)
|
+--------+---------+
| |
flatc --rust flatcc --c (or hand-rolled Odin reader)
| |
api/ (Rust) gui/ (Odin + raylib)
| ^
| HTTP / WebSocket (framed FlatBuffer binary stream)
+------------------+
```mermaid
flowchart TB
S["catalog.fbs<br/><i>single source of truth<br/>checked into repo</i>"]
R["flatc --rust"]
C["flatcc --c<br/><i>or hand-rolled Odin reader</i>"]
API["api/ · Rust"]
GUI["gui/ · Odin + raylib"]
S --> R --> API
S --> C --> GUI
API <-->|"HTTP / WebSocket<br/>framed FlatBuffer binary stream"| GUI
```
- One schema file. Two generators. Byte-for-byte identical wire format.
@@ -105,19 +105,28 @@ These are the "thou shalt" rules for keeping buffers compatible:
## Reading a buffer (conceptual)
Buffer layout:
```mermaid
flowchart LR
subgraph BUF["bytes: &[u8]"]
O["uoffset<br/>root table offset"]
FI["file_identifier<br/>(optional)"]
D["tables · vtables · data"]
end
O --> FI --> D
```
bytes: &[u8]
┌─────────────────────────────┐
│ uoffset (root table offset) │
│ file_identifier (optional) │
│ ... tables, vtables, data ...│
└─────────────────────────────┘
Access sequence — each field is a few offset dereferences and a read:
root = follow(bytes) // jump to root table via uoffset
vtable = root - root.vtable_off // locate vtable for this table
field_ra = vtable.slot_ra != 0 // present?
if present: ra = read_f64(bytes, root + slot_ra)
```mermaid
flowchart TD
R["root = follow(bytes)<br/>jump to root table via uoffset"]
V["vtable = root root.vtable_off<br/>locate vtable for this table"]
Q{"field slot present?"}
R --> V --> Q
Q -- "no" --> DEF["use schema default"]
Q -- "yes" --> RD["ra = read_f64(bytes, root + slot_ra)"]
```
There is **no parsing loop**. Each accessor is a few offset dereferences and a
@@ -196,6 +196,18 @@ Notes:
## Building buffers efficiently (Rust specifics)
Build order is **back-to-front** (children before parents):
```mermaid
flowchart TB
A["create_string / create_vector<br/>children first"]
B["create nested child tables"]
C["create parent table<br/>ObjectBatch::create(&args)"]
D["builder.finish(root, Some(&#34;DESI&#34;))"]
E["Bytes::copy_from_slice(fbb.finished_data())<br/>→ HTTP / WebSocket response"]
A --> B --> C --> D --> E
```
- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
buffer across messages. In a loop streaming batches, create one builder, reuse
it — avoid repeated reallocation.
@@ -17,6 +17,21 @@ has two C-related access points:
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
`foreign import` can consume it.
Choosing a path:
```mermaid
flowchart TD
NAT{"primary target is native desktop?"}
NAT -- "yes" --> CGO{"want to avoid C in the build?"}
CGO -- "yes" --> PATHA["Path B · pure-Odin reader<br/>hand-rolled, no C dependency"]
CGO -- "no" --> PATHA
CGO -- "prefer proven lib / less maintenance" --> PATHC["Path A · FFI to FlatCC<br/>bind generated C headers"]
PATHC --> REUSE["Path C · OdinArrow reuse<br/>or borrow its decode patterns"]
NAT -- "no · browser/WASM" --> PATHD["Path D · TS/JS interop<br/>official JS lib → typed arrays into WASM"]
```
Index of paths:
| Path | Effort | Zero-copy on reads | Notes |
|---|---|---|---|
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
@@ -175,12 +190,14 @@ typed Odin slices.
The rendering win only materializes if data stays zero-copy **into the frame
loop**:
1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime
of the frame).
2. `verify` the buffer once.
3. Get `ra_slice := ObjectBatch.ra(&buf)` → `[]f64` view.
4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight
from that slice; build `rl.Vector3`; `DrawPoint3D`.
```mermaid
flowchart LR
A["fetch frame bytes → [dynamic]u8<br/>or a slice pinned for the frame"]
B["verify the buffer once"]
C["ObjectBatch.ra(&buf) → []f64 view"]
D["per object in update()/draw()<br/>ra[i] · dec[i] · z[i] → rl.Vector3DrawPoint3D"]
A --> B --> C --> D
```
No per-object allocation. The current `Galaxy { position, color }` dynamic array
in `main.odin` is the data structure you'd replace with *slices into the
@@ -24,14 +24,17 @@ offers two built-in mechanisms plus the community pattern:
### Option 1: Size-prefixed FlatBuffers (built-in)
```mermaid
flowchart LR
A["u32 LE<br/>total buffer len<br/><i>size prefix</i>"]
B["u32 LE<br/>root table offset"]
C["file identifier<br/>(4 bytes)"]
D["tables · vtables · data"]
A --> B --> C --> D
```
```rust
builder.finish_size_prefixed(root, Some("DESI"));
// +---------------------------+
// | u32 LE: total buffer len | <-- size prefix
// | u32 LE: root table offset |
// | file identifier (4 bytes) |
// | ... data ... |
// +---------------------------+
```
Reader side:
@@ -53,8 +56,12 @@ message kinds).
### Option 2: Custom length-prefix framing (like `flatstream`)
```
[ u32 LE: message_len ] [ optional checksum (e.g. u32 crc/xxhash) ] [ flatbuffer payload ]
```mermaid
flowchart LR
A["u32 LE<br/>message_len"]
B["optional checksum<br/>(u32 crc / xxhash)"]
C["FlatBuffer payload"]
A --> B --> C
```
- `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation
@@ -260,6 +267,16 @@ FlatBuffers long-term (mmap-friendly, page-in-what-you-touch).
## Decision summary for this repo
```mermaid
flowchart TD
A["HTTP GET → one FlatBuffer body per batch<br/>validate Rust builder + Odin reader"]
B["WebSocket → one Binary message per batch<br/>interactive path · no custom framing"]
C["Self-identifying messages<br/>file_identifier &#34;DESI&#34;"]
D["size-prefixed / flatstream-style framing<br/>or HTTP-range + mmap for static catalogs"]
A --> B --> C
C -. "later, if needed" .-> D
```
1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust
builder + Odin reader (no protocol work at all).
2. Then add **WebSocket** with one `Binary` message per batch (no custom framing)
@@ -108,6 +108,15 @@ message shape — see `10-performance-benchmarks.md`.)
## Bottom line
```mermaid
flowchart TD
Q1{"zero-copy reads<br/>in the per-frame render loop?"}
Q1 -- "no" --> PB["Protobuf / gRPC<br/>decode once into draw buffers"]
Q1 -- "yes" --> Q2{"truly columnar?<br/>millions of rows"}
Q2 -- "yes" --> ARR["Apache Arrow IPC<br/>via OdinArrow"]
Q2 -- "no · batched vectors" --> FB["FlatBuffers · this proposal"]
```
- **FlatBuffers is the best default** for this project: the zero-copy read model
matches the render loop, the wire format is compact for numeric vectors, schema
evolution fits DESI's release cadence, and the Rust + WASM/JS official story
@@ -139,7 +139,23 @@ side / flatc* — see the "cross-language fixture" section below.
### 4. Cross-language conformance suite (THE key integration test)
This is the test that actually catches incompatibility. Design:
This is the test that actually catches incompatibility. Pipeline:
```mermaid
flowchart LR
S["schema/catalog.fbs"]
J["testdata/catalog_sample.json"]
S --> F["flatc --binary"]
J --> F
F --> BIN["committed .bin fixtures<br/>repo-checked-in"]
BIN --> OT["Odin tests<br/>assert identical values"]
BIN --> RT["Rust tests<br/>assert expected values"]
RT -. "deterministic builder" .-> PAR["byte-for-byte parity"]
OT -. "reads it" .-> PAR
PAR -. "catch drift" .-> F
```
Design:
1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
- Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic,
@@ -70,6 +70,16 @@ directly from the buffer each frame without allocations.
## Immediate Next Steps (when you're ready to implement)
```mermaid
flowchart TD
A["Prototype schema<br/>catalog.fbs: Catalog · CatalogObject · ServerMessage union"]
B["Generate Rust code<br/>flatc --rust → api/build.rs · serve WS via axum"]
C["Prototype the Odin reader<br/>flatcc FFI · pure-Odin · OdinArrow"]
D["Static fixture files<br/>flatc --binary → committed .bin"]
E["Cross-language tests<br/>Rust + Odin read the same fixtures identically"]
A --> B --> C --> D --> E
```
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see