updated documentation to have mermaid diagrams; updated AGENTS.md to note that all future diagrams should be mermaid diagrams first with text-based diagrams as fallback where not applicable
CI / Detect changed paths (pull_request) Successful in 6s
CI / Odin unit tests and build (pull_request) Successful in 1m23s
CI / API unit tests and lint (pull_request) Has been skipped
CI / Infra unit tests, vet, and preview (pull_request) Successful in 1m22s

This commit is contained in:
2026-09-06 14:28:48 -06:00
parent 8c74efcedb
commit 3803787fe8
9 changed files with 133 additions and 41 deletions
+1
View File
@@ -25,6 +25,7 @@ The root `Makefile` is a lean delegator: base commands (`run`, `build`, `test`,
- Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path). - Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path).
- Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly. - Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly.
- Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together. - Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together.
- **Diagrams in this repo's documentation are Mermaid flowcharts.** Gitea renders ` ```mermaid ` fenced blocks natively. Prefer a Mermaid flowchart over ASCII art / box-drawing diagrams; if a diagram genuinely can't be expressed as a flowchart, fall back to a plain text-based markdown diagram (e.g. a code block or table) rather than hand-rawn ASCII boxes.
## Gotchas ## Gotchas
- Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release. - Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release.
@@ -59,11 +59,13 @@ get_catalog_objects :: proc(url: string, catalog_name: string) // nil
## Data flow gap ## Data flow gap
``` ```mermaid
[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI] flowchart LR
^ ^ A["DESI catalog store"]
| serde JSON models | hand-mirrored structs B["Rust / axum API<br/><i>serde JSON models</i>"]
| | (stubs, never used) C["Odin + raylib GUI<br/><i>hand-mirrored structs<br/>stubs, never used</i>"]
A -. "future ingestion" .-> B
B --x|"nothing today"| C
``` ```
There is **no live data flow**. The API currently returns JSON placeholders; the There is **no live data flow**. The API currently returns JSON placeholders; the
@@ -116,17 +118,16 @@ response protocol with a cheap binary payload would fit this well.
## High-level target architecture ## High-level target architecture
``` ```mermaid
catalog.fbs (single source of truth, checked into repo) flowchart TB
| S["catalog.fbs<br/><i>single source of truth<br/>checked into repo</i>"]
+--------+---------+ R["flatc --rust"]
| | C["flatcc --c<br/><i>or hand-rolled Odin reader</i>"]
flatc --rust flatcc --c (or hand-rolled Odin reader) API["api/ · Rust"]
| | GUI["gui/ · Odin + raylib"]
api/ (Rust) gui/ (Odin + raylib) S --> R --> API
| ^ S --> C --> GUI
| HTTP / WebSocket (framed FlatBuffer binary stream) API <-->|"HTTP / WebSocket<br/>framed FlatBuffer binary stream"| GUI
+------------------+
``` ```
- One schema file. Two generators. Byte-for-byte identical wire format. - One schema file. Two generators. Byte-for-byte identical wire format.
@@ -105,19 +105,28 @@ These are the "thou shalt" rules for keeping buffers compatible:
## Reading a buffer (conceptual) ## Reading a buffer (conceptual)
Buffer layout:
```mermaid
flowchart LR
subgraph BUF["bytes: &[u8]"]
O["uoffset<br/>root table offset"]
FI["file_identifier<br/>(optional)"]
D["tables · vtables · data"]
end
O --> FI --> D
``` ```
bytes: &[u8]
┌─────────────────────────────┐ Access sequence — each field is a few offset dereferences and a read:
│ uoffset (root table offset) │
│ file_identifier (optional) │
│ ... tables, vtables, data ...│
└─────────────────────────────┘
root = follow(bytes) // jump to root table via uoffset ```mermaid
vtable = root - root.vtable_off // locate vtable for this table flowchart TD
field_ra = vtable.slot_ra != 0 // present? R["root = follow(bytes)<br/>jump to root table via uoffset"]
if present: ra = read_f64(bytes, root + slot_ra) V["vtable = root root.vtable_off<br/>locate vtable for this table"]
Q{"field slot present?"}
R --> V --> Q
Q -- "no" --> DEF["use schema default"]
Q -- "yes" --> RD["ra = read_f64(bytes, root + slot_ra)"]
``` ```
There is **no parsing loop**. Each accessor is a few offset dereferences and a There is **no parsing loop**. Each accessor is a few offset dereferences and a
@@ -196,6 +196,18 @@ Notes:
## Building buffers efficiently (Rust specifics) ## Building buffers efficiently (Rust specifics)
Build order is **back-to-front** (children before parents):
```mermaid
flowchart TB
A["create_string / create_vector<br/>children first"]
B["create nested child tables"]
C["create parent table<br/>ObjectBatch::create(&args)"]
D["builder.finish(root, Some(&#34;DESI&#34;))"]
E["Bytes::copy_from_slice(fbb.finished_data())<br/>→ HTTP / WebSocket response"]
A --> B --> C --> D --> E
```
- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the - `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
buffer across messages. In a loop streaming batches, create one builder, reuse buffer across messages. In a loop streaming batches, create one builder, reuse
it — avoid repeated reallocation. it — avoid repeated reallocation.
@@ -17,6 +17,21 @@ has two C-related access points:
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
`foreign import` can consume it. `foreign import` can consume it.
Choosing a path:
```mermaid
flowchart TD
NAT{"primary target is native desktop?"}
NAT -- "yes" --> CGO{"want to avoid C in the build?"}
CGO -- "yes" --> PATHA["Path B · pure-Odin reader<br/>hand-rolled, no C dependency"]
CGO -- "no" --> PATHA
CGO -- "prefer proven lib / less maintenance" --> PATHC["Path A · FFI to FlatCC<br/>bind generated C headers"]
PATHC --> REUSE["Path C · OdinArrow reuse<br/>or borrow its decode patterns"]
NAT -- "no · browser/WASM" --> PATHD["Path D · TS/JS interop<br/>official JS lib → typed arrays into WASM"]
```
Index of paths:
| Path | Effort | Zero-copy on reads | Notes | | Path | Effort | Zero-copy on reads | Notes |
|---|---|---|---| |---|---|---|---|
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` | | A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
@@ -175,12 +190,14 @@ typed Odin slices.
The rendering win only materializes if data stays zero-copy **into the frame The rendering win only materializes if data stays zero-copy **into the frame
loop**: loop**:
1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime ```mermaid
of the frame). flowchart LR
2. `verify` the buffer once. A["fetch frame bytes → [dynamic]u8<br/>or a slice pinned for the frame"]
3. Get `ra_slice := ObjectBatch.ra(&buf)``[]f64` view. B["verify the buffer once"]
4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight C["ObjectBatch.ra(&buf) → []f64 view"]
from that slice; build `rl.Vector3`; `DrawPoint3D`. D["per object in update()/draw()<br/>ra[i] · dec[i] · z[i] → rl.Vector3DrawPoint3D"]
A --> B --> C --> D
```
No per-object allocation. The current `Galaxy { position, color }` dynamic array No per-object allocation. The current `Galaxy { position, color }` dynamic array
in `main.odin` is the data structure you'd replace with *slices into the in `main.odin` is the data structure you'd replace with *slices into the
@@ -24,14 +24,17 @@ offers two built-in mechanisms plus the community pattern:
### Option 1: Size-prefixed FlatBuffers (built-in) ### Option 1: Size-prefixed FlatBuffers (built-in)
```mermaid
flowchart LR
A["u32 LE<br/>total buffer len<br/><i>size prefix</i>"]
B["u32 LE<br/>root table offset"]
C["file identifier<br/>(4 bytes)"]
D["tables · vtables · data"]
A --> B --> C --> D
```
```rust ```rust
builder.finish_size_prefixed(root, Some("DESI")); builder.finish_size_prefixed(root, Some("DESI"));
// +---------------------------+
// | u32 LE: total buffer len | <-- size prefix
// | u32 LE: root table offset |
// | file identifier (4 bytes) |
// | ... data ... |
// +---------------------------+
``` ```
Reader side: Reader side:
@@ -53,8 +56,12 @@ message kinds).
### Option 2: Custom length-prefix framing (like `flatstream`) ### Option 2: Custom length-prefix framing (like `flatstream`)
``` ```mermaid
[ u32 LE: message_len ] [ optional checksum (e.g. u32 crc/xxhash) ] [ flatbuffer payload ] flowchart LR
A["u32 LE<br/>message_len"]
B["optional checksum<br/>(u32 crc / xxhash)"]
C["FlatBuffer payload"]
A --> B --> C
``` ```
- `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation - `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation
@@ -260,6 +267,16 @@ FlatBuffers long-term (mmap-friendly, page-in-what-you-touch).
## Decision summary for this repo ## Decision summary for this repo
```mermaid
flowchart TD
A["HTTP GET → one FlatBuffer body per batch<br/>validate Rust builder + Odin reader"]
B["WebSocket → one Binary message per batch<br/>interactive path · no custom framing"]
C["Self-identifying messages<br/>file_identifier &#34;DESI&#34;"]
D["size-prefixed / flatstream-style framing<br/>or HTTP-range + mmap for static catalogs"]
A --> B --> C
C -. "later, if needed" .-> D
```
1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust 1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust
builder + Odin reader (no protocol work at all). builder + Odin reader (no protocol work at all).
2. Then add **WebSocket** with one `Binary` message per batch (no custom framing) 2. Then add **WebSocket** with one `Binary` message per batch (no custom framing)
@@ -108,6 +108,15 @@ message shape — see `10-performance-benchmarks.md`.)
## Bottom line ## Bottom line
```mermaid
flowchart TD
Q1{"zero-copy reads<br/>in the per-frame render loop?"}
Q1 -- "no" --> PB["Protobuf / gRPC<br/>decode once into draw buffers"]
Q1 -- "yes" --> Q2{"truly columnar?<br/>millions of rows"}
Q2 -- "yes" --> ARR["Apache Arrow IPC<br/>via OdinArrow"]
Q2 -- "no · batched vectors" --> FB["FlatBuffers · this proposal"]
```
- **FlatBuffers is the best default** for this project: the zero-copy read model - **FlatBuffers is the best default** for this project: the zero-copy read model
matches the render loop, the wire format is compact for numeric vectors, schema matches the render loop, the wire format is compact for numeric vectors, schema
evolution fits DESI's release cadence, and the Rust + WASM/JS official story evolution fits DESI's release cadence, and the Rust + WASM/JS official story
@@ -139,7 +139,23 @@ side / flatc* — see the "cross-language fixture" section below.
### 4. Cross-language conformance suite (THE key integration test) ### 4. Cross-language conformance suite (THE key integration test)
This is the test that actually catches incompatibility. Design: This is the test that actually catches incompatibility. Pipeline:
```mermaid
flowchart LR
S["schema/catalog.fbs"]
J["testdata/catalog_sample.json"]
S --> F["flatc --binary"]
J --> F
F --> BIN["committed .bin fixtures<br/>repo-checked-in"]
BIN --> OT["Odin tests<br/>assert identical values"]
BIN --> RT["Rust tests<br/>assert expected values"]
RT -. "deterministic builder" .-> PAR["byte-for-byte parity"]
OT -. "reads it" .-> PAR
PAR -. "catch drift" .-> F
```
Design:
1. **Static fixtures, committed to the repo** (`testdata/*.bin`): 1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
- Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic, - Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic,
@@ -70,6 +70,16 @@ directly from the buffer each frame without allocations.
## Immediate Next Steps (when you're ready to implement) ## Immediate Next Steps (when you're ready to implement)
```mermaid
flowchart TD
A["Prototype schema<br/>catalog.fbs: Catalog · CatalogObject · ServerMessage union"]
B["Generate Rust code<br/>flatc --rust → api/build.rs · serve WS via axum"]
C["Prototype the Odin reader<br/>flatcc FFI · pure-Odin · OdinArrow"]
D["Static fixture files<br/>flatc --binary → committed .bin"]
E["Cross-language tests<br/>Rust + Odin read the same fixtures identically"]
A --> B --> C --> D --> E
```
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`, 1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end). and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see 2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see