updated documentation to have mermaid diagrams; updated AGENTS.md to note that all future diagrams should be mermaid diagrams first with text-based diagrams as fallback where not applicable
This commit is contained in:
@@ -25,6 +25,7 @@ The root `Makefile` is a lean delegator: base commands (`run`, `build`, `test`,
|
||||
- Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path).
|
||||
- Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly.
|
||||
- Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together.
|
||||
- **Diagrams in this repo's documentation are Mermaid flowcharts.** Gitea renders ` ```mermaid ` fenced blocks natively. Prefer a Mermaid flowchart over ASCII art / box-drawing diagrams; if a diagram genuinely can't be expressed as a flowchart, fall back to a plain text-based markdown diagram (e.g. a code block or table) rather than hand-rawn ASCII boxes.
|
||||
|
||||
## Gotchas
|
||||
- Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release.
|
||||
|
||||
@@ -59,11 +59,13 @@ get_catalog_objects :: proc(url: string, catalog_name: string) // nil
|
||||
|
||||
## Data flow gap
|
||||
|
||||
```
|
||||
[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI]
|
||||
^ ^
|
||||
| serde JSON models | hand-mirrored structs
|
||||
| | (stubs, never used)
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["DESI catalog store"]
|
||||
B["Rust / axum API<br/><i>serde JSON models</i>"]
|
||||
C["Odin + raylib GUI<br/><i>hand-mirrored structs<br/>stubs, never used</i>"]
|
||||
A -. "future ingestion" .-> B
|
||||
B --x|"nothing today"| C
|
||||
```
|
||||
|
||||
There is **no live data flow**. The API currently returns JSON placeholders; the
|
||||
@@ -116,17 +118,16 @@ response protocol with a cheap binary payload would fit this well.
|
||||
|
||||
## High-level target architecture
|
||||
|
||||
```
|
||||
catalog.fbs (single source of truth, checked into repo)
|
||||
|
|
||||
+--------+---------+
|
||||
| |
|
||||
flatc --rust flatcc --c (or hand-rolled Odin reader)
|
||||
| |
|
||||
api/ (Rust) gui/ (Odin + raylib)
|
||||
| ^
|
||||
| HTTP / WebSocket (framed FlatBuffer binary stream)
|
||||
+------------------+
|
||||
```mermaid
|
||||
flowchart TB
|
||||
S["catalog.fbs<br/><i>single source of truth<br/>checked into repo</i>"]
|
||||
R["flatc --rust"]
|
||||
C["flatcc --c<br/><i>or hand-rolled Odin reader</i>"]
|
||||
API["api/ · Rust"]
|
||||
GUI["gui/ · Odin + raylib"]
|
||||
S --> R --> API
|
||||
S --> C --> GUI
|
||||
API <-->|"HTTP / WebSocket<br/>framed FlatBuffer binary stream"| GUI
|
||||
```
|
||||
|
||||
- One schema file. Two generators. Byte-for-byte identical wire format.
|
||||
|
||||
@@ -105,19 +105,28 @@ These are the "thou shalt" rules for keeping buffers compatible:
|
||||
|
||||
## Reading a buffer (conceptual)
|
||||
|
||||
Buffer layout:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph BUF["bytes: &[u8]"]
|
||||
O["uoffset<br/>root table offset"]
|
||||
FI["file_identifier<br/>(optional)"]
|
||||
D["tables · vtables · data"]
|
||||
end
|
||||
O --> FI --> D
|
||||
```
|
||||
bytes: &[u8]
|
||||
|
||||
┌─────────────────────────────┐
|
||||
│ uoffset (root table offset) │
|
||||
│ file_identifier (optional) │
|
||||
│ ... tables, vtables, data ...│
|
||||
└─────────────────────────────┘
|
||||
Access sequence — each field is a few offset dereferences and a read:
|
||||
|
||||
root = follow(bytes) // jump to root table via uoffset
|
||||
vtable = root - root.vtable_off // locate vtable for this table
|
||||
field_ra = vtable.slot_ra != 0 // present?
|
||||
if present: ra = read_f64(bytes, root + slot_ra)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
R["root = follow(bytes)<br/>jump to root table via uoffset"]
|
||||
V["vtable = root − root.vtable_off<br/>locate vtable for this table"]
|
||||
Q{"field slot present?"}
|
||||
R --> V --> Q
|
||||
Q -- "no" --> DEF["use schema default"]
|
||||
Q -- "yes" --> RD["ra = read_f64(bytes, root + slot_ra)"]
|
||||
```
|
||||
|
||||
There is **no parsing loop**. Each accessor is a few offset dereferences and a
|
||||
|
||||
@@ -196,6 +196,18 @@ Notes:
|
||||
|
||||
## Building buffers efficiently (Rust specifics)
|
||||
|
||||
Build order is **back-to-front** (children before parents):
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
A["create_string / create_vector<br/>children first"]
|
||||
B["create nested child tables"]
|
||||
C["create parent table<br/>ObjectBatch::create(&args)"]
|
||||
D["builder.finish(root, Some("DESI"))"]
|
||||
E["Bytes::copy_from_slice(fbb.finished_data())<br/>→ HTTP / WebSocket response"]
|
||||
A --> B --> C --> D --> E
|
||||
```
|
||||
|
||||
- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
|
||||
buffer across messages. In a loop streaming batches, create one builder, reuse
|
||||
it — avoid repeated reallocation.
|
||||
|
||||
@@ -17,6 +17,21 @@ has two C-related access points:
|
||||
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
|
||||
`foreign import` can consume it.
|
||||
|
||||
Choosing a path:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
NAT{"primary target is native desktop?"}
|
||||
NAT -- "yes" --> CGO{"want to avoid C in the build?"}
|
||||
CGO -- "yes" --> PATHA["Path B · pure-Odin reader<br/>hand-rolled, no C dependency"]
|
||||
CGO -- "no" --> PATHA
|
||||
CGO -- "prefer proven lib / less maintenance" --> PATHC["Path A · FFI to FlatCC<br/>bind generated C headers"]
|
||||
PATHC --> REUSE["Path C · OdinArrow reuse<br/>or borrow its decode patterns"]
|
||||
NAT -- "no · browser/WASM" --> PATHD["Path D · TS/JS interop<br/>official JS lib → typed arrays into WASM"]
|
||||
```
|
||||
|
||||
Index of paths:
|
||||
|
||||
| Path | Effort | Zero-copy on reads | Notes |
|
||||
|---|---|---|---|
|
||||
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
|
||||
@@ -175,12 +190,14 @@ typed Odin slices.
|
||||
The rendering win only materializes if data stays zero-copy **into the frame
|
||||
loop**:
|
||||
|
||||
1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime
|
||||
of the frame).
|
||||
2. `verify` the buffer once.
|
||||
3. Get `ra_slice := ObjectBatch.ra(&buf)` → `[]f64` view.
|
||||
4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight
|
||||
from that slice; build `rl.Vector3`; `DrawPoint3D`.
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["fetch frame bytes → [dynamic]u8<br/>or a slice pinned for the frame"]
|
||||
B["verify the buffer once"]
|
||||
C["ObjectBatch.ra(&buf) → []f64 view"]
|
||||
D["per object in update()/draw()<br/>ra[i] · dec[i] · z[i] → rl.Vector3 → DrawPoint3D"]
|
||||
A --> B --> C --> D
|
||||
```
|
||||
|
||||
No per-object allocation. The current `Galaxy { position, color }` dynamic array
|
||||
in `main.odin` is the data structure you'd replace with *slices into the
|
||||
|
||||
@@ -24,14 +24,17 @@ offers two built-in mechanisms plus the community pattern:
|
||||
|
||||
### Option 1: Size-prefixed FlatBuffers (built-in)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["u32 LE<br/>total buffer len<br/><i>size prefix</i>"]
|
||||
B["u32 LE<br/>root table offset"]
|
||||
C["file identifier<br/>(4 bytes)"]
|
||||
D["tables · vtables · data"]
|
||||
A --> B --> C --> D
|
||||
```
|
||||
|
||||
```rust
|
||||
builder.finish_size_prefixed(root, Some("DESI"));
|
||||
// +---------------------------+
|
||||
// | u32 LE: total buffer len | <-- size prefix
|
||||
// | u32 LE: root table offset |
|
||||
// | file identifier (4 bytes) |
|
||||
// | ... data ... |
|
||||
// +---------------------------+
|
||||
```
|
||||
|
||||
Reader side:
|
||||
@@ -53,8 +56,12 @@ message kinds).
|
||||
|
||||
### Option 2: Custom length-prefix framing (like `flatstream`)
|
||||
|
||||
```
|
||||
[ u32 LE: message_len ] [ optional checksum (e.g. u32 crc/xxhash) ] [ flatbuffer payload ]
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["u32 LE<br/>message_len"]
|
||||
B["optional checksum<br/>(u32 crc / xxhash)"]
|
||||
C["FlatBuffer payload"]
|
||||
A --> B --> C
|
||||
```
|
||||
|
||||
- `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation
|
||||
@@ -260,6 +267,16 @@ FlatBuffers long-term (mmap-friendly, page-in-what-you-touch).
|
||||
|
||||
## Decision summary for this repo
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["HTTP GET → one FlatBuffer body per batch<br/>validate Rust builder + Odin reader"]
|
||||
B["WebSocket → one Binary message per batch<br/>interactive path · no custom framing"]
|
||||
C["Self-identifying messages<br/>file_identifier "DESI""]
|
||||
D["size-prefixed / flatstream-style framing<br/>or HTTP-range + mmap for static catalogs"]
|
||||
A --> B --> C
|
||||
C -. "later, if needed" .-> D
|
||||
```
|
||||
|
||||
1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust
|
||||
builder + Odin reader (no protocol work at all).
|
||||
2. Then add **WebSocket** with one `Binary` message per batch (no custom framing)
|
||||
|
||||
@@ -108,6 +108,15 @@ message shape — see `10-performance-benchmarks.md`.)
|
||||
|
||||
## Bottom line
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Q1{"zero-copy reads<br/>in the per-frame render loop?"}
|
||||
Q1 -- "no" --> PB["Protobuf / gRPC<br/>decode once into draw buffers"]
|
||||
Q1 -- "yes" --> Q2{"truly columnar?<br/>millions of rows"}
|
||||
Q2 -- "yes" --> ARR["Apache Arrow IPC<br/>via OdinArrow"]
|
||||
Q2 -- "no · batched vectors" --> FB["FlatBuffers · this proposal"]
|
||||
```
|
||||
|
||||
- **FlatBuffers is the best default** for this project: the zero-copy read model
|
||||
matches the render loop, the wire format is compact for numeric vectors, schema
|
||||
evolution fits DESI's release cadence, and the Rust + WASM/JS official story
|
||||
|
||||
@@ -139,7 +139,23 @@ side / flatc* — see the "cross-language fixture" section below.
|
||||
|
||||
### 4. Cross-language conformance suite (THE key integration test)
|
||||
|
||||
This is the test that actually catches incompatibility. Design:
|
||||
This is the test that actually catches incompatibility. Pipeline:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
S["schema/catalog.fbs"]
|
||||
J["testdata/catalog_sample.json"]
|
||||
S --> F["flatc --binary"]
|
||||
J --> F
|
||||
F --> BIN["committed .bin fixtures<br/>repo-checked-in"]
|
||||
BIN --> OT["Odin tests<br/>assert identical values"]
|
||||
BIN --> RT["Rust tests<br/>assert expected values"]
|
||||
RT -. "deterministic builder" .-> PAR["byte-for-byte parity"]
|
||||
OT -. "reads it" .-> PAR
|
||||
PAR -. "catch drift" .-> F
|
||||
```
|
||||
|
||||
Design:
|
||||
|
||||
1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
|
||||
- Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic,
|
||||
|
||||
@@ -70,6 +70,16 @@ directly from the buffer each frame without allocations.
|
||||
|
||||
## Immediate Next Steps (when you're ready to implement)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Prototype schema<br/>catalog.fbs: Catalog · CatalogObject · ServerMessage union"]
|
||||
B["Generate Rust code<br/>flatc --rust → api/build.rs · serve WS via axum"]
|
||||
C["Prototype the Odin reader<br/>flatcc FFI · pure-Odin · OdinArrow"]
|
||||
D["Static fixture files<br/>flatc --binary → committed .bin"]
|
||||
E["Cross-language tests<br/>Rust + Odin read the same fixtures identically"]
|
||||
A --> B --> C --> D --> E
|
||||
```
|
||||
|
||||
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
|
||||
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
|
||||
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see
|
||||
|
||||
Reference in New Issue
Block a user