updated documentation to have mermaid diagrams; updated AGENTS.md to note that all future diagrams should be mermaid diagrams first with text-based diagrams as fallback where not applicable
This commit is contained in:
@@ -25,6 +25,7 @@ The root `Makefile` is a lean delegator: base commands (`run`, `build`, `test`,
|
|||||||
- Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path).
|
- Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path).
|
||||||
- Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly.
|
- Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly.
|
||||||
- Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together.
|
- Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together.
|
||||||
|
- **Diagrams in this repo's documentation are Mermaid flowcharts.** Gitea renders ` ```mermaid ` fenced blocks natively. Prefer a Mermaid flowchart over ASCII art / box-drawing diagrams; if a diagram genuinely can't be expressed as a flowchart, fall back to a plain text-based markdown diagram (e.g. a code block or table) rather than hand-rawn ASCII boxes.
|
||||||
|
|
||||||
## Gotchas
|
## Gotchas
|
||||||
- Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release.
|
- Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release.
|
||||||
|
|||||||
@@ -59,11 +59,13 @@ get_catalog_objects :: proc(url: string, catalog_name: string) // nil
|
|||||||
|
|
||||||
## Data flow gap
|
## Data flow gap
|
||||||
|
|
||||||
```
|
```mermaid
|
||||||
[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI]
|
flowchart LR
|
||||||
^ ^
|
A["DESI catalog store"]
|
||||||
| serde JSON models | hand-mirrored structs
|
B["Rust / axum API<br/><i>serde JSON models</i>"]
|
||||||
| | (stubs, never used)
|
C["Odin + raylib GUI<br/><i>hand-mirrored structs<br/>stubs, never used</i>"]
|
||||||
|
A -. "future ingestion" .-> B
|
||||||
|
B --x|"nothing today"| C
|
||||||
```
|
```
|
||||||
|
|
||||||
There is **no live data flow**. The API currently returns JSON placeholders; the
|
There is **no live data flow**. The API currently returns JSON placeholders; the
|
||||||
@@ -116,17 +118,16 @@ response protocol with a cheap binary payload would fit this well.
|
|||||||
|
|
||||||
## High-level target architecture
|
## High-level target architecture
|
||||||
|
|
||||||
```
|
```mermaid
|
||||||
catalog.fbs (single source of truth, checked into repo)
|
flowchart TB
|
||||||
|
|
S["catalog.fbs<br/><i>single source of truth<br/>checked into repo</i>"]
|
||||||
+--------+---------+
|
R["flatc --rust"]
|
||||||
| |
|
C["flatcc --c<br/><i>or hand-rolled Odin reader</i>"]
|
||||||
flatc --rust flatcc --c (or hand-rolled Odin reader)
|
API["api/ · Rust"]
|
||||||
| |
|
GUI["gui/ · Odin + raylib"]
|
||||||
api/ (Rust) gui/ (Odin + raylib)
|
S --> R --> API
|
||||||
| ^
|
S --> C --> GUI
|
||||||
| HTTP / WebSocket (framed FlatBuffer binary stream)
|
API <-->|"HTTP / WebSocket<br/>framed FlatBuffer binary stream"| GUI
|
||||||
+------------------+
|
|
||||||
```
|
```
|
||||||
|
|
||||||
- One schema file. Two generators. Byte-for-byte identical wire format.
|
- One schema file. Two generators. Byte-for-byte identical wire format.
|
||||||
|
|||||||
@@ -105,19 +105,28 @@ These are the "thou shalt" rules for keeping buffers compatible:
|
|||||||
|
|
||||||
## Reading a buffer (conceptual)
|
## Reading a buffer (conceptual)
|
||||||
|
|
||||||
|
Buffer layout:
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph BUF["bytes: &[u8]"]
|
||||||
|
O["uoffset<br/>root table offset"]
|
||||||
|
FI["file_identifier<br/>(optional)"]
|
||||||
|
D["tables · vtables · data"]
|
||||||
|
end
|
||||||
|
O --> FI --> D
|
||||||
```
|
```
|
||||||
bytes: &[u8]
|
|
||||||
|
|
||||||
┌─────────────────────────────┐
|
Access sequence — each field is a few offset dereferences and a read:
|
||||||
│ uoffset (root table offset) │
|
|
||||||
│ file_identifier (optional) │
|
|
||||||
│ ... tables, vtables, data ...│
|
|
||||||
└─────────────────────────────┘
|
|
||||||
|
|
||||||
root = follow(bytes) // jump to root table via uoffset
|
```mermaid
|
||||||
vtable = root - root.vtable_off // locate vtable for this table
|
flowchart TD
|
||||||
field_ra = vtable.slot_ra != 0 // present?
|
R["root = follow(bytes)<br/>jump to root table via uoffset"]
|
||||||
if present: ra = read_f64(bytes, root + slot_ra)
|
V["vtable = root − root.vtable_off<br/>locate vtable for this table"]
|
||||||
|
Q{"field slot present?"}
|
||||||
|
R --> V --> Q
|
||||||
|
Q -- "no" --> DEF["use schema default"]
|
||||||
|
Q -- "yes" --> RD["ra = read_f64(bytes, root + slot_ra)"]
|
||||||
```
|
```
|
||||||
|
|
||||||
There is **no parsing loop**. Each accessor is a few offset dereferences and a
|
There is **no parsing loop**. Each accessor is a few offset dereferences and a
|
||||||
|
|||||||
@@ -196,6 +196,18 @@ Notes:
|
|||||||
|
|
||||||
## Building buffers efficiently (Rust specifics)
|
## Building buffers efficiently (Rust specifics)
|
||||||
|
|
||||||
|
Build order is **back-to-front** (children before parents):
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TB
|
||||||
|
A["create_string / create_vector<br/>children first"]
|
||||||
|
B["create nested child tables"]
|
||||||
|
C["create parent table<br/>ObjectBatch::create(&args)"]
|
||||||
|
D["builder.finish(root, Some("DESI"))"]
|
||||||
|
E["Bytes::copy_from_slice(fbb.finished_data())<br/>→ HTTP / WebSocket response"]
|
||||||
|
A --> B --> C --> D --> E
|
||||||
|
```
|
||||||
|
|
||||||
- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
|
- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
|
||||||
buffer across messages. In a loop streaming batches, create one builder, reuse
|
buffer across messages. In a loop streaming batches, create one builder, reuse
|
||||||
it — avoid repeated reallocation.
|
it — avoid repeated reallocation.
|
||||||
|
|||||||
@@ -17,6 +17,21 @@ has two C-related access points:
|
|||||||
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
|
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
|
||||||
`foreign import` can consume it.
|
`foreign import` can consume it.
|
||||||
|
|
||||||
|
Choosing a path:
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
NAT{"primary target is native desktop?"}
|
||||||
|
NAT -- "yes" --> CGO{"want to avoid C in the build?"}
|
||||||
|
CGO -- "yes" --> PATHA["Path B · pure-Odin reader<br/>hand-rolled, no C dependency"]
|
||||||
|
CGO -- "no" --> PATHA
|
||||||
|
CGO -- "prefer proven lib / less maintenance" --> PATHC["Path A · FFI to FlatCC<br/>bind generated C headers"]
|
||||||
|
PATHC --> REUSE["Path C · OdinArrow reuse<br/>or borrow its decode patterns"]
|
||||||
|
NAT -- "no · browser/WASM" --> PATHD["Path D · TS/JS interop<br/>official JS lib → typed arrays into WASM"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Index of paths:
|
||||||
|
|
||||||
| Path | Effort | Zero-copy on reads | Notes |
|
| Path | Effort | Zero-copy on reads | Notes |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
|
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
|
||||||
@@ -175,12 +190,14 @@ typed Odin slices.
|
|||||||
The rendering win only materializes if data stays zero-copy **into the frame
|
The rendering win only materializes if data stays zero-copy **into the frame
|
||||||
loop**:
|
loop**:
|
||||||
|
|
||||||
1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime
|
```mermaid
|
||||||
of the frame).
|
flowchart LR
|
||||||
2. `verify` the buffer once.
|
A["fetch frame bytes → [dynamic]u8<br/>or a slice pinned for the frame"]
|
||||||
3. Get `ra_slice := ObjectBatch.ra(&buf)` → `[]f64` view.
|
B["verify the buffer once"]
|
||||||
4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight
|
C["ObjectBatch.ra(&buf) → []f64 view"]
|
||||||
from that slice; build `rl.Vector3`; `DrawPoint3D`.
|
D["per object in update()/draw()<br/>ra[i] · dec[i] · z[i] → rl.Vector3 → DrawPoint3D"]
|
||||||
|
A --> B --> C --> D
|
||||||
|
```
|
||||||
|
|
||||||
No per-object allocation. The current `Galaxy { position, color }` dynamic array
|
No per-object allocation. The current `Galaxy { position, color }` dynamic array
|
||||||
in `main.odin` is the data structure you'd replace with *slices into the
|
in `main.odin` is the data structure you'd replace with *slices into the
|
||||||
|
|||||||
@@ -24,14 +24,17 @@ offers two built-in mechanisms plus the community pattern:
|
|||||||
|
|
||||||
### Option 1: Size-prefixed FlatBuffers (built-in)
|
### Option 1: Size-prefixed FlatBuffers (built-in)
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
A["u32 LE<br/>total buffer len<br/><i>size prefix</i>"]
|
||||||
|
B["u32 LE<br/>root table offset"]
|
||||||
|
C["file identifier<br/>(4 bytes)"]
|
||||||
|
D["tables · vtables · data"]
|
||||||
|
A --> B --> C --> D
|
||||||
|
```
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
builder.finish_size_prefixed(root, Some("DESI"));
|
builder.finish_size_prefixed(root, Some("DESI"));
|
||||||
// +---------------------------+
|
|
||||||
// | u32 LE: total buffer len | <-- size prefix
|
|
||||||
// | u32 LE: root table offset |
|
|
||||||
// | file identifier (4 bytes) |
|
|
||||||
// | ... data ... |
|
|
||||||
// +---------------------------+
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Reader side:
|
Reader side:
|
||||||
@@ -53,8 +56,12 @@ message kinds).
|
|||||||
|
|
||||||
### Option 2: Custom length-prefix framing (like `flatstream`)
|
### Option 2: Custom length-prefix framing (like `flatstream`)
|
||||||
|
|
||||||
```
|
```mermaid
|
||||||
[ u32 LE: message_len ] [ optional checksum (e.g. u32 crc/xxhash) ] [ flatbuffer payload ]
|
flowchart LR
|
||||||
|
A["u32 LE<br/>message_len"]
|
||||||
|
B["optional checksum<br/>(u32 crc / xxhash)"]
|
||||||
|
C["FlatBuffer payload"]
|
||||||
|
A --> B --> C
|
||||||
```
|
```
|
||||||
|
|
||||||
- `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation
|
- `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation
|
||||||
@@ -260,6 +267,16 @@ FlatBuffers long-term (mmap-friendly, page-in-what-you-touch).
|
|||||||
|
|
||||||
## Decision summary for this repo
|
## Decision summary for this repo
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
A["HTTP GET → one FlatBuffer body per batch<br/>validate Rust builder + Odin reader"]
|
||||||
|
B["WebSocket → one Binary message per batch<br/>interactive path · no custom framing"]
|
||||||
|
C["Self-identifying messages<br/>file_identifier "DESI""]
|
||||||
|
D["size-prefixed / flatstream-style framing<br/>or HTTP-range + mmap for static catalogs"]
|
||||||
|
A --> B --> C
|
||||||
|
C -. "later, if needed" .-> D
|
||||||
|
```
|
||||||
|
|
||||||
1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust
|
1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust
|
||||||
builder + Odin reader (no protocol work at all).
|
builder + Odin reader (no protocol work at all).
|
||||||
2. Then add **WebSocket** with one `Binary` message per batch (no custom framing)
|
2. Then add **WebSocket** with one `Binary` message per batch (no custom framing)
|
||||||
|
|||||||
@@ -108,6 +108,15 @@ message shape — see `10-performance-benchmarks.md`.)
|
|||||||
|
|
||||||
## Bottom line
|
## Bottom line
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
Q1{"zero-copy reads<br/>in the per-frame render loop?"}
|
||||||
|
Q1 -- "no" --> PB["Protobuf / gRPC<br/>decode once into draw buffers"]
|
||||||
|
Q1 -- "yes" --> Q2{"truly columnar?<br/>millions of rows"}
|
||||||
|
Q2 -- "yes" --> ARR["Apache Arrow IPC<br/>via OdinArrow"]
|
||||||
|
Q2 -- "no · batched vectors" --> FB["FlatBuffers · this proposal"]
|
||||||
|
```
|
||||||
|
|
||||||
- **FlatBuffers is the best default** for this project: the zero-copy read model
|
- **FlatBuffers is the best default** for this project: the zero-copy read model
|
||||||
matches the render loop, the wire format is compact for numeric vectors, schema
|
matches the render loop, the wire format is compact for numeric vectors, schema
|
||||||
evolution fits DESI's release cadence, and the Rust + WASM/JS official story
|
evolution fits DESI's release cadence, and the Rust + WASM/JS official story
|
||||||
|
|||||||
@@ -139,7 +139,23 @@ side / flatc* — see the "cross-language fixture" section below.
|
|||||||
|
|
||||||
### 4. Cross-language conformance suite (THE key integration test)
|
### 4. Cross-language conformance suite (THE key integration test)
|
||||||
|
|
||||||
This is the test that actually catches incompatibility. Design:
|
This is the test that actually catches incompatibility. Pipeline:
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
S["schema/catalog.fbs"]
|
||||||
|
J["testdata/catalog_sample.json"]
|
||||||
|
S --> F["flatc --binary"]
|
||||||
|
J --> F
|
||||||
|
F --> BIN["committed .bin fixtures<br/>repo-checked-in"]
|
||||||
|
BIN --> OT["Odin tests<br/>assert identical values"]
|
||||||
|
BIN --> RT["Rust tests<br/>assert expected values"]
|
||||||
|
RT -. "deterministic builder" .-> PAR["byte-for-byte parity"]
|
||||||
|
OT -. "reads it" .-> PAR
|
||||||
|
PAR -. "catch drift" .-> F
|
||||||
|
```
|
||||||
|
|
||||||
|
Design:
|
||||||
|
|
||||||
1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
|
1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
|
||||||
- Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic,
|
- Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic,
|
||||||
|
|||||||
@@ -70,6 +70,16 @@ directly from the buffer each frame without allocations.
|
|||||||
|
|
||||||
## Immediate Next Steps (when you're ready to implement)
|
## Immediate Next Steps (when you're ready to implement)
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
A["Prototype schema<br/>catalog.fbs: Catalog · CatalogObject · ServerMessage union"]
|
||||||
|
B["Generate Rust code<br/>flatc --rust → api/build.rs · serve WS via axum"]
|
||||||
|
C["Prototype the Odin reader<br/>flatcc FFI · pure-Odin · OdinArrow"]
|
||||||
|
D["Static fixture files<br/>flatc --binary → committed .bin"]
|
||||||
|
E["Cross-language tests<br/>Rust + Odin read the same fixtures identically"]
|
||||||
|
A --> B --> C --> D --> E
|
||||||
|
```
|
||||||
|
|
||||||
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
|
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
|
||||||
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
|
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
|
||||||
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see
|
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see
|
||||||
|
|||||||
Reference in New Issue
Block a user