updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,218 @@
|
||||
# 04 — Odin / GUI Integration
|
||||
|
||||
**This is the highest-risk area of the whole research effort.** FlatBuffers has
|
||||
**no first-party Odin binding**. You must choose one of several integration
|
||||
paths. This doc lays out the realistic options with tradeoffs, plus the
|
||||
networking story for the GUI.
|
||||
|
||||
## The core problem
|
||||
|
||||
Odin can talk to C via `foreign` blocks (the C ABI is first-class). FlatBuffers
|
||||
has two C-related access points:
|
||||
|
||||
1. **Google's official `flatc`** generates C++ and C (C#/Java/etc.) — but its
|
||||
C support is a *separate project*, **FlatCC**.
|
||||
2. **FlatCC** (`dvidelabs/flatcc`) is an independent FlatBuffers compiler +
|
||||
runtime for **pure C**. It generates `_reader.h`/`_builder.h` headers per
|
||||
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
|
||||
`foreign import` can consume it.
|
||||
|
||||
| Path | Effort | Zero-copy on reads | Notes |
|
||||
|---|---|---|---|
|
||||
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
|
||||
| B: Pure-Odin reader (hand-rolled) | High | ✅ | Port a minimal reader; no C dependency |
|
||||
| C: OdinArrow reuse | Medium-High | ✅ | OdinArrow already has hand-rolled FlatBuffers encoder/decoder |
|
||||
| D: TS/JS interop (WASM-only) | Low-N/A | Partial | Browser builds can read FlatBuffers via JS lib instead |
|
||||
|
||||
### Path A — FFI to FlatCC (recommended starting point)
|
||||
|
||||
**Mechanics.** FlatCC generates per-schema C headers. You:
|
||||
|
||||
1. Install/build flatcc (it's a small C project, `flatbuffer`-compatible).
|
||||
2. Generate C reader+builder headers from your `.fbs`:
|
||||
```bash
|
||||
flatcc --common -a schema/catalog.fbs -o gui/src/generated
|
||||
```
|
||||
yields `catalog_reader.h`, `catalog_builder.h`, `catalog_verifier.h`, plus a
|
||||
`flatccrt.h`/`libflatccrt.a` runtime.
|
||||
3. Write (or generate with `odin-c-bindgen` /
|
||||
`Breush/odin-binding-generator`) Odin foreign bindings for the handful of
|
||||
functions your GUI actually uses.
|
||||
|
||||
Illustrative Odin binding shape (rough, not final API):
|
||||
|
||||
```odin
|
||||
package catalog_fb
|
||||
|
||||
foreign import flatcc "libflatccrt.a"
|
||||
|
||||
@(default_calling_convention = "c")
|
||||
foreign flatcc {
|
||||
// table accessors generated by flatcc look like:
|
||||
flatbuffers_verify_buffer :: proc(buf: rawptr, size: uint, id: ^byte) -> c.int ---
|
||||
catalog_Catalog_object_count :: proc(t: ^catalog_Catalog_table) -> u64 ---
|
||||
catalog_CatalogObject_ra :: proc(t: ^catalog_CatalogObject_table) -> f64 ---
|
||||
// etc.
|
||||
}
|
||||
```
|
||||
|
||||
Zero-copy: flatcc's generated **reader** macros operate directly on the buffer —
|
||||
`ra()` is a macro expanding to a bounds-checked buffer read. That maps naturally
|
||||
to Odin's `#foreign` + cstring/`^f64` access.
|
||||
|
||||
**Downsides:**
|
||||
- Odin bindings must **track schema regeneration**. Every time you add a field
|
||||
to the `.fbs`, the C headers change and the Odin foreign decls (or the
|
||||
generated bindings) must be refreshed.
|
||||
- flatcc's API is oriented to C macros; binding it faithfully through Odin
|
||||
`foreign` is doable but fiddly (macros don't transfer — you translate each
|
||||
macro into the equivalent C function or hand-roll the offset arithmetic).
|
||||
- You now have a C runtime (static lib) in the GUI build. For the **WASM
|
||||
target** this must compile under Emscripten — flatcc is plain C and does build
|
||||
for WASM, but adds to WASM binary size and toolchain coupling.
|
||||
|
||||
**When to choose:** when you want a *proven* library, don't mind C in your
|
||||
Odin build, and want to avoid writing and maintaining a reader yourself.
|
||||
|
||||
### Path B — Pure-Odin reader (hand-rolled, most effort, most control)
|
||||
|
||||
FlatBuffers read access is genuinely simple — as the ODINARROW project proves.
|
||||
OdinArrow ships a "hand-rolled FlatBuffers encoder/decoder" for the Arrow IPC
|
||||
header format. A minimal FlatBuffers table reader in Odin is ~100-300 lines:
|
||||
follow `u32` offsets, deref vtables, read little-endian scalars.
|
||||
|
||||
What you'd implement:
|
||||
- `ReadRoot(root: ^u8, size: uint) -> root_offset` (uoffset at byte 0; skip
|
||||
4-byte length prefix if size-prefixed).
|
||||
- Vtable lookup: given a table addr, read `uoffset` back to vtable, scan slots
|
||||
for a field id, read field offset (or treat as absent → default).
|
||||
- Vectors: u32 length + element stride; for `f64`/`f32`/structs this is a direct
|
||||
`^f64` slice after bounds check.
|
||||
- Verification: walk offsets checking bounds/alignment (or skip for trusted data
|
||||
— but you're on a **network stream**; verify, at least bounds, before use).
|
||||
|
||||
**Downsides:**
|
||||
- You own correctness, update discipline, and testing. Every FlatBuffers format
|
||||
nuance (file identifiers, size prefixes, unions) must be re-implemented.
|
||||
- Risk of subtle divergence from the C++/Rust/Java implementations
|
||||
(endianness, alignment, default-value semantics).
|
||||
- Cross-language conformance tests (see `08-testing-strategies.md`) absolutely
|
||||
required — you're re-implementing a spec.
|
||||
|
||||
**When to choose:** when you want zero C in the Odin build, plan long-term
|
||||
maintenance, and value full control (and you can lean on OdinArrow's already
|
||||
proven patterns).
|
||||
|
||||
### Path C — OdinArrow reuse (hybrid)
|
||||
|
||||
- https://github.com/TimeLord/OdinArrow — a mature-ish Odin implementation of
|
||||
Apache Arrow's IPC format, **including** a hand-rolled FlatBuffers
|
||||
encoder/decoder (Arrow IPC metadata is FlatBuffers).
|
||||
- You could extract/adapt OdinArrow's FlatBuffers decode machinery for your own
|
||||
schema, or (bolder) adopt Arrow IPC entirely for the data path (Arrow IPC
|
||||
*is* FlatBuffers-framed + columnar buffers — arguably an excellent fit for
|
||||
streaming galaxy positions/redshifts).
|
||||
- OdinArrow is a small, MIT-style community project (TimeLord). Verify license
|
||||
and maintenance before depending on it.
|
||||
- If you go pure Arrow IPC, you get batch semantics for free (schema message →
|
||||
record batch messages) — same framing pattern as FlatBuffers with the
|
||||
columnar layout built in.
|
||||
- Arrow IPC stream = length-prefixed (u32 LE) messages, with a
|
||||
`continuation marker 0xFFFFFFFF` for 4-byte alignment. This is a well-specified
|
||||
framing you can reuse *without* adopting Arrow's data model.
|
||||
|
||||
**When to choose:** when Arrow-style columnar data is actually what you want
|
||||
(millions of numeric rows — it is a great fit), and you're okay depending on /
|
||||
contributing to OdinArrow.
|
||||
|
||||
### Path D — WASM/JS interop (browser build only)
|
||||
|
||||
- The Web GUI (`gui/www/`) builds Odin to WASM and runs alongside JS.
|
||||
- FlatBuffers has **official JS/TS support** (`flatbuffers` npm package). For
|
||||
the web build you could do the parsing/decoding in JS (or TypeScript) and hand
|
||||
plain arrays (`Float64Array`) to the Odin WASM side — losing zero-copy at the
|
||||
WASM boundary but gaining ecosystem-tested parsing.
|
||||
- Practical hybrid: **WASM build path**: keep the WebSocket in JS, decode
|
||||
FlatBuffers in JS (official lib), then transfer typed arrays into WASM memory
|
||||
(single `Emscripten.HEAPF64.set(...)` copy). Zero-copy is not preserved across
|
||||
the WASM boundary, but the *raw-bytes → arrays* decode in JS is still far
|
||||
cheaper than JSON and uses a battle-tested library.
|
||||
- Native desktop build (`odin build` + raylib, the primary target): need one of
|
||||
A/B/C. The WEB GUI is secondary.
|
||||
|
||||
**Recommendation for this repo's roadmap:** Start with **Path B or A** for the
|
||||
native Odin build (the primary `make run` target), and use **Path D** for the
|
||||
WASM build if/when it becomes a shipping concern. The conformance-test suite
|
||||
(shared static `.mon`/`.bin` fixture files read by both Rust and Odin) is the
|
||||
safety net that makes the hand-rolled Path B safe.
|
||||
|
||||
## Networking from Odin
|
||||
|
||||
There is **no networking code in the GUI today**. Options for receiving
|
||||
FlatBuffers from the Rust API:
|
||||
|
||||
| Option | Fit | Notes |
|
||||
|---|---|---|
|
||||
| `core:net` (Odin stdlib) | Native desktop | Built-in `core:net` module has socket APIs; HTTP is manual or minimal — fine for `GET` of a binary body; WebSocket requires hand-rolling the upgrade + frame handling (doable, ~200 lines) |
|
||||
| Curl FFI (`libcurl`) | Native desktop | Battle-tested HTTP, easy `buffer` callback for `application/octet-stream`; Odin `#foreign` to curl is well-trodden (e.g., furbs). Adds libcurl dep to native build |
|
||||
| Emscripten `fetch` bridge | WASM | In the browser build, JS owns the network; call `fetch` from JS or via Odin's Emscripten bindings, then `HEAP`-copy |
|
||||
| WebSocket via JS | WASM | Same as above for the browser |
|
||||
| Community libs | Both | e.g. various `core:net`-based or `thirdparty` HTTP clients; vet for maturity |
|
||||
|
||||
`gui/src/data.odin` already has the intended procedure signatures:
|
||||
|
||||
```odin
|
||||
get_catalogs :: proc(url: string) -> ([dynamic]Catalog, ^APIError)
|
||||
get_catalog_objects :: proc(url: string, catalog_name: string) -> ([dynamic]CatalogObject, ^APIError)
|
||||
```
|
||||
|
||||
When the transport is in place, these become the seam between the network layer
|
||||
and the FlatBuffers decode layer: fetch bytes → verify → read fields → return
|
||||
typed Odin slices.
|
||||
|
||||
## ZSS / zero-copy in the render loop
|
||||
|
||||
The rendering win only materializes if data stays zero-copy **into the frame
|
||||
loop**:
|
||||
|
||||
1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime
|
||||
of the frame).
|
||||
2. `verify` the buffer once.
|
||||
3. Get `ra_slice := ObjectBatch.ra(&buf)` → `[]f64` view.
|
||||
4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight
|
||||
from that slice; build `rl.Vector3`; `DrawPoint3D`.
|
||||
|
||||
No per-object allocation. The current `Galaxy { position, color }` dynamic array
|
||||
in `main.odin` is the data structure you'd replace with *slices into the
|
||||
FlatBuffer*.
|
||||
|
||||
## WASM memory-model caveats
|
||||
|
||||
- If JS decodes FlatBuffers and hands typed arrays to Odin/WASM, the copy into
|
||||
WASM linear memory is one `HEAPF64.set()` — a single memcpy, not per-field.
|
||||
- If Odin/WASM itself decodes FlatBuffers over its own `core:net` binding, it
|
||||
reads directly from WASM heap — same as native, but you're now maintaining the
|
||||
hand-rolled decoder in WASM too (subject to WASM's 32-bit indexing, still fine
|
||||
for sub-2GiB buffers).
|
||||
- Emscripten `-sALLOW_MEMORY_GROWTH` and 4GB heap settings matter if you plan
|
||||
to hold multi-GB catalogs; keep to batched frames (e.g. ≤ 64 MB) and free per
|
||||
chunk.
|
||||
|
||||
## Build integration in this repo
|
||||
|
||||
The `gui/Makefile` owns the Odin targets. Adding FlatBuffers means:
|
||||
- A `-collection:lib=lib/local` (or a vendored submodule under `gui/lib/`) for
|
||||
either the `flatcc` runtime lib or the hand-rolled Odin package.
|
||||
- A `schema-gen` source of truth: run flatc for Rust + C (Path A) or otherwise
|
||||
regenerate, as a make target (see README root / `03-rust-integration.md`).
|
||||
- CI (`odin test` in the Gitea workflow) must pick up the new collection and any
|
||||
generated files. Commit generated files to avoid CI toolchain surprises.
|
||||
|
||||
## References
|
||||
|
||||
- Odin FFI docs: https://odin-lang.org/docs/ffi/ (foreign blocks, calling conventions)
|
||||
- Odin binding to C (official news post): https://odin-lang.org/news/binding-to-c/
|
||||
- FlatCC: https://github.com/dvidelabs/flatcc (and `flatbuffers.dev/languages/c/`)
|
||||
- OdinArrow: https://github.com/TimeLord/OdinArrow (owned FlatBuffers + Arrow IPC)
|
||||
- Odin binding generator (community): https://github.com/karl-zylinski/odin-c-bindgen
|
||||
- msgpack-odin (reference for a hand-rolled binary codec in Odin, small): https://github.com/tgolsson/msgpack-odin
|
||||
Reference in New Issue
Block a user