desi_explorer/resources/ai/research/data-streaming/04-odin-client-integration.md

04 — Odin / GUI Integration

This is the highest-risk area of the whole research effort. FlatBuffers has no first-party Odin binding. You must choose one of several integration paths. This doc lays out the realistic options with tradeoffs, plus the networking story for the GUI.

The core problem

Odin can talk to C via foreign blocks (the C ABI is first-class). FlatBuffers has two C-related access points:

1. Google's official flatc generates C++ and C (C#/Java/etc.) — but its C support is a separate project, FlatCC.
2. FlatCC (dvidelabs/flatcc) is an independent FlatBuffers compiler + runtime for pure C. It generates _reader.h/_builder.h headers per schema plus a small libflatccrt.a runtime. Works via the C ABI, so Odin's foreign import can consume it.

Path	Effort	Zero-copy on reads	Notes
A: FFI to FlatCC (C runtime)	Medium	✅	Bind generated C headers to Odin foreign
B: Pure-Odin reader (hand-rolled)	High	✅	Port a minimal reader; no C dependency
C: OdinArrow reuse	Medium-High	✅	OdinArrow already has hand-rolled FlatBuffers encoder/decoder
D: TS/JS interop (WASM-only)	Low-N/A	Partial	Browser builds can read FlatBuffers via JS lib instead

Path A — FFI to FlatCC (recommended starting point)

Mechanics. FlatCC generates per-schema C headers. You:

1. Install/build flatcc (it's a small C project, flatbuffer-compatible).
2. Generate C reader+builder headers from your .fbs:

   flatcc --common -a schema/catalog.fbs -o gui/src/generated
   

   yields catalog_reader.h, catalog_builder.h, catalog_verifier.h, plus a flatccrt.h/libflatccrt.a runtime.
3. Write (or generate with odin-c-bindgen / Breush/odin-binding-generator) Odin foreign bindings for the handful of functions your GUI actually uses.

Illustrative Odin binding shape (rough, not final API):

package catalog_fb

foreign import flatcc "libflatccrt.a"

@(default_calling_convention = "c")
foreign flatcc {
    // table accessors generated by flatcc look like:
    flatbuffers_verify_buffer       :: proc(buf: rawptr, size: uint, id: ^byte) -> c.int ---
    catalog_Catalog_object_count    :: proc(t: ^catalog_Catalog_table) -> u64 ---
    catalog_CatalogObject_ra        :: proc(t: ^catalog_CatalogObject_table) -> f64 ---
    // etc.
}

Zero-copy: flatcc's generated reader macros operate directly on the buffer — ra() is a macro expanding to a bounds-checked buffer read. That maps naturally to Odin's #foreign + cstring/^f64 access.

Downsides:

• Odin bindings must track schema regeneration. Every time you add a field to the .fbs, the C headers change and the Odin foreign decls (or the generated bindings) must be refreshed.
• flatcc's API is oriented to C macros; binding it faithfully through Odin foreign is doable but fiddly (macros don't transfer — you translate each macro into the equivalent C function or hand-roll the offset arithmetic).
• You now have a C runtime (static lib) in the GUI build. For the WASM target this must compile under Emscripten — flatcc is plain C and does build for WASM, but adds to WASM binary size and toolchain coupling.

When to choose: when you want a proven library, don't mind C in your Odin build, and want to avoid writing and maintaining a reader yourself.

Path B — Pure-Odin reader (hand-rolled, most effort, most control)

FlatBuffers read access is genuinely simple — as the ODINARROW project proves. OdinArrow ships a "hand-rolled FlatBuffers encoder/decoder" for the Arrow IPC header format. A minimal FlatBuffers table reader in Odin is ~100-300 lines: follow u32 offsets, deref vtables, read little-endian scalars.

What you'd implement:

• ReadRoot(root: ^u8, size: uint) -> root_offset (uoffset at byte 0; skip 4-byte length prefix if size-prefixed).
• Vtable lookup: given a table addr, read uoffset back to vtable, scan slots for a field id, read field offset (or treat as absent → default).
• Vectors: u32 length + element stride; for f64/f32/structs this is a direct ^f64 slice after bounds check.
• Verification: walk offsets checking bounds/alignment (or skip for trusted data — but you're on a network stream; verify, at least bounds, before use).

Downsides:

• You own correctness, update discipline, and testing. Every FlatBuffers format nuance (file identifiers, size prefixes, unions) must be re-implemented.
• Risk of subtle divergence from the C++/Rust/Java implementations (endianness, alignment, default-value semantics).
• Cross-language conformance tests (see 08-testing-strategies.md) absolutely required — you're re-implementing a spec.

When to choose: when you want zero C in the Odin build, plan long-term maintenance, and value full control (and you can lean on OdinArrow's already proven patterns).

Path C — OdinArrow reuse (hybrid)

• https://github.com/TimeLord/OdinArrow — a mature-ish Odin implementation of Apache Arrow's IPC format, including a hand-rolled FlatBuffers encoder/decoder (Arrow IPC metadata is FlatBuffers).
• You could extract/adapt OdinArrow's FlatBuffers decode machinery for your own schema, or (bolder) adopt Arrow IPC entirely for the data path (Arrow IPC is FlatBuffers-framed + columnar buffers — arguably an excellent fit for streaming galaxy positions/redshifts).
• OdinArrow is a small, MIT-style community project (TimeLord). Verify license and maintenance before depending on it.
• If you go pure Arrow IPC, you get batch semantics for free (schema message → record batch messages) — same framing pattern as FlatBuffers with the columnar layout built in.
• Arrow IPC stream = length-prefixed (u32 LE) messages, with a continuation marker 0xFFFFFFFF for 4-byte alignment. This is a well-specified framing you can reuse without adopting Arrow's data model.

When to choose: when Arrow-style columnar data is actually what you want (millions of numeric rows — it is a great fit), and you're okay depending on / contributing to OdinArrow.

Path D — WASM/JS interop (browser build only)

• The Web GUI (gui/www/) builds Odin to WASM and runs alongside JS.
• FlatBuffers has official JS/TS support (flatbuffers npm package). For the web build you could do the parsing/decoding in JS (or TypeScript) and hand plain arrays (Float64Array) to the Odin WASM side — losing zero-copy at the WASM boundary but gaining ecosystem-tested parsing.
• Practical hybrid: WASM build path: keep the WebSocket in JS, decode FlatBuffers in JS (official lib), then transfer typed arrays into WASM memory (single Emscripten.HEAPF64.set(...) copy). Zero-copy is not preserved across the WASM boundary, but the raw-bytes → arrays decode in JS is still far cheaper than JSON and uses a battle-tested library.
• Native desktop build (odin build + raylib, the primary target): need one of A/B/C. The WEB GUI is secondary.

Recommendation for this repo's roadmap: Start with Path B or A for the native Odin build (the primary make run target), and use Path D for the WASM build if/when it becomes a shipping concern. The conformance-test suite (shared static .mon/.bin fixture files read by both Rust and Odin) is the safety net that makes the hand-rolled Path B safe.

Networking from Odin

There is no networking code in the GUI today. Options for receiving FlatBuffers from the Rust API:

Option	Fit	Notes
core:net (Odin stdlib)	Native desktop	Built-in core:net module has socket APIs; HTTP is manual or minimal — fine for GET of a binary body; WebSocket requires hand-rolling the upgrade + frame handling (doable, ~200 lines)
Curl FFI (libcurl)	Native desktop	Battle-tested HTTP, easy buffer callback for application/octet-stream; Odin #foreign to curl is well-trodden (e.g., furbs). Adds libcurl dep to native build
Emscripten fetch bridge	WASM	In the browser build, JS owns the network; call fetch from JS or via Odin's Emscripten bindings, then HEAP-copy
WebSocket via JS	WASM	Same as above for the browser
Community libs	Both	e.g. various core:net-based or thirdparty HTTP clients; vet for maturity

gui/src/data.odin already has the intended procedure signatures:

get_catalogs :: proc(url: string) -> ([dynamic]Catalog, ^APIError)
get_catalog_objects :: proc(url: string, catalog_name: string) -> ([dynamic]CatalogObject, ^APIError)

When the transport is in place, these become the seam between the network layer and the FlatBuffers decode layer: fetch bytes → verify → read fields → return typed Odin slices.

ZSS / zero-copy in the render loop

The rendering win only materializes if data stays zero-copy into the frame loop:

1. Fetch frame bytes → owned [dynamic]u8 (or a slice pinned for the lifetime of the frame).
2. verify the buffer once.
3. Get ra_slice := ObjectBatch.ra(&buf) → []f64 view.
4. Per object in update()/draw(): read ra[i], dec[i], z[i] straight from that slice; build rl.Vector3; DrawPoint3D.

No per-object allocation. The current Galaxy { position, color } dynamic array in main.odin is the data structure you'd replace with slices into the FlatBuffer.

WASM memory-model caveats

• If JS decodes FlatBuffers and hands typed arrays to Odin/WASM, the copy into WASM linear memory is one HEAPF64.set() — a single memcpy, not per-field.
• If Odin/WASM itself decodes FlatBuffers over its own core:net binding, it reads directly from WASM heap — same as native, but you're now maintaining the hand-rolled decoder in WASM too (subject to WASM's 32-bit indexing, still fine for sub-2GiB buffers).
• Emscripten -sALLOW_MEMORY_GROWTH and 4GB heap settings matter if you plan to hold multi-GB catalogs; keep to batched frames (e.g. ≤ 64 MB) and free per chunk.

Build integration in this repo

The gui/Makefile owns the Odin targets. Adding FlatBuffers means:

• A -collection:lib=lib/local (or a vendored submodule under gui/lib/) for either the flatcc runtime lib or the hand-rolled Odin package.
• A schema-gen source of truth: run flatc for Rust + C (Path A) or otherwise regenerate, as a make target (see README root / 03-rust-integration.md).
• CI (odin test in the Gitea workflow) must pick up the new collection and any generated files. Commit generated files to avoid CI toolchain surprises.

References

• Odin FFI docs: https://odin-lang.org/docs/ffi/ (foreign blocks, calling conventions)
• Odin binding to C (official news post): https://odin-lang.org/news/binding-to-c/
• FlatCC: https://github.com/dvidelabs/flatcc (and flatbuffers.dev/languages/c/)
• OdinArrow: https://github.com/TimeLord/OdinArrow (owned FlatBuffers + Arrow IPC)
• Odin binding generator (community): https://github.com/karl-zylinski/odin-c-bindgen
• msgpack-odin (reference for a hand-rolled binary codec in Odin, small): https://github.com/tgolsson/msgpack-odin