desi_explorer/resources/ai/research/data-streaming/01-architecture.md

01 — Current Architecture and Where FlatBuffers Fits

What exists today

API (Rust + axum)

• Entrypoint: api/src/main.rs — binds API_BIND_ADDR (default 0.0.0.0:8080), wires tracing, serves the axum app with graceful shutdown.
• Routes (api/src/routes/):
  • GET /health → static {"status":"ok"}
  • GET /api/v1/catalogs → hardcoded list of 2 catalogs (edr, dr1)
  • GET /api/v1/objects?catalog=<name>&limit=<n> → stubbed, returns []
• Models (api/src/models.rs):

pub struct Catalog {
    pub name: String,
    pub release: String,
    pub description: &'static str,
    pub object_count: Option<u64>,
}

pub struct CatalogObject {
    pub id: String,
    pub catalog: String,
    pub object_type: String,   // galaxy / quasar / star
    pub ra: f64,               // degrees
    pub dec: f64,              // degrees
    pub redshift: f64,         // dimensionless
}

• Serialization: serde Serialize derives + axum::Json. serde_json is a dev-dependency only (used by integration tests).
• CORS: tower-http is compiled with cors + trace features, but no CORS layer is currently added to routes::app(). This matters for the WASM GUI.

GUI (Odin + raylib)

• Entrypoint: gui/src/main.odin — update() → draw() loop, orbital Camera3D, 4,000 procedurally generated points via make_universe().
• Placeholder data: Galaxy { position: rl.Vector3, color: rl.Color } generated in a 500-unit-radius sphere. Colors are a distance-based redshift stand-in.
• Data layer: gui/src/data.odin defines hand-written Odin structs that mirror the Rust API JSON contract (including pointer types ^string, ^u64 to mirror Rust Option), plus stubbed procedures:

get_catalogs :: proc(url: string) -> ([dynamic]Catalog, ^APIError)        // nil, nil
get_catalog :: proc(url: string, name: string) -> (^Catalog, ^APIError)   // nil, nil
get_catalog_objects :: proc(url: string, catalog_name: string)            // nil, nil

• No HTTP client exists on the Odin side — no core:net, no curl bindings, nothing wired into the render loop. The structs are a planned interface.
• Web build: gui/www/ — WASM shell (Emscripten) loading index.wasm + odin.js, no API URL wiring yet.

Data flow gap

[DESI catalog store]  --(future)-->  [Rust/axum API]  --(nothing today)-->  [Odin + raylib GUI]
                                        ^                                        ^
                                        | serde JSON models                    | hand-mirrored structs
                                        |                                      | (stubs, never used)

There is no live data flow. The API currently returns JSON placeholders; the GUI renders procedural points. The intended shape (per code comments):

• The /api/v1/objects endpoint "will page through the centralized DESI catalog store."
• Real survey data replaces the procedural cloud "once ingestion lands."

Where FlatBuffers fits

There are three natural insertion points:

1. Wire format for the object-stream endpoint (primary fit)

Replace the JSON array response of GET /api/v1/objects (or add an application/flatbuffer variant / new /api/v1/stream endpoint) with a framed stream of FlatBuffer messages:

• Rust side: build FlatBuffers with FlatBufferBuilder per batch (e.g. 10k-50k objects per batch), send as application/octet-stream.
• Odin side: receive the byte blob, verify once, and access ra/dec/ redshift directly from the network buffer — no per-field copy. This is the killer use case for the render loop, where every point is projected and colored per frame.

2. Catalog metadata (secondary fit)

GET /api/v1/catalogs returns tiny JSON today. The FlatBuffers win here is marginal (5-10 records), but using the same serialization everywhere keeps the codebase uniform — one schema, one reader path. Recommend serving both a JSON response (for curl/debugging) and a FlatBuffers variant (for the client).

3. Bidirectional client → server channel (future fit)

If the GUI ever sends selection/filter requests (pan region, redshift range, object type filters), a WebSocket carrying FlatBuffers both ways gives consistent framing. The get_catalog_objects() stub signature suggests the GUI is expected to query on demand as the camera moves — a WebSocket request/ response protocol with a cheap binary payload would fit this well.

Integration points (concrete anchors)

Layer	Today	FlatBuffers insertion point
Rust models	Catalog, CatalogObject (serde)	Generated catalog_generated.rs alongside or replacing
Rust routes	routes/catalogs.rs	New route (e.g. /ws/catalogs) or response-format negotiation
Rust Cargo	serde, axum::Json	flatbuffers crate + flatbuffers-build for .fbs → .rs
Odin structs	data.odin mirrors JSON	Generated/FFI FlatBuffers reader; or hand-rolled reader
Odin HTTP	none	core:net HTTP, WebSocket, or curl FFI
Build	make build, make test per project	Schema shared & compiled by both sides

High-level target architecture

        catalog.fbs (single source of truth, checked into repo)
              |
     +--------+---------+
     |                  |
flatc --rust       flatcc --c (or hand-rolled Odin reader)
     |                  |
api/ (Rust)        gui/ (Odin + raylib)
     |                  ^
     |   HTTP / WebSocket (framed FlatBuffer binary stream)
     +------------------+

• One schema file. Two generators. Byte-for-byte identical wire format.
• A shared conformance test suite (see 08-testing-strategies.md) ensures both sides stay in sync.

Key architectural decisions to make later

1. Transport: HTTP chunked/range requests vs. WebSocket vs. both (see 05-streaming-protocols.md).
2. Framing: size-prefixed FlatBuffers (size_prefixed_root_*) vs. a custom length prefix (e.g. flatstream's 4-byte LE length). Needed because a TCP/ WebSocket stream is byte-ambiguous — you must know where one message ends and the next begins.
3. Batching: per-object tables vs. packed tables of arrays (columnar-style). For millions of DESI objects, a table { ra: [double]; dec: [double]; ... } layout is dramatically smaller and faster than 1 table per object.
4. Schema ownership: options include a top-level schema/ directory shared by both api/ and gui/, checked-in generated code, or generated at build time. Checked-in generated code is the simplest CI-safe option for both cargo build and odin build isolation.

What to read next

• 02-flatbuffers-overview.md — the format's internals and guarantees
• 03-rust-integration.md — how the Rust side consumes .fbs
• 04-odin-client-integration.md — how the Odin side consumes the same format
• 05-streaming-protocols.md — transport and framing choices
• 06-schema-design.md — proposed DESI schema