desi_explorer/resources/ai/research/data-streaming/06-schema-design.md

06 — Schema Design for DESI Data

This doc sketches a FlatBuffers schema for DESI Explorer and the rationale behind each decision. It's a proposal — validate against real DESI catalog columns before finalizing.

Design goals

1. Compact bulk numeric streaming — the dominant payload is millions of (ra, dec, redshift) triples. Use columnar vectors of scalars, not one table per object. This is what makes the performance win real (see 10-performance-benchmarks.md) — vectors of scalars are contiguous and can be safe_slice'd into &[f64] views.
2. Schema evolution for future DESI releases (edr → dr1 → dr2, added columns like petro_mag, spectroscopic_class, coverage tiles).
3. Self-describing streams via file identifiers and a message union.
4. Keep the existing API contract (Catalog, CatalogObject from api/src/models.rs) as the JSON-facing mirror, so curl/debug paths stay.

Proposed schema

// schema/catalog.fbs
namespace desi;

file_identifier "DESI";

// ---------- Catalog metadata ----------

table Catalog {
  name: string;                       // "edr", "dr1", ...
  release: string;                    // "Early Data Release"
  description: string;
  object_count: uint64;               // Some(0) problem → see note below
}

// ---------- Bulk object data (columnar) ----------

// A single point, for small results / per-row paths.
struct Point {
  ra: double;                         // ICRS, degrees, [0, 360)
  dec: double;                        // ICRS, degrees, [-90, 90]
  redshift: double;                   // dimensionless
  object_type: byte;                  // 0=galaxy 1=quasar 2=star (enum → byte)
}

table ObjectBatch {
  catalog: string;                    // catalog this batch belongs to
  batch_seq: uint32;                  // for reassembling a paged stream in order
  n: uint64;                          // number of objects in this batch

  // Columnar arrays — zero-copy friendly, contiguous per column.
  ids: [uint64];                      // object ids (or bytes for survey ids)
  object_types: [byte];               // enum per object (galaxy/quasar/star)
  ra: [double];                       // degrees
  dec: [double];                      // degrees
  redshift: [double];                 // dimensionless
  // future columns (release-2): petro_mag_g/r/z (optional vectors),
  //                              is_spectroscopic: [bool], ...
}

// ---------- Server → client envelope ----------

table APIError {
  code: int32;
  message: string;
}

table CatalogList {
  catalogs: [Catalog];
}

union Message {
  CatalogList,
  Catalog,
  ObjectBatch,
  APIError,
}

table ServerMessage {
  msg: Message;
  catalog_list: CatalogList;
  catalog: Catalog;
  object_batch: ObjectBatch;
  error: APIError;
}

root_type ServerMessage;

Rationale per decision

Columnar ObjectBatch vs. per-object CatalogObject tables

	Per-object tables	Columnar arrays
Size	~vtable (≥4-16B) + per-row overhead per object	~8 bytes per f64, zero per-row metadata
Speed	vtable deref per field per object	Linear memcpy / slice iter
Odin reads	method call per field per object	[]f64 direct slice
Evolution	per-field evolveable	vector types can't structurally evolve (add new vectors instead)
Fit for render loop	ok for thousands	ideal for 100k+

3-5 columns of raw doubles per row dominate the payload. Columnar wins by a lot. Rust builds with create_vector(&flat_array_of_f64) — trivial. Odin reads with ObjectBatch_ra(obj) → []f64.

Caveat: in the columnar layout, an individual object is identified by index across columns (ra[i], dec[i], redshift[i]). That's fine for bulk rendering, and ids[i] links back to a catalog row when needed.

file_identifier "DESI"

• Enables flatbuffers::size_prefixed_root_with_opts::<ServerMessage>(opts, buf) and (on the Odin side) cheap "is this our message?" pre-checks.
• Without a file identifier, verification accepts buffers that are structurally valid but a different schema — dangerous across stream kinds.

object_count: uint64 — the Option<u64> presence trap

Current Rust model has object_count: Option<u64>. In FlatBuffers:

• A uint64 with default: 0 is not written when 0 → reader can't tell "zero" from "absent". If Some(0) is meaningful, use an optional scalar: object_count: uint64 (nullable)? No — the FBS way is:

table Catalog {
  object_count: uint64 (id: 0, default: null);  // optional scalar — presence preserved
}

Optional scalars (default: null) are always written when set, and the accessor returns the language's optional type. Rust: object_count: Option<u64>; Odin (hand-rolled): an is_set flag.

Alternatively wrap in a struct or keep as raw uint64 if you never ship a meaningful zero.

enum for object_type

enum ObjectType : byte {
  Galaxy = 0,
  Quasar = 1,
  Star = 2,
}

• Table fields can be enums; the accessor returns typed enum.
• Keep it a byte/ubyte — flat buffering stores the underlying integer; a vector of enums (object_types: [ObjectType]) is just a &[u8]/[]byte view on the wire (dense — nice for thousands of classifications).

Struct Point

• Only correct if a coordinate + type is structurally fixed forever (a struct cannot gain fields). Risky: DESI might add errors/ellipsoid fields later. For the bulk path prefer the columnar batch (add a new vector). Keep Point struct only if you need tiny random-access records that will never change — otherwise a table is safer for any "object record" shape.

Batch sequence number

• batch_seq: uint32 lets the client detect drops/reordering/restarts when paging/straming through a catalog. Server increments per (catalog, region) session. Cheap insurance on an unordered transport.

Union envelop ServerMessage

• Lets one WebSocket carry catalog lists, object batches, errors, and future message kinds, with the type tag built into the buffer (no guessing by file identifier).
• Rust: generated msg_type() + msg_as_object_batch() etc.
• Odin: hand-rolled union-tag read + payload access.

If you'd rather keep streams homogeneous, skip the union and use separate root types per endpoint (see 05-streaming-protocols.md Pattern B).

Evolution roadmap (defensive)

• v1 (this schema): as above.
• v2 (next release): append petro_mag_g/z/r: [float] (or per-band uint16 scaled), spectral_class: [byte], is_spec_selected: [bool] to ObjectBatch (all at the end of the table — v1 decoders ignore them, v2 decoders get defaults/absent for v1 data).
• v3: Catalog gains coverage_footprint: string (WKT) at the end.
• Do not reorder, remove, or deprecate anything unless the schema owner signs off; run flatc --conform in CI.

Compatibility with existing Rust/JSON models

Keep the FlatBuffers schema in sync with api/src/models.rs (serde) and gui/src/data.odin (mirrors) during the transition:

• Add a From<CatalogObject> for ObjectBatch-ish conversion inside the API, or (better) have the API build FlatBuffers directly from the DESI row fetch, bypassing serde for the binary path.
• The JSON endpoints can stay for debugging/curl and for the initial catalog list (tiny). Document in api/ that /api/v1/objects will switch to application/flatbuffer (or gain a /json variant) once the client is wired.

Where the schema lives & codegen ownership

Proposal: a top-level schema/catalog.fbs is the single source of truth.

schema/
  catalog.fbs            <- cross-language contract

• Rust: flatc → api/src/generated/catalog_generated.rs
• Odin (Path A): flatcc → gui/src/generated/catalog_reader.h (+ bindings)
• Odin (Path B): hand-rolled reader must be tested against fixtures built from this schema (see 08-testing-strategies.md).

Add a make target:

schema-gen:
	flatc --rust     -o api/src/generated ../schema/catalog.fbs
	flatcc --c --common -o gui/src/generated ../schema/catalog.fbs

and (optionally) a CI check that committed generated files match a fresh regeneration (git diff --exit-code after regen).

Open questions to resolve with real DESI data

1. DESI object IDs: u64 internal, or string (survey-id like "DESI-1234")? If string, ids: [string] (each a mini-table) costs far more than [uint64] — decide based on the actual catalog schema.
2. Coordinate range conventions: ra in [0,360) vs [-180,180)? Pick once, document, optionally store a normalized flag in the schema.
3. Redshift scale/precision: double vs float? For rendering, f32 halves the payload and is plenty; for science fidelity, keep f64 as in the model today.
4. Does the renderer need per-object colors today? If yes, precompute a packed [uint32] RGBA vector in the API — better than the client computing per frame.
5. In-stream metadata: do you need a Metadata message (region bounds, epoch, projection) before the first batch? Add now if so — retrofitting a union arm mid-release is possible but adds churn.