# 06 — Schema Design for DESI Data

This doc sketches a FlatBuffers schema for DESI Explorer and the rationale
behind each decision. It's a *proposal* — validate against real DESI catalog
columns before finalizing.

## Design goals

1. **Compact bulk numeric streaming** — the dominant payload is millions of
   (ra, dec, redshift) triples. Use **columnar vectors of scalars**, not one
   table per object. This is what makes the performance win real (see
   `10-performance-benchmarks.md`) — vectors of scalars are contiguous and can be
   `safe_slice`'d into `&[f64]` views.
2. **Schema evolution** for future DESI releases (edr → dr1 → dr2, added
   columns like `petro_mag`, `spectroscopic_class`, coverage tiles).
3. **Self-describing streams** via file identifiers and a message union.
4. **Keep the existing API contract** (`Catalog`, `CatalogObject` from
   `api/src/models.rs`) as the JSON-facing mirror, so `curl`/debug paths stay.

## Proposed schema

```fbs
// schema/catalog.fbs
namespace desi;

file_identifier "DESI";

// ---------- Catalog metadata ----------

table Catalog {
  name: string;                       // "edr", "dr1", ...
  release: string;                    // "Early Data Release"
  description: string;
  object_count: uint64;               // Some(0) problem → see note below
}

// ---------- Bulk object data (columnar) ----------

// A single point, for small results / per-row paths.
struct Point {
  ra: double;                         // ICRS, degrees, [0, 360)
  dec: double;                        // ICRS, degrees, [-90, 90]
  redshift: double;                   // dimensionless
  object_type: byte;                  // 0=galaxy 1=quasar 2=star (enum → byte)
}

table ObjectBatch {
  catalog: string;                    // catalog this batch belongs to
  batch_seq: uint32;                  // for reassembling a paged stream in order
  n: uint64;                          // number of objects in this batch

  // Columnar arrays — zero-copy friendly, contiguous per column.
  ids: [uint64];                      // object ids (or bytes for survey ids)
  object_types: [byte];               // enum per object (galaxy/quasar/star)
  ra: [double];                       // degrees
  dec: [double];                      // degrees
  redshift: [double];                 // dimensionless
  // future columns (release-2): petro_mag_g/r/z (optional vectors),
  //                              is_spectroscopic: [bool], ...
}

// ---------- Server → client envelope ----------

table APIError {
  code: int32;
  message: string;
}

table CatalogList {
  catalogs: [Catalog];
}

union Message {
  CatalogList,
  Catalog,
  ObjectBatch,
  APIError,
}

table ServerMessage {
  msg: Message;
  catalog_list: CatalogList;
  catalog: Catalog;
  object_batch: ObjectBatch;
  error: APIError;
}

root_type ServerMessage;
```

## Rationale per decision

### Columnar `ObjectBatch` vs. per-object `CatalogObject` tables

| | Per-object tables | Columnar arrays |
|---|---|---|
| Size | ~vtable (≥4-16B) + per-row overhead per object | ~8 bytes per f64, zero per-row metadata |
| Speed | vtable deref per field per object | Linear memcpy / slice iter |
| Odin reads | method call per field per object | `[]f64` direct slice |
| Evolution | per-field evolveable | vector types can't structurally evolve (add new vectors instead) |
| Fit for render loop | ok for thousands | **ideal for 100k+** |

3-5 columns of raw doubles per row dominate the payload. Columnar wins by a lot.
Rust builds with `create_vector(&flat_array_of_f64)` — trivial. Odin reads with
`ObjectBatch_ra(obj)` → `[]f64`.

Caveat: in the columnar layout, an individual object is identified by *index*
across columns (`ra[i]`, `dec[i]`, `redshift[i]`). That's fine for bulk
rendering, and `ids[i]` links back to a catalog row when needed.

### `file_identifier "DESI"`

- Enables `flatbuffers::size_prefixed_root_with_opts::<ServerMessage>(opts, buf)`
  and (on the Odin side) cheap "is this our message?" pre-checks.
- Without a file identifier, verification accepts buffers that are structurally
  valid but a *different* schema — dangerous across stream kinds.

### `object_count: uint64` — the `Option<u64>` presence trap

Current Rust model has `object_count: Option<u64>`. In FlatBuffers:

- A `uint64` with `default: 0` **is not written when 0** → reader can't tell
  "zero" from "absent". If `Some(0)` is meaningful, use an **optional scalar**:
  `object_count: uint64 (nullable)`? No — the FBS way is:

```fbs
table Catalog {
  object_count: uint64 (id: 0, default: null);  // optional scalar — presence preserved
}
```

Optional scalars (`default: null`) are always written when set, and the accessor
returns the language's optional type. Rust: `object_count: Option<u64>`;
Odin (hand-rolled): an `is_set` flag.

Alternatively wrap in a struct or keep as raw `uint64` if you never ship a
meaningful zero.

### `enum` for object_type

```fbs
enum ObjectType : byte {
  Galaxy = 0,
  Quasar = 1,
  Star = 2,
}
```

- Table fields can be enums; the accessor returns typed enum.
- Keep it a `byte`/`ubyte` — flat buffering stores the underlying integer; a
  vector of enums (`object_types: [ObjectType]`) is just a `&[u8]`/`[]byte` view
  on the wire (dense — nice for thousands of classifications).

### Struct `Point`

- Only correct if a coordinate + type is *structurally fixed forever* (a struct
  cannot gain fields). Risky: DESI might add errors/ellipsoid fields later. For
  the bulk path prefer the columnar batch (add a new vector). Keep `Point`
  struct only if you need tiny random-access records that will never change —
  otherwise a `table` is safer for any "object record" shape.

### Batch sequence number

- `batch_seq: uint32` lets the client detect drops/reordering/restarts when
  paging/straming through a catalog. Server increments per (catalog, region)
  session. Cheap insurance on an unordered transport.

### Union envelop `ServerMessage`

- Lets **one** WebSocket carry catalog lists, object batches, errors, and
  future message kinds, with the type tag built into the buffer (no guessing by
  file identifier).
- Rust: generated `msg_type()` + `msg_as_object_batch()` etc.
- Odin: hand-rolled union-tag read + payload access.

If you'd rather keep streams homogeneous, skip the union and use separate root
types per endpoint (see `05-streaming-protocols.md` Pattern B).

## Evolution roadmap (defensive)

- **v1 (this schema)**: as above.
- **v2 (next release)**: append `petro_mag_g/z/r: [float]` (or per-band uint16
  scaled), `spectral_class: [byte]`, `is_spec_selected: [bool]` to `ObjectBatch`
  (all **at the end** of the table — v1 decoders ignore them, v2 decoders get
  defaults/absent for v1 data).
- **v3**: `Catalog` gains `coverage_footprint: string` (WKT) at the end.
- Do **not** reorder, remove, or deprecate anything unless the schema owner
  signs off; run `flatc --conform` in CI.

## Compatibility with existing Rust/JSON models

Keep the FlatBuffers schema **in sync** with `api/src/models.rs` (serde) and
`gui/src/data.odin` (mirrors) during the transition:

- Add a `From<CatalogObject> for ObjectBatch`-ish conversion inside the API, or
  (better) have the API build FlatBuffers **directly from the DESI row fetch**,
  bypassing serde for the binary path.
- The JSON endpoints can stay for debugging/curl and for the initial catalog
  list (tiny). Document in `api/` that `/api/v1/objects` will switch to
  `application/flatbuffer` (or gain a `/json` variant) once the client is wired.

## Where the schema lives & codegen ownership

Proposal: a top-level `schema/catalog.fbs` is the single source of truth.

```
schema/
  catalog.fbs            <- cross-language contract
```

- Rust: flatc → `api/src/generated/catalog_generated.rs`
- Odin (Path A): flatcc → `gui/src/generated/catalog_reader.h` (+ bindings)
- Odin (Path B): hand-rolled reader must be tested against fixtures built from
  *this* schema (see `08-testing-strategies.md`).

Add a make target:

```make
schema-gen:
	flatc --rust     -o api/src/generated ../schema/catalog.fbs
	flatcc --c --common -o gui/src/generated ../schema/catalog.fbs
```

and (optionally) a CI check that committed generated files match a fresh
regeneration (`git diff --exit-code` after regen).

## Open questions to resolve with real DESI data

1. DESI object IDs: u64 internal, or string (survey-id like "DESI-1234")?
   If string, `ids: [string]` (each a mini-table) costs far more than `[uint64]`
   — decide based on the actual catalog schema.
2. Coordinate range conventions: ra in `[0,360)` vs `[-180,180)`? Pick once,
   document, optionally store a normalized flag in the schema.
3. Redshift scale/precision: double vs float? For rendering, `f32` halves the
   payload and is plenty; for science fidelity, keep `f64` as in the model today.
4. Does the renderer need per-object colors today? If yes, precompute a packed
   `[uint32]` RGBA vector in the API — better than the client computing per frame.
5. In-stream metadata: do you need a `Metadata` message (region bounds, epoch,
   projection) before the first batch? Add now if so — retrofitting a union arm
   mid-release is possible but adds churn.