updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,240 @@
|
||||
# 06 — Schema Design for DESI Data
|
||||
|
||||
This doc sketches a FlatBuffers schema for DESI Explorer and the rationale
|
||||
behind each decision. It's a *proposal* — validate against real DESI catalog
|
||||
columns before finalizing.
|
||||
|
||||
## Design goals
|
||||
|
||||
1. **Compact bulk numeric streaming** — the dominant payload is millions of
|
||||
(ra, dec, redshift) triples. Use **columnar vectors of scalars**, not one
|
||||
table per object. This is what makes the performance win real (see
|
||||
`10-performance-benchmarks.md`) — vectors of scalars are contiguous and can be
|
||||
`safe_slice`'d into `&[f64]` views.
|
||||
2. **Schema evolution** for future DESI releases (edr → dr1 → dr2, added
|
||||
columns like `petro_mag`, `spectroscopic_class`, coverage tiles).
|
||||
3. **Self-describing streams** via file identifiers and a message union.
|
||||
4. **Keep the existing API contract** (`Catalog`, `CatalogObject` from
|
||||
`api/src/models.rs`) as the JSON-facing mirror, so `curl`/debug paths stay.
|
||||
|
||||
## Proposed schema
|
||||
|
||||
```fbs
|
||||
// schema/catalog.fbs
|
||||
namespace desi;
|
||||
|
||||
file_identifier "DESI";
|
||||
|
||||
// ---------- Catalog metadata ----------
|
||||
|
||||
table Catalog {
|
||||
name: string; // "edr", "dr1", ...
|
||||
release: string; // "Early Data Release"
|
||||
description: string;
|
||||
object_count: uint64; // Some(0) problem → see note below
|
||||
}
|
||||
|
||||
// ---------- Bulk object data (columnar) ----------
|
||||
|
||||
// A single point, for small results / per-row paths.
|
||||
struct Point {
|
||||
ra: double; // ICRS, degrees, [0, 360)
|
||||
dec: double; // ICRS, degrees, [-90, 90]
|
||||
redshift: double; // dimensionless
|
||||
object_type: byte; // 0=galaxy 1=quasar 2=star (enum → byte)
|
||||
}
|
||||
|
||||
table ObjectBatch {
|
||||
catalog: string; // catalog this batch belongs to
|
||||
batch_seq: uint32; // for reassembling a paged stream in order
|
||||
n: uint64; // number of objects in this batch
|
||||
|
||||
// Columnar arrays — zero-copy friendly, contiguous per column.
|
||||
ids: [uint64]; // object ids (or bytes for survey ids)
|
||||
object_types: [byte]; // enum per object (galaxy/quasar/star)
|
||||
ra: [double]; // degrees
|
||||
dec: [double]; // degrees
|
||||
redshift: [double]; // dimensionless
|
||||
// future columns (release-2): petro_mag_g/r/z (optional vectors),
|
||||
// is_spectroscopic: [bool], ...
|
||||
}
|
||||
|
||||
// ---------- Server → client envelope ----------
|
||||
|
||||
table APIError {
|
||||
code: int32;
|
||||
message: string;
|
||||
}
|
||||
|
||||
table CatalogList {
|
||||
catalogs: [Catalog];
|
||||
}
|
||||
|
||||
union Message {
|
||||
CatalogList,
|
||||
Catalog,
|
||||
ObjectBatch,
|
||||
APIError,
|
||||
}
|
||||
|
||||
table ServerMessage {
|
||||
msg: Message;
|
||||
catalog_list: CatalogList;
|
||||
catalog: Catalog;
|
||||
object_batch: ObjectBatch;
|
||||
error: APIError;
|
||||
}
|
||||
|
||||
root_type ServerMessage;
|
||||
```
|
||||
|
||||
## Rationale per decision
|
||||
|
||||
### Columnar `ObjectBatch` vs. per-object `CatalogObject` tables
|
||||
|
||||
| | Per-object tables | Columnar arrays |
|
||||
|---|---|---|
|
||||
| Size | ~vtable (≥4-16B) + per-row overhead per object | ~8 bytes per f64, zero per-row metadata |
|
||||
| Speed | vtable deref per field per object | Linear memcpy / slice iter |
|
||||
| Odin reads | method call per field per object | `[]f64` direct slice |
|
||||
| Evolution | per-field evolveable | vector types can't structurally evolve (add new vectors instead) |
|
||||
| Fit for render loop | ok for thousands | **ideal for 100k+** |
|
||||
|
||||
3-5 columns of raw doubles per row dominate the payload. Columnar wins by a lot.
|
||||
Rust builds with `create_vector(&flat_array_of_f64)` — trivial. Odin reads with
|
||||
`ObjectBatch_ra(obj)` → `[]f64`.
|
||||
|
||||
Caveat: in the columnar layout, an individual object is identified by *index*
|
||||
across columns (`ra[i]`, `dec[i]`, `redshift[i]`). That's fine for bulk
|
||||
rendering, and `ids[i]` links back to a catalog row when needed.
|
||||
|
||||
### `file_identifier "DESI"`
|
||||
|
||||
- Enables `flatbuffers::size_prefixed_root_with_opts::<ServerMessage>(opts, buf)`
|
||||
and (on the Odin side) cheap "is this our message?" pre-checks.
|
||||
- Without a file identifier, verification accepts buffers that are structurally
|
||||
valid but a *different* schema — dangerous across stream kinds.
|
||||
|
||||
### `object_count: uint64` — the `Option<u64>` presence trap
|
||||
|
||||
Current Rust model has `object_count: Option<u64>`. In FlatBuffers:
|
||||
|
||||
- A `uint64` with `default: 0` **is not written when 0** → reader can't tell
|
||||
"zero" from "absent". If `Some(0)` is meaningful, use an **optional scalar**:
|
||||
`object_count: uint64 (nullable)`? No — the FBS way is:
|
||||
|
||||
```fbs
|
||||
table Catalog {
|
||||
object_count: uint64 (id: 0, default: null); // optional scalar — presence preserved
|
||||
}
|
||||
```
|
||||
|
||||
Optional scalars (`default: null`) are always written when set, and the accessor
|
||||
returns the language's optional type. Rust: `object_count: Option<u64>`;
|
||||
Odin (hand-rolled): an `is_set` flag.
|
||||
|
||||
Alternatively wrap in a struct or keep as raw `uint64` if you never ship a
|
||||
meaningful zero.
|
||||
|
||||
### `enum` for object_type
|
||||
|
||||
```fbs
|
||||
enum ObjectType : byte {
|
||||
Galaxy = 0,
|
||||
Quasar = 1,
|
||||
Star = 2,
|
||||
}
|
||||
```
|
||||
|
||||
- Table fields can be enums; the accessor returns typed enum.
|
||||
- Keep it a `byte`/`ubyte` — flat buffering stores the underlying integer; a
|
||||
vector of enums (`object_types: [ObjectType]`) is just a `&[u8]`/`[]byte` view
|
||||
on the wire (dense — nice for thousands of classifications).
|
||||
|
||||
### Struct `Point`
|
||||
|
||||
- Only correct if a coordinate + type is *structurally fixed forever* (a struct
|
||||
cannot gain fields). Risky: DESI might add errors/ellipsoid fields later. For
|
||||
the bulk path prefer the columnar batch (add a new vector). Keep `Point`
|
||||
struct only if you need tiny random-access records that will never change —
|
||||
otherwise a `table` is safer for any "object record" shape.
|
||||
|
||||
### Batch sequence number
|
||||
|
||||
- `batch_seq: uint32` lets the client detect drops/reordering/restarts when
|
||||
paging/straming through a catalog. Server increments per (catalog, region)
|
||||
session. Cheap insurance on an unordered transport.
|
||||
|
||||
### Union envelop `ServerMessage`
|
||||
|
||||
- Lets **one** WebSocket carry catalog lists, object batches, errors, and
|
||||
future message kinds, with the type tag built into the buffer (no guessing by
|
||||
file identifier).
|
||||
- Rust: generated `msg_type()` + `msg_as_object_batch()` etc.
|
||||
- Odin: hand-rolled union-tag read + payload access.
|
||||
|
||||
If you'd rather keep streams homogeneous, skip the union and use separate root
|
||||
types per endpoint (see `05-streaming-protocols.md` Pattern B).
|
||||
|
||||
## Evolution roadmap (defensive)
|
||||
|
||||
- **v1 (this schema)**: as above.
|
||||
- **v2 (next release)**: append `petro_mag_g/z/r: [float]` (or per-band uint16
|
||||
scaled), `spectral_class: [byte]`, `is_spec_selected: [bool]` to `ObjectBatch`
|
||||
(all **at the end** of the table — v1 decoders ignore them, v2 decoders get
|
||||
defaults/absent for v1 data).
|
||||
- **v3**: `Catalog` gains `coverage_footprint: string` (WKT) at the end.
|
||||
- Do **not** reorder, remove, or deprecate anything unless the schema owner
|
||||
signs off; run `flatc --conform` in CI.
|
||||
|
||||
## Compatibility with existing Rust/JSON models
|
||||
|
||||
Keep the FlatBuffers schema **in sync** with `api/src/models.rs` (serde) and
|
||||
`gui/src/data.odin` (mirrors) during the transition:
|
||||
|
||||
- Add a `From<CatalogObject> for ObjectBatch`-ish conversion inside the API, or
|
||||
(better) have the API build FlatBuffers **directly from the DESI row fetch**,
|
||||
bypassing serde for the binary path.
|
||||
- The JSON endpoints can stay for debugging/curl and for the initial catalog
|
||||
list (tiny). Document in `api/` that `/api/v1/objects` will switch to
|
||||
`application/flatbuffer` (or gain a `/json` variant) once the client is wired.
|
||||
|
||||
## Where the schema lives & codegen ownership
|
||||
|
||||
Proposal: a top-level `schema/catalog.fbs` is the single source of truth.
|
||||
|
||||
```
|
||||
schema/
|
||||
catalog.fbs <- cross-language contract
|
||||
```
|
||||
|
||||
- Rust: flatc → `api/src/generated/catalog_generated.rs`
|
||||
- Odin (Path A): flatcc → `gui/src/generated/catalog_reader.h` (+ bindings)
|
||||
- Odin (Path B): hand-rolled reader must be tested against fixtures built from
|
||||
*this* schema (see `08-testing-strategies.md`).
|
||||
|
||||
Add a make target:
|
||||
|
||||
```make
|
||||
schema-gen:
|
||||
flatc --rust -o api/src/generated ../schema/catalog.fbs
|
||||
flatcc --c --common -o gui/src/generated ../schema/catalog.fbs
|
||||
```
|
||||
|
||||
and (optionally) a CI check that committed generated files match a fresh
|
||||
regeneration (`git diff --exit-code` after regen).
|
||||
|
||||
## Open questions to resolve with real DESI data
|
||||
|
||||
1. DESI object IDs: u64 internal, or string (survey-id like "DESI-1234")?
|
||||
If string, `ids: [string]` (each a mini-table) costs far more than `[uint64]`
|
||||
— decide based on the actual catalog schema.
|
||||
2. Coordinate range conventions: ra in `[0,360)` vs `[-180,180)`? Pick once,
|
||||
document, optionally store a normalized flag in the schema.
|
||||
3. Redshift scale/precision: double vs float? For rendering, `f32` halves the
|
||||
payload and is plenty; for science fidelity, keep `f64` as in the model today.
|
||||
4. Does the renderer need per-object colors today? If yes, precompute a packed
|
||||
`[uint32]` RGBA vector in the API — better than the client computing per frame.
|
||||
5. In-stream metadata: do you need a `Metadata` message (region bounds, epoch,
|
||||
projection) before the first batch? Add now if so — retrofitting a union arm
|
||||
mid-release is possible but adds churn.
|
||||
Reference in New Issue
Block a user