# 02 — FlatBuffers Overview

## What it is

FlatBuffers is a **cross-platform serialization library** created by Google,
originally for game development and other performance-critical applications. Its
defining feature is **zero-copy data access**: you can read fields directly out
of the serialized byte buffer without parsing, unpacking, or allocating objects.

- **Homepage**: https://flatbuffers.dev
- **GitHub**: https://github.com/google/flatbuffers
- **License**: Apache 2.0
- **Schema language**: `.fbs` files (similar to `.proto` in spirit, but different)
- **Current version**: 25.12.19 (Dec 2025 on crates.io — ships with `flatc`)

## Core guarantees

| Property | Description |
|---|---|
| Zero-copy reads | Fields are accessed in-place via offsets — no deserialization step |
| Zero-alloc reads | Reading a buffer allocates nothing |
| Cross-language | Same schema compiles to C++, C, Rust, Go, Java, C#, JS/TS, Python, Swift, Dart, PHP, Lobster, Lua |
| Forward compatible | Code compiled against schema V1 reads data written with schema V2 (ignores unknown fields) |
| Backward compatible | Code compiled against schema V2 reads data written with schema V1 (returns defaults for missing fields) |
| Little-endian, fixed-width | Deterministic cross-platform serialization (offsets are u32 LE) |
| mmap-friendly | Works with memory-mapped files; only touched bytes are paged in |
| Thread-safe reads | Reading requires no synchronization; tables implement `Send + Sync` |

## How the format works internally

A FlatBuffer is a **contiguous binary blob** of nested objects arranged so the
data can be traversed in place like a pointer-based structure. The key concepts:

### Tables + vtables

- A **table** is an object whose fields are optional. It carries an inline
  `soffset_t` (signed 32-bit) pointing at a **vtable**.
- The **vtable** maps field slots to (offset, presence) pairs. If a field is
  absent (written by older code), the accessor returns the schema's default.
- Multiple tables with the same layout **share the same vtable** — this is where
  the zero-copy "no per-object metadata" win comes from.
- Tables are the *only* construct that supports schema evolution.

### Structs

- A **struct** is a "naked" fixed-layout record with **no vtable** and **no
  per-field optionality** (every field required).
- Layout is deterministic across platforms: scalars aligned to their own size;
  struct aligned to its largest member.
- Smaller and faster than tables, but **cannot evolve** — fields can't be added
  or removed.
- Perfect for things like `Vec3`, `Color`, or a coordinate record that will
  never change. Example: DESI's `ra`/`dec`/`redshift` triplet could be a struct,
  or (for bulk data) arrays of raw doubles.

### Vectors

- Native vector type (unlike protobuf's `repeated`). Length-prefixed, contiguous,
  and for scalars/structs can be accessed via `safe_slice()` returning a
  `&[T]` view (on little-endian machines) — **zero-copy slice access**.
- This is the single most important feature for DESI render workloads:
  `table { ra: [double]; dec: [double]; z: [double] }` gives you three
  `&[f64]` slices straight from the wire buffer.

### Unions

- A `union` is a type-discriminated tagged field (a `Type` enum + value).
- Used for things like `ServerMessage` that can carry different payloads
  (catalog list, object chunk, error).

### File identifiers + size prefixes

- **File identifier**: up to 4 ASCII chars stored at offset 4, enabled in the
  schema with `file_identifier "DESI"`. Lets a reader check "is this buffer a
  DESI buffer" before touching it.
- **Size prefix**: a `u32` LE length written at the start of the buffer,
  enabling stream framing (`size_prefixed_root_*`). This is the mechanism you
  use to frame a stream of FlatBuffer messages.

## Schema evolution rules (summary)

These are the "thou shalt" rules for keeping buffers compatible:

1. Add new fields **only at the end** of a `table`. (Or use explicit `id`
   attributes on every field to allow reordering.)
2. Never remove a field — mark it `(deprecated)` instead.
3. Never change a field's type unless the width is identical *and* there are no
   negative-value edge cases.
4. Never change a field's `default` after shipping (the default is baked into
   generated code, not the buffer).
5. Enums: only ever add values, never remove. Handle unknown values defensively.
6. Structs: **cannot** gain or lose fields. Ever.
7. `required` on a non-scalar field: the verifier fails if it's absent. Adding
   or removing `required` is a compatibility break.
8. Renaming fields/tables is safe for binary compat (buffers address by id, not
   name) but breaks code that uses the old names.
9. Run `flatc --conform schema_v1.fbs schema_v2.fbs` to mechanically check that
   `schema_v2` properly evolves from `schema_v1`.

> A subtle trap from the official docs: **scalar fields equal to their default
> are NOT written to the buffer**. Non-presence is indistinguishable from "wrote
> the default". If presence matters (e.g. `object_count: Option<u64>` where
> `Some(0)` is meaningful) use **optional scalars** (`default: null`) or wrap the
> scalar in a struct.

## Reading a buffer (conceptual)

```
bytes: &[u8]

  ┌─────────────────────────────┐
  │ uoffset (root table offset)  │
  │ file_identifier (optional)   │
  │ ... tables, vtables, data ...│
  └─────────────────────────────┘

root = follow(bytes)              // jump to root table via uoffset
vtable = root - root.vtable_off  // locate vtable for this table
field_ra  = vtable.slot_ra != 0  // present?
if present: ra = read_f64(bytes, root + slot_ra)
```

There is **no parsing loop**. Each accessor is a few offset dereferences and a
read. The Rust runtime exposes this via a `Follow` trait that compiles to near-zero
code after optimization.

## Verification (security)

- The **verifier** walks a buffer and checks offsets, alignment, required fields,
  and structural sanity *before* you access it. Protects against malicious or
  corrupted buffers and out-of-bounds reads.
- Rust: `flatbuffers::root::<T>(&opts, bytes)` runs the verifier (default
  options); `root_unchecked::<T>` skips it for trusted data.
- Available since the 25.x era in Rust; the C port (flatcc) has long had full
  verifier support; vitally, Rust verification is now mature, so the "Rust has
  no verifier" caveat from old docs is outdated.
- For untrusted network data (which is exactly your WebSocket case), **always
  verify** before zero-copy access.
- Verifier limits (`VerifierOptions`) can cap `max_tables`, `max_depth`, etc. to
  defend against resource-exhaustion attacks.

## Buffer size limits

- FlatBuffers uses `u32` offsets internally; a single buffer **cannot exceed
  2 GiB**. `FlatBufferBuilder` enforces `FLATBUFFERS_MAX_BUFFER_SIZE`.
- For DESI catalogs with millions of objects, this means **batch** (chunk) the
  data — e.g. 100k objects per buffer — rather than one giant buffer. That
  aligns naturally with streaming/pagination anyway.

## Why it fits DESI Explorer

- **Render-loop reads**: 100k Galaxy points → drop the network bytes into a
  buffer, call `verify()`, then per frame iterate `ra[i]`, `dec[i], z[i]` as raw
  `f64` slices. Zero allocations, zero GC pressure, cache-friendly contiguous
  arrays.
- **Schema versioning across DESI releases**: `edr` data vs `dr1` vs `dr2`
  differ in coverage/targets. Add `object_id` in release 2 at the end of the
  table; release-1 clients still read old buffers fine.
- **Cross-language integrity**: the same `.fbs` compiles to Rust (API) and C/Odin
  (GUI) with byte-identical wire format — no hand-maintained JSON contract drift,
  which is exactly the `data.odin` mirror-struct problem today.
- **Single obvious streaming primitives**: size-prefixed buffers make framing
  trivial; vectors make bulk data compact.

## Official resources

- Tutorial: https://flatbuffers.dev/tutorial/
- Schema guide: https://flatbuffers.dev/schema/
- Evolution rules: https://flatbuffers.dev/evolution/
- Rust usage: https://flatbuffers.dev/languages/rust/
- C (flatcc) usage: https://flatbuffers.dev/languages/c/
- White paper: https://flatbuffers.dev/white_paper/
- API docs (Rust): https://docs.rs/flatbuffers
- crates.io: https://crates.io/crates/flatbuffers