updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,240 @@
|
||||
# 08 — Unit & Integration Testing Strategies
|
||||
|
||||
FlatBuffers introduces a **cross-language wire contract**. The single most
|
||||
important testing principle: **both sides must prove they talk the identical
|
||||
bytes**, not just that each side round-trips with itself.
|
||||
|
||||
## Testing layers
|
||||
|
||||
### 0. Schema hygiene (fastest feedback)
|
||||
|
||||
- **Schema lint / conformance**: run `flatc --conform base.fbs candidate.fbs`
|
||||
in CI to catch accidental schema-evolution violations (fields added in the
|
||||
middle, removals, type changes). Early warning before any of the below.
|
||||
- **Commit-and-diff**: check generated code (`.rs`, `.h`) into the repo; CI
|
||||
re-runs codegen and `git diff --exit-code`s — catches "you forgot to
|
||||
regenerate" drift.
|
||||
|
||||
### 1. Rust-side unit tests (API crate)
|
||||
|
||||
The `flatbuffers` crate's own tests live in
|
||||
`tests/rust_usage_test/tests/integration_test.rs` — copy the patterns:
|
||||
|
||||
**Round-trip test** (write → read own buffer):
|
||||
|
||||
```rust
|
||||
#[cfg(test)]
|
||||
mod object_batch_tests {
|
||||
use crate::generated::desi::{ObjectBatch, ObjectBatchArgs, finish_object_batch_buffer, root_as_object_batch};
|
||||
use flatbuffers::FlatBufferBuilder;
|
||||
|
||||
fn build_sample_batch() -> Vec<u8> {
|
||||
let mut fbb = FlatBufferBuilder::with_capacity(4096);
|
||||
let ra = fbb.create_vector(&[0.1, 25.3, 99.9]);
|
||||
let dec = fbb.create_vector(&[-5.0, 44.0, -77.2]);
|
||||
let z = fbb.create_vector(&[0.5, 1.2, 2.8]);
|
||||
let ids = fbb.create_vector(&[1u64, 2, 3]);
|
||||
let batch = ObjectBatch::create(
|
||||
&mut fbb, &ObjectBatchArgs {
|
||||
catalog: Some(fbb.create_string("dr1")),
|
||||
batch_seq: 0, n: 3,
|
||||
ids: Some(ids), ra: Some(ra), dec: Some(dec), redshift: Some(z),
|
||||
..Default::default()
|
||||
});
|
||||
finish_object_batch_buffer(&mut fbb, batch);
|
||||
fbb.finished_data().to_vec()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn roundtrip_columnar_batch() {
|
||||
let bytes = build_sample_batch();
|
||||
let batch = root_as_object_batch(&bytes).unwrap();
|
||||
assert_eq!(batch.catalog(), Some("dr1"));
|
||||
assert_eq!(batch.n(), 3);
|
||||
assert_eq!(batch.ra().unwrap()[1], 25.3);
|
||||
assert_eq!(batch.dec().unwrap()[2], -77.2);
|
||||
assert_eq!(batch.redshift().unwrap()[0], 0.5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verifier_rejects_corruption() {
|
||||
let bytes = build_sample_batch();
|
||||
for idx in [0usize, 1, bytes.len() - 1] {
|
||||
let mut bad = bytes.clone();
|
||||
bad[idx] ^= 0xFF;
|
||||
assert!(root_as_object_batch(&bad).is_err(), "byte mutate at {idx} should fail verification");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Key Rust unit-test patterns:
|
||||
- Values equal **defaults are not serialized** → test that explicitly:
|
||||
`assert_eq!(batch.missing_field_u64(), 0)` when field absent, and that an
|
||||
explicitly-set 0 is indistinguishable (unless optional scalar). Cf. the
|
||||
`optional_scalars_test.rs` macro in the flatbuffers repo.
|
||||
- **Alignment tests**: assert struct/table accessor pointers are properly
|
||||
aligned relative to buffer start (the flatbuffers repo has
|
||||
`generated_code_alignment_and_padding` tests — replicate for `Point`/
|
||||
columnar slices).
|
||||
- **Fuzz** (proptest/quickcheck): roundtrip random vectors of `f64`/`u64`
|
||||
through the builder and back (fb's `roundtrip_vectors` uses quickcheck). Add
|
||||
property tests: for any `&[f64]` built with `create_vector`, `ra().unwrap()`
|
||||
equals the input exactly.
|
||||
|
||||
### 2. Rust-side integration tests (HTTP/WS layer)
|
||||
|
||||
The repo already has `api/tests/health.rs` (spins the app against real routes
|
||||
via axum). Add an `api/tests/catalog_fb.rs`:
|
||||
|
||||
```rust
|
||||
// Uses the same axum::Router::app() as health.rs, hits the new WS or
|
||||
// application/flatbuffer endpoint with tower::ServiceExt::oneshot(),
|
||||
// collects the body Bytes, and verifies + reads it with the generated types.
|
||||
#[tokio::test]
|
||||
async fn objects_endpoint_returns_valid_flatbuffers() {
|
||||
let app = desi_explorer_api::app(); // or routes::app()
|
||||
let resp = app.oneshot(Request::builder().uri("/api/v1/objects?catalog=dr1&limit=100").body(Body::empty()).unwrap()).await.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let body = axum::body::to_bytes(resp.into_body(), usize::MAX).await.unwrap();
|
||||
let batch = crate::generated::desi::root_as_object_batch(&body).unwrap();
|
||||
assert_eq!(batch.n(), 100);
|
||||
assert_eq!(batch.ra().unwrap().len(), 100);
|
||||
}
|
||||
```
|
||||
|
||||
For WebSocket integration, spawn the server on `127.0.0.1:0` in a test (tokio)
|
||||
and use `tungstenite`/`tokio-tungstenite` as a test client, or call the endpoint
|
||||
through axum's mock. At minimum: open a WS, receive N binary frames, verify each
|
||||
frame parses + verifies + row counts match expectations.
|
||||
|
||||
### 3. Odin-side tests (GUI crate)
|
||||
|
||||
`odin test` exists in this repo (`make test` → `odin test` in gui/). Structure:
|
||||
- **Reader unit tests** in the Odin package: given a hard-coded byte blob (from
|
||||
a fixture file), `verify` and read expected field values.
|
||||
- **Round-trip** against a builder written in Odin (if you hand-roll a builder
|
||||
for Path B) — mirror the Rust patterns.
|
||||
|
||||
```odin
|
||||
// gui/src_tests/catalog_fb_test.odin (whatever path `odin test` picks up)
|
||||
package catalog_fb
|
||||
|
||||
import "core:testing"
|
||||
|
||||
@(test)
|
||||
test_read_static_batch :: proc(t: ^testing.T) {
|
||||
data := #load("../testdata/catalog_sample.bin", []u8)
|
||||
// Read root, iterate ra/dec/redshift via the reader, assert values.
|
||||
batch := root_as_object_batch(data)
|
||||
testing.expect(t, batch.n == 3)
|
||||
testing.expect(t, batch.ra[1] == 25.3)
|
||||
testing.expect(t, batch.dec[2] == -77.2)
|
||||
testing.expect(t, batch.redshift[0] == 0.5)
|
||||
}
|
||||
```
|
||||
|
||||
The **critical** test here is the one reading a fixture *produced by the Rust
|
||||
side / flatc* — see the "cross-language fixture" section below.
|
||||
|
||||
### 4. Cross-language conformance suite (THE key integration test)
|
||||
|
||||
This is the test that actually catches incompatibility. Design:
|
||||
|
||||
1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
|
||||
- Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic,
|
||||
reproducible in CI with the pinned `flatc`), or by the Rust builder in a
|
||||
small crate that writes testdata.
|
||||
- Cover every field type: catalogs, list, empty batch, batch with defaults
|
||||
omitted, optional scalars present/absent, enum variants, nested strings.
|
||||
2. **Rust reads fixtures** + asserts expected values.
|
||||
3. **Odin reads the same fixtures** + asserts identical expected values.
|
||||
4. **Both sides parse each other's output**: a Rust test writes a buffer and a
|
||||
companion Odin test reads it (and vice versa). Practically: Rust generates a
|
||||
batch during `cargo test`, dumps to `OUT_DIR`, Odin test loads it via
|
||||
`#load` — but static committed fixtures are simpler and just as strong for
|
||||
the schema-versioned contract.
|
||||
5. **Byte-for-byte equality on builders**: for deterministic input, the Rust
|
||||
builder output should byte-identical to flatc's. Assert on fixtures.
|
||||
|
||||
Generate fixture data from a **JSON descriptor** in the repo
|
||||
(`testdata/catalog_sample.json`) via:
|
||||
|
||||
```bash
|
||||
flatc --binary --schema schema/catalog.fbs -- testdata/catalog_sample.json
|
||||
```
|
||||
|
||||
Committed `.bin` files mean: no toolchain needed to run the test suite; CI can
|
||||
regenerate and diff to catch drift.
|
||||
|
||||
**A pure-Odin reader on Path B lives or dies by this suite** — it's the only
|
||||
guarantee your hand-rolled offset/jtable/vtable arithmetic matches Google's.
|
||||
|
||||
### 5. Transport-level integration (end-to-end)
|
||||
|
||||
Two-app test: run the **real** axum server with the **real** DESI fixture data,
|
||||
drive it from an **Odin test binary** (or a Rust test client for parity):
|
||||
|
||||
```rust
|
||||
// api/tests/ws_stream.rs — spawn server on ephemeral port, connect WS
|
||||
// as the client would, assert: schema message first, then N batches,
|
||||
// then EOS; each batch verifies and cumulative n == expected.
|
||||
```
|
||||
|
||||
This closes the loop on the exact runtime path (frame boundaries, file
|
||||
identifier, union tags) that unit tests miss.
|
||||
|
||||
### 6. Property / fuzz testing
|
||||
|
||||
- **Rust**: `proptest` generating arbitrary `&[f64]`/`Vec<u64>` row sets →
|
||||
build batch → read → compare; plus fuzz the *verifier* with mutated buffers
|
||||
(the flatbuffers repo does a one-byte-corruption sweep on its own tests —
|
||||
replicate: for many positions, flip byte, assert verifier either accepts
|
||||
*correctly* (data still consistent) or rejects; never panics / OOB).
|
||||
- **Odin**: property tests are harder (no built-in fuzzer); rely on the Rust
|
||||
side for mutation fuzzing and keep Odin tests as fixed-fixture + round-trip
|
||||
sanity checks.
|
||||
- **Corrupt-length attack**: a stream with `length = 0xFFFFFFFF` must be
|
||||
rejected at framing, not OOM. Test the length-prefix path explicitly
|
||||
(size prefix + WebSocket frame-size caps).
|
||||
|
||||
### 7. CI wiring in this repo
|
||||
|
||||
`.gitea/workflows/ci.yml` currently: `odin test` + build, `cargo test` +
|
||||
clippy, `go test` + vet.
|
||||
|
||||
Additions when FlatBuffers lands:
|
||||
- A **shared schema lane**: `flatc --conform` check + `git diff --exit-code`
|
||||
after regen (catches schema drift).
|
||||
- Rust lane: compile generated code included from `api/src/generated`, run new
|
||||
unit/integration tests (no extra CI deps if generated code is committed).
|
||||
- Odin lane: run the conformance tests reading committed `.bin` fixtures (no
|
||||
extra CI deps).
|
||||
- Optional: the end-to-end WS test (server + client) in the cargo lane.
|
||||
|
||||
Keep `make test` as the single gate: `make test` = odin test + cargo test +
|
||||
go test, and each now includes the FlatBuffers conformance pieces.
|
||||
|
||||
## Testing checklist (implementation-ready)
|
||||
|
||||
- [ ] `flatc --conform` ran against schema history in CI
|
||||
- [ ] Committed generated code matches a fresh regen (CI diff check)
|
||||
- [ ] Rust: round-trip builder↔reader unit tests (all field types / enum /
|
||||
optional scalar / default-omission cases)
|
||||
- [ ] Rust: verifier-safety & corruption fuzz
|
||||
- [ ] Rust: HTTP + WS integration tests against live router
|
||||
- [ ] Odin: reader unit tests on static fixtures
|
||||
- [ ] Cross-language: Rust reads fixtures ✓; Odin reads same fixtures ✓;
|
||||
byte-for-byte parity on builder output
|
||||
- [ ] End-to-end: real server + real client (WS) with DESI-shaped rows
|
||||
- [ ] Length/frame-stealing attack tests (huge length, truncated frames)
|
||||
- [ ] No regression to existing JSON health/catalogs tests during migration
|
||||
|
||||
## References
|
||||
|
||||
- flatbuffers Rust test suite (patterns to copy):
|
||||
https://github.com/google/flatbuffers/blob/master/tests/rust_usage_test/tests/integration_test.rs
|
||||
- optional_scalars_test.rs (default-omission semantics):
|
||||
https://github.com/google/flatbuffers/blob/master/tests/rust_usage_test/tests/optional_scalars_test.rs
|
||||
- `flatc --conform` docs: https://flatbuffers.dev/evolution/
|
||||
- Existing repo test conventions: `api/tests/health.rs`, `make test`, `.gitea/workflows/ci.yml`
|
||||
Reference in New Issue
Block a user