6 Commits

Author SHA1 Message Date
sam_oneal 3a037b5fd3 got dotenv parsing working
CI / Detect changed paths (pull_request) Successful in 4s
CI / Odin unit tests and build (pull_request) Successful in 41s
CI / API unit tests and lint (pull_request) Failing after 33s
CI / Infra unit tests, vet, and preview (pull_request) Successful in 26s
2026-09-08 01:09:55 -06:00
sam_oneal fade4a17dc additional changes
CI / Detect changed paths (pull_request) Successful in 21s
CI / Odin unit tests and build (pull_request) Successful in 45s
CI / API unit tests and lint (pull_request) Failing after 28s
CI / Infra unit tests, vet, and preview (pull_request) Successful in 43s
2026-09-07 19:51:22 -06:00
sam_oneal 1443369f6c add dev resources; plumb GUI_ENV_FILE/API_ENV_FILE/API_DESI_DATA into run targets
CI / Detect changed paths (pull_request) Failing after 33m59s
CI / Infra unit tests, vet, and preview (pull_request) Has been skipped
CI / API unit tests and lint (pull_request) Failing after 31m0s
CI / Odin unit tests and build (pull_request) Failing after 31m9s
2026-09-07 14:55:49 -06:00
sam_oneal ba24005bfb changed how tests are ran; updated dotenv to handle parsing fields into structs
CI / Detect changed paths (pull_request) Failing after 31m29s
CI / Infra unit tests, vet, and preview (pull_request) Has been skipped
CI / API unit tests and lint (pull_request) Failing after 34m25s
CI / Odin unit tests and build (pull_request) Failing after 34m36s
2026-09-07 14:35:30 -06:00
sam_oneal ac2d30510c added support for dotenv files and config
CI / Detect changed paths (pull_request) Failing after 34m20s
CI / Infra unit tests, vet, and preview (pull_request) Has been skipped
CI / API unit tests and lint (pull_request) Failing after 34m36s
CI / Odin unit tests and build (pull_request) Failing after 34m49s
2026-09-07 13:27:51 -06:00
sam_oneal 3803787fe8 updated documentation to have mermaid diagrams; updated AGENTS.md to note that all future diagrams should be mermaid diagrams first with text-based diagrams as fallback where not applicable
CI / Detect changed paths (pull_request) Successful in 6s
CI / Odin unit tests and build (pull_request) Successful in 1m23s
CI / API unit tests and lint (pull_request) Has been skipped
CI / Infra unit tests, vet, and preview (pull_request) Successful in 1m22s
2026-09-06 14:28:57 -06:00
34 changed files with 1085 additions and 97 deletions
+3
View File
@@ -7,3 +7,6 @@ resources/ai/sessions
infra/Pulumi.*.yaml.backup infra/Pulumi.*.yaml.backup
infra/desi-explorer-infra infra/desi-explorer-infra
api/target/ api/target/
.env
.env.*
*.env
+1
View File
@@ -25,6 +25,7 @@ The root `Makefile` is a lean delegator: base commands (`run`, `build`, `test`,
- Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path). - Odin code lives in `gui/src/`; external deps go in `gui/lib/` and are wired via `-collection:lib=lib/local` (or a git submodule imported by relative path).
- Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly. - Everything is plain `make` — no Taskfile — so CI (Gitea Actions) can call `make` directly.
- Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together. - Keep the renderer (gui/), API (api/), and infra (infra/) logically separated; each owns its own Makefile, and the root Makefile is the only place that ties them together.
- **Diagrams in this repo's documentation are Mermaid flowcharts.** Gitea renders ` ```mermaid ` fenced blocks natively. Prefer a Mermaid flowchart over ASCII art / box-drawing diagrams; if a diagram genuinely can't be expressed as a flowchart, fall back to a plain text-based markdown diagram (e.g. a code block or table) rather than hand-rawn ASCII boxes.
## Gotchas ## Gotchas
- Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release. - Odin version is pinned in `.gitea/workflows/*.yml` (`ODIN_VERSION`) and defaults in `scripts/install_odin.sh`; bump both together when tracking a new release.
+12 -4
View File
@@ -9,6 +9,14 @@ GUI := gui
API := api API := api
INFRA := infra INFRA := infra
# Local dev/test assets live under resources/dev. These are the defaults for
# the run/*-web targets; override any of them on the command line, e.g.
# make run GUI_ENV_FILE=/path/to/gui.env API_DESI_DATA=/path/to/data.json
RESOURCE_DIR := $(CURDIR)/resources/dev
GUI_ENV_FILE ?= $(RESOURCE_DIR)/gui.env.example
API_ENV_FILE ?= $(RESOURCE_DIR)/api.env.example
API_DESI_DATA ?= $(RESOURCE_DIR)/desi_subset.json
.PHONY: help setup run run-web build build-debug build-web test clean fmt \ .PHONY: help setup run run-web build build-debug build-web test clean fmt \
renovate-validate renovate-validate
@@ -32,9 +40,9 @@ setup: ## Setup all sub-projects (submodules, gui deps, api deps, infra deps)
## ---- Renderer (Odin) ----------------------------------------------------- ## ---- Renderer (Odin) -----------------------------------------------------
run: ## Run the native app (gui/) run: ## Run the native app (gui/)
@$(MAKE) -C $(API) run & api_pid=$$!; \ @API_ENV_FILE="$(API_ENV_FILE)" API_DESI_DATA="$(API_DESI_DATA)" $(MAKE) -C $(API) run & api_pid=$$!; \
trap 'kill $$api_pid 2>/dev/null' INT TERM EXIT; \ trap 'kill $$api_pid 2>/dev/null' INT TERM EXIT; \
$(MAKE) -C $(GUI) run; \ GUI_ENV_FILE="$(GUI_ENV_FILE)" $(MAKE) -C $(GUI) run; \
kill $$api_pid 2>/dev/null kill $$api_pid 2>/dev/null
build: ## Release build (gui/ + api/) build: ## Release build (gui/ + api/)
@@ -49,9 +57,9 @@ build-web: ## WebAssembly build -> build/web (gui/, needs emscripten)
@$(MAKE) -C $(GUI) build-web @$(MAKE) -C $(GUI) build-web
run-web: ## Start WASM build + API server for web dev run-web: ## Start WASM build + API server for web dev
@$(MAKE) -C $(API) run & api_pid=$$!; \ @API_ENV_FILE="$(API_ENV_FILE)" API_DESI_DATA="$(API_DESI_DATA)" $(MAKE) -C $(API) run & api_pid=$$!; \
trap 'kill $$api_pid 2>/dev/null' INT TERM EXIT; \ trap 'kill $$api_pid 2>/dev/null' INT TERM EXIT; \
$(MAKE) -C $(GUI) build-web; \ GUI_ENV_FILE="$(GUI_ENV_FILE)" $(MAKE) -C $(GUI) build-web; \
kill $$api_pid 2>/dev/null kill $$api_pid 2>/dev/null
## ---- Aggregates ---------------------------------------------------------- ## ---- Aggregates ----------------------------------------------------------
+16
View File
@@ -79,6 +79,22 @@ make clean # remove build artifacts from all projects
make fmt # format all projects make fmt # format all projects
``` ```
`make run` / `make run-web` configure the API and renderer from the example
assets under `resources/dev/` (see the dotenv lib in `gui/lib/local/dotenv`).
Override any of them on the command line:
```sh
make run \
GUI_ENV_FILE=/path/to/gui.env \
API_ENV_FILE=/path/to/api.env \
API_DESI_DATA=/path/to/desi_data.json
```
The defaults point at `resources/dev/gui.env.example` (renderer's `API_URL`),
`resources/dev/api.env.example` (API `API_BIND_ADDR`), and
`resources/dev/desi_subset.json` (a small JSON catalog subset served by the
API's `/api/v1/catalogs` and `/api/v1/objects` endpoints).
Project-specific targets live in their own `Makefile` and are reached with Project-specific targets live in their own `Makefile` and are reached with
`make -C <dir> <target>`: `make -C <dir> <target>`:
+7
View File
@@ -99,6 +99,7 @@ version = "0.1.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"axum", "axum",
"dotenvy",
"serde", "serde",
"serde_json", "serde_json",
"tokio", "tokio",
@@ -108,6 +109,12 @@ dependencies = [
"tracing-subscriber", "tracing-subscriber",
] ]
[[package]]
name = "dotenvy"
version = "0.15.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1aaf95b3e5c8f23aa320147307562d361db0ae0d51242340f558153b4eb2439b"
[[package]] [[package]]
name = "errno" name = "errno"
version = "0.3.14" version = "0.3.14"
+5 -4
View File
@@ -5,17 +5,18 @@ edition = "2021"
description = "Backend API for DESI Explorer — serves DESI survey catalog data" description = "Backend API for DESI Explorer — serves DESI survey catalog data"
[dependencies] [dependencies]
anyhow = "1"
axum = "0.8" axum = "0.8"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal"] } dotenvy = "0.15"
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal"] }
tower-http = { version = "0.7", features = ["cors", "trace"] }
tracing = "0.1" tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] } tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tower-http = { version = "0.7", features = ["cors", "trace"] }
anyhow = "1"
[dev-dependencies] [dev-dependencies]
tower = { version = "0.5", features = ["util"] } tower = { version = "0.5", features = ["util"] }
serde_json = "1"
[profile.release] [profile.release]
lto = true lto = true
+14
View File
@@ -4,6 +4,20 @@
# `api-*` convenience targets). # `api-*` convenience targets).
CARGO ?= cargo CARGO ?= cargo
ROOT := ..
# Normalize API_ENV_FILE / API_DESI_DATA (given relative to the repo root) to
# absolute paths so the API process can open them regardless of its working
# directory. "override" is required because they are usually passed as
# command-line/env vars, which would otherwise override any assignment here.
define normalize_path
ifdef $1
ifneq ($(abspath $($1)),$($1))
override $1 := $(abspath $(ROOT)/$($1))
endif
endif
endef
$(foreach v,API_ENV_FILE API_DESI_DATA,$(eval $(call normalize_path,$v)))
.PHONY: help setup run build test check fmt clean .PHONY: help setup run build test check fmt clean
+14 -1
View File
@@ -1,5 +1,10 @@
use std::path::PathBuf;
pub struct Config { pub struct Config {
pub bind_addr: String, pub bind_addr: String,
/// Path to a DESI data file (JSON) to serve; `None` falls back to the
/// built-in placeholder catalogs.
pub desi_data: Option<PathBuf>,
} }
impl Config { impl Config {
@@ -7,6 +12,14 @@ impl Config {
let bind_addr = let bind_addr =
std::env::var("API_BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8080".to_string()); std::env::var("API_BIND_ADDR").unwrap_or_else(|_| "0.0.0.0:8080".to_string());
Ok(Self { bind_addr }) let desi_data = std::env::var("API_DESI_DATA")
.ok()
.filter(|s| !s.is_empty())
.map(PathBuf::from);
Ok(Self {
bind_addr,
desi_data,
})
} }
} }
+1
View File
@@ -1,3 +1,4 @@
pub mod config; pub mod config;
pub mod models; pub mod models;
pub mod routes; pub mod routes;
pub mod store;
+40 -2
View File
@@ -1,4 +1,7 @@
use desi_explorer_api::{config, routes}; use std::path::Path;
use std::sync::Arc;
use desi_explorer_api::{config, routes, store};
use tracing_subscriber::EnvFilter; use tracing_subscriber::EnvFilter;
@@ -11,8 +14,22 @@ async fn main() -> anyhow::Result<()> {
) )
.init(); .init();
load_env_file()?;
let config = config::Config::from_env()?; let config = config::Config::from_env()?;
let app = routes::app();
let catalog_store = match &config.desi_data {
Some(path) => {
tracing::info!(path = %path.display(), "loading DESI data");
store::CatalogStore::load(Path::new(path))?
}
None => {
tracing::warn!("API_DESI_DATA not set, serving placeholder catalogs");
store::CatalogStore::placeholder()
}
};
let app = routes::app_with_state(Arc::new(catalog_store));
let listener = tokio::net::TcpListener::bind(&config.bind_addr).await?; let listener = tokio::net::TcpListener::bind(&config.bind_addr).await?;
tracing::info!("DESI Explorer API listening on {}", config.bind_addr); tracing::info!("DESI Explorer API listening on {}", config.bind_addr);
@@ -24,6 +41,27 @@ async fn main() -> anyhow::Result<()> {
Ok(()) Ok(())
} }
/// Loads the env file named by `API_ENV_FILE` (if set) into the process
/// environment. Existing env vars are not overridden, so values passed
/// directly on the command line or by the Makefile take precedence.
fn load_env_file() -> anyhow::Result<()> {
let path = std::env::var("API_ENV_FILE").unwrap_or_default();
if path.is_empty() {
return Ok(());
}
match dotenvy::from_path(&path) {
Ok(_) => tracing::info!(%path, "loaded env file"),
Err(err) => {
return Err(anyhow::anyhow!(
"failed to load API_ENV_FILE {path:?}: {err}"
))
}
}
Ok(())
}
async fn shutdown_signal() { async fn shutdown_signal() {
let _ = tokio::signal::ctrl_c().await; let _ = tokio::signal::ctrl_c().await;
tracing::info!("shutting down"); tracing::info!("shutting down");
+4 -4
View File
@@ -1,17 +1,17 @@
use serde::Serialize; use serde::{Deserialize, Serialize};
/// Catalog metadata for a DESI data release/survey. /// Catalog metadata for a DESI data release/survey.
#[derive(Debug, Clone, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Catalog { pub struct Catalog {
pub name: String, pub name: String,
pub release: String, pub release: String,
pub description: &'static str, pub description: String,
pub object_count: Option<u64>, pub object_count: Option<u64>,
} }
/// A single catalog object (galaxy / quasar / star) with its survey /// A single catalog object (galaxy / quasar / star) with its survey
/// coordinates. `ra` and `dec` are in degrees; `redshift` is dimensionless. /// coordinates. `ra` and `dec` are in degrees; `redshift` is dimensionless.
#[derive(Debug, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
pub struct CatalogObject { pub struct CatalogObject {
pub id: String, pub id: String,
pub catalog: String, pub catalog: String,
+24 -30
View File
@@ -1,34 +1,16 @@
use std::sync::Arc;
use axum::{ use axum::{
extract::Query, extract::{Query, State},
http::StatusCode,
response::{IntoResponse, Response},
Json, Json,
}; };
use serde::Deserialize; use serde::Deserialize;
use std::sync::LazyLock;
use crate::models::{Catalog, CatalogObject}; use crate::models::{Catalog, CatalogObject};
use crate::store::CatalogStore;
/// Placeholder catalogs until real DESI EDR/DR1 ingestion lands. pub async fn list_catalogs(State(state): State<Arc<CatalogStore>>) -> Json<Vec<Catalog>> {
static CATALOGS: LazyLock<Vec<Catalog>> = LazyLock::new(|| { Json(state.catalogs.clone())
vec![
Catalog {
name: "edr".to_string(),
release: "EDR".to_string(),
description: "DESI Early Data Release",
object_count: None,
},
Catalog {
name: "dr1".to_string(),
release: "DR1".to_string(),
description: "DESI Data Release 1",
object_count: None,
},
]
});
pub async fn list_catalogs() -> Json<Vec<Catalog>> {
Json(CATALOGS.clone())
} }
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
@@ -38,17 +20,29 @@ pub struct ObjectQuery {
limit: Option<usize>, limit: Option<usize>,
} }
/// Placeholder object query. Real implementation will page through the pub async fn list_objects(
/// centralized DESI catalog store rather than return an empty result set. State(state): State<Arc<CatalogStore>>,
pub async fn list_objects(Query(query): Query<ObjectQuery>) -> Response { Query(query): Query<ObjectQuery>,
) -> Json<Vec<CatalogObject>> {
let limit = query.limit.unwrap_or(100).min(10_000); let limit = query.limit.unwrap_or(100).min(10_000);
tracing::debug!( tracing::debug!(
%limit, %limit,
catalog = query.catalog.as_deref().unwrap_or("all"), catalog = query.catalog.as_deref().unwrap_or("all"),
"querying catalog objects (placeholder)" objects = state.objects.len(),
"querying catalog objects"
); );
let objects: Vec<CatalogObject> = Vec::new(); let objects: Vec<CatalogObject> = match &query.catalog {
(StatusCode::OK, Json(objects)).into_response() Some(catalog) => state
.objects
.iter()
.filter(|o| &o.catalog == catalog)
.take(limit)
.cloned()
.collect(),
None => state.objects.iter().take(limit).cloned().collect(),
};
Json(objects)
} }
+12 -2
View File
@@ -1,13 +1,23 @@
pub mod catalogs; pub mod catalogs;
pub mod health; pub mod health;
use std::sync::Arc;
use axum::{routing::get, Router}; use axum::{routing::get, Router};
/// Builds the application router. Kept separate from `main` so tests can use crate::store::CatalogStore;
/// construct it without binding a socket.
/// Builds the application router with a static placeholder store. Kept
/// separate from `main` so tests can construct it without binding a socket.
pub fn app() -> Router { pub fn app() -> Router {
app_with_state(Arc::new(CatalogStore::placeholder()))
}
/// Builds the application router serving the given catalog store.
pub fn app_with_state(state: Arc<CatalogStore>) -> Router {
Router::new() Router::new()
.route("/health", get(health::health)) .route("/health", get(health::health))
.route("/api/v1/catalogs", get(catalogs::list_catalogs)) .route("/api/v1/catalogs", get(catalogs::list_catalogs))
.route("/api/v1/objects", get(catalogs::list_objects)) .route("/api/v1/objects", get(catalogs::list_objects))
.with_state(state)
} }
+57
View File
@@ -0,0 +1,57 @@
use std::path::Path;
use serde::Deserialize;
use crate::models::{Catalog, CatalogObject};
/// In-memory catalog store, shared (via `Arc`) across routes. Populated either
/// from a DESI data file loaded at startup or from `placeholder`.
#[derive(Debug, Clone, Default)]
pub struct CatalogStore {
pub catalogs: Vec<Catalog>,
pub objects: Vec<CatalogObject>,
}
/// JSON layout of the DESI data file referenced by `API_DESI_DATA`.
#[derive(Debug, Deserialize)]
pub struct DataFile {
pub catalogs: Vec<Catalog>,
#[serde(default)]
pub objects: Vec<CatalogObject>,
}
impl CatalogStore {
/// Static fallback catalogs used when no `API_DESI_DATA` file is given
/// (and by the `routes::app()` test helper).
pub fn placeholder() -> Self {
Self {
catalogs: vec![
Catalog {
name: "edr".to_string(),
release: "EDR".to_string(),
description: "DESI Early Data Release".to_string(),
object_count: None,
},
Catalog {
name: "dr1".to_string(),
release: "DR1".to_string(),
description: "DESI Data Release 1".to_string(),
object_count: None,
},
],
objects: Vec::new(),
}
}
/// Loads catalogs and objects from a JSON data file. Errors on unreadable
/// files or malformed JSON so the caller can fail loudly instead of
/// silently serving empty data.
pub fn load(path: &Path) -> anyhow::Result<Self> {
let text = std::fs::read_to_string(path)?;
let file: DataFile = serde_json::from_str(&text)?;
Ok(Self {
catalogs: file.catalogs,
objects: file.objects,
})
}
}
+93
View File
@@ -0,0 +1,93 @@
use axum::body::{to_bytes, Body};
use axum::http::{Request, StatusCode};
use std::sync::Arc;
use tower::ServiceExt;
use desi_explorer_api::models::CatalogObject;
use desi_explorer_api::routes;
use desi_explorer_api::store::CatalogStore;
fn sample_store() -> CatalogStore {
CatalogStore {
catalogs: Vec::new(),
objects: vec![
CatalogObject {
id: "o1".to_string(),
catalog: "edr".to_string(),
object_type: "GALAXY".to_string(),
ra: 1.5,
dec: 2.5,
redshift: 0.8,
},
CatalogObject {
id: "o2".to_string(),
catalog: "dr1".to_string(),
object_type: "STAR".to_string(),
ra: 3.5,
dec: 4.5,
redshift: 0.0,
},
],
}
}
#[tokio::test]
async fn objects_returns_all_when_no_filter() {
let app = routes::app_with_state(Arc::new(sample_store()));
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/objects")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let objects: Vec<CatalogObject> = serde_json::from_slice(&body).unwrap();
assert_eq!(objects.len(), 2);
}
#[tokio::test]
async fn objects_filters_by_catalog() {
let app = routes::app_with_state(Arc::new(sample_store()));
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/objects?catalog=edr")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let objects: Vec<CatalogObject> = serde_json::from_slice(&body).unwrap();
assert_eq!(objects.len(), 1);
assert_eq!(objects[0].id, "o1");
}
#[tokio::test]
async fn objects_respects_limit() {
let app = routes::app_with_state(Arc::new(sample_store()));
let response = app
.oneshot(
Request::builder()
.uri("/api/v1/objects?limit=1")
.body(Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(response.status(), StatusCode::OK);
let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
let objects: Vec<CatalogObject> = serde_json::from_slice(&body).unwrap();
assert_eq!(objects.len(), 1);
}
+54
View File
@@ -0,0 +1,54 @@
use desi_explorer_api::store::CatalogStore;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn load_parses_catalogs_and_objects() {
let json = r#"{
"catalogs": [
{"name":"edr","release":"EDR","description":"test","object_count":2}
],
"objects": [
{"id":"o1","catalog":"edr","object_type":"GALAXY","ra":1.5,"dec":2.5,"redshift":0.8}
]
}"#;
let dir = std::env::temp_dir().join(format!(
"desi_explorer_store_{}_{}",
std::process::id(),
line!()
));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("data.json");
std::fs::write(&path, json).unwrap();
let store = CatalogStore::load(&path).unwrap();
let _ = std::fs::remove_dir_all(&dir);
assert_eq!(store.catalogs.len(), 1);
assert_eq!(store.catalogs[0].name, "edr");
assert_eq!(store.catalogs[0].object_count, Some(2));
assert_eq!(store.objects.len(), 1);
assert_eq!(store.objects[0].id, "o1");
assert_eq!(store.objects[0].ra, 1.5);
}
#[test]
fn load_rejects_malformed_json() {
let dir = std::env::temp_dir().join(format!(
"desi_explorer_store_{}_{}",
std::process::id(),
line!()
));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("data.json");
std::fs::write(&path, "not json").unwrap();
let result = CatalogStore::load(&path);
let _ = std::fs::remove_dir_all(&dir);
assert!(result.is_err());
}
}
+27 -2
View File
@@ -8,13 +8,28 @@
# output dirs live at the repo root and are referenced through `$(ROOT)`. # output dirs live at the repo root and are referenced through `$(ROOT)`.
ODIN ?= odin ODIN ?= odin
GDB ?= gdb
ROOT := .. ROOT := ..
BIN := $(ROOT)/bin BIN := $(ROOT)/bin
BINARY := $(BIN)/desi_explorer BINARY := $(BIN)/desi_explorer
ODIN_FLAGS := -collection:lib=lib/local ODIN_FLAGS := -collection:lib=lib/local
WASM_DEFINE := RAYLIB_WASM_LIB=env.o WASM_DEFINE := RAYLIB_WASM_LIB=env.o
.PHONY: help setup add-dep run build build-debug build-web test clean fmt # Normalize GUI_ENV_FILE (given relative to the repo root) to an absolute path
# so the Odin process can open it regardless of its working directory.
# "override" is required because GUI_ENV_FILE is usually set on the command
# line (or passed as an env var to this sub-make), which would otherwise
# override any assignment made here.
ifdef GUI_ENV_FILE
ifneq ($(abspath $(GUI_ENV_FILE)),$(GUI_ENV_FILE))
override GUI_ENV_FILE := $(abspath $(ROOT)/$(GUI_ENV_FILE))
endif
endif
# "override" on a command-line variable silently drops it from the recipe
# environment; re-export it so the app can find its env file (run + gdb).
export GUI_ENV_FILE
.PHONY: help setup add-dep run build build-debug gdb build-web test clean fmt
help: ## List available targets help: ## List available targets
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \ @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
@@ -41,12 +56,22 @@ build-debug: ## Debug build -> bin/desi_explorer
@mkdir -p lib/local $(BIN) @mkdir -p lib/local $(BIN)
$(ODIN) build src $(ODIN_FLAGS) -o:none -debug -out:$(BINARY) $(ODIN) build src $(ODIN_FLAGS) -o:none -debug -out:$(BINARY)
gdb: build-debug ## Run the native app under gdb (type 'run', then 'bt' on a crash)
$(GDB) -q --args $(BINARY) $(ARGS)
build-web: ## WebAssembly build -> build/web (needs emscripten) build-web: ## WebAssembly build -> build/web (needs emscripten)
@scripts/build_web.sh @scripts/build_web.sh
test: ## Run Odin unit tests test: ## Run Odin unit tests
@mkdir -p lib/local @mkdir -p lib/local
$(ODIN) test src $(ODIN_FLAGS) @if ls test/*.odin >/dev/null 2>&1; then \
echo "== gui/test =="; \
$(ODIN) test test $(ODIN_FLAGS); \
fi
@for dir in $$(find lib/local -name '*_test.odin' -exec dirname {} \; | sort -u); do \
echo "== $$dir =="; \
$(ODIN) test "$$dir" $(ODIN_FLAGS); \
done
clean: ## Remove build artifacts clean: ## Remove build artifacts
rm -rf $(BIN) build rm -rf $(BIN) build
+167
View File
@@ -0,0 +1,167 @@
package dotenv
import "base:runtime"
import "core:os"
import "core:reflect"
import "core:strconv"
import "core:strings"
// parse parses dotenv-format source (KEY=VALUE lines) into a map allocated
// with allocator. Blank lines, lines starting with '#', and lines without a
// '=' are skipped. Keys and values are trimmed; values may be wrapped in
// double quotes. Real process environment variables take precedence over the
// file. The returned map owns its keys/values; release it with destroy.
@(private)
parse :: proc(src: string, allocator := context.allocator) -> map[string]string {
result := make(map[string]string, allocator)
it := src
for line in strings.split_lines_iterator(&it) {
tr := strings.trim_space(line)
if len(tr) == 0 || strings.has_prefix(tr, "#") {
continue
}
eq := strings.index_byte(tr, '=')
if eq < 0 {
continue
}
key := strings.trim_space(tr[:eq])
if key == "" {
continue
}
value := strings.trim_space(tr[eq + 1:])
if len(value) >= 2 && value[0] == '"' && value[len(value) - 1] == '"' {
value = value[1:len(value) - 1]
}
// real process env vars win over the file
if override, found := os.lookup_env(key, allocator); found {
result[strings.clone(key, allocator)] = override
continue
}
// clone so the map outlives the source buffer (e.g. a freed file read)
result[strings.clone(key, allocator)] = strings.clone(value, allocator)
}
return result
}
// parse_file reads a dotenv file from disk and parses it into a map. It
// returns (nil, false) when the file cannot be read (e.g. it does not exist).
parse_file :: proc(filename: string, allocator := context.allocator) -> (map[string]string, bool) {
data, err := os.read_entire_file(filename, allocator)
if err != nil {
return nil, false
}
defer delete(data, allocator)
return parse(string(data), allocator), true
}
// destroy frees the cloned keys/values and the map itself. Use it to release
// a map returned by parse/parse_file (plain delete does not free the strings).
// Any allocator passed to parse/parse_file must be passed here too.
destroy :: proc(env: map[string]string, allocator := context.allocator) {
for key, value in env {
delete(key, allocator)
delete(value, allocator)
}
delete(env)
}
// env_key returns the env key that should bind to a struct field. It prefers
// an explicit `env:"NAME"` tag; when the tag is absent or empty it falls
// back to the field's name. Matching against the parsed map is
// case-insensitive, so API_URL maps onto api_url (or an `env:"API_URL"` tag).
@(private)
env_key_for_field :: proc(field: reflect.Struct_Field) -> string {
if tag_key, ok := reflect.struct_tag_lookup(field.tag, "env"); ok && tag_key != "" {
return tag_key
}
return field.name
}
// decode populates dest's fields from env, matching each field by name
// (or by an `env:"NAME"` struct tag). Values are converted to the field's
// type: string is cloned as-is into allocator, integers are parsed with
// strconv.parse_int (decimal/hex/negative), booleans with strconv.parse_bool,
// and floats with strconv.parse_f64. Keys missing from env leave the field
// at its zero value. It returns false if a present value cannot be converted
// to the field's type.
decode :: proc(env: map[string]string, dest: ^$T, allocator := context.allocator) -> bool {
ti := reflect.type_info_base(type_info_of(T))
fields, ok := ti.variant.(runtime.Type_Info_Struct)
if !ok {
return false
}
value: string
field_ptr := rawptr(dest)
st: reflect.Struct_Field
for _, i in fields.names[:fields.field_count] {
st = reflect.struct_field_at(T, i)
name := env_key_for_field(st)
value = ""
found := false
for key, v in env {
if key == name || strings.equal_fold(key, name) {
value, found = v, true
break
}
}
if !found {
continue
}
field_ptr = rawptr(uintptr(dest) + fields.offsets[i])
field_ti := reflect.type_info_base(fields.types[i])
#partial switch variant in field_ti.variant {
case runtime.Type_Info_String:
(^string)(field_ptr)^ = strings.clone(value, allocator)
case runtime.Type_Info_Integer:
parsed, err := strconv.parse_int(value)
if !err {
return false
}
switch field_ti.size {
case 1:
(^i8)(field_ptr)^ = cast(i8)parsed
case 2:
(^i16)(field_ptr)^ = cast(i16)parsed
case 4:
(^i32)(field_ptr)^ = cast(i32)parsed
case 8:
(^i64)(field_ptr)^ = cast(i64)parsed
case:
return false
}
case runtime.Type_Info_Boolean:
parsed, err := strconv.parse_bool(value)
if !err {
return false
}
(^bool)(field_ptr)^ = parsed
case runtime.Type_Info_Float:
parsed, err := strconv.parse_f64(value)
if !err {
return false
}
switch field_ti.size {
case 4:
(^f32)(field_ptr)^ = cast(f32)parsed
case 8:
(^f64)(field_ptr)^ = parsed
case:
return false
}
case:
// unsupported field type (slices, pointers, ...) is left untouched
}
}
return true
}
+245
View File
@@ -0,0 +1,245 @@
package dotenv_tests
import "core:os"
import "core:strings"
import "core:testing"
import dotenv "lib:dotenv/src"
Test_Config :: struct {
api_url: string,
debug: bool,
port: int,
ratio: f64,
}
Tagged_Config :: struct {
api_url: string `env:"API_URL"`,
port: int `env:"PORT"`,
debug: bool `env:"DEBUG"`,
}
Tagged_Empty :: struct {
api_url: string `env:""`,
}
// load_env writes src to a unique temp file and parses it via parse_file.
// The returned map owns its strings; callers must destroy it.
load_env :: proc(t: ^testing.T, src: string) -> map[string]string {
dir, err := os.make_directory_temp("", "dotenv_test_*", context.allocator)
testing.expect(t, err == nil, "expected temp dir to be created")
defer os.remove_all(dir)
defer delete(dir)
path := strings.concatenate({dir, "/.env"})
defer delete(path)
testing.expect(
t,
os.write_entire_file(path, src) == nil,
"expected file write to succeed",
)
env, ok := dotenv.parse_file(path)
testing.expect(t, ok, "expected parse_file to succeed")
return env
}
@(test)
test_parse_basic :: proc(t: ^testing.T) {
env := load_env(t, "API_URL=http://127.0.0.1:8080\nDEBUG=true\nPORT=8080\n")
defer dotenv.destroy(env)
testing.expect(t, env["API_URL"] == "http://127.0.0.1:8080")
testing.expect(t, env["DEBUG"] == "true")
testing.expect(t, env["PORT"] == "8080")
}
@(test)
test_parse_ignores_comments_and_blank_lines :: proc(t: ^testing.T) {
env := load_env(t, "# leading comment\n\n \nFOO=bar \nBAZ = qux \n")
defer dotenv.destroy(env)
testing.expect(t, env["FOO"] == "bar", "value should be trimmed")
testing.expect(
t,
env["BAZ"] == "qux",
"key and value should be trimmed around '='",
)
testing.expect(
t,
"API_URL" not_in env,
"comment-only lines should not be parsed",
)
}
@(test)
test_parse_quoted_values :: proc(t: ^testing.T) {
env := load_env(t, "GREETING=\"hello world\"\nEMPTY=\"\"\n")
defer dotenv.destroy(env)
testing.expect(
t,
env["GREETING"] == "hello world",
"quoted value with inner space",
)
testing.expect(t, env["EMPTY"] == "", "double-quoted empty value")
}
@(test)
test_parse_skips_lines_without_equals :: proc(t: ^testing.T) {
env := load_env(t, "not-an-assignment\nOK=yep\n")
defer dotenv.destroy(env)
testing.expect(t, env["OK"] == "yep")
testing.expect(
t,
"not-an-assignment" not_in env,
"line without '=' should be skipped",
)
}
@(test)
test_parse_missing_file :: proc(t: ^testing.T) {
env, ok := dotenv.parse_file("/nonexistent/dotenv_test_does_not_exist.env")
testing.expect(t, !ok, "missing file should report failure")
testing.expect(t, env == nil, "missing file should return nil map")
}
@(test)
test_real_env_overrides_file :: proc(t: ^testing.T) {
testing.expect(t, os.set_env("DESI_EXPLORER_TEST_FOO", "from_env") == nil)
defer os.unset_env("DESI_EXPLORER_TEST_FOO")
env := load_env(t, "DESI_EXPLORER_TEST_FOO=from_file\n")
defer dotenv.destroy(env)
testing.expect(
t,
env["DESI_EXPLORER_TEST_FOO"] == "from_env",
"real env var should win over file",
)
}
@(test)
test_decode_maps_fields_case_insensitively :: proc(t: ^testing.T) {
env := load_env(
t,
"API_URL=http://127.0.0.1:8080\nDEBUG=true\nPORT=8080\nRATIO=0.5\n",
)
defer dotenv.destroy(env)
cfg := Test_Config{}
testing.expect(t, dotenv.decode(env, &cfg))
defer delete(cfg.api_url)
testing.expect(
t,
cfg.api_url == "http://127.0.0.1:8080",
"API_URL maps onto api_url",
)
testing.expect(t, cfg.debug == true, "DEBUG=true should decode to true")
testing.expect(t, cfg.port == 8080, "PORT=8080 should decode to int 8080")
testing.expect(t, cfg.ratio == 0.5, "RATIO=0.5 should decode to f64 0.5")
}
@(test)
test_decode_matches_exact_and_lowercase_keys :: proc(t: ^testing.T) {
env := load_env(t, "api_url=http://exact\nPort=9090\n")
defer dotenv.destroy(env)
cfg := Test_Config{}
testing.expect(t, dotenv.decode(env, &cfg))
defer delete(cfg.api_url)
testing.expect(
t,
cfg.api_url == "http://exact",
"exact-case key should match",
)
testing.expect(t, cfg.port == 9090, "mixed-case key should match field")
}
@(test)
test_decode_missing_keys_leave_zero_values :: proc(t: ^testing.T) {
env := load_env(t, "UNRELATED=value\n")
defer dotenv.destroy(env)
cfg := Test_Config{}
testing.expect(t, dotenv.decode(env, &cfg))
testing.expect(t, cfg.api_url == "")
testing.expect(t, !cfg.debug)
testing.expect(t, cfg.port == 0)
testing.expect(t, cfg.ratio == 0)
}
@(test)
test_decode_unparsable_int_fails :: proc(t: ^testing.T) {
env := load_env(t, "PORT=oops\n")
defer dotenv.destroy(env)
cfg := Test_Config{}
testing.expect(
t,
!dotenv.decode(env, &cfg),
"unparsable int should make decode fail",
)
}
@(test)
test_decode_unparsable_bool_fails :: proc(t: ^testing.T) {
env := load_env(t, "DEBUG=maybe\nAPI_URL=http://127.0.0.1:8080\n")
defer dotenv.destroy(env)
cfg := Test_Config{}
testing.expect(
t,
!dotenv.decode(env, &cfg),
"unparsable bool should make decode fail",
)
defer delete(cfg.api_url)
}
@(test)
test_decode_hex_and_negative_ints :: proc(t: ^testing.T) {
env := load_env(t, "PORT=0x1F\n")
defer dotenv.destroy(env)
cfg := Test_Config{}
testing.expect(t, dotenv.decode(env, &cfg))
testing.expect(t, cfg.port == 31, "hex int should decode")
}
@(test)
test_decode_honors_env_tags :: proc(t: ^testing.T) {
env := load_env(t, "API_URL=http://127.0.0.1:8080\nPORT=9090\nDEBUG=true\n")
defer dotenv.destroy(env)
cfg := Tagged_Config{}
testing.expect(t, dotenv.decode(env, &cfg))
defer delete(cfg.api_url)
testing.expect(
t,
cfg.api_url == "http://127.0.0.1:8080",
"env tag should bind API_URL",
)
testing.expect(t, cfg.port == 9090, "env tag should bind PORT")
testing.expect(t, cfg.debug == true, "env tag should bind DEBUG")
}
@(test)
test_decode_empty_env_tag_falls_back_to_field_name :: proc(t: ^testing.T) {
env := load_env(t, "api_url=http://fallback\n")
defer dotenv.destroy(env)
cfg := Tagged_Empty{}
testing.expect(t, dotenv.decode(env, &cfg))
defer delete(cfg.api_url)
testing.expect(
t,
cfg.api_url == "http://fallback",
"empty env tag should use field name",
)
}
+44
View File
@@ -0,0 +1,44 @@
package main
import "core:log"
import "core:os"
import dotenv "lib:dotenv/src"
Config :: struct {
api_url: string `env:"API_URL"`,
}
get_config :: proc(env_file: ^string = nil) -> (^Config, ^Error) {
path := ".env"
if env_file != nil && env_file^ != "" {
path = env_file^
} else if from_env, ok := os.lookup_env("GUI_ENV_FILE", context.temp_allocator); ok {
path = from_env
}
log.debugf("getting config from file: %s", path)
env, _ := dotenv.parse_file(path, context.temp_allocator)
defer dotenv.destroy(env, context.temp_allocator)
c := new(Config)
if env == nil {
c.api_url = os.get_env("API_URL", context.temp_allocator)
} else if !dotenv.decode(env, c) {
return nil, new_clone(Error{.Config, "failed to decode .env into Config"})
}
if err := validate_config(c); err != nil {
return nil, err
}
return c, nil
}
validate_config :: proc(c: ^Config) -> (err: ^Error) {
if c.api_url == "" {
err = new_clone(Error{.Config, "'API_URL' is required"})
}
return err
}
+7
View File
@@ -1,5 +1,8 @@
package main package main
import "core:net"
import "vendor:curl"
Catalog :: struct { Catalog :: struct {
name: string, name: string,
release: string, release: string,
@@ -22,6 +25,10 @@ APIError :: struct {
} }
get_catalogs :: proc(url: string) -> ([dynamic]Catalog, ^APIError) { get_catalogs :: proc(url: string) -> ([dynamic]Catalog, ^APIError) {
ucurl := curl.url()
defer curl.url_cleanup(ucurl)
return nil, nil return nil, nil
} }
+20
View File
@@ -0,0 +1,20 @@
package main
import "core:fmt"
import "core:strings"
ErrorType :: enum {
Config,
API,
}
Error :: struct {
type: ErrorType,
message: string,
}
format_error :: proc(err: ^Error) -> string {
sb := strings.builder_make(context.temp_allocator)
return fmt.sbprintf(&sb, "[%s] => %s", err.type, err.message)
}
+13 -7
View File
@@ -1,7 +1,9 @@
package main package main
import "base:runtime"
import "core:math" import "core:math"
import "core:math/rand" import "core:math/rand"
import "core:os"
import rl "vendor:raylib" import rl "vendor:raylib"
WIDTH :: 1280 WIDTH :: 1280
@@ -24,6 +26,16 @@ universe: [dynamic]Galaxy
rng: rand.Default_Random_State rng: rand.Default_Random_State
main :: proc() { main :: proc() {
c: ^Config
err: ^Error
s := os.get_env("GUI_ENV_FILE", context.temp_allocator)
if c, err = get_config(&s); err != nil {
panic(format_error(err))
}
rng = rand.create(0xDE51_0000) rng = rand.create(0xDE51_0000)
context.random_generator = rand.default_random_generator(&rng) context.random_generator = rand.default_random_generator(&rng)
@@ -103,11 +115,5 @@ draw :: proc() {
} }
rl.DrawFPS(10, 10) rl.DrawFPS(10, 10)
rl.DrawText( rl.DrawText("DESI Explorer — drag to rotate, scroll to zoom", 10, 34, 18, rl.RAYWHITE)
"DESI Explorer — drag to rotate, scroll to zoom",
10,
34,
18,
rl.RAYWHITE,
)
} }
@@ -59,11 +59,13 @@ get_catalog_objects :: proc(url: string, catalog_name: string) // nil
## Data flow gap ## Data flow gap
``` ```mermaid
[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI] flowchart LR
^ ^ A["DESI catalog store"]
| serde JSON models | hand-mirrored structs B["Rust / axum API<br/><i>serde JSON models</i>"]
| | (stubs, never used) C["Odin + raylib GUI<br/><i>hand-mirrored structs<br/>stubs, never used</i>"]
A -. "future ingestion" .-> B
B --x|"nothing today"| C
``` ```
There is **no live data flow**. The API currently returns JSON placeholders; the There is **no live data flow**. The API currently returns JSON placeholders; the
@@ -116,17 +118,16 @@ response protocol with a cheap binary payload would fit this well.
## High-level target architecture ## High-level target architecture
``` ```mermaid
catalog.fbs (single source of truth, checked into repo) flowchart TB
| S["catalog.fbs<br/><i>single source of truth<br/>checked into repo</i>"]
+--------+---------+ R["flatc --rust"]
| | C["flatcc --c<br/><i>or hand-rolled Odin reader</i>"]
flatc --rust flatcc --c (or hand-rolled Odin reader) API["api/ · Rust"]
| | GUI["gui/ · Odin + raylib"]
api/ (Rust) gui/ (Odin + raylib) S --> R --> API
| ^ S --> C --> GUI
| HTTP / WebSocket (framed FlatBuffer binary stream) API <-->|"HTTP / WebSocket<br/>framed FlatBuffer binary stream"| GUI
+------------------+
``` ```
- One schema file. Two generators. Byte-for-byte identical wire format. - One schema file. Two generators. Byte-for-byte identical wire format.
@@ -105,19 +105,28 @@ These are the "thou shalt" rules for keeping buffers compatible:
## Reading a buffer (conceptual) ## Reading a buffer (conceptual)
Buffer layout:
```mermaid
flowchart LR
subgraph BUF["bytes: &[u8]"]
O["uoffset<br/>root table offset"]
FI["file_identifier<br/>(optional)"]
D["tables · vtables · data"]
end
O --> FI --> D
``` ```
bytes: &[u8]
┌─────────────────────────────┐ Access sequence — each field is a few offset dereferences and a read:
│ uoffset (root table offset) │
│ file_identifier (optional) │
│ ... tables, vtables, data ...│
└─────────────────────────────┘
root = follow(bytes) // jump to root table via uoffset ```mermaid
vtable = root - root.vtable_off // locate vtable for this table flowchart TD
field_ra = vtable.slot_ra != 0 // present? R["root = follow(bytes)<br/>jump to root table via uoffset"]
if present: ra = read_f64(bytes, root + slot_ra) V["vtable = root root.vtable_off<br/>locate vtable for this table"]
Q{"field slot present?"}
R --> V --> Q
Q -- "no" --> DEF["use schema default"]
Q -- "yes" --> RD["ra = read_f64(bytes, root + slot_ra)"]
``` ```
There is **no parsing loop**. Each accessor is a few offset dereferences and a There is **no parsing loop**. Each accessor is a few offset dereferences and a
@@ -196,6 +196,18 @@ Notes:
## Building buffers efficiently (Rust specifics) ## Building buffers efficiently (Rust specifics)
Build order is **back-to-front** (children before parents):
```mermaid
flowchart TB
A["create_string / create_vector<br/>children first"]
B["create nested child tables"]
C["create parent table<br/>ObjectBatch::create(&args)"]
D["builder.finish(root, Some(&#34;DESI&#34;))"]
E["Bytes::copy_from_slice(fbb.finished_data())<br/>→ HTTP / WebSocket response"]
A --> B --> C --> D --> E
```
- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the - `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
buffer across messages. In a loop streaming batches, create one builder, reuse buffer across messages. In a loop streaming batches, create one builder, reuse
it — avoid repeated reallocation. it — avoid repeated reallocation.
@@ -17,6 +17,21 @@ has two C-related access points:
schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's schema plus a small `libflatccrt.a` runtime. Works via the C ABI, so Odin's
`foreign import` can consume it. `foreign import` can consume it.
Choosing a path:
```mermaid
flowchart TD
NAT{"primary target is native desktop?"}
NAT -- "yes" --> CGO{"want to avoid C in the build?"}
CGO -- "yes" --> PATHA["Path B · pure-Odin reader<br/>hand-rolled, no C dependency"]
CGO -- "no" --> PATHA
CGO -- "prefer proven lib / less maintenance" --> PATHC["Path A · FFI to FlatCC<br/>bind generated C headers"]
PATHC --> REUSE["Path C · OdinArrow reuse<br/>or borrow its decode patterns"]
NAT -- "no · browser/WASM" --> PATHD["Path D · TS/JS interop<br/>official JS lib → typed arrays into WASM"]
```
Index of paths:
| Path | Effort | Zero-copy on reads | Notes | | Path | Effort | Zero-copy on reads | Notes |
|---|---|---|---| |---|---|---|---|
| A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` | | A: FFI to FlatCC (C runtime) | Medium | ✅ | Bind generated C headers to Odin `foreign` |
@@ -175,12 +190,14 @@ typed Odin slices.
The rendering win only materializes if data stays zero-copy **into the frame The rendering win only materializes if data stays zero-copy **into the frame
loop**: loop**:
1. Fetch frame bytes → owned `[dynamic]u8` (or a slice pinned for the lifetime ```mermaid
of the frame). flowchart LR
2. `verify` the buffer once. A["fetch frame bytes → [dynamic]u8<br/>or a slice pinned for the frame"]
3. Get `ra_slice := ObjectBatch.ra(&buf)``[]f64` view. B["verify the buffer once"]
4. Per object in `update()`/`draw()`: read `ra[i]`, `dec[i]`, `z[i]` straight C["ObjectBatch.ra(&buf) → []f64 view"]
from that slice; build `rl.Vector3`; `DrawPoint3D`. D["per object in update()/draw()<br/>ra[i] · dec[i] · z[i] → rl.Vector3DrawPoint3D"]
A --> B --> C --> D
```
No per-object allocation. The current `Galaxy { position, color }` dynamic array No per-object allocation. The current `Galaxy { position, color }` dynamic array
in `main.odin` is the data structure you'd replace with *slices into the in `main.odin` is the data structure you'd replace with *slices into the
@@ -24,14 +24,17 @@ offers two built-in mechanisms plus the community pattern:
### Option 1: Size-prefixed FlatBuffers (built-in) ### Option 1: Size-prefixed FlatBuffers (built-in)
```mermaid
flowchart LR
A["u32 LE<br/>total buffer len<br/><i>size prefix</i>"]
B["u32 LE<br/>root table offset"]
C["file identifier<br/>(4 bytes)"]
D["tables · vtables · data"]
A --> B --> C --> D
```
```rust ```rust
builder.finish_size_prefixed(root, Some("DESI")); builder.finish_size_prefixed(root, Some("DESI"));
// +---------------------------+
// | u32 LE: total buffer len | <-- size prefix
// | u32 LE: root table offset |
// | file identifier (4 bytes) |
// | ... data ... |
// +---------------------------+
``` ```
Reader side: Reader side:
@@ -53,8 +56,12 @@ message kinds).
### Option 2: Custom length-prefix framing (like `flatstream`) ### Option 2: Custom length-prefix framing (like `flatstream`)
``` ```mermaid
[ u32 LE: message_len ] [ optional checksum (e.g. u32 crc/xxhash) ] [ flatbuffer payload ] flowchart LR
A["u32 LE<br/>message_len"]
B["optional checksum<br/>(u32 crc / xxhash)"]
C["FlatBuffer payload"]
A --> B --> C
``` ```
- `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation - `flatstream-rs` (see `03-rust-integration.md`) is a reference implementation
@@ -260,6 +267,16 @@ FlatBuffers long-term (mmap-friendly, page-in-what-you-touch).
## Decision summary for this repo ## Decision summary for this repo
```mermaid
flowchart TD
A["HTTP GET → one FlatBuffer body per batch<br/>validate Rust builder + Odin reader"]
B["WebSocket → one Binary message per batch<br/>interactive path · no custom framing"]
C["Self-identifying messages<br/>file_identifier &#34;DESI&#34;"]
D["size-prefixed / flatstream-style framing<br/>or HTTP-range + mmap for static catalogs"]
A --> B --> C
C -. "later, if needed" .-> D
```
1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust 1. Start with **HTTP GET → one FlatBuffer body per batch** to validate the Rust
builder + Odin reader (no protocol work at all). builder + Odin reader (no protocol work at all).
2. Then add **WebSocket** with one `Binary` message per batch (no custom framing) 2. Then add **WebSocket** with one `Binary` message per batch (no custom framing)
@@ -108,6 +108,15 @@ message shape — see `10-performance-benchmarks.md`.)
## Bottom line ## Bottom line
```mermaid
flowchart TD
Q1{"zero-copy reads<br/>in the per-frame render loop?"}
Q1 -- "no" --> PB["Protobuf / gRPC<br/>decode once into draw buffers"]
Q1 -- "yes" --> Q2{"truly columnar?<br/>millions of rows"}
Q2 -- "yes" --> ARR["Apache Arrow IPC<br/>via OdinArrow"]
Q2 -- "no · batched vectors" --> FB["FlatBuffers · this proposal"]
```
- **FlatBuffers is the best default** for this project: the zero-copy read model - **FlatBuffers is the best default** for this project: the zero-copy read model
matches the render loop, the wire format is compact for numeric vectors, schema matches the render loop, the wire format is compact for numeric vectors, schema
evolution fits DESI's release cadence, and the Rust + WASM/JS official story evolution fits DESI's release cadence, and the Rust + WASM/JS official story
@@ -139,7 +139,23 @@ side / flatc* — see the "cross-language fixture" section below.
### 4. Cross-language conformance suite (THE key integration test) ### 4. Cross-language conformance suite (THE key integration test)
This is the test that actually catches incompatibility. Design: This is the test that actually catches incompatibility. Pipeline:
```mermaid
flowchart LR
S["schema/catalog.fbs"]
J["testdata/catalog_sample.json"]
S --> F["flatc --binary"]
J --> F
F --> BIN["committed .bin fixtures<br/>repo-checked-in"]
BIN --> OT["Odin tests<br/>assert identical values"]
BIN --> RT["Rust tests<br/>assert expected values"]
RT -. "deterministic builder" .-> PAR["byte-for-byte parity"]
OT -. "reads it" .-> PAR
PAR -. "catch drift" .-> F
```
Design:
1. **Static fixtures, committed to the repo** (`testdata/*.bin`): 1. **Static fixtures, committed to the repo** (`testdata/*.bin`):
- Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic, - Built once by `flatc --binary <schema>.fbs <data>.json` (deterministic,
@@ -70,6 +70,16 @@ directly from the buffer each frame without allocations.
## Immediate Next Steps (when you're ready to implement) ## Immediate Next Steps (when you're ready to implement)
```mermaid
flowchart TD
A["Prototype schema<br/>catalog.fbs: Catalog · CatalogObject · ServerMessage union"]
B["Generate Rust code<br/>flatc --rust → api/build.rs · serve WS via axum"]
C["Prototype the Odin reader<br/>flatcc FFI · pure-Odin · OdinArrow"]
D["Static fixture files<br/>flatc --binary → committed .bin"]
E["Cross-language tests<br/>Rust + Odin read the same fixtures identically"]
A --> B --> C --> D --> E
```
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`, 1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end). and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see 2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see
+8
View File
@@ -0,0 +1,8 @@
# Example API environment — passed via API_ENV_FILE (see root Makefile).
#
# The API also reads API_DESI_DATA from the environment (set by the root
# Makefile to resources/dev/desi_subset.json by default), so it is not
# repeated here. Values here win unless the same key is already set in the
# real process environment.
API_BIND_ADDR=127.0.0.1:8080
+58
View File
@@ -0,0 +1,58 @@
{
"catalogs": [
{
"name": "edr",
"release": "EDR",
"description": "DESI Early Data Release (local dev subset)",
"object_count": 3
},
{
"name": "dr1",
"release": "DR1",
"description": "DESI Data Release 1 (local dev subset)",
"object_count": 2
}
],
"objects": [
{
"id": "DESI_EDR_000000001",
"catalog": "edr",
"object_type": "GALAXY",
"ra": 150.123456,
"dec": 2.345678,
"redshift": 0.5521
},
{
"id": "DESI_EDR_000000002",
"catalog": "edr",
"object_type": "GALAXY",
"ra": 254.987654,
"dec": -15.203041,
"redshift": 1.1045
},
{
"id": "DESI_EDR_000000003",
"catalog": "edr",
"object_type": "QSO",
"ra": 75.001234,
"dec": 38.765432,
"redshift": 2.8756
},
{
"id": "DESI_DR1_000000001",
"catalog": "dr1",
"object_type": "STAR",
"ra": 188.556677,
"dec": 47.112233,
"redshift": 0.0001
},
{
"id": "DESI_DR1_000000002",
"catalog": "dr1",
"object_type": "GALAXY",
"ra": 300.445566,
"dec": 12.778899,
"redshift": 0.7742
}
]
}
+7
View File
@@ -0,0 +1,7 @@
# Example GUI environment — passed via GUI_ENV_FILE (see root Makefile).
#
# Point the renderer at the local dev API: `make run` also starts the API
# server, so http://127.0.0.1:8080 is the default. Values here win unless
# the same key is already set in the real process environment.
API_URL=http://127.0.0.1:8080