Architecture Overview¶
Layered Design¶
The platform uses a layered architecture to separate concerns and maintain clarity in responsibilities. Each layer provides defined services and exposes controlled interfaces. This allows components to evolve independently while ensuring governance and auditability across the system.
Execution Flow¶
The platform executes data processing in a sequence of controlled steps. Each step is defined through metadata and audited for traceability.
- Plan — Job, connector, model, and rule definitions are stored in the metadata service. These objects are versioned for reproducibility.
- Schedule — The scheduler emits run requests based on defined triggers or SLAs. The orchestrator builds execution DAGs, manages dependencies, and applies retry policies.
- Connect — Connector adapters load credentials from the secrets vault and establish secure links to source systems through VPN or private link.
- Ingest — Ingestion runners extract data into landing storage (bronze). Logs are collected through the observability stack, and run details are written to the audit service.
- Standardize — Standardization services apply typing, deduplication, and business key alignment. Outputs are stored as standardized datasets (silver).
- Model — The model builder generates Golden Data Point (GDP) schemas and star models (gold). Lineage is recorded in the catalogue.
- Quality — Data quality rules are applied. Failures trigger events on the message bus and are logged in the audit service.
- Anomaly — The anomaly engine computes statistical or rule-based signals. Results are available for consumption or further processing.
- Assemble Insights (optional) — Metadata-bound prompt templates are combined with anomaly results to generate controlled narratives.
- Serve — Dashboards, reports, APIs, and scheduled exports deliver outputs to consumers.
- Observe — Logs, metrics, and traces are aggregated in the observability stack for operations monitoring.
- Access Control — Authentication, authorization, and tenant isolation are enforced across all interactions.
This sequence ensures data flows from source systems to executive consumption in a repeatable, governed manner. Control flows and audit trails are embedded throughout the process.