Solutions / 02
Security Data Platform
A designed security data estate: sources, pipelines, schema, lineage and quality — the foundation every detection and AI capability inherits.
The problem
Every capability built on security telemetry inherits the properties of that telemetry, including the ones nobody chose. A detection is only as reliable as the field it matches on. A model is only as trustworthy as the lineage of its training data. A compliance answer is only as fast as the metadata that supports it.
In most estates that inheritance is invisible until something breaks. An upstream format change stops an analytic from matching, and nothing alerts because absence of a match is indistinguishable from absence of a threat. This is the quiet failure that a security data platform exists to make loud.
Architecture
What the system looks like.
Ingestion
Collection with source identity attached at the edge.
Pipelines
Parsing, normalisation, enrichment and routing — as reviewed, tested code.
Quality
Continuous schema, volume, cardinality and freshness checks. A failure is an incident.
Governance
Classification, retention, access and named ownership per source.
Metadata & lineage
Emitted by the pipelines themselves, so impact analysis is a query.
Consumption
Detection, analytics and AI read versioned interfaces, not raw fields.
Data moves through a designed sequence of layers, each with an explicit contract. Verified data proceeds; unknown data is marked as unknown rather than silently promoted; data that fails a quality gate is quarantined where it can be inspected rather than dropped where it cannot.
- Ingestion
- Collection with source identity attached at the edge. A record's origin is established once, at the point it enters the estate, and travels with it.
- Pipelines
- Parsing, normalisation, enrichment and routing. Transformations are code: reviewed, tested and versioned like any other production logic.
- Quality
- Continuous checks on schema conformance, volume, cardinality and freshness. A failing check is an incident, not a dashboard entry.
- Governance
- Classification, retention, access and ownership. Every source has a named owner and a stated retention rationale.
- Metadata & lineage
- Emitted by the pipelines themselves, so 'what breaks if this field changes' is answerable from the system rather than from memory.
- Consumption
- Detection, analytics and AI workloads read from documented, versioned interfaces rather than from whatever field happened to be present.
Capabilities
- Source inventory with owner, schema, volume, fidelity and retention
- Normalisation to a documented schema standard
- Enrichment from asset, identity and threat context
- Tiered routing and storage driven by an explicit cost model
- Continuous data quality checks with alerting
- Lineage emitted as a pipeline output
- Classification, retention and access policy per source
- Versioned consumption interfaces for detection, analytics and AI
Security controls
- Source provenance attached at ingestion and preserved end to end
- Classification-driven access control on downstream datasets
- Quarantine path for records that fail validation, retained for inspection
- Retention enforced by the platform rather than by convention
- Change to any transformation follows code review and deployment control
- Sensitive-field handling — masking, tokenisation or exclusion — decided at design time
Integration approach
The platform is deliberately not a product choice. It is an architecture that can be realised on a SIEM, a lakehouse, a streaming platform, or the combination most estates actually run. What matters is that the contracts between layers exist and are enforced, not which vendor implements them.
- Inventory every source, including the ones without a current owner.
- Establish the schema standard and the naming convention before migrating anything.
- Rebuild the highest-value pipelines against that standard, with tests and quality gates.
- Emit lineage and metadata from the pipelines as they are rebuilt.
- Introduce classification, retention and access policy per source.
- Move consumers onto versioned interfaces and retire direct field access.
Expected outcomes
- Upstream changes that fail visibly rather than silently
- A cost position that reflects deliberate decisions
- Lineage that makes impact analysis a query rather than an investigation
- One schema standard shared by detection, analytics and AI
- Governance answers produced by the platform, not assembled by hand