Projects / 05
Security Data Platform & Governance Lab
Experimental work on security telemetry architecture, pipelines, medallion layering, metadata, lineage, data quality and governance.
Overview
This lab treats security telemetry as a data engineering problem. Detection quality, investigation speed and every downstream AI application depend on the structure, completeness and lineage of the data underneath them, and those properties are decided long before an analyst sees an alert.
We investigate how a medallion architecture behaves when the data is security telemetry rather than business events, and what metadata, lineage and quality controls are needed for an investigator to trust a derived table enough to act on it.
The governance question runs alongside the engineering one. Security data is among the most sensitive data an organisation holds, and a platform that improves analytical access without a governance model has simply moved the risk.
The problem
Security telemetry arrives as high-volume, heterogeneous, poorly documented streams whose schemas change without warning when a vendor updates a product. Pipelines built against those streams break silently, and a silently broken security pipeline is indistinguishable from a quiet environment.
Derived data compounds this. Detection logic and AI-assisted analysis increasingly run against normalised and aggregated tables rather than raw events, so an error in a transformation propagates into every decision made downstream, with no trace back to the source unless lineage was designed in.
Governance is usually retrofitted. Access control, retention and classification are applied to raw sources and then lost through the transformation chain, so the derived table that is easiest to query is often the one with the weakest controls attached.
Architecture
The architecture separates ingestion, layered refinement and consumption. Ingestion lands raw telemetry unmodified so the original record is always recoverable. Refinement follows a medallion pattern: a raw bronze layer, a normalised and validated silver layer, and curated gold datasets shaped for detection, investigation and reporting.
- Bronze
- Raw telemetry retained as received, with source and arrival metadata attached and no transformation applied.
- Silver
- Normalised, schema-validated events where quality checks run and failures are recorded rather than dropped silently.
- Gold
- Curated datasets built for specific consumers, each carrying the lineage back to the silver and bronze records it derives from.
- Metadata and lineage
- A catalogue describing what each dataset contains, where it came from and which transformations produced it, maintained as part of the pipeline rather than alongside it.
- Data quality
- Explicit expectations — completeness, freshness, schema conformance — evaluated continuously, with violations raised as operational signals.
- Governance
- Classification, access policy and retention expressed against datasets and carried through derivation, so controls survive transformation.
We study metadata and governance platforms in this ecosystem, including OpenMetadata and Collibra, to understand how catalogue, lineage and policy models are expressed in practice. These are studied as ecosystems only. No partnership, endorsement or commercial relationship of any kind is implied.
Security model
The platform's own contents are the sensitive asset. Security telemetry describes the estate in detail, which makes the analytical layer a high-value target: an attacker with read access learns the monitoring coverage, and one with write access can shape what detection sees.
- Trusted
- The pipeline definitions, the catalogue and the governance policy store, administered through infrastructure controls and change review.
- Untrusted
- Ingested telemetry content. Log fields are attacker-influenceable in many source systems and must never be treated as authoritative simply because they arrived through a pipeline.
- Integrity requirement
- The bronze layer must be effectively append-only. If raw records can be altered after landing, lineage becomes an assertion rather than evidence.
The property we care about most is traceability: any figure presented to an analyst should be traceable to the raw records that produced it, because an investigation that cannot show its provenance cannot be defended afterwards.
Current status
The work is experimental. The layering, metadata and governance model have been designed and are being tested against telemetry generated in our own lab environments. They have not been operated at production volume or against a broad set of vendor sources.
We are not publishing quantitative results from this lab. The volumes and source diversity here are not representative enough for any number to mean something outside it.
Limitations
One limitation is source diversity. A lab generates telemetry from a small number of system types, so schema drift — the failure mode that hurts most in production — is under-represented here.
A second is that lineage at column level is significantly harder than lineage at dataset level, and it is column-level lineage that an investigator actually needs when questioning a single field.
A third is cost. Retaining a full raw layer alongside refined layers is expensive, and we have not studied the trade-offs organisations face when that cost forces raw retention windows down.
A fourth is that governance metadata is only as accurate as the process maintaining it. A catalogue that drifts from reality is worse than no catalogue, because it is trusted.
Roadmap
- Define the quality expectations for each layer and make violations visible as operational signals rather than log entries.
- Test schema-drift handling by deliberately changing source formats and observing where pipelines fail silently.
- Prototype column-level lineage for a single investigation-facing gold dataset.
- Model how classification and retention policy should propagate through derivation.
- Document the governance model in enough detail for external critique.
Detail
Stack
- PostgreSQL
- MinIO
- OpenTelemetry
- Loki
- Prometheus
- Grafana
Topics
- security data
- telemetry pipelines
- medallion architecture
- data lineage
- data quality
- data governance
- metadata