Projects / 03
Sovereign Agentic AI Lab
A self-hosted agentic AI environment on owned hardware, used to study inference, retrieval and agent security without external model dependencies.
- Prefill throughput
- 22.5 tok/s
- Aggregate generation
- 33.5 tok/s
- Context under test
- 64K
- Prefix cache hit rate
- 94.4%
Selected lab measurement from one tested configuration on this single node. Not a portable benchmark and not comparable across hardware, models or settings.
Selected lab measurement from one tested configuration on this single node. Not a portable benchmark and not comparable across hardware, models or settings.
The context length used in the tested configuration, recorded so the other figures can be read in context. Not a maximum, a target or a benchmark result.
Selected lab measurement from one tested configuration with a workload whose prompts shared long prefixes. Highly workload-dependent and not a portable benchmark.
Lab measurement from a tested configuration in an experimental environment. Not a benchmark, and not portable to other hardware, models or workloads.
Overview
The Sovereign Agentic AI Lab is an agentic AI environment that runs entirely on hardware we own and administer. It exists so that our agent security research can be conducted on a stack we can inspect at every layer, without sending research data to a third-party inference provider.
Owning the stack changes what is observable. Inference behaviour, memory retrieval, tool invocation and the surrounding telemetry can all be instrumented directly, which is what makes experiments on agent security tractable. A hosted API gives us outputs; the lab gives us the intermediate state that the outputs depend on.
The environment is a research instrument, not a service. It is sized for experiments rather than for capacity, and every performance figure reported from it should be read in that light.
The problem
Agent security research on hosted models runs into a visibility ceiling. The parts of the system that matter most — how retrieval assembles context, what the model actually received, how a tool call was formed — sit behind an API boundary, and the provider's behaviour can change between runs without notice.
There is also a data problem. Research on memory poisoning and agent misuse involves constructing adversarial content deliberately. Sending that content to a third-party service is both a confidentiality question and a reproducibility question, since provider-side filtering may silently alter the experiment.
The lab addresses both by keeping models, data, retrieval and telemetry on hardware under our control, and by using open-weight models whose behaviour we can pin for the duration of a study.
Architecture
The hardware is a single AMD EPYC 7451 node — 24 cores, 48 threads, 256 GB of ECC memory, NVMe storage — presenting four NUMA domains, virtualised under Proxmox. The four NUMA domains matter more than the core count for CPU inference: memory locality dominates throughput, and placement decisions are a recurring subject of testing.
The architecture separates inference, state and observability. Inference runs on llama.cpp with Qwen-family open-weight models. State is split across PostgreSQL for relational data, Qdrant for vector retrieval and MinIO for object storage. Observability runs on Prometheus, Grafana, Loki and Tempo, so that metrics, logs and traces from experiments are collected the same way each time. Remote access to the environment is over WireGuard.
Keeping observability in the same environment as the workload is a deliberate choice: an experiment that cannot be traced end to end is an experiment we cannot interpret afterwards.
Security model
The lab is a research environment and is modelled as one. It is isolated from any production system, it holds no customer or third-party data, and access is limited to the operators of the lab over an authenticated tunnel.
- Trusted
- The physical host, the hypervisor and the operators. Compromise at these levels is out of scope for the experiments the lab is used to run.
- Untrusted
- Model output, retrieved documents and any adversarial content constructed for an experiment. All of it is assumed hostile by design, because much of it is written to be.
- Deliberately not assumed
- That self-hosting is itself a security property. Running a model locally removes a third-party dependency; it does not make the model's output trustworthy.
Because adversarial content is created here on purpose, containment is the priority: the lab must not be a path into anything else, and experiments must not depend on network reachability beyond the environment.
Current status
The lab is active and in continuous use as a research environment. The inference, storage and observability stack is stable enough to run repeatable experiments, which is the bar it was built to meet.
It is a single node. There is no redundancy, no capacity planning and no service commitment, and the measurements below come from one tested configuration rather than from a benchmarking programme.
Limitations
One limitation is that CPU inference on a four-domain NUMA node makes throughput sensitive to placement. Small changes in thread and memory affinity move the numbers, which makes cross-run comparison harder than it appears.
A second is that the prefix cache hit rate reflects a workload with heavily shared prompt prefixes. A workload without that property would produce a materially different figure, so the number says as much about the test as about the system.
A third is scale. Findings from a single node with open-weight models of a given size do not necessarily transfer to larger models or to distributed serving, and we do not claim they do.
Roadmap
- Document the tested configuration in full so lab measurements can be interpreted and repeated internally.
- Characterise the sensitivity of throughput to NUMA placement across a defined set of configurations.
- Extend tracing coverage so that retrieval and tool invocation appear in the same trace as inference.
- Run controlled memory-poisoning experiments against the local retrieval path.
- Establish a repeatable procedure for pinning model versions for the duration of a study.
Detail
Stack
- llama.cpp
- Qwen-family open models
- PostgreSQL
- Qdrant
- MinIO
- Prometheus
- Grafana
- Loki
- Tempo
- WireGuard
- Proxmox
Topics
- self-hosted AI
- sovereignty
- inference
- open models
- NUMA
- observability
- lab infrastructure