Lab Notes • Architecture of the Dark Gravity experiment

10-Namespace Topology, OpenZiti Zero-Trust Security & Limited-Resource Engineering

The current architecture of the Dark Gravity lab: an experimental multi-agent software factory across 10 Kubernetes namespaces, with an OpenZiti overlay and gVisor micro-sandboxing. These are design choices under test; some are implemented, some are still being validated. Check the repository README for the up-to-date status.

Explore 10-Namespace Inter-Relations OpenZiti Dark Network Mesh
Kubernetes Architecture

The 10 Namespaces & Their Precise Inter-Relations

Every capability in Dark Gravity is compartmentalized into a dedicated Kubernetes namespace. Outbound NetworkPolicies enforce a Deny-All posture; communication occurs strictly over authenticated channels with cryptographic identities.

10-Namespace Production Mesh & Service Matrix

Structured across 4 architectural operational planes with zero-trust egress and isolated runtimes.

Layer 1 — Autonomous Execution & Sandboxing Runtimes
ns / agents Deny-All Egress

Autonomous Agent Runtime

Executes Rust-based agents (Rustant, ZeroClaw, DocAgent) and deep research processes. Runs with native sidecars (restartPolicy: Always) for resilient pod lifecycles.

factory-worker factory-mcp-server (:8100) deep-research (:8200) hatchet-proxy (:7070)
Connected Namespaces:
orchestrators (gRPC) confluent (events) development (runsc) security (JIT tokens)
ns / development gVisor Hardware Sandbox

Ephemeral Micro-Sandboxes

Runs untrusted code synthesis, syntax checking, and unit test suites inside strictly isolated user-space kernels via runsc. Configured target limits: ≤ 30 MiB RAM and ≤ 250m CPU (still being validated against real workloads).

gVisor (runsc) Ephemeral Jobs Memory Clamp: 30MiB CPU Clamp: 250m
Connected Namespaces:
agents (dispatched via MCP)
Layer 2 — Workflow Orchestration & Asynchronous Event Streaming
ns / orchestrators Durable State Checkpoints

Hatchet Workflow Engine

Drives crash-resilient 6-phase autonomous DAGs (Ingestion → Plan → Code → Validation → Review → Delivery). Automatically retries flaky steps and checkpoints DAG states directly to PostgreSQL.

hatchet-engine (:7077/:8000) keda-operator gRPC Workflow Dispatch
Connected Namespaces:
agents (gRPC task dispatch) storage (Postgres state) confluent (triggers)
ns / confluent KRaft Consensus

Enterprise Kafka Event Mesh

Zero-dependency high-throughput distributed event streaming for mission ingestion, real-time agent internal thoughts, and cryptographic audit events across all autonomous nodes.

Apache Kafka (KRaft) Schema Registry topic: mission-ingestion topic: agent-thought
Connected Namespaces:
agents (publishes thoughts) orchestrators (triggers DAG)
Layer 3 — AI Intelligence, GraphRAG & Sovereign Data Layer
ns / llm-apps FinOps Budget Controls

LiteLLM AI Gateway & Semantica

Centralized LLM router injecting FinOps virtual tags, enforced spending caps, and caching. Includes Semantica-AGI for tracking causal code provenance and R2R GraphRAG.

LiteLLM Proxy (:4000) Semantica Lineage R2R GraphRAG (:7272)
Connected Namespaces:
agents (model queries) security (API keys)
ns / knowledge AST Trees & Vectors

Semantic Code Memory

Maintains high-dimensional code vectors and symbol dependency graphs using deepwiki-rs (Tree-sitter AST engine) and pgvector for zero-latency semantic codebase lookups.

pgvector CNPG Cluster deepwiki-rs Daemon Code AST Graph Engine
Connected Namespaces:
agents (symbol resolution) storage (persistence)
ns / storage Sovereign S3 + DirectPV

PostgreSQL & S3 Storage Tier

Primary on-premise storage tier. High-performance CloudNativePG PostgreSQL 16 cluster, MinIO S3 object storage for MLflow runs and DAG step artifacts, mounted via direct-attached DirectPV NVMe disks.

CloudNativePG 16 MinIO S3 Sovereign DirectPV CSI Driver
Connected Namespaces:
orchestrators (checkpoints) agents (build artifacts)
Layer 4 — Zero-Trust Security, Dark Network Fabric & GitOps
ns / openziti Zero Open Ports

OpenZiti Dark Network Mesh

Removes the traditional perimeter by eliminating all open listening ports. Encapsulates pod-to-pod communications into an identity-based mTLS 1.3 cryptographic dark overlay mesh.

Ziti Controller (:8441) Ziti Edge Routers (:3022) Dark Egress / Dial-Out
Connected Namespaces:
All 10 Namespaces (Encrypted Overlay)
ns / security 5-Minute JIT Tokens

Vault & SealedSecrets Identity

Issues dynamic, non-renewable 5-minute (300s) Just-In-Time Git tokens for pull request creation. Tokens are designed to be zeroized on drop (Zeroize crate).

HashiCorp Vault Bitnami SealedSecrets Keycloak OIDC
Connected Namespaces:
agents (credential provider) orchestrators (auth)
ns / flux-system Pull Reconciliation

FluxCD v2 GitOps Engine

Continuous reconciliation engine that monitors Git repositories and applies declarative Kubernetes manifests automatically. Completely immune to configuration drift.

Flux Source Controller Kustomize Controller Helm Controller
Connected Namespaces:
All 10 Namespaces (Declarative State)
Namespace Core Microservices & Components Namespace Role in Autonomous Loop Connected Namespaces
1. agents factory-worker, factory-mcp-server (:8100), deep-research-worker (:8200) Hosts the AI personas (*Rustant*, *ZeroClaw*, *DocAgent*). Uses native sidecars with restartPolicy: Always to reduce startup ordering failures. orchestrators, confluent, security, development, llm-apps, knowledge
2. orchestrators hatchet-engine (gRPC :7077, API :8000), keda-operator Coordinates crash-resilient 6-phase durable DAGs. Dispatches steps to workers and persists state transitions. agents, storage, confluent
3. confluent Apache Kafka (KRaft consensus), Schema Registry Decoupled event streaming for mission-ingestion, real-time agent-thought, and audit telemetry. agents, orchestrators
4. storage CloudNativePG PostgreSQL 16, Sovereign MinIO S3 cluster, DirectPV CSI Primary state store: database checkpoints, MLflow experiment artifacts, and encrypted DAG state in S3. orchestrators, agents, knowledge
5. knowledge pgvector CNPG cluster, deepwiki-rs AST engine Maintains semantic code embeddings and Tree-sitter AST symbol graphs for instantaneous code indexing. agents, storage
6. llm-apps LiteLLM Gateway (:4000/:4001), Semantica-AGI, R2R GraphRAG (:7272) Central AI gateway injecting FinOps virtual tags, tracking causal code provenance, and serving GraphRAG memory. agents, security
7. security HashiCorp Vault, Bitnami SealedSecrets, Keycloak OIDC Issues unrenewable 5-minute Just-In-Time (JIT) access tokens for Git and databases; decrypts cluster secrets. agents, orchestrators, storage
8. openziti OpenZiti Controller (:8441), Ziti Edge Routers (:3022) Dark network fabric closing all inbound ports. Provides programmable mTLS 1.3 overlays between pods. All namespaces (mesh overlay)
9. development Ephemeral gVisor (runsc) sandboxed Pods and Jobs Isolated execution environment where untrusted code, compilation, and test suites run with a target limit of ≤ 30 MiB RAM. agents
10. flux-system FluxCD v2 Controllers (Source, Kustomize, Helm, Notification) GitOps pull-based engine aimed at detecting and reconciling cluster drift without exposing write-access credentials to CI/CD. All namespaces (GitOps declarative state)
Network Security

OpenZiti Dark Network Overlay & Zero-Ingress Architecture

Why Dark Gravity is invisible to port scanners and immune to perimeter penetration. All network connections are dial-out, authenticated by cryptographic keypairs, and encrypted in transit via OpenZiti mTLS 1.3.

OpenZiti Dark Fabric vs. Traditional Vulnerable Perimeter

Traditional webhooks require public IP ingress and open firewall ports. Dark Gravity eliminates all public listening ports.

graph LR subgraph ATTACKER["Public Internet Scanners and Adversaries"] SCAN["Nmap, Shodan, Port Scanners"] end subgraph TRADITIONAL["Traditional Vulnerable Setup - REJECTED"] PUB_IP["Public Ingress IP"] PORT["Open Ports 80, 443, 8080"] WEBHOOK["Inbound Webhook Listener"] PUB_IP --> PORT --> WEBHOOK end subgraph DARK_GRAVITY["Dark Gravity Sovereign Cluster - ZERO INGRESS"] FIREWALL["Deny-All Perimeter Firewall
Zero Inbound Ports Open"] subgraph ZITI_FABRIC["OpenZiti Programmable Dark Network Mesh"] Z_ROUTER["Ziti Edge Router
Dial-Out Dark Tunnel"] Z_CTRL["OpenZiti Controller
mTLS 1.3 Zero-Trust Auth"] end subgraph INTERNAL_SERVICES["Internal Kubernetes Microservices"] POLLER["factory-cli poller
Outbound API Queries"] WORKERS["factory-worker Pods"] S3["Sovereign MinIO S3"] end POLLER -->|Outbound TLS Polling| GITHUB["GitHub API"] Z_ROUTER ---|Encrypted Pod-to-Pod Mesh| WORKERS Z_ROUTER ---|mTLS 1.3| S3 Z_CTRL -->|Identity Validation| Z_ROUTER end SCAN -.->|Blocked - No Listening Ports| FIREWALL classDef red stroke:#f43f5e,stroke-width:2px,fill:#1f131a,color:#fff; classDef green stroke:#10b981,stroke-width:2px,fill:#05070c,color:#fff; classDef purple stroke:#a855f7,stroke-width:2px,fill:#0d121f,color:#fff; classDef blue stroke:#00f0ff,stroke-width:2px,fill:#101726,color:#fff; class SCAN,PUB_IP,PORT,WEBHOOK red; class FIREWALL,Z_ROUTER,Z_CTRL,POLLER,WORKERS,S3 green; class ZITI_FABRIC purple;
🛡️

No Public Ingress (design goal)

Traditional webhooks require opening ports (:80, :443) or configuring reverse tunnels. Dark Gravity uses outbound polling: the cluster initiates all connections; no external traffic can probe internal pods.

0 Open Ports Dark Service Anti-DDoS
🔐

mTLS 1.3 Dark Overlay

All pod-to-pod communications (Worker to Hatchet, Worker to Vault, Worker to MinIO) run inside the OpenZiti dark network overlay, encrypted with mutual TLS 1.3 and validated against cryptographic hardware identities.

mTLS 1.3 Edge Routers Encrypted overlay
🚫

Deny-All Egress Policies

Kubernetes NetworkPolicies restrict pod egress. Even if untrusted code executes inside a sandbox, it cannot reach arbitrary IP addresses on the internet or intranet; only designated Ziti edge endpoints are routable.

NetworkPolicy Egress Clamped Air-Gap Strict
Cryptographic Pipeline

Zero-Trust Security Lifecycle (as designed): From Ingestion to Credential Wipe

Cryptographic enforcement at every boundary: Ed25519 Non-Human Identities, HashiCorp Vault 5-minute dynamic session tokens, and Criterion-verified null-byte RAM wiping.

Mission Ingestion to Credential Destruction Lifecycle

sequenceDiagram autonumber actor Human as Human Operator (PO) participant Poller as GitPlatformPoller (Rust) participant Vault as HashiCorp Vault (JIT) participant Kafka as Kafka (mission-input) participant Hatchet as Hatchet Durable Engine participant Worker as factory-worker (ZeroClaw) participant Sandbox as gVisor (runsc) Sandbox participant RAM as Process RAM (ZeroizeOnDrop) Human->>Poller: Tags Issue with autonomous-mission (RAM max 30MiB, Timeout 300s) Poller->>Vault: Requests ephemeral JIT Git token (secret/data/repos) Vault-->>Poller: Issues dynamic token with 5-minute non-renewable TTL Poller->>Poller: Signs payload with Ed25519 (W3C Verifiable Credential) Poller->>Kafka: Publishes signed mission payload Kafka->>Hatchet: Triggers cacd-mission-dag execution Hatchet->>Worker: Dispatches Phase 2 (zeroclaw:execute) Worker->>Sandbox: Spawns ephemeral gVisor container (RAM max 30 MiB, CPU max 250m) Sandbox->>Sandbox: Applies AST refactor and runs unit tests in user-space Sandbox-->>Worker: Emits code patch and test results Worker->>RAM: Drops credential struct (ZeroizeOnDrop) RAM-->>RAM: Null-byte overwrite wipe on drop (micro-benchmarked, not a production guarantee)
Resource Optimization

Operating an Industrial Autonomous Factory on Limited Resources

How Dark Gravity runs on modest hardware or bare-metal servers by replacing heavy virtual machines with sub-second gVisor micro-sandboxes and predictive FinOps budget clamps.

📦

gVisor Micro-Sandboxing (target ≤ 30 MiB RAM)

Traditional VM-based sandboxes require gigabytes of RAM and take 30+ seconds to boot. Dark Gravity uses gVisor (runsc):

  • User-Space Kernel: gVisor intercepts Linux syscalls in user-space, preventing kernel exploits without full virtualization overhead.
  • Resource limits under test: Agent sandbox pods are configured with RAM ≤ 30 MiB and CPU ≤ 250m. Sidecars are restricted to RAM ≤ 20 MiB and CPU ≤ 100m.
  • Sub-Second Boot Time: Micro-sandboxes launch in milliseconds, allowing hundreds of tests to execute concurrently on a single commodity node.
💰

Predictive FinOps & Velocity Clamping

Uncontrolled LLM agent loops can bankrupt a development budget in hours. Dark Gravity enforces hardcoded financial guardrails:

  • Virtual Tags (Vtags): LiteLLM injects HTTP tracking headers (x-vtags-team, x-vtags-epic, x-vtags-cost_center) to attribute every token.
  • Spend Velocity Guard: If burn rate exceeds +$1.00 / 60 seconds, the circuit breaker trips instantly.
  • HardStop Budget Cap: When 90% of the daily budget ($45 of $50) is consumed, the Hatchet DAG halts, JIT tokens revoke, and a Kafka alert is broadcast.
Open-Source Sovereignty

The Complete Open-Source Toolchain

The lab is built on open-source projects, some of them maintained as independent forks. Avoiding vendor lock-in is a goal with its own cost: every fork has to be patched and kept up to date by me.

Layer Component & Project License / Governance Dark Gravity Architectural Function
Language & Core Rust 2024 / Tokio MIT / Apache 2.0 Memory-safe, high-concurrency runtime without a garbage collector.
Orchestration Hatchet Engine Apache 2.0 Crash-resilient durable DAG workflow orchestration with PostgreSQL state persistence.
Event Streaming Apache Kafka (KRaft) Apache 2.0 High-throughput pub/sub event bus streaming cognitive thoughts and audit events.
Relational Database CloudNativePG (PostgreSQL 16) PostgreSQL Open Source HA database cluster managing mission state checkpoints and pgvector embeddings.
Sovereign S3 Storage MinIO Hardened Core (@lgcorzo) Apache 2.0 (Sovereign Fork) Air-gapped object storage for checkpoints, compliance artifacts, and MLflow models.
Hardware SIMD sha256-simd / simdjson-go Apache 2.0 (Sovereign Fork) AVX-512 & ARM64 hardware-accelerated crypto hashing and multi-gigabyte JSON parsing.
Runtime Isolation Google gVisor (runsc) Apache 2.0 User-space kernel sandboxing for untrusted code execution (target ≤ 30 MiB RAM).
Network Security OpenZiti (mTLS 1.3) Apache 2.0 Dark network overlay closing all external ingress ports across Kubernetes clusters.
Secrets Management HashiCorp Vault & SealedSecrets MPL 2.0 / Apache 2.0 Short-lived Just-In-Time (JIT) 5-minute dynamic session tokens and encrypted GitOps secrets.
GitOps Deployment FluxCD v2 Apache 2.0 (CNCF Graduated) Pull-based continuous delivery reconciling desired cluster state from Git repositories.