Live Engineering Logbook · Real Runs & Architectural Decisions

Engineering Lab Notebooks

This is a persistent public record of system tests, architectural evaluations, documented failures, and empirical trade-offs. Rather than curated marketing case studies, each entry answers five core questions: intent, architecture used, actual outcome, lessons/limitations, and next steps.

1. INTENT

What specific hypothesis or capability was tested?

2. CONFIGURATION

What architecture, tools, and constraints were used?

3. REALITY

What actually happened, including error logs & drift?

4. LESSONS

What limits emerged and what architectural insight was gained?

5. NEXT STEP

How does this inform the next experiment or industrial pattern?

TECHNICAL DEMO (RENDERIZADO) 1080p · 60 FPS · 5:00 Walkthrough

Repo: lgcorzo/rust_CACD_autonomous_factory EU AI Act Readiness

Dark Gravity: 5-Minute Technical Demonstration Screencast & Autonomous Pipeline Walkthrough

Walkthrough: docs/screencast_walkthrough.md · Script: scripts/render_screencast.py

Video Specs: 1080p HD · 60 FPS · 5:00 Duration · Renderizado a partir de un guion (render_screencast.py) Watch on YouTube

Esta demostración técnica (renderizada programáticamente a partir de un guion automatizado con trazas reales de misiones) exhibe la orquestación de Dark Gravity, el aislamiento gVisor, la firma de credenciales NHI Ed25519 y la gobernanza estricta Human-in-the-Loop (HITL).

🎬 Screencast Timeline & Section Breakdown

Section 1 (0:00 - 0:45) Ingestion & NHI

Issue Creation & Cryptographic Ingestion

• Issue #104 created in rust_CACD_autonomous_factory with labels [autonomous-mission, dark-gravity].
• Explicit resource constraint parsing: CPU: 500m, RAM: 512Mi, Timeout: 300s.
• Outbound polling daemon (factory-cli poller) detects issue without inbound webhooks.
• Ed25519 NHI engine signs a W3C Verifiable Credential under did:factory:dark-gravity-euskadi.
• Payload dispatched to Confluent Kafka topic mission-input with offset idempotency.

Section 2 (0:45 - 2:00) Orchestration & gVisor

Cluster Orchestration & Sandboxed Execution

• Hatchet orchestrates 6-phase DAG: Ingestion → Plan → Code Sandbox → Validation → SAST Audit → Review.
• ZeroClaw executor spawned inside Kubernetes pod with RuntimeClass: gvisor-runsc (Kernel Isolation L3).
• Real-time cgroup gauges confirm active enforcement: CPU capped at 340m/500m (68%), RAM clamped to 218Mi/512Mi (42%).
• Durable state step checkpoints (StepCheckpoint#03) written to PostgreSQL for fault recovery.
• Zero trust OpenZiti mTLS 1.3 overlay mesh secures inter-agent tool calls with zero public ingress ports.

Section 3 (2:00 - 3:15) Delivery & SAST Gates

GitOps Delivery & Quality Triad Gates

• ZeroClaw performs surgical tree-sitter AST mutations on crates/factory-mcp-server/src/tools/spec_kit_tool.rs.
• Sandboxed unit test suite (cargo test -p factory-mcp-server) achieves 18/18 passes in gVisor sandbox.
• Auditor Agent completes automated SAST scan, verifying 0 high/critical CVEs and a 10.0/10.0 security score.
• Automatically opens PR #42 with unified git diffs, test logs, and cryptographic proof of lineage.

Section 4 (3:15 - 4:15) Governance & HITL

Interactive Directives & HITL Merge Shield

• Human architect comments @darkgravity /status on PR #42; bot reacts with status matrix & FinOps metrics.
• Architect issues @darkgravity /refine Add strict validation for empty payload strings.
• ZeroClaw re-enters gVisor sandbox, applies AST mutation, passes tests (19/19), and pushes commit e9f8a12.
• Strict Governance Shield: Git merge capability is PERMANENTLY BLOCKED (🔒 MERGE BLOCKED - Awaiting Human Maintainer Approval).

Section 5 (4:15 - 5:00) Telemetry & Compliance Audit

Observability, Cost Metrics & EU AI Act Compliance

Execution Telemetry

60,570 Tokens

42,150 prompt + 18,420 completion

FinOps Budget

$0.5097 USD Total

$0.4285 LLM + $0.0812 compute

Stage Latencies

875s Total Run

Sandbox (380s), Val (210s), SAST (128s)

Cryptographic W3C Verifiable Credential audit trail stored for EU AI Act Conformity Assessment & SOC 2 Type II compliance in evidence/missions/mission-001/report.json.

FAILURE & FIX EXP-2026-001 · Oct 11, 2026

Repo: lgcorzo/gitops_internal_lgcorzo

FluxCD CI Pipeline: Kubeconform Validation False Positives on Helm Values & Raw Pod Templates

Commit: 4839b91 (fix: exclude Helm values and raw job templates from kubeconform)

1. What I wanted to verify

Enforce strict schema validation on all Kubernetes manifests in GitOps before merging PRs, eliminating malformed manifests and unparseable CRDs via Kubeconform.

2. Architecture & Setup

GitHub Actions CI pipeline running kubeconform -strict with Flux CRD schemas across all *.yaml files discovered under clusters/ and infrastructure/.

3. What actually happened

The pipeline failed systematically on valid Helm values fragments and parameter-injected sandbox job templates (sandbox-job.yaml) because they were not standalone, schema-complete Kubernetes resource documents.

4. Lessons & Limitations

Static AST/schema tools cannot blindly walk recursive Git trees without discriminating between compiled manifests, parameter files, and template fragments. Naive CI gates create friction and noise.

5. Action & Evolution

Restructured file exclusion rules in the workflow to specifically filter out *values*.yaml and raw templated batch jobs, reserving Kubeconform strictly for hydrated/declarative manifests.

# Pipeline pattern update
find clusters infrastructure -type f $ -name "*.yaml" $
! -name "*values*.yaml" ! -path "*/sandbox-job.yaml" | xargs kubeconform

TRADE-OFF DECISION EXP-2026-002 · Oct 13, 2026

Repo: lgcorzo/rust_CACD_autonomous_factory

Cargo Audit Gates vs. Unmaintained Transitive Crates: Managing Real Dependency Trade-offs

Commit: 087e6247 (fix(ci): format qa_observer, update quinn-proto, and ignore unmaintained advisory)

1. What I wanted to verify

Maintain an automated zero-advisory security standard across the Rust factory codebase via automated cargo audit checks on every branch.

2. Architecture & Setup

Rust 2024 autonomous factory workspace with multiple microservices, Hatchet worker integration, and OpenZiti/Quinn networking stack.

3. What actually happened

Transitive dependency unsafe-libyaml triggered RUSTSEC-2024-0449 (unmaintained warning). Upgrading immediately was impossible without upstream breaking changes in YAML parser consumers.

4. Lessons & Limitations

Total automated security cannot mean blind zero-tolerance halts. Real systems require explicit risk acceptance registers (.cargo/audit.toml) with tracked rationale rather than unvetted hacks or blocking all delivery.

5. Industrial Relevance

Directly translates to enterprise software management: automated scanners flag unmaintained packages, but architectural decisions require distinguishing unexploitable parser edges from critical remote CVEs.

ARCHITECTURAL DECISION EXP-2026-003 · Continuous

Architecture: Hatchet Engine vs. Custom State Machine

Durable Orchestration: Evaluating Hatchet DAGs vs. In-House Actor State Machines

1. What I wanted to verify

Whether a custom state machine in Rust (Tokio + Postgres) offers better operational simplicity than adopting an external durable workflow engine (Hatchet) for multi-step agent tasks.

2. Findings & Failure Modes

Custom state machines failed on node eviction during long-running LLM inferences. Replaying partial DAG state without durable step checkpointing required building a full scheduler from scratch.

3. Lessons Learned

Offloading step-level durability and retry backoff to Hatchet drastically reduces bespoke concurrency code in agent executors, freeing focus for spec verification and sandbox containment.

4. Industrial Application Transfer

In manufacturing environments with multi-step CAD/CAM processing, durable execution patterns prevent orphaned compute jobs when worker pods scale or rebalance under load.

SECURITY BENCHMARK EXP-2026-004 · May 2026

Tools: kube-bench (CIS Benchmark v1.8) & Trivy

Kubernetes Control Plane & Worker Node Hardening: CIS Benchmark Audit and RBAC Least-Privilege

Context: Sanitized audit logs and remediation backlog from gitops_internal_lgcorzo/docs/kube-bench and trivy

1. What I wanted to verify

Evaluate cluster security posture against the official CIS Kubernetes Benchmark and NSA hardening guidance, pinpointing control plane authorization drift, non-root pod compliance, and RBAC privilege inflation.

2. Architecture & Methodology

Automated job execution of kube-bench across control plane / worker profiles alongside cluster-wide trivy k8s scans targeting RBAC roles and container runtime configuration.

3. Findings & Gaps Detected

Identified non-compliant control plane defaults: API server authorization permitting permissive modes, anonymous authentication flags active by default, and third-party operator ClusterRoles (e.g. storage and autoscaling controllers) requesting broad wildcard permissions.

4. Remediation & Lessons

Restricted API server flags to strict Node,RBAC modes, disabled anonymous auth, enabled automatic certificate rotation, clamped file permissions to 600, and applied least-privilege scoping to custom ClusterRoles.

5. Industrial Application Transfer

Default lightweight or developer Kubernetes distributions often ship permissive security configurations for developer convenience. In industrial OT/IT integration, automated CIS auditing gates are mandatory before hosting factory telemetry or execution agents.

GITOPS & CRYPTO DISASTER RECOVERY EXP-2026-005 · Dec 2025 – Present

Framework: Bitnami SealedSecrets & FluxCD

Asymmetric Secret Governance in Public Repositories: Testing SealedSecrets Key Rotation & Disaster Recovery

Context: Sanitized procedure from gitops_internal_lgcorzo/docs/sealed_secrets_recovery.md

1. What I wanted to verify

Validate a zero-plaintext-in-Git architecture using Bitnami SealedSecrets and verify cluster migration and disaster recovery protocols without exposing unsealed keys or locking out automation.

2. Architecture & Key Lifecycle

Public/private asymmetric keypair managed in-cluster by the sealed-secrets controller. CI/CD developers encrypt secrets locally using the public certificate; only the in-cluster private key can decrypt them into native Kubernetes secrets.

3. Disaster Recovery Scenario Tested

Simulated total cluster teardown and rebuild from scratch. The primary operational risk tested: if the master sealing key is lost, every encrypted manifest in Git becomes permanently unrecoverable, halting Flux reconciliation.

4. Lessons & Operational Controls

Established an offline out-of-band master key recovery procedure, verified key injection prior to Flux root sync, and tested secret unsealing verification steps to eliminate bootstrap deadlocks.

5. Industrial Application Transfer

Enables strict GitOps compliance in regulated industrial settings: Git repositories can be audited publicly or semi-publicly without risking database passwords, API tokens, or OT gateway credentials.

AUTONOMOUS RESILIENCE EXP-2026-006 · Oct 2026

Repo: lgcorzo/rust_CACD_autonomous_factory

Durable Execution & Idempotency: Crash Resilience Under Node Termination in Hatchet Orchestrator

Artifact: evidence/missions/mission-001/recovery-test.json · Lab: EXP-2026-006

1. What I wanted to verify

Prove that terminating an autonomous agent worker mid-execution with SIGKILL / pod eviction does not leave orphan branches or create duplicate pull requests upon recovery.

2. Architecture & Setup

Hatchet durable task engine with persistent step state stored in PostgreSQL. Worker executed in Kubernetes sandbox with Ed25519-signed checkpointing.

3. What actually happened

Simulated pod kill during the Code step. Upon orchestrator restart, the task resumed from the exact step checkpoint in 3.4s without re-running completed git initializations or producing duplicate PRs.

4. Lessons & Operational Controls

Agent tasks must be strictly partitioned into pure read/plan phases and transactional write steps with deterministic idempotency keys derived from task UUID and commit SHA.

5. Industrial Application Transfer

In physical automation (CNC machines, automated storage systems), interrupted jobs must never re-dispatch destructive physical actions. Durable state guarantees zero-duplicate side effects.

SANDBOX SECURITY EXP-2026-007 · Oct 2026

Runtime: gVisor runsc & Calico Egress Policies

Zero-Trust Execution Boundary: Syscall Filtering and Calico Egress Blocking Under Adversarial Injection

Artifact: evidence/security/threat-model.md · Lab: EXP-2026-007

1. What I wanted to verify

Verify that untrusted LLM-generated code running inside agent workers cannot execute host kernel escapes, access host device nodes, or exfiltrate credentials over public networks.

2. Architecture & Setup

gVisor (runsc) user-space kernel sandbox paired with strict Calico Default Deny egress network policy and non-root UID execution.

3. What actually happened

Kernel exploit simulations (sys_ptrace, klogctl) were blocked at the user-space boundary (Operation not permitted). Outbound exfiltration probes to public IPs timed out with 100% packet drop.

4. Lessons & Operational Controls

Standard Linux namespaces and cgroups do not provide sufficient isolation for autonomous code execution. Sandboxed user-space virtualization kernels (gVisor or Firecracker) are non-negotiable.

5. Industrial Application Transfer

Edge industrial computing nodes connected to PLCs and OT fieldbuses must run untrusted analytic extensions in microVMs to prevent lateral movement into physical process controllers.

BENCHMARK & EVALUATION EXP-2026-008 · Oct 2026

Repo: lgcorzo/mlops-python-package

Verification Triad: Deterministic Pytest Suites vs. LLM-as-a-Judge Plausibility Bias (44% False Negatives)

Artifact: evidence/evaluations/llm-judge-vs-pytest-benchmark.json · Lab: EXP-2026-008

1. What I wanted to verify

Measure the accuracy gap and false-negative rate when using LLM code reviewers to evaluate PR safety vs. deterministic automated test suites (pytest, mypy, Semgrep).

2. Architecture & Dataset

50 code mutations on ML inference routines (25 correct, 25 subtle defects: silent slicing, divide-by-zero, off-by-one). Evaluated by GPT-4o reviewer vs automated pytest suite.

3. What actually happened

The LLM judge suffered a 44% False Negative rate (approving 11 flawed PRs due to convincing docstrings and plausible syntax). The automated test suite caught 100% (25/25) of defects.

4. Lessons & Operational Controls

LLM judges are useful for readability and style guidance, but must never act as PR merge gates. Deterministic test suites and strict type checkers are mathematically required.

5. Industrial Application Transfer

In robotic welding and CNC kinematic controllers, plausible-looking code with geometric sign errors causes catastrophic physical collisions. Safety relies on mathematical proof, not LLM opinion.

CONTEXT OPTIMIZATION EXP-2026-009 · Oct 2026

Tools: Graphify AST Knowledge Graph & R2R

AST Subgraphs vs. Raw Grep: Achieving 91.8% Token Reduction and 94% First-Pass Code Generation

Artifact: evidence/benchmarks/ast-graph-token-efficiency.json · Lab: EXP-2026-009

1. What I wanted to verify

Measure token consumption, retrieval latency, and refactoring accuracy when providing agents with a 2-hop AST dependency subgraph instead of full-file dumps or keyword grep.

2. Architecture & Methodology

Evaluated on a 120-file code repository (>45k LoC). Baseline: ripgrep dumping matching files into prompt context. Graph: 2-hop neighborhood extraction from Graphify graph.json.

3. What actually happened

The AST graph slashed prompt tokens from 38,400 to 3,150 (91.8% token cut) and retrieval latency from 3.8s to 420ms. First-pass compilation rate jumped from 62% to 94%.

4. Lessons & Operational Controls

Eliminating "lost-in-the-middle" attention degradation by pruning irrelevant symbols is the single most effective way to improve agent code quality and cut inference costs.

5. Industrial Application Transfer

In massive industrial software codebases (millions of lines of C++/C#/Rust), targeted AST dependency subgraphs are the only technically viable and cost-effective mechanism for automated refactoring.

RESOURCE GOVERNANCE EXP-2026-010 · Oct 2026

Runtime: gVisor / cgroups v2 (512Mi Clamping)

Predictable Cloud-Native Execution: Strict 512Mi Memory Clamping and Safe Out-of-Memory Termination

Artifact: evidence/benchmarks/resource-profiling-512mi.json · Lab: EXP-2026-010

1. What I wanted to verify

Verify that autonomous factory tasks strictly adhere to declared resource budgets (CPU: 500m, RAM: 512Mi, Timeout: 300s) and handle intentional memory exhaustion gracefully.

2. Architecture & Setup

Kubernetes pod configured with strict cgroups v2 resource limits running on gVisor runtime, profiled via Prometheus/cAdvisor exporters.

3. What actually happened

Standard test runs peaked at 285 MiB RAM (55.6% budget) and 380m CPU. When subjected to intentional memory exhaustion, gVisor terminated the pod cleanly in 2.8s (exit code 137, OOMKilled) with zero host memory pressure.

4. Lessons & Operational Controls

Compiling large Rust workspaces requires 1.5–2 GiB; realistic micro-sandboxes (512Mi) must execute pre-built binaries or targeted test suites. Honest resource profiling prevents production surprise.

5. Industrial Application Transfer

On shopfloor edge industrial PCs, autonomous AI diagnostic workers must operate within hard memory ceilings to ensure they never starve real-time deterministic machine control cycles.

Laboratory Boundary Notice

Experiments conducted in this lab are personal research implementations. They do not expose confidential client infrastructure, proprietary enterprise code, or unvetted production configurations. Learnings inform reusable architectural patterns tested in enterprise contexts through separate, formalized governance.

← Return to Main Overview