vxedge documentation

Specification, results and user documentation for vxedge.

vxedge is an open, all-software testbed for mapping 5G network slices onto zonal and central software-defined-vehicle (SDV) E/E architectures on a single machine. Every published number ships with the manifest and image digests that produced it.

Pre-release (0.x): the manifest schema and vxAPI may change between minor versions; published results are frozen.

The vxedge specification and user documentation.

Repositories

Links

• Site: https://vxedge.dev
• Architecture: docs/ARCHITECTURE.md
• Results: docs/RESULTS.md · Roadmap: docs/ROADMAP.md
• Contributing: CONTRIBUTING · Security: SECURITY
• License: Apache-2.0 (LICENSE) for code; CC BY 4.0 (LICENSE-DOCS) for docs, results, figures and models

Getting started

vxedge is an all-software research platform for mapping 5G network slices onto zonal & central SDV (software-defined vehicle) E/E architectures: a virtual edge gateway (vxgateway), QEMU/KVM virtual ECUs, containerized AD compute, and a Rust control plane (vxedged) driven entirely by YAML manifests. Read ARCHITECTURE.md for the full picture; this page gets you from zero to a measured experiment; day-to-day usage lives in USER-GUIDE.md.

Pre-release (0.x). The manifest schema and vxAPI may change between minor versions (VERSIONING.md); published results are frozen and do not change.

1. Prerequisites

Verified versions as of 2026-07:

Later phases additionally need QEMU/KVM + libvirt, Open vSwitch, and an NVIDIA GPU (CARLA/Autoware/Alpamayo workloads) — none of that is required to build and run the control plane today.

2. Workspace layout

vxedge is a meta-workspace of independent git repos, cloned side by side (the parent folder is not a git repo):

mkdir vxedge && cd vxedge
for r in docs vxedge vxparts vxota vxagents vxtelemetry vxperiments vxmodels; do
  gh repo clone vxedge/$r
done

Every repo has an AGENTS.md (scope + rules) and TASKS.md (phased work items); docs/ROADMAP.md is the program-level board with the cross-repo critical path.

3. Build everything

for r in vxedge vxagents vxtelemetry; do (cd $r && cargo build --workspace); done

Quick unprivileged smoke (daemon + CLI over a Unix socket):

cd vxedge
./target/debug/vxedged --listen unix:///tmp/vxedge.sock --runtime-dir /tmp/vxrt &
./target/debug/vxedge --endpoint unix:///tmp/vxedge.sock status
# gateway vxgateway: state=1 detail=no gateway provisioned
./target/debug/vxedge --help          # full command tree

Real provisioning (up, zone up) needs CAP_NET_ADMIN — run vxedged under sudo, or sudo setcap cap_net_admin+ep target/debug/vxedged. Without it, up validates the manifest and then fails with an actionable message (by design).

4. Run your first experiment (exp-001)

The flagship walkthrough: a full slice-isolation measurement, one command per step.

cd vxperiments/experiments/exp-001-ws
sudo -E env "PATH=$PATH" ./scripts/run.sh --quick                              # ~1 min pipeline validation
python3 ../../notebooks/analyze_exp001.py ../../results/exp-001-quick          # table + H1 verdict

After pulling new commits, rebuild all three binaries before a run (cargo build --workspace in vxedge, vxagents and vxtelemetry, as your own user, not root). The runner refuses to measure with a binary older than its source and tells you the exact command, including a touch of the rebuilt binaries.

Zone images are pulled from ghcr.io/vxedge/parts/.... While those packages are private, run sudo env "PATH=$PATH" oras login ghcr.io once (the daemon runs as root); the exp-001 runner does not need it.

Results land in vxperiments/results/. Quick runs write to results/exp-001-quick/ (disposable); the frozen campaign owns results/exp-001/, and reruns use --repro (date-suffixed dir) — a dir holding a frozen campaign's archives is never overwritten in place.

The --quick numbers are pipeline validation only (marked NOT reportable). The full campaign is ./scripts/run.sh --repro (it never overwrites the frozen results/exp-001/): ~35 minutes, 5 repetitions x 5 load levels, 60k samples per point — keep the machine otherwise idle and prefer the performance CPU governor. Reference result and methodology: RESULTS.md and the experiment-methodology skill.

5. Zones and VM guest bundles

Container-backed zones work out of the box (vxedge zone up --vxid … after up). The VM backends (qemu-microvm, libvirt) boot a guest bundle built once per host:

sudo chmod 644 /boot/vmlinuz-$(uname -r)        # Ubuntu ships the kernel root-only
cd vxparts && ./parts/zone/linux/stack/probe-bundle/build.sh   # -> ./bundle + digests

See USER-GUIDE.md §3c for wiring a bundle into a manifest.

6. Poke the API directly

The contract lives in vxedge/proto/vxedge/v1/ (package vxedge.v1). Any gRPC tool works; with grpcurl, from the workspace root:

grpcurl -unix -plaintext -import-path vxedge/proto \
  -proto vxedge/v1/lifecycle.proto \
  /tmp/vxedge.sock vxedge.v1.GatewayService/Status

Proto hygiene: buf lint in the vxedge repo (CI enforces it).

7. Manifests — the only input

Everything the platform does is declared in YAML manifests (schema: ARCHITECTURE.md §6; worked example: §16). The parser is strict — unknown fields are errors — and semantic validation covers slice references, vxid uniqueness, and slice-mode consistency. Experiment manifests live in vxperiments/experiments/, vehicle manifests in vxperiments/vehicles/.

8. Troubleshooting

• protoc not found during cargo build — install protoc (table above); the proto codegen runs inside the build via crates/vxedge-api/build.rs.
• cannot reach vxedged at unix:///run/vxedge.sock — the default socket path needs root-writable /run; for development pass a user-writable path to both sides (--listen/--endpoint unix:///tmp/vxedge.sock) or set VXEDGE_ENDPOINT.
• A verb prints "not wired to the daemon yet (scaffold)" — truthfully unimplemented; USER-GUIDE.md §2 lists each verb's state. Check it before assuming a bug.
• An experiment runner refuses with "older than source file …" — the staleness gate: rebuild as your user (cargo build --workspace && touch <binary>; cargo doesn't relink unchanged binaries, so mtime alone can look stale).
• insufficient privileges … CAP_NET_ADMIN — run the daemon under sudo or setcap it (see §3); provisioning refuses to half-apply without privileges.
• Bundle build: "vmlinuz not readable" — Ubuntu ships kernels 0600; sudo chmod 644 /boot/vmlinuz-$(uname -r).

9. Where to contribute next

Check docs/ROADMAP.md (program board) and RESULTS.md (what's done vs open). The phase-1 gate has passed (2026-07-13): H1, H3, and H4 all hold with second-campaign reproducibility checks, and v0.1.0 is tagged across the delivering repos. Current open fronts, in order:

1. vxedge — Open5GS/UERANSIM driver (mode: s-nssai) + URSP-style slice_policy → the H2 numbers (phase-2 gate)
2. vxparts (parts/hpc/) — Autoware/ROS2 + CARLA + carla-ros-bridge, closed loop
3. vxparts (parts/zone/) — QNX SDP 8 / AAOS images + real-RTOS guest bundles (Zephyr/FreeRTOS) beyond the linux-probe bundle
4. Track M — M0/M1: vxmodels repo + the retroactive H1–H4 SysML v2 traceability model over the frozen archives (docs/ROADMAP.md track M; zero new experiments)
5. Release/artifact pipeline — GHCR images with digests, cargo-dist binaries

Conventions to know before your first PR: manifests are the sole source of truth; no orchestration logic in any frontend; every image is digest-tagged; commits without a Co-Authored-By trailer. Details per repo in its AGENTS.md.

User guide

For SDV / automotive engineers using vxedge as a testbed. Onboarding lives in GETTING-STARTED.md; the full spec in ARCHITECTURE.md; host requirements in TESTBED-SETUP.md; live progress in RESULTS.md.

Honest state banner: vxedge is a research-grade executable virtual prototype of a zonal SDV E/E architecture — not a certified tool and not a "digital twin" of any physical vehicle (that word is reserved for hardware-in-the-loop parity, phase 4). What is verified today is listed per-verb below; everything else fails loudly rather than pretending.

1. Concepts

• A workspace is the unit of work — a folder, not a single file. An experiment is a directory holding a .vxp package descriptor (id, version, entry, requires) plus top-level YAML documents, each declaring its own kind — experiment (the test), vehicle (the system under test), road (the environment, optional), baseline (which software), component — and id. Every cross-reference is by id, never by path, so files can be renamed or reorganized without breaking anything; a whole workspace can also be one file of ----separated documents. The CLI/daemon keep no other configuration; every result archive records the workspace's SHA-256.
• A workspace is a sandbox. It owns its own scripts/run.sh (the campaign: repetitions, warmup, measurement window), parts/ (its loadables), assets/ (scenario files), tools/ (its own analysis), and results/ (one archive per run) — a folder that can be zipped, sent, unzipped and run standalone. vxedge experiment new <id> [--template minimal|slicing] scaffolds one; vxedge experiment validate runs every cross-file check and every model invariant before anything is provisioned. Full schema: ARCHITECTURE.md §6.0.
• Structure and software are separate files. One vehicle.yaml carries many baseline files, so an A/B arm is two baselines over one unchanged vehicle, not two vehicle files that drift apart. The vehicle names a slice (links[].slice: adas); the experiment defines it (network.slices[].mode: vlan|s-nssai) — the only reference that crosses the structure/test boundary, and experiment validate checks it.
• Slices are the network partitions under test: mode: vlan (per-slice VLAN + HTB queue-tree reservation on the virtual gateway — the H1 mechanism, verified) or mode: s-nssai (real 5G slices via Open5GS/UERANSIM — the H2 mechanism, verified).
• vxgateway is the software edge gateway the daemon provisions from the vehicle/experiment: a VLAN-filtering bridge (network.data_plane: kernel-bridge, default) or an OVS bridge (ovs) — same plan, pluggable rendering backend — a shaped trunk uplink (the shared bottleneck where slice contention is measured), and one access port per zone-node interface. A link's impairments block (rate/delay/jitter/loss, via netem) and declarative harness block (medium/length/weight, reporting-only) model the physical wiring between two nodes.
• Zones, nodes, and roles. Zones are the six regions (front-left, front-right, rear-left, rear-right, central-hpc, gateway); a node is a device with a role (zone-controller, hpc, gateway, ecu, sensor, actuator, switch) sitting in one. Each node runs under a packaging backend — container (network namespace), qemu-microvm (raw QEMU, direct kernel boot) or libvirt (full VM) — with identical network attachment; the packaging choice is itself an experimental variable (H3). ComputeService is role: hpc filtered over that same backend split, not a separate provisioning path.
• Jobs: every mutating command returns a job with live streamed progress; a job that couldn't finish reports FAILED with the real error, never a hollow success.
• Results archives (results/<exp>/<run>.zip) are the reproducibility unit: raw metric scrapes, structured samples, agent summaries, environment metadata, the workspace itself. v0.1.0 (2026-07-13) tags the checkpoint where this became true end to end — H1/H3/H4 all reproducible from a tagged manifest, not just runnable once.

2. CLI reference

All lifecycle verbs follow up|down[|status]; operations are idempotent.

Global flags: --endpoint (default local Unix socket), --json.

3. Typical workflows

a) Slicing sandbox (no measurement)

sudo -E env "PATH=$PATH" ./target/debug/vxedged --listen unix:///tmp/vx.sock --runtime-dir /tmp/vxrt &
vxedge --endpoint unix:///tmp/vx.sock up --manifest exp.yaml     # bridge + VLANs + HTB
vxedge --endpoint unix:///tmp/vx.sock status                     # per-slice observed state
vxedge --endpoint unix:///tmp/vx.sock zone up --vxid vxzone-front-left-01
vxedge --endpoint unix:///tmp/vx.sock zone list
vxedge --endpoint unix:///tmp/vx.sock down --manifest exp.yaml

b) A measurement campaign (the exp-001 pattern)

Campaigns are orchestrated by a runner script that encodes the methodology (warmup outside the probe window, ≥5 repetitions, environment metadata, archive per run):

cd vxperiments/experiments/exp-001-ws
sudo -E env "PATH=$PATH" ./scripts/run.sh --quick   # ~1 min pipeline validation -> ../../results/exp-001-quick/
sudo -E env "PATH=$PATH" ./scripts/run.sh --repro   # full rerun (~35 min, idle host) -> ../../results/exp-001-repro-<date>/
python3 ../../notebooks/analyze_exp001.py ../../results/exp-001   # archives -> table + verdict

A dir already holding a campaign's archives is never overwritten in place — full runs refuse it, repro checks and quick runs get their own dirs (scripts/lib/results_dir.sh).

Numbers from --quick runs are pipeline validation only and are marked NOT reportable. Reference result (2026-08-03 corrected campaign, superseding the original 2026-07-10 run — see RESULTS.md's Phase 1 row for why): ADAS slice p99 held at 0.91x baseline under 1.2x offered overload from the infotainment slice, zero loss (see RESULTS.md).

c) VM-backed zones (guest bundles)

VM backends boot a bundle (kernel + initramfs with an autostarting payload), built once:

sudo chmod 644 /boot/vmlinuz-$(uname -r)      # Ubuntu ships the kernel root-only
cd vxparts && ./parts/zone/linux/stack/probe-bundle/build.sh # -> ./bundle (digests in meta.json)

Point a node's image.url at the bundle dir with kind: rtimage and backend: qemu-microvm or backend: libvirt. Guest network comes from the manifest's ip: via the kernel cmdline; the payload (a vxprobe reflector) starts automatically.

4. Measurement agents & telemetry (standalone today)

• vxprobe reflect|send — RTT p50/p95/p99, jitter, loss; live Prometheus /metrics; JSON summary on stdout (stdout is machine-readable; logs go to stderr).
• vxtraffic — rate-accurate UDP load (--pps or --bandwidth-mbps).
• vxtelemetry ingest|collect — scrapes manifest targets, archives results/<exp>.zip with metadata (experiment id, manifest SHA-256, tool versions, ingest completeness).

5. Reading results

Every archive contains metadata.json (reproducibility fields), manifest.yaml, ingest/raw/**.prom (timestamped scrapes), ingest/samples.jsonl (structured samples), agent summaries, and run-env.json (kernel, RT flag, governor, pinning, timestamping method, load level, repetition). Analysis reads archives only; figures must regenerate headlessly. Report medians with IQR across repetitions; state sample counts next to tail percentiles; keep the recorded caveats attached (shared-host gateway, non-PREEMPT_RT kernel, userspace timestamps).

6. Mapping to production SDV vocabulary

7. Getting help / contributing

Each repo carries AGENTS.md (scope + rules) and TASKS.md (work items); docs/ROADMAP.md is the program board. Quality bars are codified as repo skills (.claude/skills/): rust-patterns, testbed-quality, experiment-methodology, gateway-verification (vxedge), sysml-authoring (vxmodels). A verb that prints "not wired to the daemon yet (scaffold)" is truthfully unimplemented — check RESULTS.md before assuming a bug.

Results

What vxedge has measured so far, one row per hypothesis, with the caveat that matters most for each. Hypotheses are defined in ARCHITECTURE.md §2; methodology in §12. Every number below comes from a frozen summary in vxperiments/results/, which records the input manifest's sha256 alongside it. Each one is registered in FACTS.md, which also holds the full per-load and per-arm tables. This page explains the numbers; that file is where they are quoted from.

Summary

Every campaign above is 5 repetitions per arm or load point, 1000 pps × 60 s per point (60 000 probes), with a 10 s warmup discarded. Each result has been reproduced by an independent second campaign (--repro) that reached the same verdict.

H1 — isolation (exp-001)

The ADAS tail does not grow with load: p99 is 89.6 µs idle and 81.9 µs at 600 mbit, and loss is 0.0 % at every level. Full table: FACTS.md § H1.

Caveat: the gateway shares its host with the zones it serves, and timestamps are taken in userspace (SO_TIMESTAMPING is a tracked follow-up). An earlier version of this campaign measured loopback delivery instead of the gateway path, because probe and reflector sat in the same network namespace. That defect was fixed with real netns isolation and is now covered by a CI regression test. The numbers above are the corrected re-run. Summary

H2 — real slicing vs static mapping (exp-010)

S-NSSAI PDU sessions through Open5GS + UERANSIM, compared against exp-001's VLAN arm. The S-NSSAI path has a higher absolute floor (p99 ≈ 317–337 µs, since it crosses a software UPF and GTP-U), but its tail grows no faster under load than the VLAN path does, which is what the hypothesis tests.

Caveat: the first campaign did not hold, because nothing in the 5G path applied per-slice QoS. The fix was real HTB enforcement on the slice path. We checked first whether 5QI/AMBR configuration alone would do it, and it did not change anything. The campaign was then re-run with host-noise mitigations (CPU governor, core pinning, randomized load order). All intermediate campaigns are kept in the results directory. Summary

H3 — virtualization choice (exp-002b)

Six packaging arms, from container to microvm-mmio, with p50/p99, CPU and RSS per arm: FACTS.md § H3.

Decomposition: virtio transport (MMIO vs PCI) −26.0 µs; libvirt vs raw QEMU −17.3 µs, likely because libvirt enables vhost/optimized virtio settings that a bare QEMU invocation does not; vhost-net alone −9.5 µs. Machine type (q35 vs microvm) has no measurable effect (−0.3 µs, within IQR).

Caveat: exp-002's original conclusion ("microVMs are slowest") was right about the ranking but wrong about the cause. exp-002b was designed specifically to separate the factors. Summary

H4 — OTA co-location (exp-003)

Both arms, with p50/p95/p99 and OTA deliveries (361 ok, 0 failed per repetition): FACTS.md § H4.

Caveat: the v1 OTA transport binds a plain host address. That means this measures host-level CPU and scheduling contention between OTA and the ADAS probe, not gateway-QoS-enforced network contention. A second campaign moved the absolute numbers outside the first campaign's IQR, which is session-to-session host drift, but it did not change the comparative verdict. Summary

Other completed experiments

• exp-005 — cold-start / first-packet latency after zone boot. The effect was confirmed in 20 of 20 repetitions. It explains the inflated first-arm tails seen incidentally in exp-001/002/003.
• exp-014 — wiring-harness topology as a variable: star vs daisy-chained zones. The expected hop-count effect holds.

Methodology caveats that apply to everything above

These are properties of the host class used for phase 1–2, not of one machine:

• Shared-host gateway: vxgateway runs on the same host as the zones.
• Non-PREEMPT_RT kernel, performance CPU governor, measurement cores pinned.
• Userspace monotonic timestamps.

Host requirements per experiment tier are in TESTBED-SETUP.md.

Facts registry

This file is the single registry of published facts: measured numbers, verdicts, phase states, the parity ledger and what has been delivered. The website (vxedge.dev) builds every number and state it shows from this file and nothing else. The GitHub READMEs and RESULTS.md quote it.

Each measured value is copied from a frozen summary in vxperiments/results/. Those summaries record the input manifest's sha256. This file does not replace them; it is the one place they are quoted from. RESULTS.md explains the numbers: what each hypothesis tests and the caveat that matters most.

To change a fact: edit it here first, in the same change as the summary or state that moved. Then run python3 scripts/facts.py check from this repo. The website picks the change up on its next build. Its CI fails if the vendored copy is stale, or if a page shows a number that is not registered here.

Format. Every table whose first column is Key is machine-read by scripts/facts.py. A table with a Value column holds one fact per row. Any other keyed table is a row table: each cell becomes <key>.<column>, where the column name is the first word of its header, lowercased (p99 µs (IQR) → p99). Keys are lowercase and never change meaning once published. A fact that is retired is deleted, not renamed.

State words are fixed. For deliveries: measured (frozen, reproduced results), live (works today, no frozen number), building (partly landed), planned (specified, not built). For the parity ledger: real (the production software or kernel path itself), emulated (real software on modelled hardware), planned, caveat (works, with a stated limit on what it can tell you).

1 · Project

Phases

Tracks

2 · Results

The tables under each hypothesis heading (H1–H4) and under Other completed experiments are measurements. The website paints those values, and only those, in its measured-value colour. The method, parameters and verdicts are not measurements and are not painted.

Method and parameters

The same method for every campaign, and the parameters each hypothesis was tested with. These are configuration, not measurements.

Verdicts

H1 — isolation (exp-001)

Per load level. Latency in µs with the IQR across repetitions in brackets. Source: exp-001-summary.md.

H2 — real slicing vs static mapping (exp-010)

Per load level, S-NSSAI path. Source: exp-010-summary.md.

H3 — virtualization choice (exp-002b)

Per packaging arm. CPU and RSS are means across repetitions. Source: exp-002b-summary.md.

H4 — OTA co-location (exp-003)

Per arm. Source: exp-003-summary.md.

Other completed experiments

3 · Deliveries

Parity ledger

How close each part of the virtual vehicle is to production today, and what closes the gap (p3, p4: the phase that does).

Use cases

Agent path

How a generated experiment reaches the bench.

vxmcp tools

Other deliveries

Roadmap

Where vxedge stands, phase by phase. The full phase definitions are in ARCHITECTURE.md §15, and the measured results are in RESULTS.md. Per-repo work items live in each repository's own TASKS.md.

References elsewhere in this repository to numbered program-board items (#14, #28, …) and dated state snapshots point to the maintainers' internal working board. This page is the public summary of that board.

Phase status

The phase, track and delivery states below are registered in FACTS.md; change them there first.

Cross-cutting tracks:

• M — SysML v2 model governance (vxmodels): M0–M3 are closed. M4 (scenario models, fidelity ledger) is waiting on a GPU host for CARLA.
• C — containerization: complete through C3.2. The 5G stack and platform services run in containers, and the measured path stays on the host.
• P — vxparts unification: complete. The earlier zone and compute repositories are merged into the unified OCI/ORAS loadable registry.

Open work

Phase 2 carry-overs

• exp-004, exp-012 and exp-020 are each blocked on a real prerequisite: a compute-timing agent, the TSN gateway variant, and CARLA respectively.
• vxagents: DSCP marking, clock-sync probing, and PCAP helpers.
• SO_TIMESTAMPING (kernel timestamps) in place of userspace-monotonic probing.

Phase 3

• vxedge vehicle up end to end (the gate).
• exp-011: real ADAS traffic, CARLA → ROS 2 → Autoware closed loop. It needs a Tier 2 GPU host (TESTBED-SETUP.md).
• QNX SDP 8 and Android Automotive zones, and an embedded Qt/QML IVI loadable.
• vxmcp: the remaining read tools, and mutating tools gated by confirmation.
• On-demand reproduction pipeline for published experiments.

Release engineering

• A tag-triggered GitHub Release per repository.
• cargo-dist binaries for vxedge/vxedged.
• GHCR images with recorded digests.
• A cross-repository compatibility matrix.

Publication

• Paper #1 (the phase-1 dataset: H1/H3/H4) is in preparation for arXiv.
• A citable, archived reproducibility bundle (Zenodo DOI) will accompany it.

Contributing to the roadmap

Open an issue on the repository the work belongs to. For anything that spans repositories, open it in vxedge/docs.

Architecture

The master specification for vxedge: research goals, topology, vxAPI, manifest schemas, the network, service and virtualization models, methodology, and the phased roadmap.

vxedge is a research-grade orchestration and experiment platform that evaluates 5G network-slice mapping onto zonal and central software-defined-vehicle (SDV) E/E architectures, entirely in software. In short:

• Gateway. vxgateway, built on the Linux networking stack, replaces physical edge gear.
• Zones. KVM/QEMU VMs managed through libvirt provide zonal ECUs (QNX, Zephyr, FreeRTOS, Android Automotive).
• Compute. Containerized central compute runs Autoware/ROS 2 and NVIDIA AD workloads (Alpamayo, DeepStream, TensorRT).
• Control plane. A Rust daemon (vxedged) orchestrates manifest-driven vehicles, experiments, services and telemetry, and exposes a versioned gRPC/Protobuf API (vxAPI).
• Frontends. Thin clients of that API: the vxedge CLI today; web/TypeScript and Electron frontends next; and an MCP server (vxmcp) for AI agent tooling. Locally over a Unix socket, or cloud-hosted behind mTLS.

Qt/QML is not an operator-dashboard frontend here; see §5.1 for its two real, unrelated roles: an in-vehicle embedded IVI loadable, and a future vxmodels graphical design tool.

Table of contents

1. Executive summary
2. Research goals & hypotheses
3. High-level architecture (topology & components)
4. Repositories & single-source rules
5. vxedge control plane & API (vxedged, vxAPI, frontends, CLI)
6. Manifests (experiment & vehicle schemas)
7. Network model — vxgateway (SDN slicing)
8. Service model (OTA, telemetry, extensibility)
9. Virtualization & zone provisioning (libvirt, QEMU/KVM, QNX, RTOS, containers)
10. Compute model (Autoware/ROS2, Alpamayo, inference, OTA hosting)
11. Edge agents & telemetry collection (vxagents / vxtelemetry)
12. Experiment methodology, metrics, and analysis plan
13. Reproducibility, CI/CD, artifact publication
14. Safety, security, licensing, and industrialization notes
15. Implementation policy & phased roadmap
16. Appendix: example manifests, gateway snippets, run sequence

1 — Executive summary

vxedge is a reproducible, modular framework to define and run SDV experiments that combine zone virtual ECUs, central compute, a software-defined virtual edge gateway with 5G-style slicing, and services (OTA, telemetry). Manifests declare the whole system — up to and including a complete software-bootstrapped vehicle (vxedge vehicle up). The Rust CLI orchestrates the lifecycle using a simple up|down model across all subsystems.

The platform is designed to:

• generate publishable results (latency/jitter percentiles, OTA reliability, isolation metrics);
• stay production-representative: the same middleware (Eclipse SDV: Kuksa/VSS, Zenoh, uProtocol), network stack (OVS/tc/FRR/WireGuard, SONiC as production NOS twin), and AD workloads (Autoware, NVIDIA Alpamayo) used or converged on by OEMs;
• run on a single NVIDIA GPU host (scaling to several), with an explicit later path onto NVIDIA hardware (Jetson AGX, DRIVE AGX / DRIVE OS, BlueField/DOCA offload);
• be API-first: all orchestration lives behind vxAPI (gRPC/Protobuf) served by the vxedged daemon. The CLI, GUIs (Electron), web dashboards, and the vxmcp MCP server are interchangeable thin clients — human-operated or agent-operated, same rule either way — and the same API hosts locally or in the cloud (controller mode federating multiple hosts).

No physical router is required: the edge-gateway role is fully absorbed by vxgateway.

2 — Research goals & hypotheses

Primary goal: Quantify whether a software-defined virtual gateway — built from the production Linux SDN stack (OVS/Linux bridge, tc queue trees, VXLAN, WireGuard, FRR) and running co-located with virtualized zonal ECUs and central compute — can provide slice isolation and deterministic performance for SDV workloads mapped to logical 5G slices.

Hypotheses to test:

• H1 (isolation): Per-slice VLAN/VXLAN mapping with hierarchical tc queue-tree reservations in vxgateway can keep p95/p99 latency for ADAS (URLLC-like) traffic near baseline under heavy infotainment load.
• H2 (real slicing vs. static mapping): True 5G slicing (S-NSSAI PDU sessions via Open5GS + UERANSIM) provides measurably different — and under contention, stronger — isolation than static VLAN-to-slice mapping.
• H3 (virtualization choice): Virtual ECU packaging (full VM via libvirt vs. container vs. microVM via raw QEMU/QMP or cloud-hypervisor) measurably affects latency/jitter observable at zone-to-compute flows.
• H4 (service co-location): OTA service co-located in central compute (deployed on demand) does not compromise ADAS slice SLAs when slices and queue reservations are correctly configured.

Deliverables: Reproducible experiment artifacts, datasets, scripts, and publishable analysis — plus a reference "vehicle-in-software" composition demonstrating the full stack end to end.

Framing note: unlike physical edge gear, the virtual gateway shares the host with the workloads it isolates. Determinism controls (PREEMPT_RT kernel, isolcpus, CPU pinning, hugepages) are part of the method, and their effect is itself reported.

3 — High-level architecture (topology & components)

Figure 3.1: the topology (click for full size). Box states follow the parity ledger in FACTS.md: solid is built today (real, emulated or building), dashed is specified and not built yet. Source: diagrams/topology.py.

Key components:

• vxedged (Rust) — control-plane daemon: orchestration + provisioning core behind vxAPI (gRPC/Protobuf). See §5.
• vxedge CLI (Rust) — thin vxAPI client; the primary frontend.
• vxgateway — the software-defined edge gateway: Linux-native data plane (OVS or Linux bridge, tc, VXLAN, WireGuard, FRR), provisioned declaratively by the CLI. See §7.
• vxzones — authoritative zone manifests, VM images (QNX SDP 8, Zephyr, FreeRTOS, AAOS), RTOS builds.
• vxcompute — central compute manifests, Autoware/ROS2 images, Alpamayo/AlpaSim, inference.
• vxota — OTA server & signing tooling (artifact store).
• vxagents — performant Rust agents for measurements and metrics.
• vxtelemetry — telemetry ingestion & Prometheus export (VSS-aligned; see §11).
• vxperiments — manifest-driven experiments, vehicle manifests, and results.

4 — Repositories & single-source rules

Canonical repos (under vxedge org):

• vxedge/vxedge — Rust workspace: vxedge-core (domain lib), vxedged (daemon, vxAPI), vxedge-cli (thin client), proto/ (vxAPI Protobuf IDL — the source for all generated SDKs: Rust/tonic, TypeScript/connect-es), plus gateway provisioning templates (netlink/OVS/nftables/tc, libvirt domain XML templates).
• vxedge/vxparts — the unified loadable registry: zone VM/kernel/RTOS bundles, central-compute container images, gateway/TCU images, agent binaries, and platform-service images — every loadable the platform boots, published to ghcr.io/vxedge/parts/... via OCI/ORAS, classified by class/subtype/layer (see §13's own artifact-publication rules). Supersedes vxzones and vxcompute (Track P, decided 2026-07-26): those two repos split loadables by deployment mechanism (VM vs. container) rather than by domain, which is exactly the kind of implementation detail this workspace's own single-source-of-truth discipline exists to keep out of repo topology. Both were archived 2026-08-04 (GitHub archive, not deleted — their history stays reachable, read-only) once their content was confirmed fully present in vxparts — see vxparts/TASKS.md Track P for the migration account.
• vxedge/vxota — OTA server, signing, manifests.
• vxedge/vxagents — Rust agents, metrics exporters.
• vxedge/vxtelemetry — telemetry ingestion and exporters.
• vxedge/vxperiments — experiment manifests, vehicle manifests, notebooks, results.
• vxedge/vxmodels — (track M) SysML v2 system models: requirements/traceability (H1–H4 verify-linked to frozen result archives), vehicle structure models, the analysis-case/variant library, and the vxmodel renderer (Rust) that generates manifests from models. See §6.3 and §15 (track M).
• vxedge/docs — papers, slides, reproducibility guide.

Single-source rules:

• A zone/compute's manifest is authoritative for its images and node IDs.
• Services declared under services in an experiment manifest are the authoritative runtime descriptors for that experiment (image + config). The code lives in vxota or vxtelemetry as appropriate.
• A vehicle manifest is the authoritative composition of zones + compute + network + services for a software-bootstrapped vehicle; experiments reference a vehicle rather than restating it.
• CLI uses manifests as sole input; state stored in ephemeral runtime directories and vxedge logs.
• Exactly two cross-language interfaces exist: manifests (YAML content) and vxAPI (Protobuf IDL under vxedge/vxedge/proto/, authoritative; client SDKs are generated, never hand-written). No bespoke ad-hoc RPC.
• Model governance (track M, when vxmodels lands): the SysML v2 model is the design authority upstream of manifests; manifests remain the sole execution contract — the daemon/CLI never read models. A model-rendered manifest records the digest of the model version that produced it and must pass vxedge manifest validate like any hand-written manifest. The model never duplicates what zone/compute manifests are authoritative for (images, node IDs). See §6.3.

5 — vxedge control plane & API (vxedged, vxAPI, frontends, CLI)

Principle (API-first): the split every production SDV toolchain converges on — Tesla-style fleet tooling, NVIDIA DriveWorks tools, AUTOSAR Adaptive development tooling — is a headless core behind a strongly-typed IDL, with every UI a disposable thin client. vxedge adopts it outright: no orchestration logic lives in any frontend.

5.1 Layers & frontends

Figure 5.1: layers and frontends (click for full size). Box states follow the parity ledger in FACTS.md. Source: diagrams/layers.py.

• vxedge-core (Rust lib): pure domain logic; no user-facing I/O.
• vxedged (Rust daemon, tonic + tonic-web): hosts vxAPI. Every mutating verb is an async job with server-streamed progress; the reconciler compares desired (manifest) vs observed state — that is what makes up|down idempotent.
• Frontends (all thin clients of the same API): platform administration, live status monitoring, and experiment operation — for two kinds of operator: OEM engineers and SDV researchers directly, and (via vxmcp) AI agent tooling acting on their behalf. None of these are the in-vehicle HMI — that role belongs to Android Automotive (AAOS), already the infotainment zone's guest OS, or an embedded Qt/QML IVI loadable (running under a QEMU ARM zone node or hosted atop QNX — the production pattern several OEMs actually run; both are real, alternate in-vehicle-HMI baselines, §9). Frontends here are external tooling that operates the testbed, not something running inside a simulated vehicle. Correction (2026-08-06): there is no Qt/QML operator dashboard — any platform monitoring dashboard is web-based (TS) or the CLI, never Qt. A separate, future, not-yet-built Qt/QML tool is planned for vxmodels' MBSE tooling (§6.3) — a graphical experiment designer, not an operator dashboard and not a vxAPI frontend the way the items below are.
  • vxedge CLI (Rust) — unchanged UX; local Unix socket by default, --endpoint for remote; auto-spawns an ephemeral local daemon for single-host lab use.
  • Web dashboard / Electron (TypeScript; connect-es-generated SDK over gRPC-Web) — cloud console first, "vxedge Studio" desktop app later.
  • vxedge-tray (Rust, StatusNotifierItem/SNI system tray) — at-a-glance gateway/experiment state + quick up/down from the desktop; pairs with a systemd unit for vxedged (socket-activated where possible). Same thin-client rule: 1:1 vxAPI calls, no logic.
  • vxmcp (Rust, rmcp — the official MCP Rust SDK) — an MCP server exposing vxAPI to AI agent tooling (Claude Code and others), for the same reason every frontend above exists: a thin, 1:1 client over the same RPCs, this time addressed by an LLM agent's tool-call surface instead of a terminal or a GUI. Every MCP tool is a direct wrapper over one vxAPI RPC (GatewayService/ZoneService/ComputeService/ExperimentService/TrafficService/ SvcService/JobService/EventService) — never a multi-step workflow baked into the server, the hard rule stated explicitly for this frontend too. Sits directly on vxedge-api's generated tonic bindings (a Rust-to-Rust dependency, not a second codegen pipeline the way the TS/C++ SDKs are). Targets MCP spec 2026-07-28: stateless request-response (no initialize handshake), with Multi Round-Trip Requests (InputRequiredResult/InputRequest::Elicitation) as the mechanism for the one thing a thin wrapper still needs — a mutating tool (up/down/traffic up) asking its caller to confirm before it runs, the same posture a human operator gets from the CLI's own explicit verbs, not something an LLM can chain unsupervised into a destructive sequence. Read tools (status/list/events subscribe/results-archive summaries) ship first, since they carry no such confirmation question and are useful standalone; mutating tools follow once the MRTR confirmation flow is built and reviewed. New repo, vxedge/vxmcp, phase 3 (docs/ROADMAP.md) — frontend/tooling investment, not hardware-gated, so it doesn't wait on anything phase 3's GPU items do.

5.2 API surface (vxAPI, proto/vxedge/v1/)

Rules: mutating RPCs carry manifest documents (or references to them) — the API introduces no second source of truth beside manifests; daemon state stays in ephemeral runtime dirs + logs. The proto package is versioned (vxedge.v1): additive changes stay in v1, breaking changes mean vxedge.v2.

Why gRPC/Protobuf (not SOME/IP, not bare REST): SOME/IP–ara::com is the in-vehicle service plane — the wrong tool for an orchestration/tooling plane; bare REST lacks first-class streaming and codegen rigor. One Protobuf IDL generates every SDK, gRPC-Web covers browsers and Electron, and Protobuf is what Eclipse uProtocol already speaks — schema reuse when the signal plane lands in phase 3. An optional REST/OpenAPI façade can be added later by transcoding, without touching the core.

5.3 Transport, security, deployment modes

• Local (default, live today): Unix domain socket only — vxedged --listen rejects any other scheme outright, and vxedge --endpoint rejects any non-unix:// value the same way (found while scoping Phase 5, 2026-08-05: neither side has ever accepted anything else). The access boundary today is filesystem permissions on the socket path, not a per-call identity check — no SO_PEERCRED/UCred peer-credential extraction exists anywhere in the code yet. "Peer-credential auth" below is the target for this mode, not its current behavior; stated honestly here per this doc's own fidelity rule (§14 / testbed-quality) rather than implying it already gates local connections.
• Remote (planned, Phase 5): TCP + mTLS, OIDC bearer tokens; roles viewer/operator/admin; every mutating call audit-logged with the calling actor's identity attached — a second gap found during the same scoping pass: Audit::record(rpc, detail) (vxedged/src/audit.rs) has no actor field at all today, because there has only ever been one, local, unauthenticated caller. Remote access without that field would produce an audit log that says what happened but not who did it — not acceptable once "who" is a real question. Full scope: docs/ROADMAP.md's Phase 5 section.
• Cloud-hosted (planned, Phase 3): vxedged --controller runs anywhere and federates one or more site daemons (one per GPU host, reached over WireGuard + mTLS) behind the same vxAPI. Frontends never know whether they talk to a single host or a fleet controller, and never touch hypervisors directly. A related but distinct concern from Remote access above: federation is site-daemon-to-controller plumbing, not the human/CLI-to-daemon auth path — see docs/ROADMAP.md's Phase 5 section for why the two are scoped separately.

5.4 CLI — commands & semantics

Location: vxedge/vxedge (Rust workspace: vxedge-core, vxedged, vxedge-cli, proto/). Binaries: vxedge (thin client), vxedged (daemon). Global behavior: all lifecycle commands follow up|down semantics. Where staging is required, up performs staging + start; down stops + cleanup. All operations are idempotent.

Commands (canonical)

# Daemon / control plane
vxedged serve [--listen unix:///run/vxedge.sock | https://0.0.0.0:8443] [--controller]

# Client endpoint (applies to every command below; defaults to the local Unix socket)
vxedge [--endpoint <url>] ...

# Gateway (vxgateway — virtual edge gateway)
vxedge up|down|status

# Zones
vxedge zone up --vxid <vxid>
vxedge zone down --vxid <vxid>
vxedge zone list
vxedge zone add --manifest vxzones/...yaml
vxedge zone rm --vxid <vxid>
vxedge zone image push --vxid <vxid> --from <path> --registry <url?>

# Compute
vxedge compute up --vxid <vxid>
vxedge compute down --vxid <vxid>
vxedge compute add --manifest vxcompute/...yaml
vxedge compute rm --vxid <vxid>
vxedge compute image push --vxid <vxid> --from <path> --registry <url?>

# Services (OTA, telemetry)
vxedge ota up [--experiment <path>]      # cwd-resolved by default, like `experiment up`;
vxedge ota down [--experiment <path>]    # --experiment overrides with an explicit workspace
vxedge telemetry up|down|collect         # not yet wired to the daemon (planned);
                                          # vxtelemetry runs standalone via its own binary today

# Vehicle (software-bootstrapped SDV: gateway + zones + compute + signal plane + services)
vxedge vehicle up --manifest vxperiments/vehicles/<vehicle>.yaml
vxedge vehicle down --manifest vxperiments/vehicles/<vehicle>.yaml
vxedge vehicle status --manifest vxperiments/vehicles/<vehicle>.yaml

# Experiments (manifest-driven; may reference a vehicle manifest)
vxedge experiment up --manifest vxperiments/exp-001.yaml
vxedge experiment down --manifest vxperiments/exp-001.yaml

# Traffic generation
vxedge traffic up --manifest ...

vxedge vehicle up is sugar for the ordered composition: gateway up → zones up → compute up → signal plane up (Kuksa, phase 3) → services up. vehicle down tears down in reverse.

Every CLI verb is a 1:1 wrapper over a vxAPI RPC — anything the CLI can do, every other frontend (web, Electron, scripts against the TS SDK) can do identically.

6 — Manifests — final schemas (YAML)

6.0 The workspace model (Track D)

An experiment is a folder. Every file declares its kind and id; the loader indexes on that pair, and every cross-reference is by id, never by path — so files can be renamed or reorganised without breaking anything.

h1-slice-isolation/
  h1-slice-isolation.vxp   # package descriptor: id, version, entry, requires
  main.yaml                # kind: experiment  — the test
  vehicle.yaml             # kind: vehicle     — the system under test
  road1.yaml               # kind: road        — the environment (omit if unused)
  parts.yaml               # kind: baseline    — which software
  scripts/run.sh           # the campaign: repetitions, warmup, measurement window
  parts/                   # this experiment's own loadables
  assets/                  # scenario files, traces, maps
  tools/                   # analysis specific to this experiment
  results/                 # archives, one per run

A workspace is a sandbox. It owns everything the experiment needs, so it can be zipped, sent, unzipped and run — which is what makes an experiment shareable at all. A workspace whose parts live three directories up is not something a recipient can unpack. Only the top-level *.yaml are workspace documents; the subdirectories are content, and the loader ignores them (a part.yaml inside parts/ describes a loadable, not an experiment).

Kinds: experiment | vehicle | road | baseline | component. Loading is: read the .vxp, glob *.yaml (non-recursive, so a results/ archive is never absorbed), index by (kind, id), resolve references by id. No includes, no import boilerplate, no ordering rules. Files exist only when the experiment needs them.

A file may hold several documents, ----separated, and a whole workspace may be one file. That follows from every reference being by id: the file boundary carries no meaning, so it is free to move. Separate files stay the scaffold default because they teach the model, but an engineer who wants one file — or a caller with no directory to point at, such as an inline manifest over vxAPI — has one.

Structure and software are separate files. One vehicle.yaml with many baselines is what an OTA campaign is, what an RxSWIN identifies, and what makes an A/B arm two baselines rather than two vehicle files that drift apart.

The vehicle names a slice; the experiment defines it. links[].slice: adas in the vehicle, network.slices[].mode: vlan|s-nssai in the experiment. H2's comparison is therefore one block changed over a provably unchanged vehicle, enforced by the schema rather than asserted in a README. This is the only reference that crosses the structure/test boundary, and vxedge experiment validate checks it.

network: (and everything under it — slices, tunnels, data_plane, nos_check) lives in the experiment document only, never inside a vehicle document, for a second reason beyond H2: there is exactly one vxgateway per experiment run (one bridge, one HTB tree — a deliberate fidelity choice, one vehicle has one gateway ECU, not an oversight). When an experiment references multiple vehicles (vehicles: [car1, car2]), they share that one gateway, so they share one network: block too — a per-vehicle network: would imply each vehicle gets its own separate gateway, which the system doesn't build. Vehicles that need distinct, isolated segments still get them: the experiment declares every slice either vehicle needs (car1-adas/car2-adas on separate VLANs, say), and each vehicle's own links references only the slice IDs that belong to it — car2.yaml never has to mention car1-adas at all.

Local-first. Every ref has a local form (./images/foo) and a published form (ghcr.io/… + digest), same key, resolved by shape. digest is optional for local refs (content-hashed into the results archive) and required for published ones — a moving tag is not reproducible for anyone else. So a run is always reproducible for its author, and publishing is what makes it reproducible for others. vxedge experiment up must work offline with zero published artifacts.

Tooling: vxedge experiment new <id> [--template minimal|slicing] scaffolds a workspace; vxedge experiment validate [path] runs every cross-file check and reports all problems at once.

6.0.1 kind: vehicle

kind: vehicle
id: car1
class: road                          # taxonomy spine: kind > class > subtype
subtype: car

zones:                               # REGIONS of the entity, not devices
  front-left: {}
  central: { name: central compute bay, reference: <external zoning std> }

devices:                             # units LOCATED IN zones
  zonefl: { role: zone-controller, zone: front-left, backend: container }
  hpc1:   { role: hpc, zone: central, backend: container, resources: { cpu: 4, memory: 8 } }
  gw1:    { role: gateway, backend: container }

functions:                           # units of BEHAVIOUR, deployed onto devices
  adas:    { deployed_on: zonefl, asil: B }
  percept: { deployed_on: hpc1 }

links:                               # transport: how bytes move (the harness)
  - { id: trunk-fl,  endpoints: [gw1:eth0, zonefl:eth0], slice: adas }
  - id: arm-cont                     # object form, when an attachment needs a fixed address
    endpoints: [{ node: zonefl, port: eth0, ip: 10.101.0.11/24 }]
    slice: adas
  - id: seg-t1                       # a slice-bound link may emulate its PHY
    endpoints: [node-sensor-01:eth0]
    slice: adas
    impairments: { rate_mbps: 100, delay_us: 250, jitter_us: 40, loss_pct: 0.01 }
    harness: { medium: 100BASE-T1, length_m: 2.4, weight_g: 85 }

channels:                            # interaction: what is exchanged, between FUNCTIONS
  - id: adas.object-list
    from: percept
    to:   [adas]
    protocol: dds | zenoh | someip | udp
    qos: { reliability: reliable, deadline_ms: 50 }
    carried_by: [trunk-hpc, trunk-fl]

Zone, device, function are three different things. A zone is a place — a named region (front-left, cabin, nacelle-2, an ATA zone), not an enum, which is what lets a non-road entity have zones at all. A device is a unit located in one. A function is behaviour deployed onto a device: AUTOSAR calls it a software component, and separating it from the ECU it runs on is the whole point of the Virtual Function Bus. parts: targets functions, and channels connect functions — a topic flows between behaviours, and which devices they sit on is a deployment decision a study should be able to vary without rewriting the interaction topology.

role is zone-controller | hpc | gateway | ecu | sensor | actuator | switch. backend is container | qemu-microvm | libvirt and is required for any device attached to a slice: the planner has to know whether to build a veth or a tap, and guessing produced a workspace that validated and then could not boot.

Addressing is <id>[:<port>]. Ids are flat and unique workspace-wide (zonefl, not zones.front-left): structural paths break when the model is restructured, and flat ids do not. Uniqueness is an enforced check, and so is the zone/device collision it makes possible.

A link endpoint takes two forms. The bare zonefl:eth0 is right whenever nothing has to be reachable at a known address, which is most attachments. The object form { node, port, ip } pins a host address on that attachment, for the case with no other expression: exp-002's three arms share one VLAN and are told apart only by address. An address belongs to the attachment rather than the device — a device on two slices has two — and it lives in the vehicle rather than the experiment because it states a structural fact ("this ECU answers here"), which is what keeps one vehicle usable by many baselines.

carried_by binds a channel to the links carrying it, and validation requires the channel's endpoints to be genuinely reachable over those links. This is what makes latency and loss attributable per topic per slice rather than only per link — the measurement §12 exists to produce. Without the reachability check, carried_by would be an unchecked assertion.

A slice-bound link may add impairments (tc netem: rate_mbps/delay_us/jitter_us/ loss_pct) to emulate the PHY it stands for — the mechanism behind §7's "Harness modeling." Every value is per traversal, one direction: a round trip pays the declared delay twice. Exactly one of the link's access ports renders it, in both directions (the reverse direction rides an IFB, since a qdisc only shapes egress); which port is not observable in a measurement, only that the segment's declared characteristics apply once per crossing. Any link, impaired or not, may separately carry harness (medium, length_m, weight_g) — the declarative record a zonal-versus-domain harness-reduction study reports on, distinct from impairments' emulated half: nothing about a length in meters is a tc argument.

Behaviour splits in two: interaction (channels) belongs to the vehicle; stimulus (traffic_profiles, scenario) belongs to the experiment and the road. The vehicle is the same vehicle whether it is driven hard or gently.

6.0.2 kind: baseline

kind: baseline
id: bl-2026-07
rxswin: RX-VXE-0001-0003             # optional; UN R156, a property of the vehicle TYPE

parts:
  - ref: ghcr.io/vxedge/parts/zone/qnx/os/sdp8-base:1.0.0
    digest: sha256:…                 # required for published refs
    target_ids: [zonefl, zonefr]     # scalar or list — declared once, however many nodes
  - ref: ./images/my-autoware        # local; no digest required
    target_ids: hpc1

Each entry carries ref, digest, target_ids, and — only where the reference is silent about it — kind: container | bundle. Class, subtype, layer, name and the VXPN part number are all derivable from the ref path; restating them is the drift the computed-part-number design exists to prevent.

kind exists because a part knows what it is and a device knows how it is run, and the two have to agree: a container backend boots a container image, libvirt and qemu-microvm boot a guest bundle. The local/… namespace is conclusive on its own (a host process, never a bundle), and a published vxparts ref is deliberately not read for this — its class says what kind of node the part loads onto, not how it is packaged, and a zone guest is legitimately a container under the netns backend. Only a bare path is genuinely ambiguous, and there kind is how the author says so. A stated kind is validated against the device's backend, so a baseline pointing an H3 libvirt arm at a container image is rejected with both halves named rather than silently accepted and failed at boot. vxedge manifest bom derives them for display, SBOM export and the RxSWIN roll-up.

target_ids declared once for many components is deliberate: repeating a shared part per-component is how fleet baselines silently drift when one copy is bumped and the others are not.

6.0.3 kind: experiment and the .vxp descriptor

kind: experiment carries network (slices, tunnels, nos_check), traffic_profiles, telemetry, services and results, plus vehicles: [<id>…], baseline: <id>, world: <id>, and a method block (duration_s, repetitions, warmup_s). It carries no zones/compute inline: structure is the vehicle's, and software is the baseline's.

Every one of those blocks is typed and validated when the model is built — a malformed services: entry is one error against the file the author wrote, not a surprise at svc up.

network.tunnels (docs/ROADMAP.md #16) declares a point-to-point link the gateway itself terminates, reaching a remote peer (another site's gateway) — distinct from network.slices, which partitions the local bridge fabric zone nodes attach to. Two kinds today, each with only the fields it needs:

network:
  tunnels:
    - id: ota-uplink
      kind: wireguard                  # an encrypted, routed L3 endpoint
      listen_port: 51820
      address: 10.200.0.1/24           # this side's address on the tunnel's own overlay subnet
      peer:
        public_key: <base64>           # the peer's key, known out of band — never this side's own
        endpoint: 198.51.100.7:51820   # omit if this side only listens (WireGuard roaming)
        allowed_ips: [10.200.0.0/24]
    - id: site-b-adas
      kind: vxlan                      # extends a local slice's L2 domain across hosts
      vni: 100                         # 24-bit VXLAN network identifier
      remote: 198.51.100.7             # the remote gateway's reachable address (bare IP)
      dstport: 4789                    # optional; defaults to the IANA-assigned VXLAN port
      slices: [adas]                   # which of this experiment's own slices this link carries

wireguard carries no local key material, deliberately: a manifest is a versioned document, and a private key does not belong in one. The gateway mints its own identity on first up — piped straight from wg genkey into wg set's stdin, never written to disk — and vxedge status reports the resulting public key back, which is what an operator hands to the peer out of band. Re-applying an already-existing tunnel reconfigures its peer/listen-port declaratively without touching that identity, so a routine reconcile pass can never silently invalidate what the peer already trusts.

vxlan has no key material at all — an unencrypted L2 extension, the same trust model as a physical inter-switch link — and is structurally the opposite of wireguard: a WireGuard interface is a routed L3 endpoint, never enslaved to the bridge/OVS switch, while a VXLAN interface joins the bridge as a trunk-like port, so slices (not an address/peer block) is what a vxlan tunnel actually needs — which local slice(s) it extends, validated against network.slices the same way a links[].slice reference is. Model and validation only (docs/ROADMAP.md #16, delivered 2026-08-04) — GatewayPlan refuses to plan a kind: vxlan tunnel today (PlanError::VxlanTunnelNotYetProvisionable), loudly rather than silently dropping it, since provisioning it is a separate, not-yet-built follow-on.

network.nos_check (docs/ROADMAP.md #16/#31, schema settled 2026-08-04) opts an experiment into checking its network.slices VLAN intent against a real production network OS, alongside — never instead of — the actual measured path (network.data_plane). Absent by default; most experiments don't declare it:

network:
  nos_check:
    kind: sonic-vs                     # the only kind today
    trunk_port: Ethernet0              # validated against the pinned image's actual port list
    # part_ref / digest: optional, default to the pinned build vxparts/parts/gw/nos/os/sonic-vs

Deliberately not named "twin": the whole testbed already is one (an executable virtual prototype of a vehicle), so reusing that word for one internal field would describe nothing. What this does is a check — boot a real instance of the named NOS, hand it the same VLAN declaration network.slices already expresses, and confirm it accepts and reports back exactly that. It never carries measured traffic (§7: SONiC-VS is functional but not timing-representative), and it is not part of experiment up's provisioning sequence — a missing or misconfigured nos_check block never blocks or changes what an experiment actually measures.

kind names which NOS is being checked against — sonic-vs today, the shape a second kind would extend rather than a reason to design one now (network.data_plane's own kernel-bridge | ovs precedent). trunk_port is not a name vxedge invents: it is one of the pinned image's own, fixed port names (SONiC-VS's default reference platform exposes exactly 32, Ethernet0/Ethernet4/.../Ethernet124 — confirmed by booting it, not assumed), validated at vxedge experiment validate time against that concrete list rather than a free-form string or a loose pattern.

The .vxp descriptor holds packaging rather than content, so a version bump never touches an experiment file:

id: h1-slice-isolation
version: 0.1.0
entry: main.yaml
requires: { gpu: none }              # surfaced at scaffold time, checked by validate

6.0.4 Requirements vs. allocation

Two different things that both sound like "resources":

• requires on a part (part.yaml in vxparts) — what the software needs: arch, libc, cpu, memory_gb, gpu.vram_gb, gpu.compute_capability.
• resources on a component (in a vehicle) — what the node is allocated.

vxedge manifest validate checks part → component → host and fails early, so "this host cannot run this workspace" is reported before build time is spent rather than after. depends_on between parts is metadata only: validated for presence in the same baseline, never version-solved — a resolver is a package manager, and that would be a separate, deliberate decision.

6.0.5 Interface naming

A device's host interfaces are derived from its id: vz-<stem> for the bridge access port (the veth host side, or the tap itself) and vzp-<stem> for the veth peer. The stem is the sanitised id, shortened to a readable head plus a hash of the whole id when it does not fit the kernel's 15-byte IFNAMSIZ limit — node-cont-01 becomes vz-node-c-32cc.

This is a contract, not an implementation detail: campaign scripts address these names directly (ip addr replace … dev vzp-node-a-c06a, qemu … ifname=vz-node-q-691e). Names used to be positional — vz0, vz1, … in traversal order — which made ordering load-bearing in the worst way: adding a device renumbered the rest, and two views of the same topology could number it differently. exp-002 declares its arms cont, microvm, libvirt and sorts to cont, libvirt, microvm, so two H3 arms swapped interfaces between the two schema paths — the campaign would have measured libvirt and reported microvm, passing every test on the way.

A name now depends on its own device and nothing else: not declaration order, not sorting, not how many other devices exist. Two ids that shorten to one name are refused with both named. vxedge zone list prints each device's port= and peer=, so the mapping is always answerable without recomputing a hash.

6.0.6 Migration status

§6.0 is the only authored schema, and the only runnable one (2026-07-31). Every planner takes the domain model, which is built from a workspace; vxedge up, vxedge experiment up and vxtelemetry ingest|collect all read a workspace directory. All five experiments — exp-001, exp-002, exp-002b, exp-003, exp-010 — have workspaces, each verified to produce plans identical to the frozen §6.1 manifest it replaced.

§6.1 and §6.2 are deleted — structs, adapter and lowering alike. vxmodel render (track M) emits workspaces too, so vxedge manifest validate has one shape to check and CI asserts it.

The frozen vxperiments/experiments/exp-*.yaml files are kept as provenance for the results already recorded against them, not as inputs. Pointing a tool at one now reports what the document is and where to go instead.

6.1–6.2 Experiment and vehicle manifests — retired 2026-07-31

These two schemas are gone: the structs are deleted from vxedge-core, nothing parses them, and no tool emits them. §6.0 is the only schema.

They are named here rather than silently dropped because results archives, RESULTS.md entries and the frozen vxperiments/experiments/exp-*.yaml files all reference them. A §6.1 experiment manifest was one document carrying experiment/network/zones/compute/links/ traffic_profiles/telemetry/results, with software attached to a node as a flat image:; a §6.2 vehicle manifest carried the same composition blocks under a vehicle: header. §6.0 splits both across kind: experiment / kind: vehicle / kind: baseline documents and separates zone (a place) from device (a unit) from function (behaviour) — the conflation that made channels unexecutable and a slice's definition unlocatable.

The frozen manifests stay in vxperiments/experiments/ as provenance for results recorded against them. To read one against the schema it was written to, use the v0.1.0 tag, which is where phase-1 reproducibility is anchored.

6.3 Model governance — SysML v2 (track M, vxmodels)

Why SysML v2 here: unlike v1 it is textual (git-diffable), has a standardized REST/JSON API, and composes typed elements — the first systems-modeling standard practical for GenAI authoring. vxedge is unusually well-placed for it: the system is already fully declarative (manifests) and fully executable (up|down), so a model layer is live, not documentation.

Construct mapping (authoritative for track M):

Pipeline: .sysml sources (authored in vxmodels) → vxmodel render (Rust) → YAML manifest (+ recorded model digest) → vxedge manifest validate --json → the normal CLI lifecycle. GenAI-authored experiments therefore clear two typed gates (model typing, then manifest validation) before touching the testbed — the safety argument for generated experiments is structural, not aspirational.

Out of model scope, by rule: Rust internals, the vxAPI IDL (Protobuf stays the contract, SDKs stay generated), and anything zone/compute manifests are authoritative for.

Future graphical frontend (not yet built, added 2026-08-06): a Qt/QML desktop tool — analogous to the planned Electron "vxedge Studio," an MBSE/CAD-style variant of the same idea — to help OEM engineers design experiments visually instead of hand-authoring .sysml text directly. It would still read/write the same textual, git-diffable .sysml sources (the construct-mapping table above stays the contract either way); a graphical editor is a view onto that source, not a second model format. Explicitly deferred — no design work done here yet.

Terminology (matches ROADMAP.md): track M makes vxedge a model-governed executable virtual prototype. "Digital twin" is an earned claim reserved for phase-4 hardware parity — track M supplies the model half; phase 4 supplies the physical half.

7 — Network model — vxgateway (SDN slicing)

The edge gateway is a virtual network function built entirely from the production open-source Linux SDN stack, provisioned declaratively by vxedge (Rust rtnetlink/OVSDB, no shell templating):

• Data plane: Open vSwitch (or Linux bridge) carrying an 802.1q trunk; one VLAN per slice. Deployed either as a set of network namespaces on the host (default; lowest overhead, fully inspectable) or as a dedicated VM (stronger isolation of the gateway itself).
• Queueing / slice QoS: hierarchical tc (HTB) queue trees per slice with guaranteed min-rate for URLLC-like slices — the object of hypothesis H1.
• Real 5G slicing (H2): Open5GS core + UERANSIM gNB/UE terminate PDU sessions per S-NSSAI; slice definitions in the manifest (mode: s-nssai) provision both the 5G core and the gateway. Zone nodes may additionally carry a URSP-style slice_policy (per-traffic-class selection across concurrent PDU sessions), mirroring UE-side slicing as deployed on production automotive modems (Qualcomm Snapdragon Auto 5G; cf. the DT/BMW/Valeo/Ericsson/Qualcomm 5G SA slicing trial, which vxedge reproduces openly in software). Optional radio-channel realism via NVIDIA Sionna (6G Developer Program); full Aerial CUDA-Accelerated RAN is a stretch goal requiring specific NVIDIA platforms.
• TSN variant (phase 3): time-aware scheduling via taprio/cbs qdiscs + gPTP time sync (linuxptp, 802.1AS) as an alternative gateway shaping mode — enables HTB-vs-TSN isolation comparisons matching production automotive Ethernet backbones.
• Legacy bus edges (type: can, phase 3): SocketCAN/vcan buses per zone, exposed to guests via QEMU CAN controllers and tunneled across the backbone with cannelloni; CAN frames map to VSS signals via Kuksa feeders — the zonal-controller CAN-aggregation role that makes the prototype vehicle-shaped rather than datacenter-shaped.
• Harness modeling: the manifest links section is the wiring harness. Per-link impairments (tc netem: rate cap, latency/jitter, loss) emulate PHY characteristics such as 100/1000BASE-T1 — delivered: rendered onto the elected access port's egress qdisc and, for the reverse direction, an IFB the port's ingress traffic is redirected onto (a qdisc only shapes egress, so the return direction needs a device of its own to shape). harness (medium/length_m/weight_g) carries the same segment's physical properties declaratively, for a harness-reduction report, independent of whether it is impaired. CAN/LIN drops ride type: can links (LIN modeled as a constrained bus, phase 3). Because harness topology is manifest data, it becomes an experimental variable — star vs daisy-chained zones, sensor-to-ECU path placement — iterable in minutes, which no physical testbed offers. Power distribution is modeled only as VSS signals (vxplant power tree, optional); electrical/thermal fidelity is explicitly out of scope. gPTP/TSN (above) is the harness's timing layer.
• Tunnels: VXLAN for inter-host communication; WireGuard (in-kernel) for secure telemetry and OTA transport — network.tunnels (§6.0.3), one gateway-terminated point-to-point link per entry. WireGuard delivered (docs/ROADMAP.md #16): backend-independent (a WireGuard interface is a routed L3 endpoint, never enslaved to the bridge/OVS switch — the same mechanism regardless of network.data_plane). No local key material in the manifest; the gateway mints its own identity on first up and reports the public key back via status. Kernel-level correctness (a real handshake between two simulated hosts, not just interface presence) verified by gateway_netns.rs's tunnel_completes_a_real_handshake_between_two_simulated_hosts. VXLAN: model and validation delivered 2026-08-04 (docs/ROADMAP.md #16) — kind: vxlan (§6.0.3), structurally the opposite of WireGuard: it joins the bridge as a trunk-like port, extending one or more local slices' L2 domain across hosts. GatewayPlan provisioning is a separate, not-yet-built follow-on; declaring one today is refused loudly at plan time, not silently dropped.
• Routing: static for lab experiments; FRR provides BGP for cloud interconnect.
• Firewalling: nftables per VLAN/slice.
• Production NOS twin: the same slice/VLAN/VXLAN/BGP intent is also rendered onto a SONiC-VS virtual switch (community qcow2). SONiC is the production open-source NOS (Linux Foundation; NVIDIA ships production SONiC on Spectrum), which demonstrates that vxedge slice configs express identically on a production NOS. The SONiC-VS data plane is functional but not timing-representative, so all latency/jitter measurements run against the Linux-native vxgateway path. (NVIDIA Cumulus VX is deprecated as a standalone image and is not used; NVIDIA Air remains only an optional cloud sandbox.) Manifest field: network.nos_check (§6.0.3).
• Determinism controls: gateway forwarding cores pinned and isolated (isolcpus), PREEMPT_RT host kernel, hugepages. An optional FD.io VPP (DPDK) data-plane variant exists for URLLC-focused runs needing minimal per-packet latency variance.
• Hardware offload path (phase 4): the same OVS/tc intent offloads onto NVIDIA BlueField DPU / ConnectX via DOCA — the production exit ramp from the virtual gateway.
• Monitoring: gateway counters, tc/OVS statistics, and syslog exported to the central Prometheus node.

8 — Service model (OTA, telemetry, extensibility)

• All services defined under top-level services in the manifest.
• ota and telemetry are the core supported types.
• Services can be referenced in zones or compute nodes.
• CLI deploys services via vxedge ota up|down [--experiment <path>] (cwd-resolved by default), or as part of experiment up --stage services — the path an actual campaign takes, since it sequences services after the gateway/zones they depend on.
• Extensible: future vxsvc-* repos can be added and referenced in manifests. The phase-3 Kuksa databroker (vehicle signal plane, COVESA VSS) is deployed via the vehicle manifest's signal_plane block rather than as an experiment service.
• Each service definition includes id, type, image, config.

OTA design (H4, vxota): the service exists to be a realistic, measurable OTA workload for H4, not a fleet-management platform — two industry concerns are kept deliberately separate:

• Trust model (adopted as design vocabulary, not as a dependency): artifact metadata follows the shape of Uptane (the automotive-specific, TUF-derived OTA security framework — the closest thing to an ISO/SAE 21434 / UNECE R156-aligned standard) — signed root/targets-style manifests, image-repo vs. director-repo separation. The reference implementation aktualizr (C++) is not adopted as a dependency: it would contradict the Rust-first service policy (§15) for no benefit vxota needs today.
• Delivery/orchestration (out of scope for vxota v1): production fleet tools — Eclipse hawkBit (Java rollout server + DDI API) and Mender (Go, server+client) — are named here only as the production analogs an OEM engineer would recognize (mapped in docs/USER-GUIDE.md §6); neither is wired in, since vxota's zones/compute nodes are ephemeral per-experiment instances, not persistent fleet devices needing campaign/rollout management.
• Slot/image-swap updates (phase 3, optional): if a later experiment wants a real atomic image swap on a running zone (vs. vxota v1's signed-blob delivery), the candidate is Rugix Ctrl (Rust, memory-safe, secure-by-default signature verification, native content-defined-chunking delta updates — chosen over the more established RAUC specifically for the Rust-policy fit; RAUC remains the fallback if Rugix's narrower deployment history or its gaps — no NAND/UBI, no encrypted updates, both irrelevant to our QEMU/libvirt block-storage bundles — become a blocker). Maps naturally onto vxzones guest bundles, which are already kernel+initramfs "slots" in Rugix/RAUC's sense.
• v1 service backend (fidelity caveat, stated honestly): SvcService.Up (§5) spawns the service binary (vxota serve) as a host-local child process co-located with vxedged, tracked by a pidfile the same way the microVM zone backend tracks QEMU — not yet scheduled onto a vxcompute node, since that backend doesn't exist. config.listen (new manifest field, alongside config.artifacts) is the address the process binds; today that is still the gateway's own bridge address on the slice under test, an address zone nodes can already reach without a tunnel. network.tunnels (§7) now exists as a mechanism — a manifest can declare a WireGuard link the gateway terminates — but vxota is not yet wired to bind config.listen onto one; that integration, and scheduling onto a real vxcompute node, remain open (vxedge/TASKS.md §4).

9 — Virtualization & zone provisioning (libvirt, QEMU/KVM, QNX, RTOS, containers)

Stack, bottom to top: host Linux (PREEMPT_RT, isolcpus, hugepages) → KVM → QEMU — managed by libvirt, driven by vxedge through the Rust virt crate.

• libvirt is the default zone backend for kind: vm | rtimage. The CLI compiles zone manifests into libvirt domain XML: lifecycle (define/start/destroy/undefine) gives idempotent up|down and queryable status; vNICs attach to the vxgateway OVS/bridge trunk with per-vNIC VLAN tags; <cputune> (vcpupin/emulatorpin), hugepages, and virtio multiqueue express the manifest's realtime: block; storage pools hold the SHA256-digest-tagged images.
• microVM escape hatch: backend: qemu-microvm drives raw QEMU (QMP, microvm machine type) — or cloud-hypervisor/Firecracker — bypassing libvirt's device model for the minimal-latency arm of hypothesis H3. Both backends hide behind the same vxedge zone up interface.
• Guest palette:
  • QNX SDP 8 (free non-commercial license) — QEMU x86-64/ARM guest; the production-representative ADAS/safety guest for front-left/front-right zones. QNX is what production ADAS and cockpit controllers run, which anchors OEM relevance.
  • Zephyr / FreeRTOS — microVM images for sensor/ECU nodes.
  • Android Automotive (AAOS) — infotainment zone; QEMU with virtio-gpu or GPU passthrough.
  • Embedded Qt/QML IVI/cluster loadable — an alternate infotainment/cluster baseline to AAOS, not desktop tooling (correction, 2026-08-06: earlier drafts of this document wrongly scoped Qt/QML as an operator dashboard only — it has no such role, see §5.1). Runs either as an app/middleware layer atop the QNX guest above (the production QNX+Qt cluster/IVI pattern several OEMs actually ship) or under a bare QEMU ARM zone node — exercises the same "one vehicle, many baselines" mechanism (ARCHITECTURE.md §6.0) that lets an experiment swap AAOS for Qt/QML (or vice versa) on the same infotainment zone without touching the vehicle's structural definition.
  • Containers — for nodes where packaging as a container is the experimental variable.
• Fidelity caveat (stated honestly in publications): production zonal controllers invert this stacking — silicon runs a Type-1 hypervisor (QNX Hypervisor and peers) hosting guests, whereas here KVM plays the hypervisor and QNX is a guest. Functionally equivalent for slice-isolation research; the gap closes via the hypervisor fidelity ladder below and, ultimately, DRIVE OS on DRIVE AGX hardware (phase 4).
• Hypervisor fidelity ladder (production-grade rungs only — no research-toy hypervisors):
  • Rung 0 (software-only, phase 1–3): KVM with strict static partitioning — isolcpus/cpuset pinned cells, hugepages — the Linux approximation of partitioning and the baseline arm of the freedom-from-interference experiment (exp-004).
  • Rung 1 (phase 4): Xen dom0less as a dedicated boot-entry host profile — production open-source with an active functional-safety track (Xen 4.21 dom0less refactoring; AMD/Honda automotive demos, Xen Summit 2025); Zephyr domUs boot sub-second.
  • Rung 2 (phase 4): QNX Hypervisor — the production zonal-controller stack itself (license permitting; pairs with DRIVE OS QNX). Each rung converts part of the stated caveat into a measured claim (exp-004 reruns per rung).
• CLI stages images, creates instances, and configures network interfaces; persistent storage optional per zone; logs and telemetry collected centrally.

Sensor, actuator & I/O virtualization (four layers)

1. Perception sensors (camera, lidar, radar, GNSS/IMU) — synthesized by the world simulator (CARLA; AlpaSim / Sensor RTX as the higher-fidelity path). In a real vehicle these ride GMSL/FPD-Link serdes into the ADAS ECU; here the transport is slice traffic through vxgateway (traffic_profiles.source: carla) — sensor streams are the measured ADAS load, not synthetic filler.
2. Body sensors/actuators (doors, HVAC, wipers, BMS…) — restbus simulation: simulated nodes emit/consume CAN frames per DBC definitions (sensors:/actuators: in the node schema); Kuksa's CAN provider maps DBC signals ↔ VSS. Closed-loop behavior comes from the vxplant agent (in vxagents): subscribes to actuator commands, integrates a plant model (first-order default; FMI/FMU import in phase 4 so OEMs can drop in Simulink-exported models), publishes sensor feedback.
3. Register-level I/O (GPIO/ADC/PWM/I2C/SPI) — application-class ECUs (QNX, AAOS) use QEMU device models (io.devices); MCU-class ECUs needing register fidelity use Renode (backend: renode): full peripheral models running unmodified Zephyr/FreeRTOS firmware.
4. Actuation loop closure — control commands (steering/throttle/brake) return to CARLA via carla-ros-bridge vehicle control, closing the loop world → sensors → slices → AD stack → actuation → world. End-to-end control-loop latency under slice contention is a first-class metric (§12).

Boundary standards (phase 4): ASAM OSI at the simulator↔stack boundary, FMI/FMU for plant models, XIL-style semantics for the bench API — so the virtual testbed and OEM HIL rigs speak the same protocols (the SIL→HIL migration story).

10 — Compute model (vxcompute — Autoware/ROS2, Alpamayo, inference, OTA hosting)

• Compute nodes run containerized workloads. Two AD stacks are supported as ADAS-slice workloads, which usefully stress the network differently:
  • Autoware / ROS2 (Apache-2.0) — classical modular AV stack; consumes CARLA sensor streams via carla-ros-bridge.
  • NVIDIA Alpamayo / AlpaSim (open-sourced at CES 2026) — reasoning-VLA driving model and open simulation framework; runs on the host GPU without DRIVE hardware. Fine-tuning and DRIVE Hyperion integration are out of scope (phase 4).
• Perception/inference workloads use DeepStream / TensorRT; container definitions are optimized for GPU pass-through and TensorRT runtime (see resources.gpu in the schema).
• OTA server may be deployed in a compute node if referenced in the experiment manifest.
• Resources declared in manifest (cpu, memory, gpu); services deployed dynamically per manifest references.
• Optional in-vehicle-style workload orchestration via Eclipse Ankaios/BlueChi (phase 3) to mirror production SDV runtimes.

11 — Edge agents & telemetry collection (vxagents / vxtelemetry)

• Rust agents deployed per experiment to collect metrics, latency, and traffic.
• Prometheus exporters for centralized metric collection; GPU metrics tracked alongside (DCGM/NVML: utilization, VRAM boundaries, thermal/power throttling) since AD workloads share the host with the gateway.
• VSS alignment: vehicle-signal telemetry follows the COVESA VSS catalog via the Kuksa databroker (phase 3). The open-source C++ AWS IoT FleetWise edge agent serves as a design reference for signal collection/campaign semantics — as a reference only: the AWS-hosted FleetWise service closes to new customers in April 2026 and is not a dependency.
• Optional PCAP capture for detailed traffic analysis.
• Agents and telemetry services configured via manifest and CLI commands.
• Designed to be lightweight to avoid perturbing experiment results.

12 — Experiment methodology, metrics, and analysis plan

Methodology:

1. Preparation:

   • Validate manifest syntax and semantics (experiment + referenced vehicle manifest).
   • Stage images on host(s) using CLI (libvirt storage pools / container registry).
   • vxedge up provisions vxgateway slices (VLANs, tc queue trees; S-NSSAI slices in the 5G core for mode: s-nssai).

2. Execution:

   • vxedge vehicle up (or vxedge experiment up referencing a vehicle) to provision gateway, zones, compute, signal plane, and services.
   • Generate traffic per traffic_profiles — synthetic (pps/bandwidth) and/or real perception traffic (source: carla).
   • Collect telemetry (Prometheus + GPU metrics + PCAP) and service logs.

3. Postprocessing:

   • Extract metrics from Prometheus exporters.
   • Parse PCAPs for latency, jitter, packet loss.
   • Aggregate results per slice and per node.

Metrics:

• Network: p50/p95/p99 latency, jitter, packet loss per slice.
• Closed loop (phase 2+): end-to-end control-loop latency — world → sensors → slices → AD stack → actuation → world — under slice contention.
• Compute: CPU/memory/GPU utilization, service response times.
• OTA: deployment success rate, artifact delivery latency.
• Telemetry agents: data ingestion rates, metric completeness.

Analysis plan:

• Compare latency/jitter under different virtualization choices (H3: libvirt VM vs container vs microVM).
• Evaluate slice isolation under heavy multi-zone traffic (H1), and static VLAN mapping vs real S-NSSAI slicing (H2).
• Quantify OTA and telemetry impact on URLLC slices (H4).
• Quantify the effect of determinism controls (pinning, PREEMPT_RT, VPP data plane) on tail latency.
• Generate reproducible tables, charts, and CSV datasets for publication.

13 — Reproducibility, CI/CD, artifact publication (registries)

• Workspace identity — one rule. A run's provenance is a SHA-256 over every file the experiment is made of: the relative path, a NUL, the bytes, a NUL, in sorted path order, recursing into subdirectories and excluding results/.

  Each part of that is load-bearing. The path is hashed with the bytes, so renaming a file is a change. Order is sorted rather than as-read, so the answer does not depend on the filesystem. results/ is excluded because outputs are not inputs — including them would change a run's identity the moment it produced one. The recursion is what distinguishes the identity from the document set: a sandbox's parts/, assets/, scripts/ and tools/ are what ran even though they are not workspace documents, and a local part that changes is a different experiment.

  It answers one question — did every repetition of this campaign run from the same input? — and it is one rule because it is recorded in two places that must agree: the daemon's audit records and the results archive's workspace_sha256. vxedge-core and vxtelemetry implement it independently (the latter deliberately does not depend on the former) and pin the same test vector over the same fixture. Two implementations of one rule stay honest by agreeing on an answer, not by sharing code; if the rule is revised, both move together.

• Reproducibility:

  • All manifests under vxperiments are versioned and authoritative.
  • Images and RTOS builds tagged with SHA256 digest.
  • CLI ensures idempotent up|down operations (libvirt domain lifecycle; declarative gateway provisioning).
  • Because the gateway is software, its full provisioning is unit/integration-testable in CI against network namespaces — no physical router in the loop.

• CI/CD:

  • GitHub Actions pipelines for building images, testing CLI, and publishing manifests.
  • Artifacts automatically uploaded to private registry or S3 bucket.
  • Experimental runs can be triggered and reproduced on demand.

• Artifact publication:

  • Results archived per results_prefix in experiment manifest.
  • Metadata includes experiment ID, workspace identity (above), .vxp id/version, CLI version, and — when the campaign opts in — the content-addressed digests of the container images it ran against. The archive bundles the workspace itself under workspace/, so a result is re-runnable from what is inside it rather than merely described by it.
  • Registry uploads optional; ensures compliance with reproducible research principles.

14 — Safety, security, licensing, and industrialization notes

• Safety:

  • Non-safety-critical lab environment; no real vehicle control signals.
  • Experiments simulate zonal ECU traffic in virtual/isolated networks.
  • The platform is production-representative, not safety-certified; the QNX-as-guest inversion (§9) and shared-host gateway are documented fidelity caveats.

• Security:

  • WireGuard for encrypted telemetry and OTA transport.
  • Images validated with SHA256 digests.
  • No root exposure in containerized services.
  • nftables firewall per VLAN/slice on vxgateway.

• Licensing:

  • Open-source Rust CLI and agents (MIT/Apache-2.0).
  • QNX SDP 8 under QNX free non-commercial license (research use); Zephyr/FreeRTOS under their respective licenses.
  • Autoware/ROS2 under Apache-2.0; Alpamayo models/AlpaSim under NVIDIA's open-model licenses; SONiC, OVS, FRR, Open5GS, UERANSIM, Eclipse SDV components under their upstream OSS licenses.

• Industrialization notes:

  • Middleware and patterns match the production SDV convergence: Eclipse SDV (Kuksa/VSS, Zenoh, uProtocol, Ankaios), SOAFEE-style cloud/vehicle virtual-ECU parity, NVIDIA DRIVE-adjacent compute, and UE-side slicing semantics matching production automotive modems (Qualcomm Snapdragon Auto, URSP). This makes results directly legible to NVIDIA-DRIVE-partner OEMs (JLR-class zonal E/E programs) and Qualcomm-Digital-Chassis programs alike.
  • ARM64 zone/compute variants (phase 3) target NVIDIA Jetson and the cloud virtual-ECU parity paths both silicon vendors ship: Snapdragon vSoC / Virtual Platforms on Graviton and Google Cloud.
  • All manifests modular, reusable across multiple experiments.
  • Gateway patterns have an explicit hardware exit ramp: SONiC on Spectrum, or OVS/DOCA offload on BlueField/ConnectX.
  • CI/CD pipelines support automated image and manifest validation.
  • Publishable results designed to integrate into research papers or internal dashboards.

15 — Implementation policy & phased roadmap

Languages

• Rust — vxedge-core, vxedged daemon (vxAPI/tonic), vxedge CLI, vxagents, gateway provisioning (rtnetlink/OVSDB, virt crate for libvirt).
• C++ — everything touching ROS 2/DDS/Autoware, DeepStream/TensorRT plugins, QNX-side components (automotive-native, AUTOSAR-adjacent), and the embedded Qt/QML IVI/cluster loadable (an in-vehicle-HMI baseline alongside AAOS — see §5.1, §9).
• Python — Sionna, notebooks, analysis pipelines.
• TypeScript — web dashboard, Electron "vxedge Studio", generated vxAPI SDK (connect-es).
• Qt/QML (future, not yet built) — a graphical vxmodels MBSE frontend for designing experiments visually (see §5.1, §6.3); an Electron-style desktop variant, not an operator dashboard.
• Shell — glue only.
• SysML v2 (track M) — authoring-time system models in vxmodels, rendered to YAML manifests by vxmodel. Deliberately not a third runtime interface.
• Manifests (YAML) and the generated vxAPI (Protobuf) are the only cross-language interfaces.

Containerization policy (hybrid, Podman — decided 2026-07-14)

The measured path stays on the host: kernel data plane (bridge/VLAN/tc/netem/ WireGuard/vcan), KVM/libvirt zone execution, vxedged, and measurement agents. Containers share the kernel, so containerizing these would put container networking inside the measurement and break comparability with frozen campaigns. Everything else runs containerized under Podman (daemonless/rootless; OCI images stay Docker-compatible; matches the automotive ecosystem — Ankaios, AGL, RH In-Vehicle): the 5G stack (Open5GS NFs + UERANSIM + MongoDB as one pod sharing a network namespace, so FiveGPlan's loopback addressing is unchanged in or out of the pod — vxedge/containers/fiveg/), service images (vxota/vxtelemetry), the build/dev toolchain, and phase-2 workloads (CARLA/Autoware/ DeepStream, already container-first per §10). Image digests are version pins recorded into results metadata (§13). Host keeps only: distro-standard kernel tooling, QEMU/libvirt, and one modprobe sctp for UERANSIM's NGAP (SCTP is not namespaced). This mirrors the SOAFEE/EWAOL production pattern (containerized workloads over a thin KVM host) and is the portability story for non-Linux dev machines (build/analysis anywhere; measurement always on a Linux kernel).

Phases

1. Phase 1 — slicing core, API-first from day one: vxedge-core + vxedged + vxAPI v1 (Gateway/Zone/Compute/Job/Event services) with the CLI as thin client; vxgateway (OVS/tc/FRR/WireGuard) + synthetic traffic + vxagents/vxtelemetry → the H1/H3/H4 isolation numbers. This alone is the paper.
2. Phase 2 — real slices & workloads: Open5GS/UERANSIM S-NSSAI slicing (H2) incl. URSP-style slice_policy (UE-side selection over concurrent PDU sessions); CARLA → ROS2 → Autoware as real ADAS traffic, closed-loop (actuation back into CARLA; control-loop latency metric); per-link harness impairments (netem, BASE-T1 emulation); QNX SDP 8 and AAOS zones; SONiC-VS production-NOS twin; TypeScript vxAPI SDK + minimal web dashboard (live slice metrics over gRPC-Web).
3. Phase 3 — production SDV alignment: vehicle manifest end-to-end (vxedge vehicle up); Kuksa/VSS signal plane; rmw_zenoh/uProtocol comms; Ankaios workload orchestration; Sionna channel realism (NVIDIA 6G Developer Program); Alpamayo/AlpaSim workload; optional Rugix-Ctrl-based atomic slot updates on vxzones guest bundles (§8 OTA design); embedded Qt/QML IVI/cluster loadable (an alternate in-vehicle-HMI baseline to AAOS, §9); cloud-hosted vxedged --controller federating multiple GPU hosts; Electron "vxedge Studio" groundwork; vxedge-tray (Rust SNI tray, thin vxAPI client) + vxedged systemd unit; ARM64 zone/compute/agent variants (Jetson + Snapdragon vSoC / Graviton parity); CAN/LIN edge modeling (type: can, SocketCAN/vcan + Kuksa CAN feeders); TSN gateway variant (taprio/cbs + gPTP); UDS/DoIP diagnostic probe for OTA runs; V2X groundwork — multi-vehicle composition (second vehicle-manifest instance, V2V) + rsu nodes (V2I); sensor/actuator virtualization — restbus simulation (DBC + Kuksa CAN provider), vxplant plant-model agent, QEMU io.devices, Renode backend for MCU-class virtual ECUs.
4. Phase 4 — hardware (optional): Jetson AGX in the loop (affordable DRIVE-lineage stand-in); DRIVE AGX + DRIVE OS (Linux/QNX); BlueField/DOCA gateway offload; Aerial CUDA-Accelerated RAN; V2X radio realism — C-V2X PC5 sidelink (Sionna channel models; PAWR/AERPAW outdoor validation); ASAM boundary standards — OSI (simulator↔stack), FMI/FMU plant-model import, XIL-style bench semantics; optional VSS power-tree modeling; hypervisor fidelity ladder rungs 1–2 — Xen dom0less host profile and QNX Hypervisor (freedom-from-interference measured per rung via exp-004).
5. Phase 5 — remote/multi-user/cloud deployment (backlog, no gate yet): mTLS + OIDC auth path for vxAPI's TCP transport, with viewer/operator/admin roles (§5.3's own target design — today only the local Unix-socket transport is live, and even that lacks the peer-credential check §5.3 names as its own target, see that section's 2026-08-05 correction). Deliberately not scheduled into an earlier phase: it secures remote access, and every frontend that would use it remotely (web dashboard, Electron, vxmcp) is itself deferred to phase 3/4 (docs/ROADMAP.md's 2026-08-01 re-scope — "the CLI is the only frontend any experiment uses," and the CLI only ever talks over the local socket). Building the auth layer before a real remote consumer exists would be infrastructure with nothing to test it against; revisit once a phase-3/4 frontend actually needs it. Scoped 2026-08-05 (full account, docs/ROADMAP.md Phase 5 section) into six concrete sub-items — TCP listener support in vxedged/vxedge-cli (neither accepts anything but unix:// today), mTLS (cert provisioning is an open design question, not just a library choice), OIDC bearer-token validation, a per-RPC role/authorization model, actor-aware audit logging (a real, present gap — Audit::record carries no identity field today), and the concurrent-multi-experiment question this makes sharper (EdgeState.active is a single slot, not a collection, already flagged under "Beyond phase 4"). FRR (BGP for cloud interconnect) joins it for the same reason, moved here 2026-08-04: unlike VXLAN (which extends an already-documented network.tunnels schema hook), FRR has no manifest schema field anywhere in this document — it would need one designed from scratch — and its only stated purpose, cloud interconnect, has no current or near-term consumer; every experiment today uses static routing, which §7 already states is sufficient for lab experiments. Revisit alongside mTLS+OIDC once a real multi-host/cloud deployment scenario exists to design and test the schema against. Deliberately excluded from this scope: vxedged --controller federation (still Phase 3 — a site-daemon-to-controller plumbing concern, not the human/CLI-to-daemon auth path this item covers) and the cloud-hosted persistence/multi-tenant-session/event-streaming question (docs/ROADMAP.md's "Beyond phase 4" — explicitly speculative, no phase assigned, its own separate sign-off needed before scoping).

Cross-cutting track M — SysML v2 model governance (twin-readiness)

Not phase-gated; runs alongside phases 2–3 and gates nothing in phases 1–4. All public claims keep the executable virtual prototype vocabulary until phase-4 hardware parity earns the term "digital twin" (ROADMAP.md terminology rule); track M builds the model half of that claim. Details in §6.3; work items on the program board (docs/ROADMAP.md) and, once created, vxmodels/TASKS.md.

• M0 — governance & home: this section + §4/§6.3 scoping; vxedge/vxmodels created and seeded (AGENTS.md/TASKS.md/skills) — all done 2026-07-13.
• M1 — retroactive traceability: H1–H4 as requirement definitions with verify links to the frozen exp-001/exp-002b/exp-003 archives (manifest digests). Zero new experiments; demonstrates the requirement → analysis-case → manifest → archive chain on already-published-quality data.
• M2 — structure model + renderer: vx-vehicle-01 E/E architecture as part/connection/interface definitions; vxmodel render (Rust) emits the vehicle manifest, gated by vxedge manifest validate --json; model digest recorded in the generated manifest (§4 single-source rule).
• M3 — analysis cases + variants (the GenAI generation target): experiments as analysis cases; arms (packaging choice, vlan vs s-nssai) as variant points enumerated mechanically; a sysml-authoring skill so LLM sessions compose typed model elements — generated experiments pass the §6.3 double gate before touching the testbed. Aligns with phase 2 (the S-NSSAI driver gives variants something real to enumerate).
• M4 — scenarios + fidelity ledger: traffic/transportation scenarios as use case/occurrence structures mapped toward ASAM OpenSCENARIO (extending the phase-4 OSI commitment); fidelity caveats as first-class model annotations so every generated experiment carries its validity envelope; environmental KPIs (energy per slice/workload via the DCGM/NVML hooks) as verified requirements; V2X multi-vehicle scenario families in phase 3.

Paper #3 (ROADMAP.md) draws on M1–M4; the twin claim attaches only if phase 4 lands.

16 — Appendix: example manifests, gateway snippets, run sequence

Example experiment manifest snippet

experiment:
  id: exp-001
  name: ADAS slice isolation test
  duration_s: 1800
  results_prefix: results/exp-001
  vehicle: vxperiments/vehicles/vx-vehicle-01.yaml

network:
  slices:
    - id: adas
      mode: vlan
      vlan: 101
      qos: { min_rate_mbps: 100, max_rate_mbps: 200, priority: 1 }
    - id: infotainment
      mode: vlan
      vlan: 103
      qos: { min_rate_mbps: 0, max_rate_mbps: 500, priority: 5 }

services:
  ota:
    id: ota-01
    type: ota
    image:
      url: registry/vxota:latest
      kind: container
    config:
      artifacts: /opt/ota/artifacts
      listen: 10.101.0.1:9900   # reachable from zone nodes on the same slice
  telemetry:
    id: telemetry-01
    type: telemetry
    image:
      url: registry/vxtelemetry:latest
      kind: container
    config:
      scrape_interval_s: 5

vxgateway provisioning (rendered by the CLI; shown as equivalent shell for illustration)

# OVS trunk + per-slice VLANs (one access port per zone vNIC, tagged)
ovs-vsctl add-br vxbr0
ovs-vsctl add-port vxbr0 vz-front-left  tag=101   # adas slice
ovs-vsctl add-port vxbr0 vz-front-right tag=102
ovs-vsctl add-port vxbr0 vz-rear-left   tag=103   # infotainment slice
ovs-vsctl add-port vxbr0 vz-central-hpc tag=105

# Per-slice queue tree (HTB): guaranteed min-rate for ADAS, capped infotainment
tc qdisc add dev vxup0 root handle 1: htb default 50
tc class add dev vxup0 parent 1:  classid 1:1  htb rate 1gbit
tc class add dev vxup0 parent 1:1 classid 1:10 htb rate 100mbit ceil 200mbit prio 1  # adas
tc class add dev vxup0 parent 1:1 classid 1:50 htb rate 1mbit   ceil 500mbit prio 5  # infotain

Run sequence (CLI)

0. vxedged serve # or let the CLI auto-spawn a local daemon
1. vxedge up # vxgateway: bridges, VLANs, queue trees
2. vxedge zone up --vxid vxzone-front-left
3. vxedge zone up --vxid vxzone-front-right
4. vxedge compute up --vxid vxcompute-central
5. vxedge ota up --experiment vxperiments/exp-001.yaml
6. vxedge telemetry up --experiment vxperiments/exp-001.yaml (not yet wired to the daemon)
7. vxedge experiment up --manifest vxperiments/exp-001.yaml
8. Collect results: vxedge telemetry collect --manifest vxperiments/exp-001.yaml --out results/exp-001.zip
9. Tear down: vxedge experiment down --manifest vxperiments/exp-001.yaml then services, compute, zones, gateway

Or, end to end with a vehicle manifest: vxedge vehicle up --manifest vxperiments/vehicles/vx-vehicle-01.yaml → vxedge experiment up --manifest vxperiments/exp-001.yaml → collect → experiment down → vehicle down.

Testbed setup

Target: the full vxedge testbed — control plane, vxgateway, virtual ECUs, AD workloads — running flawlessly on a single machine. Requirements are tiered by phase so you only install what the work in front of you needs. vxedge doctor --tier N (delivered 2026-07-13) verifies tiers 0–3 automatically with actionable fix hints; --json for CI.

Tier 0 — control-plane development (works today)

Tier 1 — H1 experiments: gateway + probes (phase 1)

Everything in Tier 0, plus:

No OVS and no WireGuard needed for H1 — the gateway uses the kernel bridge.

Tier 2 — real workloads: zones, 5G, CARLA (phase 2)

Everything above, plus:

Tier 3 — full vehicle + HMI (phase 3)

Adds: Kuksa databroker container, CARLA assets (~20 GB), AAOS emulator images, Qt ≥ 6.8 (QML HMI), Node 20+ (web dashboard build). RAM comfortably 96 GB for the six-zone vehicle manifest with AAOS + QNX guests concurrently.

Tier M — modeling workstation (optional, any phase; track M)

Authoring-time only — none of this runs during a measured experiment, so it has zero effect on results and no host-resource requirements beyond an editor.

Rules

• Versions above are the verified set; bumps go through this file in the same PR.
• One machine is the design target through phase 3; multi-host (vxedged --controller, VXLAN inter-host) is optional scale-out, not a requirement.
• Anything the testbed needs that this file doesn't list is a bug in this file.

Provisioning

Single, self-contained working document for this initiative: design, use-case manifests, functional requirements, and task-board status, all in one place (consolidated 2026-08-10 from a separate NODE-PROVISIONING.md + docs/ROADMAP.md Track V split — this work is substantial enough to pick up in isolation without juggling files). Governs crates/vxedge-core/src/zones/, the new crates/vxedge-core/src/containers/ module, their relationship to crates/vxedge-core/src/ fiveg/, and (Track F) a multi-vehicle-instance layer above VehicleService. Read ARCHITECTURE.md §9 (virtualization) and §15 (containerization policy) first — this doc extends both, contradicts neither.

How to read this document, in order: §1–2 grounds scope and reviews an external proposal that prompted this work. §3 is the use-case-driven methodology this initiative follows — real YAML manifests first, validated, before any Rust abstraction is designed. §4 is the current draft use case (a six-zone reference vehicle) awaiting that validation. §§5–9 are the design derived from existing patterns (written before §3's methodology was adopted as house process for this initiative — kept because it's still correct, but §3's use case is what governs new decisions from here on, not the other way around). §10 is the task board — Track V (single-vehicle node provisioning) and Track F (multi-vehicle fleet layer), with live status.

1. What this layer is for, and what it is not for

vxedge is a research platform evaluating 5G network slicing over zonal/central SDV architectures, running on commodity dev-host / GPU-host hardware — not a production automotive ECU, and not a functional-safety-certified target. ARCHITECTURE.md's own framing: "entirely in software," QEMU/KVM VMs "managed via libvirt by default with a raw-QEMU microVM backend option." The metrics this platform is built to produce (§12: p50/p95/p99 slice latency, jitter, packet loss, OTA delivery latency, telemetry completeness) are all measured above the virtualization layer, not inside it. Nothing in the platform's phase gates (H1–H4, docs/ROADMAP.md) depends on hypervisor boot latency, vCPU-exit handling cost, or virtio ring-buffer throughput — those would matter enormously if vxedge were building production ECU firmware; they are not what phase 1–3 gates measure.

This matters because it sets the actual bar for this layer: reliable, idempotent, low-drama provisioning of heterogeneous nodes, using components mature and audited enough that engineering time goes into the 5G-slicing research question, not into re-deriving virtualization primitives the industry has already spent a decade hardening. A design that optimizes for ECU-grade determinism and in-process vCPU thread loops is solving a problem this project does not have, at the cost of one it does: shipping the actual provisioning layer soon, on a foundation that doesn't need a security audit before the next experiment campaign can trust it.

If vxedge's mission changes — if a later phase genuinely targets a real embedded ECU deployment target rather than a dev-host research testbed — revisit §2's verdict. Nothing here forecloses that; it just isn't today's problem.

2. Review of the external proposal ("Gemini brief")

An external proposal (pasted into this repo's task history 2026-08-07, not otherwise recorded) argued for a from-scratch, in-process VMM: vxedge-core opens /dev/kvm directly, issues KVM_CREATE_VM/KVM_RUN ioctls itself, runs vCPU loops on native threads, implements VirtIO ring buffers by hand, and adds a WASM-capsule execution mode for RTOS/MCU-class nodes — explicitly to avoid shelling out to qemu-system-*/firecracker/libvirtd/incus. It asked for this "as fast as possible."

Verdict: reject the from-scratch in-process VMM; adopt two of the proposal's real components on their own merits, at a much narrower scope than proposed.

2.1 Fact-check of the named components

The proposal cites specific crates/projects as the stack. Checked against real sources (not taken on faith, since a proposal this specific deserves the same verification this repo applies to any other technical claim):

2.2 The scope problem, independent of the fact-check

Even where every cited component were real and mature, building a from-scratch VMM is the wrong call for this project right now, for reasons that would hold even if the fact-check above had come back clean:

• "As fast as possible" and "raw KVM ioctls + hand-rolled VirtIO rings + custom seccomp filters" are contradictory asks. Firecracker and Cloud Hypervisor are exactly this design, built by dedicated teams over years, security-audited, and battle-tested at hyperscale. Re-deriving that from primitives is one of the slowest, highest-risk paths available — not a shortcut to avoiding "external binary execution," a multi-year systems-engineering undertaking wearing the clothes of one.
• The stated justification (storage bloat, brittle CLI text-parsing, functional-safety determinism) describes a resource-constrained production ECU, not this project's actual deployment target (§1). Cloud Hypervisor and Firecracker already solve "no CLI text-parsing" — both are API-first over a Unix socket, no stdout-scraping involved — without requiring this repo to own vCPU-exit handling.
• Safety-relevant framing ("Freedom from Interference," ISO-26262-flavored mixed-criticality isolation) implies a certification bar this repo has no process for and does not need for its actual research questions. Claiming FFI-grade isolation from a from-scratch, unaudited VMM would be a false claim of rigor this project doesn't have the process to back up — worse than not claiming it at all.

2.3 What's worth keeping

Two ideas in the proposal are genuinely good, decoupled from the in-process-VMM framing:

• A lightweight, sandboxed execution mode for RTOS/MCU-class control-loop nodes is a real gap today — ARCHITECTURE.md §9's guest palette already names Zephyr/FreeRTOS as microVM images for "sensor/ECU nodes," which means today's only option for that zone class is a full microVM boot (qemu-microvm or, per §6/§7, eventually Cloud Hypervisor) even for a node that might genuinely be simulable as a sandboxed control loop with no real kernel underneath. Wasmtime directly (mature, audited, in production use well beyond research contexts) is worth prototyping for exactly this — decoupled from treVM/Ariel-OS specifically, which this repo has no reason to depend on. See §7.3.
• fdt-rs stays a legitimate, cheap-to-adopt-later component if/when this platform boots ARM guests that need synthetic device-tree injection (AAOS on ARM is on the roadmap). Not needed today; no action now; noted for when it is.

3. Use-case-driven methodology (governs how this initiative proceeds from here)

Directive from the project architect, 2026-08-08/2026-08-10: design this bottom-up from real use cases, not top-down from abstract principle. Concretely, the order is:

1. Real YAML use cases first — actual kind: vehicle + kind: baseline manifests representing a genuine software baseline: the virtual appliances/devices it's built from, the resources they need, and the requirements they carry. Not a toy two-device example the way most of today's experiment vehicles are (deliberately minimal, scoped to one hypothesis each) — a topology closer to what a real prototype vehicle's E/E architecture actually looks like: many nodes, many roles, many vxparts references, across all six zone positions.
2. Validate the use case — reviewed by the architect before anything downstream is built on it. §4 below is that draft, awaiting exactly this review. Nothing in §§5–9 is re-derived from §4 yet; §§5–9 predate this methodology (written 2026-08-07, kept because still correct) and §4 is checked against them for consistency, not the other way around, going forward.
3. Only then, functional requirements / abstractions — what does provisioning this topology actually demand of the Rust API layer, derived from what's really in the manifest (how many node kinds, which backends they actually need, what a realistic parts: list references) — not from a hypothetical worst case.
4. Only then, implementation — the provisioning-backend work (Track V's task board, §10).

Why this order, not the reverse: §§5–9's design (spec-translation split, the generic containers module, the CLI-vs-library test) was derived from what already exists in code plus reasoning about what backends are available — sound, but not yet checked against a real, full-scale vehicle topology. A "true prototype car" manifest is the cheapest possible way to find out whether that design actually holds at realistic scale (does the containers module's group-per-pod shape still fit when central-hpc hosts three containerized functions instead of one podman-pod's worth? does VmSpec still cover every zone-class device once there are a dozen of them, not two?) before committing Rust API surface to it.

4. Draft use case: a six-zone reference vehicle (awaiting validation)

This populates a real, already-scoped vxparts backlog item, not an invented example: vxparts/parts/veh/car/config/reference-sedan-6zone (ghcr.io/vxedge/parts/veh/car/config/ reference-sedan-6zone) has existed since Track D as a named, todo-status placeholder — its own README states the two ways to populate it: "author a genuinely new six-zone reference topology… or defer to Track M/vxmodels to render it from SysML." This draft takes the first path, as a concrete, reviewable YAML artifact rather than waiting on SysML tooling that doesn't exist yet.

4.1 Topology

All six canonical zone roles (ARCHITECTURE.md §3) except gateway — consistent with every existing vehicle manifest in vxperiments, none of which declares a gateway zone entry; the gateway is provisioned separately by GatewayService from the experiment's own network/slices, never as a vehicle device. central-hpc hosts the AD compute stack; the four corner zones each carry a small, realistic device mix (an ECU plus a sensor or actuator) rather than one device each, per the "so many vxparts, true prototype car" framing this draft is checking §§5–9 against.

Honesty constraint, stated once here rather than per line below: today's only real, live, registry-verified zone-class guest bundle is zone/linux/stack/probe-bundle:0.1.0 (sha256:33c9c8d8625bd5c6d1f4d3d0c561fdaa930e057fcd1fb24fb3badb402f25429d, re-verified via oras resolve while drafting this). QNX/AAOS/Zephyr have zero published parts — confirmed by listing vxparts/parts/zone/{qnx,aaos,zephyr}/os/, all empty. hpc/ad/stack/ad-baseline (the real composite this draft wants for central-hpc) has real source files in the vxparts repo but failed live oras resolve (ghcr.io/vxedge/parts/hpc/ad/stack/ad-baseline:0.1.0: not found) — its own composite-publish script requires every constituent (including hpc/ad/config/ autoware-params) published first, and at least one hasn't been. So every device below uses probe-bundle as its actual bundle today, with a comment naming the RTOS/guest image a real deployment would eventually use once that part exists — exactly vx-vehicle-01's own precedent ("one device is enough… without waiting on a production-representative guest image"), extended to six zones instead of one.

Deliberately not included: an AAOS/IVI device (no real or realistically-composable part exists yet for it — adding one here would mean inventing a fictional part ref, which this draft won't do) and svc-class parts (kuksa-databroker, vxota) — those are experiment-level services: block entries per §6.0's "vehicle names a slice; the experiment defines it" rule extended to services, not vehicle-manifest functions:, so they belong in a companion experiment manifest, not here.

4.2 vehicle.yaml (draft)

kind: vehicle
id: vx-reference-sedan-01
class: road
subtype: car

# Six-zone reference topology (vxparts veh/car/config/reference-sedan-6zone). Every zone-class
# device uses the one real, registry-verified guest bundle (zone/linux/stack/probe-bundle)
# today; the `role` below is the honest structural label this manifest is making a claim about
# -- the differentiated RTOS image that role implies (QNX for ecu, Zephyr/FreeRTOS for
# sensor/actuator) does not exist as a published vxparts part yet. Grow this file part-by-part
# as those land, per vx-vehicle-01's own precedent.
zones:
  front-left: {}
  front-right: {}
  rear-left: {}
  rear-right: {}
  central-hpc: {}

devices:
  fl-safety-ecu:      { role: ecu,      zone: front-left,  backend: qemu-microvm }
  fl-camera-sensor:   { role: sensor,   zone: front-left,  backend: qemu-microvm }
  fr-body-ecu:        { role: ecu,      zone: front-right, backend: qemu-microvm }
  fr-steer-actuator:  { role: actuator, zone: front-right, backend: qemu-microvm }
  rl-radar-sensor:    { role: sensor,   zone: rear-left,   backend: qemu-microvm }
  rr-brake-actuator:  { role: actuator, zone: rear-right,  backend: qemu-microvm }
  hpc-ad-01:          { role: hpc,      zone: central-hpc, backend: container }

functions:
  fl-safety-fn:     { deployed_on: fl-safety-ecu }
  fl-camera-fn:     { deployed_on: fl-camera-sensor }
  fr-body-fn:       { deployed_on: fr-body-ecu }
  fr-steer-fn:      { deployed_on: fr-steer-actuator }
  rl-radar-fn:      { deployed_on: rl-radar-sensor }
  rr-brake-fn:      { deployed_on: rr-brake-actuator }
  hpc-ad-fn:        { deployed_on: hpc-ad-01 }

links:
  - id: link-fl-safety
    endpoints: [{ node: fl-safety-ecu, port: eth0, ip: 10.120.1.11/24 }]
    slice: adas
  - id: link-fl-camera
    endpoints: [{ node: fl-camera-sensor, port: eth0, ip: 10.120.1.12/24 }]
    slice: adas
  - id: link-fr-body
    endpoints: [{ node: fr-body-ecu, port: eth0, ip: 10.120.2.11/24 }]
    slice: adas
  - id: link-fr-steer
    endpoints: [{ node: fr-steer-actuator, port: eth0, ip: 10.120.2.12/24 }]
    slice: adas
  - id: link-rl-radar
    endpoints: [{ node: rl-radar-sensor, port: eth0, ip: 10.120.3.11/24 }]
    slice: adas
  - id: link-rr-brake
    endpoints: [{ node: rr-brake-actuator, port: eth0, ip: 10.120.4.11/24 }]
    slice: adas
  - id: link-hpc-ad
    endpoints: [{ node: hpc-ad-01, port: eth0, ip: 10.120.5.11/24 }]
    slice: adas

4.3 parts.yaml (draft baseline)

kind: baseline
id: bl-vx-reference-sedan-01

parts:
  # Real, live, registry-verified (re-checked via `oras resolve` while drafting this section).
  - ref: ghcr.io/vxedge/parts/zone/linux/stack/probe-bundle:0.1.0
    digest: sha256:33c9c8d8625bd5c6d1f4d3d0c561fdaa930e057fcd1fb24fb3badb402f25429d
    target_ids: [fl-safety-fn, fl-camera-fn, fr-body-fn, fr-steer-fn, rl-radar-fn, rr-brake-fn]

  # NOT verified live -- `oras resolve` on this tag failed (2026-08-10, "not found"). Source
  # files exist in vxparts but the composite has not actually been published; at least one
  # constituent (hpc/ad/config/autoware-params) is itself unpublished. Do not treat this
  # manifest as executable until this entry is either resolved live or replaced.
  - ref: ghcr.io/vxedge/parts/hpc/ad/stack/ad-baseline:0.1.0
    digest: "TBD -- unverified, see note above"
    target_ids: [hpc-ad-fn]

4.4 What validating this means

Not "does it parse" — vxedge experiment validate-shaped correctness is necessary but not the point. Validation here means the architect confirming: (a) this is a realistic-enough shape for a prototype car (device mix per zone, role choices, whether seven devices is the right scale or too few/many for what "so many vxparts" was asking for); (b) the honesty framing (real vs. aspirational parts) is the right way to handle the QNX/AAOS/Zephyr/ad-baseline gaps, versus some other resolution (e.g., blocking on publishing ad-baseline for real first); (c) whether a second draft use case is needed (e.g., one that includes an experiment-level services: block alongside this vehicle, showing kuksa-databroker/vxota in their correct place) before moving to §3 step 3.

5. What already exists (do not re-derive)

• crates/vxedge-core/src/zones/backend.rs — ZoneBackend trait, three implementations (ContainerBackend netns-only, QemuMicrovmBackend raw QEMU CLI, LibvirtBackend virsh CLI), one dispatch site backend_for(kind). kernel_cmdline/bundle_paths are already pure, shared spec-translation functions — the split principle below extends what's already there.
• crates/vxedge-core/src/fiveg/lifecycle.rs — a separate, pod-shaped podman CLI integration for the 5G core, unrelated to ZoneBackend.
• NodePlan (zones/plan.rs) — pure desired-state data, already carries bundle/remote_bundle (mutually exclusive, "planning records intent only, never fetches") as the precedent for any new optional per-backend field.
• Shell (gateway/netops.rs) — the CLI-shelling primitive, escalate injected not computed.
• Sequential, single-node-at-a-time provisioning (stage_zones in crates/vxedged/src/ experiment.rs) is deliberate: exp-002b showed concurrent guest startup causes CPU contention that invalidates measurement campaigns. Nothing in this doc changes that.

6. The provisioning stack this repo is actually building

                    vxedge-core (in-process, pure-Rust orchestration)
   ┌──────────────────────────────────────────────────────────────────────┐
   │  ZoneBackend seam (zones/backend.rs)     containers/ module (new)    │
   └──────────────────────────────────────────────────────────────────────┘
        │                    │                    │              │
        ▼                    ▼                    ▼              ▼
 ┌─────────────┐     ┌──────────────┐     ┌──────────────┐  ┌───────────┐
 │  libvirt     │     │ qemu-microvm │     │Cloud Hyperv- │  │  bollard  │
 │  (virt crate)│     │ (unchanged,  │     │ isor client  │  │ (Podman   │
 │              │     │  no further  │     │ (Phase B,    │  │  Docker-  │
 │              │     │  investment) │     │  spiked)     │  │  compat)  │
 └──────┬───────┘     └──────┬───────┘     └──────┬───────┘  └─────┬─────┘
        │                    │                    │                │
        ▼                    ▼                    ▼                ▼
   libvirtd,           /dev/kvm via         /dev/kvm via       Podman daemon,
   in-process FFI,      QEMU's own           CH's own          rootless user
   no sudo needed        process              process             socket

Every box here is an existing, audited, already-maintained VMM or container engine, reached via its own stable API (FFI or a Unix-socket REST client) instead of a CLI's stdout. None of it requires vxedge-core to own /dev/kvm directly, a vCPU thread loop, or a VirtIO ring-buffer implementation — that engineering already exists, upstream, security-audited, and this repo consumes it rather than re-derives it.

7. Design principles

7.1 Split spec translation from launch, at the grain that's real

One pure VmSpec (kernel/initrd/cmdline/pinned_cpus/endpoint) shared by every VM-shaped engine (qemu-microvm today, libvirt, Cloud Hypervisor once spiked); one pure ContainerSpec (image/entrypoint/env) for container-shaped engines. Neither is part of ZoneBackend's object-safe trait signature (an associated type would break &dyn ZoneBackend) — each is an internal discipline a backend module follows, consolidating what kernel_cmdline/bundle_paths already do today into one named, reusable spec per execution model.

7.2 A generic container-provisioning module — not owned by fiveg, not owned by zones

New module: crates/vxedge-core/src/containers/mod.rs. Owns a bollard-backed ContainerClient and a ContainerGroup primitive (an infra container providing a shared netns, joined via NetworkMode: container:<infra_id> — bollard has no native "pod" concept, so this makes the group explicit). Consumed by two real call sites: fiveg::lifecycle (today's podman CLI calls become ContainerClient calls) and the zone container backend's future OCI payload (currently netns-only). Two real call sites from day one is what earns this as a seam under this repo's own service-boundaries rule ("two implementations, or two call sites") — not a speculative abstraction.

7.3 A lightweight capsule mode for control-loop nodes — prototyped, not assumed

Decoupled from treVM/Ariel-OS: a WasmCapsuleBackend (fourth ZoneBackend impl, gated behind its own manifest backend: wasm-capsule value) that runs a .wasm module via wasmtime directly, in-process, for zone nodes that are genuinely simulable as a sandboxed control loop rather than a full guest kernel — a real, currently-unserved gap for the Zephyr/FreeRTOS "sensor/ECU node" class (§9's own guest palette). This is a prototype-and-evaluate item, not committed infrastructure: build one, run it against one real sensor-node manifest, and decide based on results whether it earns permanent status. wasmtime itself is the only new dependency this needs — mature, widely audited, no in-process-VMM ioctl work required.

7.4 Considered and rejected

• Runtime backend registry. backend_for's static match already satisfies "dispatch once, at one site." A registry only earns its complexity if selection needs to be dynamic; Backend is a closed, manifest-declared enum — it doesn't.
• Typestate lifecycle enum (Defined→Starting→Running→Stopped as distinct types). Competes with NodeState/Observable as a second source of truth for the same fact; the reconciler model (up/down converge desired vs. observed) doesn't want a client-side FSM.
• In-process raw-KVM hypervisor. §2.

7.5 CLI vs. library — the decision test

Does adopting a backend's library avoid requiring anything beyond what the testbed already installs for other reasons, or is there no meaningful CLI to shell out to in the first place (the backend is API-first)? If yes to either, the library wins.

8. Testing approach (TDD, three tiers — this repo's existing convention, applied here)

1. Unit — pure spec-translation logic first, before any backend touches it. VmSpec/ ContainerSpec construction from a NodePlan, and the containers module's ContainerGroup naming/idempotency logic, are pure functions — write their tests before the bollard/virt call that consumes them, exactly the discipline core_run_specs already demonstrates in fiveg::lifecycle (pure, unit-tested without touching podman).
2. Unprivileged e2e in plain cargo test. Anything that can run without root/KVM/a real daemon (error-path construction, idempotency-check logic against a fake NodeState) stays in the default cargo test run.
3. Privileged/environmental behind #[ignore], with a reason string, mirroring gateway-verification's existing convention: a live libvirt domain define/start/destroy, a live Podman group stand-up, a live Cloud Hypervisor VM boot. Plain cargo test must always pass unprivileged.

Every new backend lands with all three tiers present before being wired into backend_for's match — not after, per this repo's existing rust-patterns house rule.

9. Roadmap

See §10 for the phased, numbered task list (Track V + Track F) this document's design maps to. Summary: Track V phase 1 (containers module + fiveg migration, libvirt→virt crate) is scoped and ready to start once §4's use case is validated; Track V phase 2 (Cloud Hypervisor spike) follows once phase 1 lands; the WASM-capsule prototype (§7.3) is an independent, parallel-track evaluation; the in-process-VMM direction (§2) is recorded as reviewed and rejected, not silently dropped, so a future session doesn't re-propose it without this document's reasoning in view; Track F (multi-vehicle fleet layer) does not start until Track V is at a high success rate.

10. Task board

Moved here from docs/ROADMAP.md 2026-08-10, consolidating design and task status into one file. Status as of 2026-08-10: design phase. §4's use case is drafted and awaiting validation; no Track V or Track F item has started implementation. This section is the live status — update it in place as items move, rather than appending a separate changelog.

Track V — node-provisioning engine (single vehicle)

Extends Track C (docs/ROADMAP.md), gates nothing in phases 1–3. Blocked on §4's use case being validated (§3) before V0/V1 implementation starts — the use case is what confirms the design in §§5–9 actually holds at realistic scale before Rust API surface commits to it.

• V0 — LibvirtBackend: virsh CLI → virt crate (vxedge-core::zones::backend). Removes the last CLI-shell privilege-escalation exposure on the VM side entirely (in-process FFI against libvirtd, group-membership/polkit authorized, no sudo at all — stronger than the qemu-microvm tap-ownership fix, docs/ROADMAP.md 2026-08-06 entry). New build-time dependency on libvirt headers (libvirt-dev) — add to docs/TESTBED-SETUP.md Tier 2 in the same change, not as a follow-up. libvirt_domain_xml's XML generation is untouched (pure, transport-independent); only virsh()'s CLI transport is replaced.
• V1 — generic containers module (crates/vxedge-core/src/containers/mod.rs, new): a bollard-backed ContainerClient + ContainerGroup primitive (infra container owns the shared netns; joiners attach via NetworkMode: container:<infra_id>, since bollard has no native pod concept). Two real call sites from day one, satisfying service-boundaries' own "two call sites" rule for a real seam, not a speculative one: fiveg::lifecycle (V2, below) and the zone container backend's future OCI payload (V3). Must connect to the rootless user Podman socket specifically, never the system socket — a different identity resolves a different, possibly-unbuilt image store (real incident, 2026-07-17, documented in fiveg::lifecycle's history). docs/TESTBED-SETUP.md gains the socket enablement line (systemctl --user enable --now podman.socket) in the same change.
• V2 — migrate fiveg::lifecycle's podman CLI calls (podman(), pod_pid, pod_exists, run_detached, pod_containers) onto V1's ContainerClient. core_run_specs and the veth/bridge/nsenter networking (host_ip/nsenter_run/setup_transit_link/ setup_upf_tun) stay in fiveg unchanged — N2/N3-equivalent networking built on top of two container groups, not container management itself, does not move to containers/. Async ripple: fiveg_up/bridge_up/ue_up/fiveg_down become async fn; the four blocking(...) wrappers around them in vxedged::services::ZoneSvc::lifecycle come off in favor of direct .await.
• V3 — zone container backend gets a real OCI payload via V1's ContainerClient (currently netns-only, zones::backend::container_up). Open design question to resolve before starting, not mid-implementation: bollard has no "join a namespace by name" primitive (only "join another container's namespace by ID/PID") — likely direction is inverting control so the group's infra container provides the netns and the zone backend's veth peer moves in by the infra container's PID instead of by name.
• V4 — spike CloudHypervisorBackend (fourth ZoneBackend impl) using the cloud-hypervisor-client crate (OpenAPI-generated against CH's own REST-over-UDS API). Recommended over Firecracker for this project specifically: CH's richer device model (PCI, CPU/memory/device hotplug) is a better fit for the guest palette actually on the roadmap (QNX SDP 8, AAOS, Zephyr/FreeRTOS — ARCHITECTURE.md §9) than Firecracker's deliberately spartan five-device model, and no mature/canonical Firecracker Rust client was confirmed during evaluation (2026-08-07) where cloud-hypervisor-client was. Scoping caution: CH is headless — AAOS's own §9 line requires "virtio-gpu or GPU passthrough," so that guest stays on libvirt/QEMU regardless of this item. Validate first against the already-live-tested linux-probe bundle before attempting a richer guest; don't assert QNX-on-CH works without that empirical step. Companion decision, not a separate item: once V4 lands, stop further investment in QemuMicrovmBackend beyond its current (2026-08-06) fixed state — raw qemu-system-x86_64 has no comparable typed Rust binding to reach for (QMP is a hand-rollable protocol, not a maintained crate), and CH is the gap that closes.
• V5 — prototype WasmCapsuleBackend (fifth ZoneBackend impl, backend: wasm-capsule): run a .wasm module via wasmtime directly, in-process, for zone nodes simulable as a sandboxed control loop rather than a full guest kernel — a real, currently-unserved gap for the Zephyr/FreeRTOS "sensor/ECU node" class (today's only option is a full microVM boot). Deliberately decoupled from the external proposal's treVM/Ariel-OS coupling — wasmtime alone is the only new dependency, mature and audited independent of that pairing. Evaluate-and-decide, not committed infrastructure: build one, run it against one real sensor-node manifest, decide based on results whether it earns permanent status.
• V6 (parked, not scheduled) — fdt-rs for synthetic device-tree injection, if/when an ARM guest (AAOS on ARM) needs it. No action until that's a real, scheduled guest target.

Explicitly rejected, recorded so it isn't re-proposed without this reasoning in view: a from-scratch, in-process KVM hypervisor (raw kvm-ioctls VM/vCPU management, hand-rolled VirtIO ring buffers, custom seccomp filters) as this project's provisioning backend. Full reasoning in §2.

Track F — vehicle-fleet layer (multiple concurrent vehicle instances over vxAPI)

Hard gate, stated by the architect 2026-08-08: Track F cannot start until Track V is done with a high success rate. Not a parallel track — provisioning one vehicle reliably is the prerequisite for provisioning many. Nothing below is scheduled until that gate opens.

Real current blocker, confirmed in code 2026-08-08 (not speculative): EdgeState.active: Option<ActiveExperiment> (crates/vxedged/src/services.rs:66) is a single slot, and GatewayPlan::from_model hardcodes "vxbr0"/"vxup0"/"vxup0p" as literal bridge/tap names (crates/vxedge-core/src/gateway/plan.rs:461-463). Two concurrent vehicle up calls today collide on the same bridge, full stop — this has to be fixed before any fleet layer can exist above it, not as an implementation detail underneath one.

• F0 (blocked on Track V) — per-instance resource namespacing: derive bridge/tap/cache/ runtime-dir names from the vehicle id (reuse sha2, already a workspace dependency, for a short stable hash — no new hashing crate) instead of the current hardcoded vxbr0-family literals.
• F1 (blocked on Track V) — EdgeState.active becomes a collection (HashMap<VehicleId, ActiveInstance> or similar) instead of Option<ActiveExperiment>'s single slot.
• F2 (blocked on F0/F1) — a vehicle-instance registry: dashmap (mature, sharded concurrent map — matches this codebase's own "don't reinvent a concurrent map" instinct) keyed by vehicle id, populated/removed by VehicleService's existing up/down, queried by a new fleet-scoped RPC surface (list/status-all). Explicitly not an actor/supervision framework (ractor was considered and set aside 2026-08-08): for a handful of concurrent research-testbed vehicle instances, plain tokio::task::JoinSet + dashmap is proportionate and consistent with every other concurrency choice already in this codebase (rust-patterns' own mutex/channel rules) — revisit only if real fleet scale (dozens+, genuine fault-isolation needs between instances) materializes.
• F3 (blocked on F2) — fleet-scoped vxAPI surface (list active vehicles, status-all, bulk down) — proto design not started; needs its own pass once F0–F2 land and the shape of "an instance" (what state it carries beyond today's per-vehicle VehicleStatus) is clear from real use.

Versioning and changelogs

Each repo under github.com/vxedge/ is an independent git repo with its own version history (see workspace CLAUDE.md) — there is no single "vxedge version." This document is the convention every repo follows so an OEM engineer can clone one, read one file, and know exactly what they're getting.

Rules

1. SemVer per repo. Tags are vMAJOR.MINOR.PATCH (e.g. v0.1.0), following Semantic Versioning 2.0.0. Pre-1.0 (all repos today): breaking changes bump MINOR, everything else bumps PATCH — normal SemVer-0.x practice.
2. CHANGELOG.md per repo, Keep a Changelog format: an ## [Unreleased] section accumulates entries as work lands; cutting a tag moves that section under a dated version heading and opens a fresh ## [Unreleased]. Categories: Added, Changed, Fixed, Deprecated, Removed, Security. Every CHANGELOG.md gets a repo-local AGENTS.md/CLAUDE.md pointer alongside the other docs it already lists.
3. No tag before the entry exists. A version is cut by moving [Unreleased] content under a new heading in the same commit as the tag — never tag first and backfill later.
4. v0.1.0 cut 2026-07-13 — the phase-1 gate closed that day (H1+H3+H4 numbers in hand). Tagged: docs, vxedge, vxagents, vxtelemetry, vxota, vxperiments, vxzones — the seven repos that actually delivered phase-1 content. Not tagged: vxcompute, vxgo, vxslice — empty, a tag on nothing is meaningless. Tagging and visibility are decoupled: a tag is cut the moment a milestone is real, independent of whether the repository is public yet.
5. Compatibility across repos. The only thing that couples repos is vxAPI (vxedge/proto/vxedge/v1/*.proto) — a breaking proto change is a MINOR bump (pre-1.0) in vxedge and must be called out in every dependent repo's next CHANGELOG.md entry (vxagents, vxtelemetry, vxota all speak vxAPI-adjacent conventions even before they use the client SDK directly). Manifest schema changes (ARCHITECTURE.md §6) are the second coupling point — same rule.
6. Reproducibility tags stay separate from release tags. Experiment results already pin an exact state via metadata.json (manifest SHA-256, and now tool_versions — see vxperiments/scripts/run-exp-002b.sh's env_metadata() for the current shape) — that's a finer-grained provenance mechanism than a repo release tag and is not replaced by this policy; a future result should record tool_versions per repo (already 3 of 3 in place for the exp-002b runner: vxedge, vxagents, vxtelemetry) rather than a coarser release tag, since a campaign can run between tags.

What OEM-adoption needs beyond this doc

Cutting tags and writing changelogs by hand doesn't scale past the first release, and none of the CI automation for it exists yet (no git-cliff/Conventional-Commits enforcement, no tag-triggered GitHub Release workflow, no cross-repo compatibility-matrix generation). That work is tracked as its own devops board section — see docs/ROADMAP.md "Release engineering & versioning (devops)" — rather than implemented ad hoc here.

Acronyms and glossary

Every acronym used across ARCHITECTURE.md and the other program docs, explained in plain language and grouped by field. Not auto-generated — update it by hand alongside ARCHITECTURE.md when a new acronym enters the vocabulary (same discipline as AGENTS.md's other cross-referenced docs).

Two acronyms are genuinely ambiguous in this project's own docs, depending on context — flagged inline where they occur: UDS (Unix Domain Socket vs. Unified Diagnostic Services) and OSI (ASAM's simulator interface vs. the classic 7-layer networking model — vxedge only ever means the former).

5G / RAN / mobile core

The vxedge core-network/RAN stack (Open5GS + UERANSIM), relevant to H2 and the open5gs-ueransim-patterns skill.

• 5G — Fifth-generation cellular network standard; the slicing technology H2 tests.
• RAN — Radio Access Network. The base-station side of the network (here: UERANSIM's simulated gNB).
• CN — Core Network. The non-radio side (AMF, SMF, UPF, etc.) — here: Open5GS.
• gNB — "next-generation NodeB," the 5G base station.
• UE — User Equipment. The device connecting to the network — here, a simulated modem (UERANSIM) inside a zone/pod, not a phone.
• UERANSIM — Open-source UE + RAN simulator; vxedge's RAN implementation.
• PDU (session) — Packet Data Unit session: the data connection a UE establishes to reach a network (roughly, "the tunnel that carries its traffic").
• S-NSSAI — Single Network Slice Selection Assistance Information: the identifier for one network slice, made of an SST + optional SD.
• SST — Slice/Service Type: the broad category of a slice (e.g. "URLLC-like").
• SD — Slice Differentiator: a finer-grained tag distinguishing slices that share an SST.
• URSP — UE Route Selection Policy: rules that tell a UE which slice to send a given traffic class over.
• URLLC — Ultra-Reliable Low-Latency Communication: the 5G traffic class the ADAS slice is modeled on.
• DNN — Data Network Name: the network a PDU session connects to (successor to the older term APN, Access Point Name).
• AMF — Access and Mobility Management Function: handles UE registration/mobility.
• SMF — Session Management Function: manages PDU sessions.
• UPF — User Plane Function: actually forwards user traffic (the data-plane NF).
• NRF — Network Repository Function: lets core NFs discover each other.
• AUSF — Authentication Server Function.
• UDM / UDR — Unified Data Management / Unified Data Repository: subscriber data functions.
• PCF — Policy Control Function.
• BSF — Binding Support Function.
• NSSF — Network Slice Selection Function: picks the right slice for a UE.
• NF — Network Function: generic term for any of the boxes above.
• SBI — Service-Based Interface: the HTTP-based protocol core NFs use to talk to each other.
• PFCP — Packet Forwarding Control Protocol: the SMF↔UPF control channel.
• NGAP — NG Application Protocol: the signaling protocol between gNB and AMF.
• NAS — Non-Access Stratum: UE↔core signaling that rides over, but is logically independent of, the radio link.
• N2 / N3 — 3GPP reference points: N2 is gNB↔AMF (control), N3 is gNB↔UPF (data) — vxedge's CN-pod↔RAN-pod transit subnet stands in for both.
• GTP / GTP-U — GPRS Tunneling Protocol (User plane): how UPF tunnels a UE's actual traffic.
• SCTP — Stream Control Transmission Protocol: the transport NGAP rides on.
• SA — Standalone (5G SA = a 5G core end-to-end, not anchored on 4G) — the architecture the DT/BMW/Valeo/Ericsson/Qualcomm slicing trial vxedge cites reproduced.
• V2X — Vehicle-to-Everything: umbrella term for vehicle wireless comms.
• V2V / V2I — Vehicle-to-Vehicle / Vehicle-to-Infrastructure.
• RSU — Roadside Unit: a V2I endpoint (phase-3 node type).
• PC5 — The 3GPP sidelink interface used for direct V2X (no core network hop).

SDN & Linux networking (vxgateway)

The vxgateway data plane, relevant to network-provisioning-patterns.

• SDN — Software-Defined Networking: separating the control logic from the packet forwarding — vxgateway's whole premise.
• OVS — Open vSwitch: a software switch; one of vxgateway's two bridge backends.
• OVSDB — Open vSwitch Database protocol: how OVS is configured/queried.
• VLAN — Virtual LAN: tags that separate traffic on one physical/virtual wire — how vxgateway maps slices today (mode: vlan).
• VXLAN — Virtual Extensible LAN: tunnels VLAN-style segments across hosts.
• HTB — Hierarchical Token Bucket: the Linux tc queueing discipline vxgateway uses for per-slice bandwidth guarantees.
• rtnetlink — The Linux kernel's netlink socket family for network configuration; the planned successor to today's iproute2-shell-out backend.
• FRR — FRRouting: open-source routing daemon suite (provides vxgateway's BGP).
• BGP — Border Gateway Protocol: the routing protocol FRR speaks for cloud interconnect.
• TSN — Time-Sensitive Networking: deterministic-latency Ethernet extensions (phase-3 gateway variant).
• gPTP / PTP — (generalized) Precision Time Protocol: clock sync TSN depends on.
• NOS — Network Operating System: SONiC (SONiC-VS) is vxedge's "production NOS twin."
• BASE-T1 (100BASE-T1 / 1000BASE-T1) — Automotive Ethernet PHY standards vxgateway's link impairments emulate.
• DPDK — Data Plane Development Kit; VPP — Vector Packet Processing (FD.io): a kernel-bypass data-plane option for URLLC-focused runs.
• DOCA — NVIDIA's SDK for offloading OVS/tc-style intent onto a BlueField DPU (phase-4 hardware exit ramp).
• CAN — Controller Area Network; LIN — Local Interconnect Network: legacy automotive buses, modeled via type: can links (phase 3).
• SocketCAN / vcan — Linux's native CAN subsystem / its virtual-CAN device.
• DBC — A CAN database file format defining frame layouts (used by restbus simulation to map CAN ↔ VSS).

SDV & automotive E/E architecture

• SDV — Software-Defined Vehicle: a vehicle whose behavior is primarily defined by software, not fixed wiring — vxedge's whole subject.
• E/E — Electrical/Electronic (architecture): how a vehicle's electronics are wired and organized (zonal vs. domain-based, etc.).
• ECU — Electronic Control Unit: an embedded computer controlling one vehicle function — what vxedge's "zones" stand in for.
• ADAS — Advanced Driver-Assistance Systems: the URLLC-like traffic class H1/H2 test.
• AD — Autonomous Driving.
• HMI — Human-Machine Interface. In-vehicle: Android Automotive (AAOS) or an embedded Qt/QML IVI/cluster loadable — two real, alternate infotainment-zone guest baselines (§9). There is no Qt/QML operator dashboard (corrected 2026-08-06); any platform monitoring dashboard is web-based or the CLI (ARCHITECTURE.md §5.1).
• AUTOSAR — AUTomotive Open System ARchitecture: the standard automotive software architecture vxedge's QNX/C++ conventions are described as "adjacent" to.
• OEM — Original Equipment Manufacturer: the vehicle maker — vxedge's target audience/adopter.
• VSS — Vehicle Signal Specification: COVESA's standard catalog of vehicle signal names (speed, door state, etc.), used via the Kuksa databroker.
• COVESA — Connected Vehicle Systems Alliance: the standards body behind VSS/Kuksa.
• BMS — Battery Management System: one of the comfort/EV zone workloads modeled.
• HVAC — Heating, Ventilation, and Air Conditioning: another comfort-zone workload.
• GMSL / FPD-Link — Gigabit Multimedia Serial Link / Flat Panel Display Link: the physical transport real camera/sensor data rides in a vehicle (vxedge substitutes slice traffic through vxgateway instead).
• UDS (automotive sense) — Unified Diagnostic Services: the ISO 14229 diagnostic protocol behind the phase-3 UDS/DoIP probe. Not to be confused with the networking sense below.
• DoIP — Diagnostics over IP: carries UDS diagnostics over Ethernet/IP.
• SAE — Society of Automotive Engineers: standards body (e.g. the WCX venue).

Virtualization & zones

• VM — Virtual Machine.
• KVM — Kernel-based Virtual Machine: the Linux hypervisor vxedge builds on.
• QEMU — Quick EMUlator: the machine emulator KVM uses; also runs standalone as the qemu-microvm zone backend.
• QMP — QEMU Machine Protocol: how the microVM backend controls QEMU directly.
• microVM — A minimal-device-model QEMU machine type, for lower overhead than a full VM.
• QNX / SDP — QNX Software Development Platform 8: the real-time OS used for ADAS/safety zone guests.
• RTOS — Real-Time Operating System (Zephyr, FreeRTOS, QNX).
• AAOS — Android Automotive OS: the infotainment zone guest.
• MCU — Microcontroller Unit: the class of chip Renode-backed zones emulate at register level.
• GPIO / I2C / SPI / PWM / ADC — Low-level hardware I/O buses/peripherals (General-Purpose I/O, Inter-Integrated Circuit, Serial Peripheral Interface, Pulse Width Modulation, Analog-to-Digital Converter) — the register-level I/O layer QEMU device models / Renode expose to guests.
• NAND / UBI — Flash storage technology; came up as a reason Rugix (not RAUC) fits vxedge's QEMU/libvirt block-storage bundles (no real NAND to manage).

Compute, AI & GPU (vxparts: hpc-class parts)

• GPU — Graphics Processing Unit.
• CUDA — NVIDIA's parallel-compute platform/API.
• TensorRT — NVIDIA's inference-optimization SDK (DeepStream pipelines target it).
• DCGM — Data Center GPU Manager; NVML — NVIDIA Management Library: how GPU utilization/VRAM/thermal metrics get collected.
• VRAM — Video RAM (GPU memory) — one of the tracked GPU metrics.
• MIG — Multi-Instance GPU: NVIDIA's hardware GPU-partitioning feature (resources.gpu: mig-slice).
• ROS2 — Robot Operating System 2: the middleware Autoware runs on.
• DDS — Data Distribution Service: the pub/sub protocol standard under ROS2.
• rmw — ROS MiddleWare: ROS2's pluggable transport layer (rmw_zenoh vs. rmw_fastrtps).
• CARLA — Open-source autonomous-driving simulator (proper name, not an acronym); the source of realistic sensor/perception traffic.
• LLM — Large Language Model: relevant to track M's GenAI-authored-experiment goal.

API & software architecture (vxAPI)

• API — Application Programming Interface.
• gRPC — Google's RPC framework (originally "gRPC Remote Procedure Calls"); the transport vxAPI is built on.
• RPC — Remote Procedure Call: a function call made across a network boundary.
• IDL — Interface Definition Language: the Protobuf .proto files describing vxAPI's contract.
• Protobuf — Protocol Buffers: Google's binary serialization format, the wire format gRPC uses.
• SDK — Software Development Kit: the generated per-language vxAPI client libraries (Rust/tonic, TypeScript/connect-es).
• CLI — Command-Line Interface: the vxedge binary, a thin vxAPI client.
• UDS (networking sense) — Unix Domain Socket: vxedge's default local transport between the CLI and vxedged. Not to be confused with the automotive sense above.
• mTLS — mutual Transport Layer Security: both client and server present certificates — vxedge's remote-transport auth.
• TLS — Transport Layer Security: the encryption protocol mTLS builds on.
• OIDC — OpenID Connect: the bearer-token auth scheme for remote vxAPI access.
• REST — Representational State Transfer: the alternative API style vxAPI deliberately isn't (it's gRPC), though an optional REST façade is possible later.
• gRPC-Web — A gRPC variant browsers can call directly; how the web dashboard/Electron frontends reach vxAPI.
• JSON — JavaScript Object Notation: used for --json CLI output and iproute2's machine-readable reads.
• YAML — the manifest file format (experiment/vehicle manifests).
• FFI — Foreign Function Interface: cross-language function calling (relevant to any native bindings).
• OCI — Open Container Initiative: the standard container image format Podman images conform to.
• GHCR — GitHub Container Registry: where container images get published with digests.
• MCP — Model Context Protocol: the open standard AI agent tooling (Claude Code and others) uses to call external tools. vxmcp (phase 3) is vxAPI's MCP server — a thin frontend for agents, the same role the CLI plays for a human at a terminal.
• MRTR — Multi Round-Trip Request: MCP spec 2026-07-28's mechanism for a tool call that needs a follow-up (e.g. a confirmation) before it can complete, without a held-open bidirectional stream — how vxmcp's mutating tools ask their caller to confirm before running, the agent-tooling equivalent of the CLI's own explicit verbs.
• rmcp — the official Rust SDK for MCP (modelcontextprotocol/rust-sdk); vxmcp's chosen implementation, reached spec-2026-07-28 Tier-1 conformance 2026-07-31.

Standards, modeling & simulation (track M)

• SysML — Systems Modeling Language: the modeling language vxmodels authors in (v2 specifically — textual, git-diffable).
• OMG — Object Management Group: the standards body that owns SysML.
• MBSE — Model-Based Systems Engineering: the discipline SysML/track M applies.
• INCOSE — International Council on Systems Engineering: an MBSE-adjacent standards community.
• ASAM — Association for Standardization of Automation and Measurement Systems: the body behind OSI/OpenSCENARIO.
• OSI (ASAM sense) — Open Simulation Interface: the boundary standard between a world simulator (CARLA) and the AD stack (phase 4). vxedge never means the classic networking "7-layer OSI model."
• OpenSCENARIO — ASAM's standard format for traffic/scenario definitions (phase-4 mapping target for CARLA scenarios).
• FMI / FMU — Functional Mock-up Interface / Functional Mock-up Unit: the standard for importing plant models (e.g. Simulink-exported) into vxplant.
• XIL — X-in-the-Loop: umbrella term for SIL (Software-in-the-Loop) and HIL (Hardware-in-the-Loop) testing — vxedge aims to speak the same bench-API semantics as OEM HIL rigs.

OTA & security (vxota)

• OTA — Over-The-Air (update): vxota's whole subject, and H4's workload.
• Uptane — An automotive-specific, TUF-derived OTA security framework; vxota follows its metadata shape without depending on its reference implementation.
• TUF — The Update Framework: the general (non-automotive) update-security spec Uptane derives from.
• RAUC — Robust Auto-Update Controller: an established atomic-update tool, considered as vxota's phase-3 slot-update fallback.
• Rugix (Ctrl) — A Rust-native atomic-update tool; vxota's preferred phase-3 candidate over RAUC (Rust-policy fit).

Methodology, reproducibility & DevOps

• CI / CD — Continuous Integration / Continuous Delivery: automated build/test/publish pipelines (GitHub Actions here).
• PR — Pull Request.
• RTT — Round-Trip Time: the core latency metric vxprobe measures.
• IQR — Interquartile Range: the spread measure used to judge whether a rerun's numbers are "reproducible" (within the original's IQR).
• p50 / p95 / p99 — Percentile latency notation (the value below which 50%/95%/99% of samples fall) — not an acronym, but ubiquitous enough to define: p99 is the "worst-case-ish" tail latency figure.
• RSS — Resident Set Size: a process's actual in-memory footprint, one of H3's resource-cost metrics.
• SHA256 — The cryptographic hash algorithm used to digest-tag every manifest, image, and archive for reproducibility.
• CSV — Comma-Separated Values: the tabular result-export format.
• SemVer — Semantic Versioning: the MAJOR.MINOR.PATCH convention every repo follows.
• LTS — Long-Term Support (e.g. an OS/toolchain release channel).
• PPA — Personal Package Archive: an Ubuntu third-party package source (the now-removed host Open5GS install came from one).
• NOPASSWD — A sudoers option letting a specific command run without a password prompt — how vxedged gets CAP_NET_ADMIN-requiring privileges without running as root.

Changelog

All notable changes to this repo are documented here. Format follows Keep a Changelog; versioning follows Semantic Versioning — see VERSIONING.md.

[Unreleased]

[0.2.0] - 2026-10-13

Added

• FACTS.md: the registry of every published number and state, read by scripts/facts.py (CI runs check); the website and READMEs quote it.
• Generated architecture diagrams (diagrams/topology.py, layers.py) replacing the ASCII figures in ARCHITECTURE.md, with a CI sync check.
• Open-source release preparation: LICENSE (Apache-2.0), NOTICE, rewritten README.md. Program board, status journal and publication plan moved out; public summaries now live in RESULTS.md and ROADMAP.md.

Changed

• Reproduction commands in GETTING-STARTED.md and USER-GUIDE.md corrected to experiments/exp-001-ws/scripts/run.sh; vehicle up documented as live on a bare vehicle.
• Page titles cleaned up for the rendered docs.

[0.1.0] - 2026-07-13

Phase-1 gate passed: ARCHITECTURE.md §2's H1, H3, and H4 hypotheses all have reproducible numbers (vxperiments' exp-001/exp-002/exp-002b/exp-003). First tagged release across the phase-1-participating repos.

Added

• ARCHITECTURE.md (16 sections) — the master spec: hypotheses H1-H4 (§2), vxAPI control-plane design (§5), manifest schemas for experiments and vehicles (§6), vxgateway network model (§7), service model incl. OTA/Uptane design (§8), virtualization/hypervisor fidelity ladder (§9), compute model (§10), metrics/methodology (§12), reproducibility rules (§13), language policy + phase roadmap (§15), worked examples (§16).
• ROADMAP.md — program-level task board: critical path, phase gates 1-4, the devops/ release-engineering board.
• RESULTS.md — living progress report, updated every push that moves progress.
• ROADMAP.md — arXiv/venue plan, paper skeletons, partner-selection procedure, the publication gate.
• GETTING-STARTED.md, TESTBED-SETUP.md (host requirements tiered by phase), USER-GUIDE.md (public-facing usage doc, CLI verb state table).
• VERSIONING.md — the SemVer + Keep-a-Changelog convention this release itself follows.

Notable decisions folded in along the way

• Cumulus VX deprecated in favor of a from-scratch software-defined vxgateway (bridge/VLAN/tc HTB, OVS variant planned) — no physical/vendor router anywhere.
• Jailhouse hypervisor evaluated and rejected (stale upstream, no OEM adoption); the hypervisor fidelity ladder targets Xen dom0less and QNX Hypervisor instead (phase 4).
• OTA design (§8) adopts Uptane's trust-model vocabulary (signed root/targets metadata, image-repo vs. director-repo split) without taking aktualizr as a dependency; Rugix Ctrl recommended over RAUC for phase-3 atomic slot updates, evaluated against https://rugix.org's OTA engine comparison.
• "Digital twin" avoided as project terminology in publications — reserved for phase-4 hardware-in-the-loop; this is an "executable virtual prototype."

docs/diagrams

Editable architecture diagrams for the vxedge platform. Source format is native draw.io (.drawio = mxGraphModel XML) — diffable in git, editable in draw.io Desktop or app.diagrams.net.

Both pages derive from ../ARCHITECTURE.md. That file stays the master spec — if a diagram and ARCHITECTURE.md disagree, the diagram is wrong. Change the spec first, then the diagram.

The two perspectives

Page 1 — Hardware. Blocks are physical/virtualised hardware entities, connectors are physical link types. A single VEHICLE container holds all six zones (front-left, front-right, rear-left, rear-right, central-hpc, gateway), each with its zone controller (ZCU), zone-local bus segment, ECUs, sensors and actuators. Zones attach to a shared Ethernet backbone (802.1Q trunk, one VLAN per slice); body zones additionally ride a CAN-FD/LIN backbone. Off-board elements (5G core, RAN, OTA repo, Prometheus, SONiC-VS twin) sit outside the vehicle boundary.

Front zones are laid out mirrored (ZCU at the bottom, nearest the backbone) so every ZCU→backbone connector is a short vertical drop — front and rear zones read symmetrically about the spine.

Page 2 — Software. The same six zones read as a layered stack: applications → middleware (signal/comms plane) → guest OS → vxedge zone/compute backend, over a shared vxgateway data-plane band and host-platform band. Columns align 1:1 with the hardware zones, so a block on page 1 has a stack on page 2 at the same horizontal position. The right-hand column is the API-first control plane (CLI/frontends → vxAPI → vxedged → vxedge-core, fed by manifests and, upstream of those, the SysML v2 model).

Conventions

Connector semantics map to a vehicle's links (ARCHITECTURE.md §6.0.1). Ethernet backbone, zone access link, CAN-FD, LIN, GMSL2 serdes, actuator drive, 5G Uu and internal fabric each have a fixed stroke colour/weight/dash — the legend panel on page 1 is the authority, and it is built from the same style strings the diagram uses.

Zone containers carry manifest metadata. Each zone is an <object> with data attributes — vxid, role, kind, backend, slice, spec — visible in draw.io via right-click → Edit Data. This keeps the picture traceable to the manifests rather than drifting into decoration. Populate them on any zone you add.

Fill colour encodes domain, not vendor: blue = ADAS/safety, green = body/comfort/energy, amber = connectivity, purple = central compute.

The diagram is descriptive; the manifest is the contract. Nothing exists in the testbed because it is drawn here. A block added to the diagram must land in a vehicle or experiment manifest (ARCHITECTURE.md §6) before it is real.

Extending

Page 1's legend panel doubles as an extension palette:

1. Copy the dashed ZONE SLOT group into the VEHICLE container.
2. Right-click → Edit Data and set vxid / role / backend / slice.
3. Fill in ZCU, sensors, ECUs, actuators from the block palette above it.
4. Connect the ZCU to the backbone with the Ethernet backbone connector style.
5. Add the matching column on page 2 (headers at the top; each band takes one box per column).

Everything is a plain container/child hierarchy with absolute geometry — no layout engine to fight. Zone containers are container=1, so dragging a zone moves its contents with it.

Page geometry and printing

Both pages are 2420 × 1711 units — exactly √2 : 1, A-series landscape, with the content inset by a uniform 100-unit margin (content box 2220 × 1511 at 100,100 → 2320,1611). One page is one sheet: File ▸ Print ▸ Fit to page, landscape, fills the paper with no tiling and nothing clipped.

Legibility is the real constraint, not fit. Scaled onto each sheet size, the 12-unit body text lands at:

Screen and PDF are unaffected — zoom is free there. If you need a genuinely readable A4, print a region rather than shrinking the whole page: select a zone (or the legend) and use File ▸ Print ▸ Selection Only.

If you change the layout, keep the page ratio at √2 — set File ▸ Page Setup to a custom size whose width ÷ height is 1.4142, or the fit-to-page print stops being a clean single sheet.

Editing and export

drawio docs/diagrams/sdv-ee-architecture.drawio          # open in the desktop app

# export a page to PNG/SVG/PDF with the diagram XML embedded (stays editable)
drawio -x -f png -e -b 10 -p 1 -o hardware.drawio.png docs/diagrams/sdv-ee-architecture.drawio
drawio -x -f svg -e -b 10 -p 2 -o software.drawio.svg docs/diagrams/sdv-ee-architecture.drawio

# print-ready PDF, both pages, page geometry preserved
drawio -x -f pdf -e -a -o sdv-ee-architecture.pdf docs/diagrams/sdv-ee-architecture.drawio

SVG is the best choice for papers and slides — it scales without the type-size penalty above.

Exported images are deliberately not committed — they go stale silently, and the .drawio renders on GitHub. Generate them on demand.

One gotcha worth knowing

draw.io typesets backtick-delimited text as AsciiMath. A label reading `links[]` renders as l ∈ ks[] (in → ∈). Do not use backticks for code spans in labels — write links[] plain. Setting math="0" on the model does not suppress it in CLI export.