Compare commits
1 Commits
feat/nixos
...
feat/restr
| Author | SHA1 | Date | |
|---|---|---|---|
| 993b9c559c |
@@ -1,64 +0,0 @@
|
||||
name: NixOS Build & Test
|
||||
run-name: Build ${{ gitea.event_name == 'push' && gitea.ref_name || format('PR #{0}', gitea.event.pull_request.number) }}
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
paths:
|
||||
- '**.nix'
|
||||
- 'flake.lock'
|
||||
- 'secrets/**'
|
||||
- 'hosts/**'
|
||||
- 'modules/**'
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
paths:
|
||||
- '**.nix'
|
||||
- 'flake.lock'
|
||||
- 'secrets/**'
|
||||
- 'hosts/**'
|
||||
- 'modules/**'
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: nixos-builder
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Nix environment
|
||||
run: |
|
||||
echo "extra-experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf
|
||||
cat ~/.config/nix/nix.conf
|
||||
|
||||
- name: Install nh (nix helper)
|
||||
run: |
|
||||
nix --extra-experimental-features "nix-command flakes" \
|
||||
profile add nixpkgs#nh
|
||||
nh --version
|
||||
|
||||
- name: Build NixOS configuration (lazyworkhorse)
|
||||
run: |
|
||||
nh os build .#lazyworkhorse
|
||||
env:
|
||||
NIX_CONFIG: "extra-experimental-features = nix-command flakes"
|
||||
|
||||
- name: Build NixOS configuration (cyt-pi)
|
||||
run: |
|
||||
nh os build .#cyt-pi
|
||||
env:
|
||||
NIX_CONFIG: "extra-experimental-features = nix-command flakes"
|
||||
|
||||
- name: Integration tests (placeholder)
|
||||
run: |
|
||||
echo "TODO: Add integration tests here"
|
||||
echo ""
|
||||
echo "Suggested future checks:"
|
||||
echo " - nix flake check (evaluate all NixOS configs)"
|
||||
echo " - Validate agenix secrets are decryptable"
|
||||
echo " - Check services are defined correctly"
|
||||
echo " - Run VM test if nixos-test infrastructure exists"
|
||||
@@ -59,7 +59,6 @@
|
||||
./modules/nixos/filesystem/hoardingcow-mount.nix
|
||||
./modules/nixos/services/docker_manager.nix
|
||||
./modules/nixos/services/open_code_server.nix
|
||||
./modules/nixos/services/staging-vm.nix
|
||||
./modules/nixos/services/ollama_init_custom_models.nix
|
||||
./modules/nixos/services/openclaw_node.nix
|
||||
./modules/nixos/security/ai-worker-restricted.nix
|
||||
|
||||
@@ -1,64 +1,74 @@
|
||||
# AI Worker Restricted Access
|
||||
|
||||
This module provides SSH access for the AI worker (hermes-agent) to run ollama benchmarks on the host.
|
||||
This module provides SSH access for the AI worker (hermes-agent) to run docker commands on the host with restrictions.
|
||||
|
||||
## Security Model
|
||||
|
||||
The `ai-worker` user has:
|
||||
### Overview
|
||||
|
||||
The `ai-worker` user is a member of the `docker` group, but the `docker` binary is wrapped with a script that **blocks dangerous subcommands** while allowing safe operations.
|
||||
|
||||
### Blocked Commands
|
||||
|
||||
These commands are intercepted by the docker wrapper and rejected:
|
||||
|
||||
| Command | Risk | Reason |
|
||||
|---------|------|--------|
|
||||
| `docker exec` | Execute arbitrary commands inside running containers | FILE MODIFICATION |
|
||||
| `docker cp` | Copy files between containers and host | FILE ACCESS |
|
||||
| `docker commit` | Create images from running containers | DATA EXFIL |
|
||||
| `docker diff` | Inspect filesystem changes | INFO LEAK |
|
||||
| `docker export` | Export container filesystem as tar archive | DATA EXFIL |
|
||||
| `docker import` | Import a tar archive to create filesystem | FILE INJECTION |
|
||||
| `docker load` | Load images from tar archive | FILE INJECTION |
|
||||
| `docker save` | Save images to tar archive | DATA EXFIL |
|
||||
| `docker attach` | Attach to running container's stdio | INTERACTIVE ACCESS |
|
||||
| `docker push` | Push images to remote registries | DATA EXFIL |
|
||||
| `docker tag` | Tag/rename images | DATA EXFIL |
|
||||
|
||||
Also blocked in compose context: `docker compose exec`, `docker compose cp`, etc.
|
||||
|
||||
### Allowed Commands
|
||||
|
||||
These commands work normally:
|
||||
|
||||
- `docker ps` — list containers
|
||||
- `docker images` — list images
|
||||
- `docker inspect` — inspect containers/images
|
||||
- `docker logs` — view container logs
|
||||
- `docker start` — start a stopped container
|
||||
- `docker stop` — stop a running container
|
||||
- `docker restart` — restart a container
|
||||
- `docker rm` — remove a stopped container
|
||||
- `docker rmi` — remove an image
|
||||
- `docker pull` — pull an image
|
||||
- `docker build` — build an image
|
||||
- `docker run` — create and start a container
|
||||
- `docker compose` — compose orchestration (but not `compose exec`)
|
||||
- `docker system` — disk management
|
||||
- `docker network ls` — list networks
|
||||
- `docker volume ls` — list volumes
|
||||
|
||||
### How It Works
|
||||
|
||||
1. A wrapper script intercepts `docker` calls in the user's PATH
|
||||
2. It parses the first non-flag argument to determine the subcommand
|
||||
3. If the subcommand is in the blocklist, it prints an error and exits
|
||||
4. Otherwise, it passes through to the real Docker binary
|
||||
|
||||
The wrapper is installed both as a system package and in ai-worker's personal profile to ensure it takes precedence over the real docker binary.
|
||||
|
||||
### Why Not Use Docker Authorization Plugins?
|
||||
|
||||
Docker's native authorization plugin system requires Docker-managed plugins (images) which is complex to deploy in NixOS. A CLI wrapper is simpler, maintainable, and effective for the primary threat model (an LLM agent that uses the docker CLI).
|
||||
|
||||
Note: A determined attacker in the docker group can bypass the wrapper by calling the Docker API directly via `/var/run/docker.sock`. For the LLM agent threat model, this is a theoretical bypass — the agent uses CLI commands and `docker exec` returning an error is sufficient to stop it.
|
||||
|
||||
### Filesystem Access
|
||||
- **Home directory**: `/home/ai-worker` (standard user home)
|
||||
- **No bind mounts**: Cannot access `/home/gortium/infra` or other host files
|
||||
- **Cannot access**: Any files outside standard system paths
|
||||
|
||||
### Sudo Access
|
||||
- **NONE**: ai-worker has no sudo privileges
|
||||
- Cannot run `nh`, `nixos-rebuild`, `nixpkgs-fmt`, or `nix` with elevated permissions
|
||||
|
||||
### Docker Access
|
||||
- Member of `docker` group - can run `docker` and `docker exec` commands
|
||||
- Primary use: `docker exec ollama ollama ...` for benchmarking
|
||||
- Can run `docker exec --privileged ollama rocm-smi ...` for VRAM monitoring
|
||||
|
||||
## Workflow: SSH + Docker Benchmarking
|
||||
|
||||
The AI worker connects from the Hermes container to the host via SSH, runs ollama benchmarks, then returns to save results.
|
||||
|
||||
### Example Workflow
|
||||
|
||||
```bash
|
||||
# From Hermes container, SSH to host
|
||||
ssh -i /path/to/ssh/key ai-worker@host.docker.internal
|
||||
|
||||
# On host, run ollama benchmarks via docker
|
||||
docker exec ollama ollama pull devstral-small-2:24b
|
||||
|
||||
# Create test modelfile
|
||||
docker exec ollama bash -c 'cat <<EOF > /root/.ollama/test.modelfile
|
||||
FROM devstral-small-2:24b
|
||||
PARAMETER num_ctx 65536
|
||||
PARAMETER num_gpu 99
|
||||
PARAMETER flash_attn true
|
||||
EOF'
|
||||
|
||||
# Create and test model
|
||||
docker exec ollama ollama create test-model -f /root/.ollama/test.modelfile
|
||||
docker exec ollama ollama run test-model "Write a Python async function"
|
||||
|
||||
# Check VRAM usage
|
||||
docker exec --privileged ollama rocm-smi --showmeminfo vram
|
||||
|
||||
# Cleanup
|
||||
docker exec ollama ollama rm test-model
|
||||
|
||||
# Exit SSH, return to Hermes container
|
||||
exit
|
||||
|
||||
# Save results in Hermes container
|
||||
# /opt/data/ai-optimizer/state.json
|
||||
# /opt/data/ai-optimizer/results.csv
|
||||
```
|
||||
|
||||
## SSH Access
|
||||
|
||||
Connect as:
|
||||
@@ -70,32 +80,42 @@ The working directory will be `/home/ai-worker`. No infra repo access.
|
||||
|
||||
## Verification
|
||||
|
||||
Check ai-worker permissions:
|
||||
```bash
|
||||
# On the host, as root or gortium:
|
||||
sudo -u ai-worker sudo -l
|
||||
# Should show: no sudo access
|
||||
# Verify wrapper is in PATH
|
||||
sudo -u ai-worker which docker
|
||||
# Should show: /home/ai-worker/.nix-profile/bin/docker (wrapped version)
|
||||
|
||||
# Check docker group membership
|
||||
# Test blocked command (should fail)
|
||||
sudo -u ai-worker docker exec ollama ollama list
|
||||
# Expected: ERROR: docker 'exec' is blocked by security policy
|
||||
|
||||
# Test allowed command (should work)
|
||||
sudo -u ai-worker docker ps
|
||||
# Expected: CONTAINER ID IMAGE ...
|
||||
|
||||
# Verify docker group membership
|
||||
groups ai-worker
|
||||
# Should show: ai-worker docker
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If ai-worker cannot run docker commands:
|
||||
If docker commands fail unexpectedly:
|
||||
|
||||
```bash
|
||||
# Check docker group membership
|
||||
groups ai-worker
|
||||
# Check which docker binary is being used
|
||||
which docker
|
||||
# If this shows /run/current-system/sw/bin/docker, the wrapper is not in PATH
|
||||
|
||||
# Verify ollama container is running
|
||||
docker ps | grep ollama
|
||||
# Check if the wrapper is installed
|
||||
ls -la $(which docker)
|
||||
|
||||
# Test docker access
|
||||
sudo -u ai-worker docker exec ollama ollama list
|
||||
# Verify you're running as the right user
|
||||
whoami
|
||||
```
|
||||
|
||||
If SSH connection fails:
|
||||
|
||||
```bash
|
||||
# Check SSH key is authorized
|
||||
cat /home/ai-worker/.ssh/authorized_keys
|
||||
|
||||
@@ -2,16 +2,123 @@
|
||||
|
||||
with lib;
|
||||
|
||||
let
|
||||
# Docker subcommands that are BLOCKED for ai-worker
|
||||
# These commands allow file modification inside containers or data exfiltration.
|
||||
blockedCommands = [
|
||||
"exec" # Execute arbitrary commands in containers (FILE MODIFICATION)
|
||||
"cp" # Copy files between containers and host (FILE ACCESS)
|
||||
"commit" # Create images from running containers (DATA EXFIL)
|
||||
"diff" # Inspect filesystem changes of containers (INFO LEAK)
|
||||
"export" # Export container filesystem as tar archive (DATA EXFIL)
|
||||
"import" # Import a tar archive to create filesystem (FILE INJECTION)
|
||||
"load" # Load images from tar archive (FILE INJECTION)
|
||||
"save" # Save images to tar archive (DATA EXFIL)
|
||||
"attach" # Attach to running container's stdio (INTERACTIVE ACCESS)
|
||||
"push" # Push images to remote registries (DATA EXFIL)
|
||||
"tag" # Tag/rename images (used with push)
|
||||
];
|
||||
|
||||
blockedDockerArgs = lib.concatStringsSep "|" blockedCommands;
|
||||
|
||||
# Docker wrapper script that blocks dangerous subcommands
|
||||
# Must handle: docker exec, docker compose exec, docker cp, etc.
|
||||
restrictedDockerScript = pkgs.writeShellScriptBin "docker" ''
|
||||
set -e
|
||||
|
||||
# Blocklist pattern
|
||||
BLOCKED_PATTERN="^(${blockedDockerArgs})$"
|
||||
|
||||
# Parse the first non-flag argument to find the docker subcommand
|
||||
# Flags: -H, --host, -D, --debug, --config, --context, --log-level, -l
|
||||
# Also handle: docker compose <subcommand> (subcommand may be after 'compose')
|
||||
SUBCOMMAND=""
|
||||
COMPOSE_MODE=false
|
||||
FOUND_ARG=false
|
||||
|
||||
for arg in "$@"; do
|
||||
# Skip flags and their values
|
||||
case "$arg" in
|
||||
-H|--host|-l|--log-level|--config|--context|-D|--debug)
|
||||
FOUND_ARG=true
|
||||
continue
|
||||
;;
|
||||
--tls|--tlsverify|--tlscacert|--tlscert|--tlskey)
|
||||
if $FOUND_ARG; then FOUND_ARG=false; else continue; fi
|
||||
;;
|
||||
# Skip flag values (the next arg after a flag that takes a value)
|
||||
-*)
|
||||
continue
|
||||
;;
|
||||
*)
|
||||
# This is a positional argument — first one is the subcommand (or 'compose')
|
||||
if [ -z "$SUBCOMMAND" ]; then
|
||||
if [ "$arg" = "compose" ]; then
|
||||
COMPOSE_MODE=true
|
||||
continue
|
||||
fi
|
||||
SUBCOMMAND="$arg"
|
||||
break
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
FOUND_ARG=false
|
||||
done
|
||||
|
||||
# If in compose mode, the subcommand is after 'compose'
|
||||
if $COMPOSE_MODE; then
|
||||
# In compose mode, we check the sub-subcommand
|
||||
NEXT_GOT=""
|
||||
for arg in "$@"; do
|
||||
if [ "$NEXT_GOT" = "true" ]; then
|
||||
if echo "$arg" | grep -qE "$BLOCKED_PATTERN"; then
|
||||
echo "ERROR: docker compose '$arg' is blocked by security policy" >&2
|
||||
echo "This command can modify files inside containers." >&2
|
||||
exit 1
|
||||
fi
|
||||
break
|
||||
fi
|
||||
if [ "$arg" = "compose" ]; then
|
||||
NEXT_GOT="true"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# Check if the subcommand is blocked
|
||||
if [ -n "$SUBCOMMAND" ]; then
|
||||
if echo "$SUBCOMMAND" | grep -qE "$BLOCKED_PATTERN"; then
|
||||
echo "ERROR: docker '$SUBCOMMAND' is blocked by security policy" >&2
|
||||
echo "This command can modify files inside containers." >&2
|
||||
echo "" >&2
|
||||
echo "Allowed commands: ps, images, inspect, logs, start, stop, restart," >&2
|
||||
echo " rm, rmi, pull, build, run, compose, system, network ls, volume ls" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Execute the real docker binary
|
||||
exec ${pkgs.docker}/bin/docker "$@"
|
||||
'';
|
||||
in
|
||||
{
|
||||
options.services.aiWorkerAccess = mkOption {
|
||||
type = types.bool;
|
||||
default = false;
|
||||
description = "Enable AI worker SSH access with docker group membership for ollama benchmarking";
|
||||
description = "Enable AI worker SSH access with restricted docker commands";
|
||||
};
|
||||
|
||||
config = mkIf config.services.aiWorkerAccess {
|
||||
# ai-worker is member of docker group - can run docker commands via SSH
|
||||
# No bind mounts, no sudo access - docker-only for ollama benchmarking
|
||||
# ai-worker is in docker group for normal docker operations
|
||||
users.groups.docker.members = [ "ai-worker" ];
|
||||
|
||||
# Install the docker wrapper for ai-worker
|
||||
# This puts a filtered 'docker' script in ai-worker's PATH that blocks
|
||||
# dangerous commands like exec, cp, commit, etc.
|
||||
# The real docker binary is still available at its store path, but the
|
||||
# wrapper intercepts it because ~/.nix-profile/bin/ comes before /run/.../sw/bin/ in PATH.
|
||||
users.users.ai-worker.packages = [ restrictedDockerScript ];
|
||||
|
||||
# Also install the wrapper system-wide for consistency
|
||||
environment.systemPackages = [ restrictedDockerScript ];
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,415 +0,0 @@
|
||||
{ config, lib, pkgs, ... }:
|
||||
|
||||
with lib;
|
||||
|
||||
let
|
||||
cfg = config.services.staging-vm;
|
||||
|
||||
# Resolve the first IP in the subnet (the gateway address for the NAT network)
|
||||
networkIp = head (splitString "/" cfg.network.subnet);
|
||||
|
||||
# ── pr-test-vm helper script ──────────────────────────────────────────
|
||||
pr-test-vm = pkgs.writeShellScriptBin "pr-test-vm" ''
|
||||
set -euo pipefail
|
||||
|
||||
LIBVIRT_URI="qemu:///system"
|
||||
STORAGE_POOL="${cfg.storagePool}"
|
||||
VM_DIR="${cfg.dataDir}"
|
||||
NETWORK="${cfg.network.name}"
|
||||
SCRIPT_NAME="$(basename "$0")"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $SCRIPT_NAME <command> [options]
|
||||
|
||||
Commands:
|
||||
build <nixos-config> [--name <name>] Build VM image from a NixOS config
|
||||
start <vm-name> Start a VM
|
||||
stop <vm-name> Gracefully shut down a VM
|
||||
destroy <vm-name> Force-power-off and undefine a VM
|
||||
ssh [user@]<vm-name> SSH into a running VM
|
||||
console <vm-name> Connect to VM serial console
|
||||
list List all staging VMs
|
||||
status <vm-name> Show VM status
|
||||
rebuild <vm-name> Redeploy the VM (stop + start)
|
||||
|
||||
Examples:
|
||||
$SCRIPT_NAME build ./vm-config.nix --name my-test
|
||||
$SCRIPT_NAME start my-test
|
||||
$SCRIPT_NAME ssh root@my-test
|
||||
EOF
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Find the VM's IP address from the DHCP lease
|
||||
vm_ip() {
|
||||
local name="$1"
|
||||
local mac
|
||||
mac=$(${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" domiflist "$name" 2>/dev/null \
|
||||
| ${pkgs.gawk}/bin/awk 'NR>2 && $1 ~ /^vnet/ {print $NF; exit}')
|
||||
[ -z "$mac" ] && { echo "error: cannot find MAC for VM '$name'"; exit 1; }
|
||||
|
||||
local ip
|
||||
ip=$(${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" net-dhcp-leases "$NETWORK" 2>/dev/null \
|
||||
| ${pkgs.gawk}/bin/awk -v mac="$mac" '$0 ~ mac {gsub(/-.*/, "", $3); print $3; exit}')
|
||||
[ -z "$ip" ] && { echo "error: no DHCP lease found for VM '$name' (MAC: $mac)"; exit 1; }
|
||||
echo "$ip"
|
||||
}
|
||||
|
||||
case "''${1:-help}" in
|
||||
build)
|
||||
shift
|
||||
CONFIG="''${1:?Missing NixOS config path}"
|
||||
VM_NAME="''${2:-}"
|
||||
[ -f "$CONFIG" ] || { echo "error: config file not found: $CONFIG"; exit 1; }
|
||||
|
||||
# Extract name from --name flag or config basename
|
||||
if [ "''${2:-}" = "--name" ] && [ -n "''${3:-}" ]; then
|
||||
VM_NAME="$3"
|
||||
elif [ -z "$VM_NAME" ] || [ "''${VM_NAME#--}" != "$VM_NAME" ]; then
|
||||
VM_NAME="$(basename "$CONFIG" .nix)"
|
||||
fi
|
||||
|
||||
BUILD_DIR="$VM_DIR/$VM_NAME"
|
||||
IMAGE="$BUILD_DIR/disk-image.qcow2"
|
||||
|
||||
echo "==> Building VM '$VM_NAME' from config: $CONFIG"
|
||||
mkdir -p "$BUILD_DIR"
|
||||
|
||||
# Build the NixOS VM derivation
|
||||
nix build --no-link -f "$CONFIG" vm 2>&1 || {
|
||||
# Fallback: try as flake output
|
||||
echo "Trying flake build..."
|
||||
nix build "''${CONFIG%/.nix}#nixosConfigurations.$VM_NAME.config.system.build.vm" --no-link 2>&1 || {
|
||||
echo "error: failed to build VM (tried both import and flake)"
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
echo "==> Build complete. Run 'pr-test-vm start $VM_NAME' to launch."
|
||||
;;
|
||||
|
||||
start)
|
||||
VM_NAME="''${1:?Missing VM name}"
|
||||
IMAGE="$VM_DIR/$VM_NAME/disk-image.qcow2"
|
||||
[ -f "$IMAGE" ] || { echo "error: no disk image found at $IMAGE. Build first."; exit 1; }
|
||||
|
||||
# Check if already running
|
||||
STATE=$(${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" domstate "$VM_NAME" 2>/dev/null || echo "undefined")
|
||||
if [ "$STATE" = "running" ]; then
|
||||
echo "VM '$VM_NAME' is already running."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "==> Starting VM '$VM_NAME'..."
|
||||
|
||||
# Undefine if defined but not running
|
||||
if [ "$STATE" != "undefined" ]; then
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" undefine "$VM_NAME" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Define and start with virt-install
|
||||
${pkgs.virt-manager}/bin/virt-install \
|
||||
--connect "$LIBVIRT_URI" \
|
||||
--name "$VM_NAME" \
|
||||
--memory "${toString cfg.memory}" \
|
||||
--vcpus "${toString cfg.vcpus}" \
|
||||
--disk "$IMAGE",bus=virtio \
|
||||
--import \
|
||||
--network network="$NETWORK",model=virtio \
|
||||
--graphics none \
|
||||
--console pty,target_type=virtio \
|
||||
--serial pty \
|
||||
--memballoon virtio \
|
||||
--rng /dev/urandom \
|
||||
--noautoconsole \
|
||||
--os-variant detect=on,name=generic
|
||||
|
||||
echo "==> VM '$VM_NAME' started. Get IP with: pr-test-vm status $VM_NAME"
|
||||
;;
|
||||
|
||||
stop)
|
||||
VM_NAME="''${1:?Missing VM name}"
|
||||
echo "==> Stoping VM '$VM_NAME'..."
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" shutdown "$VM_NAME" 2>/dev/null && {
|
||||
echo "Waiting for VM to shut down..."
|
||||
for i in $(seq 1 30); do
|
||||
STATE=$(${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" domstate "$VM_NAME" 2>/dev/null || echo "undefined")
|
||||
[ "$STATE" != "running" ] && { echo "VM stopped."; exit 0; }
|
||||
sleep 2
|
||||
done
|
||||
echo "warning: VM did not shut down gracefully, use 'destroy' for force"
|
||||
} || {
|
||||
echo "VM '$VM_NAME' not running or does not exist."
|
||||
}
|
||||
;;
|
||||
|
||||
destroy)
|
||||
VM_NAME="''${1:?Missing VM name}"
|
||||
echo "==> Destroying VM '$VM_NAME'..."
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" destroy "$VM_NAME" 2>/dev/null || true
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" undefine "$VM_NAME" 2>/dev/null || true
|
||||
echo "==> VM '$VM_NAME' destroyed and undefined."
|
||||
;;
|
||||
|
||||
ssh)
|
||||
TARGET="''${1:?Usage: $SCRIPT_NAME ssh [user@]<vm-name>}"
|
||||
# Split user@hostname if present
|
||||
if echo "$TARGET" | ${pkgs.gnugrep}/bin/grep -q '@'; then
|
||||
USER="''${TARGET%@*}"
|
||||
VM_NAME="''${TARGET#*@}"
|
||||
else
|
||||
VM_NAME="$TARGET"
|
||||
USER=""
|
||||
fi
|
||||
|
||||
IP=$(vm_ip "$VM_NAME") || exit 1
|
||||
if [ -n "$USER" ]; then
|
||||
exec ${pkgs.openssh}/bin/ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "''${USER}@''${IP}"
|
||||
else
|
||||
exec ${pkgs.openssh}/bin/ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$IP"
|
||||
fi
|
||||
;;
|
||||
|
||||
console)
|
||||
VM_NAME="''${1:?Missing VM name}"
|
||||
exec ${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" console "$VM_NAME"
|
||||
;;
|
||||
|
||||
list)
|
||||
echo "Staging VMs:"
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" list --all
|
||||
echo ""
|
||||
echo "Active networks:"
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" net-list
|
||||
echo ""
|
||||
echo "Storage pools:"
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" pool-list
|
||||
;;
|
||||
|
||||
status)
|
||||
VM_NAME="''${1:?Missing VM name}"
|
||||
echo "VM: $VM_NAME"
|
||||
STATE=$(${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" domstate "$VM_NAME" 2>/dev/null || echo "not found")
|
||||
echo "State: $STATE"
|
||||
if [ "$STATE" = "running" ]; then
|
||||
IP=$(vm_ip "$VM_NAME" 2>/dev/null || echo "N/A")
|
||||
echo "IP: $IP"
|
||||
${pkgs.libvirt}/bin/virsh -c "$LIBVIRT_URI" dommemstat "$VM_NAME" 2>/dev/null | head -3 || true
|
||||
fi
|
||||
;;
|
||||
|
||||
rebuild)
|
||||
VM_NAME="''${1:?Missing VM name}"
|
||||
"$0" destroy "$VM_NAME"
|
||||
"$0" start "$VM_NAME"
|
||||
;;
|
||||
|
||||
help|--help|-h)
|
||||
usage
|
||||
;;
|
||||
|
||||
*)
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
'';
|
||||
in
|
||||
{
|
||||
options.services.staging-vm = {
|
||||
enable = mkEnableOption "Staging VM infrastructure with libvirt/KVM";
|
||||
|
||||
network = {
|
||||
name = mkOption {
|
||||
type = types.str;
|
||||
default = "staging";
|
||||
description = "Name of the libvirt NAT network for staging VMs";
|
||||
};
|
||||
|
||||
bridge = mkOption {
|
||||
type = types.str;
|
||||
default = "virbr1";
|
||||
description = "Bridge interface name for the staging network";
|
||||
};
|
||||
|
||||
subnet = mkOption {
|
||||
type = types.str;
|
||||
default = "192.168.122.0/24";
|
||||
description = "NAT network subnet in CIDR notation";
|
||||
};
|
||||
|
||||
dhcpStart = mkOption {
|
||||
type = types.str;
|
||||
default = "192.168.122.2";
|
||||
description = "Start of the DHCP range";
|
||||
};
|
||||
|
||||
dhcpEnd = mkOption {
|
||||
type = types.str;
|
||||
default = "192.168.122.254";
|
||||
description = "End of the DHCP range";
|
||||
};
|
||||
};
|
||||
|
||||
storagePool = mkOption {
|
||||
type = types.str;
|
||||
default = "/var/lib/libvirt/images";
|
||||
description = "Directory path for the libvirt storage pool";
|
||||
};
|
||||
|
||||
dataDir = mkOption {
|
||||
type = types.str;
|
||||
default = "/var/lib/staging-vm";
|
||||
description = "Directory for staging VM test data (images, cloud-init configs)";
|
||||
};
|
||||
|
||||
memory = mkOption {
|
||||
type = types.int;
|
||||
default = 2048;
|
||||
description = "Default RAM in MiB for staging VMs";
|
||||
};
|
||||
|
||||
vcpus = mkOption {
|
||||
type = types.int;
|
||||
default = 2;
|
||||
description = "Default number of vCPUs for staging VMs";
|
||||
};
|
||||
};
|
||||
|
||||
config = mkIf cfg.enable {
|
||||
# ── libvirtd with QEMU/KVM ──────────────────────────────────────────
|
||||
virtualisation.libvirtd = {
|
||||
enable = true;
|
||||
|
||||
qemu = {
|
||||
package = pkgs.qemu_kvm;
|
||||
runAsRoot = true;
|
||||
|
||||
swtpm = {
|
||||
enable = true;
|
||||
};
|
||||
|
||||
ovmf = {
|
||||
enable = true;
|
||||
packages = [ pkgs.OVMF ];
|
||||
};
|
||||
};
|
||||
|
||||
# Allow the staging bridge for guest networking
|
||||
allowedBridges = [ cfg.network.bridge ];
|
||||
};
|
||||
|
||||
# ── System packages ─────────────────────────────────────────────────
|
||||
environment.systemPackages = with pkgs; [
|
||||
libvirt # virsh, virt-admin
|
||||
qemu_kvm # QEMU/KVM
|
||||
swtpm # Software TPM
|
||||
OVMF # UEFI firmware for VMs
|
||||
virt-manager # GUI management
|
||||
virt-viewer # SPICE/VNC viewer
|
||||
libguestfs # virt-install, virt-customize, guestfish
|
||||
cdrtools # genisoimage for cloud-init ISOs
|
||||
jq # JSON parsing
|
||||
gawk # awk for DHCP lease parsing
|
||||
gnugrep # grep
|
||||
];
|
||||
|
||||
# ── User permissions ────────────────────────────────────────────────
|
||||
users.users.gortium.extraGroups = [ "libvirtd" ];
|
||||
|
||||
# ── Directories ─────────────────────────────────────────────────────
|
||||
systemd.tmpfiles.rules = [
|
||||
"d ${cfg.storagePool} 0755 root root -"
|
||||
"d ${cfg.dataDir} 0755 root root -"
|
||||
];
|
||||
|
||||
# ── Default NAT network definition ──────────────────────────────────
|
||||
systemd.services.define-staging-network = {
|
||||
description = "Define the staging libvirt NAT network";
|
||||
after = [ "libvirtd.service" ];
|
||||
requires = [ "libvirtd.service" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
before = [ "define-staging-pool.service" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
};
|
||||
script = let
|
||||
networkXml = pkgs.writeText "staging-network.xml" ''
|
||||
<network>
|
||||
<name>${cfg.network.name}</name>
|
||||
<forward mode='nat'/>
|
||||
<bridge name='${cfg.network.bridge}' stp='on' delay='0'/>
|
||||
<ip address='${networkIp}' netmask='255.255.255.0'>
|
||||
<dhcp>
|
||||
<range start='${cfg.network.dhcpStart}' end='${cfg.network.dhcpEnd}'/>
|
||||
</dhcp>
|
||||
</ip>
|
||||
</network>
|
||||
'';
|
||||
in ''
|
||||
set -e
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system net-info "${cfg.network.name}" 2>/dev/null && {
|
||||
echo "Network '${cfg.network.name}' already exists"
|
||||
} || {
|
||||
echo "Defining network '${cfg.network.name}'..."
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system net-define "${networkXml}"
|
||||
}
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system net-autostart "${cfg.network.name}"
|
||||
# Start the network if not already active
|
||||
STATE=$(${pkgs.libvirt}/bin/virsh -c qemu:///system net-state "${cfg.network.name}" 2>/dev/null || echo "inactive")
|
||||
if [ "$STATE" != "active" ]; then
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system net-start "${cfg.network.name}"
|
||||
fi
|
||||
echo "Network '${cfg.network.name}' is ready."
|
||||
'';
|
||||
};
|
||||
|
||||
# ── Storage pool definition ─────────────────────────────────────────
|
||||
systemd.services.define-staging-pool = {
|
||||
description = "Define the staging libvirt storage pool";
|
||||
after = [ "libvirtd.service" "define-staging-network.service" ];
|
||||
requires = [ "libvirtd.service" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
};
|
||||
script = ''
|
||||
set -e
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system pool-info staging 2>/dev/null && {
|
||||
echo "Storage pool 'staging' already exists"
|
||||
} || {
|
||||
echo "Defining storage pool 'staging' at ${cfg.storagePool}..."
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system pool-define-as \
|
||||
--name staging --type dir --target "${cfg.storagePool}"
|
||||
}
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system pool-autostart staging
|
||||
STATE=$(${pkgs.libvirt}/bin/virsh -c qemu:///system pool-state staging 2>/dev/null || echo "inactive")
|
||||
if [ "$STATE" != "running" ]; then
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system pool-build staging
|
||||
${pkgs.libvirt}/bin/virsh -c qemu:///system pool-start staging
|
||||
fi
|
||||
echo "Storage pool 'staging' is ready."
|
||||
'';
|
||||
};
|
||||
|
||||
# ── Firewall rules for libvirt guests ───────────────────────────────
|
||||
networking.firewall = {
|
||||
# Trust the bridge interface to allow guest traffic
|
||||
trustedInterfaces = [ cfg.network.bridge ];
|
||||
|
||||
extraCommands = mkAfter ''
|
||||
# Allow forwarding between the staging bridge and the outside world
|
||||
iptables -I FORWARD -i ${cfg.network.bridge} -j ACCEPT 2>/dev/null || true
|
||||
iptables -I FORWARD -o ${cfg.network.bridge} -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || true
|
||||
|
||||
# NAT for guest outbound traffic
|
||||
iptables -t nat -I POSTROUTING -s ${cfg.network.subnet} -j MASQUERADE 2>/dev/null || true
|
||||
'';
|
||||
};
|
||||
|
||||
# ── pr-test-vm helper script ────────────────────────────────────────
|
||||
environment.systemPackages = [ pr-test-vm ];
|
||||
};
|
||||
}
|
||||
@@ -4,6 +4,8 @@
|
||||
group = "ai-worker";
|
||||
home = "/home/ai-worker";
|
||||
createHome = true;
|
||||
# ai-worker stays in docker group for normal docker operations (ps, start, stop, compose, ...)
|
||||
# Dangerous commands (exec, cp, commit) are blocked by a wrapper script.
|
||||
extraGroups = [ "docker" ];
|
||||
shell = pkgs.bashInteractive;
|
||||
openssh.authorizedKeys.keys = [
|
||||
@@ -14,17 +16,14 @@
|
||||
};
|
||||
users.groups.ai-worker = {};
|
||||
|
||||
# Enable restricted AI worker SSH access for ollama benchmarking
|
||||
# SECURITY: ai-worker can only:
|
||||
# - SSH into host from Hermes container
|
||||
# - Run docker commands (docker exec ollama ...) via docker group
|
||||
# - Run specific security audit commands
|
||||
# - NO access to infra repo (no bind mount)
|
||||
# - NO sudo access (no nh, nixos-rebuild, nixpkgs-fmt, nix)
|
||||
# WORKFLOW: SSH from Hermes container, run docker benchmarks, return and save results to /opt/data/ai-optimizer/
|
||||
# Enable restricted AI worker SSH access
|
||||
# SECURITY: ai-worker is in docker group but docker commands are filtered:
|
||||
# ALLOWED: ps, images, logs, start, stop, restart, rm, rmi, pull, build, run, compose
|
||||
# BLOCKED: exec, cp, commit, diff, export, import, load, save, attach, push
|
||||
# The filtering is done by a docker wrapper in ai-worker's PATH.
|
||||
services.aiWorkerAccess = true;
|
||||
|
||||
# Restricted sudo for ai-worker - security checks only
|
||||
|
||||
# Restricted sudo for ai-worker - security checks only (not for docker)
|
||||
security.sudo.extraRules = [
|
||||
{
|
||||
users = [ "ai-worker" ];
|
||||
@@ -69,15 +68,6 @@
|
||||
command = "/run/current-system/sw/bin/sshd -T";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
# Docker service checks
|
||||
{
|
||||
command = "/run/current-system/sw/bin/docker ps";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
{
|
||||
command = "/run/current-system/sw/bin/docker inspect *";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
# Network diagnostics
|
||||
{
|
||||
command = "/run/current-system/sw/bin/ss -tlnp";
|
||||
|
||||
Reference in New Issue
Block a user