Compare commits

..

3 Commits

Author SHA1 Message Date
Robert
af38655170 Adding the runner key
Some checks failed
Build NixOS config / build (pull_request) Failing after 55s
2026-05-10 22:23:29 -04:00
dcf5ac6b5e fix: use nixos-builder label for NixOS CI 2026-05-10 22:15:53 -04:00
d78de94f94 feat: add NixOS build CI for infra PRs 2026-05-10 22:15:02 -04:00
7 changed files with 153 additions and 238 deletions

View File

@@ -0,0 +1,33 @@
name: Build NixOS config
on:
pull_request:
branches: [ master ]
paths:
- '**.nix'
- 'flake.lock'
- 'secrets/**'
- 'hosts/**'
- 'modules/**'
push:
branches: [ master ]
paths:
- '**.nix'
- 'flake.lock'
- 'secrets/**'
- 'hosts/**'
- 'modules/**'
jobs:
build:
runs-on: nixos-builder
steps:
- name: Checkout
run: |
git clone -b "${{ github.head_ref || github.ref_name }}" \
https://gitea:${{ secrets.GITHUB_TOKEN }}@code.lazyworkhorse.net/gortium/infra.git .
git log --oneline -3
- name: Build NixOS config (lazyworkhorse)
run: |
nix --version
nh os build .#lazyworkhorse 2>&1

View File

@@ -207,7 +207,6 @@
ai = {
path = self + "/assets/compose/ai";
envFile = config.age.secrets.containers_env.path;
ports = [ 22000 ]; # Syncthing TCP sync
};
cloudstorage = {
@@ -475,7 +474,7 @@
services.openssh.settings = {
PermitRootLogin = "no";
MaxAuthTries = 3;
MaxSessions = 20;
MaxSessions = 10;
LoginGraceTime = 30;
ClientAliveInterval = 300;
ClientAliveCountMax = 2;

View File

@@ -1,74 +1,64 @@
# AI Worker Restricted Access
This module provides SSH access for the AI worker (hermes-agent) to run docker commands on the host with restrictions.
This module provides SSH access for the AI worker (hermes-agent) to run ollama benchmarks on the host.
## Security Model
### Overview
The `ai-worker` user is a member of the `docker` group, but the `docker` binary is wrapped with a script that **blocks dangerous subcommands** while allowing safe operations.
### Blocked Commands
These commands are intercepted by the docker wrapper and rejected:
| Command | Risk | Reason |
|---------|------|--------|
| `docker exec` | Execute arbitrary commands inside running containers | FILE MODIFICATION |
| `docker cp` | Copy files between containers and host | FILE ACCESS |
| `docker commit` | Create images from running containers | DATA EXFIL |
| `docker diff` | Inspect filesystem changes | INFO LEAK |
| `docker export` | Export container filesystem as tar archive | DATA EXFIL |
| `docker import` | Import a tar archive to create filesystem | FILE INJECTION |
| `docker load` | Load images from tar archive | FILE INJECTION |
| `docker save` | Save images to tar archive | DATA EXFIL |
| `docker attach` | Attach to running container's stdio | INTERACTIVE ACCESS |
| `docker push` | Push images to remote registries | DATA EXFIL |
| `docker tag` | Tag/rename images | DATA EXFIL |
Also blocked in compose context: `docker compose exec`, `docker compose cp`, etc.
### Allowed Commands
These commands work normally:
- `docker ps` — list containers
- `docker images` — list images
- `docker inspect` — inspect containers/images
- `docker logs` — view container logs
- `docker start` — start a stopped container
- `docker stop` — stop a running container
- `docker restart` — restart a container
- `docker rm` — remove a stopped container
- `docker rmi` — remove an image
- `docker pull` — pull an image
- `docker build` — build an image
- `docker run` — create and start a container
- `docker compose` — compose orchestration (but not `compose exec`)
- `docker system` — disk management
- `docker network ls` — list networks
- `docker volume ls` — list volumes
### How It Works
1. A wrapper script intercepts `docker` calls in the user's PATH
2. It parses the first non-flag argument to determine the subcommand
3. If the subcommand is in the blocklist, it prints an error and exits
4. Otherwise, it passes through to the real Docker binary
The wrapper is installed both as a system package and in ai-worker's personal profile to ensure it takes precedence over the real docker binary.
### Why Not Use Docker Authorization Plugins?
Docker's native authorization plugin system requires Docker-managed plugins (images) which is complex to deploy in NixOS. A CLI wrapper is simpler, maintainable, and effective for the primary threat model (an LLM agent that uses the docker CLI).
Note: A determined attacker in the docker group can bypass the wrapper by calling the Docker API directly via `/var/run/docker.sock`. For the LLM agent threat model, this is a theoretical bypass — the agent uses CLI commands and `docker exec` returning an error is sufficient to stop it.
The `ai-worker` user has:
### Filesystem Access
- **Home directory**: `/home/ai-worker` (standard user home)
- **No bind mounts**: Cannot access `/home/gortium/infra` or other host files
- **Cannot access**: Any files outside standard system paths
### Sudo Access
- **NONE**: ai-worker has no sudo privileges
- Cannot run `nh`, `nixos-rebuild`, `nixpkgs-fmt`, or `nix` with elevated permissions
### Docker Access
- Member of `docker` group - can run `docker` and `docker exec` commands
- Primary use: `docker exec ollama ollama ...` for benchmarking
- Can run `docker exec --privileged ollama rocm-smi ...` for VRAM monitoring
## Workflow: SSH + Docker Benchmarking
The AI worker connects from the Hermes container to the host via SSH, runs ollama benchmarks, then returns to save results.
### Example Workflow
```bash
# From Hermes container, SSH to host
ssh -i /path/to/ssh/key ai-worker@host.docker.internal
# On host, run ollama benchmarks via docker
docker exec ollama ollama pull devstral-small-2:24b
# Create test modelfile
docker exec ollama bash -c 'cat <<EOF > /root/.ollama/test.modelfile
FROM devstral-small-2:24b
PARAMETER num_ctx 65536
PARAMETER num_gpu 99
PARAMETER flash_attn true
EOF'
# Create and test model
docker exec ollama ollama create test-model -f /root/.ollama/test.modelfile
docker exec ollama ollama run test-model "Write a Python async function"
# Check VRAM usage
docker exec --privileged ollama rocm-smi --showmeminfo vram
# Cleanup
docker exec ollama ollama rm test-model
# Exit SSH, return to Hermes container
exit
# Save results in Hermes container
# /opt/data/ai-optimizer/state.json
# /opt/data/ai-optimizer/results.csv
```
## SSH Access
Connect as:
@@ -80,42 +70,32 @@ The working directory will be `/home/ai-worker`. No infra repo access.
## Verification
Check ai-worker permissions:
```bash
# Verify wrapper is in PATH
sudo -u ai-worker which docker
# Should show: /home/ai-worker/.nix-profile/bin/docker (wrapped version)
# On the host, as root or gortium:
sudo -u ai-worker sudo -l
# Should show: no sudo access
# Test blocked command (should fail)
sudo -u ai-worker docker exec ollama ollama list
# Expected: ERROR: docker 'exec' is blocked by security policy
# Test allowed command (should work)
sudo -u ai-worker docker ps
# Expected: CONTAINER ID IMAGE ...
# Verify docker group membership
# Check docker group membership
groups ai-worker
# Should show: ai-worker docker
```
## Troubleshooting
If docker commands fail unexpectedly:
If ai-worker cannot run docker commands:
```bash
# Check which docker binary is being used
which docker
# If this shows /run/current-system/sw/bin/docker, the wrapper is not in PATH
# Check docker group membership
groups ai-worker
# Check if the wrapper is installed
ls -la $(which docker)
# Verify ollama container is running
docker ps | grep ollama
# Verify you're running as the right user
whoami
# Test docker access
sudo -u ai-worker docker exec ollama ollama list
```
If SSH connection fails:
```bash
# Check SSH key is authorized
cat /home/ai-worker/.ssh/authorized_keys

View File

@@ -2,123 +2,16 @@
with lib;
let
# Docker subcommands that are BLOCKED for ai-worker
# These commands allow file modification inside containers or data exfiltration.
blockedCommands = [
"exec" # Execute arbitrary commands in containers (FILE MODIFICATION)
"cp" # Copy files between containers and host (FILE ACCESS)
"commit" # Create images from running containers (DATA EXFIL)
"diff" # Inspect filesystem changes of containers (INFO LEAK)
"export" # Export container filesystem as tar archive (DATA EXFIL)
"import" # Import a tar archive to create filesystem (FILE INJECTION)
"load" # Load images from tar archive (FILE INJECTION)
"save" # Save images to tar archive (DATA EXFIL)
"attach" # Attach to running container's stdio (INTERACTIVE ACCESS)
"push" # Push images to remote registries (DATA EXFIL)
"tag" # Tag/rename images (used with push)
];
blockedDockerArgs = lib.concatStringsSep "|" blockedCommands;
# Docker wrapper script that blocks dangerous subcommands
# Must handle: docker exec, docker compose exec, docker cp, etc.
restrictedDockerScript = pkgs.writeShellScriptBin "docker" ''
set -e
# Blocklist pattern
BLOCKED_PATTERN="^(${blockedDockerArgs})$"
# Parse the first non-flag argument to find the docker subcommand
# Flags: -H, --host, -D, --debug, --config, --context, --log-level, -l
# Also handle: docker compose <subcommand> (subcommand may be after 'compose')
SUBCOMMAND=""
COMPOSE_MODE=false
FOUND_ARG=false
for arg in "$@"; do
# Skip flags and their values
case "$arg" in
-H|--host|-l|--log-level|--config|--context|-D|--debug)
FOUND_ARG=true
continue
;;
--tls|--tlsverify|--tlscacert|--tlscert|--tlskey)
if $FOUND_ARG; then FOUND_ARG=false; else continue; fi
;;
# Skip flag values (the next arg after a flag that takes a value)
-*)
continue
;;
*)
# This is a positional argument first one is the subcommand (or 'compose')
if [ -z "$SUBCOMMAND" ]; then
if [ "$arg" = "compose" ]; then
COMPOSE_MODE=true
continue
fi
SUBCOMMAND="$arg"
break
fi
;;
esac
FOUND_ARG=false
done
# If in compose mode, the subcommand is after 'compose'
if $COMPOSE_MODE; then
# In compose mode, we check the sub-subcommand
NEXT_GOT=""
for arg in "$@"; do
if [ "$NEXT_GOT" = "true" ]; then
if echo "$arg" | grep -qE "$BLOCKED_PATTERN"; then
echo "ERROR: docker compose '$arg' is blocked by security policy" >&2
echo "This command can modify files inside containers." >&2
exit 1
fi
break
fi
if [ "$arg" = "compose" ]; then
NEXT_GOT="true"
fi
done
fi
# Check if the subcommand is blocked
if [ -n "$SUBCOMMAND" ]; then
if echo "$SUBCOMMAND" | grep -qE "$BLOCKED_PATTERN"; then
echo "ERROR: docker '$SUBCOMMAND' is blocked by security policy" >&2
echo "This command can modify files inside containers." >&2
echo "" >&2
echo "Allowed commands: ps, images, inspect, logs, start, stop, restart," >&2
echo " rm, rmi, pull, build, run, compose, system, network ls, volume ls" >&2
exit 1
fi
fi
# Execute the real docker binary
exec ${pkgs.docker}/bin/docker "$@"
'';
in
{
options.services.aiWorkerAccess = mkOption {
type = types.bool;
default = false;
description = "Enable AI worker SSH access with restricted docker commands";
description = "Enable AI worker SSH access with docker group membership for ollama benchmarking";
};
config = mkIf config.services.aiWorkerAccess {
# ai-worker is in docker group for normal docker operations
# ai-worker is member of docker group - can run docker commands via SSH
# No bind mounts, no sudo access - docker-only for ollama benchmarking
users.groups.docker.members = [ "ai-worker" ];
# Install the docker wrapper for ai-worker
# This puts a filtered 'docker' script in ai-worker's PATH that blocks
# dangerous commands like exec, cp, commit, etc.
# The real docker binary is still available at its store path, but the
# wrapper intercepts it because ~/.nix-profile/bin/ comes before /run/.../sw/bin/ in PATH.
users.users.ai-worker.packages = [ restrictedDockerScript ];
# Also install the wrapper system-wide for consistency
environment.systemPackages = [ restrictedDockerScript ];
};
}

View File

@@ -1,36 +1,36 @@
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHNzaC1lZDI1NTE5IEdoTUQ4QSBWNEpt
cGFNeVBBaDRqb3pLSEZGQW0wb3VmVnBoZCswUFkzbnBLUnJ0QTNrClRqVkk4RUVO
d29KYjd5YUcwankvaTFmVHUxQVpDT2ZYWHRaY3JXTUtQMU0KLT4gKXBtQ3UsXi1n
cmVhc2UgNnwxYCBXVyA/KCQmIHt9NAo3OTZVUHR2UXkvaEFwY0ZBdEJsaFpsbHJ0
cklKcDVHcEdWMEdPSkpnN0FiRU43RW5hUWFMdjR3WFRRSFBLSGlmClM3cTNJWlNM
TExkdHdXUHJISkNIaE1TTUxUc1NUWkV1a09HeFU3bVZwQXMKLS0tIGhOcXFTUElS
azJJNnEreUhMWTJBaVZGSTJPRUFqQkVYS01KRENUVVpZSDgK8+8onFejroBo7MeO
dW+so4lOsq4zJKn3f0cxmCFg1f0X8zt6h4Uc3A5Cvr1uU+6yw1FWmJ7xa3jJz3lO
EEaKQJXYC+xIIKGcA7qILa0SFp4a/4OuYjcg27HrlPhg7u5wDhQrd0LdVEe1Xngp
ZivX7P7HwIna3X8C+TL+K2v/AG2N/z86cdKfRvxyMKNbHhYw+CfHEnWgh8tJ++4h
G9evNniuNqte6cQaRe7jODfPNW4FuY/Sb7barlJ/M9iAQdYAdyLAzU1LABeHeUfD
wtHjxy9DUZ55Vg8bB8M2JJU9MkoRT4ewiVd9LeC1GWeVmKsm93wsmrov714i7U2j
wHtDkjqEF2MmzuQc18sjNaAHiwz8j6o5xU2L/Q4+Q707yISWG7RGZYh389Cr1rnw
siUq/Vunqw2wk13+J/4vu9nqt5mMktBaCtp+QiWIurjwB5LUAyChrSm+dg5lb0Mt
UhSc0lq1+E3vxAXM2Hmk+vP86VD+6WJvAU82VFApF1s6zG2FU1/AcOVVf54nan/q
f+rgSFfASHQCYSblUJHyEtwLNsWEmTGmOEn1buUKD/H0zatPQnc0rYpjlx2V0Sjd
6yB5+wPrZ0AkN1pjcsPKOv8Kaog2DzqIjib+SaSTaRxWHQEb9uzvaReAcYI5HOpE
gkC040HN33BItATbo4+hz70Im8Ni/VXD+g6yzM6Hj1hJL+PinTKeg5keQRFIZjMx
grzievB2wVBBgLgN3qMdTFmpplaL7iL702JjXZUTTK9Izp+9wiCsV1fTa53FWDht
ylFL5SWElqXjK+QBXxAe+Jk6VQov5HI21YDXL67S554ABeRok23wxrQ31TCI4xq9
PQV7VtNRjyVud7S29m3OwpWOsgTZhn+JclHj2v4bNJzJkJnZRTmcvGPktzRI5+R4
e5vxVhGnJDzI71txaHl8+xS1lu9VzCQUrxX6TXyTRV4KjIOz0g06JOBgmBRBvJca
7MZbC65xpisl/gyLRbgkVga3t94dPV+dpZsn8eq6427IyRbKslJefatggR9//c6I
5N5fl0fR3gJQMB+HRbipBH2YsdbdWJyb4Nn6STZxIfrqoG/xC6C1raF0xK7hUx6i
4DUDSPohM8fOIswQPfE+FH3eygfzu/Ln5+ghsgHTEhgFvmgMvyxaAt6kHIzIUhMX
M3dASr4VPDpIXuXsRWwYLEifhzxsuvwVxfwtsnCaR6XKijsYECWGDdYOWHdleeqx
wDPhxEesfFVhKxhrKY9Ir8k9/FFBKQU/3GjW4+SMAg5Al1YEzxshP9vKuVcsei7W
JDwAwotNXaCm6NBckiyZJE53ou6+gckPY7V9cOfnuH74Z9ywkFzB3HW3ZlonaGyM
oGmLGcccavFtyhg5s/As4i6X8ARIpDiwe59Pn3GNXMctySqIrrr2ogUoXgrfFCie
6GOTdeMW7GeOSdJUxCofghlspS/nq01Og77VI/beWYrIwLubSka6Zaltww9zgObk
/FGEMgFkEpq7iyCvYSPA8F46pJKvnMP3S84AWCPmcTcHeg4lwGPvs6btexXBGdoz
nkCyq7wdH5Nngm7jUbl88LtaLZPAQkuqXphBVTnrF9Ofbnb4iRZ2Op4xpx9rGyvx
mO6UEhL6V1i2YZFNkNMg/W8aoMiUgBdqbkxaxblT9L0aNdlFU9+LbWYolURVEadd
Qjv0Z1gMA+tsuBbVszwsMfneZ5+B9Q==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHNzaC1lZDI1NTE5IEdoTUQ4QSB5c1B3
anRidlRvdUIxUGZWVEw0VDQ5L1QzRHdlVHJHRDRvNEF6M2dNQVNrCk50aXF6ekxq
WnVCUXBZbEJsVk1BMmQvN2ZpNmp3ZHYvSUlvakN2VTdZUVkKLT4gMyR8Syh7ZyMt
Z3JlYXNlIFs3ezgmZDAKZ2hhNzZmU2wzdkNkTnliL0NiWWlQbnppcmRLdHZVYWdm
bG5LZ0p3Ci0tLSBMYmxYa0hCbnVham05dVo5eEJXTEhob2F1YXhzNlNoVXBwNmNz
M0RVS0xRCt2jk8SwZ4McFmBkWknNs0phZSD3deFAh1nvRE9fSp82yNG0SoFE8O8e
HaMP5fO1Y7gW7Nqvvtr0Fs2AXwxzLtVRc8+XYxDuhRY9Rq5w/VHaEt9fS2OQioml
hcTw3MhBGAN4JVhFz/lUOEO82jd3lKig95fGjA2SpwfxlAMhepe7OmJ4Hpe+QrPt
orXgJHG2Ssu2mLgpnIr3mYVXKfNMy9TUGkh5hM1qR2TidtcefExYegCesjRNmSw/
KVe8wTdF1dr+R+vQDGXZleLpLzKZHATw36+GxGUfpm6iWuDc3lBsjHrVVlm2mz7o
XM+bHDnnUHSFXhHh5gDIoRi/kI3w5b1F2psKXFAGOQq4osrYInoGdvMrGWDvhGZS
GbVyAoZKYOmlqjM/AY4vMc5N7ICsDJqugFi9fDSmckfgzvUYVbzuv7D+1gtmjAzi
+e/vhujRjE2xaD7kIeO6UFq0lZpMoQV3tX5kJG0OOuF1GlZzw22SUc67w25JNHSE
GRnm+MN7XfvtlRZyyYERMs6j829H6zOQL96YBpzxc1yx07K+SMKewHhbm4pyrcDD
dVZxNMHv+B+F2tw1KFMl+Go3pyQp0p+CgiSWUoyvBzK0BwZ7XXy93GWmuAN/dhpU
YhXIiqXgexN6a/0Z6OTah8xW1sr6EYvIGWnXdUUB+azEI2uykX00MO10szDJ/vkh
DZUafMtlgMKU+fBbM0Tr0QM1R30BWNJCjHgsoU7mxbyiuFYuyAC72T70z5WdQTst
jf4NJdc4FKOHA6ydLoyzi7ey05nxm8zquHW1kCwLVUMhvjUPbQhpWQbU8DvlB6l1
nGqWBUTqeorstDcvTTEjySPrp7PIUMj+Vt2MiFCQa5eh2sZWFJsYrA6CEtGFTpyU
og8rGtS7UBukgsp7rucLO8MYt0/ZauUcEg0jcbWc+KOWtC9MPedv6gTLfaxpk1xy
iY+7Y+H860Pt1WvVFYg9/0fj0wz92DqR08oWvOjR7+EekhacqAHfTm6PpLLKSeHw
M1D0dElJ6L1hkmHr88q7YwPfMzObmRbKh28ayeoLOprx3ezYr4t0qWWAKysvkkuo
uhvfqngZefj7CloVCaQVM0Du6p74wV5UEkxopJyMc3s4oEkauX2DgVHzb0NJ+a3n
KyatZoIEOK+yBDuRw9+c3FdxOmSc8EEdDPh5tXHOfixdxHlMXeNWo7VwtR7ThPcj
wRHwN9dgzW4AbYQAHKSOCGGqlhINYVxDviwhTT+PYB/w7x42id8aMW2KOhgGcVCD
fyQQJN5ANCxglu/T0+SmXikPQkuJqzOuE0+qqz1sMuKF3KLJB5ppys1Ygmpm27Up
oaqBa7JS4yJc53rPSEy4r10GqBbvYNDmcEWZilSrj5aTB7nhq+RVyTwmAU+qfLmK
NgGi05oyVGYBX135c6eXkEtKg1HiH5LkqkvLGIDYkAeiLmqXv6N7hGphBF7fdLXw
g8Anrn+JYaOc/6ZntCPLwm4cV4KE2hswSReHFrposb6M4/Kbx8Yf2iMx07IQ3v/R
z0sIihHjfOqLi3kafL4N0BDoLAQdjbxqkx1z0VcGVkPliS0mDro17b1KfxPsI04/
k6xTWoBQAhcCkH+NpT0VRNCM7f6aycsJekfakTRRqg5rwFimqReMSJcKCLZdoezi
iAIMHPqPW1+sjwA7Rroj3Th0NbC8vGJ47vobK8jQ4XgrwaUGhhloCTpAjzFLFTUS
YdiJN2qBQfABhxD9Owqo98HXDigqb9jhh/LrPYYDhtP6c4J+zz3+gPRiho7T03H3
kZ+YU3Q=
-----END AGE ENCRYPTED FILE-----

View File

@@ -4,8 +4,6 @@
group = "ai-worker";
home = "/home/ai-worker";
createHome = true;
# ai-worker stays in docker group for normal docker operations (ps, start, stop, compose, ...)
# Dangerous commands (exec, cp, commit) are blocked by a wrapper script.
extraGroups = [ "docker" ];
shell = pkgs.bashInteractive;
openssh.authorizedKeys.keys = [
@@ -16,14 +14,17 @@
};
users.groups.ai-worker = {};
# Enable restricted AI worker SSH access
# SECURITY: ai-worker is in docker group but docker commands are filtered:
# ALLOWED: ps, images, logs, start, stop, restart, rm, rmi, pull, build, run, compose
# BLOCKED: exec, cp, commit, diff, export, import, load, save, attach, push
# The filtering is done by a docker wrapper in ai-worker's PATH.
# Enable restricted AI worker SSH access for ollama benchmarking
# SECURITY: ai-worker can only:
# - SSH into host from Hermes container
# - Run docker commands (docker exec ollama ...) via docker group
# - Run specific security audit commands
# - NO access to infra repo (no bind mount)
# - NO sudo access (no nh, nixos-rebuild, nixpkgs-fmt, nix)
# WORKFLOW: SSH from Hermes container, run docker benchmarks, return and save results to /opt/data/ai-optimizer/
services.aiWorkerAccess = true;
# Restricted sudo for ai-worker - security checks only (not for docker)
# Restricted sudo for ai-worker - security checks only
security.sudo.extraRules = [
{
users = [ "ai-worker" ];
@@ -68,6 +69,15 @@
command = "/run/current-system/sw/bin/sshd -T";
options = [ "NOPASSWD" ];
}
# Docker service checks
{
command = "/run/current-system/sw/bin/docker ps";
options = [ "NOPASSWD" ];
}
{
command = "/run/current-system/sw/bin/docker inspect *";
options = [ "NOPASSWD" ];
}
# Network diagnostics
{
command = "/run/current-system/sw/bin/ss -tlnp";