Compare commits
1 Commits
feat/herme
...
feat/restr
| Author | SHA1 | Date | |
|---|---|---|---|
| 993b9c559c |
Submodule assets/compose updated: ebad994d60...d3f2e3b7b9
@@ -9,10 +9,6 @@
|
||||
ai-worker = {
|
||||
main = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAXeGtPPcsP2IYRQNvII41NVWhJsarEk8c4qxs/a5sXf";
|
||||
};
|
||||
|
||||
hermes_agent = {
|
||||
age = "age178ypgaxn3fldh2aeqz37ncpk7jrplaxacrca8kkcycre3ahjef4s2dp3rp";
|
||||
};
|
||||
};
|
||||
|
||||
hosts = {
|
||||
|
||||
@@ -1,64 +1,74 @@
|
||||
# AI Worker Restricted Access
|
||||
|
||||
This module provides SSH access for the AI worker (hermes-agent) to run ollama benchmarks on the host.
|
||||
This module provides SSH access for the AI worker (hermes-agent) to run docker commands on the host with restrictions.
|
||||
|
||||
## Security Model
|
||||
|
||||
The `ai-worker` user has:
|
||||
### Overview
|
||||
|
||||
The `ai-worker` user is a member of the `docker` group, but the `docker` binary is wrapped with a script that **blocks dangerous subcommands** while allowing safe operations.
|
||||
|
||||
### Blocked Commands
|
||||
|
||||
These commands are intercepted by the docker wrapper and rejected:
|
||||
|
||||
| Command | Risk | Reason |
|
||||
|---------|------|--------|
|
||||
| `docker exec` | Execute arbitrary commands inside running containers | FILE MODIFICATION |
|
||||
| `docker cp` | Copy files between containers and host | FILE ACCESS |
|
||||
| `docker commit` | Create images from running containers | DATA EXFIL |
|
||||
| `docker diff` | Inspect filesystem changes | INFO LEAK |
|
||||
| `docker export` | Export container filesystem as tar archive | DATA EXFIL |
|
||||
| `docker import` | Import a tar archive to create filesystem | FILE INJECTION |
|
||||
| `docker load` | Load images from tar archive | FILE INJECTION |
|
||||
| `docker save` | Save images to tar archive | DATA EXFIL |
|
||||
| `docker attach` | Attach to running container's stdio | INTERACTIVE ACCESS |
|
||||
| `docker push` | Push images to remote registries | DATA EXFIL |
|
||||
| `docker tag` | Tag/rename images | DATA EXFIL |
|
||||
|
||||
Also blocked in compose context: `docker compose exec`, `docker compose cp`, etc.
|
||||
|
||||
### Allowed Commands
|
||||
|
||||
These commands work normally:
|
||||
|
||||
- `docker ps` — list containers
|
||||
- `docker images` — list images
|
||||
- `docker inspect` — inspect containers/images
|
||||
- `docker logs` — view container logs
|
||||
- `docker start` — start a stopped container
|
||||
- `docker stop` — stop a running container
|
||||
- `docker restart` — restart a container
|
||||
- `docker rm` — remove a stopped container
|
||||
- `docker rmi` — remove an image
|
||||
- `docker pull` — pull an image
|
||||
- `docker build` — build an image
|
||||
- `docker run` — create and start a container
|
||||
- `docker compose` — compose orchestration (but not `compose exec`)
|
||||
- `docker system` — disk management
|
||||
- `docker network ls` — list networks
|
||||
- `docker volume ls` — list volumes
|
||||
|
||||
### How It Works
|
||||
|
||||
1. A wrapper script intercepts `docker` calls in the user's PATH
|
||||
2. It parses the first non-flag argument to determine the subcommand
|
||||
3. If the subcommand is in the blocklist, it prints an error and exits
|
||||
4. Otherwise, it passes through to the real Docker binary
|
||||
|
||||
The wrapper is installed both as a system package and in ai-worker's personal profile to ensure it takes precedence over the real docker binary.
|
||||
|
||||
### Why Not Use Docker Authorization Plugins?
|
||||
|
||||
Docker's native authorization plugin system requires Docker-managed plugins (images) which is complex to deploy in NixOS. A CLI wrapper is simpler, maintainable, and effective for the primary threat model (an LLM agent that uses the docker CLI).
|
||||
|
||||
Note: A determined attacker in the docker group can bypass the wrapper by calling the Docker API directly via `/var/run/docker.sock`. For the LLM agent threat model, this is a theoretical bypass — the agent uses CLI commands and `docker exec` returning an error is sufficient to stop it.
|
||||
|
||||
### Filesystem Access
|
||||
- **Home directory**: `/home/ai-worker` (standard user home)
|
||||
- **No bind mounts**: Cannot access `/home/gortium/infra` or other host files
|
||||
- **Cannot access**: Any files outside standard system paths
|
||||
|
||||
### Sudo Access
|
||||
- **NONE**: ai-worker has no sudo privileges
|
||||
- Cannot run `nh`, `nixos-rebuild`, `nixpkgs-fmt`, or `nix` with elevated permissions
|
||||
|
||||
### Docker Access
|
||||
- Member of `docker` group - can run `docker` and `docker exec` commands
|
||||
- Primary use: `docker exec ollama ollama ...` for benchmarking
|
||||
- Can run `docker exec --privileged ollama rocm-smi ...` for VRAM monitoring
|
||||
|
||||
## Workflow: SSH + Docker Benchmarking
|
||||
|
||||
The AI worker connects from the Hermes container to the host via SSH, runs ollama benchmarks, then returns to save results.
|
||||
|
||||
### Example Workflow
|
||||
|
||||
```bash
|
||||
# From Hermes container, SSH to host
|
||||
ssh -i /path/to/ssh/key ai-worker@host.docker.internal
|
||||
|
||||
# On host, run ollama benchmarks via docker
|
||||
docker exec ollama ollama pull devstral-small-2:24b
|
||||
|
||||
# Create test modelfile
|
||||
docker exec ollama bash -c 'cat <<EOF > /root/.ollama/test.modelfile
|
||||
FROM devstral-small-2:24b
|
||||
PARAMETER num_ctx 65536
|
||||
PARAMETER num_gpu 99
|
||||
PARAMETER flash_attn true
|
||||
EOF'
|
||||
|
||||
# Create and test model
|
||||
docker exec ollama ollama create test-model -f /root/.ollama/test.modelfile
|
||||
docker exec ollama ollama run test-model "Write a Python async function"
|
||||
|
||||
# Check VRAM usage
|
||||
docker exec --privileged ollama rocm-smi --showmeminfo vram
|
||||
|
||||
# Cleanup
|
||||
docker exec ollama ollama rm test-model
|
||||
|
||||
# Exit SSH, return to Hermes container
|
||||
exit
|
||||
|
||||
# Save results in Hermes container
|
||||
# /opt/data/ai-optimizer/state.json
|
||||
# /opt/data/ai-optimizer/results.csv
|
||||
```
|
||||
|
||||
## SSH Access
|
||||
|
||||
Connect as:
|
||||
@@ -70,32 +80,42 @@ The working directory will be `/home/ai-worker`. No infra repo access.
|
||||
|
||||
## Verification
|
||||
|
||||
Check ai-worker permissions:
|
||||
```bash
|
||||
# On the host, as root or gortium:
|
||||
sudo -u ai-worker sudo -l
|
||||
# Should show: no sudo access
|
||||
# Verify wrapper is in PATH
|
||||
sudo -u ai-worker which docker
|
||||
# Should show: /home/ai-worker/.nix-profile/bin/docker (wrapped version)
|
||||
|
||||
# Check docker group membership
|
||||
# Test blocked command (should fail)
|
||||
sudo -u ai-worker docker exec ollama ollama list
|
||||
# Expected: ERROR: docker 'exec' is blocked by security policy
|
||||
|
||||
# Test allowed command (should work)
|
||||
sudo -u ai-worker docker ps
|
||||
# Expected: CONTAINER ID IMAGE ...
|
||||
|
||||
# Verify docker group membership
|
||||
groups ai-worker
|
||||
# Should show: ai-worker docker
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If ai-worker cannot run docker commands:
|
||||
If docker commands fail unexpectedly:
|
||||
|
||||
```bash
|
||||
# Check docker group membership
|
||||
groups ai-worker
|
||||
# Check which docker binary is being used
|
||||
which docker
|
||||
# If this shows /run/current-system/sw/bin/docker, the wrapper is not in PATH
|
||||
|
||||
# Verify ollama container is running
|
||||
docker ps | grep ollama
|
||||
# Check if the wrapper is installed
|
||||
ls -la $(which docker)
|
||||
|
||||
# Test docker access
|
||||
sudo -u ai-worker docker exec ollama ollama list
|
||||
# Verify you're running as the right user
|
||||
whoami
|
||||
```
|
||||
|
||||
If SSH connection fails:
|
||||
|
||||
```bash
|
||||
# Check SSH key is authorized
|
||||
cat /home/ai-worker/.ssh/authorized_keys
|
||||
|
||||
@@ -2,16 +2,123 @@
|
||||
|
||||
with lib;
|
||||
|
||||
let
|
||||
# Docker subcommands that are BLOCKED for ai-worker
|
||||
# These commands allow file modification inside containers or data exfiltration.
|
||||
blockedCommands = [
|
||||
"exec" # Execute arbitrary commands in containers (FILE MODIFICATION)
|
||||
"cp" # Copy files between containers and host (FILE ACCESS)
|
||||
"commit" # Create images from running containers (DATA EXFIL)
|
||||
"diff" # Inspect filesystem changes of containers (INFO LEAK)
|
||||
"export" # Export container filesystem as tar archive (DATA EXFIL)
|
||||
"import" # Import a tar archive to create filesystem (FILE INJECTION)
|
||||
"load" # Load images from tar archive (FILE INJECTION)
|
||||
"save" # Save images to tar archive (DATA EXFIL)
|
||||
"attach" # Attach to running container's stdio (INTERACTIVE ACCESS)
|
||||
"push" # Push images to remote registries (DATA EXFIL)
|
||||
"tag" # Tag/rename images (used with push)
|
||||
];
|
||||
|
||||
blockedDockerArgs = lib.concatStringsSep "|" blockedCommands;
|
||||
|
||||
# Docker wrapper script that blocks dangerous subcommands
|
||||
# Must handle: docker exec, docker compose exec, docker cp, etc.
|
||||
restrictedDockerScript = pkgs.writeShellScriptBin "docker" ''
|
||||
set -e
|
||||
|
||||
# Blocklist pattern
|
||||
BLOCKED_PATTERN="^(${blockedDockerArgs})$"
|
||||
|
||||
# Parse the first non-flag argument to find the docker subcommand
|
||||
# Flags: -H, --host, -D, --debug, --config, --context, --log-level, -l
|
||||
# Also handle: docker compose <subcommand> (subcommand may be after 'compose')
|
||||
SUBCOMMAND=""
|
||||
COMPOSE_MODE=false
|
||||
FOUND_ARG=false
|
||||
|
||||
for arg in "$@"; do
|
||||
# Skip flags and their values
|
||||
case "$arg" in
|
||||
-H|--host|-l|--log-level|--config|--context|-D|--debug)
|
||||
FOUND_ARG=true
|
||||
continue
|
||||
;;
|
||||
--tls|--tlsverify|--tlscacert|--tlscert|--tlskey)
|
||||
if $FOUND_ARG; then FOUND_ARG=false; else continue; fi
|
||||
;;
|
||||
# Skip flag values (the next arg after a flag that takes a value)
|
||||
-*)
|
||||
continue
|
||||
;;
|
||||
*)
|
||||
# This is a positional argument — first one is the subcommand (or 'compose')
|
||||
if [ -z "$SUBCOMMAND" ]; then
|
||||
if [ "$arg" = "compose" ]; then
|
||||
COMPOSE_MODE=true
|
||||
continue
|
||||
fi
|
||||
SUBCOMMAND="$arg"
|
||||
break
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
FOUND_ARG=false
|
||||
done
|
||||
|
||||
# If in compose mode, the subcommand is after 'compose'
|
||||
if $COMPOSE_MODE; then
|
||||
# In compose mode, we check the sub-subcommand
|
||||
NEXT_GOT=""
|
||||
for arg in "$@"; do
|
||||
if [ "$NEXT_GOT" = "true" ]; then
|
||||
if echo "$arg" | grep -qE "$BLOCKED_PATTERN"; then
|
||||
echo "ERROR: docker compose '$arg' is blocked by security policy" >&2
|
||||
echo "This command can modify files inside containers." >&2
|
||||
exit 1
|
||||
fi
|
||||
break
|
||||
fi
|
||||
if [ "$arg" = "compose" ]; then
|
||||
NEXT_GOT="true"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# Check if the subcommand is blocked
|
||||
if [ -n "$SUBCOMMAND" ]; then
|
||||
if echo "$SUBCOMMAND" | grep -qE "$BLOCKED_PATTERN"; then
|
||||
echo "ERROR: docker '$SUBCOMMAND' is blocked by security policy" >&2
|
||||
echo "This command can modify files inside containers." >&2
|
||||
echo "" >&2
|
||||
echo "Allowed commands: ps, images, inspect, logs, start, stop, restart," >&2
|
||||
echo " rm, rmi, pull, build, run, compose, system, network ls, volume ls" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Execute the real docker binary
|
||||
exec ${pkgs.docker}/bin/docker "$@"
|
||||
'';
|
||||
in
|
||||
{
|
||||
options.services.aiWorkerAccess = mkOption {
|
||||
type = types.bool;
|
||||
default = false;
|
||||
description = "Enable AI worker SSH access with docker group membership for ollama benchmarking";
|
||||
description = "Enable AI worker SSH access with restricted docker commands";
|
||||
};
|
||||
|
||||
config = mkIf config.services.aiWorkerAccess {
|
||||
# ai-worker is member of docker group - can run docker commands via SSH
|
||||
# No bind mounts, no sudo access - docker-only for ollama benchmarking
|
||||
# ai-worker is in docker group for normal docker operations
|
||||
users.groups.docker.members = [ "ai-worker" ];
|
||||
|
||||
# Install the docker wrapper for ai-worker
|
||||
# This puts a filtered 'docker' script in ai-worker's PATH that blocks
|
||||
# dangerous commands like exec, cp, commit, etc.
|
||||
# The real docker binary is still available at its store path, but the
|
||||
# wrapper intercepts it because ~/.nix-profile/bin/ comes before /run/.../sw/bin/ in PATH.
|
||||
users.users.ai-worker.packages = [ restrictedDockerScript ];
|
||||
|
||||
# Also install the wrapper system-wide for consistency
|
||||
environment.systemPackages = [ restrictedDockerScript ];
|
||||
};
|
||||
}
|
||||
|
||||
Binary file not shown.
@@ -4,7 +4,6 @@ let
|
||||
keys.users.gortium.main
|
||||
keys.hosts.lazyworkhorse.main
|
||||
keys.hosts.lazyworkhorse.bootstrap
|
||||
keys.users.hermes_agent.age
|
||||
];
|
||||
in
|
||||
{
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
group = "ai-worker";
|
||||
home = "/home/ai-worker";
|
||||
createHome = true;
|
||||
# ai-worker stays in docker group for normal docker operations (ps, start, stop, compose, ...)
|
||||
# Dangerous commands (exec, cp, commit) are blocked by a wrapper script.
|
||||
extraGroups = [ "docker" ];
|
||||
shell = pkgs.bashInteractive;
|
||||
openssh.authorizedKeys.keys = [
|
||||
@@ -14,17 +16,14 @@
|
||||
};
|
||||
users.groups.ai-worker = {};
|
||||
|
||||
# Enable restricted AI worker SSH access for ollama benchmarking
|
||||
# SECURITY: ai-worker can only:
|
||||
# - SSH into host from Hermes container
|
||||
# - Run docker commands (docker exec ollama ...) via docker group
|
||||
# - Run specific security audit commands
|
||||
# - NO access to infra repo (no bind mount)
|
||||
# - NO sudo access (no nh, nixos-rebuild, nixpkgs-fmt, nix)
|
||||
# WORKFLOW: SSH from Hermes container, run docker benchmarks, return and save results to /opt/data/ai-optimizer/
|
||||
# Enable restricted AI worker SSH access
|
||||
# SECURITY: ai-worker is in docker group but docker commands are filtered:
|
||||
# ALLOWED: ps, images, logs, start, stop, restart, rm, rmi, pull, build, run, compose
|
||||
# BLOCKED: exec, cp, commit, diff, export, import, load, save, attach, push
|
||||
# The filtering is done by a docker wrapper in ai-worker's PATH.
|
||||
services.aiWorkerAccess = true;
|
||||
|
||||
# Restricted sudo for ai-worker - security checks only
|
||||
|
||||
# Restricted sudo for ai-worker - security checks only (not for docker)
|
||||
security.sudo.extraRules = [
|
||||
{
|
||||
users = [ "ai-worker" ];
|
||||
@@ -69,15 +68,6 @@
|
||||
command = "/run/current-system/sw/bin/sshd -T";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
# Docker service checks
|
||||
{
|
||||
command = "/run/current-system/sw/bin/docker ps";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
{
|
||||
command = "/run/current-system/sw/bin/docker inspect *";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
# Network diagnostics
|
||||
{
|
||||
command = "/run/current-system/sw/bin/ss -tlnp";
|
||||
|
||||
Reference in New Issue
Block a user