fix: ai-worker docker-only access for ollama benchmarking
Remove infra repo bind mount and sudo access from ai-worker user. Now ai-worker can only: - SSH into host from Hermes container - Run docker commands via docker group membership - Execute ollama benchmarks via docker exec Results saved to /opt/data/ai-optimizer/ in Hermes container.
This commit is contained in:
@@ -1,54 +1,62 @@
|
||||
# AI Worker Restricted Access
|
||||
|
||||
This module provides restricted access for the AI worker (hermes-agent) to manage the infra repository.
|
||||
This module provides SSH access for the AI worker (hermes-agent) to run ollama benchmarks on the host.
|
||||
|
||||
## Security Model
|
||||
|
||||
The `ai-worker` user has:
|
||||
|
||||
### Filesystem Access
|
||||
- **Bind mount**: `/home/ai-worker/infra` → `/home/gortium/infra` (read-write)
|
||||
- **Cannot access**: Any other files outside the bind mount and standard system paths
|
||||
- **Home directory**: `/home/ai-worker` (standard user home)
|
||||
- **No bind mounts**: Cannot access `/home/gortium/infra` or other host files
|
||||
- **Cannot access**: Any files outside standard system paths
|
||||
|
||||
### Sudo Access (Whitelist Only)
|
||||
The following commands are allowed via sudo without password:
|
||||
- `/run/current-system/sw/bin/nh` - NixOS home manager
|
||||
- `/run/current-system/sw/bin/nixos-rebuild` - System rebuild
|
||||
- `/run/current-system/sw/bin/nixpkgs-fmt` - Nix formatter
|
||||
- `/run/current-system/sw/bin/nix` - Nix package manager
|
||||
### Sudo Access
|
||||
- **NONE**: ai-worker has no sudo privileges
|
||||
- Cannot run `nh`, `nixos-rebuild`, `nixpkgs-fmt`, or `nix` with elevated permissions
|
||||
|
||||
### Docker Access
|
||||
- Member of `docker` group - can manage containers
|
||||
- Cannot modify host system directly
|
||||
- Member of `docker` group - can run `docker` and `docker exec` commands
|
||||
- Primary use: `docker exec ollama ollama ...` for benchmarking
|
||||
- Can run `docker exec --privileged ollama rocm-smi ...` for VRAM monitoring
|
||||
|
||||
### Audit Logging
|
||||
- All changes to `/home/gortium/infra` are logged via Linux audit subsystem
|
||||
- Audit rule: `-w /home/gortium/infra -p wa -k infra_changes`
|
||||
## Workflow: SSH + Docker Benchmarking
|
||||
|
||||
## Workflow: Ask First, Always
|
||||
|
||||
**CRITICAL**: Before running any deployment command (`nh os switch` or `nixos-rebuild`), the AI MUST:
|
||||
|
||||
1. **Show the planned changes** to the user
|
||||
2. **Explain the impact** of the changes
|
||||
3. **Wait for explicit confirmation** before executing
|
||||
The AI worker connects from the Hermes container to the host via SSH, runs ollama benchmarks, then returns to save results.
|
||||
|
||||
### Example Workflow
|
||||
|
||||
```bash
|
||||
# AI prepares changes
|
||||
cd /home/ai-worker/infra
|
||||
# ... edits files ...
|
||||
nixpkgs-fmt .
|
||||
# From Hermes container, SSH to host
|
||||
ssh -i /path/to/ssh/key ai-worker@host.docker.internal
|
||||
|
||||
# AI shows diff to user
|
||||
git diff
|
||||
# On host, run ollama benchmarks via docker
|
||||
docker exec ollama ollama pull devstral-small-2:24b
|
||||
|
||||
# AI asks: "Ready to deploy? This will restart the ai_stack service."
|
||||
# User responds: "Yes, proceed"
|
||||
# Create test modelfile
|
||||
docker exec ollama bash -c 'cat <<EOF > /root/.ollama/test.modelfile
|
||||
FROM devstral-small-2:24b
|
||||
PARAMETER num_ctx 65536
|
||||
PARAMETER num_gpu 99
|
||||
PARAMETER flash_attn true
|
||||
EOF'
|
||||
|
||||
# Only then does AI run:
|
||||
sudo nh os switch --flake .#lazyworkhorse
|
||||
# Create and test model
|
||||
docker exec ollama ollama create test-model -f /root/.ollama/test.modelfile
|
||||
docker exec ollama ollama run test-model "Write a Python async function"
|
||||
|
||||
# Check VRAM usage
|
||||
docker exec --privileged ollama rocm-smi --showmeminfo vram
|
||||
|
||||
# Cleanup
|
||||
docker exec ollama ollama rm test-model
|
||||
|
||||
# Exit SSH, return to Hermes container
|
||||
exit
|
||||
|
||||
# Save results in Hermes container
|
||||
# /opt/data/ai-optimizer/state.json
|
||||
# /opt/data/ai-optimizer/results.csv
|
||||
```
|
||||
|
||||
## SSH Access
|
||||
@@ -58,7 +66,7 @@ Connect as:
|
||||
ssh ai-worker@lazyworkhorse
|
||||
```
|
||||
|
||||
The working directory will be `/home/ai-worker`, with infra repo accessible at `/home/ai-worker/infra`.
|
||||
The working directory will be `/home/ai-worker`. No infra repo access.
|
||||
|
||||
## Verification
|
||||
|
||||
@@ -66,27 +74,32 @@ Check ai-worker permissions:
|
||||
```bash
|
||||
# On the host, as root or gortium:
|
||||
sudo -u ai-worker sudo -l
|
||||
```
|
||||
# Should show: no sudo access
|
||||
|
||||
Expected output should show only the whitelisted commands.
|
||||
# Check docker group membership
|
||||
groups ai-worker
|
||||
# Should show: ai-worker docker
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If ai-worker cannot access infra:
|
||||
If ai-worker cannot run docker commands:
|
||||
```bash
|
||||
# Check bind mount
|
||||
mount | grep ai-worker/infra
|
||||
# Check docker group membership
|
||||
groups ai-worker
|
||||
|
||||
# Check permissions
|
||||
ls -la /home/gortium/infra
|
||||
ls -la /home/ai-worker/infra
|
||||
# Verify ollama container is running
|
||||
docker ps | grep ollama
|
||||
|
||||
# Test docker access
|
||||
sudo -u ai-worker docker exec ollama ollama list
|
||||
```
|
||||
|
||||
If sudo commands fail:
|
||||
If SSH connection fails:
|
||||
```bash
|
||||
# Check sudo rules
|
||||
sudo cat /etc/sudoers.d/* | grep ai-worker
|
||||
# Check SSH key is authorized
|
||||
cat /home/ai-worker/.ssh/authorized_keys
|
||||
|
||||
# Check audit logs
|
||||
sudo ausearch -k infra_changes
|
||||
# Check SSH service
|
||||
systemctl status sshd
|
||||
```
|
||||
|
||||
@@ -6,52 +6,12 @@ with lib;
|
||||
options.services.aiWorkerAccess = mkOption {
|
||||
type = types.bool;
|
||||
default = false;
|
||||
description = "Enable restricted AI worker access to infra repo with deployment capabilities";
|
||||
description = "Enable AI worker SSH access with docker group membership for ollama benchmarking";
|
||||
};
|
||||
|
||||
config = mkIf config.services.aiWorkerAccess {
|
||||
# Bind mount for infra repo access (read-write for editing)
|
||||
fileSystems."/home/ai-worker/infra" = {
|
||||
device = "/home/gortium/infra";
|
||||
fsType = "none";
|
||||
options = [ "bind" ];
|
||||
};
|
||||
|
||||
# Restricted sudo access - only specific commands allowed
|
||||
security.sudo.extraRules = [
|
||||
{
|
||||
users = [ "ai-worker" ];
|
||||
commands = [
|
||||
{
|
||||
command = "/run/current-system/sw/bin/nh";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
{
|
||||
command = "/run/current-system/sw/bin/nixos-rebuild";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
{
|
||||
command = "/run/current-system/sw/bin/nixpkgs-fmt";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
{
|
||||
command = "/run/current-system/sw/bin/nix";
|
||||
options = [ "NOPASSWD" ];
|
||||
}
|
||||
];
|
||||
}
|
||||
];
|
||||
|
||||
# Ensure ai-worker has necessary tools available
|
||||
environment.systemPackages = with pkgs; [
|
||||
nh
|
||||
nixpkgs-fmt
|
||||
];
|
||||
|
||||
# Audit logging for ai-worker actions on infra directory
|
||||
security.audit.enable = mkDefault true;
|
||||
security.audit.rules = [
|
||||
"-w /home/gortium/infra -p wa -k infra_changes"
|
||||
];
|
||||
# ai-worker is member of docker group - can run docker commands via SSH
|
||||
# No bind mounts, no sudo access - docker-only for ollama benchmarking
|
||||
users.groups.docker.members = [ "ai-worker" ];
|
||||
};
|
||||
}
|
||||
|
||||
@@ -14,12 +14,12 @@
|
||||
};
|
||||
users.groups.ai-worker = {};
|
||||
|
||||
# Enable restricted AI worker access with deployment capabilities
|
||||
# Enable restricted AI worker SSH access for ollama benchmarking
|
||||
# SECURITY: ai-worker can only:
|
||||
# - Access /home/ai-worker/infra (bind-mounted to /home/gortium/infra)
|
||||
# - Run: nh, nixos-rebuild, nixpkgs-fmt, nix (via sudo, no password)
|
||||
# - Manage docker containers (via docker group)
|
||||
# - All changes to infra/ are logged via audit subsystem
|
||||
# WORKFLOW: AI must ask for user confirmation before running nh os switch
|
||||
# - SSH into host from Hermes container
|
||||
# - Run docker commands (docker exec ollama ...) via docker group
|
||||
# - NO access to infra repo (no bind mount)
|
||||
# - NO sudo access (no nh, nixos-rebuild, nixpkgs-fmt, nix)
|
||||
# WORKFLOW: SSH from Hermes container, run docker benchmarks, return and save results to /opt/data/ai-optimizer/
|
||||
services.aiWorkerAccess = true;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user