fix: ai-worker docker-only access for ollama benchmarking
Remove infra repo bind mount and sudo access from ai-worker user. Now ai-worker can only: - SSH into host from Hermes container - Run docker commands via docker group membership - Execute ollama benchmarks via docker exec Results saved to /opt/data/ai-optimizer/ in Hermes container.
This commit is contained in:
@@ -1,54 +1,62 @@
|
|||||||
# AI Worker Restricted Access
|
# AI Worker Restricted Access
|
||||||
|
|
||||||
This module provides restricted access for the AI worker (hermes-agent) to manage the infra repository.
|
This module provides SSH access for the AI worker (hermes-agent) to run ollama benchmarks on the host.
|
||||||
|
|
||||||
## Security Model
|
## Security Model
|
||||||
|
|
||||||
The `ai-worker` user has:
|
The `ai-worker` user has:
|
||||||
|
|
||||||
### Filesystem Access
|
### Filesystem Access
|
||||||
- **Bind mount**: `/home/ai-worker/infra` → `/home/gortium/infra` (read-write)
|
- **Home directory**: `/home/ai-worker` (standard user home)
|
||||||
- **Cannot access**: Any other files outside the bind mount and standard system paths
|
- **No bind mounts**: Cannot access `/home/gortium/infra` or other host files
|
||||||
|
- **Cannot access**: Any files outside standard system paths
|
||||||
|
|
||||||
### Sudo Access (Whitelist Only)
|
### Sudo Access
|
||||||
The following commands are allowed via sudo without password:
|
- **NONE**: ai-worker has no sudo privileges
|
||||||
- `/run/current-system/sw/bin/nh` - NixOS home manager
|
- Cannot run `nh`, `nixos-rebuild`, `nixpkgs-fmt`, or `nix` with elevated permissions
|
||||||
- `/run/current-system/sw/bin/nixos-rebuild` - System rebuild
|
|
||||||
- `/run/current-system/sw/bin/nixpkgs-fmt` - Nix formatter
|
|
||||||
- `/run/current-system/sw/bin/nix` - Nix package manager
|
|
||||||
|
|
||||||
### Docker Access
|
### Docker Access
|
||||||
- Member of `docker` group - can manage containers
|
- Member of `docker` group - can run `docker` and `docker exec` commands
|
||||||
- Cannot modify host system directly
|
- Primary use: `docker exec ollama ollama ...` for benchmarking
|
||||||
|
- Can run `docker exec --privileged ollama rocm-smi ...` for VRAM monitoring
|
||||||
|
|
||||||
### Audit Logging
|
## Workflow: SSH + Docker Benchmarking
|
||||||
- All changes to `/home/gortium/infra` are logged via Linux audit subsystem
|
|
||||||
- Audit rule: `-w /home/gortium/infra -p wa -k infra_changes`
|
|
||||||
|
|
||||||
## Workflow: Ask First, Always
|
The AI worker connects from the Hermes container to the host via SSH, runs ollama benchmarks, then returns to save results.
|
||||||
|
|
||||||
**CRITICAL**: Before running any deployment command (`nh os switch` or `nixos-rebuild`), the AI MUST:
|
|
||||||
|
|
||||||
1. **Show the planned changes** to the user
|
|
||||||
2. **Explain the impact** of the changes
|
|
||||||
3. **Wait for explicit confirmation** before executing
|
|
||||||
|
|
||||||
### Example Workflow
|
### Example Workflow
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# AI prepares changes
|
# From Hermes container, SSH to host
|
||||||
cd /home/ai-worker/infra
|
ssh -i /path/to/ssh/key ai-worker@host.docker.internal
|
||||||
# ... edits files ...
|
|
||||||
nixpkgs-fmt .
|
|
||||||
|
|
||||||
# AI shows diff to user
|
# On host, run ollama benchmarks via docker
|
||||||
git diff
|
docker exec ollama ollama pull devstral-small-2:24b
|
||||||
|
|
||||||
# AI asks: "Ready to deploy? This will restart the ai_stack service."
|
# Create test modelfile
|
||||||
# User responds: "Yes, proceed"
|
docker exec ollama bash -c 'cat <<EOF > /root/.ollama/test.modelfile
|
||||||
|
FROM devstral-small-2:24b
|
||||||
|
PARAMETER num_ctx 65536
|
||||||
|
PARAMETER num_gpu 99
|
||||||
|
PARAMETER flash_attn true
|
||||||
|
EOF'
|
||||||
|
|
||||||
# Only then does AI run:
|
# Create and test model
|
||||||
sudo nh os switch --flake .#lazyworkhorse
|
docker exec ollama ollama create test-model -f /root/.ollama/test.modelfile
|
||||||
|
docker exec ollama ollama run test-model "Write a Python async function"
|
||||||
|
|
||||||
|
# Check VRAM usage
|
||||||
|
docker exec --privileged ollama rocm-smi --showmeminfo vram
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
docker exec ollama ollama rm test-model
|
||||||
|
|
||||||
|
# Exit SSH, return to Hermes container
|
||||||
|
exit
|
||||||
|
|
||||||
|
# Save results in Hermes container
|
||||||
|
# /opt/data/ai-optimizer/state.json
|
||||||
|
# /opt/data/ai-optimizer/results.csv
|
||||||
```
|
```
|
||||||
|
|
||||||
## SSH Access
|
## SSH Access
|
||||||
@@ -58,7 +66,7 @@ Connect as:
|
|||||||
ssh ai-worker@lazyworkhorse
|
ssh ai-worker@lazyworkhorse
|
||||||
```
|
```
|
||||||
|
|
||||||
The working directory will be `/home/ai-worker`, with infra repo accessible at `/home/ai-worker/infra`.
|
The working directory will be `/home/ai-worker`. No infra repo access.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
@@ -66,27 +74,32 @@ Check ai-worker permissions:
|
|||||||
```bash
|
```bash
|
||||||
# On the host, as root or gortium:
|
# On the host, as root or gortium:
|
||||||
sudo -u ai-worker sudo -l
|
sudo -u ai-worker sudo -l
|
||||||
```
|
# Should show: no sudo access
|
||||||
|
|
||||||
Expected output should show only the whitelisted commands.
|
# Check docker group membership
|
||||||
|
groups ai-worker
|
||||||
|
# Should show: ai-worker docker
|
||||||
|
```
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
If ai-worker cannot access infra:
|
If ai-worker cannot run docker commands:
|
||||||
```bash
|
```bash
|
||||||
# Check bind mount
|
# Check docker group membership
|
||||||
mount | grep ai-worker/infra
|
groups ai-worker
|
||||||
|
|
||||||
# Check permissions
|
# Verify ollama container is running
|
||||||
ls -la /home/gortium/infra
|
docker ps | grep ollama
|
||||||
ls -la /home/ai-worker/infra
|
|
||||||
|
# Test docker access
|
||||||
|
sudo -u ai-worker docker exec ollama ollama list
|
||||||
```
|
```
|
||||||
|
|
||||||
If sudo commands fail:
|
If SSH connection fails:
|
||||||
```bash
|
```bash
|
||||||
# Check sudo rules
|
# Check SSH key is authorized
|
||||||
sudo cat /etc/sudoers.d/* | grep ai-worker
|
cat /home/ai-worker/.ssh/authorized_keys
|
||||||
|
|
||||||
# Check audit logs
|
# Check SSH service
|
||||||
sudo ausearch -k infra_changes
|
systemctl status sshd
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -6,52 +6,12 @@ with lib;
|
|||||||
options.services.aiWorkerAccess = mkOption {
|
options.services.aiWorkerAccess = mkOption {
|
||||||
type = types.bool;
|
type = types.bool;
|
||||||
default = false;
|
default = false;
|
||||||
description = "Enable restricted AI worker access to infra repo with deployment capabilities";
|
description = "Enable AI worker SSH access with docker group membership for ollama benchmarking";
|
||||||
};
|
};
|
||||||
|
|
||||||
config = mkIf config.services.aiWorkerAccess {
|
config = mkIf config.services.aiWorkerAccess {
|
||||||
# Bind mount for infra repo access (read-write for editing)
|
# ai-worker is member of docker group - can run docker commands via SSH
|
||||||
fileSystems."/home/ai-worker/infra" = {
|
# No bind mounts, no sudo access - docker-only for ollama benchmarking
|
||||||
device = "/home/gortium/infra";
|
users.groups.docker.members = [ "ai-worker" ];
|
||||||
fsType = "none";
|
|
||||||
options = [ "bind" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
# Restricted sudo access - only specific commands allowed
|
|
||||||
security.sudo.extraRules = [
|
|
||||||
{
|
|
||||||
users = [ "ai-worker" ];
|
|
||||||
commands = [
|
|
||||||
{
|
|
||||||
command = "/run/current-system/sw/bin/nh";
|
|
||||||
options = [ "NOPASSWD" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
command = "/run/current-system/sw/bin/nixos-rebuild";
|
|
||||||
options = [ "NOPASSWD" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
command = "/run/current-system/sw/bin/nixpkgs-fmt";
|
|
||||||
options = [ "NOPASSWD" ];
|
|
||||||
}
|
|
||||||
{
|
|
||||||
command = "/run/current-system/sw/bin/nix";
|
|
||||||
options = [ "NOPASSWD" ];
|
|
||||||
}
|
|
||||||
];
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
# Ensure ai-worker has necessary tools available
|
|
||||||
environment.systemPackages = with pkgs; [
|
|
||||||
nh
|
|
||||||
nixpkgs-fmt
|
|
||||||
];
|
|
||||||
|
|
||||||
# Audit logging for ai-worker actions on infra directory
|
|
||||||
security.audit.enable = mkDefault true;
|
|
||||||
security.audit.rules = [
|
|
||||||
"-w /home/gortium/infra -p wa -k infra_changes"
|
|
||||||
];
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,12 +14,12 @@
|
|||||||
};
|
};
|
||||||
users.groups.ai-worker = {};
|
users.groups.ai-worker = {};
|
||||||
|
|
||||||
# Enable restricted AI worker access with deployment capabilities
|
# Enable restricted AI worker SSH access for ollama benchmarking
|
||||||
# SECURITY: ai-worker can only:
|
# SECURITY: ai-worker can only:
|
||||||
# - Access /home/ai-worker/infra (bind-mounted to /home/gortium/infra)
|
# - SSH into host from Hermes container
|
||||||
# - Run: nh, nixos-rebuild, nixpkgs-fmt, nix (via sudo, no password)
|
# - Run docker commands (docker exec ollama ...) via docker group
|
||||||
# - Manage docker containers (via docker group)
|
# - NO access to infra repo (no bind mount)
|
||||||
# - All changes to infra/ are logged via audit subsystem
|
# - NO sudo access (no nh, nixos-rebuild, nixpkgs-fmt, nix)
|
||||||
# WORKFLOW: AI must ask for user confirmation before running nh os switch
|
# WORKFLOW: SSH from Hermes container, run docker benchmarks, return and save results to /opt/data/ai-optimizer/
|
||||||
services.aiWorkerAccess = true;
|
services.aiWorkerAccess = true;
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user