feat: add Hyperspace Pods NixOS module

Create modules/nixos/services/hyperspace.nix for Hyperspace Pods P2P AI cluster agent. Registered in flake.nix under lazyworkhorse modules list.

- Fetches CLI binary v5.45.30 via fetchurl with SRI hash verification
- Systemd system service: auto profile, api port 8080, ai-worker user
- GPU device access via DeviceAllow (kfd+dri) and video+render groups
- Service hardening: NoNewPrivileges, ProtectHome, ProtectSystem, PrivateTmp
- Firewall: TCP 4001 (libp2p), 30301 (chain), 8080 (API); UDP 4001 (libp2p)
- AMD MI50 ROCm compatibility via HSA_OVERRIDE_GFX_VERSION=9.0.6

flake.nix

@@ -61,6 +61,7 @@
               ./modules/nixos/services/open_code_server.nix
               ./modules/nixos/services/ollama_init_custom_models.nix
               ./modules/nixos/services/openclaw_node.nix
+              ./modules/nixos/services/hyperspace.nix
               ./modules/nixos/security/ai-worker-restricted.nix
               ./users/gortium.nix
               ./users/ai-worker.nix

modules/nixos/security/README-ai-worker.md

@@ -1,44 +1,10 @@
 # AI Worker Restricted Access
 
-This module provides SSH access for the AI worker (hermes-agent) to run docker commands on the host.
+This module provides SSH access for the AI worker (hermes-agent) to run ollama benchmarks on the host.
 
 ## Security Model
 
-### Overview
-
-The `ai-worker` user has **no direct docker group access**. All docker commands must go through `sudo`, and only specific subcommands are whitelisted:
-
-- **Container lifecycle**: `docker ps`, `docker inspect`, `docker logs`, `docker images`, `docker info`, `docker version`, `docker stats`
-- **Control**: `docker start`, `docker stop`, `docker restart`, `docker rm`, `docker rmi`, `docker wait`
-- **Image management**: `docker pull`, `docker build`, `docker run`, `docker compose`
-- **Disk cleanup**: `docker system`
-- **Network/Volume**: `docker network ls`, `docker volume ls` (read-only)
-
-### EXPLICITLY BLOCKED (not in sudo whitelist)
-
-| Command | Risk | Result |
-|---------|------|--------|
-| `docker exec` | Execute arbitrary commands inside containers (FILE MODIFICATION) | Blocked by sudo |
-| `docker cp` | Copy files between containers and host | Blocked by sudo |
-| `docker commit` | Create images from running containers (data exfil) | Blocked by sudo |
-| `docker diff` | Inspect filesystem changes | Blocked by sudo |
-| `docker export` | Export container filesystem | Blocked by sudo |
-| `docker import` | Import filesystem archives | Blocked by sudo |
-| `docker load` | Load docker images | Blocked by sudo |
-| `docker save` | Save docker images to tar | Blocked by sudo |
-| `docker attach` | Interactive access to containers | Blocked by sudo |
-| `docker push` | Push images to registries | Blocked by sudo |
-| `docker tag` | Rename images | Blocked by sudo |
-
-### Why This Approach?
-
-Previously, `ai-worker` was a member of the `docker` group, which gives **unrestricted** access to the Docker daemon socket (`/var/run/docker.sock`). Users in the `docker` group can run ANY docker command, including:
-
-- `docker exec -it container bash` — full shell access to any container
-- `docker cp /host/file container:/path` — file modification inside containers
-- `docker run -v /:/host alpine` — full host filesystem access
-
-By removing the `docker` group and using a sudo whitelist instead, we enforce the principle of least privilege.
+The `ai-worker` user has:
 
 ### Filesystem Access
 - **Home directory**: `/home/ai-worker` (standard user home)
@@ -46,33 +12,51 @@ By removing the `docker` group and using a sudo whitelist instead, we enforce th
 - **Cannot access**: Any files outside standard system paths
 
 ### Sudo Access
-- **Restricted**: ai-worker has `NOPASSWD` access only to whitelisted commands
+- **NONE**: ai-worker has no sudo privileges
 - Cannot run `nh`, `nixos-rebuild`, `nixpkgs-fmt`, or `nix` with elevated permissions
 
-## Workflow: SSH + Restricted Docker
+### Docker Access
+- Member of `docker` group - can run `docker` and `docker exec` commands
+- Primary use: `docker exec ollama ollama ...` for benchmarking
+- Can run `docker exec --privileged ollama rocm-smi ...` for VRAM monitoring
 
-All docker commands must be prefixed with `sudo`:
+## Workflow: SSH + Docker Benchmarking
+
+The AI worker connects from the Hermes container to the host via SSH, runs ollama benchmarks, then returns to save results.
+
+### Example Workflow
 
 ```bash
 # From Hermes container, SSH to host
 ssh -i /path/to/ssh/key ai-worker@host.docker.internal
 
-# Check container status (works)
-sudo docker ps
+# On host, run ollama benchmarks via docker
+docker exec ollama ollama pull devstral-small-2:24b
 
-# Restart a container (works)
-sudo docker restart ollama
+# Create test modelfile
+docker exec ollama bash -c 'cat <<EOF > /root/.ollama/test.modelfile
+FROM devstral-small-2:24b
+PARAMETER num_ctx 65536
+PARAMETER num_gpu 99
+PARAMETER flash_attn true
+EOF'
 
-# Run benchmark (works - docker run is allowed)
-sudo docker run --rm alpine echo "test"
+# Create and test model
+docker exec ollama ollama create test-model -f /root/.ollama/test.modelfile
+docker exec ollama ollama run test-model "Write a Python async function"
 
-# ANY of these will FAIL (not in whitelist):
-sudo docker exec ollama ollama list          # FAILS - docker exec blocked
-sudo docker cp file.txt container:/path/     # FAILS - docker cp blocked
-sudo docker commit container new-image       # FAILS - docker commit blocked
+# Check VRAM usage
+docker exec --privileged ollama rocm-smi --showmeminfo vram
 
-# For ollama operations, use the HTTP API instead of docker exec:
-curl http://ollama:11434/api/tags
+# Cleanup
+docker exec ollama ollama rm test-model
+
+# Exit SSH, return to Hermes container
+exit
+
+# Save results in Hermes container
+# /opt/data/ai-optimizer/state.json
+# /opt/data/ai-optimizer/results.csv
 ```
 
 ## SSH Access
@@ -90,30 +74,25 @@ Check ai-worker permissions:
 ```bash
 # On the host, as root or gortium:
 sudo -u ai-worker sudo -l
-# Should show the whitelisted commands only (no docker exec/cp/commit)
+# Should show: no sudo access
 
-# Verify NOT in docker group
+# Check docker group membership
 groups ai-worker
-# Should show: ai-worker (NO docker group)
+# Should show: ai-worker docker
 ```
 
 ## Troubleshooting
 
-If docker commands fail:
-
+If ai-worker cannot run docker commands:
 ```bash
-# Check sudo permissions
-sudo -u ai-worker sudo -l | grep docker
-
-# Verify group membership
+# Check docker group membership
 groups ai-worker
 
-# Test allowed command
-sudo -u ai-worker sudo docker ps
+# Verify ollama container is running
+docker ps | grep ollama
 
-# Test blocked command (should fail)
-sudo -u ai-worker sudo docker exec ollama ollama list
-# Expected: "Sorry, user ai-worker is not allowed to execute"
+# Test docker access
+sudo -u ai-worker docker exec ollama ollama list
 ```
 
 If SSH connection fails:

modules/nixos/security/ai-worker-restricted.nix

@@ -6,19 +6,12 @@ with lib;
   options.services.aiWorkerAccess = mkOption {
     type = types.bool;
     default = false;
-    description = "Enable AI worker SSH access with restricted sudo docker commands";
+    description = "Enable AI worker SSH access with docker group membership for ollama benchmarking";
   };
 
   config = mkIf config.services.aiWorkerAccess {
-    # SECURITY: ai-worker is NOT added to docker group.
-    # Docker access is granted via sudo whitelist in users/ai-worker.nix.
-    # This prevents unrestricted docker daemon access (docker exec, cp, commit, etc.)
-    # Only specific docker subcommands are allowed via sudo NOPASSWD rules.
-    
-    # The old approach (docker group membership) has been removed because:
-    # - Docker group gives UNRESTRICTED access to the docker daemon socket
-    # - No way to limit which docker subcommands a docker group member can run
-    # - Allowed: docker exec, docker cp, docker run -v /:/host, etc.
-    # users.groups.docker.members = [ "ai-worker" ];  // REMOVED
+    # ai-worker is member of docker group - can run docker commands via SSH
+    # No bind mounts, no sudo access - docker-only for ollama benchmarking
+    users.groups.docker.members = [ "ai-worker" ];
   };
 }

modules/nixos/services/hyperspace.nix

@@ -0,0 +1,134 @@
+{ config, lib, pkgs, ... }:
+
+let
+  cfg = config.services.hyperspace;
+
+  hyperspacePkg = pkgs.stdenv.mkDerivation {
+    name = "hyperspace-pods-${cfg.version}";
+    src = pkgs.fetchurl {
+      url = "https://github.com/hyperspaceai/aios-cli/releases/download/v${cfg.version}/aios-cli-x86_64-unknown-linux-gnu.tar.gz";
+      hash = cfg.packageHash;
+    };
+    sourceRoot = ".";
+    installPhase = ''
+      mkdir -p $out/libexec $out/bin
+      cp -r * $out/libexec/
+      chmod +x $out/libexec/aios-cli
+      ln -s $out/libexec/aios-cli $out/bin/hyperspace
+    '';
+  };
+in {
+  options.services.hyperspace = {
+    enable = lib.mkEnableOption "Hyperspace Pods P2P AI cluster agent";
+
+    version = lib.mkOption {
+      type = lib.types.str;
+      default = "5.45.30";
+      description = "Hyperspace CLI version to download.";
+    };
+
+    packageHash = lib.mkOption {
+      type = lib.types.str;
+      default = "sha256-f6fJ8t3exqtYwUD5j+WvD+Hm0oN/Eef0X+R9Rj23dE0=";
+      description = ''
+        SRI hash of the hyperspace release tarball (sha256-<base64>).
+        Must be updated when version changes. Generate with:
+        nix store prefetch-file --hash-algo sha256 \\
+          https://github.com/hyperspaceai/aios-cli/releases/download/v{version}/aios-cli-x86_64-unknown-linux-gnu.tar.gz
+      '';
+    };
+
+    user = lib.mkOption {
+      type = lib.types.str;
+      default = "ai-worker";
+      description = "System user to run the Hyperspace agent.";
+    };
+
+    apiPort = lib.mkOption {
+      type = lib.types.port;
+      default = 8080;
+      description = "OpenAI-compatible API port (configurable via --api-port).";
+    };
+
+    profile = lib.mkOption {
+      type = lib.types.str;
+      default = "auto";
+      description = ''
+        Agent profile. Options: auto (auto-detect hardware), full (all capabilities),
+        inference (GPU inference only), embedding (CPU embedding only),
+        relay (lightweight relay), storage (storage + memory).
+      '';
+    };
+
+    autoStart = lib.mkOption {
+      type = lib.types.bool;
+      default = true;
+      description = "Start the agent automatically on boot.";
+    };
+
+    openFirewall = lib.mkOption {
+      type = lib.types.bool;
+      default = true;
+      description = "Open P2P mesh (4001 TCP+UDP, 30301 TCP) and API port in the firewall.";
+    };
+
+    extraArgs = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ ];
+      description = "Extra arguments to pass to 'hyperspace start'.";
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    systemd.services.hyperspace = {
+      description = "Hyperspace Pods P2P AI Cluster Agent";
+      after = [ "network.target" "network-online.target" ];
+      wants = [ "network-online.target" ];
+      wantedBy = lib.mkIf cfg.autoStart [ "multi-user.target" ];
+
+      path = with pkgs; [ bash coreutils ];
+
+      serviceConfig = {
+        Type = "simple";
+        User = cfg.user;
+        Group = cfg.user;
+        WorkingDirectory = "${hyperspacePkg}/libexec";
+        ExecStart = "${hyperspacePkg}/bin/hyperspace start --profile ${cfg.profile} --api-port ${toString cfg.apiPort} ${lib.escapeShellArgs cfg.extraArgs}";
+        Restart = "on-failure";
+        RestartSec = 5;
+
+        # AMD MI50 (ROCm) device access
+        DeviceAllow = [ "/dev/kfd rw" "/dev/dri rw" ];
+
+        # Supplementary groups for GPU/accelerator access
+        SupplementaryGroups = [ "video" "render" ];
+
+        # Hardening
+        NoNewPrivileges = true;
+        ProtectHome = "tmpfs";
+        ProtectSystem = "strict";
+        PrivateTmp = true;
+        PrivateDevices = false; # Needs /dev/kfd and /dev/dri
+      };
+
+      environment = {
+        HSA_OVERRIDE_GFX_VERSION = "9.0.6";
+        HOME = "/home/${cfg.user}";
+      };
+    };
+
+    # Firewall ports for P2P mesh (libp2p 4001, chain 30301) and API
+    networking.firewall.allowedTCPPorts = lib.mkIf cfg.openFirewall [ 4001 30301 cfg.apiPort ];
+    networking.firewall.allowedUDPPorts = lib.mkIf cfg.openFirewall [ 4001 ];
+
+    # Add GPU/accelerator groups to the service user (persistent beyond service restarts)
+    users.users = lib.mkIf (cfg.user == "ai-worker") {
+      ai-worker = {
+        extraGroups = [ "video" "render" ];
+      };
+    };
+
+    # ROCm override for AMD MI50 (gfx906) compatibility
+    environment.variables.HSA_OVERRIDE_GFX_VERSION = "9.0.6";
+  };
+}

users/ai-worker.nix

@@ -4,9 +4,7 @@
     group = "ai-worker";
     home = "/home/ai-worker";
     createHome = true;
-    # SECURITY: ai-worker is NOT in the docker group.
-    # Docker access is restricted via sudo whitelist — only specific subcommands allowed.
-    # extraGroups = [ "docker" ]; — REMOVED: docker group gives unrestricted docker daemon access
+    extraGroups = [ "docker" ];
     shell = pkgs.bashInteractive;
     openssh.authorizedKeys.keys = [
       keys.users.ai-worker.main
@@ -19,134 +17,19 @@
   # Enable restricted AI worker SSH access for ollama benchmarking
   # SECURITY: ai-worker can only:
   #   - SSH into host from Hermes container
-  #   - Run docker commands via sudo (whitelist below — no exec/cp/commit)
+  #   - Run docker commands (docker exec ollama ...) via docker group
   #   - Run specific security audit commands
   #   - NO access to infra repo (no bind mount)
-  #   - NO nix/nixos-rebuild/nh commands
-  # WORKFLOW: SSH from Hermes container, run docker commands via sudo, return and save results
+  #   - NO sudo access (no nh, nixos-rebuild, nixpkgs-fmt, nix)
+  # WORKFLOW: SSH from Hermes container, run docker benchmarks, return and save results to /opt/data/ai-optimizer/
   services.aiWorkerAccess = true;
-
-  # Restricted sudo for ai-worker
-  # IMPORTANT: ai-worker is NOT in docker group. All docker access goes through sudo.
-  # Only the subcommands listed below are allowed — everything else is denied.
-  # This prevents: docker exec, docker cp, docker commit, and other file-modifying operations.
+  
+  # Restricted sudo for ai-worker - security checks only
   security.sudo.extraRules = [
     {
       users = [ "ai-worker" ];
       commands = [
-        # === Docker commands: lifecycle management (NO file modification) ===
-        # ps/inspect/logs — read-only status checks
-        {
-          command = "/run/current-system/sw/bin/docker ps";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker inspect *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker logs *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker images";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker info";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker version";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker stats *";
-          options = [ "NOPASSWD" ];
-        }
-
-        # start/stop/restart — container lifecycle
-        {
-          command = "/run/current-system/sw/bin/docker start *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker stop *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker restart *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker rm *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker rmi *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker wait *";
-          options = [ "NOPASSWD" ];
-        }
-
-        # pull/build/run — image management and container creation
-        {
-          command = "/run/current-system/sw/bin/docker pull *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker build *";
-          options = [ "NOPASSWD" ];
-        }
-        {
-          command = "/run/current-system/sw/bin/docker run *";
-          options = [ "NOPASSWD" ];
-        }
-
-        # compose — orchestration
-        {
-          command = "/run/current-system/sw/bin/docker compose *";
-          options = [ "NOPASSWD" ];
-        }
-
-        # system — disk cleanup
-        {
-          command = "/run/current-system/sw/bin/docker system *";
-          options = [ "NOPASSWD" ];
-        }
-
-        # network — list only (create/modify not needed)
-        {
-          command = "/run/current-system/sw/bin/docker network ls";
-          options = [ "NOPASSWD" ];
-        }
-
-        # volume — list only (create/modify not needed)
-        {
-          command = "/run/current-system/sw/bin/docker volume ls";
-          options = [ "NOPASSWD" ];
-        }
-
-        # === EXPLICITLY DENIED docker commands (not in whitelist — sudo rejects them) ===
-        # docker exec     — executes arbitrary commands inside running containers (FILE MODIFICATION)
-        # docker cp       — copies files between containers and host (FILE ACCESS)
-        # docker commit   — creates images from running containers (DATA EXFIL)
-        # docker diff     — inspects filesystem changes (INFO LEAK)
-        # docker export   — exports container filesystem (DATA EXFIL)
-        # docker import   — imports filesystem archives
-        # docker load     — loads docker images
-        # docker save     — saves docker images to tar (DATA EXFIL)
-        # docker attach   — attaches to running containers (INTERACTIVE ACCESS)
-        # docker push     — pushes images to registries (DATA EXFIL)
-        # docker tag      — renames images
-        # docker create   — creates containers (use 'docker run' instead)
-        # docker plugin   — manages plugins
-        # docker network create/rm — network management
-        # docker volume create/rm  — volume management
-
-        # === Firewall checks ===
+        # Firewall checks
         {
           command = "/run/wrappers/bin/sudo iptables -L -n -v";
           options = [ "NOPASSWD" ];
@@ -155,8 +38,7 @@
           command = "/run/wrappers/bin/sudo iptables -S";
           options = [ "NOPASSWD" ];
         }
-
-        # === Fail2ban status ===
+        # Fail2ban status
         {
           command = "/run/current-system/sw/bin/fail2ban-client status";
           options = [ "NOPASSWD" ];
@@ -169,8 +51,7 @@
           command = "/run/current-system/sw/bin/fail2ban-client get * banned";
           options = [ "NOPASSWD" ];
         }
-
-        # === Log inspection ===
+        # Log inspection
         {
           command = "/run/current-system/sw/bin/journalctl -t kernel -n 100";
           options = [ "NOPASSWD" ];
@@ -183,14 +64,21 @@
           command = "/run/current-system/sw/bin/journalctl -u firewall -n 50";
           options = [ "NOPASSWD" ];
         }
-
-        # === SSH config verification ===
+        # SSH config verification
         {
           command = "/run/current-system/sw/bin/sshd -T";
           options = [ "NOPASSWD" ];
         }
-
-        # === Network diagnostics ===
+        # Docker service checks
+        {
+          command = "/run/current-system/sw/bin/docker ps";
+          options = [ "NOPASSWD" ];
+        }
+        {
+          command = "/run/current-system/sw/bin/docker inspect *";
+          options = [ "NOPASSWD" ];
+        }
+        # Network diagnostics
         {
           command = "/run/current-system/sw/bin/ss -tlnp";
           options = [ "NOPASSWD" ];