feat: add Hyperspace Pods NixOS module and enable on lazyworkhorse

Hyperspace Pods let multiple machines pool their GPUs into one private P2P mesh AI cluster. Models are split across all connected GPUs — e.g. two machines with 16GB VRAM each can run Qwen 3.5 32B together. Changes: - Add modules/nixos/services/hyperspace.nix — NixOS module that: * Fetches the Hyperspace CLI binary (v5.45.30) via fetchurl * Sets up systemd service for the agent * Opens firewall ports (libp2p 4001, chain 30301, API 8080) * Configures GPU passthrough for AMD MI50 (ROCm) - Register module in flake.nix for lazyworkhorse - Enable hyperspace service on lazyworkhorse (ai-worker user, port 8080) Usage after deployment: hyperspace pod create "tdnde-lab" # create pod hyperspace pod invite # share invite with cyt-pi curl http://localhost:8080/v1/chat/completions # OpenAI API See skill: nixos-hyperspace-pods
chore: update compose submodule to traefik logging branch
2026-05-02 15:36:15 +00:00 · 2026-05-02 15:30:28 +00:00 · 2026-05-02 15:30:28 +00:00 · 2026-04-30 21:54:47 -04:00 · 2026-04-27 06:11:34 -04:00 · 2026-04-27 06:02:25 -04:00
17 changed files with 734 additions and 45 deletions
--- a/.planning/ROADMAP.md
+++ b/.planning/ROADMAP.md
@@ -13,7 +13,9 @@ None
 - ✅ **Phase 1: Foundation Setup** - Establish core NixOS configuration with flakes
 - ✅ **Phase 2: Docker Service Integration** - Integrate Docker Compose services
 - ✅ **Phase 3: AI Assistant Integration** - Enable AI-assisted infrastructure management
- [ ] **Phase 4: Internet Access & MCP** - MCP server for web access
+- ✅ **Phase 4: Internet Access & MCP** - MCP server for web access
+- 🚨 **Security Hardening** - CRITICAL: Firewall, fail2ban, SSH hardening (PR #28)
+- [ ] **Phase 5: TAK Server** - Research, implementation, and validation


 ## Phase Details
@@ -133,8 +135,25 @@ Plans:

 ## Progress

+**Merge Priority Order** (CRITICAL - merge in this order):
+
+| Priority | PR | Description | Status | Notes |
+|----------|-----|-------------|--------|-------|
+| 🚨 1 | #28 | **Security hardening** (firewall, fail2ban, SSH) | Open | **MERGE FIRST** - protects all other services |
+| 2 | #22 | Matrix bridge dependency fix | Open | Blocks Hermes functionality |
+| 3 | #21 | Backup network creation fix | Open | Infrastructure fix |
+| 4 | #25 | Hermes voice GPU support | Open | Feature enhancement |
+| 5 | #24 | uConsole CM5 host | Open | New hardware support |
+| 6 | #23 | NixOS deployment infrastructure | Open | Deployment tooling |
+| 7 | #1 | AI worker restricted access | Open | Legacy PR (superseded by hardening) |
+
 **Execution Order:**
-Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7
+Phases execute in numeric order: 1 → 2 → 3 → 4 → Security → 5 → 6 → 7
+
+**Merge vs Phase Execution:**
+- PRs can merge independently (no strict phase ordering for merges)
+- **EXCEPTION:** Security hardening (#28) must merge before any new services are exposed
+- After security merge, deploy with: `nh os switch --flake .#lazyworkhorse`

 | Phase | Milestone | Plans Complete | Status | Completed |
 |-------|-----------|----------------|--------|-----------|
--- a/assets/compose
+++ b/assets/compose
--- a/flake.lock
+++ b/flake.lock
@@ -23,6 +23,20 @@
        "type": "github"
      }
    },
+    "flake-compat": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1751685974,
+        "narHash": "sha256-NKw96t+BgHIYzHUjkTK95FqYRVKB8DHpVhefWSz/kTw=",
+        "rev": "549f2762aebeff29a2e5ece7a7dc0f955281a1d1",
+        "type": "tarball",
+        "url": "https://git.lix.systems/api/v1/repos/lix-project/flake-compat/archive/549f2762aebeff29a2e5ece7a7dc0f955281a1d1.tar.gz"
+      },
+      "original": {
+        "type": "tarball",
+        "url": "https://git.lix.systems/lix-project/flake-compat/archive/main.tar.gz"
+      }
+    },
    "home-manager": {
      "inputs": {
        "nixpkgs": [
@@ -44,7 +58,125 @@
        "type": "github"
      }
    },
+    "lix": {
+      "inputs": {
+        "flake-compat": "flake-compat",
+        "nix2container": "nix2container",
+        "nix_2_18": "nix_2_18",
+        "nixpkgs": [
+          "nixpkgs"
+        ],
+        "nixpkgs-regression": "nixpkgs-regression",
+        "pre-commit-hooks": "pre-commit-hooks"
+      },
+      "locked": {
+        "lastModified": 1774721317,
+        "narHash": "sha256-KS0ElyhZKdUFcfaxfwid3yi2Id3EP9i+dGL16/wx1T8=",
+        "ref": "main",
+        "rev": "d0190cff6f2314cc1c727ff113aea20e086f4bcc",
+        "revCount": 19103,
+        "type": "git",
+        "url": "https://git.lix.systems/lix-project/lix"
+      },
+      "original": {
+        "ref": "main",
+        "type": "git",
+        "url": "https://git.lix.systems/lix-project/lix"
+      }
+    },
+    "lowdown-src": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1633514407,
+        "narHash": "sha256-Dw32tiMjdK9t3ETl5fzGrutQTzh2rufgZV4A/BbxuD4=",
+        "owner": "kristapsdz",
+        "repo": "lowdown",
+        "rev": "d2c2b44ff6c27b936ec27358a2653caaef8f73b8",
+        "type": "github"
+      },
+      "original": {
+        "owner": "kristapsdz",
+        "repo": "lowdown",
+        "type": "github"
+      }
+    },
+    "nix2container": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1767195068,
+        "narHash": "sha256-+OMnL79ZjqM/PCz2hoQ12MnXNoSSfBGnsYBOZnA9XbI=",
+        "owner": "nlewo",
+        "repo": "nix2container",
+        "rev": "bb6801be998ba857a62c002cb77ece66b0a57298",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nlewo",
+        "repo": "nix2container",
+        "type": "github"
+      }
+    },
+    "nix_2_18": {
+      "inputs": {
+        "flake-compat": [
+          "lix",
+          "flake-compat"
+        ],
+        "lowdown-src": "lowdown-src",
+        "nixpkgs": "nixpkgs",
+        "nixpkgs-regression": [
+          "lix",
+          "nixpkgs-regression"
+        ]
+      },
+      "locked": {
+        "lastModified": 1730375271,
+        "narHash": "sha256-RrOFlDGmRXcVRV2p2HqHGqvzGNyWoD0Dado/BNlJ1SI=",
+        "owner": "NixOS",
+        "repo": "nix",
+        "rev": "0f665ff6779454f2117dcc32e44380cda7f45523",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "2.18.9",
+        "repo": "nix",
+        "type": "github"
+      }
+    },
    "nixpkgs": {
+      "locked": {
+        "lastModified": 1705033721,
+        "narHash": "sha256-K5eJHmL1/kev6WuqyqqbS1cdNnSidIZ3jeqJ7GbrYnQ=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "a1982c92d8980a0114372973cbdfe0a307f1bdea",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixos-23.05-small",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-regression": {
+      "locked": {
+        "lastModified": 1643052045,
+        "narHash": "sha256-uGJ0VXIhWKGXxkeNnq4TvV3CIOkUJ3PAoLZ3HMzNVMw=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "215d4d0fd80ca5163643b03a33fde804a29cc1e2",
+        "type": "github"
+      }
+    },
+    "nixpkgs_2": {
      "locked": {
        "lastModified": 1774386573,
        "narHash": "sha256-4hAV26quOxdC6iyG7kYaZcM3VOskcPUrdCQd/nx8obc=",
@@ -60,10 +192,27 @@
        "type": "github"
      }
    },
+    "pre-commit-hooks": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1769939035,
+        "narHash": "sha256-Fok2AmefgVA0+eprw2NDwqKkPGEI5wvR+twiZagBvrg=",
+        "owner": "cachix",
+        "repo": "git-hooks.nix",
+        "rev": "a8ca480175326551d6c4121498316261cbb5b260",
+        "type": "github"
+      },
+      "original": {
+        "owner": "cachix",
+        "repo": "git-hooks.nix",
+        "type": "github"
+      }
+    },
    "root": {
      "inputs": {
        "agenix": "agenix",
-        "nixpkgs": "nixpkgs"
+        "lix": "lix",
+        "nixpkgs": "nixpkgs_2"
      }
    },
    "systems": {
--- a/flake.nix
+++ b/flake.nix
@@ -8,10 +8,14 @@
      inputs.darwin.follows = "";
      inputs.nixpkgs.follows = "nixpkgs";
    };
+    lix = {
+      url = "git+https://git.lix.systems/lix-project/lix?ref=main";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
    self.submodules = true;
  };

-  outputs = { self, nixpkgs, agenix, ... }@inputs:
+  outputs = { self, nixpkgs, agenix, lix, ... }@inputs:
    let
      system = "x86_64-linux";
      keys = import ./lib/keys.nix;
@@ -26,6 +30,9 @@
      pkgs = import nixpkgs {
        inherit system overlays;
        config.allowUnfree = true;
+        config.permittedInsecurePackages = [
+          "openclaw-2026.3.12"
+        ];
      };

      devShell = import ./shells/nix_dev.nix {
@@ -35,12 +42,16 @@
      {
        nixosConfigurations = {
          lazyworkhorse = nixpkgs.lib.nixosSystem {
-            specialArgs = { inherit system self keys paths; };
+            specialArgs = { inherit system self keys paths inputs; };
            modules = [
              {
                nixpkgs.overlays = overlays;
                nixpkgs.config.allowUnfree = true;
                nixpkgs.config.rocmSupport = true;
+                nixpkgs.config.permittedInsecurePackages = [
+                  "openclaw-2026.3.12"
+                ];
+                nix.package = lix.packages.${system}.default;
              }
              agenix.nixosModules.default
              ./hosts/lazyworkhorse/configuration.nix
@@ -49,7 +60,24 @@
              ./modules/nixos/services/docker_manager.nix
              ./modules/nixos/services/open_code_server.nix
              ./modules/nixos/services/ollama_init_custom_models.nix
+              ./modules/nixos/services/openclaw_node.nix
+              ./modules/nixos/services/hyperspace.nix
              ./users/gortium.nix
+              ./users/ai-worker.nix
+            ];
+          };
+
+          cyt-pi = nixpkgs.lib.nixosSystem {
+            specialArgs = { inherit self keys paths inputs; };
+            modules = [
+              {
+                nixpkgs.overlays = overlays;
+                nixpkgs.config.allowUnfree = true;
+                nixpkgs.hostPlatform = "aarch64-linux";
+                nix.package = lix.packages."aarch64-linux".default;
+              }
+              ./hosts/cyt-pi/configuration.nix
+              ./hosts/cyt-pi/hardware-configuration.nix
            ];
          };
        };
--- a/hosts/cyt-pi/configuration.nix
+++ b/hosts/cyt-pi/configuration.nix
@@ -0,0 +1,98 @@
+{ config, lib, pkgs, paths, self, ... }:
+
+{
+  # Basic Host Info
+  networking.hostName = "cyt-pi";
+  time.timeZone = "America/Montreal";
+  i18n.defaultLocale = "en_CA.UTF-8";
+
+  # System State
+  system.stateVersion = "25.05";
+
+  # Boot & Hardware (Pi Zero 2 W is ARM64)
+  boot.loader.grub.enable = false;
+  boot.loader.generic-extlinux-compatible.enable = true;
+  boot.kernelPackages = pkgs.linuxPackages_latest;
+
+  # Networking
+  networking.networkmanager.enable = true;
+  services.openssh = {
+    enable = true;
+    settings.PermitRootLogin = "prohibit-password";
+  };
+
+  # User
+  users.users.gortium = {
+    isNormalUser = true;
+    extraGroups = [ "wheel" "networkmanager" "kismet" ];
+    openssh.authorizedKeys.keys = [
+      # Populate with your public key
+    ];
+  };
+
+  # CYT Project Dependencies (Headless)
+  environment.systemPackages = with pkgs; [
+    git
+    python311
+    python311Packages.opencv4
+    python311Packages.numpy
+    python311Packages.pillow
+    autossh # For the reverse tunnel
+    kismet # Wi-Fi monitoring
+  ];
+
+  # Kismet Service
+  systemd.services.kismet = {
+    description = "Kismet Wi-Fi Monitor";
+    after = [ "network-online.target" ];
+    wantedBy = [ "multi-user.target" ];
+    serviceConfig = {
+      User = "gortium";
+      Group = "kismet";
+      ExecStart = ''
+        ${pkgs.kismet}/bin/kismet -c panda --log-base=/home/gortium/kismet_logs --no-nc-ui
+      '';
+      Restart = "always";
+      RestartSec = "10s";
+    };
+  };
+
+  # Reverse SSH Tunnel Service
+  systemd.services.cyt-tunnel = {
+    description = "Reverse SSH Tunnel to lazyworkhorse.net";
+    after = [ "network-online.target" ];
+    wantedBy = [ "multi-user.target" ];
+    serviceConfig = {
+      User = "gortium";
+      ExecStart = ''
+        ${pkgs.autossh}/bin/autossh -M 0 -N \
+          -o "ServerAliveInterval 30" \
+          -o "ServerAliveCountMax 3" \
+          -R 19999:localhost:22 \
+          gortium@lazyworkhorse.net -p 2425 \
+          -i /home/gortium/.ssh/cyt_tunnel_key
+      '';
+      Restart = "always";
+      RestartSec = "10s";
+    };
+  };
+
+  # CYT Application Service
+  systemd.services.cyt-app = {
+    description = "Chasing Your Tail - Target Detector";
+    after = [ "network-online.target" "kismet.service" ];
+    wantedBy = [ "multi-user.target" ];
+    serviceConfig = {
+      User = "gortium";
+      WorkingDirectory = "/home/gortium/Chasing-Your-Tail-NG";
+      ExecStart = ''
+        ${pkgs.python311}/bin/python3 target_detector_cli.py --min-ssids 2
+      '';
+      Restart = "on-failure";
+      RestartSec = "60s";
+      Environment = [
+        "CYT_KISMET_LOGS=/home/gortium/kismet_logs"
+      ];
+    };
+  };
+}
--- a/hosts/cyt-pi/hardware-configuration.nix
+++ b/hosts/cyt-pi/hardware-configuration.nix
@@ -0,0 +1,24 @@
+{ config, lib, pkgs, modulesPath, ... }:
+
+{
+  imports =
+    [ (modulesPath + "/installer/scan/not-detected.nix")
+    ];
+
+  boot.initrd.availableKernelModules = [ "xhci_pci" "usbhid" "sdhci_pci" ];
+  boot.initrd.kernelModules = [ ];
+  boot.kernelModules = [ ];
+  boot.extraModulePackages = [ ];
+
+  # Pi Zero 2 W specific filesystem
+  fileSystems."/" =
+    { device = "/dev/disk/by-label/NIXOS_SD";
+      fsType = "ext4";
+      options = [ "noatime" ];
+    };
+
+  swapDevices = [ ];
+
+  nixpkgs.hostPlatform = lib.mkDefault "aarch64-linux";
+  hardware.enableRedistributableFirmware = true;
+}
--- a/hosts/lazyworkhorse/configuration.nix
+++ b/hosts/lazyworkhorse/configuration.nix
@@ -9,7 +9,7 @@
  hoardingcow-mount.enable = true;

  # Flakesss
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [ "nix-command" "flakes" "flake-self-attrs" ];
  nix.settings.trusted-users = [ "root" "gortium" ];

  # Garbage collection
@@ -135,6 +135,7 @@
    kitty.terminfo
    nodejs_22
    uv
+    openclaw
    (python3.withPackages (ps: with ps; [
      openai-whisper
    ]))
@@ -209,6 +210,7 @@

    coms = {
      path = self + "/assets/compose/coms";
+      envFile = config.age.secrets.containers_env.path;
    };

    finance = {
@@ -248,16 +250,43 @@
        mode = "0600";
        path = "/etc/ssh/ssh_host_ed25519_key";
      };
-      # n8n_ssh_key = {
-      #   file = ../../secrets/n8n_ssh_key.age;
-      #   owner = "root";
-      #   group = "root";
-      #   mode = "0600";
-      #   path = "/home/n8n-worker/.ssh/n8n_ssh_key";
-      # };
+      ai_ssh_key = {
+        file = ../../secrets/ai_ssh_key.age;
+        owner = "root";
+        group = "root";
+        mode = "0600";
+        path = "/home/ai-worker/.ssh/ai_ssh_key";
+      };
+      openclaw_gateway_token = {
+        file = ../../secrets/openclaw_gateway_token.age;
+        owner = "root";
+        group = "ai-worker";
+        mode = "0440";
+        path = "/run/secrets/openclaw_gateway_token";
+      };
    };
  };

+  # OpenClaw Node service (host-side execution for Docker gateway)
+  services.openclaw-node = {
+    enable = true;
+    user = "ai-worker";
+    gatewayHost = "127.0.0.1";
+    gatewayPort = 18789;
+    gatewayTokenFile = "/run/secrets/openclaw_gateway_token";
+    displayName = "lazyworkhorse-host";
+  };
+
+  # Hyperspace Pods — P2P mesh AI cluster (combine GPUs across machines)
+  services.hyperspace = {
+    enable = true;
+    user = "ai-worker";
+    apiPort = 8080;
+    profile = "auto";
+    openFirewall = true;
+    extraArgs = [ "--verbose" ];
+  };
+
  # Public host ssh key (kept in sync with the private one)
  environment.etc."ssh/ssh_host_ed25519_key.pub".text =
    "${keys.hosts.lazyworkhorse.main}";
--- a/lib/keys.nix
+++ b/lib/keys.nix
@@ -6,7 +6,7 @@
      gitea = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN9tKezYidZglWBRI9/2I/cBGUUHj2dHY8rHXppYmf7F";
    };
    
-    n8n-worker = {
+    ai-worker = {
      main = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAXeGtPPcsP2IYRQNvII41NVWhJsarEk8c4qxs/a5sXf";
    };
  };
--- a/modules/nixos/services/docker_manager.nix
+++ b/modules/nixos/services/docker_manager.nix
@@ -29,6 +29,11 @@ with lib;
    systemd.services = mapAttrs' (name: value: nameValuePair "${name}_stack" {
      description = "Docker Compose stack: ${name}";
      
+      # Forces systemd to restart when the files change
+      reloadTriggers = [
+        "${builtins.hashFile "sha256" (toString value.path + "/compose.yml")}"
+      ] ++ (lib.optional (value.envFile != null) "${value.envFile}");
+
      after = [ "network.target" "docker.service" "docker.socket" "agenix.service" ];
      wants = [ "docker.socket" "agenix.service" ];
      requires = [ "docker.service" ];
--- a/modules/nixos/services/hyperspace.nix
+++ b/modules/nixos/services/hyperspace.nix
@@ -0,0 +1,235 @@
+{ config, lib, pkgs, ... }:
+
+with lib;
+
+let
+  cfg = config.services.hyperspace;
+
+  # Hyperspace CLI release from github.com/hyperspaceai/aios-cli
+  # The binary bundles Node.js runtime + llama.cpp + sidecars (~914MB)
+  # It auto-updates via `hyperspace update` post-install
+  hyperspacePkg = pkgs.stdenv.mkDerivation rec {
+    pname = "hyperspace";
+    version = cfg.release;
+
+    src = pkgs.fetchurl {
+      url = "https://github.com/hyperspaceai/aios-cli/releases/download/v${version}/aios-cli-x86_64-unknown-linux-gnu.tar.gz";
+      hash = "sha256-f6fJ8t3exqtYwUD5j+WvD+Hm0oN/Eef0X+R9Rj23dE0=";
+    };
+
+    sourceRoot = ".";
+
+    installPhase = ''
+      mkdir -p $out/bin $out/lib/hyperspace
+
+      # Main CLI binary
+      cp aios-cli $out/bin/hyperspace
+      chmod +x $out/bin/hyperspace
+
+      # Sidecar binaries
+      for f in _aios-cli pod-raft hyperspace-*; do
+        [ -f "$f" ] && install -m755 "$f" $out/lib/hyperspace/ || true
+      done
+
+      # WASM, native modules, Python shards
+      cp -r *.wasm $out/lib/hyperspace/ 2>/dev/null || true
+      cp -r *.node $out/lib/hyperspace/ 2>/dev/null || true
+      mkdir -p $out/lib/hyperspace/python
+      cp -r python/* $out/lib/hyperspace/python/ 2>/dev/null || true
+
+      # Skills directory
+      mkdir -p $out/share/hyperspace
+      cp -r skills $out/share/hyperspace/ 2>/dev/null || true
+
+      # Set HYPERSPACE_PATH so the binary finds sidecars
+      wrapProgram $out/bin/hyperspace \
+        --set HYPERSPACE_PATH "$out/lib/hyperspace" \
+        --set HYPERSPACE_SKILLS_DIR "$out/share/hyperspace/skills"
+    '';
+
+    nativeBuildInputs = with pkgs; [ makeWrapper ];
+
+    meta = {
+      description = "Hyperspace CLI — P2P mesh AI inference network (Pods)";
+      longDescription = ''
+        Hyperspace Pods let multiple machines pool their GPUs into one private
+        AI cluster. Install the CLI, create a pod, share an invite link — your
+        machines form a P2P mesh and can run models split across all connected
+        GPUs. Exposes an OpenAI-compatible API for use with Cursor, Claude Code,
+        Aider, etc.
+      '';
+      homepage = "https://hyperspace.sh";
+      sourceProvenance = with lib; [ sourceTypes.binaryNativeCode ];
+      license = lib.licenses.unfree;
+      platforms = [ "x86_64-linux" ];
+      maintainers = [ ];
+    };
+  };
+
+in {
+  options.services.hyperspace = {
+    enable = mkEnableOption "Hyperspace P2P AI agent (Pods)";
+
+    release = mkOption {
+      type = types.str;
+      default = "5.45.30";
+      description = "Hyperspace CLI release version (from GitHub releases).";
+    };
+
+    user = mkOption {
+      type = types.str;
+      default = "ai-worker";
+      description = "System user to run the Hyperspace agent.";
+    };
+
+    apiPort = mkOption {
+      type = types.port;
+      default = 8080;
+      description = "Port for the OpenAI-compatible API server.";
+    };
+
+    autoStart = mkOption {
+      type = types.bool;
+      default = true;
+      description = "Auto-start the Hyperspace agent on boot.";
+    };
+
+    openFirewall = mkOption {
+      type = types.bool;
+      default = true;
+      description = "Open firewall ports for P2P traffic (libp2p 4001, chain 30301, API).";
+    };
+
+    profile = mkOption {
+      type = types.enum [ "auto" "full" "inference" "embedding" "relay" "storage" ];
+      default = "auto";
+      description = ''
+        Agent profile:
+        - auto: auto-detect hardware
+        - full: all 9 capabilities
+        - inference: GPU inference only
+        - embedding: CPU embedding only
+        - relay: lightweight relay
+        - storage: storage + memory
+      '';
+    };
+
+    extraArgs = mkOption {
+      type = types.listOf types.str;
+      default = [ ];
+      description = "Extra arguments passed to `hyperspace start`.";
+    };
+
+    dataDir = mkOption {
+      type = types.str;
+      default = "/var/lib/hyperspace";
+      description = "Data directory for agent state (models, config, logs).";
+    };
+  };
+
+  config = mkIf cfg.enable {
+    # Ensure the service user exists
+    users.users.${cfg.user} = {
+      isSystemUser = true;
+      group = cfg.user;
+      home = "/home/${cfg.user}";
+      createHome = true;
+      shell = pkgs.bash;
+    };
+    users.groups.${cfg.user} = { };
+
+    # Install the hyperspace binary
+    environment.systemPackages = [ hyperspacePkg ];
+
+    # Data directories
+    systemd.tmpfiles.rules = [
+      "d ${cfg.dataDir} 0755 ${cfg.user} ${cfg.user} -"
+      "d ${cfg.dataDir}/models 0755 ${cfg.user} ${cfg.user} -"
+      "d ${cfg.dataDir}/data 0755 ${cfg.user} ${cfg.user} -"
+    ];
+
+    # Systemd service: runs the Hyperspace agent as a system daemon
+    systemd.services.hyperspace = {
+      description = "Hyperspace P2P AI Agent — Pods mesh cluster";
+      documentation = [ "https://hyperspace.sh" "https://github.com/hyperspaceai/aios-cli" ];
+      after = [ "network-online.target" ];
+      wants = [ "network-online.target" ];
+      wantedBy = mkIf cfg.autoStart [ "multi-user.target" ];
+
+      environment = {
+        HYPERSPACE_HOME = cfg.dataDir;
+        HYPERSPACE_API_PORT = toString cfg.apiPort;
+        HYPERSPACE_PATH = "${hyperspacePkg}/lib/hyperspace";
+      };
+
+      path = with pkgs; [ bash curl nodejs ];
+
+      script = ''
+        # Wait for network connectivity before starting
+        ${pkgs.bash}/bin/bash -c '
+          for i in $(seq 1 30); do
+            ping -c 1 -W 1 8.8.8.8 >/dev/null 2>&1 && break
+            sleep 2
+          done
+        ' || true
+
+        exec ${hyperspacePkg}/bin/hyperspace start \
+          --profile ${cfg.profile} \
+          --api-port ${toString cfg.apiPort} \
+          ${lib.escapeShellArgs cfg.extraArgs}
+      '';
+
+      serviceConfig = {
+        Type = "exec";
+        User = cfg.user;
+        Group = cfg.user;
+        WorkingDirectory = cfg.dataDir;
+        Restart = "always";
+        RestartSec = 10;
+        TimeoutStartSec = 180;
+        TimeoutStopSec = 30;
+        KillMode = "mixed";
+
+        # File limits for network-heavy P2P agent
+        LimitNOFILE = 65536;
+        LimitNPROC = 4096;
+
+        # GPU access — AMD MI50 (ROCm) through /dev/kfd and /dev/dri
+        DeviceAllow = [
+          "/dev/kfd" "rw"
+          "/dev/dri" "rw"
+        ];
+        SupplementaryGroups = [ "video" "render" ];
+
+        # Security hardening
+        NoNewPrivileges = true;
+        ProtectSystem = "strict";
+        ProtectHome = true;
+        PrivateTmp = true;
+        PrivateDevices = false;  # needs GPU access
+        ReadWritePaths = [
+          cfg.dataDir
+          "/tmp"
+        ];
+        BindPaths = [
+          # GPU devices for AMD MI50
+          "/dev/kfd"
+          "/dev/dri"
+        ];
+      };
+    };
+
+    # Firewall: open P2P ports for the mesh network
+    networking.firewall = mkIf cfg.openFirewall {
+      allowedTCPPorts = [
+        4001    # libp2p P2P (agent gossip, DHT, circuits)
+        30301   # Chain P2P (blockchain consensus)
+        cfg.apiPort  # OpenAI-compatible API
+      ];
+      allowedUDPPorts = [
+        4001    # libp2p QUIC transport
+        30301   # Chain UDP discovery
+      ];
+    };
+  };
+}
--- a/modules/nixos/services/ollama_init_custom_models.nix
+++ b/modules/nixos/services/ollama_init_custom_models.nix
@@ -14,8 +14,25 @@
        local base_model=$2
        if ! ${pkgs.docker}/bin/docker exec ollama ollama list | grep -q "$model_name"; then
          echo "$model_name not found, creating from $base_model..."
+          
+          # We use a custom TEMPLATE block to strip the 'currentDate' function 
+          # which is unsupported in Ollama 0.5.7 but present in Devstral's default manifest.
          ${pkgs.docker}/bin/docker exec ollama sh -c "cat <<EOF > /root/.ollama/$model_name.modelfile
 FROM $base_model
+TEMPLATE \"\"\"{{- if .System }}
+[SYSTEM_PROMPT]
+{{ .System }}
+[/SYSTEM_PROMPT]
+{{- end }}
+{{- range .Messages }}
+{{- if eq .Role \"user\" }}
+[INST]
+{{ .Content }}
+[/INST]
+{{- else if eq .Role \"assistant\" }}
+{{ .Content }}
+{{- end }}
+{{- end }}\"\"\"
 PARAMETER num_ctx 131072
 PARAMETER num_predict 4096
 PARAMETER num_keep 1024
@@ -26,6 +43,7 @@ PARAMETER stop \"[/INST]\"
 PARAMETER stop \"</s>\"
 EOF"
          ${pkgs.docker}/bin/docker exec ollama ollama create "$model_name" -f "/root/.ollama/$model_name.modelfile"
+          ${pkgs.docker}/bin/docker exec ollama rm "/root/.ollama/$model_name.modelfile"
        else
          echo "$model_name already exists, skipping."
        fi
@@ -36,6 +54,10 @@ EOF"
      
      # Create Devstral
      create_model_if_missing "devstral-small-2:24b-128k" "devstral-small-2:24b" 
+      
+      # create_model_if_missing "qwen2.5-coder:32b-128k" "qwen2.5-coder:32b"
+      
+      # create_model_if_missing "mistral-large-planner:123b" "mistral-large:123b-instruct-v2407-q4_K_S"
    '';
    serviceConfig = {
      Type = "oneshot";
--- a/modules/nixos/services/openclaw_node.nix
+++ b/modules/nixos/services/openclaw_node.nix
@@ -0,0 +1,64 @@
+{ config, lib, pkgs, ... }:
+
+let
+  cfg = config.services.openclaw-node;
+  openclawPkg = pkgs.openclaw;
+in {
+  options.services.openclaw-node = {
+    enable = lib.mkEnableOption "OpenClaw Node service";
+
+    user = lib.mkOption {
+      type = lib.types.str;
+      default = "ai-worker";
+      description = "User to run the OpenClaw headless node as.";
+    };
+
+    gatewayHost = lib.mkOption {
+      type = lib.types.str;
+      default = "127.0.0.1";
+      description = "Gateway host (IP or hostname).";
+    };
+
+    gatewayPort = lib.mkOption {
+      type = lib.types.int;
+      default = 18789;
+      description = "Gateway WebSocket port.";
+    };
+
+    gatewayTokenFile = lib.mkOption {
+      type = lib.types.str;
+      default = "";
+      description = "Path to file containing the gateway auth token.";
+    };
+
+    displayName = lib.mkOption {
+      type = lib.types.str;
+      default = "lazyworkhorse-host";
+      description = "Display name for this node (shown in pairing).";
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    systemd.services.openclaw-node = {
+      description = "OpenClaw Headless Node Service";
+      after = [ "network.target" ];
+      wantedBy = [ "multi-user.target" ];
+
+      serviceConfig = {
+        Type = "exec";
+        User = cfg.user;
+        Group = cfg.user;
+        WorkingDirectory = "/home/${cfg.user}";
+        ExecStart = ''
+          ${pkgs.bash}/bin/bash -c 'export OPENCLAW_GATEWAY_TOKEN=$(cat ${cfg.gatewayTokenFile}) && exec ${openclawPkg}/bin/openclaw node run --host ${cfg.gatewayHost} --port ${toString cfg.gatewayPort} --display-name "${cfg.displayName}"'
+        '';
+        Restart = "always";
+        RestartSec = 5;
+      };
+
+      environment = {
+        NODE_ENV = "production";
+      };
+    };
+  };
+}
--- a/secrets/n8n_ssh_key.age
+++ b/secrets/n8n_ssh_key.age
--- a/secrets/containers.env.age
+++ b/secrets/containers.env.age
@@ -1,32 +1,34 @@
 -----BEGIN AGE ENCRYPTED FILE-----
-YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHNzaC1lZDI1NTE5IEdoTUQ4QSBmeWR3
-UzRxRGlkU2h2cjVjQlJrcjhYcm5oRWt3SFdSb0t4Wjd2ZTFKNTJjCjNIVmRtRmoz
-RTMyTDB5a1NJMU56RnFJRVFLSW1oMERGZ2RRSFgxQ0ZuSzgKLT4gVkAtZ3JlYXNl
-ICw+WDxrIFIsCk9MRDQ2ZWlPN2JUWDVyZWlQUGN3Ci0tLSB4WGhCdWdkN3M2THJZ
-VnB2SFFqa1NTcUh0bG9qTWNzT3BBUW5qQ0M4aUFzCsFpZE1btvUR1BwkUNC8qy3m
-0SwXk/gUS1519LuEnvZg7Mc+EB23e6nmz8rK34ycR+stTbVNv1xV2xCLxLoTg9wf
-+ThXsVrf18kv0N92X3d5v7clMVC4eMr9CcyfBY+HaMgNa72aRyVyyxKgg/v6oks+
-QEHssNw8+TKxjfeoxdCmsYVDEQME4id8vqoDOkyAg2IAXPCVVhN9G9fuMPyT1TWk
-yJD1RgpyzBkR0yBEQkxgY1GJ76TI0h85hveNbXQXZTuU2yj0KJbdj2gXDGdrqbu7
-r/6ZlRGlC2tSqtRBot6BatVIhtGZNVQnXbiVlQCmO1mh4XyxF7rKsCa7r3yVuvFN
-XybugrWSdG7dJF6ne/dMMsnwhvrKZFwUosjMnoH/x/LF2bOLAcA6i2WA6ivWzo9c
-6NmND6sLkQJWyychbLu4AmRg4MgVTlTGwTCizOe3xEo9qRrQBX7PmvuXSs+IE1o4
-l7pb0DSzIa80BT0Otj9tFlei1nwRh8wzEVECV0FUjilUvUp19mJ6Cn+/RnHTSOp9
-1UGrOFxbamx4L4yFWL3rWoqBpbO4CBSCGM7moDEhAQn/OsZgeUhKeIDvrEBtCeZ3
-vC/v0lVgfXZDd+aRSLPbGaRNwifyc5UeBWF1WvkJXi3jDUK7qFOT/RInVQDDF3u9
-YbvnHPler1UfbbPihHTFbCJu8lJHMLHfpe07j2cx4hCPMv/4Yx+xBAstPXwtaOuw
-/9PCvPvvGvygdzljKTksnsMVN11cQzmU3l1dKHvr5sNk1n+U+uW0xDrT9Nv1ZETg
-IY64EtzsqH48YAJ6SV6h4dZ8D9R5qTg4T5yP7D4PLuFtNGeqd7++zhBCZLZ3HEQ6
-M1SlHzWk59xBN4agrLKX0VjPYBwmg8wkpRfU5A4Rg36H4mZLHEUKqFVx6BaHfDZ2
-5P3o7GbZB39Zs9mZb70ZZJ5TFUsCEISfJHz/u5u4/duSBLeyHXah2dmXrQ1eUWT4
-MNNcJ6+53Us4LTe96ttYNa/v5RQVoarTwNM7x7ux5j59QHozVOK1NO8Z4+oHD/ZD
-rJQlXAeAUrhkZLluzzy1JL45tBpPm3oAfU3xB178c+fMoWtZxyWrBfu1iRzwyDWC
-MKgK29h9HeGwQc9dB8exQr2cj5NhqUOiaWP8dH1N/g+KYIPVNRgKjdDucsxTcbDN
-bIIz2qus6jQkOfmbtdoHWMp+kwXSHRF7MwECKxkAIcNdxnLI1DecNhjbiItnPlgI
-1uy0fERRc12BLg3dLV3YkBL358SRww+pxho87IQuS9x9aQeExksk0Y10QR8J/1g0
-cEXUhDNfeI+mKyuISxV6Zs4Fp7+6P6bd5Bs2Xyxw3A3PTdWn12brb62O1N81LiAv
-yccIDR24lb0VDD+aIq28FBUPQ62tVdtZgRfJhkVxelgzHuGATOTluDZH+6GE3rEj
-z1OoormFX/2TovCNnTVJRs1ifWUe+a2QHcAFFfL0Y1RBbIPYDMykfjCPNaWqarlX
-Z50QIWv6Ov1oDBZY59fjx5Bfm+Es+edMC4b2GibRKS5wwpOzGDEKDXVoTEv3NX+B
-NV4p3oDKEE8anYffrB+v
+YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHNzaC1lZDI1NTE5IEdoTUQ4QSBOL29w
+eGk1N2xxTHJtaUEvWWZmbkh1bk11Tjk3anNnMDB1cCtPYUMzdTNJCkdhQ08vblNG
+UlV1K2xVTGZVTzFWYXAzcjZaMWs0RTFWdStKSmlSTURvK1EKLT4gLC1zKU8zVkgt
+Z3JlYXNlIFUiXFcpS302IHByVn5jOy0gRDMKQjV3SHpDWUIybGFyQUg3ZlR0R2hV
+eWM3SFlCVW5mdlpBVUF3a0xpNlZCeGNUd1oxTTlkc1RkTXdZS0lFTmN3Ci0tLSA3
+VlBqM1VLWllZc0JnOTMvUFRjMU13OTdzMmhsdGJubkk5eGpERVVLYUk4Cnzh5UbU
+FlgqpM8jkJ6XlsaIDCw/G3D6uJ/GRJW4gIekuhAUxpZJrc8eOA8ZuHfGrBbH3acV
+tVafX5F0Kr2oOblqZ6gduZOUS52KmWH8stiBJM+e5ZZ7zRQVE4PJUKUPCzi+WdcH
+zr295T//FOdicrYHdsjfziKEHzBtUCFiATW05+O2zMjYjO6cPzePcCzPWinwiID6
+V+f6ngfkkQaj3wBGkzaieQJzRcdSwky21aVhGCCX/bvqx61iW2d5QAKxGbtQ2RcG
+X1okr+xunAM94nzDMv46vyN97KxY7cZd4pAaOxoICc2Tfhtw6F+iS6QkQh1odJzO
+7ZH+sSQCvndG+8z9shXGiHalASF5tdguM+JlEvAGljcaiAUtsQWxr9CoWiEkC6c6
+NCaECSYO8Il+SXBQnSZSGJSNDhuPYCYrsjXGSAONFixuyeslAkq9x2WUaUS4H063
+1QvRF7XO2tBPtgCLsSjdiGp0h+ImUaGdu6fDR7zrDsGsaAFCSFeH/rGNNXRQ2vP2
+CSfPfDDCqpUSCn0WuA30BtaPLxGmZT6OjFevKzYMNDmdeq9ia/q8K0hmjLUBdN3k
+tdYWbwoaf4gYbUWxSleD768b0Jgxss9Vod+sFQ+NYRksdGIeyND+aQIc312XehfA
+qHFBS8nlj7eUF5bdvCYQ64z741mH4cNlGxyjPBH1x8FHnEOocJXYt1l2AZSRJmJA
+c3z0QGXyuCbsrLBXWK1EKa/Juo4PGGsEVoLRhwJAQy9+i1JN0yrfRvSPyzvD4px6
+wRPzlZ80MQdb2lv84WS/zcOEZmZzlLntszTRRdIfAsuaavP2Rquh4rEXABYeTZwp
+5dem79s8bdW2nFsGMNz1OQKQwocyjYu1jJMHu6Gp7Ngdl1xyW7xfg0dezE1c0cIh
+xt1aLER9YJp4n5to5cOH16l3mjDHnAvABx38xE9loNL3399J/evw7LxpTYQ4v2Xv
+x8xnDHcqJ+deFSwyuUnMS5DkUeYuHmUl0Q2WYcfY+ibCmcgCb2ObTtuN1/ZxNYrL
+OKrnmfuSvBgyuIOj5e6uWW0+Zs8dHKXu2TgV8WignxOhl5zQgCpCBlqVfO0t+NCu
+Gi26hU/fhGWQ/1oQa3VkpGsypZbJpgQvfWxfcGHP/MMhnl01zzlP8/aexSY3pAxf
+fz9v0IVh6xxtu3zbiiVzUsXbfG7t+xY98jMphf4AS2mWva3GWVmhhu0lS3J3P+go
+YEEP4rOFHeU0Y1/6kLydTXvz4jMH0H92XQIzshd7vzQnEJPUPAzqRmw3LKYGgCI+
+wZEnxJ6ckqTkGBFnxTpy9LLllwmnz2Ky87nY3XAmqxlhb2Ap1XFAlfgszmGjc+Il
+KkIgoWQHTUm6QM9ta++oUTIDneOvxGd0zZsqoEhiC/7E01BNNZ6E58TeJU3fDlA3
+mX6n05XjwPRpgXZfayPoAgBlZc2H4KeiynxwNZ/dWu7qz7L6Ppk6Nvtly8giTbFx
+CA+tto7vq+D+CAEJ4bgyq4BCH4GL4APrhPcWp98Mko1WCiRTIKgkZxQCYvlg/LZq
+LNhMacP9T1qTvNC+yR1NEMiegE3APzk6CkDpVaO9+5f/sqifNPINCMothenI9ePw
+zjQLI3Mo1m73bkomytUZ7i1VstP5sEZ5LF72Sq7BpR3oQ3Gp0CAN9w==
 -----END AGE ENCRYPTED FILE-----
--- a/secrets/openclaw_gateway_token.age
+++ b/secrets/openclaw_gateway_token.age
@@ -0,0 +1,11 @@
+-----BEGIN AGE ENCRYPTED FILE-----
+YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IHNzaC1lZDI1NTE5IEdoTUQ4QSBCWEpO
+cG9yNnFpcHFqTkNzTngxU1MxN0NYK0hrZFhUTjVORWFrK3JNd2tZCmtMTGpwQk1E
+WlUwL3N6SGRWblpnNEkrWkkyU2hQMkRIK0M3R0pOVEREV3MKLT4gY2osLWdyZWFz
+ZSBacSozVVQgUCAxRS1OQSAuKXxDPCoKbStWNW1BZjBZQzNDaTlDbU5EZkxsRWxM
+cXJ3dDU1RDNpOXRlV0tzdEp2NUo3S1lhRG5Md0RHTGlJdkFSYmt5YQo4R1hiQWRG
+V2VxekJKZwotLS0geG1XSi9VbkhXZHQzcEFVS3hKNzVueXFLa2xnZTc3Q2tJTVZ5
+eXJabWk5Ywp6bJCP3s0xxzjE+eTR+cv7ZUnkoliT/n7uIprq1BTn/LIRLkUTUqs3
+NiDwrXcoq4/QKd0Dt+8ap3vFAuusjGxRlnYMaRrZie2AGtTV8U7Q7durm9o2K+/4
+QzRQ/MtumIQm
+-----END AGE ENCRYPTED FILE-----
--- a/secrets/secrets.nix
+++ b/secrets/secrets.nix
@@ -10,4 +10,5 @@ in
  "containers.env.age".publicKeys = authorizedKeys;
  "lazyworkhorse_host_ssh_key.age".publicKeys = authorizedKeys;
  "n8n_ssh_key.age".publicKeys = authorizedKeys;
+  "openclaw_gateway_token.age".publicKeys = authorizedKeys;
 }
--- a/users/ai-worker.nix
+++ b/users/ai-worker.nix
@@ -2,6 +2,8 @@
  users.users.ai-worker = {
    isSystemUser = true;
    group = "ai-worker";
+    home = "/home/ai-worker";
+    createHome = true;
    extraGroups = [ "docker" ];
    shell = pkgs.bashInteractive;
    openssh.authorizedKeys.keys = [
Author	SHA1	Message	Date
Hermes Agent	f4b666284a	feat: add Hyperspace Pods NixOS module and enable on lazyworkhorse Hyperspace Pods let multiple machines pool their GPUs into one private P2P mesh AI cluster. Models are split across all connected GPUs — e.g. two machines with 16GB VRAM each can run Qwen 3.5 32B together. Changes: - Add modules/nixos/services/hyperspace.nix — NixOS module that: * Fetches the Hyperspace CLI binary (v5.45.30) via fetchurl * Sets up systemd service for the agent * Opens firewall ports (libp2p 4001, chain 30301, API 8080) * Configures GPU passthrough for AMD MI50 (ROCm) - Register module in flake.nix for lazyworkhorse - Enable hyperspace service on lazyworkhorse (ai-worker user, port 8080) Usage after deployment: hyperspace pod create "tdnde-lab" # create pod hyperspace pod invite # share invite with cyt-pi curl http://localhost:8080/v1/chat/completions # OpenAI API See skill: nixos-hyperspace-pods	2026-05-02 15:36:15 +00:00
Hermes Agent	815ca3afa6	chore: update compose submodule to traefik logging branch	2026-05-02 15:30:28 +00:00
Hermes Agent	e983775c04	docs: add merge priority order with security hardening as #1 priority - Updated roadmap phase status (Phase 4 complete) - Added merge priority table with PR #28 (security) at top - Documented that security must merge before new services exposed - Added deployment command reference	2026-05-02 15:30:28 +00:00
Robert	bcf5cadaa0	olllama template fix to remove currenttime	2026-04-30 21:54:47 -04:00
Thierry Pouplier	7efba3ac5b	Compose update	2026-04-27 06:11:34 -04:00
Robert	cf1373cd68	Forced restart for docker services	2026-04-27 06:02:25 -04:00
Robert	bc875ef9fb	feat: isolate docker networks and add cyt-pi remote node config - Refactor all 12 compose stacks to use isolated networks with Traefik as the hub - Add openclaw-ssh sidecar to ai stack for reverse tunneling (port 2425) - Add sshnode entrypoint to Traefik configuration - Add cyt-pi host configuration for Pi Zero 2 W (headless) - Include kismet and target_detector_cli services for remote Wi-Fi monitoring - Add reverse SSH tunnel service via autossh	2026-04-06 19:14:57 -04:00
Robert	c579b07843	fix: read gateway token from secret file via bash	2026-04-04 17:49:39 -04:00
Robert	d3f50cdadc	fix: always restart node service on exit	2026-04-04 17:43:03 -04:00
Robert	8aa85e62e5	feat: add openclaw CLI to system packages	2026-04-04 17:23:15 -04:00
Robert	b9cf8a47f7	fix: set openclaw secret group to ai-worker	2026-04-04 17:15:24 -04:00
Robert	2e749228bb	fix: set correct working directory and create home for ai-worker	2026-04-04 17:07:13 -04:00
Robert	ce20fad4d3	fix: enable flake-self-attrs for lix compatibility	2026-04-04 16:54:10 -04:00
Robert	401b23ce46	feat: add openclaw node service and migrate to lix - Add headless openclaw node systemd service for host execution - Migrate from nix to lix package manager - Permit openclaw-2026.3.12 (insecure package warning) - Use ai-worker user for node service	2026-04-04 16:26:33 -04:00