Compare commits
4 Commits
feat/nix-d
...
feat/k3s-p
| Author | SHA1 | Date | |
|---|---|---|---|
| f4b666284a | |||
| 815ca3afa6 | |||
| e983775c04 | |||
|
bcf5cadaa0 |
@@ -13,7 +13,9 @@ None
|
|||||||
- ✅ **Phase 1: Foundation Setup** - Establish core NixOS configuration with flakes
|
- ✅ **Phase 1: Foundation Setup** - Establish core NixOS configuration with flakes
|
||||||
- ✅ **Phase 2: Docker Service Integration** - Integrate Docker Compose services
|
- ✅ **Phase 2: Docker Service Integration** - Integrate Docker Compose services
|
||||||
- ✅ **Phase 3: AI Assistant Integration** - Enable AI-assisted infrastructure management
|
- ✅ **Phase 3: AI Assistant Integration** - Enable AI-assisted infrastructure management
|
||||||
- [ ] **Phase 4: Internet Access & MCP** - MCP server for web access
|
- ✅ **Phase 4: Internet Access & MCP** - MCP server for web access
|
||||||
|
- 🚨 **Security Hardening** - CRITICAL: Firewall, fail2ban, SSH hardening (PR #28)
|
||||||
|
- [ ] **Phase 5: TAK Server** - Research, implementation, and validation
|
||||||
|
|
||||||
|
|
||||||
## Phase Details
|
## Phase Details
|
||||||
@@ -133,8 +135,25 @@ Plans:
|
|||||||
|
|
||||||
## Progress
|
## Progress
|
||||||
|
|
||||||
|
**Merge Priority Order** (CRITICAL - merge in this order):
|
||||||
|
|
||||||
|
| Priority | PR | Description | Status | Notes |
|
||||||
|
|----------|-----|-------------|--------|-------|
|
||||||
|
| 🚨 1 | #28 | **Security hardening** (firewall, fail2ban, SSH) | Open | **MERGE FIRST** - protects all other services |
|
||||||
|
| 2 | #22 | Matrix bridge dependency fix | Open | Blocks Hermes functionality |
|
||||||
|
| 3 | #21 | Backup network creation fix | Open | Infrastructure fix |
|
||||||
|
| 4 | #25 | Hermes voice GPU support | Open | Feature enhancement |
|
||||||
|
| 5 | #24 | uConsole CM5 host | Open | New hardware support |
|
||||||
|
| 6 | #23 | NixOS deployment infrastructure | Open | Deployment tooling |
|
||||||
|
| 7 | #1 | AI worker restricted access | Open | Legacy PR (superseded by hardening) |
|
||||||
|
|
||||||
**Execution Order:**
|
**Execution Order:**
|
||||||
Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7
|
Phases execute in numeric order: 1 → 2 → 3 → 4 → Security → 5 → 6 → 7
|
||||||
|
|
||||||
|
**Merge vs Phase Execution:**
|
||||||
|
- PRs can merge independently (no strict phase ordering for merges)
|
||||||
|
- **EXCEPTION:** Security hardening (#28) must merge before any new services are exposed
|
||||||
|
- After security merge, deploy with: `nh os switch --flake .#lazyworkhorse`
|
||||||
|
|
||||||
| Phase | Milestone | Plans Complete | Status | Completed |
|
| Phase | Milestone | Plans Complete | Status | Completed |
|
||||||
|-------|-----------|----------------|--------|-----------|
|
|-------|-----------|----------------|--------|-----------|
|
||||||
|
|||||||
Submodule assets/compose updated: fb0f2cbe84...a79fe9dffa
@@ -1,474 +0,0 @@
|
|||||||
# Nix Installation for Hermes Agent Container
|
|
||||||
|
|
||||||
This guide covers several approaches for installing Nix in the Hermes Agent Docker
|
|
||||||
container to enable remote NixOS deployment via `nixos-rebuild`. It covers both
|
|
||||||
x86_64 (lazyworkhorse) and aarch64 (cyt-pi, uConsole) architectures.
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
1. [Why Nix in a Container?](#why-nix-in-a-container)
|
|
||||||
2. [Prerequisites](#prerequisites)
|
|
||||||
3. [Installation Methods](#installation-methods)
|
|
||||||
- [Method A: Determinate Systems Installer](#method-a-determinate-systems-installer-recommended)
|
|
||||||
- [Method B: Vanilla Nix Installer](#method-b-vanilla-nix-installer)
|
|
||||||
- [Method C: NixOS-Based Container Image](#method-c-nixos-based-container-image)
|
|
||||||
4. [Architecture-Specific Notes](#architecture-specific-notes)
|
|
||||||
- [x86_64 (lazyworkhorse)](#x86_64-lazyworkhorse)
|
|
||||||
- [aarch64 (cyt-pi, uConsole)](#aarch64-cyt-pi-uconsole)
|
|
||||||
- [Cross-Compilation](#cross-compilation)
|
|
||||||
5. [Post-Install Configuration](#post-install-configuration)
|
|
||||||
6. [Verification](#verification)
|
|
||||||
7. [Container-Specific Considerations](#container-specific-considerations)
|
|
||||||
- [Persistence](#persistence)
|
|
||||||
- [Disk Space](#disk-space)
|
|
||||||
- [Security](#security)
|
|
||||||
- [Resource Constraints](#resource-constraints)
|
|
||||||
8. [Integration with deploy.sh](#integration-with-deploysh)
|
|
||||||
9. [Troubleshooting](#troubleshooting)
|
|
||||||
10. [References](#references)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Why Nix in a Container?
|
|
||||||
|
|
||||||
The Hermes Agent container runs on an Ubuntu/Debian base. To deploy NixOS
|
|
||||||
configurations to remote hosts, we need:
|
|
||||||
|
|
||||||
- `nix` — the Nix package manager (for building configurations)
|
|
||||||
- `nixos-rebuild` — the NixOS deployment tool
|
|
||||||
- Access to the infra repo with flake configuration
|
|
||||||
|
|
||||||
Installing Nix inside the container avoids:
|
|
||||||
- Host-level Nix installation on the Docker host
|
|
||||||
- Cross-container volume mounts of /nix/store
|
|
||||||
- Dependencies on the host's Nix daemon (which may be a different version)
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
- Docker host running Linux (x86_64 and/or aarch64)
|
|
||||||
- Container base: Debian/Ubuntu (apt-based)
|
|
||||||
- 1-2 GB additional disk space for Nix store
|
|
||||||
- Network access to cache.nixos.org (or a local binary cache)
|
|
||||||
- Git access to the infra repository
|
|
||||||
|
|
||||||
## Installation Methods
|
|
||||||
|
|
||||||
### Method A: Determinate Systems Installer (Recommended)
|
|
||||||
|
|
||||||
The Determinate Systems installer is the recommended approach. It is non-interactive,
|
|
||||||
sets up flakes by default, and handles multi-user installation cleanly.
|
|
||||||
|
|
||||||
**Dockerfile additions:**
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
# Install Nix (Determinate Systems installer)
|
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
||||||
curl \
|
|
||||||
xz-utils \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
# Download and run Nix installer (non-interactive)
|
|
||||||
RUN curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix \
|
|
||||||
-o /tmp/nix-install.sh \
|
|
||||||
&& chmod +x /tmp/nix-install.sh \
|
|
||||||
&& sh /tmp/nix-install.sh install --no-confirm \
|
|
||||||
&& rm /tmp/nix-install.sh
|
|
||||||
|
|
||||||
# Configure Nix for flakes
|
|
||||||
RUN mkdir -p /root/.config/nix \
|
|
||||||
&& echo 'experimental-features = nix-command flakes' > /root/.config/nix/nix.conf
|
|
||||||
|
|
||||||
# Add Nix to PATH for all users
|
|
||||||
ENV PATH="/nix/var/nix/profiles/default/bin:$PATH"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Pros:**
|
|
||||||
- Fully non-interactive (--no-confirm)
|
|
||||||
- Enables flakes automatically
|
|
||||||
- Sets up multi-user daemon
|
|
||||||
- Auto-selects correct architecture
|
|
||||||
- Handles upgrades gracefully
|
|
||||||
|
|
||||||
**Cons:**
|
|
||||||
- Downloads ~100 MB installer
|
|
||||||
- Requires systemd in container (works with --privileged or cgroupv2)
|
|
||||||
- Daemon mode may conflict with container exit semantics
|
|
||||||
|
|
||||||
**Container runtime additions:**
|
|
||||||
|
|
||||||
For the Nix daemon to work properly inside a container, you may need:
|
|
||||||
```dockerfile
|
|
||||||
# Ensure /nix is a volume for persistence
|
|
||||||
VOLUME /nix
|
|
||||||
|
|
||||||
# Or mount tmpfs for ephemeral builds:
|
|
||||||
# docker run --tmpfs /nix:exec,size=4G ...
|
|
||||||
```
|
|
||||||
|
|
||||||
### Method B: Vanilla Nix Installer
|
|
||||||
|
|
||||||
The official single-user Nix installer is lighter but requires manual flake setup.
|
|
||||||
|
|
||||||
**Dockerfile additions:**
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
# Install Nix (single-user, official installer)
|
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
||||||
curl \
|
|
||||||
sudo \
|
|
||||||
xz-utils \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
# Install Nix as root (single-user)
|
|
||||||
RUN curl -L https://nixos.org/nix/install -o /tmp/nix-install.sh \
|
|
||||||
&& chmod +x /tmp/nix-install.sh \
|
|
||||||
&& sh /tmp/nix-install.sh --no-daemon \
|
|
||||||
&& rm /tmp/nix-install.sh
|
|
||||||
|
|
||||||
# Enable flakes
|
|
||||||
RUN mkdir -p /root/.config/nix \
|
|
||||||
&& echo 'experimental-features = nix-command flakes' > /root/.config/nix/nix.conf
|
|
||||||
|
|
||||||
# Source Nix in shell
|
|
||||||
RUN echo '. /root/.nix-profile/etc/profile.d/nix.sh' >> /root/.bashrc
|
|
||||||
ENV PATH="/root/.nix-profile/bin:$PATH"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Pros:**
|
|
||||||
- Smaller installer
|
|
||||||
- No daemon needed (single-user mode)
|
|
||||||
- Works in containers without systemd
|
|
||||||
- Simpler container lifecycle
|
|
||||||
|
|
||||||
**Cons:**
|
|
||||||
- Manual flake configuration required
|
|
||||||
- Single-user only (no multi-user isolation)
|
|
||||||
- PATH must be set manually
|
|
||||||
- No automatic garbage collection
|
|
||||||
|
|
||||||
### Method C: NixOS-Based Container Image
|
|
||||||
|
|
||||||
For maximum isolation, use an official NixOS base image for the build stage.
|
|
||||||
|
|
||||||
**Multi-stage Dockerfile:**
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
# Build stage: NixOS builder
|
|
||||||
FROM nixos/nix:latest AS builder
|
|
||||||
|
|
||||||
COPY infra /infra
|
|
||||||
WORKDIR /infra
|
|
||||||
|
|
||||||
# Build the configuration once
|
|
||||||
RUN nix build '.#nixosConfigurations.lazyworkhorse.config.system.build.toplevel'
|
|
||||||
|
|
||||||
# Final stage: Hermes container
|
|
||||||
FROM ubuntu:22.04
|
|
||||||
|
|
||||||
# Copy the Nix closure and binary cache
|
|
||||||
COPY --from=builder /nix /nix
|
|
||||||
|
|
||||||
# ... rest of Hermes setup
|
|
||||||
```
|
|
||||||
|
|
||||||
**Pros:**
|
|
||||||
- Purely declarative build environment
|
|
||||||
- No installation at runtime
|
|
||||||
- Easy to pin Nix version
|
|
||||||
- Good for CI/CD pipelines
|
|
||||||
|
|
||||||
**Cons:**
|
|
||||||
- Requires multi-stage Docker build
|
|
||||||
- Larger initial image build
|
|
||||||
- Harder to update Nix version at runtime
|
|
||||||
- Overkill if Nix is only needed for `nixos-rebuild`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture-Specific Notes
|
|
||||||
|
|
||||||
### x86_64 (lazyworkhorse)
|
|
||||||
|
|
||||||
The Hermes container likely runs on x86_64 hardware for the primary server.
|
|
||||||
Nix will download x86_64 binaries from cache.nixos.org by default.
|
|
||||||
|
|
||||||
**No special configuration needed** — the standard installer works out of the box.
|
|
||||||
|
|
||||||
If the container is running on an AMD Ryzen/EPYC or Intel Xeon, consider:
|
|
||||||
```bash
|
|
||||||
# Enable CPU-specific optimizations (optional)
|
|
||||||
echo 'extra-platforms = x86_64-v1 x86_64-v2 x86_64-v3' >> /root/.config/nix/nix.conf
|
|
||||||
```
|
|
||||||
|
|
||||||
### aarch64 (cyt-pi, uConsole)
|
|
||||||
|
|
||||||
When building for aarch64 targets from an x86_64 container, you need either:
|
|
||||||
1. Remote builder (aarch64 machine does the build), or
|
|
||||||
2. QEMU-based emulation (slower but self-contained), or
|
|
||||||
3. Build directly on the aarch64 target using `--build-host`
|
|
||||||
|
|
||||||
**For QEMU emulation in the container:**
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
# Enable binfmt for aarch64 emulation
|
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
||||||
qemu-user-static \
|
|
||||||
binfmt-support \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
# Register aarch64 binfmt
|
|
||||||
RUN update-binfmts --enable qemu-aarch64
|
|
||||||
```
|
|
||||||
|
|
||||||
**Container runtime (for QEMU):**
|
|
||||||
```bash
|
|
||||||
docker run --privileged --rm ... hermes-agent
|
|
||||||
# Or with specific capability:
|
|
||||||
docker run --cap-add=SYS_ADMIN --security-opt seccomp=unconfined ... hermes-agent
|
|
||||||
```
|
|
||||||
|
|
||||||
### Cross-Compilation
|
|
||||||
|
|
||||||
For native cross-compilation (without emulation), add to your Nix configuration:
|
|
||||||
|
|
||||||
```nix
|
|
||||||
# In your flake.nix or nix.conf
|
|
||||||
{
|
|
||||||
nix.settings.extra-platforms = [ "aarch64-linux" "x86_64-linux" ];
|
|
||||||
nix.settings.extra-sandbox-paths = [ ];
|
|
||||||
boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Or in `nix.conf`:
|
|
||||||
```
|
|
||||||
extra-platforms = x86_64-linux aarch64-linux
|
|
||||||
extra-sandbox-paths =
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Post-Install Configuration
|
|
||||||
|
|
||||||
### nix.conf for Container Usage
|
|
||||||
|
|
||||||
Recommended `/root/.config/nix/nix.conf`:
|
|
||||||
|
|
||||||
```ini
|
|
||||||
experimental-features = nix-command flakes
|
|
||||||
substituters = https://cache.nixos.org/
|
|
||||||
trusted-users = root
|
|
||||||
max-jobs = auto
|
|
||||||
cores = 0
|
|
||||||
sandbox = false
|
|
||||||
```
|
|
||||||
|
|
||||||
Note: `sandbox = false` is needed inside containers that lack full sandbox
|
|
||||||
support. This is safe in a single-tenant container environment.
|
|
||||||
|
|
||||||
### PATH Setup
|
|
||||||
|
|
||||||
Add to your Dockerfile:
|
|
||||||
```dockerfile
|
|
||||||
ENV PATH="/nix/var/nix/profiles/default/bin:/root/.nix-profile/bin:${PATH}"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Shell Integration
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
RUN echo 'source /root/.nix-profile/etc/profile.d/nix.sh' >> /root/.bashrc
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
After installation, verify with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check Nix is available
|
|
||||||
nix --version
|
|
||||||
|
|
||||||
# Check nixos-rebuild
|
|
||||||
nixos-rebuild --help | head -3
|
|
||||||
|
|
||||||
# Verify flakes are enabled
|
|
||||||
nix flake --help
|
|
||||||
|
|
||||||
# Test a build (must be in infra repo)
|
|
||||||
cd /opt/data/infra
|
|
||||||
nix build --no-link '.#nixosConfigurations.lazyworkhorse.config.system.build.toplevel'
|
|
||||||
|
|
||||||
# Check available systems
|
|
||||||
nix eval --impure --expr 'builtins.currentSystem'
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Container-Specific Considerations
|
|
||||||
|
|
||||||
### Persistence
|
|
||||||
|
|
||||||
The `/nix` directory should be a Docker volume to avoid re-downloading
|
|
||||||
packages on every container restart:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# docker-compose.yml
|
|
||||||
volumes:
|
|
||||||
- nix-store:/nix
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
nix-store:
|
|
||||||
```
|
|
||||||
|
|
||||||
Without persistence, every container restart requires re-downloading the
|
|
||||||
entire Nix store (~500 MB - 2 GB depending on packages used).
|
|
||||||
|
|
||||||
### Disk Space
|
|
||||||
|
|
||||||
The Nix store grows over time as old generations accumulate. Set up garbage
|
|
||||||
collection:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Manual GC
|
|
||||||
nix store gc
|
|
||||||
|
|
||||||
# Remove old generations
|
|
||||||
nix-collect-garbage --delete-older-than 30d
|
|
||||||
|
|
||||||
# Automatic GC (in nix.conf)
|
|
||||||
# Currently not supported in nix.conf, but you can run a cron job:
|
|
||||||
# nix store gc --max 10G
|
|
||||||
```
|
|
||||||
|
|
||||||
In Docker, limit store growth with:
|
|
||||||
```dockerfile
|
|
||||||
# Configure max store size
|
|
||||||
RUN mkdir -p /etc/nix && \
|
|
||||||
echo 'min-free = 5368709120' > /etc/nix/nix.conf # Keep 5GB free
|
|
||||||
```
|
|
||||||
|
|
||||||
### Security
|
|
||||||
|
|
||||||
Running Nix in a container introduces some security considerations:
|
|
||||||
|
|
||||||
1. **Sandboxing:** `sandbox = false` disables build isolation. In a multi-tenant
|
|
||||||
container, this means Nix builds can affect the container filesystem.
|
|
||||||
**Mitigation:** Only build configs you trust (your own infra repo).
|
|
||||||
|
|
||||||
2. **Network access:** The container needs outbound access to cache.nixos.org.
|
|
||||||
If using a restricted network, set up a local binary cache:
|
|
||||||
```nix
|
|
||||||
substituters = https://cache.nixos.org/ https://nix-cache.internal/
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Privileged mode:** QEMU emulation for aarch64 builds may need `--privileged`
|
|
||||||
or `--security-opt seccomp=unconfined`. This reduces container isolation.
|
|
||||||
**Mitigation:** Use remote builders or build natively on the target.
|
|
||||||
|
|
||||||
4. **Supply chain:** Nix derivations pin exact inputs via hashes. Verify
|
|
||||||
flake.lock is committed and reviewed.
|
|
||||||
|
|
||||||
### Resource Constraints
|
|
||||||
|
|
||||||
Nix builds can be memory and CPU intensive:
|
|
||||||
|
|
||||||
```nix
|
|
||||||
# Limit build parallelism in nix.conf
|
|
||||||
max-jobs = 2
|
|
||||||
cores = 4
|
|
||||||
|
|
||||||
# Or set per-build:
|
|
||||||
# nix build --max-jobs 2 --cores 4
|
|
||||||
```
|
|
||||||
|
|
||||||
For containers with limited memory (< 2 GB), consider:
|
|
||||||
- Building on the target host instead (`--build-host`)
|
|
||||||
- Using the deploy script's `build` action separately
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Integration with deploy.sh
|
|
||||||
|
|
||||||
The deployment script at `scripts/deploy.sh` expects:
|
|
||||||
|
|
||||||
1. **Nix installed** with flakes enabled
|
|
||||||
2. **SSH key** at `/opt/data/home/.ssh/id_hermes_gitea` (or via SSH_KEY env)
|
|
||||||
3. **Infra repo** cloned at the script's parent directory
|
|
||||||
4. **Network access** to:
|
|
||||||
- `code.lazyworkhorse.net:2222` (Gitea for git operations)
|
|
||||||
- Target hosts via SSH (see deploy-ssh-config)
|
|
||||||
- `cache.nixos.org` or a local substitute
|
|
||||||
|
|
||||||
Typical usage from Hermes:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Full deployment
|
|
||||||
./scripts/deploy.sh lazyworkhorse master switch
|
|
||||||
|
|
||||||
# Build-only check (no remote deployment)
|
|
||||||
./scripts/deploy.sh cyt-pi master build
|
|
||||||
|
|
||||||
# Dry run
|
|
||||||
./scripts/deploy.sh uConsole feat/test dry-activate
|
|
||||||
|
|
||||||
# Override SSH key
|
|
||||||
SSH_KEY=/opt/data/home/.ssh/my-custom-key ./deploy.sh lazyworkhorse
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### "nix: command not found"
|
|
||||||
|
|
||||||
- Ensure Nix is installed and PATH is set:
|
|
||||||
```bash
|
|
||||||
export PATH="/nix/var/nix/profiles/default/bin:/root/.nix-profile/bin:$PATH"
|
|
||||||
```
|
|
||||||
- Check installation: `ls -la /nix/` should exist
|
|
||||||
- Re-source profile: `. /root/.nix-profile/etc/profile.d/nix.sh`
|
|
||||||
|
|
||||||
### "error: unable to download ... cache.nixos.org"
|
|
||||||
|
|
||||||
- Check network connectivity: `ping cache.nixos.org`
|
|
||||||
- Check DNS resolution from inside the container
|
|
||||||
- If behind a proxy, set `http_proxy` / `https_proxy` environment variables
|
|
||||||
|
|
||||||
### "sandbox: cannot run build in sandbox"
|
|
||||||
|
|
||||||
- Add `sandbox = false` to nix.conf
|
|
||||||
- Or run container with `--privileged` or `--security-opt seccomp=unconfined`
|
|
||||||
|
|
||||||
### "aarch64-linux builds fail on x86_64"
|
|
||||||
|
|
||||||
- QEMU binfmt not registered. Check: `ls /proc/sys/fs/binfmt_misc/`
|
|
||||||
- Rebuild QEMU registration: `docker run --privileged --rm tonistiigi/binfmt --install all`
|
|
||||||
- Or use `--build-host` to build on the target directly
|
|
||||||
|
|
||||||
### "nixos-rebuild fails with SSH errors"
|
|
||||||
|
|
||||||
- Verify SSH key exists and has correct permissions:
|
|
||||||
```bash
|
|
||||||
ls -la /opt/data/home/.ssh/id_hermes_gitea
|
|
||||||
chmod 600 /opt/data/home/.ssh/id_hermes_gitea
|
|
||||||
```
|
|
||||||
- Test SSH manually: `ssh -p 2424 -i /opt/data/home/.ssh/id_hermes_gitea ai-worker@lazyworkhorse.net`
|
|
||||||
- Check target host is reachable: `nc -zv lazyworkhorse.net 2424`
|
|
||||||
|
|
||||||
### "git fetch fails from Gitea"
|
|
||||||
|
|
||||||
- Verify GIT_SSH_COMMAND is set: `echo $GIT_SSH_COMMAND`
|
|
||||||
- Test git SSH: `ssh -T git@code.lazyworkhorse.net -p 2222`
|
|
||||||
- Check the infra repo remote: `git remote -v`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## References
|
|
||||||
|
|
||||||
- [Determinate Systems Nix Installer](https://github.com/DeterminateSystems/nix-installer)
|
|
||||||
- [NixOS Manual: Installation](https://nixos.org/manual/nix/stable/installation/)
|
|
||||||
- [NixOS Wiki: Flakes](https://nixos.wiki/wiki/Flakes)
|
|
||||||
- [NixOS Wiki: nixos-rebuild](https://nixos.wiki/wiki/Nixos-rebuild)
|
|
||||||
- [NixOS Wiki: Cross Compilation](https://nixos.wiki/wiki/Cross_Compilation)
|
|
||||||
- [Multi-arch Docker with QEMU](https://github.com/multiarch/qemu-user-static)
|
|
||||||
@@ -61,6 +61,7 @@
|
|||||||
./modules/nixos/services/open_code_server.nix
|
./modules/nixos/services/open_code_server.nix
|
||||||
./modules/nixos/services/ollama_init_custom_models.nix
|
./modules/nixos/services/ollama_init_custom_models.nix
|
||||||
./modules/nixos/services/openclaw_node.nix
|
./modules/nixos/services/openclaw_node.nix
|
||||||
|
./modules/nixos/services/hyperspace.nix
|
||||||
./users/gortium.nix
|
./users/gortium.nix
|
||||||
./users/ai-worker.nix
|
./users/ai-worker.nix
|
||||||
];
|
];
|
||||||
|
|||||||
@@ -277,6 +277,16 @@
|
|||||||
displayName = "lazyworkhorse-host";
|
displayName = "lazyworkhorse-host";
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# Hyperspace Pods — P2P mesh AI cluster (combine GPUs across machines)
|
||||||
|
services.hyperspace = {
|
||||||
|
enable = true;
|
||||||
|
user = "ai-worker";
|
||||||
|
apiPort = 8080;
|
||||||
|
profile = "auto";
|
||||||
|
openFirewall = true;
|
||||||
|
extraArgs = [ "--verbose" ];
|
||||||
|
};
|
||||||
|
|
||||||
# Public host ssh key (kept in sync with the private one)
|
# Public host ssh key (kept in sync with the private one)
|
||||||
environment.etc."ssh/ssh_host_ed25519_key.pub".text =
|
environment.etc."ssh/ssh_host_ed25519_key.pub".text =
|
||||||
"${keys.hosts.lazyworkhorse.main}";
|
"${keys.hosts.lazyworkhorse.main}";
|
||||||
|
|||||||
235
modules/nixos/services/hyperspace.nix
Normal file
235
modules/nixos/services/hyperspace.nix
Normal file
@@ -0,0 +1,235 @@
|
|||||||
|
{ config, lib, pkgs, ... }:
|
||||||
|
|
||||||
|
with lib;
|
||||||
|
|
||||||
|
let
|
||||||
|
cfg = config.services.hyperspace;
|
||||||
|
|
||||||
|
# Hyperspace CLI release from github.com/hyperspaceai/aios-cli
|
||||||
|
# The binary bundles Node.js runtime + llama.cpp + sidecars (~914MB)
|
||||||
|
# It auto-updates via `hyperspace update` post-install
|
||||||
|
hyperspacePkg = pkgs.stdenv.mkDerivation rec {
|
||||||
|
pname = "hyperspace";
|
||||||
|
version = cfg.release;
|
||||||
|
|
||||||
|
src = pkgs.fetchurl {
|
||||||
|
url = "https://github.com/hyperspaceai/aios-cli/releases/download/v${version}/aios-cli-x86_64-unknown-linux-gnu.tar.gz";
|
||||||
|
hash = "sha256-f6fJ8t3exqtYwUD5j+WvD+Hm0oN/Eef0X+R9Rj23dE0=";
|
||||||
|
};
|
||||||
|
|
||||||
|
sourceRoot = ".";
|
||||||
|
|
||||||
|
installPhase = ''
|
||||||
|
mkdir -p $out/bin $out/lib/hyperspace
|
||||||
|
|
||||||
|
# Main CLI binary
|
||||||
|
cp aios-cli $out/bin/hyperspace
|
||||||
|
chmod +x $out/bin/hyperspace
|
||||||
|
|
||||||
|
# Sidecar binaries
|
||||||
|
for f in _aios-cli pod-raft hyperspace-*; do
|
||||||
|
[ -f "$f" ] && install -m755 "$f" $out/lib/hyperspace/ || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# WASM, native modules, Python shards
|
||||||
|
cp -r *.wasm $out/lib/hyperspace/ 2>/dev/null || true
|
||||||
|
cp -r *.node $out/lib/hyperspace/ 2>/dev/null || true
|
||||||
|
mkdir -p $out/lib/hyperspace/python
|
||||||
|
cp -r python/* $out/lib/hyperspace/python/ 2>/dev/null || true
|
||||||
|
|
||||||
|
# Skills directory
|
||||||
|
mkdir -p $out/share/hyperspace
|
||||||
|
cp -r skills $out/share/hyperspace/ 2>/dev/null || true
|
||||||
|
|
||||||
|
# Set HYPERSPACE_PATH so the binary finds sidecars
|
||||||
|
wrapProgram $out/bin/hyperspace \
|
||||||
|
--set HYPERSPACE_PATH "$out/lib/hyperspace" \
|
||||||
|
--set HYPERSPACE_SKILLS_DIR "$out/share/hyperspace/skills"
|
||||||
|
'';
|
||||||
|
|
||||||
|
nativeBuildInputs = with pkgs; [ makeWrapper ];
|
||||||
|
|
||||||
|
meta = {
|
||||||
|
description = "Hyperspace CLI — P2P mesh AI inference network (Pods)";
|
||||||
|
longDescription = ''
|
||||||
|
Hyperspace Pods let multiple machines pool their GPUs into one private
|
||||||
|
AI cluster. Install the CLI, create a pod, share an invite link — your
|
||||||
|
machines form a P2P mesh and can run models split across all connected
|
||||||
|
GPUs. Exposes an OpenAI-compatible API for use with Cursor, Claude Code,
|
||||||
|
Aider, etc.
|
||||||
|
'';
|
||||||
|
homepage = "https://hyperspace.sh";
|
||||||
|
sourceProvenance = with lib; [ sourceTypes.binaryNativeCode ];
|
||||||
|
license = lib.licenses.unfree;
|
||||||
|
platforms = [ "x86_64-linux" ];
|
||||||
|
maintainers = [ ];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
in {
|
||||||
|
options.services.hyperspace = {
|
||||||
|
enable = mkEnableOption "Hyperspace P2P AI agent (Pods)";
|
||||||
|
|
||||||
|
release = mkOption {
|
||||||
|
type = types.str;
|
||||||
|
default = "5.45.30";
|
||||||
|
description = "Hyperspace CLI release version (from GitHub releases).";
|
||||||
|
};
|
||||||
|
|
||||||
|
user = mkOption {
|
||||||
|
type = types.str;
|
||||||
|
default = "ai-worker";
|
||||||
|
description = "System user to run the Hyperspace agent.";
|
||||||
|
};
|
||||||
|
|
||||||
|
apiPort = mkOption {
|
||||||
|
type = types.port;
|
||||||
|
default = 8080;
|
||||||
|
description = "Port for the OpenAI-compatible API server.";
|
||||||
|
};
|
||||||
|
|
||||||
|
autoStart = mkOption {
|
||||||
|
type = types.bool;
|
||||||
|
default = true;
|
||||||
|
description = "Auto-start the Hyperspace agent on boot.";
|
||||||
|
};
|
||||||
|
|
||||||
|
openFirewall = mkOption {
|
||||||
|
type = types.bool;
|
||||||
|
default = true;
|
||||||
|
description = "Open firewall ports for P2P traffic (libp2p 4001, chain 30301, API).";
|
||||||
|
};
|
||||||
|
|
||||||
|
profile = mkOption {
|
||||||
|
type = types.enum [ "auto" "full" "inference" "embedding" "relay" "storage" ];
|
||||||
|
default = "auto";
|
||||||
|
description = ''
|
||||||
|
Agent profile:
|
||||||
|
- auto: auto-detect hardware
|
||||||
|
- full: all 9 capabilities
|
||||||
|
- inference: GPU inference only
|
||||||
|
- embedding: CPU embedding only
|
||||||
|
- relay: lightweight relay
|
||||||
|
- storage: storage + memory
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
extraArgs = mkOption {
|
||||||
|
type = types.listOf types.str;
|
||||||
|
default = [ ];
|
||||||
|
description = "Extra arguments passed to `hyperspace start`.";
|
||||||
|
};
|
||||||
|
|
||||||
|
dataDir = mkOption {
|
||||||
|
type = types.str;
|
||||||
|
default = "/var/lib/hyperspace";
|
||||||
|
description = "Data directory for agent state (models, config, logs).";
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
config = mkIf cfg.enable {
|
||||||
|
# Ensure the service user exists
|
||||||
|
users.users.${cfg.user} = {
|
||||||
|
isSystemUser = true;
|
||||||
|
group = cfg.user;
|
||||||
|
home = "/home/${cfg.user}";
|
||||||
|
createHome = true;
|
||||||
|
shell = pkgs.bash;
|
||||||
|
};
|
||||||
|
users.groups.${cfg.user} = { };
|
||||||
|
|
||||||
|
# Install the hyperspace binary
|
||||||
|
environment.systemPackages = [ hyperspacePkg ];
|
||||||
|
|
||||||
|
# Data directories
|
||||||
|
systemd.tmpfiles.rules = [
|
||||||
|
"d ${cfg.dataDir} 0755 ${cfg.user} ${cfg.user} -"
|
||||||
|
"d ${cfg.dataDir}/models 0755 ${cfg.user} ${cfg.user} -"
|
||||||
|
"d ${cfg.dataDir}/data 0755 ${cfg.user} ${cfg.user} -"
|
||||||
|
];
|
||||||
|
|
||||||
|
# Systemd service: runs the Hyperspace agent as a system daemon
|
||||||
|
systemd.services.hyperspace = {
|
||||||
|
description = "Hyperspace P2P AI Agent — Pods mesh cluster";
|
||||||
|
documentation = [ "https://hyperspace.sh" "https://github.com/hyperspaceai/aios-cli" ];
|
||||||
|
after = [ "network-online.target" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
wantedBy = mkIf cfg.autoStart [ "multi-user.target" ];
|
||||||
|
|
||||||
|
environment = {
|
||||||
|
HYPERSPACE_HOME = cfg.dataDir;
|
||||||
|
HYPERSPACE_API_PORT = toString cfg.apiPort;
|
||||||
|
HYPERSPACE_PATH = "${hyperspacePkg}/lib/hyperspace";
|
||||||
|
};
|
||||||
|
|
||||||
|
path = with pkgs; [ bash curl nodejs ];
|
||||||
|
|
||||||
|
script = ''
|
||||||
|
# Wait for network connectivity before starting
|
||||||
|
${pkgs.bash}/bin/bash -c '
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
ping -c 1 -W 1 8.8.8.8 >/dev/null 2>&1 && break
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
' || true
|
||||||
|
|
||||||
|
exec ${hyperspacePkg}/bin/hyperspace start \
|
||||||
|
--profile ${cfg.profile} \
|
||||||
|
--api-port ${toString cfg.apiPort} \
|
||||||
|
${lib.escapeShellArgs cfg.extraArgs}
|
||||||
|
'';
|
||||||
|
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "exec";
|
||||||
|
User = cfg.user;
|
||||||
|
Group = cfg.user;
|
||||||
|
WorkingDirectory = cfg.dataDir;
|
||||||
|
Restart = "always";
|
||||||
|
RestartSec = 10;
|
||||||
|
TimeoutStartSec = 180;
|
||||||
|
TimeoutStopSec = 30;
|
||||||
|
KillMode = "mixed";
|
||||||
|
|
||||||
|
# File limits for network-heavy P2P agent
|
||||||
|
LimitNOFILE = 65536;
|
||||||
|
LimitNPROC = 4096;
|
||||||
|
|
||||||
|
# GPU access — AMD MI50 (ROCm) through /dev/kfd and /dev/dri
|
||||||
|
DeviceAllow = [
|
||||||
|
"/dev/kfd" "rw"
|
||||||
|
"/dev/dri" "rw"
|
||||||
|
];
|
||||||
|
SupplementaryGroups = [ "video" "render" ];
|
||||||
|
|
||||||
|
# Security hardening
|
||||||
|
NoNewPrivileges = true;
|
||||||
|
ProtectSystem = "strict";
|
||||||
|
ProtectHome = true;
|
||||||
|
PrivateTmp = true;
|
||||||
|
PrivateDevices = false; # needs GPU access
|
||||||
|
ReadWritePaths = [
|
||||||
|
cfg.dataDir
|
||||||
|
"/tmp"
|
||||||
|
];
|
||||||
|
BindPaths = [
|
||||||
|
# GPU devices for AMD MI50
|
||||||
|
"/dev/kfd"
|
||||||
|
"/dev/dri"
|
||||||
|
];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
# Firewall: open P2P ports for the mesh network
|
||||||
|
networking.firewall = mkIf cfg.openFirewall {
|
||||||
|
allowedTCPPorts = [
|
||||||
|
4001 # libp2p P2P (agent gossip, DHT, circuits)
|
||||||
|
30301 # Chain P2P (blockchain consensus)
|
||||||
|
cfg.apiPort # OpenAI-compatible API
|
||||||
|
];
|
||||||
|
allowedUDPPorts = [
|
||||||
|
4001 # libp2p QUIC transport
|
||||||
|
30301 # Chain UDP discovery
|
||||||
|
];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -14,8 +14,25 @@
|
|||||||
local base_model=$2
|
local base_model=$2
|
||||||
if ! ${pkgs.docker}/bin/docker exec ollama ollama list | grep -q "$model_name"; then
|
if ! ${pkgs.docker}/bin/docker exec ollama ollama list | grep -q "$model_name"; then
|
||||||
echo "$model_name not found, creating from $base_model..."
|
echo "$model_name not found, creating from $base_model..."
|
||||||
|
|
||||||
|
# We use a custom TEMPLATE block to strip the 'currentDate' function
|
||||||
|
# which is unsupported in Ollama 0.5.7 but present in Devstral's default manifest.
|
||||||
${pkgs.docker}/bin/docker exec ollama sh -c "cat <<EOF > /root/.ollama/$model_name.modelfile
|
${pkgs.docker}/bin/docker exec ollama sh -c "cat <<EOF > /root/.ollama/$model_name.modelfile
|
||||||
FROM $base_model
|
FROM $base_model
|
||||||
|
TEMPLATE \"\"\"{{- if .System }}
|
||||||
|
[SYSTEM_PROMPT]
|
||||||
|
{{ .System }}
|
||||||
|
[/SYSTEM_PROMPT]
|
||||||
|
{{- end }}
|
||||||
|
{{- range .Messages }}
|
||||||
|
{{- if eq .Role \"user\" }}
|
||||||
|
[INST]
|
||||||
|
{{ .Content }}
|
||||||
|
[/INST]
|
||||||
|
{{- else if eq .Role \"assistant\" }}
|
||||||
|
{{ .Content }}
|
||||||
|
{{- end }}
|
||||||
|
{{- end }}\"\"\"
|
||||||
PARAMETER num_ctx 131072
|
PARAMETER num_ctx 131072
|
||||||
PARAMETER num_predict 4096
|
PARAMETER num_predict 4096
|
||||||
PARAMETER num_keep 1024
|
PARAMETER num_keep 1024
|
||||||
@@ -26,6 +43,7 @@ PARAMETER stop \"[/INST]\"
|
|||||||
PARAMETER stop \"</s>\"
|
PARAMETER stop \"</s>\"
|
||||||
EOF"
|
EOF"
|
||||||
${pkgs.docker}/bin/docker exec ollama ollama create "$model_name" -f "/root/.ollama/$model_name.modelfile"
|
${pkgs.docker}/bin/docker exec ollama ollama create "$model_name" -f "/root/.ollama/$model_name.modelfile"
|
||||||
|
${pkgs.docker}/bin/docker exec ollama rm "/root/.ollama/$model_name.modelfile"
|
||||||
else
|
else
|
||||||
echo "$model_name already exists, skipping."
|
echo "$model_name already exists, skipping."
|
||||||
fi
|
fi
|
||||||
@@ -36,6 +54,10 @@ EOF"
|
|||||||
|
|
||||||
# Create Devstral
|
# Create Devstral
|
||||||
create_model_if_missing "devstral-small-2:24b-128k" "devstral-small-2:24b"
|
create_model_if_missing "devstral-small-2:24b-128k" "devstral-small-2:24b"
|
||||||
|
|
||||||
|
# create_model_if_missing "qwen2.5-coder:32b-128k" "qwen2.5-coder:32b"
|
||||||
|
|
||||||
|
# create_model_if_missing "mistral-large-planner:123b" "mistral-large:123b-instruct-v2407-q4_K_S"
|
||||||
'';
|
'';
|
||||||
serviceConfig = {
|
serviceConfig = {
|
||||||
Type = "oneshot";
|
Type = "oneshot";
|
||||||
|
|||||||
@@ -1,63 +0,0 @@
|
|||||||
# Hermes Container SSH Configuration
|
|
||||||
# For NixOS deployment to remote hosts
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# cp scripts/deploy-ssh-config ~/.ssh/config.d/hermes-include
|
|
||||||
# Or: cat scripts/deploy-ssh-config >> ~/.ssh/config
|
|
||||||
#
|
|
||||||
# This config covers all NixOS hosts managed from the Hermes container.
|
|
||||||
# Lazyworkhorse has two users: ai-worker (primary automation) and gortium (admin).
|
|
||||||
# Cyt-pi connects via reverse SSH tunnel on port 19999.
|
|
||||||
# uConsole is a placeholder until LAN-hostname resolution is confirmed.
|
|
||||||
|
|
||||||
# ── Global defaults ──────────────────────────────────────────────────
|
|
||||||
Host *
|
|
||||||
ServerAliveInterval 60
|
|
||||||
ServerAliveCountMax 3
|
|
||||||
TCPKeepAlive yes
|
|
||||||
Compression yes
|
|
||||||
CompressionLevel 6
|
|
||||||
ControlMaster auto
|
|
||||||
ControlPath ~/.ssh/controlmasters/%r@%h:%p
|
|
||||||
ControlPersist 10m
|
|
||||||
StrictHostKeyChecking no
|
|
||||||
UserKnownHostsFile /dev/null
|
|
||||||
|
|
||||||
# ── Hosts ──────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
# Lazyworkhorse — x86_64 main server (ai-worker@lazyworkhorse.net:2424)
|
|
||||||
Host lazyworkhorse
|
|
||||||
HostName lazyworkhorse.net
|
|
||||||
User ai-worker
|
|
||||||
Port 2424
|
|
||||||
IdentityFile /opt/data/home/.ssh/id_hermes_gitea
|
|
||||||
|
|
||||||
# Lazyworkhorse — admin access (gortium@lazyworkhorse.net:2425)
|
|
||||||
Host lazyworkhorse-admin
|
|
||||||
HostName lazyworkhorse.net
|
|
||||||
User gortium
|
|
||||||
Port 2425
|
|
||||||
IdentityFile /opt/data/home/.ssh/id_hermes_gitea
|
|
||||||
|
|
||||||
# Cyt-pi — aarch64 Pi Zero 2 W
|
|
||||||
# Connected via reverse SSH tunnel (gortium directs tunnel to :19999)
|
|
||||||
Host cyt-pi
|
|
||||||
HostName localhost
|
|
||||||
User gortium
|
|
||||||
Port 19999
|
|
||||||
IdentityFile /opt/data/home/.ssh/id_hermes_gitea
|
|
||||||
|
|
||||||
# uConsole — aarch64 ClockworkPi (placeholder hostname)
|
|
||||||
# Replace uconsole.lan with actual IP/hostname when deployed
|
|
||||||
Host uConsole uconsole
|
|
||||||
HostName uconsole.lan
|
|
||||||
User gortium
|
|
||||||
Port 22
|
|
||||||
IdentityFile /opt/data/home/.ssh/id_hermes_gitea
|
|
||||||
|
|
||||||
# ── Gitea host — for git operations ──────────────────────────────────
|
|
||||||
Host code
|
|
||||||
HostName code.lazyworkhorse.net
|
|
||||||
Port 2222
|
|
||||||
User gortium
|
|
||||||
IdentityFile /opt/data/home/.ssh/id_hermes_gitea
|
|
||||||
@@ -1,286 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# NixOS Deployment Helper Script
|
|
||||||
# Remote NixOS deployment from Hermes container to target hosts.
|
|
||||||
#
|
|
||||||
# Usage: ./deploy.sh <hostname> [branch] [action]
|
|
||||||
#
|
|
||||||
# Actions:
|
|
||||||
# switch Activate configuration now (default)
|
|
||||||
# boot Activate on next reboot
|
|
||||||
# test Activate without switching generations
|
|
||||||
# build Build locally only, no remote activation
|
|
||||||
# dry-activate Show what would change without applying
|
|
||||||
#
|
|
||||||
# Examples:
|
|
||||||
# ./deploy.sh lazyworkhorse # deploy master/switch to lazyworkhorse
|
|
||||||
# ./deploy.sh cyt-pi feat/test boot # deploy feat/test branch, activate on boot
|
|
||||||
# ./deploy.sh uConsole master build # just build, don't deploy
|
|
||||||
# NO_BUILD_CHECK=1 ./deploy.sh uConsole # skip the pre-flight nix build
|
|
||||||
#
|
|
||||||
# Environment variables:
|
|
||||||
# SSH_USER SSH user (default: auto-detected per host)
|
|
||||||
# SSH_PORT SSH port (default: auto-detected per host)
|
|
||||||
# SSH_KEY SSH identity file
|
|
||||||
# BUILD_HOST Build flake for this host (default: same as target host)
|
|
||||||
# NO_BUILD_CHECK Set to 1 to skip local nix build before deployment
|
|
||||||
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
# ── Colors ──────────────────────────────────────────────────────────────
|
|
||||||
RED='\033[0;31m'
|
|
||||||
GREEN='\033[0;32m'
|
|
||||||
YELLOW='\033[1;33m'
|
|
||||||
BLUE='\033[0;34m'
|
|
||||||
CYAN='\033[0;36m'
|
|
||||||
NC='\033[0m' # No Color
|
|
||||||
|
|
||||||
info() { echo -e "${BLUE}[INFO]${NC} $*"; }
|
|
||||||
ok() { echo -e "${GREEN}[OK]${NC} $*"; }
|
|
||||||
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
|
|
||||||
error() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
|
|
||||||
step() { echo -e "\n${CYAN}━━━ $* ━━━${NC}"; }
|
|
||||||
|
|
||||||
# ── Cleanup trap ───────────────────────────────────────────────────────
|
|
||||||
cleanup() {
|
|
||||||
local ec=$?
|
|
||||||
if [ $ec -ne 0 ]; then
|
|
||||||
error "Deployment failed with exit code $ec"
|
|
||||||
fi
|
|
||||||
exit $ec
|
|
||||||
}
|
|
||||||
trap cleanup EXIT
|
|
||||||
|
|
||||||
# ── Usage / Help ───────────────────────────────────────────────────────
|
|
||||||
show_usage() {
|
|
||||||
cat <<EOF
|
|
||||||
Usage: $0 <hostname> [branch] [action]
|
|
||||||
|
|
||||||
Remote NixOS deployment from Hermes container to target hosts.
|
|
||||||
|
|
||||||
HOSTNAME (required):
|
|
||||||
lazyworkhorse x86_64 main server
|
|
||||||
cyt-pi aarch64 Pi Zero 2 W (via reverse tunnel)
|
|
||||||
uConsole aarch64 ClockworkPi
|
|
||||||
|
|
||||||
BRANCH (optional, default: master):
|
|
||||||
Git branch or tag to deploy. Fetched from origin.
|
|
||||||
|
|
||||||
ACTION (optional, default: switch):
|
|
||||||
switch Activate configuration now (default)
|
|
||||||
boot Activate on next reboot
|
|
||||||
test Activate without switching generations
|
|
||||||
build Build locally only, skip remote deployment
|
|
||||||
dry-activate Show what would change without applying
|
|
||||||
|
|
||||||
Environment variables:
|
|
||||||
SSH_USER SSH username override
|
|
||||||
SSH_PORT SSH port override
|
|
||||||
SSH_KEY SSH identity file path
|
|
||||||
BUILD_HOST Build flake hostname (default: same as HOSTNAME)
|
|
||||||
NO_BUILD_CHECK Skip local nix build validation (set to 1)
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
$0 lazyworkhorse # deploy master/switch
|
|
||||||
$0 cyt-pi feat/test boot # deploy feature branch, boot
|
|
||||||
$0 uConsole master build # just build, no remote
|
|
||||||
NO_BUILD_CHECK=1 $0 uConsole # skip build check
|
|
||||||
|
|
||||||
EOF
|
|
||||||
exit 0
|
|
||||||
}
|
|
||||||
|
|
||||||
# ── Argument parsing ───────────────────────────────────────────────────
|
|
||||||
HOSTNAME="${1:-}"
|
|
||||||
BRANCH="${2:-master}"
|
|
||||||
ACTION="${3:-switch}"
|
|
||||||
NO_BUILD_CHECK="${NO_BUILD_CHECK:-0}"
|
|
||||||
|
|
||||||
if [ "$HOSTNAME" = "--help" ] || [ "$HOSTNAME" = "-h" ] || [ -z "$HOSTNAME" ]; then
|
|
||||||
show_usage
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── Host configuration ─────────────────────────────────────────────────
|
|
||||||
case "$HOSTNAME" in
|
|
||||||
lazyworkhorse)
|
|
||||||
DEFAULT_SSH_USER="ai-worker"
|
|
||||||
DEFAULT_SSH_PORT="2424"
|
|
||||||
ARCH="x86_64-linux"
|
|
||||||
;;
|
|
||||||
cyt-pi)
|
|
||||||
DEFAULT_SSH_USER="gortium"
|
|
||||||
DEFAULT_SSH_PORT="19999"
|
|
||||||
ARCH="aarch64-linux"
|
|
||||||
;;
|
|
||||||
uConsole)
|
|
||||||
DEFAULT_SSH_USER="gortium"
|
|
||||||
DEFAULT_SSH_PORT="22"
|
|
||||||
ARCH="aarch64-linux"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
error "Unknown host: $HOSTNAME"
|
|
||||||
echo "Supported hosts: lazyworkhorse, cyt-pi, uConsole"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
SSH_USER="${SSH_USER:-$DEFAULT_SSH_USER}"
|
|
||||||
SSH_PORT="${SSH_PORT:-$DEFAULT_SSH_PORT}"
|
|
||||||
SSH_KEY="${SSH_KEY:-/opt/data/home/.ssh/id_hermes_gitea}"
|
|
||||||
BUILD_HOST="${BUILD_HOST:-$HOSTNAME}"
|
|
||||||
|
|
||||||
SSH_OPTS="-p $SSH_PORT -i $SSH_KEY -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
|
|
||||||
SSH_TARGET="${SSH_USER}@${HOSTNAME}"
|
|
||||||
export GIT_SSH_COMMAND="ssh -i $SSH_KEY -p 2222 -o StrictHostKeyChecking=no"
|
|
||||||
export PATH="/nix/var/nix/profiles/default/bin:$PATH"
|
|
||||||
|
|
||||||
# ── Banner ─────────────────────────────────────────────────────────────
|
|
||||||
echo "╔══════════════════════════════════════════════╗"
|
|
||||||
echo "║ NixOS Remote Deployment ║"
|
|
||||||
echo "╚══════════════════════════════════════════════╝"
|
|
||||||
info "Host: $HOSTNAME ($ARCH)"
|
|
||||||
info "Branch: $BRANCH"
|
|
||||||
info "Action: $ACTION"
|
|
||||||
info "SSH: ${SSH_USER}@${HOSTNAME}:${SSH_PORT}"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# ── Pre-flight checks ─────────────────────────────────────────────────
|
|
||||||
step "Pre-flight checks"
|
|
||||||
|
|
||||||
# 1. Check required tools
|
|
||||||
for cmd in nix git ssh; do
|
|
||||||
if ! command -v "$cmd" &>/dev/null; then
|
|
||||||
error "Required tool not found: $cmd"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
ok "Required tools available (nix, git, ssh)"
|
|
||||||
|
|
||||||
# 2. Check infra repo
|
|
||||||
INFRA_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
|
||||||
if [ ! -d "$INFRA_DIR/.git" ]; then
|
|
||||||
error "Not a git repository: $INFRA_DIR"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
ok "Infra repo found at $INFRA_DIR"
|
|
||||||
|
|
||||||
# 3. Check SSH connectivity (skip for build-only actions)
|
|
||||||
if [ "$ACTION" != "build" ]; then
|
|
||||||
if ssh $SSH_OPTS -o ConnectTimeout=5 "$SSH_TARGET" "echo connected" &>/dev/null; then
|
|
||||||
ok "SSH connectivity to $HOSTNAME verified"
|
|
||||||
else
|
|
||||||
warn "Cannot reach $HOSTNAME via SSH — deployment step will fail later"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── Git sync ───────────────────────────────────────────────────────────
|
|
||||||
step "Git sync"
|
|
||||||
|
|
||||||
cd "$INFRA_DIR"
|
|
||||||
|
|
||||||
# Stash local changes if any
|
|
||||||
if ! git diff --quiet HEAD; then
|
|
||||||
warn "Local changes detected, stashing..."
|
|
||||||
git stash push -m "auto-stash before deploy $(date -Iseconds)"
|
|
||||||
STASHED=1
|
|
||||||
else
|
|
||||||
STASHED=0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Fetch and checkout
|
|
||||||
git fetch origin "$BRANCH" 2>/dev/null || git fetch origin master
|
|
||||||
if git rev-parse --verify "origin/$BRANCH" &>/dev/null 2>&1; then
|
|
||||||
# Remote branch exists — fast-forward merge
|
|
||||||
git checkout -B "$BRANCH" "origin/$BRANCH"
|
|
||||||
elif git rev-parse --verify "$BRANCH" &>/dev/null 2>&1; then
|
|
||||||
# Local branch or tag
|
|
||||||
git checkout "$BRANCH"
|
|
||||||
else
|
|
||||||
error "Branch/tag not found: $BRANCH"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
ok "Checked out $BRANCH ($(git rev-parse --short HEAD))"
|
|
||||||
|
|
||||||
# Update submodules
|
|
||||||
if [ -f .gitmodules ]; then
|
|
||||||
git submodule update --init --recursive
|
|
||||||
ok "Submodules updated"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── Build validation ──────────────────────────────────────────────────
|
|
||||||
if [ "$NO_BUILD_CHECK" != "1" ]; then
|
|
||||||
step "Build validation"
|
|
||||||
info "Building nixosConfigurations.$BUILD_HOST (no link)..."
|
|
||||||
|
|
||||||
if nix build --no-link --print-build-logs \
|
|
||||||
".#nixosConfigurations.${BUILD_HOST}.config.system.build.toplevel" 2>&1; then
|
|
||||||
ok "Build succeeded for $BUILD_HOST"
|
|
||||||
else
|
|
||||||
error "Build failed for $BUILD_HOST"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
warn "Build check skipped (NO_BUILD_CHECK=1)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ── Deployment ─────────────────────────────────────────────────────────
|
|
||||||
if [ "$ACTION" = "build" ]; then
|
|
||||||
step "Build complete (no deployment)"
|
|
||||||
info "Use one of: switch, boot, test, dry-activate to deploy"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
step "Deployment ($ACTION)"
|
|
||||||
|
|
||||||
# Build the nixos-rebuild command
|
|
||||||
case "$ACTION" in
|
|
||||||
switch|boot|test)
|
|
||||||
nixos-rebuild "$ACTION" \
|
|
||||||
--flake ".#$HOSTNAME" \
|
|
||||||
--target-host "$SSH_TARGET" \
|
|
||||||
--build-host "localhost" \
|
|
||||||
--use-remote-sudo \
|
|
||||||
--max-jobs 4
|
|
||||||
;;
|
|
||||||
dry-activate)
|
|
||||||
nixos-rebuild dry-activate \
|
|
||||||
--flake ".#$HOSTNAME" \
|
|
||||||
--target-host "$SSH_TARGET" \
|
|
||||||
--build-host "localhost" \
|
|
||||||
--use-remote-sudo
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
error "Unknown action: $ACTION"
|
|
||||||
echo "Valid actions: switch, boot, test, build, dry-activate"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
# ── Check result ───────────────────────────────────────────────────────
|
|
||||||
DEPLOY_EXIT=$?
|
|
||||||
if [ $DEPLOY_EXIT -eq 0 ]; then
|
|
||||||
echo ""
|
|
||||||
ok "Deployment to $HOSTNAME ($ACTION) completed successfully"
|
|
||||||
case "$ACTION" in
|
|
||||||
switch|test)
|
|
||||||
info "Configuration is now active"
|
|
||||||
;;
|
|
||||||
boot)
|
|
||||||
info "Configuration will activate on next reboot"
|
|
||||||
;;
|
|
||||||
dry-activate)
|
|
||||||
info "Dry-run complete — no changes applied"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
else
|
|
||||||
error "Deployment failed with exit code $DEPLOY_EXIT"
|
|
||||||
exit $DEPLOY_EXIT
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "╔══════════════════════════════════════════════╗"
|
|
||||||
echo "║ Deployment Complete ║"
|
|
||||||
echo "╚══════════════════════════════════════════════╝"
|
|
||||||
info "Host: $HOSTNAME"
|
|
||||||
info "Branch: $BRANCH ($(git rev-parse --short HEAD))"
|
|
||||||
info "Action: $ACTION"
|
|
||||||
info "Time: $(date -Iseconds)"
|
|
||||||
Reference in New Issue
Block a user