Compare commits

..

11 Commits

Author SHA1 Message Date
bcebf18676 fix: move filter into jail settings (NixOS submodule doesn't pass string filters) 2026-05-01 11:59:33 +00:00
0370d784a0 fix: http-botsearch logpath must be string, not list 2026-05-01 04:02:06 +00:00
260b2d2756 fix: restructure fail2ban jails per NixOS module - recidive in jails, settings attr, str bantime 2026-05-01 03:59:32 +00:00
2477acdfc7 fix: services.fail2ban top-level options - no findtime, maxretry lowercase 2026-05-01 03:57:21 +00:00
81c25d3f20 fix: use security.auditd instead of services.auditd 2026-05-01 03:55:09 +00:00
9b1f467db9 fix: remove invalid networking.firewall.defaultAllow option 2026-05-01 03:52:57 +00:00
65fa778b2b fix: add custom traefik fail2ban filters for http-auth and http-botsearch jails 2026-05-01 03:40:59 +00:00
5d3bbe99f3 chore: update compose submodule for traefik access logs 2026-05-01 03:33:34 +00:00
3e04ccc1e8 security: remove deployment commands from ai-worker sudo rules
ai-worker only needs security audit commands, not deployment access.

Removed:
- nh os switch
- nixos-rebuild switch

Kept:
- Firewall checks (iptables)
- Fail2ban status
- Log inspection (journalctl)
- SSH config (sshd -T)
- Docker service checks
- Network diagnostics
2026-04-30 17:46:39 +00:00
21bd4bb283 security: add restricted sudo for ai-worker with security audit commands
- Deployment: nh os switch, nixos-rebuild switch (flake path locked)
- Firewall checks: iptables -L, iptables -S
- Fail2ban: status, banned IPs
- Logs: journalctl for kernel and fail2ban
- SSH config: sshd -T for verification
- Docker: ps, inspect (service health)
- Network: ss -tlnp, /proc/net/tcp

All commands are whitelisted with NOPASSWD.
No shell access, no ALL command - principle of least privilege.
2026-04-30 17:46:39 +00:00
7994aad8d8 security: harden lazyworkhorse with firewall, fail2ban, SSH hardening
- Firewall (default deny):
  - Allow only essential ports: SSH(2424), Gitea(2222), HTTP(80), HTTPS(443)
  - Rate limit SSH (max 4 new connections/60s)
  - Rate limit HTTP/HTTPS (25/minute)
  - Drop invalid packets, log dropped packets

- Fail2ban (auto-ban attackers):
  - SSH jail: 3 strikes = 1 hour ban
  - HTTP auth failures: 5 strikes = 1 hour ban
  - HTTP scanning: 2 strikes = 2 hour ban
  - Recidive jail: repeat offenders = 1 week ban

- SSH hardening:
  - No root login
  - Max 3 auth tries, 5 sessions
  - 30s login grace time
  - No X11/TCP/agent forwarding
  - Verbose logging

- Kernel network hardening:
  - SYN flood protection (syncookies)
  - IP spoofing protection (rp_filter)
  - Disable source routing, redirects
  - Log martian packets
  - Connection tuning for high load

- Audit logging enabled

Ports commented for review (likely internal-only):
- 8000 (Portainer), 4242 (Coms), 5000/8087/8089 (TAK)
2026-04-30 17:46:39 +00:00
10 changed files with 266 additions and 614 deletions

View File

@@ -5,6 +5,7 @@ This document outlines the development conventions for this NixOS-based infrastr
## Build & Deployment
- **Build/Deploy:** Use `nixos-rebuild switch --flake .#<hostname>` to build and deploy the configuration for a specific host.
- **CRITICAL — Validate before pushing:** Always `nix build --no-link '.#nixosConfigurations.<hostname>.config.system.build.toplevel'` (or `nh os build`) and confirm it succeeds before pushing any changes. Never push untested NixOS configs.
- **Development Shell:** Activate the development environment with `nix develop`.
## Linting & Formatting

View File

@@ -1,203 +0,0 @@
# AI Model Optimization Cron Job - EXECUTION PROMPT
**When this cron runs, follow these instructions exactly:**
---
## Your Role
You are an AI model optimization agent. Your task is to find the best ollama/llama.cpp configuration for maximum context size and hardware utilization.
**Hardware:**
- 2× AMD MI50 GPUs (32GB VRAM each, 64GB total)
- 128GB system RAM
- ROCm: HSA_OVERRIDE_GFX_VERSION=9.0.6, HIP_VISIBLE_DEVICES=0,1
---
## File Locations
```
STATE: /opt/data/infra/assets/ai-optimizer/state.json
RESULTS: /opt/data/infra/assets/ai-optimizer/results.csv
INFRA_REPO: /opt/data/infra
```
---
## Model Queues
### GPU Track (Coding - prioritize speed + context on GPU)
1. `devstral-small-2:24b`
2. `qwen2.5-coder:32b`
3. `codellama:34b-instruct`
### RAM Track (Knowledge - prioritize max context)
1. `qwen2.5:72b`
2. `nemotron-3-nano:30b`
3. `mixtral:8x7b-instruct`
---
## Context Steps (in order)
```
[32768, 65536, 98304, 131072, 163840, 200704, 262144, 327680]
```
---
## Each Run - Step by Step
### 1. Read State
```bash
cd /opt/data/infra
cat assets/ai-optimizer/state.json
```
### 2. Determine Next Test
- Read `track` (gpu or ram)
- Read `current_model` from queue at `model_index`
- Read `current_config` for parameters to test
- Select next context step from `context_steps` based on `phase`
### 3. Pull Model (if needed)
```bash
docker exec ollama ollama list | grep -q "<model>" || docker exec ollama ollama pull <model>
```
### 4. Create Test Modelfile
```bash
docker exec ollama bash -c "cat <<EOF > /root/.ollama/test_${model}.modelfile
FROM ${model}
PARAMETER num_ctx ${current_config.num_ctx}
PARAMETER num_gpu ${current_config.num_gpu}
PARAMETER flash_attn ${current_config.flash_attn}
PARAMETER num_predict 4096
PARAMETER num_keep 1024
PARAMETER repeat_penalty 1.1
EOF"
docker exec ollama ollama create test-model -f /root/.ollama/test_${model}.modelfile
```
### 5. Run Benchmark
```bash
# Warm up
docker exec ollama ollama run test-model "Hello" > /dev/null
# Coding prompt
START=$(date +%s%N)
docker exec ollama ollama run test-model "Write a Python async context manager that retries a function with exponential backoff, max 5 retries, and logs each attempt using structlog. Include type hints."
END=$(date +%s%N)
# Calculate tokens/sec from output
```
### 6. Measure VRAM (if possible)
```bash
# Try host first
rocm-smi --showmeminfo vram 2>/dev/null || \
# Try via docker
docker exec --privileged ollama rocm-smi --showmeminfo vram 2>/dev/null || \
# Fallback
echo "VRAM measurement unavailable"
```
### 7. Record Results
- Parse tokens/sec from ollama output
- Record VRAM/RAM usage
- Determine if this is best config so far for this model
- Update `best_configs` if tokens/sec improved or context increased
### 8. Update State
```python
# Logic:
if test_successful:
if context_step < max_reached:
phase = "context_scaling"
current_config.num_ctx = next_context_step
else:
# Move to next model
model_index += 1
phase = "context_scaling"
current_config.num_ctx = context_steps[0]
else:
# OOM or error - record last good as best
best_configs[track][current_model] = last_good_config
model_index += 1
phase = "context_scaling"
```
### 9. Commit to Repo
```bash
cd /opt/data/infra
git add assets/ai-optimizer/
git commit -m "ai-optimizer: tested ${model} at ${num_ctx} ctx - ${status}"
git push origin master
```
### 10. Matrix Notification (if available)
```python
import os
if os.getenv("MATRIX_HOME_SERVER") and os.getenv("MATRIX_ACCESS_TOKEN"):
# Send notification to Matrix room
# Room ID from env or config
pass
# Else: silent
```
---
## Stop Conditions
1. All models in both queues have `best_configs` recorded
2. Manual intervention needed (error in state.json `error` field)
3. No progress for 3 consecutive runs (stuck)
---
## Error Handling
If any step fails:
1. Log error to state.json: `"error": {"message": "...", "timestamp": "..."}`
2. Do NOT increment model_index (retry next run)
3. Commit state with error field
4. Exit gracefully
---
## Important Notes
- **No num_parallel**: Do not use this parameter
- **Two tracks**: Complete GPU track first, then RAM track
- **Backend**: Start with ollama, llama.cpp testing is optional (requires uncommenting in compose.yml)
- **Host access**: Some commands need host - use docker exec or SSH if available
- **Ask before deploy**: If config changes needed in NixOS modules, show diff and wait for user confirmation before `nh os switch`
---
## Example State Transitions
**Start:**
```json
{"track": "gpu", "model_index": 0, "current_model": "devstral-small-2:24b", "current_config": {"num_ctx": 32768, ...}}
```
**After successful test at 32k:**
```json
{"track": "gpu", "model_index": 0, "current_model": "devstral-small-2:24b", "current_config": {"num_ctx": 65536, ...}}
```
**After OOM at 131k:**
```json
{
"track": "gpu",
"model_index": 1,
"current_model": "qwen2.5-coder:32b",
"best_configs": {
"gpu": {
"devstral-small-2:24b": {"num_ctx": 98304, "num_gpu": 99, "tokens_per_sec": 11.2}
}
}
}
```

View File

@@ -1,283 +0,0 @@
# AI Model Optimization Cron Job
**Goal:** Find optimal configurations for maximum context size with full hardware utilization.
**Hardware:**
- 2× AMD MI50 GPUs (32GB VRAM each, 64GB total)
- 128GB system RAM
- ROCm: HSA_OVERRIDE_GFX_VERSION=9.0.6, HIP_VISIBLE_DEVICES=0,1
---
## Model Queue
### GPU-Optimized (Coding - prioritize speed + context on GPU)
1. `devstral-small-2:24b` - Best coding model
2. `qwen2.5-coder:32b` - Strong coder, fits on GPU+offload
3. `codellama:34b-instruct` - Legacy but solid
### RAM-Optimized (Knowledge - prioritize max context, accept slower)
1. `qwen2.5:72b` - Best knowledge, needs heavy offload
2. `nemotron-3-nano:30b` - Good general knowledge
3. `mixtral:8x7b-instruct` - MoE, efficient for knowledge
---
## Optimization Strategy
**Two separate tracks:**
### Track A: GPU-Focused (Coding)
```
Baseline: num_ctx=32768, num_gpu=99, flash_attn=true
Steps:
1. Increase context: 32k → 65k → 98k → 131k → 163k
2. At each step, verify VRAM usage < 60GB (leave headroom)
3. If OOM: reduce num_gpu until stable, record best
4. Measure tokens/sec - if < 5 tok/s, consider context too high
```
### Track B: RAM-Focused (Knowledge)
```
Baseline: num_ctx=65536, num_gpu=50, flash_attn=true
Steps:
1. Increase context: 65k → 131k → 200k → 262k → 327k
2. Allow heavy RAM offload (system RAM up to 100GB)
3. If OOM: reduce context or num_gpu
4. Speed less critical - focus on max stable context
```
---
## Backend-Specific Configs
### Ollama (Modelfile parameters)
```
PARAMETER num_ctx <value>
PARAMETER num_gpu <layers>
PARAMETER flash_attn true/false
PARAMETER num_predict 4096
PARAMETER num_keep 1024
PARAMETER repeat_penalty 1.1
```
### Llama.cpp (CLI flags)
```
--ctx-size <value>
--n-gpu-layers <layers>
--flash-attn on/off
--n-predict 4096
--batch-size 4096
--ubatch-size 512
--cache-type-k f16
--cache-type-v f16
--split-mode layer
--no-mmap
```
---
## Host Test Instructions
**The cron runs inside the hermes container. Some tests require host access:**
### 1. VRAM Monitoring (HOST)
```bash
# Run on host to check VRAM usage during/after benchmark
sudo rocm-smi --showmeminfo vram
# Or via docker exec if rocm-smi available in container
docker exec --privileged ollama rocm-smi --showmeminfo vram
```
### 2. Running Ollama Benchmarks (CONTAINER)
```bash
# Pull model
docker exec ollama ollama pull <model>
# Create custom modelfile
docker exec ollama bash -c 'cat <<EOF > /root/.ollama/test.modelfile
FROM <model>
PARAMETER num_ctx 65536
PARAMETER num_gpu 99
PARAMETER flash_attn true
EOF'
# Create model from modelfile
docker exec ollama ollama create test-model -f /root/.ollama/test.modelfile
# Run benchmark (warm model first)
docker exec ollama ollama run test-model "Write a Python async context manager with exponential backoff"
# Cleanup
docker exec ollama ollama rm test-model
```
### 3. Running Llama.cpp Benchmarks (CONTAINER - needs llama.cpp container)
```bash
# Uncomment llama_cpp_devstral in compose.yml first
# Then rebuild: sudo nh os switch --flake .#lazyworkhorse
# Test via HTTP API
curl http://localhost:8300/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "devstral-2-small-llama_cpp",
"prompt": "Write a Python function",
"max_tokens": 100
}'
```
### 4. Deploying Changes (HOST via ai-worker)
```bash
# After optimization, commit results
cd /home/ai-worker/infra
git add assets/ai-optimizer/
git commit -m "ai-optimizer: new best config for <model>"
git push
# If config changes needed in ollama_init_custom_models.nix:
# 1. Edit the file
# 2. nixpkgs-fmt .
# 3. Show diff to user
# 4. Wait for confirmation
# 5. sudo nh os switch --flake .#lazyworkhorse
```
### 5. Accessing Host from Hermes Container
```bash
# SSH to host as ai-worker (key should be mounted)
ssh -i /path/to/key ai-worker@host.docker.internal
# Or via docker socket if mounted
# (not recommended for security)
```
---
## Benchmark Prompts
### Coding (Track A)
```
"Write a Python async context manager that retries a function with exponential backoff, max 5 retries, and logs each attempt using structlog. Include type hints and error handling."
```
### Knowledge (Track B)
```
"Explain the complete memory hierarchy in modern GPUs, from registers through L1/L2 caches to VRAM, and how data moves between them during matrix multiplication. Include bandwidth considerations for each level."
```
### Measurement
- Tokens per second (generation speed)
- Time to first token (latency)
- VRAM usage (via rocm-smi)
- System RAM usage (via free -h)
- Context success (did it complete without OOM?)
---
## State File Structure
`/opt/data/infra/assets/ai-optimizer/state.json`
```json
{
"track": "gpu",
"current_model": "devstral-small-2:24b",
"model_index": 0,
"phase": "context_scaling",
"backend": "ollama",
"current_config": {
"num_ctx": 65536,
"num_gpu": 99,
"flash_attn": true
},
"best_configs": {
"gpu": {
"devstral-small-2:24b": {
"backend": "ollama",
"num_ctx": 131072,
"num_gpu": 99,
"flash_attn": true,
"tokens_per_sec": 12.5,
"vram_used_gb": 58.2,
"tested_at": "2026-04-28T17:00:00Z"
}
},
"ram": {}
},
"completed_models": [],
"gpu_queue": ["devstral-small-2:24b", "qwen2.5-coder:32b", "codellama:34b-instruct"],
"ram_queue": ["qwen2.5:72b", "nemotron-3-nano:30b", "mixtral:8x7b-instruct"]
}
```
---
## Results CSV
`/opt/data/infra/assets/ai-optimizer/results.csv`
```csv
timestamp,track,model,backend,phase,num_ctx,num_gpu,flash_attn,tokens_per_sec,vram_gb,ram_gb,status,is_best
2026-04-28T17:00:00Z,gpu,devstral-small-2:24b,ollama,context_scaling,65536,99,true,15.2,52.1,18.4,success,false
```
---
## Cron Job Flow
```
1. Read state.json
2. If both queues empty → STOP (all models tested)
3. Select next model from current track queue
4. Pull model if needed (docker exec ollama ollama pull)
5. Create Modelfile / llama.cpp config with current test params
6. Run benchmark (both prompts)
7. Measure: tokens/sec, VRAM (rocm-smi), RAM (free -h)
8. If successful:
- Increase context (next step)
- Update current_config in state
9. If OOM/error:
- Record last good config as best_configs[track][model]
- Move to next model in queue
10. Update state.json
11. Append to results.csv
12. Git commit + push to /opt/data/infra
13. Send Matrix notification if available, else silent
```
---
## Matrix Notification (Optional)
```python
# If matrix credentials available in environment
if os.getenv("MATRIX_HOME_SERVER") and os.getenv("MATRIX_ACCESS_TOKEN"):
# Send completion notification
# Room: !ai-optimizer:lazyworkhorse.net (or similar)
pass
# Else: silent, just commit
```
---
## Files to Create
```
/opt/data/infra/assets/ai-optimizer/
├── state.json # Current progress
├── results.csv # All test results
├── best_configs.json # Final best configs (human-readable)
└── CRON_JOB_DRAFT.md # This file
```
---
## Notes
- **No num_parallel**: Removed to avoid limiting other settings
- **Two tracks**: GPU (coding/speed) vs RAM (knowledge/context)
- **Both backends**: Test ollama first, then llama.cpp if available
- **Host tests**: rocm-smi must run on host or privileged container
- **Deploy**: ai-worker has sudo for nh/nixos-rebuild, must ask user first

View File

@@ -1 +0,0 @@
timestamp,track,model,backend,phase,num_ctx,num_gpu,flash_attn,tokens_per_sec,vram_gb,ram_gb,status,is_best
1 timestamp track model backend phase num_ctx num_gpu flash_attn tokens_per_sec vram_gb ram_gb status is_best

View File

@@ -1,21 +0,0 @@
{
"track": "gpu",
"current_model": "devstral-small-2:24b",
"model_index": 0,
"phase": "context_scaling",
"backend": "ollama",
"current_config": {
"num_ctx": 32768,
"num_gpu": 99,
"flash_attn": true
},
"best_configs": {
"gpu": {},
"ram": {}
},
"completed_models": [],
"gpu_queue": ["devstral-small-2:24b", "qwen2.5-coder:32b", "codellama:34b-instruct"],
"ram_queue": ["qwen2.5:72b", "nemotron-3-nano:30b", "mixtral:8x7b-instruct"],
"context_steps": [32768, 65536, 98304, 131072, 163840, 200704, 262144, 327680],
"last_updated": "2026-04-28T17:00:00Z"
}

View File

@@ -18,7 +18,10 @@ RUN apt-get update && \
curl poppler-utils imagemagick \
chromium xvfb fonts-noto-color-emoji fonts-unifont fonts-liberation fonts-ipafont-gothic fonts-wqy-zenhei fonts-tlwg-loma-otf fonts-freefont-ttf \
libasound2t64 libatk-bridge2.0-0t64 libatk1.0-0t64 libatspi2.0-0t64 libcairo2 libcups2t64 libdbus-1-3 libdrm2 libgbm1 libglib2.0-0t64 libnspr4 libnss3 libpango-1.0-0 libx11-6 libxcb1 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxkbcommon0 libxrandr2 \
texlive-latex-base texlive-latex-extra texlive-fonts-recommended texlive-xetex texlive-science && \
texlive-latex-base texlive-latex-extra texlive-fonts-recommended texlive-xetex texlive-science \
qemu-user-static binfmt-support qemu-user-binfmt \
emacs-nox \
libportaudio2 && \
rm -rf /var/lib/apt/lists/*
# Non-root user for runtime; UID can be overridden via HERMES_UID at runtime
@@ -57,7 +60,8 @@ RUN chmod -R a+rX /opt/hermes
# ---------- Python virtualenv ----------
RUN uv venv && \
uv pip install --no-cache-dir -e ".[all]"
uv pip install --no-cache-dir -e ".[all]" && \
uv pip install --no-cache-dir sounddevice numpy faster-whisper
# ---------- Runtime ----------
ENV HERMES_WEB_DIST=/opt/hermes/hermes_cli/web_dist

View File

@@ -1,102 +0,0 @@
#!/bin/bash
# Docker/Podman entrypoint: bootstrap config files into the mounted volume, then run hermes.
set -e
HERMES_HOME="${HERMES_HOME:-/opt/data}"
INSTALL_DIR="/opt/hermes"
# --- Privilege dropping via gosu ---
# When started as root (the default for Docker, or fakeroot in rootless Podman),
# optionally remap the hermes user/group to match host-side ownership, fix volume
# permissions, then re-exec as hermes.
if [ "$(id -u)" = "0" ]; then
if [ -n "$HERMES_UID" ] && [ "$HERMES_UID" != "$(id -u hermes)" ]; then
echo "Changing hermes UID to $HERMES_UID"
usermod -u "$HERMES_UID" hermes
fi
if [ -n "$HERMES_GID" ] && [ "$HERMES_GID" != "$(id -g hermes)" ]; then
echo "Changing hermes GID to $HERMES_GID"
# -o allows non-unique GID (e.g. macOS GID 20 "staff" may already exist
# as "dialout" in the Debian-based container image)
groupmod -o -g "$HERMES_GID" hermes 2>/dev/null || true
fi
# Fix ownership of the data volume. When HERMES_UID remaps the hermes user,
# files created by previous runs (under the old UID) become inaccessible.
# Always chown -R when UID was remapped; otherwise only if top-level is wrong.
actual_hermes_uid=$(id -u hermes)
needs_chown=false
if [ -n "$HERMES_UID" ] && [ "$HERMES_UID" != "10000" ]; then
needs_chown=true
elif [ "$(stat -c %u "$HERMES_HOME" 2>/dev/null)" != "$actual_hermes_uid" ]; then
needs_chown=true
fi
if [ "$needs_chown" = true ]; then
echo "Fixing ownership of $HERMES_HOME to hermes ($actual_hermes_uid)"
# In rootless Podman the container's "root" is mapped to an unprivileged
# host UID — chown will fail. That's fine: the volume is already owned
# by the mapped user on the host side.
chown -R hermes:hermes "$HERMES_HOME" 2>/dev/null || \
echo "Warning: chown failed (rootless container?) — continuing anyway"
fi
echo "Dropping root privileges"
exec gosu hermes "$0" "$@"
fi
# --- Running as hermes from here ---
source "${INSTALL_DIR}/.venv/bin/activate"
# Create essential directory structure. Cache and platform directories
# (cache/images, cache/audio, platforms/whatsapp, etc.) are created on
# demand by the application — don't pre-create them here so new installs
# get the consolidated layout from get_hermes_dir().
# The "home/" subdirectory is a per-profile HOME for subprocesses (git,
# ssh, gh, npm …). Without it those tools write to /root which is
# ephemeral and shared across profiles. See issue #4426.
mkdir -p "$HERMES_HOME"/{cron,sessions,logs,hooks,memories,skills,skins,plans,workspace,home}
# .env
if [ ! -f "$HERMES_HOME/.env" ]; then
cp "$INSTALL_DIR/.env.example" "$HERMES_HOME/.env"
fi
# config.yaml
if [ ! -f "$HERMES_HOME/config.yaml" ]; then
cp "$INSTALL_DIR/cli-config.yaml.example" "$HERMES_HOME/config.yaml"
fi
# Ensure the main config file remains accessible to the hermes runtime user
# even if it was edited on the host after initial ownership setup.
if [ -f "$HERMES_HOME/config.yaml" ]; then
chown hermes:hermes "$HERMES_HOME/config.yaml"
chmod 640 "$HERMES_HOME/config.yaml"
fi
# SOUL.md
if [ ! -f "$HERMES_HOME/SOUL.md" ]; then
cp "$INSTALL_DIR/docker/SOUL.md" "$HERMES_HOME/SOUL.md"
fi
# Sync bundled skills (manifest-based so user edits are preserved)
if [ -d "$INSTALL_DIR/skills" ]; then
python3 "$INSTALL_DIR/tools/skills_sync.py"
fi
# Final exec: two supported invocation patterns.
#
# docker run <image> -> exec `hermes` with no args (legacy default)
# docker run <image> chat -q "..." -> exec `hermes chat -q "..."` (legacy wrap)
# docker run <image> sleep infinity -> exec `sleep infinity` directly
# docker run <image> bash -> exec `bash` directly
#
# If the first positional arg resolves to an executable on PATH, we assume the
# caller wants to run it directly (needed by the launcher which runs long-lived
# `sleep infinity` sandbox containers — see tools/environments/docker.py).
# Otherwise we treat the args as a hermes subcommand and wrap with `hermes`,
# preserving the documented `docker run <image> <subcommand>` behavior.
if [ $# -gt 0 ] && command -v "$1" >/dev/null 2>&1; then
exec "$@"
fi
exec hermes "$@"

View File

@@ -158,7 +158,7 @@
settings = {
PasswordAuthentication = false;
KbdInteractiveAuthentication = false;
PermitRootLogin = "prohibit-password";
# Additional hardening settings below in SERVER HARDENING section
};
hostKeys = [
{
@@ -308,6 +308,196 @@
# Or disable the firewall altogether.
# networking.firewall.enable = false;
# =============================================================================
# SERVER HARDENING - Firewall, Fail2ban, SSH, Kernel
# =============================================================================
# Firewall - default deny, explicit allow
networking.firewall = {
# Enable firewall with default deny policy (NixOS firewall denies all by default)
enable = true;
allowPing = true;
# Only essential ports exposed to internet
allowedTCPPorts = [
2424 # SSH (non-standard port)
2222 # Gitea (version control)
80 # HTTP (Traefik redirect)
443 # HTTPS (Traefik)
# 8000 # Portainer - REVIEW: internal only?
# 4242 # Coms - REVIEW: internal only?
# 5000 # TAK API - REVIEW: internal only?
# 8087 # TAK Connect - REVIEW: internal only?
# 8089 # TAK Management - REVIEW: internal only?
];
allowedUDPPorts = [
# Add UDP ports if required
];
# Rate limiting and attack prevention
extraCommands = ''
# Rate limit SSH connections (max 4 new connections per 60 seconds)
iptables -A INPUT -p tcp --dport 2424 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 2424 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
# Rate limit HTTP/HTTPS (protects Traefik)
iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -m state --state NEW -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
# Drop invalid packets
iptables -A INPUT -m state --state INVALID -j DROP
# Log dropped packets (rate limited)
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
'';
};
# Fail2ban - automatic IP banning
services.fail2ban = {
enable = true;
maxretry = 3;
bantime = "1h";
banaction = "iptables-multiport";
jails = {
# SSH brute force protection (uses systemd journal backend)
sshd = {
enabled = true;
settings = {
filter = "sshd";
port = "2424";
maxretry = 3;
bantime = "1h";
};
};
# Recidive - ban repeat offenders for 1 week
recidive = {
enabled = true;
settings = {
filter = "recidive";
logpath = "/var/log/fail2ban.log";
bantime = "1w";
findtime = "1d";
maxretry = 3;
};
};
# HTTP authentication failures (Traefik)
http-auth = {
enabled = true;
settings = {
filter = "traefik-auth";
port = "80,443";
logpath = "/var/log/traefik/access.log";
maxretry = 5;
bantime = "1h";
};
};
# HTTP scanning/attacks (Traefik)
http-botsearch = {
enabled = true;
settings = {
filter = "traefik-botsearch";
port = "80,443";
logpath = "/var/log/traefik/access.log";
maxretry = 2;
bantime = "2h";
};
};
};
};
# Custom fail2ban filters for Traefik
environment.etc."fail2ban/filter.d/traefik-auth.conf".text = ''
[Definition]
failregex = ^<HOST> -.*"(GET|POST|HEAD|PUT|DELETE).*" (401|403) \d+.*$
ignoreregex =
'';
environment.etc."fail2ban/filter.d/traefik-botsearch.conf".text = ''
[Definition]
failregex = ^<HOST> -.*"(GET|POST|HEAD|PUT|DELETE).*" 404 \d+.*$
^<HOST> -.*"(GET|POST|HEAD|PUT|DELETE).*/(\.|wp-|php|admin|login|xmlrpc|\.env|\.git|\.aws|\.azure).*" \d+.*$
ignoreregex =
'';
# SSH hardening
services.openssh.settings = {
PermitRootLogin = "no";
MaxAuthTries = 3;
MaxSessions = 5;
LoginGraceTime = 30;
ClientAliveInterval = 300;
ClientAliveCountMax = 2;
PermitEmptyPasswords = "no";
ChallengeResponseAuthentication = "no";
UsePAM = true;
LogLevel = "VERBOSE";
X11Forwarding = false;
AllowTcpForwarding = "no";
AllowAgentForwarding = "no";
PermitTunnel = "no";
};
# Kernel network hardening
boot.kernel.sysctl = {
# IP Spoofing protection
"net.ipv4.conf.all.rp_filter" = 1;
"net.ipv4.conf.default.rp_filter" = 1;
# Ignore ICMP broadcasts
"net.ipv4.icmp_echo_ignore_broadcasts" = 1;
# Disable source routing
"net.ipv4.conf.all.accept_source_route" = 0;
"net.ipv4.conf.default.accept_source_route" = 0;
"net.ipv6.conf.all.accept_source_route" = 0;
"net.ipv6.conf.default.accept_source_route" = 0;
# Disable redirects
"net.ipv4.conf.all.send_redirects" = 0;
"net.ipv4.conf.default.send_redirects" = 0;
# SYN flood protection
"net.ipv4.tcp_syncookies" = 1;
"net.ipv4.tcp_max_syn_backlog" = 2048;
"net.ipv4.tcp_synack_retries" = 2;
"net.ipv4.tcp_syn_retries" = 5;
# Log martian packets
"net.ipv4.conf.all.log_martians" = 1;
"net.ipv4.conf.default.log_martians" = 1;
# Ignore redirects
"net.ipv4.conf.all.accept_redirects" = 0;
"net.ipv4.conf.default.accept_redirects" = 0;
"net.ipv4.conf.all.secure_redirects" = 0;
"net.ipv4.conf.default.secure_redirects" = 0;
"net.ipv6.conf.all.accept_redirects" = 0;
"net.ipv6.conf.default.accept_redirects" = 0;
# Connection tuning
"net.core.somaxconn" = 4096;
"net.core.netdev_max_backlog" = 65536;
"net.ipv4.tcp_max_orphans" = 65536;
"net.ipv4.tcp_fin_timeout" = 15;
"net.ipv4.tcp_keepalive_time" = 300;
"net.ipv4.tcp_keepalive_probes" = 5;
"net.ipv4.tcp_keepalive_intvl" = 15;
};
# Audit logging
security.auditd.enable = true;
# Fail2ban log directory
systemd.tmpfiles.rules = [
"d /var/log/fail2ban 0755 root root -"
"d /var/log/traefik 0755 root root -"
];
# Copy the NixOS configuration file and link it from the resulting system
# (/run/current-system/configuration.nix). This is useful in case you
# accidentally delete configuration.nix.

View File

@@ -11,4 +11,71 @@
];
};
users.groups.ai-worker = {};
# Restricted sudo for ai-worker - security checks only
security.sudo.extraRules = [
{
users = [ "ai-worker" ];
commands = [
# Firewall checks
{
command = "/run/wrappers/bin/sudo iptables -L -n -v";
options = [ "NOPASSWD" ];
}
{
command = "/run/wrappers/bin/sudo iptables -S";
options = [ "NOPASSWD" ];
}
# Fail2ban status
{
command = "/run/current-system/sw/bin/fail2ban-client status";
options = [ "NOPASSWD" ];
}
{
command = "/run/current-system/sw/bin/fail2ban-client status *";
options = [ "NOPASSWD" ];
}
{
command = "/run/current-system/sw/bin/fail2ban-client get * banned";
options = [ "NOPASSWD" ];
}
# Log inspection
{
command = "/run/current-system/sw/bin/journalctl -t kernel -n 100";
options = [ "NOPASSWD" ];
}
{
command = "/run/current-system/sw/bin/journalctl -u fail2ban -n 50";
options = [ "NOPASSWD" ];
}
{
command = "/run/current-system/sw/bin/journalctl -u firewall -n 50";
options = [ "NOPASSWD" ];
}
# SSH config verification
{
command = "/run/current-system/sw/bin/sshd -T";
options = [ "NOPASSWD" ];
}
# Docker service checks
{
command = "/run/current-system/sw/bin/docker ps";
options = [ "NOPASSWD" ];
}
{
command = "/run/current-system/sw/bin/docker inspect *";
options = [ "NOPASSWD" ];
}
# Network diagnostics
{
command = "/run/current-system/sw/bin/ss -tlnp";
options = [ "NOPASSWD" ];
}
{
command = "/run/current-system/sw/bin/cat /proc/net/tcp";
options = [ "NOPASSWD" ];
}
];
}
];
}