gortium/compose
1
1
Fork 0
Code Issues Pull Requests 17 Actions Packages Projects Releases Wiki Activity

feat: add llama-cpp-hermes service with ROCm 6.1 + gfx906 support #54

Merged
gortium merged 3 commits from feat/hermes43-llamacpp into master 2026-07-07 19:00:08 +00:00
Conversation 0 Commits 3 Files Changed 2 +62 -55
Hermes commented 2026-06-11 15:46:18 +00:00
Collaborator
Copy Link

Summary

Add llama-cpp-hermes service with custom ROCm 6.1 + gfx906 Docker image to serve Hermes 4.3 on dual MI50 GPUs.

Changes

ai/compose.yml

  • New service llama-cpp-hermes: Custom llama.cpp (b9596) built with ROCm 6.1 + gfx906 target. Serves Hermes 4.3-36B (GGUF Q4_K_M) on both MI50s with 160K context. ~19 t/s. Replaces the old commented-out llama_cpp_devstral config.
  • Modified ollama service: Removed ROCm/HSA env vars and GPU device mappings. Ollama now runs CPU-only for bge-m3 embeddings (Honcho integration).

ai/llama-cpp/Dockerfile

Multi-stage build:

  • Builder: rocm/dev-ubuntu-22.04:6.1.2-complete, compiles llama.cpp b9596 with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx906:xnack-
  • Runtime: ubuntu:24.04 with ROCm 6.1 runtime libs (libamdhip64, libhipblas, libdrm, libelf, libnuma)

Why

Ollama's built-in ggml-hip backend (v0.23.2) puts all layers on CPU due to a bug in the old engine. The official llama.cpp Docker image ships ROCm 7.2 which dropped gfx906 support. This custom build bridges the gap.

Context

  • Hermes 4.3 GGUF model (21.7 GB, seed_oss architecture) is bind-mounted from the existing Ollama blob cache
  • Both MI50s used: 94% + 98% VRAM at 160K context
  • Single card fallback possible with HIP_VISIBLE_DEVICES=0 and --split-mode none
## Summary Add `llama-cpp-hermes` service with custom ROCm 6.1 + gfx906 Docker image to serve Hermes 4.3 on dual MI50 GPUs. ## Changes ### `ai/compose.yml` - **New service `llama-cpp-hermes`**: Custom llama.cpp (b9596) built with ROCm 6.1 + gfx906 target. Serves Hermes 4.3-36B (GGUF Q4_K_M) on both MI50s with 160K context. ~19 t/s. Replaces the old commented-out `llama_cpp_devstral` config. - **Modified `ollama` service**: Removed ROCm/HSA env vars and GPU device mappings. Ollama now runs CPU-only for bge-m3 embeddings (Honcho integration). ### `ai/llama-cpp/Dockerfile` Multi-stage build: - Builder: `rocm/dev-ubuntu-22.04:6.1.2-complete`, compiles llama.cpp b9596 with `-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx906:xnack-` - Runtime: `ubuntu:24.04` with ROCm 6.1 runtime libs (libamdhip64, libhipblas, libdrm, libelf, libnuma) ## Why Ollama's built-in ggml-hip backend (v0.23.2) puts all layers on CPU due to a bug in the old engine. The official llama.cpp Docker image ships ROCm 7.2 which dropped gfx906 support. This custom build bridges the gap. ## Context - Hermes 4.3 GGUF model (21.7 GB, seed_oss architecture) is bind-mounted from the existing Ollama blob cache - Both MI50s used: 94% + 98% VRAM at 160K context - Single card fallback possible with `HIP_VISIBLE_DEVICES=0` and `--split-mode none`
Hermes added 1 commit 2026-06-11 15:46:19 +00:00
feat: add llama-cpp-hermes service with ROCm 6.1 + gfx906 support
Some checks failed
Build Hermes agent / build (pull_request) Has been cancelled
Details
Build ollama (gfx906) / build (pull_request) Has been cancelled
Details
3c92d93366 
- Add custom llama.cpp Dockerfile with ROCm 6.1 + gfx906 (MI50) build
- Add llama-cpp-hermes service serving Hermes 4.3 on dual MI50 GPUs
- Strip GPU devices/ROCm env from ollama service (CPU-only for embeddings)

Hermes 4.3 runs at ~19 t/s on dual MI50s with 160K context.
Hermes added 1 commit 2026-07-07 18:29:34 +00:00
fix: rename ollama→ollama-cpu, fix llama-cpp-hermes YAML indentation
Some checks failed
Build Hermes agent / build (pull_request) Has been cancelled
Details
Build ollama (gfx906) / build (pull_request) Has been cancelled
Details
80c9906757 
- Rename ollama service to ollama-cpu (CPU-only for bge-m3 embeddings)
- Fix llama-cpp-hermes indented under networks instead of as top-level service
- Update hermes OLLAMA_HOST to point to ollama-cpu

Part of PR #54 GPU/ROCm refactor
Hermes added 1 commit 2026-07-07 18:51:24 +00:00
Merge branch 'master' into feat/hermes43-llamacpp
Some checks failed
Build Hermes agent / build (pull_request) Has been cancelled
Details
Build ollama (gfx906) / build (pull_request) Has been cancelled
Details
1e290410de 
Conflicts resolved:
- hermes env: keep OLLAMA_HOST=ollama-cpu + master's new vars
- ollama→ollama-cpu rename: keep our rename over master's ollama
gortium merged commit e595381173 into master 2026-07-07 19:00:07 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Hermes gortium
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: gortium/compose#54
No description provided.
Delete Branch "feat/hermes43-llamacpp"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?