ComfyUI on k3s (bobcat)

Run ComfyUI as an on-demand GPU workload on the k3s worker bobcat, managed with ArgoCD, and using this workstation's SD 1.5 anime model library.

Goal

Offload Stable Diffusion 1.5 image generation from the workstation to the cluster's NVIDIA node, reachable at http://comfy.maus.local. The GPU node is shared with longmarch/llama-teacher, so ComfyUI is started on demand and the two swap the single GPU.

Hardware reality check

bobcat has a GTX 1060 6GB (Pascal, sm_61, driver 550 / CUDA 12.4). Of the ~6 GB, ~5.9 GB is usable after the CUDA context. That is enough for SD 1.5-class checkpoints (~2 GB) but not for the heavier models in the workstation library:

Model class	Size	GTX 1060 6GB
SD 1.5 anime checkpoints	~2 GB	works
SDXL	~6.5 GB	--lowvram only, slow
Flux / Z-Image	12-24 GB	not feasible
Wan 2.2 video	9-15 GB	not feasible
Anima (2.09B DiT + Qwen3-0.6B TE)	~5.6 GB	not feasible

Keep Flux / Z-Image / Wan / Anima on the workstation's Radeon (16 GB, ROCm).

Architecture

comfy.maus.local (Traefik ingress)
        │
        ▼
  Service comfyui:8188  ──►  Deployment comfyui (replicas 0/1)
                                  nodeSelector: bobcat
                                  runtimeClassName: nvidia
                                  nvidia.com/gpu: 1
                                  image: harbor.maus.local/library/comfyui:v1
        │
        ▼
  hostPath /srv/comfyui  (1.8 TB disk on bobcat)
    ├── models/            ← models are not in the image
    ├── output/ input/ temp/
    └── user/              ← settings + seeded workflows

The image contains only ComfyUI + addons; models and output live on /srv/comfyui. A small entrypoint.sh seeds the baked default workflows into the hostPath user directory on first start.

Repository layout

.
├── argo/app.yaml                 ArgoCD Application (source: deploy/)
├── deploy/                       Kubernetes manifests (ArgoCD sync path)
│   ├── namespace.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── ingress.yaml
├── container/                    Container image build
│   ├── Containerfile
│   ├── entrypoint.sh
│   ├── addons.txt                core custom nodes
│   ├── build.sh                  build + push to Harbor
│   └── workflows/                seeded SD 1.5 workflows
└── ansible/                      Node prep + GPU handover
    ├── inventory.ini
    ├── vars.yaml
    ├── comfyui.yaml
    └── scale.yaml

Prerequisites

• podman on this workstation, logged in to Harbor: podman login harbor.maus.local
• Ansible (ansible-core + the ansible.posix collection) and kubectl
• SSH access to bobcat, and the comfyanonymous/ComfyUI sources are cloned in the image (no local checkout required)
• The ArgoCD repo https://git.pr0.tech/rs/comfyui created and this directory pushed to it

Build and push the image

cd container
./build.sh              # builds harbor.maus.local/library/comfyui:v1 and :latest

Override when needed:

TAG=v2 COMFYUI_REF=v0.24.0 ./build.sh

After a rebuild, update the image tag in deploy/deployment.yaml (or use :latest with imagePullPolicy: Always) and let ArgoCD sync.

Prepare models on bobcat

cd ansible
ansible-playbook -i inventory.ini comfyui.yaml

This creates /srv/comfyui, downloads Kenshi (f16), Hassaku v12 and the kl-f8-anime2 VAE from Hugging Face, and rsyncs the local anime checkpoints. See ansible/vars.yaml for the exact filenames. Re-runs are idempotent.

Deploy

kubectl apply -f argo/app.yaml

ArgoCD creates the comfyui namespace and syncs deploy/. The Deployment is declared with replicas: 0; the Application uses ignoreDifferences on /spec/replicas, so a manual scale is not reverted by selfHeal.

Use

Start ComfyUI (stops llama-teacher so the GPU is free):

cd ansible
ansible-playbook -i inventory.ini scale.yaml -e state=up

Open http://comfy.maus.local. In the Workflows sidebar load one of the seeded templates:

• sd15_txt2img - 512x768, DPM++ 2M Karras, 25 steps, CFG 7
• sd15_hires - txt2img + latent upscale to 768x1152 + second pass (denoise 0.5)
• sd15_img2img - LoadImage + VAEEncode, denoise 0.6

Stop ComfyUI and give the GPU back to llama-teacher:

ansible-playbook -i inventory.ini scale.yaml -e state=down

Manual equivalent

kubectl -n longmarch scale deploy/llama-teacher --replicas=0
kubectl -n comfyui   scale deploy/comfyui      --replicas=1
# ... use it ...
kubectl -n comfyui   scale deploy/comfyui      --replicas=0
kubectl -n longmarch scale deploy/llama-teacher --replicas=1

Managing models and addons

• Models: drop files into the matching folder under /srv/comfyui/models/ on bobcat (checkpoints/, vae/, loras/, embeddings/, upscale_models/). They appear after a ComfyUI refresh; the pod does not need a restart.
• Addons: edit container/addons.txt, rebuild, push, and refresh the Deployment. ComfyUI-Manager is included for installing/updating nodes from the web UI at runtime.

Default models

Model	Source	Notes
AOM3	local	classic anime (see ansible/vars.yaml)
xenoflex	local	anime (see ansible/vars.yaml)
fiamixReboot	local	anime (see ansible/vars.yaml)
KENSHI01_Pruned_f16.safetensors	HF SweetLuna/Kenshi	semi-realistic anime
Hassaku v12	HF hanafuusen2001/Hassaku	anime (see ansible/vars.yaml)
kl-f8-anime2.ckpt	HF hakurei/waifu-diffusion-v1-4	anime VAE (optional)

Troubleshooting

• podman push fails with unauthorized - run podman login harbor.maus.local first (not committed anywhere).
• no space left on device when pushing - the image is ~11.5 GB. Harbor's harbor-registry PVC is nominally 5 Gi but is backed by maus's local-path directory (no quota enforced); check free space on maus.
• Large pushes fail with 499 / client disconnected during blob PATCH - Traefik's default entrypoint readTimeout is 60s and aborts uploads bigger than that. Set --entryPoints.websecure.transport.respondingTimeouts.readTimeout=0 and the same for web on the Traefik deployment. Pulls are unaffected.
• ImagePullBackOff - the node cannot pull from Harbor. bobcat is expected to already trust the internal CA; verify with ssh bobcat 'curl -s -o /dev/null -w "%{http_code}\n" https://harbor.maus.local/v2/'.
• CUDA OOM at 512x768 - lower resolution to 512x512, start ComfyUI with --reserve-vram 0 (edit args in deploy/deployment.yaml), or close ComfyUI previews. Do not expect SDXL to fit.
• GPU not available / pod Pending - llama-teacher still holds the GPU: kubectl -n longmarch scale deploy/llama-teacher --replicas=0.
• Workflows missing - they are seeded into /srv/comfyui/user/default/workflows; re-run ansible/comfyui.yaml or copy container/workflows/*.json there manually.
• Slow startup - 4 vCPU / 16 GB; model load and first generation take a while.

Notes

• The ingress has no authentication. Keep comfy.maus.local on the internal network only.
• Models are anime-focused. Many are released under CreativeML OpenRAIL-M; use internally and respect the individual model licenses.