fix: failed-run exit codes, hung pods, and Dockerfile Python skew #5

Merged

rs merged 5 commits from fix/reliability-defects into main

2026-09-02 21:51:58 +00:00

rs commented

2026-09-02 21:07:53 +00:00

Owner

Fixes the three top defects found in the architecture review of docs/arc42-architecture.md against the code. One commit per defect.

1. Failed runs exited 0 — failures were invisible to Kubernetes (`7902312`)

main.py caught every exception (invalid token, channel not found), logged it, and returned 0. With restartPolicy: Never the CronJob marked every run successful, so nothing could alert on failure. Ironically the only non-zero exit path was a Pushgateway outage, because metrics.push() sat outside the try block.

WaifuBot now tracks a success flag, set when a post completes
main() returns 1 on failure and logs tracebacks (exc_info=True)
metric push moved into its own try/except so a Pushgateway outage cannot crash the run

2. Exceptions escaping `on_ready` hung the pod forever (`7e201e9`)

client.close() was not in a finally, so any exception escaping taskrun() skipped it. discord.py's default on_error swallows event exceptions, so client.start() never returned and the pod hung — and the CronJob had no activeDeadlineSeconds/concurrencyPolicy, so stuck pods accumulated while new runs kept firing. Realistic escape paths: KeyError in get_img_url() on an API shape change, KeyError/JSON errors in get_tags() (only TaggerError was caught), and aiohttp.ClientError on connection failures.

on_ready closes the client in finally
image_api / tagger convert connection errors, timeouts, and unexpected response shapes into their typed errors, so they hit the existing retry / graceful-degradation paths instead of escaping
CronJob: concurrencyPolicy: Forbid + activeDeadlineSeconds: 300

3. Dockerfile build/runtime Python version skew (`f6616e0`)

Dependencies were built on python:3.14-slim but copied into distroless python3-debian13, which ships Python 3.13 — cp314-compiled C extensions (aiohttp and friends) grafted onto a 3.13 interpreter. Builder pinned to python:3.13-slim; site-packages path and PYTHONPATH updated to match.

Verification

mypy . clean
black --check clean for all changed lines (pre-existing violation on waifu/discord_bot.py:119 left untouched — it exists on main; CI's black . without --check never catches it, see honorable-mention CI defect)
Smoke tests: dead hosts and malformed payloads raise ImageApiError/TaggerError (6/6 pass); both API response shapes (url, items[0].url) still parse
End-to-end with an invalid Discord token: logs bot run failed and exits 1 (was 0)
Rendered CronJob template parses; concurrencyPolicy: Forbid and activeDeadlineSeconds: 300 present

Fixes the three top defects found in the architecture review of `docs/arc42-architecture.md` against the code. One commit per defect. ## 1. Failed runs exited 0 — failures were invisible to Kubernetes (`7902312`) `main.py` caught every exception (invalid token, channel not found), logged it, and returned `0`. With `restartPolicy: Never` the CronJob marked every run successful, so nothing could alert on failure. Ironically the only non-zero exit path was a Pushgateway outage, because `metrics.push()` sat outside the try block. - `WaifuBot` now tracks a `success` flag, set when a post completes - `main()` returns `1` on failure and logs tracebacks (`exc_info=True`) - metric push moved into its own try/except so a Pushgateway outage cannot crash the run ## 2. Exceptions escaping `on_ready` hung the pod forever (`7e201e9`) `client.close()` was not in a `finally`, so any exception escaping `taskrun()` skipped it. discord.py's default `on_error` swallows event exceptions, so `client.start()` never returned and the pod hung — and the CronJob had no `activeDeadlineSeconds`/`concurrencyPolicy`, so stuck pods accumulated while new runs kept firing. Realistic escape paths: `KeyError` in `get_img_url()` on an API shape change, `KeyError`/JSON errors in `get_tags()` (only `TaggerError` was caught), and `aiohttp.ClientError` on connection failures. - `on_ready` closes the client in `finally` - `image_api` / `tagger` convert connection errors, timeouts, and unexpected response shapes into their typed errors, so they hit the existing retry / graceful-degradation paths instead of escaping - CronJob: `concurrencyPolicy: Forbid` + `activeDeadlineSeconds: 300` ## 3. Dockerfile build/runtime Python version skew (`f6616e0`) Dependencies were built on `python:3.14-slim` but copied into distroless `python3-debian13`, which ships Python 3.13 — cp314-compiled C extensions (aiohttp and friends) grafted onto a 3.13 interpreter. Builder pinned to `python:3.13-slim`; site-packages path and `PYTHONPATH` updated to match. ## Verification - `mypy .` clean - `black --check` clean for all changed lines (pre-existing violation on `waifu/discord_bot.py:119` left untouched — it exists on `main`; CI's `black .` without `--check` never catches it, see honorable-mention CI defect) - Smoke tests: dead hosts and malformed payloads raise `ImageApiError`/`TaggerError` (6/6 pass); both API response shapes (`url`, `items[0].url`) still parse - End-to-end with an invalid Discord token: logs `bot run failed` and exits `1` (was `0`) - Rendered CronJob template parses; `concurrencyPolicy: Forbid` and `activeDeadlineSeconds: 300` present

rs added 3 commits

2026-09-02 21:07:53 +00:00