Health Checks, Metrics & Dependency Monitoring

Overview

webhook-go takes the heaviest traffic of any service in the platform, and LINE will not retry if we answer slowly. The observability layer is therefore built so that Kubernetes can pull an unready pod out of the load balancer quickly, and so that a pod whose dependency has been down for too long kills itself and gets restarted.

Three principles govern the design:

1. Liveness is separate from readiness. /livez only checks that the process is alive; it must never depend on a backend, or a Redis outage would have Kubernetes killing the entire fleet. /readyz returns 503 until every enabled dependency passes its ping and the pod is not draining.
2. /metrics and pprof live exclusively on the admin port :9100 and must never be exposed through the ingress.
3. Health endpoints exist on both ports, including the business port, because some ALB and Kubernetes probes target it directly.

Business Flow

Health endpoints

Endpoint	Business port (APP_PORT, default 6570)	Admin port (:9100)	Meaning
/livez	✅	✅	The process is alive; dependencies are not checked
/healthz	✅	✅	Identical to /livez
/readyz	✅	✅	Returns 503 until every dependency pings successfully and the pod is not draining
/metrics	❌	✅	Prometheus format via the VictoriaMetrics client
/version	❌	✅	Service name and version
/debug/pprof/*	❌	✅ when PPROF_ENABLED	Profiling
GET /api/health	✅	—	Terminus-style response, for NestJS parity

Both ports read from the same health.Checker, constructed exactly once in app.New.

The dependency contract

Every backend implements a four-method interface — Name, Connect, Ping, and Close. The deps.Registry then connects each one at boot (duplicate names fail fast), converts Ping into a health.Probe shared by readiness and the supervisor, and closes everything in reverse order at shutdown.

The api role registers three groups: the sqlx pool (deps/sqldb), every named Redis client (deps/redisclient), and the AMQP publisher (deps/amqppub). A dependency activates automatically once its host or URL environment variable is set (TYPEORM_HOST, REDIS_URL, AMQP_URLS). Leave it unset and the dependency is neither registered nor checked, while the service still boots normally.

What happens when a dependency fails (the supervisor)

1. Retry every 5 seconds for the first minute.
2. Retry every 10 seconds thereafter.
3. After one minute — or after the configured number of consecutive failures — fire an alert to the chat webhook. ALERT_WEBHOOK_URL supports both Google Chat and Slack; leaving it unset makes alerting a no-op.
4. After three minutes, alert again, cancel the root context for a gentle drain, and call os.Exit(1) once the grace period expires, so Kubernetes restarts the pod.

Every threshold is tunable through DEP_-prefixed environment variables, and the decision itself lives in the pure function supervisor.evaluate, which is unit tested.

Metrics collected

• RecordHTTP(method, route, code, duration) — per-route RED metrics, labelled by route pattern (for example /api/line/:id) rather than raw path, to keep label cardinality bounded.
• SetDepConnected(dep, up) — connection status per dependency.
• RecordDepOp(dep, op, d, err) — per-dependency operation timings.
• NewTask(name).Track(...) — business metrics.
• LogMemStats — periodic memory statistics.

Graceful shutdown

The order matters. On SIGTERM the service first calls Health.SetDraining() so that readiness fails and the load balancer withdraws the pod. It then calls srv.Shutdown(ctx) to close the HTTP server and wait for in-flight requests, bounded by SHUTDOWN_TIMEOUT. Next, app.Shutdown() waits out SHUTDOWN_DRAIN_DELAY, closes the admin port, and finally closes each dependency in reverse order.

signalx.Multiplex keeps SIGTERM and SIGINT (which mean drain) separate from SIGHUP (which means reload LOG_LEVEL).

Logging

Logs are single-line JSON written through log/slog to stdout only — fmt.Println is forbidden. Timestamps are timezone-aware, logx.Err(err) attaches a stack trace, the level can be changed at runtime via SIGHUP, and CLS fields (userId, sessionId, transactionId) come from Core HTTP & the Middleware Pipeline.

Key Files & Functions

File	Highlights
internal/health/health.go	Checker, Probe, New, Run, SetDraining
internal/health/handler.go	hello (GET /api), health (GET /api/health)
internal/health/register.go	Register(api, appName) — mounts two routes under /api
internal/obs/admin.go	ServeAdmin(addr, checker, opts, log) for :9100
internal/obs/metrics.go	RecordHTTP, RecordDepOp, SetDepConnected, NewTask, WriteMetrics
internal/obs/pprof.go	pprof, gated behind PPROF_ENABLED
internal/supervisor/supervisor.go	Start and evaluate (a tested pure function)
internal/alert/alert.go	Chat webhook delivery in {"text": ...} form
internal/signalx/signalx.go	Multiplex(ctx, cancel, reload, log)
internal/deps/registry.go, dep.go	Registry and the Dependency interface
internal/logx/logx.go	JSON logger with a dynamic level
deploy/values.yaml	The Helm contract — probes on /livez and /readyz, scraping :9100, a CPU limit of at least 1000m

Connections to Other Services

• Kubernetes — the liveness probe targets /livez and the readiness probe targets /readyz. The supervisor relies on Kubernetes to restart the pod after os.Exit(1).
• Prometheus / VictoriaMetrics — a ServiceMonitor scrapes :9100/metrics.
• Chat webhook — the outage notification destination, configured via ALERT_WEBHOOK_URL.
• Monitored dependencies — the Postgres pool, each Redis instance, and the AMQP publisher. If any one of them stays down for more than three minutes, the whole pod restarts, even if the others are healthy.

:::caution Operational warnings Never expose port :9100 through the ingress, and never add a second /metrics endpoint on the business port. :::