Kubernetes Deployment (Helm)

中文

The chart lives at deploy/helm/imcore. It deploys the websocket server plus optional persist_consumer / ai_worker / sync_worker deployments, with rolling upgrades, graceful connection draining, a PodDisruptionBudget, and an HPA. Kubernetes is one of three supported shapes — compose (deploy/compose/) and systemd (deploy/systemd/ + imctl) remain first-class; pick one per environment.

Prerequisites

Multi-instance rules that the chart cannot enforce for you:

• All replicas must share one Redis and one broker (RabbitMQ, NATS, or Redis Streams) — cross-instance fan-out, presence, and conversation state depend on them. Redis/broker/DB run outside this chart (operators, cloud services, or your own manifests).
• The AI subsystem (ai_worker, RAG, handover, call analysis) works on RabbitMQ, NATS, or Redis Streams. Redis Streams reuses the required Redis connection; production requires AOF, redis.stream.ai_claim_min_idle_ms longer than the slowest task, and a maxlen sized above the recovery-window backlog. The Helm chart still bundles RabbitMQ, not NATS, as its optional in-cluster broker (see below); Redis can be both the required cache/state service and the selected broker.
• Migrations are applied manually before first start — the image ships /app/bin/migrate (see below).
• High-throughput multi-instance: prefer NATS cluster, or RabbitMQ with quorum queues (deployment config, no code change).

Install

1. Create the config Secret. Start from config/app.yaml.example; app.yaml holds secrets, so it always lives in a Secret:

kubectl create namespace imcore
kubectl -n imcore create secret generic imcore-config --from-file=app.yaml

For the admin console to be reachable in-cluster, set admin.console.listen_addr: "0.0.0.0:7080" in app.yaml (default binds loopback) and see service.exposeAdmin.

2. Run migrations (one-off pod; repeats are safe — the runner skips applied files):

kubectl -n imcore run imcore-migrate --rm -i --restart=Never \
  --image=balalaim/imcore-basic:latest \
  --overrides='{"spec":{"containers":[{"name":"imcore-migrate","image":"balalaim/imcore-basic:latest","env":[{"name":"APP_BIN","value":"migrate"}],"volumeMounts":[{"name":"config","mountPath":"/app/config/app.yaml","subPath":"app.yaml"}]}],"volumes":[{"name":"config","secret":{"secretName":"imcore-config"}}]}}'

3. Install the chart:

helm install imcore deploy/helm/imcore -n imcore \
  --set config.existingSecret=imcore-config \
  --set persistConsumer.enabled=true

persistConsumer is required somewhere in the system for room-history persistence (binary: jetstream_consumer for NATS mode). Verify: kubectl -n imcore rollout status deploy/imcore.

Full-stack install (bundled dependencies) — PoC / offline / 信创 only

For a self-contained cluster (dev, air-gapped, 信创 单机式私有化) the chart can bundle Postgres (pgvector), Redis, RabbitMQ and MinIO as subcharts and run migrations automatically, so one command brings up the whole stack:

# 1. Create the config Secret from the bundled example (edit as needed):
cp deploy/helm/imcore/examples/app.bundled.yaml app.yaml
kubectl create secret generic imcore-config --from-file=app.yaml

# 2. Install with the bundled profile:
helm install imcore deploy/helm/imcore \
  -f deploy/helm/imcore/values-bundled.yaml \
  --set config.existingSecret=imcore-config

The bundled dependencies are single-instance and NOT highly available, run no backups, and are intended only for PoC / dev / offline / 信创 单机式私有化. For production, leave postgresql.enabled / redis.enabled / rabbitmq.enabled / minio.enabled at their default false and point app.yaml at external managed services (or the operators below). RabbitMQ is the bundled broker (not NATS) as a packaging choice — this chart has no bundled-NATS subchart, not because the AI subsystem needs it (it now runs on all three supported brokers, see Prerequisites above). Backup/restore of bundled data is out of scope for this chart — see docs/RUNBOOK_BACKUP_RESTORE.md.

Migrations: values-bundled.yaml sets migration.enabled=true, so a <release>-migrate Job applies the schema before app pods start, and each app pod gets a wait-for-migrations initContainer that blocks until that Job completes. You can turn on just this behavior against an external DB with --set migration.enabled=true, without enabling any bundled dependency.

Production dependency operators (reference)

For production HA, run the stateful dependencies via mature operators outside this chart instead of the bundled subcharts: CloudNativePG (Postgres), the RabbitMQ Cluster Operator, a Redis operator (e.g. Spotahome Redis Operator), and the MinIO Operator. Keep postgresql/redis/rabbitmq/minio.enabled=false and point app.yaml at the operator-provisioned endpoints.

Key values

Rolling upgrades and graceful drain

How a rollout stays hitless, matching the shutdown path in cmd/websocket/main.go:

1. maxSurge: 1, maxUnavailable: 0 — a new pod must be Ready (/healthz) before any old pod is asked to stop.
2. The pod is removed from Service endpoints; preStop sleeps preStopSleepSeconds (5s) so the LB/kube-proxy deprograms it while it still serves.
3. SIGTERM: DrainClients(3s) notifies connected WebSocket clients (they reconnect to surviving pods and catch up), then HTTP/gRPC shutdown (5s), audit-log flush, persister stop, broker teardown.
4. terminationGracePeriodSeconds: 30 comfortably covers 2+3; only then would SIGKILL fire.

Reconnect pressure during a rollout is bounded by pod count — clients of one pod reconnect per step, not the whole fleet. websocket.max_connections_per_instance (app.yaml) sheds excess handshakes with a 503 so the LB rebalances onto the freshly-surged pod.

helm rollback follows the same choreography in reverse order of versions.

Upgrading an existing NATS deployment to AI-on-NATS

One manual step, NATS-only, one-time: gowebsocket_persist changed from a push consumer to a pull consumer (needed so multiple jetstream_consumer replicas can actually compete — previously a second replica panicked at startup). JetStream does not allow converting an existing push consumer to pull in place, so the old one must be deleted before the new binary starts:

1. Scale jetstream_consumer to 0 (and stop any websocket pod running nats.embedded_persist_inproc: true).
2. nats consumer rm gowebsocket gowebsocket_persist
3. Roll out the new image — the consumer is recreated automatically; DeliverAll picks up from whatever is still on the stream, so nothing within the core stream’s 5-minute MaxAge is lost.

Embedded NATS (nats.embedded: true) cannot follow that order — the NATS server is the websocket pod, so once it is stopped there is nothing to run step 2 against, while nats.embedded_store_dir still holds the old push durable. The binary panics on cannot pull subscribe to push based consumer, so the pod crash-loops. Either start once with nats.embedded_persist_inproc: false, run step 2 against nats.embedded_listen, and revert; or delete nats.embedded_store_dir (costs at most 5 minutes of undelivered fan-out and no durable business state). Full procedure: docs/RUNBOOK_BACKUP_RESTORE.md §6.3.

The two new AI streams (<stream_name>_ai_work / _ai_push) are created fresh by natsjets.New() on first boot — no equivalent step for them, and no data migration is needed moving AI traffic from RabbitMQ to NATS (AI subjects carry no durable business state; that lives in the database).

PodDisruptionBudget

maxUnavailable: 1 lets node drains and cluster upgrades proceed one pod at a time instead of evicting several WebSocket pods at once (each eviction = that pod’s clients reconnecting). The template renders only with real redundancy (replicas > 1 or HPA), because a PDB over one replica deadlocks drains.

Autoscaling

CPU at 70% is the safe default. The better signal for a WebSocket server is connections per pod — export it from the built-in /metrics (imcore_current_connections) through prometheus-adapter:

# prometheus-adapter values.yaml (rules.custom)
rules:
  custom:
    - seriesQuery: 'imcore_current_connections{namespace!="",pod!=""}'
      resources: { overrides: { namespace: {resource: namespace}, pod: {resource: pod} } }
      name: { matches: "imcore_current_connections" }
      metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

# imcore values.yaml
autoscaling:
  enabled: true
  customMetrics:
    - type: Pods
      pods:
        metric: { name: imcore_current_connections }
        target: { type: AverageValue, averageValue: "8000" }  # from your capacity baseline

Set the target from a measured baseline (docs/CHAOS_TESTING.md), not a guess. The HPA’s scaleDown behavior is deliberately slow (1 pod / 2 min after a 5-min window) so long-lived connections don’t slosh between shrinking pods.

Metrics scraping

GET /metrics sits behind the same ops gate as /stat: source must be loopback or inside admin.allowed_cidrs, plus an admin token while admin.require_token is true (the default). For in-cluster Prometheus either:

• add the cluster pod CIDR to admin.allowed_cidrs and keep the token → serviceMonitor.bearerTokenSecret: {name, key}, or
• also set admin.require_token: false if the CIDR gate alone is acceptable in your threat model.

Alert rules ship in deploy/prometheus/imcore-alerts.yml.

Relationship to compose / systemd

The chart deploys imcore only. The compose stack bundles Postgres/Redis/RabbitMQ/MinIO for a single host; systemd + imctl covers bare-metal fleets. Kubernetes assumes you bring managed/operator-run dependencies. app.yaml is identical across all three — only the injection mechanism differs.