Kubernetes Deployment (Helm)

中文

The chart lives at deploy/helm/imcore. It deploys the websocket server plus optional persist_consumer / ai_worker / sync_worker deployments, with rolling upgrades, graceful connection draining, a PodDisruptionBudget, and an HPA. Kubernetes is one of three supported shapes — compose (deploy/compose/) and systemd (deploy/systemd/ + imctl) remain first-class; pick one per environment.

Prerequisites

Multi-instance rules that the chart cannot enforce for you:

• All replicas must share one Redis and one broker (RabbitMQ, NATS, or Redis Streams) — cross-instance fan-out, presence, and conversation state depend on them. Redis/broker/DB run outside this chart (operators, cloud services, or your own manifests).
• The AI subsystem (ai_worker, RAG, handover, call analysis) works on RabbitMQ, NATS, or Redis Streams. Redis Streams reuses the required Redis connection; production requires AOF, redis.stream.ai_claim_min_idle_ms longer than the slowest task, and a maxlen sized above the recovery-window backlog. The Helm chart still bundles RabbitMQ, not NATS, as its optional in-cluster broker (see below); Redis can be both the required cache/state service and the selected broker.
• Migrations are applied manually before first start — the image ships /app/bin/migrate (see below).
• High-throughput multi-instance: prefer NATS cluster, or RabbitMQ with quorum queues (deployment config, no code change).

Install

1. Create the config Secret. Start from config/app.yaml.example; app.yaml holds secrets, so it always lives in a Secret:

kubectl create namespace imcore
kubectl -n imcore create secret generic imcore-config --from-file=app.yaml

For the admin console to be reachable in-cluster, set admin.console.listen_addr: "0.0.0.0:7080" in app.yaml (default binds loopback) and see service.exposeAdmin.

2. Run migrations (one-off pod; repeats are safe — the runner skips applied files):

kubectl -n imcore run imcore-migrate --rm -i --restart=Never \
  --image=balalaim/imcore-basic:latest \
  --overrides='{"spec":{"containers":[{"name":"imcore-migrate","image":"balalaim/imcore-basic:latest","env":[{"name":"APP_BIN","value":"migrate"}],"volumeMounts":[{"name":"config","mountPath":"/app/config/app.yaml","subPath":"app.yaml"}]}],"volumes":[{"name":"config","secret":{"secretName":"imcore-config"}}]}}'

3. Install the chart:

helm install imcore deploy/helm/imcore -n imcore \
  --set config.existingSecret=imcore-config \
  --set persistConsumer.enabled=true

persistConsumer is required somewhere in the system for room-history persistence (binary: jetstream_consumer for NATS mode). Verify: kubectl -n imcore rollout status deploy/imcore.

Full-stack install (bundled dependencies) — PoC / offline / 信创 only

For a self-contained cluster (dev, air-gapped, 信创 单机式私有化) the chart can bundle Postgres (pgvector), Redis, RabbitMQ and MinIO as subcharts and run migrations automatically, so one command brings up the whole stack:

# 1. Create the config Secret from the bundled example (edit as needed):
cp deploy/helm/imcore/examples/app.bundled.yaml app.yaml
kubectl create secret generic imcore-config --from-file=app.yaml

# 2. Install with the bundled profile:
helm install imcore deploy/helm/imcore \
  -f deploy/helm/imcore/values-bundled.yaml \
  --set config.existingSecret=imcore-config

The bundled dependencies are single-instance and NOT highly available, run no backups, and are intended only for PoC / dev / offline / 信创 单机式私有化. For production, leave postgresql.enabled / redis.enabled / rabbitmq.enabled / minio.enabled at their default false and point app.yaml at external managed services (or the operators below). RabbitMQ is the bundled broker (not NATS) as a packaging choice — this chart has no bundled-NATS subchart, not because the AI subsystem needs it (it now runs on all three supported brokers, see Prerequisites above). Backup/restore of bundled data is out of scope for this chart — see docs/RUNBOOK_BACKUP_RESTORE.md.

Migrations: values-bundled.yaml sets migration.enabled=true, so a <release>-migrate Job applies the schema before app pods start, and each app pod gets a wait-for-migrations initContainer that blocks until that Job completes. You can turn on just this behavior against an external DB with --set migration.enabled=true, without enabling any bundled dependency.

Production dependency operators (reference)

For production HA, run the stateful dependencies via mature operators outside this chart instead of the bundled subcharts: CloudNativePG (Postgres), the RabbitMQ Cluster Operator, a Redis operator (e.g. Spotahome Redis Operator), and the MinIO Operator. Keep postgresql/redis/rabbitmq/minio.enabled=false and point app.yaml at the operator-provisioned endpoints.

Key values

Network isolation

Set networkPolicy.enabled=true in a hardened cluster. The chart then renders one policy for the websocket pods and, when workers are enabled, one policy for the worker pods. Each policy declares both Ingress and Egress; omitted rules are denied. The defaults allow only DNS queries to CoreDNS and worker metrics from pods in the namespace labelled kubernetes.io/metadata.name: monitoring. Add the ingress controller or trusted client namespaces/CIDRs to networkPolicy.ingress.app.from, and add PostgreSQL, Redis, the broker, object storage, and the optional license control plane to networkPolicy.egress.extraRules.

For example, this permits an ingress controller, a Prometheus namespace, and a PostgreSQL CIDR while keeping admin access limited to an operations CIDR:

networkPolicy:
  enabled: true
  ingress:
    app:
      from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
    workerObservability:
      from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
    admin:
      from:
        - ipBlock: { cidr: 10.20.0.0/16 }
  egress:
    extraRules:
      - to: [{ ipBlock: { cidr: 10.30.0.0/16 } }]
        ports: [{ protocol: TCP, port: 5432 }]

The policy does not infer dependency addresses from app.yaml, so a release with egress isolation must list every destination it needs. Validate the rendered policy with kubectl diff and an in-cluster connectivity check before cutover; a missing rule intentionally fails closed.

Rolling upgrades and graceful drain

How a rollout stays hitless, matching the shutdown path in cmd/websocket/main.go:

1. maxSurge: 1, maxUnavailable: 0 — a new pod must be Ready (/healthz) before any old pod is asked to stop.
2. The pod is removed from Service endpoints; preStop sleeps preStopSleepSeconds (5s) so the LB/kube-proxy deprograms it while it still serves.
3. SIGTERM: DrainClients(3s) notifies connected WebSocket clients (they reconnect to surviving pods and catch up); the WS listener and gRPC fanout close (5s each), periodic/push tasks drain, audit-log flush, persister stop, broker teardown. The main HTTP server — /metrics plus the already-failing /readyz — closes last, so the whole drain window stays scrapeable (fenced new requests get 503 throughout).
4. terminationGracePeriodSeconds: 30 comfortably covers 2+3; only then would SIGKILL fire. Independently of the orchestrator, every binary installs a shutdown watchdog when draining begins: a second SIGINT/SIGTERM force-exits immediately, and an internal deadline (websocket 30s; workers = drain budget + 15s) does the same — a wedged drain step can never hang the process until SIGKILL. The docker-compose stacks set stop_grace_period per service for the same reason (docker’s 10s default would SIGKILL mid-flush).

Reconnect pressure during a rollout is bounded by pod count — clients of one pod reconnect per step, not the whole fleet. websocket.max_connections_per_instance (app.yaml) sheds excess handshakes with a 503 so the LB rebalances onto the freshly-surged pod.

helm rollback follows the same choreography in reverse order of versions.

Upgrading an existing NATS deployment to AI-on-NATS

One manual step, NATS-only, one-time: gowebsocket_persist changed from a push consumer to a pull consumer (needed so multiple jetstream_consumer replicas can actually compete — previously a second replica panicked at startup). JetStream does not allow converting an existing push consumer to pull in place, so the old one must be deleted before the new binary starts:

1. Scale jetstream_consumer to 0 (and stop any websocket pod running nats.embedded_persist_inproc: true).
2. nats consumer rm gowebsocket gowebsocket_persist
3. Roll out the new image — the consumer is recreated automatically; DeliverAll picks up from whatever is still on the stream, so nothing within the core stream’s 5-minute MaxAge is lost.

Embedded NATS (nats.embedded: true) cannot follow that order — the NATS server is the websocket pod, so once it is stopped there is nothing to run step 2 against, while nats.embedded_store_dir still holds the old push durable. The binary panics on cannot pull subscribe to push based consumer, so the pod crash-loops. Either start once with nats.embedded_persist_inproc: false, run step 2 against nats.embedded_listen, and revert; or delete nats.embedded_store_dir (costs at most 5 minutes of undelivered fan-out and no durable business state). Full procedure: docs/RUNBOOK_BACKUP_RESTORE.md §6.3.

The two new AI streams (<stream_name>_ai_work / _ai_push) are created fresh by natsjets.New() on first boot — no equivalent step for them, and no data migration is needed moving AI traffic from RabbitMQ to NATS (AI subjects carry no durable business state; that lives in the database).

PodDisruptionBudget

maxUnavailable: 1 lets node drains and cluster upgrades proceed one pod at a time instead of evicting several WebSocket pods at once (each eviction = that pod’s clients reconnecting). The template renders only with real redundancy (replicas > 1 or HPA), because a PDB over one replica deadlocks drains.

Autoscaling

CPU at 70% is the safe default. The better signal for a WebSocket server is connections per pod — export it from the built-in /metrics (imcore_current_connections) through prometheus-adapter:

# prometheus-adapter values.yaml (rules.custom)
rules:
  custom:
    - seriesQuery: 'imcore_current_connections{namespace!="",pod!=""}'
      resources: { overrides: { namespace: {resource: namespace}, pod: {resource: pod} } }
      name: { matches: "imcore_current_connections" }
      metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

# imcore values.yaml
autoscaling:
  enabled: true
  customMetrics:
    - type: Pods
      pods:
        metric: { name: imcore_current_connections }
        target: { type: AverageValue, averageValue: "8000" }  # from your capacity baseline

Set the target from a measured baseline (docs/CHAOS_TESTING.md), not a guess. The HPA’s scaleDown behavior is deliberately slow (1 pod / 2 min after a 5-min window) so long-lived connections don’t slosh between shrinking pods.

Resource sizing and the Go runtime

Numbers below are from the 2026-10 phase-boundary memory profile (heap snapshots at idle / connected / load / post-disconnect under the loadtest; RSS sampled every second). Treat them as the shape and the starting formula, not as your production numbers — re-measure at your message mix (the methodology: heap profile per phase + pprof -base diffs; RSS alone lies, because the Go runtime retains freed arena pages at the high-water mark).

• Per WS connection: ~41 KiB live heap (NewClient channel buffers dominate; websocket.reliable_send_buffer is the lever), so 100k connections ≈ 4.1 GiB live.
• RSS runs 2-3x the live heap (GOGC headroom + arena retention): size limits.memory ≈ 8-12 Gi for a 100k-connection pod, requests.memory from steady-state usage.
• No post-disconnect retention was measured — heap returns to idle; goroutine hygiene is enforced by the TestMain leak gates.

GOMEMLIMIT

Go does not derive GOMEMLIMIT from limits.memory. Without it the GC only follows the heap goal and an overshoot is an uncatchable OOMKill (exit 137 — the process cannot log, dump stacks, or run cleanup). Wire it at ~90% of the limit, leaving room for non-Go memory:

# imcore values.yaml
env:
  - name: GOMEMLIMIT
    value: "920Mi"   # 90% of a 1Gi limit

Re-verify GC frequency and p99 after enabling: as live heap approaches the soft limit, GC cycles per allocated GiB multiply (the limiter is a soft target — it would rather exceed the limit than starve).

GOMAXPROCS and CPU limits

imcore’s go.mod go directive is 1.27+, so GOMAXPROCS is container-aware: derived from the CPU limit (rounded up, floor of 2), re-checked on cgroup resizes. Do not add automaxprocs — it pins the value and disables the live updates. Recommended default stays no CPU limit for the WebSocket loop (CFS throttling shows up directly in p99); if you must set one, keep it ≥ 1000m or verify throttling (container_cpu_cfs_throttled_periods) under load.

HPA prerequisite

targetCPUUtilizationPercentage is usage ÷ requests — it silently cannot act unless resources.requests is set. With the chart default resources: {}, prefer the connection-count custom metric (previous section), which has no such dependency.

Metrics scraping

GET /metrics sits behind the same ops gate as /stat: source must be loopback or inside admin.allowed_cidrs, plus an admin token while admin.require_token is true (the default). For in-cluster Prometheus either:

• add the cluster pod CIDR to admin.allowed_cidrs and keep the token → serviceMonitor.bearerTokenSecret: {name, key}, or
• also set admin.require_token: false if the CIDR gate alone is acceptable in your threat model.

Alert rules ship in deploy/prometheus/imcore-alerts.yml.

Relationship to compose / systemd

The chart deploys imcore only. The compose stack bundles Postgres/Redis/RabbitMQ/MinIO for a single host; systemd + imctl covers bare-metal fleets. Kubernetes assumes you bring managed/operator-run dependencies. app.yaml is identical across all three — only the injection mechanism differs.