Architecture Deep Dive

This document is the comprehensive functional and technical documentation for imcore, written for newly onboarded developers, solution architects, and technical reviewers. It is organized in four parts: ① Project Overview → ② Feature Documentation → ③ Core Technical Implementation → ④ Standalone Deep Dives into High-Complexity Topics. The full design reasoning for each topic is recorded separately in CLAUDE.md and docs/superpowers/specs/; this document does not repeat all of that argumentation, but provides structured explanations that can be read on their own.

Table of Contents

• Part I Project Overview
• Part II Feature Documentation
• Part III Core Technical Implementation
• Part IV Standalone Deep Dives into High-Complexity Topics
  • Topic 1: Dynamic Binding and Reconnect Reconciliation for Cross-Instance Broadcast
  • Topic 2: Concurrency Model of the High-Concurrency Connection Layer
  • Topic 3: Semantic Unification Across Three Message Brokers
  • Topic 4: SQL Dialect Runtime Rewriter
  • Topic 5: Three Layers of Defense for Message Idempotency and Reliable Delivery
  • Topic 6: Race Control for AI Streaming Generation, Cancellation, and Persistence
  • Topic 7: Idempotency Gate and Dual Audit for LLM Paid Side Effects
  • Topic 8: Concurrency Correctness of Customer-Service Session Boundaries
  • Topic 9: Budget Degradation Ladder for Memory Injection
  • Topic 10: License Signature Verification and Compile-Time Symbol Isolation
  • Topic 11: Audit Log HMAC Hash Chain
  • Topic 12: Transaction Boundaries for Data-Governance Erasure and Archival
  • Topic 13: SSRF and DNS Rebinding Protection
  • Topic 14: Privilege-Escalation Protection for Webhook Callbacks
  • Topic 15: Dual-Dialect Database Migration System
  • Topic 16: Protocol-Driven SDK Generation for Four Platforms
  • Topic 17: Workflow Orchestration of the AI Reply Pipeline
  • Topic 18: Streaming Accumulation and Controlled Side-Effect Execution for Tool Calling
  • Topic 19: Task Flow Chain: From Broker Subjects to the Queue-Routing State Machine

Part I Project Overview

1.1 Project Positioning

imcore is a commercial WebSocket IM system implemented in Go (github.com/balala/imcore, Go 1.27), covering three real-time messaging scenarios (1:1 DM / group chat / live rooms), on top of which it builds complete AI-powered customer service, a human agent workbench, an admin console, data governance, and multi-platform SDKs, delivered commercially under a SKU (software edition) + signed license model.

1.2 Technology Stack Overview

1.3 Process / Binary Matrix

1.4 Two-Axis Commercial Model: Build SKUs × Sellable Editions

The project splits “compile-time capability trimming” and “runtime license unlock” into two orthogonal axes:

Build SKUs (Go build tags, 6 total)

Sellable editions (runtime-gated by lib/license, 4 total): basic (free) / pro (¥9,800/year) / ultimate (¥29,800/year) / customer_service (¥19,800/year). The single source of truth for edition pricing and feature comparison is lib/license (capabilities and quotas) + packaging/editions.yaml (display metadata); cmd/packagegen generates the marketing-site SKU data, feature-comparison docs, and packaging/sku.mk from these.

Release binaries do not trust plaintext configuration: signature verification is mandatory (Ed25519), the public key is burned in at compile time via ldflags, and editing yaml cannot upgrade the edition; after a license expires the service does not go down — it degrades to basic and keeps running (core IM stays alive, paid capabilities are turned off in real time).

1.5 Overall Architecture Diagram

                         ┌────────────────────────────────────────────────┐
                         │                Clients (multi-platform)         │
                         │  Web SDK / Electron / RN / Mini Program /       │
                         │  Flutter / Swift / Kotlin / chat-studio /       │
                         │  Customer-service Widget                        │
                         └───────────────┬────────────────────────────────┘
                          WebSocket(JSON) │ HTTP API / OpenAPI / Webhook
             ┌───────────────────────────┼──────────────────────────────┐
             │              websocket main service (multi-instance)      │
             │  ┌────────────┐ ┌──────────────┐ ┌─────────────────────┐  │
             │  │ Connection │ │ Command      │ │ Business            │  │
             │  │ layer:     │ │ routing:     │ │ controllers:        │  │
             │  │ sharded    │ │ ~130 WS cmd  │ │ DM/group/room/      │  │
             │  │ registry   │ │ + HTTP/ADMIN │ │ friend/RTC/modera-  │  │
             │  │            │ │              │ │ tion/push/session   │  │
             │  │            │ │              │ │ state               │  │
             │  └─────┬──────┘ └──────┬───────┘ └─────────┬───────────┘  │
             │        │        ┌──────┴────────┐   ┌──────┴──────────┐  │
             │  ┌─────┴──────┐ │ bus.go glue   │   │ gRPC server     │  │
             │  │ Admin      │ │ layer (bind/  │   │ (presence/      │  │
             │  │ Console    │ │ unbind        │   │ direct          │  │
             │  │ :7080,     │ │ refcounting)  │   │ delivery)       │  │
             │  │ standalone │ │               │   │                 │  │
             │  └────────────┘ └──────┬────────┘   └──────┬──────────┘  │
             └───────────────────────┼──────────────────┼──────────────┘
                                     │ Publish/Subscribe│
        ┌────────────────────────────┼──────────────────┼───────────────────┐
        │               Message bus (pluggable, pick one of three)          │
        │  RabbitMQ(topic/direct) / NATS JetStream(3 streams) / Redis Streams│
        └───────┬─────────────────────┬───────────────────┬───────────────┘
                │                     │                   │
     ┌──────────┴─────────┐ ┌─────────┴────────┐ ┌────────┴─────────────┐
     │ persist_consumer / │ │ ai_worker (multi │ │ sync_worker / other  │
     │ jetstream_consumer │ │ replicas):       │ │ workers: business    │
     │                    │ │ LLM/RAG/tools/   │ │ sync / governance    │
     │                    │ │ transcription    │ │ cron jobs            │
     └──────────┬─────────┘ └─────────┬────────┘ └────────┬─────────────┘
                │                     │                    │
     ┌──────────┴─────────────────────┴────────────────────┴─────────────┐
     │  MySQL / PostgreSQL (dual dialect, optional read/write splitting)  │
     │  messages/sessions/CS/AI/governance ~50 tables + pgvector         │
     │  Object storage (S3/OSS/MinIO: media, governance archives)        │
     │  Redis (presence/cache/locks/rate limiting/streams)               │
     └──────────────────────────────────────────────────────────────────┘

Key data flows (summary; details in Part III):

1. Uplink: client WS frame → auth/rate limiting/dedup → moderation (sensitive words/callback) → synchronous idempotent persistence → local delivery + broker cross-instance delivery → offline push task → conversation summary → ack.
2. Downlink: broker subscriptions (room.> / user.> / group.> etc.) → handleBroadcast identity re-verification → local room/group/user index fan-out → per-connection dual queues (reliable/best-effort) write pump.
3. AI: a DM hits a bot → ai.request (workqueue, exactly one worker consumes) → streaming delta/done/error (fanout, each WS instance delivers to local sessions).

Part II Feature Documentation

2.1 IM Core Messaging

2.1.1 Three Conversation Scenarios

2.1.2 Message Operations and UX

• Message operations: edit, recall (with MutationVersion version chain), emoji reactions, quoted replies, read and delivery receipts (chat_message_receipts table), typing indicators.
• Message body structure: a segments union — text / emoji / image / file / voice / location / contact card / nudge / quick_reply / card (rich interactive; see 2.3.6); message IDs are UUIDv7 (time-ordered, enforced by the server); requests carry a tempID for client-side dedup and ack correlation.
• Conversation state: unread counts (atomic Redis Lua increment + DB persistence in chat_conversation_states), pin/mute/tags/drafts, recent conversation list (ZSet top-100), multi-device sync (chat_device_sync event stream + full-snapshot fallback).
• RTC audio/video: 1v1 calls + multi-party meetings (rtc_* / rtc_multi_*), TURN HMAC ephemeral credentials (RFC 5766-bis), call records persisted to DB and able to trigger AI call analysis.
• Relationships: friend request/accept/remove, blocklist, DM allowlist policy, enterprise org structure (department sync + @department).
• Media: uploads go through the server-side gateway (S3/OSS/MinIO/local), private buckets + time-limited presigned GET; DM media is not publicly accessible by default.

2.1.3 Content Safety

• Sensitive-word moderation (moderation): built-in word list of ~2k Chinese terms (go:embed) + admin-side CRUD of the word list; hit policy block/replace/pass is configurable; async audit records land in chat_moderation_logs; a word-list load failure is fail-open (normal messages are not blocked because the moderation list hiccups).
• Pre-send callback (message_callback): a synchronous HTTP callback that can moderate, rewrite, or reject messages; HMAC-SHA256 signature headers prevent forgery; fail_open configurable (see Topic 14).
• Post-delivery callback (message_route): asynchronous event fanout to business systems (RongCloud-style signature compatibility).

2.2 AI-Powered Customer Service (aiagent build)

Evolved by phases; current capabilities (P0–P6+):

2.3 Human Customer-Service Workbench

2.3.1 Agents and Organization

• Agent roster (cs_agents): the single source of truth for agent identity; bootstrap seeds an empty table + admin CRUD; roles are agent / supervisor; agents belong to teams (cs_teams).
• Availability: being online (determined by cross-instance presence) ≠ being available; automatic routing requires enabled + online + ready state + within shift + skill/language match + exact active count below max_concurrent.
• Team routing: transfers can target a team (teamID), auto-picking an agent least-loaded; with no candidate the conversation enters a persistent queue.

2.3.2 Service Sessions (cs_sessions)

A single service engagement is modeled as a first-class concept:

• Session boundary: a conversation is judged a CS session if any participant is a bot or a routing row exists; the first message anchors the session open, and the activity watermark advances monotonically; auto-close after 5 idle minutes triggers the summary (independent of the 30-minute memory write-back threshold).
• Transfer / reassignment: agent-to-agent transfer (chat_dm_transfer, a core command), supervisor forced reassignment (chat_dm_force_reassign, fail-closed + pre-written audit row), in-group claim/release/reassign (chat_group_cs_claim/release/reassign, reusing the routing table, groupcs: conversation-id shape).
• Queueing system: cs_routing_queue persistent queue (priority + enqueue-time retries; DB short leases coordinate multiple workers); outside business hours offline_policy=queue also queues persistently; ai_worker.routing_queue.interval_sec (default 5s) scans it.
• Satisfaction rating: after a session closes, a rating card (rich interactive message) is sent to the customer — 1/3/5 scores + a hard expiry window for supplementary ratings + a cooldown + a minimum-message-count gate, three gates in all; operations gets detailed review (review_note/tags), a rating nudge (at most once), and satisfaction trends in daily_stats.
• Reply templates (cs_reply_templates): agent quick replies with {{var}} placeholders rendered client-side; returns the true total when capped at 500 entries.
• Internal notes (cs_customer_notes): append-only free-text agent notes, never fed into AI prompts (preventing an LLM from paraphrasing things like “this customer is difficult”); id-cursor pagination.
• Customer profile card (ai_user_profile_values): agents can view/edit a customer’s structured profile; values with source='operator' are never overwritten by the model; authorization fail-closed.
• Agent workbench (/home/agent-workbench) and CS demo (/home/cs-demo, visitor bubble → transfer-to-human creates a dedicated CS group → queue-claim drill).

2.4 Admin Console (Ultimate, standalone Gin engine on :7080)

• Security: JWT (access/refresh/2FA — three token kinds isolated by audience, jti dual-channel revocation), RBAC (35+ permission codes module:action, four-level wildcards, built-in superadmin/ops/moderator/readonly + cached DB custom roles), TOTP two-factor (RFC 6238, secrets encrypted at rest, 10 recovery codes), login rate limiting, full operation audit.
• Management surface: dashboard, operations trends (daily_stats: DAU/new users/senders/active groups & rooms/peak online/message volume/satisfaction), admins & roles, user/group/room/message management (including message logs and broadcast), sensitive words, notification templates, announcements, push tasks, blocklist, system configuration (runtime hot-update allowlisted keys + history/rollback), license status, AI bot management (bot CRUD/testing/usage, KB documents, memory, tools, hooks, QA), customer-service center (agents/teams/CS groups/templates/satisfaction/queue live pages), call records, data governance.
• Audit: all console operations land in admin_audit_logs, protected by an HMAC hash chain against tampering (Topic 11).

2.5 Data Governance (Ultimate)

• Data inventory: 14 categories of data assets (table-level inventory: category/purpose/sensitivity/ subject key).
• Subject access/erasure (DSR): request state machine (pending→approved→processing→completed), export matched per customer; erasure = rewriting DM conversation ids + pseudonymizing messages + 44 delete specs + 5 pseudonymization specs inside a single transaction (erasing my data must not destroy other people’s conversations) + object-storage erasure-marker replay compensation.
• Message archival: tiered retention policies (DM/group/room/community × session/group/version scope), archive packages (including related tables) + SHA-256 checksums + manifests stored in object storage, idempotent restore after checksum + allowlist validation; messages under legal hold are not archived.
• Legal hold: legal hold blocks erasure; retention-policy execution and the scheduler fail closed independently.
• Execution safety: governance.execution_enabled defaults to false; destructive operations from both the admin page and the CLI require an explicit -execute.

2.6 Open Capabilities

• Inbound Push OpenAPI (/openapi/v1): customer business backends push user profiles/friendships/ call records one-way; database-scoped credentials (SHA-256 stored, token shown only once) + rate limiting (local bucket + optional cross-instance per-minute quota in Redis); when disabled, 404 does not reveal that the endpoint exists.
• External sync sync_worker: periodically pulls business APIs (users/friends/DMs/rooms/room-chat history), version numbers prevent rollback (chat_sync_entity_versions), cursors/watermarks persisted, no advance on failure; multi-instance Redis lease leader election (active-passive).
• Webhook outbound contracts: docs/WEBHOOK_CONTRACTS.md + JSON Schema + examples, all generated by cmd/protocolgen from clients/protocol.yaml, with CI drift protection.
• Multi-platform SDKs: TypeScript (browser/Node/Electron/RN/Mini Program), Dart/Flutter (with UIKit), Swift (SPM), Kotlin (Gradle); protocol is additive forward-compatible (server ≥ v1.2.0); SDK public API surface snapshots (sdk-api/snapshots/*.api) + default-behavior baselines + migration-report policy.
• Customer-service Widget: iframe embedding + host-page loader script (IMCoreWidget API), brandable.

2.7 Operations and Delivery

• Deployment forms: Docker Compose (one-command demo stack: pgvector+Redis+RabbitMQ+MinIO+migrate Job), Kubernetes Helm (main deployment rolling strategy + PDB + HPA (can consume the custom metric imcore_current_connections) + worker deployment + ServiceMonitor), systemd + imctl TUI, Xinchuang three-architecture tarballs.
• Observability: /healthz /livez /readyz (dependency readiness fail-closed, not-ready during drain), /stat (connections/throughput/errors/goroutines/cluster instances), /metrics (50+ custom series: message counts/queue depth/worker stream lag & backlog/DB connections & long transactions/ webhook success rate/error-code distribution), 16 Prometheus alert rules; worker binaries have a separate consumer.obs_addr observability endpoint + bounded drain.
• Capacity and performance (4C8G single instance, multi-room 200 msg/s): 10,000 concurrent connections p99 = 152.8ms, fan-out 38,506 msg/s, 0 errors and 0 lost messages (sat-20260724, the officially published figures); 5,000 connections recommended for low-latency capacity (p99 143.9ms); 39KiB memory per connection; Protobuf bus decoding is 5.6× faster than JSON.
• Chaos testing: 8 fault-injection scripts (DB/Redis/broker outage, slow consumers, queue flooding, reconnect storms, network matrix, expired upload credentials), with zero loss proven by database counts (a one-shot comparison of sent count vs deduplicated room_messages count proves zero loss and zero duplication).
• Quality pipeline: make ci = fmt/vet/lint + 4-SKU test matrix + migration consistency + SKU symbol isolation + protocol/packaging artifact drift protection; PGO (a committed default.pgo takes effect automatically); gosec + govulncheck + npm audit; secret-check prevents credentials entering the repo.

Part III Core Technical Implementation

3.1 Connection Layer (servers/websocket)

Connection establishment (init_acc.go): HTTP → WS upgrade, with the auth chain executed in order — IP handshake rate limit → JWT (carried in query/header, three forms; verifies issuer/audience, extracts user_id/scopes/jti/device_id) → DB revocation check → per-user rate limit → ban check → Origin allowlist (normalized scheme+authority) → per-instance connection cap (503 when exceeded, letting the LB rebalance). After the upgrade there are two goroutines per connection (read/write).

Client structure (client.go): dual send queues — Send (reliable, default 256) and BestEffort (best effort, default 32); outboundFrame is a value type, so during broadcast fan-out the same frame (including the *websocket.PreparedMessage) is shared by all receivers with zero extra allocations.

Heartbeats: 75s read timeout (refreshed by pong); server pings use a sharded scheduler + bounded worker pool (ping_scheduler.go) to flatten the huge number of timers; application-level cleanup after 180s of inactivity (expired-JWT connections are removed at the same time).

Registry (client_registry.go): shardedRegistry shards connection sets into 32 shards by FNV hash of session ID + shards by user ID (map[userID]map[sessionID]*Client, naturally multi-device), with no global lock; cross-shard snapshots are not globally atomic (all callers can tolerate this). ClientManager.start() is a single-goroutine event loop, selecting over six channels of capacity 1000 (Register/Login/Unregister/JoinRoom/LeaveRoom/Broadcast).

3.2 Message Uplink Path

ProcessData (acc_process.go): JSON envelope {seq, cmd, data} → 256KiB cap → global rate limit → non-public commands require login → protected commands re-check revocation before every dispatch (closing the race window between revoke and socket close) → handler-map dispatch → a uniform {seq, cmd, code, data} ack.

The three send controllers share the same skeleton (dedup → permission → moderation → rate limit → UUIDv7 → callback gate → persistence → delivery → offline push → summary → ack), differing in delivery routing and persistence approach:

3.3 Message Downlink and Cross-Instance Broadcast (bus.go)

Subscription topology: each instance declares 5 broadcast queues (gowebsocket.broadcast.{global|user|room|group|community}.{instanceID}) subscribing to wildcard subjects; persistence uses the fixed queue name gowebsocket.persist.room to form fleet-wide competing consumption (each message is persisted exactly once).

handleBroadcast, the glue core: dispatches by RoomType to local primitives (global/user/room/group/ targeted-group), with three safety hardenings — identity re-verification of the payload’s embedded sender (the sender field compared against BusMessage.UserID), group targeted messages re-validating membership on the consumer side (fail-closed, defending against forged payloads), and room message permission re-checks (lifecycle commands pass through; everything else checks membership state; the admin prefix compares the persisted recall state and version).

Same-instance direct-delivery optimization (user-targeted): deliver locally first, then consult Redis presence to decide whether remote sessions exist; publish only when there are remote sessions, stamping OriginInstance; when the message bounces back to this instance it is skipped (origin dedup), avoiding double delivery. Presence misjudgment is backstopped by P0 catch-up sync + offline push.

Dynamic Bind/Unbind: per-user/room/group/community reference counts (local online member counts) — Bind on 0→1, Unbind on 1→0; under RabbitMQ these are real server-side bindings; under NATS/Redis, Bind degrades to first-time wildcard-subscription setup (local filtering). After reconnect: snapshot — execute outside the lock — reconcile inside the lock (see Topic 1).

3.4 Message Bus Abstraction (lib/broker)

A unified interface Publish / Subscribe / Bind / Unbind / OnReconnect / Close plus the extension SubscribeWork (the competing-consumption primitive); subjects are uniformly NATS-style, and the RabbitMQ adapter translates > into # at the boundary.

Codec (lib/buscodec): encoding follows configuration (Protobuf by default, measured 5.6× faster); decoding sniffs the first byte ({ = JSON, otherwise Protobuf), so old and new instances can inter-translate during rolling upgrades.

3.5 Persistence Layer (lib/dblib)

Dual-channel design:

• Channel A (synchronous): the send-path controller performs an inline idempotent insert — the message is in the DB before ack (the first pillar of reliability);
• Channel B (asynchronous): history-backfill-type writes go through 4 buffered channels (room/roomChat/ groupChat/dm, default cap 10000) + a single-goroutine persistLoop batch flush (batch 2005000 / 100ms60s, hot-tunable); when a channel is full, DB writes are dropped but Redis history is still written (tiered degradation); after 3 flush failures with exponential backoff, the drop is logged.

MySQL/PostgreSQL dual dialect: sql_dialect.go is a runtime rewriter translating MySQL-dialect SQL into Postgres (see Topic 4); connection pool defaults maxIdle=maxOpen; the PG DSN auto-injects idle_in_transaction_session_timeout; optional read/write splitting (deep-paginated history and search routed to read replicas; latest pages/transactions/read-after-write always go to the primary).

Migrations: migrations/{mysql,postgres}/ dual-dialect directories + schema_migrations bookkeeping

• CI set-consistency checks (see Topic 15).

3.6 Redis Layer (lib/cache / lib/redislib)

• Presence: three-key structure (session Hash TTL 24h + user-session Set + legacy single-record compatibility), the basis for cross-instance delivery routing and gRPC fan-out;
• Instance directory: balala:hash:servers Hash + readiness lifecycle reporting;
• Hot history cache: per-scenario list RPush+LTrim(1000), DB backfill on miss;
• Conversation summaries: ZSet conversation list + Lua atomic unread increment + DB persistence;
• Request-dedup state machines / rate-limit windows / mutation mutexes / group & relationship caches;
• Redis cluster compatibility: conversation-state keys carry a {userID} hash tag to keep Lua in a single slot.

Message identity has no centralized sequence number — UUIDv7 message ID + DB auto-increment id (the history pagination cursor).

Fixed — per-conversation write lock makes id order == commit order (regression gate: lib/dblib/conversation_write_lock_test.go). Auto-increment ids are assigned at INSERT while rows become visible at COMMIT, both under read-committed. Without serialization, concurrent senders in one conversation can interleave commits so a reader sees id 7 while id 6 is still in flight, advances its cursor to 7 (prevCursor = max id in page), and message 6 is then permanently skipped by the id > cursor catch-up channel (a client-view skip, not data loss). The fix serializes “assign id → commit” per conversation in every chat-insert DAO (DM/group/room idempotent inserts, legacy upserts, synced/AI-reply inserts — lib/dblib/conversation_write_lock.go): a session-scoped advisory lock (PG pg_try_advisory_lock(hashtextextended(key,0)) / MySQL GET_LOCK), acquired via a try-and-backoff loop that pins no pool connection while waiting; the holder pins one connection and runs the insert on it, so concurrent holders can never deadlock against the pool. Lock for order instead of escalating isolation, per the Materialize “linearizable log on read-committed” playbook. Known bypass-writers outside the gate (recorded): demo seeding and the room_messages event persister — neither feeds the chat-history cursor path. The old cursor-cancellation fallout (CancelPendingPushTasksUpToCursor UUIDv7 <= readCursor) shares the healed premise: reads can no longer overtake an uncommitted smaller id from the locked path.

3.7 AI Pipeline (lib/aiagent + cmd/ai_worker)

DM persisted → dispatchAIBotDM (bot: prefix / routing-table decision)
  → RegisterQueuedStream (Redis registers cancellable metadata) → publish ai.request
  → [ai_worker] SubscribeWork (exactly one replica consumes; always acks, errors go to stream.error)
    → rate limiting (Redis Lua sliding window, in-process fallback on failure)
    → speech transcription (optional, SSRF-guarded webhook)
    → beforeChat hooks (rules can stop/handover)
    → fork: native pipeline / external workflow (Dify/Coze/webhook own memory and RAG)
       native pipeline:
         memory assembly (profile → summary → vector facts, budget degradation)
       + KB retrieval (pgvector top-k, low-confidence policy)
       + history window (role attribution: bot/taken-over agent = assistant)
       → provider.Stream (SSE → chunk channel)
       → per-delta publish chat.stream.delta (Seq monotonically increasing)
       → tool loop (≤3 rounds, idempotent claim, dual audit)
       → persist dm_messages → publish chat.stream.done
  → [websocket instance] ai_push fanout → local-session frame delivery (no-op if not on this instance)

Consumption semantics: work subjects (ai.request/kb.ingest/call.analyze/ai.handover.brief/ ai.survey.ready) are processed by exactly one replica; push subjects (chat.stream.* etc.) fan out to all instances. The source of truth for the subject→stream mapping is lib/broker/aisubjects, guarded by drift tests. Background sweeper (single-flight under a single Redis lock): session close (5min) → memory write-back (30min) → memory TTL cleanup → routing-queue scan (5s), with per-phase budgets over a shared clock (see Topic 8).

3.8 HTTP / gRPC Surface

• HTTP (routers/web_routers.go): health probes (public), user group (online/limits/config/upload/ transcription), admin group (adminOps gateway = loopback + CIDR + token: message dispatch/ announcements/system/stats/metrics/pprof/chat-studio debug console), OpenAPI (Bearer).
• gRPC (servers/grpcserver): QueryUsersOnline / SendMsg / SendMsgAll / GetUserList — the cross-instance delivery fallback when the bus is disabled, plus the presence-query channel; instances that lose contact are automatically removed from the Redis directory.

3.9 Security Architecture (Cross-Cutting)

Part IV Standalone Deep Dives into High-Complexity Topics

The following 19 topics are the parts of this project with the highest implementation difficulty and complexity. Each topic is presented independently in the structure “problem background → why it is hard → implementation → key code → design trade-offs” and can be read on its own.

Topic 1: Dynamic Binding and Reconnect Reconciliation for Cross-Instance Broadcast

Location: servers/websocket/bus.go (~900 lines of core logic).

Problem Background

In a multi-instance deployment, when user A is connected to instance 1 and user B to instance 2, a message from A to B must be delivered across instances. Delivery relies on the message bus, but each instance must not subscribe to every subject — wildcard subscriptions over tens of thousands of rooms × hundreds of thousands of users would drown effective bandwidth in irrelevant messages. Bindings must therefore be dynamic: an instance subscribes to a room/group/user subject only while it holds local connections for it, and unsubscribes once they leave.

Why It Is Hard

Binding state is a compound problem of distributed consistency + high-concurrency lifecycle:

1. join/leave/login/logout for connections happen concurrently across multiple goroutines (the event loop, controller synchronous sections, per-user worker pools), and counts must be exact — over-counting leaks stale bindings, while under-counting silently drops cross-instance messages (a RabbitMQ topic message that hits no binding is discarded with no buffering and no error);
2. after a broker reconnect, all bindings must be rebuilt while counts are still being mutated concurrently — in the gap between snapshot and execution a key’s count may already have dropped to zero, and the rebuilt binding would be wrong;
3. Bind is a network round trip and must not be executed while holding any lock, or a single network hiccup stalls every login.

Implementation

• Four reference-count tables: userBindCounts / roomBindCounts / groupBindCounts (+ the groupLocalUsers map[groupID]map[userID]int two-level index); Bind on 0→1, Unbind on 1→0; groups count “user × group × number of sessions”; snapshots are cleaned up when a group dissolves.
• Three-phase reconnect reconciliation (rebind) (rebindCounts, a generic function):
  1. snapshot all current non-zero counts under the lock;
  2. perform each Bind outside the lock (network operations hold no lock);
  3. afterwards reconcile under the lock: entries whose counts hit zero mid-flight get a compensating Unbind. No matter what happens concurrently during the rebuild, the final state matches the count tables.
• Ack ordering guarantee: the join/leave controllers complete the broker bind and the client.RoomID assignment synchronously while holding roomStateMu, before side effects are dispatched into the event loop — guaranteeing that when the ack reaches the client the state has already landed, and a message sent immediately after the ack is not dropped by an unbound routing key.
• Network I/O stripped out: broker binds and presence broadcasts for login/leave go to a per-user sharded worker pool (16 shards × cap 2048, FIFO keeps per-user ordering); when a shard queue is full, execution degrades to inline rather than being dropped (dropping a bind = silently cutting delivery).
• RabbitMQ publishPool: binding operations used to contend on the internal lock of a single shared channel; the pool builds its own connections + N independently locked channels (round-robin), degrading back to the shared channel on failure — “never worse than before”.

Design Trade-offs

Under NATS/Redis, Bind degrades to a no-op (wildcard subscription + local filtering), so the value of this counting system is concentrated on RabbitMQ; but the code uniformly goes through reference counting so the bus layer behaves identically across the three brokers. Debugging count drift is extremely costly (no errors, only lost messages), so all cleanup paths (dissolution, eviction, reconnect) go through the same reconciliation logic instead of each doing its own thing.

Topic 2: Concurrency Model of the High-Concurrency Connection Layer

Location: servers/websocket/client.go, client_registry.go, client_manager.go, ping_scheduler.go, send_policy.go.

Problem Background

A single instance must stably carry 10,000+ long connections and 38,000+ msg/s fan-out (measured figures), with p99 latency under 200ms. Goroutines are cheap, but one timer per connection, one lock contention per message, and slow consumers blocking broadcasts are the three things that destroy performance at scale.

Why It Is Hard

1. Slow writers must not drag down fast ones: broadcast fan-out sends the same frame to thousands of connections, and any single connection with a full TCP window can block the shared path;
2. Heap overhead of massive timers: if every connection’s heartbeat ping runs its own time.After, the timer-heap churn at 10k connections is significant;
3. Backpressure policy must differentiate message classes: users complain when chat messages are lost, but nobody notices lost typing/status signals — a one-size-fits-all policy will not do.

Implementation

• Dual queues + tiered backpressure: per connection, two channels — Send (reliable, 256) and BestEffort (best effort, 32); classifyOutboundPriority (send_policy.go) classifies by cmd allowlist (typing/status/conversation updates are best-effort). The send path uses non-blocking select + default: reliable queue full → judged a slow consumer, actively shutdown(CloseTryAgainLater) to disconnect (the client re-pulls history after reconnecting); best-effort full → drop and count (imcore_best_effort_dropped_total).
• Batch-drain write pump (nextQueuedMessage): the write loop polls both queues non-blockingly; under backlog, an entire batch sets the write deadline only once, reducing timer-heap churn from O(N) to O(1) per batch; without backlog it blocks in select. Broadcast frames use a process-wide shared PreparedMessage (likewise for ping frames): serialize once, send N times.
• Meter before enqueue: the queue-watermark gauge is incremented before the channel send, avoiding a race with the writer’s immediate receive that would permanently inflate the metric.
• Sharded registry: connection sets are sharded 32 ways by SessionID (UserID may still be empty at registration; SessionID is stable from NewClient onward), and the user index is sharded by userID — each per-user operation touches a single shard with a single lock; the total connection count is a lock-free atomic (handshake cap checks take zero locks).
• Sharded ping scheduler: interval = pingPeriod / shardCount; each tick snapshots only one shard, spreading 10k ping writes across the whole period; a bounded worker pool prevents one slow shard from stalling everything.
• Layered event loop and worker pools: ClientManager.start() is a single goroutine serially processing the six lifecycle channels (in-memory index add/remove); heavy I/O (broker binds, presence broadcasts, stats writes) is pushed down to per-user sharded worker pools, degrading inline when queues are full.

Measured Results

4C8G single instance: 10,000 connections multi-room p99 152.8ms, fan-out 38,506 msg/s, 0 errors and 0 lost messages; 39KiB memory per connection. Two key fixes are on record: the single-goroutine login event-loop avalanche (~11× improvement on the same host), and a conversation-summary batching change that saved CPU but regressed p99 by 2× (located via go tool nm symbol isolation and reverted) — performance work in this project is data-driven.

Topic 3: Semantic Unification Across Three Message Brokers

Location: lib/broker/ (interface + 3 adapters), lib/broker/aisubjects.

Problem Background

The same business code must run on RabbitMQ, NATS JetStream, and Redis Streams, switchable by configuration without code changes. But the three differ radically in capability model: RabbitMQ has real bindings but no ack deadlines; JetStream has the two-level stream/consumer abstraction and WorkQueuePolicy; Redis Streams only has consumer groups + PEL. More critically, the paid side effects of the AI subsystem require “each task processed by exactly one worker”, so all three brokers must offer consistent competing-consumption semantics.

Why It Is Hard

1. fanout and workqueue are two opposite delivery semantics, expressed differently by each broker (RabbitMQ/Redis via queue/group names, NATS via stream policy + consumer shape);
2. NATS push consumers have two famous traps: a durable push consumer without a deliver group does not allow a second subscriber (the second replica panics on startup); push + deliver group piles messages in the memory of whichever member the server picks (a slow worker hoarding work it cannot do);
3. subject registration must be a closed set: in NATS, publishing a subject that no stream captures produces no error, no log, no metric — a silent black hole.

Implementation

• SubscribeWork as the single entry point: all competing consumption goes through it; RabbitMQ/Redis express it with fixed queue/group names (per-instance suffix = fanout, fixed name = competing), NATS with a shared durable pull consumer (a Fetch(5s) loop + backoff, with reconnects de-duped by subscription key to prevent goroutine leaks).
• Why pull over push (code comments record two real incidents): both persist and AI work use pull — it is the only shape that supports multi-replica competition without letting a slow worker privately hoard messages. A push-durable-without-deliver-group once caused the second replica to panic on startup; it was converted to pull, with a one-time upgrade step documented (docs/RUNBOOK_BACKUP_RESTORE.md).
• Dual AI stream design: <stream>_ai_work (WorkQueuePolicy, ack deletes; JetStream rejects a second consumer on the same subject — “no fanout” is enforced by the broker rather than by code review) and <stream>_ai_push (LimitsPolicy + one durable per instance, FileStorage — MemoryStorage loses consumers when NATS restarts). Why the core stream cannot host AI subjects: the core stream’s MaxAge of 5min is an instantaneous distribution buffer, so AI tasks for a fleet of workers down for 5 minutes would be silently swallowed; and Retention is a stream-level attribute that cannot be overridden per subject.
• Subject registry as source of truth: lib/broker/aisubjects declares the ownership (work/push) of all ai.* subjects; a build-tag-free drift test intercepts literals that bypass registration.
• Redis specifics: XAUTOCLAIM reclaims the PELs of dead consumers (AI uses a longer minIdle threshold so a running LLM handler is not wrongly reclaimed); NOGROUP self-healing (on group loss, evict the local cache and rebuild; a workqueue’s first creation reads from 0 so tasks published before startup are not lost); Close’s wg.Wait() is a bounded wait (go-redis v9 blocking reads are immune to ctx cancel; only the Block window can be waited out).
• Operational differences documented explicitly: two NATS semantics RabbitMQ lacks — redelivery on ack timeout (AckWait must be budgeted for the whole handler, not a single model call) and discard when MaxDeliver is exhausted; Redis production must enable AOF + maxlen covering the maximum backlog over the recovery window.

Design Trade-offs

What is guaranteed is “each delivery goes to only one replica”; after crash/timeout redelivery it is still at-least-once, not business-level exactly-once — hence every paid/non-idempotent path has a separate DB claim gate (Topic 7). The broker only dispatches; idempotency is always the business layer’s responsibility.

Topic 4: SQL Dialect Runtime Rewriter

Location: lib/dblib/sql_dialect.go (~400 lines of core logic).

Problem Background

The same set of DAOs must run on both MySQL and PostgreSQL. The conventional approach — writing every SQL twice or adopting an ORM — does not fit a project with ~50 tables and a large amount of hand-written high-performance SQL (batch INSERT, upsert, CTE); duplicate maintenance inevitably drifts. The choice: MySQL dialect as the authoring language, rewritten to Postgres at runtime before execution.

Why It Is Hard

Rewriting is not string substitution but a small compiler front end with context:

1. INSERT IGNORE → ON CONFLICT ... DO NOTHING needs a conflict target (the unique-key list), which Postgres syntax never omits and MySQL has no concept of — a hand-maintained conflict-column registry is required (postgresConflictTarget, ~50 tables);
2. in ON DUPLICATE KEY UPDATE, VALUES(col) must become EXCLUDED.col, IF(a,b,c) must be recursively parsed into CASE WHEN, and bare columns in the update clause need table qualification (Postgres rejects bare columns in DO UPDATE as ambiguous) — distinguishing “SET left-hand side” from “comparison expression” requires paren-depth analysis (isUpdateAssignmentLHS);
3. all of this happens on the hot path of every Exec/Query and must cost nearly zero.

Implementation

• dialectDB/dialectTx wrappers: when driver=postgres, SQL passes through prepareSQL for rewriting; MySQL passes through untouched (Dameng DM8 goes through the MySQL-compatibility passthrough).
• Fast-path short circuit: allocation-free hand-written hasPrefixFold/containsFold (ASCII lower-casing) scan first, so the vast majority of keyword-free queries pass at zero cost; regexes are compiled once at package level.
• Rewrite rule list: INSERT IGNORE→ON CONFLICT DO NOTHING (registry lookup), ON DUPLICATE KEY UPDATE→ON CONFLICT DO UPDATE (including VALUES→EXCLUDED, IF→CASE, column qualification), boolean literals, UNIX_TIMESTAMP→EXTRACT(EPOCH FROM), backticks→double quotes, ?→$N (quote-aware rebind that never treats a ? inside a string literal as a placeholder).
• Fail loud: when ON DUPLICATE KEY finds no registered conflict target, it errors immediately instead of silently passing through (Postgres would inevitably produce a syntax error; failing early beats failing late).
• Performance: after regex optimization the dialect translation is 12.6× faster (recorded in docs/perf); the hot-path overhead on PG deployments is negligible.

Design Trade-offs

The registry is hand-maintained (new tables must register), in exchange for a single source of SQL and zero ORM tax. The rewriter covers only the SQL dialect subset the project actually uses (it is not a general SQL translator), which keeps the rule set tractable and exhaustively testable — more realistic than “universally correct”.

Topic 5: Three Layers of Defense for Message Idempotency and Reliable Delivery

Location: lib/cache/request_state_cache.go, lib/dblib/db_lib_dm_history.go, servers/websocket/chat_dm_controller.go.

Problem Background

IM send semantics require: client resends (timeout retries), broker redelivery (at-least-once), and batch-channel retries — under any of these, the same logical message must produce its side effect exactly once (one DB insert, one push, one unread increment), while a resend must still receive the same ack as the first time (clients correlate the UI via tempID).

Why It Is Hard

1. a resend can arrive after any time gap — once the Redis window expires, only the DB unique key can identify it, and once identified the code must decide precisely “which side effects to skip and what the ack carries”;
2. request-dedup scope matters: seq is a connection-level counter and overlaps after reconnect — mutation dedup must concatenate the SessionID into the scope, or cross-connection misjudgment follows;
3. the ack cache itself must resist impersonation: a third party guessing a tempID and replaying it must not walk away with someone else’s ack.

Implementation (Three Layers)

1. Redis request state machine (millisecond-to-hour window): SETNX balala:request:state:{user:cmd}:{tempID} (processing, TTL 24h); the completed state caches the ack result for direct replay; failures delete the key to allow retry. extractCachedAck verifies ack ownership (user+cmd match) to prevent tempID impersonation; mutation scopes concatenate the SessionID.
2. DB persistent idempotency (fallback after the window expires): dm_messages unique key (conversation_id, sender_id, client_dedup_id), MySQL INSERT IGNORE / PG ON CONFLICT DO NOTHING; when RowsAffected=0, the original message_id is looked back up, and the controller receives inserted=false.
3. Conditional side-effect execution: the inserted=false branch skips re-delivery/offline push/summary update and returns only the ack carrying the original ID — every branch has a comment arguing its case (why no re-delivery: the broker already got it the first time; why return the original ID: the client merges two tempIDs pointing at the same message into one UI entry).

All batch async channels use INSERT IGNORE, safe alongside broker at-least-once redelivery; room persistence consumption uses database dedup counts as the final zero-loss criterion (chaos tests verify exactly this).

Related Mechanisms

• Offline messages have no separate queue: the message is the offline message (DB persistence + Redis hot history top-1000); after reconnecting, clients page through in both directions from a cursor. Online delivery uses a three-step decision (local → gRPC → Redis presence); only on a miss does the message enter push_tasks for offline push (filtered by quiet hours/notification level/ mention level).
• The current state of device sync and the sliding-sync plan are in docs/superpowers/specs/2026-08-26-sliding-sync-sticky-events-research.md (conclusion: build a narrow purpose-built protocol; do not copy Matrix MSC4186).

Topic 6: Race Control for AI Streaming Generation, Cancellation, and Persistence

Location: lib/aiagent/worker.go, stream_cancel.go, dispatch.go, reply_delivery.go.

Problem Background

AI replies are a cross-process streaming pipeline: a WS instance accepts the user’s “stop generating”, some ai_worker replica owns the generation goroutine, and delta frames reach the user through multiple broker hops. Cancellation races forward with generation/persistence, and the guarantees are: a request the user has stopped must never publish done, while a legitimate completion where “done precedes the cancel’s arrival” must not be killed by mistake.

Why It Is Hard

1. cancellation can appear at any point in the pipeline: while queued (not yet taken by a worker), mid-stream, at stream end, before persistence, before publishing done — every gap must re-check;
2. there is no shared memory across processes: the cancel signal must travel “WS instance ↔ worker replica” and must not depend on pub/sub arrival ordering;
3. between ack and first frame there is a “cannot cancel” window: the task is already in the broker but no worker has taken it when the user hits stop — who claims this cancellation?
4. a worker panic mid-stream: the outer recover has no message ID and cannot emit a terminal error carrying the ID; the client stream would hang forever.

Implementation

• Cancellable while queued: DispatchRequest calls RegisterQueuedStream before publishing (Redis writes balala:ai:stream:active:<request_message_id> authorization metadata) — cancellers can hit this key from the first millisecond onward; if publishing fails, it is cleaned up.
• Cross-process cancellation = Redis polling, not pub/sub: watchStreamCancellation runs EXISTS cancel key every 100ms — any WS instance can accept the cancel and any worker replica can hold the generation; Redis is the sole rendezvous point, with no ordering dependency.
• Three explicit re-checks: between every chunk mid-stream, at stream end, and before persistence, each checks streamCanceled.Load(); the external path mirrors the same three spots. This makes “a canceled request never publishes done” hold under all interleavings.
• Panic fallback: recoverStream acts as the stream goroutine’s recover and can emit a terminal error carrying the messageID (the outer handler’s recover cannot — the ID does not exist yet at that point).
• Semantics explicitly accepted and documented: “completion may beat cancellation” — persistence done, done published, and only then the cancel arrives; it is treated as completed (see the RequestStreamCancel comment); linearizability is not pursued.
• Persistence boundary: first insertSyncedDMMessage lands the row in dm_messages, then chat.stream.done is published (with FullText) — when done reaches the client, this message is guaranteed to already be in history.
• Conversation lease: acquireConversationLease (a Redis mutex, TTL 10min) ensures only one generation per conversation across instances at a time; acquiring an occupied lock returns operation_in_flight; Redis failure is fail-open.

Reply Size Governance

Inline replies are capped at 500 runes and fail-closed without truncation (deltas may already have been shown to the user; a truncated done text would disagree with what was shown); when the client declares accepted_modes=["object_ref"], oversized replies are externalized to object storage and sent as reply_externalized (recoverable).

Topic 7: Idempotency Gate and Dual Audit for LLM Paid Side Effects

Location: lib/aiagent/tool_executor.go, lib/toolcatalog/, lib/toolaudit/.

Problem Background

AI tool calling lets the LLM trigger real side effects (HTTP calls into customer business systems: refunds, orders, status changes). Such executions face three kinds of duplication at once: broker at-least-once redelivery, the LLM replaying the same call from context, and recovery after a process crash. “Refunding twice” is an unacceptable incident. Enterprise scenarios also demand human confirmation for high-risk operations (suspend → admin approves → resume).

Why It Is Hard

1. the idempotency key must be content-addressed: a redelivered message and an LLM-regenerated identical call must hit the same key, yet the byte representation of the arguments JSON may differ (whitespace, key order);
2. suspend-resume is a cross-process state machine: the LLM conversation cannot wait forever, admin approval may come minutes later, and the execution row must sit safely in an intermediate state meanwhile;
3. the ordering of audit versus execution is a security boundary: execute when the audit write fails? After a timeout, did the side effect happen or not?

Implementation

• Unified audit-first, fail-closed: beginAgentToolAudit writes the attempted event; if that write fails, execution is refused (no audit, no side effect); on completion, the completed event is written. Parameters and requests are stored only as SHA-256 digests (privacy + row width), and the idempotency key is a digest too.
• Content-addressed idempotency key: sha256(bot + userMessageID + toolID + name + arguments) → ClaimAIToolExecution: if a terminal state already exists (succeeded/failed/degraded), the persisted result is replayed directly and nothing re-executes — redelivery/replay returns the previous result, not a second side effect.
• Human-confirmation state machine: when a tool is marked RequiresConfirmation, claim returns pending_confirmation to the LLM (the conversation continues, telling the user to await confirmation); after the admin calls ApproveAIToolExecution on the admin side, ExecuteApprovedExecution resumes from the same execution row (bypassing the confirmation gate but preserving the full audit chain).
• SideEffect tri-state: not_started / committed / unknown — timeout or fallback scenarios are explicitly marked unknown (the side effect may have happened); never pretend to know; Decision: allowed/denied/pending.
• Catalog contract as single source: lib/toolcatalog defines the stable authorization contract for 20 tools (Key/Version/RequiredScope/five-level Risk/RequiresConfirmation/AllowedActors); the HTTP, MCP, and Agent adapters all project this catalog; OpenAPI inbound writes share the same tool_call_audits lifecycle evidence table.
• Permission scopes: PermissionScopes from the WS session, supporting * / module:* wildcards.

Similar Idempotency Gates

kb.ingest (document ingestion) uses TryClaimKBDocumentForIngest CAS state transitions, and call.analyze uses TryClaimCallRecordForAnalysis — one DB claim gate per paid/non-idempotent subject is the project’s fixed pattern; ai.request has no claimable row and relies on the AckWait budget as a fallback (the ops docs state its redelivery cost explicitly: the customer receives a second reply + a second paid completion).

Topic 8: Concurrency Correctness of Customer-Service Session Boundaries

Location: lib/dblib/cs_session_db.go, lib/aiagent/cs_session_hook.go, cs_session_closer.go, handover_sweeper.go.

Problem Background

IM DMs are an endless stream, but CS KPIs (first-response time, session counts, the satisfaction denominator, summaries) need “one service engagement” to have boundaries. The boundary must be overlaid without touching the DM data model, and multi-replica workers scanning and closing concurrently must not err.

Why It Is Hard

1. the “at most one in progress” uniqueness constraint has two traps on MySQL: unique indexes treat multiple NULLs as distinct (does not block concurrent double inserts); a TIMESTAMP sentinel has a 1970 lower bound and time-zone drift issues;
2. closing runs an LLM summary (up to 60s) — if a new message arrives during the summary, the aggregation window is wrong;
3. multi-replica sweepers contend for the same session, and two phases (closing + memory write-back) share one ticker and lock — a mis-set budget permanently starves one of the phases;
4. an aggregation window over a time range excludes the opening message itself (started_at differs from the DB-defaulted created_at by milliseconds).

Implementation

• A closed_seq integer marker instead of NULL/timestamps: in-progress is always 0; combined with UNIQUE (conversation_id, closed_seq) this guarantees at most one in-progress session; on close it is set to the row’s own id (unique per row), so the slot is freed and history rows never conflict. “Ensure an in-progress session exists” is a single INSERT ... ON CONFLICT DO NOTHING — atomic and cross-driver; concurrent double inserts open only one row.
• The hot path does exactly one thing: once a DM is persisted and judged a CS session, one ensure-upsert runs; subsequent messages only monotonically advance the last_message_seq activity watermark; aggregates like message_count/first_response_at are not written on the hot path — at close, a single SQL recomputes them from an id range of dm_messages (an id range, not a time range — a time range would miss the opening message itself).
• Optimistic-concurrency close: CloseCSSessionIfActivityUnchanged carries WHERE last_message_seq = ? — if new messages arrived during the 60s summary the watermark has changed, the close fails, and the next round retries; aggregation and the closing UPDATE compare within the same window, so no message falls into the gap between aggregation and UPDATE.
• Two-phase budget split: a single Redis lock (TTL 600s, Del at the end of each round; TTL is only a crash fallback) with single-flight; the two phases share tickStart, and phase 1 (session close, idle 5min) gets an extra sweepTickBudget/2 = 120s sub-budget — otherwise 20 due sessions × 12s summaries eat the whole 240s budget and phase 2 (memory write-back) never runs again. Worst-case round ≤ 300s < lock TTL.
• Two-sided scan window (idleBefore / staleAfter): the 7-day lower bound is correctness, not optimization — fully extracted sessions never leave the candidate set (routing row stays human + last_bot_id non-empty), sorting oldest-first keeps them perpetually at the front, and once they cumulatively fill each round’s cap new sessions never get a turn (no error, no log, no metric; the human→AI memory write-back is permanently stalled). The lower bound knocks such sessions out of the window in one pass.
• Explicit enumeration of watermark advancement: WS DM sends and the two bot reply paths advance it; the three paths deliberately not advancing (survey-card / rating-nudge DMs inserted only after session close; extsync history backfill never opens a session) each carry a comment; a known gap (RTC call-record insertion does not advance the watermark) is documented, not fixed — fixing it would be a behavior change.
• Summary failure does not block closing: the boundary value (duration/counts/first response) does not depend on model availability; the summary has its own 60s ctx timeout (preventing a gateway that “accepts the connection but never emits a byte” from wedging the ticker forever and taking down memory write-back with it).

Group CS Extension

cs_sessions.group_id (0=DM) is the single source of truth for “which group’s session this is” (forbidden to reverse-parse or LIKE-prefix the conversation_id — two sources of truth inevitably drift; and SQL reverse-parsing defeats index seeks); group session metrics are not equivalent to DM ones (message_count counts only the customer’s own messages — chat_group_messages cannot attribute which customer an agent’s reply was for); the docs state “not horizontally comparable”.

Topic 9: Budget Degradation Ladder for Memory Injection

Location: lib/aiagent/memory_retrieve.go, memory_writer.go, memory_profile.go.

Problem Background

AI long-term memory injects three layers into the system prompt: the structured customer profile, the rolling summary, and pgvector long-tail facts. The prompt budget is limited (block cap 1200 runes); when it is exceeded, content must be dropped — but commitment/taboo-type profile entries hand-written by agents (source=‘operator’) must never be dropped: that is “this customer was promised a refund” level information, and dropping it means the AI rejects the customer again.

Why It Is Hard

This is a closed-loop design problem where the read-side constraint must be pushed back into the write-side API:

1. if the read side marks operator/always_inject as undroppable while the write side has no total cap, three privileged fields of 400 chars each can exceed the budget, and the degradation ladder would drop all vector facts, then all summaries, and finally truncate even the privileged fields themselves — no logs, no metrics, and every turn of the customer’s conversation runs without memory;
2. “the model never overwrites operator values” must be guaranteed at the upsert SQL layer (syntax differs across the two dialects);
3. is the extraction watermark stored per (bot, user) or per conversation? A pair-level slot holding a conversation-level count means new conversations are never extracted while big conversations extract on every message.

Implementation

• Four-level degradation ladder (read side, formatMemoryAsSystem): ① drop free-text long-tail facts one by one → ② drop historical summaries whole → ③ drop non-privileged profile fields from the tail one by one → ④ privileged fields are never proactively dropped; only under a pathological configuration where the privileged fields alone exceed the budget are they truncated as a last resort (the comment marks this as a forced floor).
• Write-side total cap (MaxOperatorProfileTotalRunes = 400): over-limit agent writes are rejected, not truncated, with the cap reported; existing over-limit data uses OperatorProfileTotalBefore to allow monotonic “decrease-only” convergence. This is the necessary condition for the read-side “never drop” promise — the undroppable set itself must be unable to overflow the budget.
• Operator priority guaranteed at the SQL level: Postgres ON CONFLICT ... DO UPDATE ... WHERE source <> 'operator' OR EXCLUDED.source = 'operator'; MySQL ON DUPLICATE KEY UPDATE has no WHERE, and bare column names on the right side read old values — equivalent semantics built from exactly these two facts.
• Watermark stored per conversation (ai_memory_conversation_marks, keyed (bot, conversation)): summaries accumulate per (bot, customer) across conversations (the profile is right), but the trigger compares the conversation’s message count, so the scopes must match; extraction/parse failures also advance the watermark (otherwise a persistent outage degrades into one LLM call per message); baseline read failure is fail-closed (treating an unreadable baseline as 0 = one call per message).
• MySQL compatibility: the profile and summary tables have no vector column and work across drivers; vector recall is Postgres-only, and VectorRecallSupported() on MySQL short-circuits before the paid embedding call.
• Human-AI memory loop: the product has no “conversation ended” event; a sweeper scans handed-over conversations to write memory back (agent speech is classified as assistant by decodeHistoryRow, so human-handling conclusions naturally enter the summary); transfers keep last_bot_id to record “who took over last” (the memory primary key is (bot, user); without it, memory would be written to the wrong bot).

Topic 10: License Signature Verification and Compile-Time Symbol Isolation

Location: lib/license/, Makefile, scripts/verify_sku.sh, cmd/packagegen.

Problem Background

The trust model of commercial software delivery: the free and paid versions are built from the same codebase, and it must hold that — ① editing configuration cannot unlock paid features; ② swapping public keys cannot forge licenses; ③ the free binary truly does not contain the paid code (not hidden — absent); ④ expiry does not take the service down (degrade, keep the core).

Why It Is Hard

1. the trust anchor must be compiled into the binary: any runtime-modifiable configuration (public key path, enforce switch) is attack surface;
2. “not in the binary” needs a verification method — what can still be probed after Go strips symbols? The embedded frontend JS bundle contains many behavior strings (ai_bots, processRequest) that make plain grep pure false positives;
3. expiry semantics cut both ways: checking at startup is easy, but when the license expires on day 3, should the service stop? Stopping angers customers; not stopping is breach.

Implementation

• Three layers of isolation working together:
  1. Build tags: AI/Ultimate-exclusive files are tagged file-wide; shared files use function-variable hooks (default no-op, overridden by init() under the tag); neutral helpers (lib/idutil) are callable from basic;
  2. ldflags key burning: -X lib/license.embeddedPublicKey=<key>; in release builds resolveLicensePublicKey() returns only the embedded key — configuration/environment variables/files are all ignored. An empty key (free basic) means every signed license is rejected, permanently locked to basic;
  3. pclntab package-path probing: verify_sku.sh probes gated package paths (function names survive -s -w stripping inside pclntab), not behavior symbols (frontend JS false positives) or string constants (license aliases, table names, command names are supposed to be in basic anyway). CI runs it on every push; Docker images are additionally verified not to contain the ai_worker/sync_worker binaries.
• Ed25519 verification (signed.go): SignedDocument{Algorithm, Payload, Signature}; the signature is computed over the byte sequence with “whitespace compressed but keys not sorted” — the verifier re-compresses the same payload stored in the document to obtain identical bytes. Key constraint: this must never be changed to re-serializing the parsed object, which would change the byte sequence and invalidate every issued license (sign the stored bytes, not the semantics). The public key supports PEM/base64 (4 variants)/hex forms; base64 and hex are strictly discriminated as fixed 32-byte lengths.
• Release/dev dual posture: trust.go defines the release defaults (distrust plaintext configuration, enforce verification); trust_dev.go (the devlicense tag) inverts them — that tag appears only in dev targets such as make run/build and never in release builds.
• Dual-track expiry semantics:
  • Require/Allows (expiry-unaware): describes “what was granted at signing time”, used for startup fail-fast (ValidateViperEntitlements infers from configuration which paid features this process enables and refuses to start if unlicensed);
  • PermitsFeature (expiry-aware): runtime defense-in-depth; after expiry it degrades to the defaultFeatures(EditionBasic) decision — core IM stays alive, Pro features shut off immediately, a degradation warning is printed, no restart needed. The two contracts must not be mixed.
  • Admin console: servers/admin.LicensePermitsConsole checks admin_console / rbac / audit_log / data_governance and validity against the same installed license; with no license loaded it refuses; the explicitly installed dev-state enforce=false preserves the integration-testing behavior. With no valid console license at startup it returns (nil, nil) — no admin bootstrap, no background tasks, no port listening, while core IM keeps running. HTTP engine-level gating covers login, extended APIs, and static assets; new requests after mid-run expiry return 403 / license_unavailable; the scheduled announcement and data-retention jobs re-check before starting the next item. Operations already in flight are not forcibly interrupted. The license file is still loaded at process start; changing the file requires a restart.
• Single-source-of-truth generation chain: lib/license (capabilities/quotas) + packaging/editions.yaml (display metadata) → cmd/packagegen generates the marketing-site SKU data / feature-comparison docs / sku.mk; CI packagegen-check prevents drift, and unit tests re-render and diff the same artifacts directly (no separate golden — the two guards cannot disagree).

Known Boundaries

Enforced resource quotas: ai_bots and kb_documents are enforced in the shared database write layer, covering the legacy admin API, the console, config bootstrap, and draft publication. EffectiveQuotaLimit uses the same runtime license: missing or expired degrades to Basic; the explicitly installed enforce=false dev license is unlimited; 0 forbids additions; only -1 means unlimited. Issued quotas and the product catalog are unchanged.

• Bots count all existing rows in the database; disabled bots still occupy quota; the knowledge base counts ordinary documents across the whole library, not split per bot, and pending/error documents count too. Deletion releases quota; when existing resources exceed the limit nothing is auto-deleted — updates and reductions are allowed, additions are blocked. Existing supported KB/RAG deployments remain PostgreSQL.
• license_quota_locks fixed row locks + READ COMMITTED transactions serialize counting, checking, and writing for instances sharing the same primary database and running this version; if the primary/lock table is unavailable, writes fail. The lock table is created by 20260924_create_license_quota_locks.sql; upgrades must apply the migration first.
• Draft saves do not pre-reserve live-document quota; publication checks net growth against the final count, and bot edits, KB add/update/delete, and version-state changes commit in one transaction. Full quota still allows equal-quantity replacement; quota rejections/SQL failures roll back the whole thing, and concurrent or duplicate publication of the same draft cannot create documents twice. Post-publication enqueue still follows the existing broker flow and is not part of that database transaction.
• Ephemeral evaluation vector payloads do not consume user document quota; they are constructed from real eval run IDs by a dedicated internal method, and regular bot/document writes reject the reserved eval_run: namespace. API over-quota returns 403 / license_quota_exceeded, including quota and limit.

Gateway instance and connection quotas: lib/licensecapacity meters atomically within a single Redis DB.

• cmd/websocket claims an instance slot before starting the listener; every WebSocket handshake first reserves a connection slot, covering both the upgrading and the connected states. Failed upgrades return the slot; disconnects close the transport first and then decrement; duplicate releases and Redis command retries never double-decrement other connections. Instance counts do not include standalone persist/sync/AI workers.
• The ledger and the optional instance directory used for ops are separate, using a fixed Cluster hash slot, a random process identity, and Redis server-side time. Leases are 45 seconds with 10-second renewal; the local monotonic clock provides a more conservative 30-second service deadline. Redis commands use a 2-second context deadline, with the client socket deadline enabled.
• After Redis errors, a missing ledger, or a lost lease, admission stops, connections close, and the process exits — no automatic re-claiming. Main HTTP, the admin console, and gRPC check the local lease when a request enters; WebSocket send/receive checks too; requests already executing have no transaction-level fencing per database/broker side effect. Normal shutdown closes admission and transports first, then tears down services, and finally returns the instance slot; crashes are recovered by lease expiry.
• Over-limit connections return 429 / license_quota_exceeded; capacity-dependency failure returns 503 / license_capacity_unavailable. Missing/expired licenses degrade to Basic; the explicit dev license is unlimited but still requires an available ledger. Lowering the connection allowance does not forcibly kick old connections; admission resumes after draining below the limit; lowering the instance allowance retains a fixed subset ordered by random process identity, and the remaining instances stop serving.
• Deployments must use the same version, the same license, and the same Redis DB; mixed old versions do not form an enforcement boundary. imcore:{license-capacity}:* is a reserved ledger — Redis should avoid key eviction and protect write access. Redis flush/eviction, server clock jumps, primary-replica failover, or replica rollback can briefly skew counts before old processes notice the invalidation; therefore consensus-grade global mutual exclusion or activation binding across independent Redis instances cannot be claimed. This matches Redis’s official documentation on the limits of asynchronous replication for mutual exclusion during failover (https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/).

Daily message-volume quota: send admission is metered atomically by Redis Lua in lib/licenseusage. The Redis server-side time determines the UTC day; ordinary DMs and group/broadcast sends each count once; AI replies (including deterministic refusals and external workflows), satisfaction surveys, the first slice of OpenAPI streams, and outbound SMTP emails each count once before their first side effect. Sharding, receipts, typing state, heartbeats, internal fanout/redelivery, history sync/import, and system lifecycle events do not count. No refunds after admission — database, broker, SMTP, or streaming failures may make delivery undecidable; idempotent ACKs cached by callers return before the quota, and retries after the cache expires may be counted again. Missing ledger fields, service unavailability, or server-time rollback fail closed; the ledger survives restarts and license changes. Inbound external mail and history imports do not count; outbound mail does. The mechanism depends on a protected Redis and untampered service code, and cannot stop an operator with Redis/source access from editing the ledger directly or removing the check.

License time rollback protection: the entitlement windows for features and resource quotas uniformly use an in-process monotonic-clock guard; rolling the system clock back mid-run cannot reactivate an expired license; a license whose issue time is in the future stays at the Basic baseline until issued_at arrives. After Redis is initialized, the server TIME is sampled every 10 seconds and the highest microsecond time persisted to imcore:{license-time}:floor:v1; when samples lag by more than 30 seconds, fall below the persisted floor, when server time rolls back, or when Redis is unavailable, license checks are forced fail-closed until a new sample catches up to the floor. It is a sampled external clock on a protected Redis, not consensus time; deleting the time key, restoring an old Redis snapshot, failover, or an operator with Redis write access can still influence time.

License ledger binding: the gateway-capacity and daily-message ledgers store the SHA-256 fingerprint of the full license configuration. A different license fingerprint appearing in the same Redis billing domain stops further admission; renewals or accidental mixed deployments cannot reset counters by changing key names. Different Redis deployments remain different billing domains — a global deployment binding cannot be claimed.

The signed license payload may optionally include deployment_id. When present, every process must be configured with the same deployment.id or IMCORE_DEPLOYMENT_ID; a missing or mismatched value fails closed immediately; when absent, old licenses stay compatible. What it binds is an operator-defined deployment name; the real root of trust remains the issuing private key and the protected Redis/configuration.

License key rotation and revocation: signed documents may optionally carry key_id; configured public keys support a key_id:pubkey,key_id-2:pubkey rotation set; legacy documents without key_id remain compatible as long as exactly one trusted public key exists. An optional signed revocation list license.revocation_file can reject licenses by license ID or deployment ID before startup; license_tool revoke generates the list. Both the revocation list and public keys must be protected; file updates require a restart to take effect. It provides offline revocation for self-hosted deployments but does not stop someone with local write access from replacing the license, public key, and revocation list simultaneously.

Binary integrity and obfuscation: lib/integrity can generate SHA-256 records for every service binary in the release directory and sign the manifest with a separate Ed25519 private key. With hardened builds enabled, the verification public key and the enforce switch are written into the binary via ldflags, and the process verifies its own files before installing a license; replacing binaries or the manifest is therefore detectable, and editing ordinary configuration cannot disable verification. cmd/integrity_tool handles key generation, signing, and offline verification; make integrity-* builds through an explicit garble toolchain entry; the package-integrity-* targets place the signed manifest into config/. Obfuscation only raises reverse-engineering cost; it is not a cryptographic boundary. An attacker who can simultaneously rewrite binaries, the manifest, the embedded public key, or the boot environment can still bypass it; production should combine read-only images, minimal file permissions, and Release cosign/SHA256/SBOM verification.

Remote license service recommendations for self-hosted deployment: well suited as an optional “online renewal / revocation / device registration” control plane, not as a hard dependency of every message or request. Offline customers, intranet isolation, license-service outages, and cross-region latency all require the locally signed license to keep working independently; the online service only issues short-lived or revocable Ed25519 credentials, the last valid credential is cached locally with an explicit grace period, and after the grace period only paid capabilities are disabled. The server should not treat quota state in Redis or the database as the licensing root of trust, nor accept client-uploaded public keys; publisher private keys stay in the license service/HSM, and instances authenticate via mTLS or one-time registration tokens. This adds revocation and tenant lifecycle management, but cannot stop an operator with root on the customer host, binary write access, or the full source from bypassing local checks.

Not yet implemented: license binding across independent deployments. License time rollback across restarts is covered by the Redis-persisted floor but does not constitute consensus-grade time or tamper-proof storage. Binary integrity and obfuscation provide a hardened build entry point, but they still take an unmodified boot environment and a protected filesystem as their boundary; they cannot stop someone with database/Redis write access from editing ledgers directly, nor someone with the source and build capability from removing the checks.

Topic 11: Audit Log HMAC Hash Chain

Location: lib/auditlog/, lib/dblib/admin_audit_log_db.go.

Problem Background

The admin console’s audit log is accountability evidence: who changed what, when. But the log lives in the DB, and a DB administrator (or an attacker with DB privileges) can quietly alter or delete a row. A hash chain makes any tampering detectable — the classic compliance application of hash chains outside blockchain (RFC 6962-style thinking).

Why It Is Hard

1. the signed input must match byte-for-byte “the value read back from the database” — production MySQL TIMESTAMP columns have only second precision; hashes signed from Go-side sub-second timestamps cannot be reproduced on read-back (the v1 chain died here);
2. chain-tail races under concurrent inserts: two logs claiming the same predecessor fork the chain;
3. verification’s own consistency: the verifier scans the log rows and then reads the chain-tail state; if a new log commits between the two autocommit queries, a spurious tail mismatch appears — a verifier that cries wolf is worse than no verifier at all;
4. audit writes must not drag down requests (async), yet must not be lost (accountability evidence) — the backpressure policy is the opposite of ordinary logging.

Implementation

• Write pipeline: a goroutine pool (2 workers / 1024 buffer) writes asynchronously; Submit applies blocking backpressure and never drops — slowing requests down is preferable to silently losing audit entries (deliberately the opposite of webhookaudit’s best-effort; the two differ in semantics: audit is evidence, webhook observation is droppable telemetry). RedactJSON recursively masks credential-like keys (password/secret/token) before they enter the immutable log.
• Single-transaction chained signing (InsertAuditLog):
  1. ensureAuditChainState INSERT IGNORE initializes the chain-state row;
  2. SELECT last_entry_id, last_hash ... FOR UPDATE — a row lock serializes the chain tail; concurrent inserts queue here, so the chain cannot fork;
  3. predecessor-hash linking (only v2-signed rows participate);
  4. EntryHash = HMAC-SHA256(integrity_key, canonical), where canonical joins 14 fields with the \x1f unit separator (field values cannot ambiguously contain the separator);
  5. INSERT the row + UPDATE the chain state in the same transaction.
• Time normalization: created_at is normalized to UTC truncated to seconds — aligned with MySQL TIMESTAMP column precision, guaranteeing “signed input == read-back value”. The v1 chain was scrapped and rebuilt over non-reproducible sub-second precision (signature_version=2); legacy rows are excluded from verification.
• Consistent-snapshot verification: VerifyAuditChain wraps the log scan and the chain-state read in a REPEATABLE READ read-only transaction — eliminating spurious mismatches from concurrent commits. It verifies prev_hash linkage row by row + constant-time hmac.Equal comparison + reconciles against the chain-tail state.
• Monitoring-friendly: the Integrity endpoint returns 200 + valid:false on tampering (not 500) — tampering is anticipated evidence, not a transport error; monitoring alerts on the body, not the status code.
• Key resolution chain: governance.audit_integrity_key → legacy admin.audit_integrity_key. admin.console.jwt_secret is no longer a fallback; the two purposes must use independent keys, so a leaked JWT secret cannot also break the audit chain’s tamper-evidence.

Topic 12: Transaction Boundaries for Data-Governance Erasure and Archival

Location: lib/governance/, lib/dblib/data_governance_execution.go, data_governance_message_retention.go, cmd/governance.

Problem Background

GDPR/PIPL scenarios: when a customer exercises the right to erasure, their messages, profiles, AI memory, and CS sessions must be verifiably deleted; meanwhile other people’s conversations must not be destroyed (deleting customer A must not mangle the agent’s conversation with customer B). A second requirement sits alongside: messages archived to object storage under retention policies are deleted from the DB and must be restorable.

Why It Is Hard

1. DB transactions and object storage cannot share a transaction: erasing a customer means deleting the objects they uploaded — what if the DB rolls back after the objects are gone? Archival inverts it: the objects upload fine but the DB commit fails, leaving orphan objects;
2. pseudonymization must be irreversible yet stable (two erasures of the same person produce the same pseudonym — linkable for statistics, not identifiable);
3. archive restoration is high-risk: the manifest itself lives in the DB, and a tampered manifest must not become an injection channel for arbitrary INSERTs;
4. subject roles are asymmetric: a customer’s deletion = deleting rows; an agent’s “deletion” = only pseudonymizing records they handled (other customers’ conversations must stay).

Implementation

• Data-asset inventory driven: 14 DataAsset declarations of table-level metadata + SubjectColumn precisely declaring each identity column’s delete | pseudonymize action — the executor iterates the inventory rather than hardcoding.
• Erasure transaction (EraseSubjectDataWithStore): legal-hold pre-check block → collect attachment object keys → within a single transaction: DM conversation-ID rewrite (dm:a:b canonical ordering; user ids may contain :) → pseudonymize 4 message tables (sender replacement + search_text + payload JSON rewrite — the payload replaces only string values exactly equal to the userID, never free-text substring replacement) → group/community ownership transferred to the pseudonym → 44 delete specs + 5 pseudonymization specs (rows about other people are preserved, only the subject’s own columns replaced; CASE two-arm UPDATEs cover both directional columns at once).
• Object-storage compensation: data_erasure_markers records the object-key list; ReplayErasureMarkers replays deletion for late-written objects that arrived after erasure — eventually consistent compensation, accepting a “DB succeeds first, objects deleted later” window.
• Pseudonymization function: "deleted-" + HMAC-SHA256(salt, userID)[:20] — stable per person with a stable salt, irreversible without it.
• Archival loop: a transaction with SELECT ... FOR UPDATE locks the batch → packs an envelope with related receipts/audiences/mentions rows (SchemaVersion/Columns/Rows/Related/Policy) → JSON + SHA-256 checksum → upload to object storage at governance/message-archives/yyyy/mm/dd/<uuid>.json → write the manifest → delete related rows and primary rows → commit; if uncommitted, a deferred cleanup deletes the uploaded objects (reverse compensation). Restoration triple-checks: checksum + envelope/manifest consistency + a table/column double allowlist (validArchiveColumns, defending against tampered-manifest injection) → idempotent re-insert with INSERT IGNORE.
• Fail-closed without overstatement: execution_enabled defaults to false; legacy/unavailable tables degrade gracefully to Warnings instead of failing the whole run (isUnknownTableError) — reports honestly reflect coverage.
• Both the CLI and the admin entry support preview (default) / execute (explicit); the backup-restore drill script scripts/governance_drill.sh requires an explicitly empty DSN for the target database and never drops anything.

Topic 13: SSRF and DNS Rebinding Protection

Location: lib/netguard/netguard.go.

Problem Background

The system has many places where administrators/operators configure URLs and the server initiates requests: message callbacks, push webhooks, speech transcription, AI webhook providers, beforeChat hook webhooks, KB document URL fetching. Once admin credentials leak, these URLs become SSRF springboards — an attacker points the URL at http://169.254.169.254/ (cloud metadata) or an intranet service. Every outbound network point must pass unified anti-SSRF validation.

Why It Is Hard

A naive “check whether the host is an intranet IP” does not survive DNS rebinding: at validation time the domain resolves to a public A record (passes validation); at actual dial time DNS returns an intranet address (a second resolution with an extremely short TTL) — the resolution window between validation and dialing (TOCTOU) is the root problem. HTTP_PROXY in environment variables also triggers another resolution outside the guard, a second bypass path.

Implementation

• GuardURLHost (pre-request validation): resolves all A/AAAA records and rejects loopback / link-local / RFC1918 / multicast / unspecified addresses; validates after resolution rather than string matching (a public domain resolving to an intranet IP is caught too). Failed domain resolution is rejected outright.
• NewGuardedHTTPClient (pinned at dial time, closing the TOCTOU window): the custom Transport’s DialContext re-resolves and validates, then dials the validated IP directly — the URL stays a hostname (the standard transport preserves the Host header and TLS SNI; HTTPS certificate validation is unaffected), but the actual connection goes to the validated IP, so a second rebinding resolution never happens. Proxy = nil explicitly disables environment proxies, preventing the proxy from resolving outside the guard.
• CheckRedirect: rejects cross-authority redirects (http://public/redirect→intranet chains are blocked as well).
• All outbound callers reuse this uniformly: callback, push webhook, speech, AI workflow provider (the guardWorkflowHost seam), KB URL fetching, hook webhooks. The fixed recipe when adding a new outbound point is the netguard.GuardURLHost + NewGuardedHTTPClient pair.

Design Trade-offs

Validation is a fast-fail before the request (configuration errors surface early); dial-time pinning is the real security boundary — both layers are required; neither alone suffices. In TLS contexts SNI/certificate validation depend on the hostname, so the implementation chose the transparent “URL keeps the hostname, socket uses the IP” approach rather than rewriting the URL (rewriting would break certificate validation).

Topic 14: Privilege-Escalation Protection for Webhook Callbacks

Location: servers/websocket/chat_message_callback.go, lib/chatseg/.

Problem Background

The pre-send callback (message_callback) allows an operator-configured webhook to wholly rewrite user messages (text, type, segments) — a legitimate capability (content processing, translation, risk-control rewriting). But imcore’s message segments include privileged types: quick_reply/ card (rich interactive segments carrying “click to send text on the user’s behalf” buttons) and AI citation segments. If a compromised operator webhook could rewrite an ordinary user message into a button-carrying card, that would escalate the operator’s rewriting capability into the bot/agent’s sending capability — a user clicking the button would “personally” say whatever the button says.

Why It Is Hard

Trust has two tiers: rewriting capability (operator level) and send-gate permission (bot/agent level). The rewrite path naturally bypasses the send gate (the message has already passed it), so the validation layer must guarantee that rewriting cannot cross capability tiers. And message segments are a shared flat union (ChatSegment) — a text segment can technically smuggle an entire Actions array — so validation must cover every smuggling path.

Implementation

• Callbacks cannot mint privileged segments: if the message rewritten by applyMessageCallbackReplacement contains quick_reply/card interactive segments that were absent inbound → the rewrite is rejected; AI citation segments likewise (applyCallbackSegmentsIfNotNewlyInteractive). “Interactive capabilities that were not inbound cannot be created by the callback.”
• The send gate and validation share the same rule set: lib/chatseg (a standalone package, because dblib cannot import servers/websocket — the dependency direction dictates where the shared package lives) provides ValidateInteractiveSegments:
  • message-wide action-id uniqueness (cross-segment dedup — historically, server-constructed cards that skipped cross-segment dedup produced a wrong-response bug, the direct reason this package exists);
  • non-interactive segments must prove Actions/Rows/Answer are all empty (blocking union-struct smuggling);
  • URLs limited to http/https (blocking javascript:), a style allowlist, per-field rune caps.
• Answer is a server-authoritative field: inbound client messages carrying Answer are always rejected — otherwise a sender could self-declare “already answered”, rendering the button grey from birth. AccessURL is the precedent for the same approach.
• Interactive segments are not editable: ordinary messages cannot be edited into interactive ones either — the edit path goes through neither prepareSendContent (where the DM check lives) nor the send gate; “send text first, then edit it into a card” was a complete bypass chain; and allowing button-text edits = allowing tampering with “what the customer actually clicked back then” after they answered.
• Answer-side on-behalf sending is executed by the server: chat_interactive_reply constructs a DM request and calls ChatDMSendController directly (the whole chain of idempotency/moderation/ session hook/quota/offline push is inherited automatically) instead of letting the client send it — the invariant “button id == the sentence actually sent” cannot be entrusted to the client (satisfaction statistics depend on it). Failures distinguish transient vs permanent: permanent ones roll back Answer so the customer can click again; otherwise it is a dead card that stays grey forever.
• Asymmetric send-gate design: on the send side an empty allowlist fails closed (rich interactivity is a new capability with no “pre-upgrade” state to inherit — copying the empty-table-allows convention would let anyone with a routing row send an “I agree to pay 5000” card); on the answer side there is no allowlist gate at all (the answerer is the customer, never the agent).

Topic 15: Dual-Dialect Database Migration System

Location: lib/migrate/, cmd/migrate, migrations/, scripts/check_migrations.sh.

Problem Background

The same product must support MySQL and PostgreSQL (+ Dameng DM8 reusing the MySQL dialect); migration files must correspond one-to-one across dialects, and execution semantics must adapt to a fundamental difference between the two databases — Postgres DDL is transactional and rollback-able; MySQL DDL implicitly autocommits and cannot be rolled back.

Why It Is Hard

1. consistency: drift between the two directories’ file sets is a silent incident (a table existing on only one side);
2. atomicity fork: Postgres can do “file execution + bookkeeping” in a single transaction; on MySQL a mid-run crash leaves “DDL executed, bookkeeping not written”, and a rerun errors on duplicate DDL;
3. onboarding existing databases: legacy customers created tables by hand; running migrations directly fails across the board.

Implementation

• Directory conventions: the 47 files in the root are the frozen MySQL historical set; new migrations land in both migrations/mysql/ and migrations/postgres/; migrations/manual/ is manual-only. FilesFor: mysql/dm runs root + mysql/; postgres runs postgres/ only (its 20250101_create_schema.sql is a self-contained merged version).
• Bookkeeping: schema_migrations(filename PK, applied_at) in dialect-compatible DDL; Applied/Pending bookkeep by relative path and apply in lexicographic order.
• Execution fork (Apply): postgres — file execution + record commit atomically in one transaction; MySQL — execute first, record after, with the mid-crash recovery path explicitly documented (a rerun errors on duplicate DDL → manual reconciliation or Baseline).
• Baseline: onboarding existing databases — mark every file applied without executing.
• CI set consistency: check_migrations.sh verifies that the mysql/postgres file sets correspond; single-dialect files must be registered in an explicit allowlist (ALLOW_MY_ONLY / pgvector-only and other “known intentional differences”); root-directory freeze checks (new filenames must not be later than the freeze line).
• Runner UX: -status / -dry-run / -baseline / -until <exclusive prefix stop point> / -timeout (default 10 minutes, extendable for big-table backfills); DSN parsing identical to dblib; migrations are embedded into the binary via go:embed (deployment packages need no SQL directory).
• Runtime compatibility fallback: sql_dialect.go (Topic 4) + the DAO layer’s isUnknownColumnError detects old schemas and automatically degrades to the legacy column set, so old and new schemas can coexist and serve during rolling upgrades.

Topic 16: Protocol-Driven SDK Generation for Four Platforms

Location: clients/protocol.yaml, cmd/protocolgen, sdk-api/, scripts/sdk_api_surface.py.

Problem Background

SDKs in four languages (TypeScript/Dart/Swift/Kotlin) + Chinese/English API docs + webhook contracts + JSON Schema must all stay consistent with the server’s ~130 WS commands. Manual syncing inevitably drifts: the server adds a command and the SDKs and docs catch up three months later (or never).

Why It Is Hard

1. drift detection needs a bidirectional closed loop: checking only “everything in the manifest is registered” is not enough; “everything registered is in the manifest” (or explicitly allowlisted) must also hold — skip the reverse check and undeclared commands become hidden APIs outside the contract;
2. API surface snapshots must be reviewable: diffing source code directly is too noisy; a stable public-API baseline must be distilled;
3. SDK behavior changes need a migration narrative: customers upgrading SDKs across four platforms need to know which default behaviors changed.

Implementation

• A single protocol manifest clients/protocol.yaml: commands + DTOs + webhook declarations + the unexposed allowlist — one file is the single source of truth for every endpoint.
• --check bidirectional cross-validation: manifest commands must already be registered in routers/ (forward); registered commands must be declared or allowlisted (reverse); DTOs aligned with models/chat_model.go; webhook contracts consistent.
• 16 files generated: four-language commands + four-language types, Chinese/English API_REFERENCE, WEBHOOK_CONTRACTS, contracts/webhooks.schema.json + examples, a Go webhook consumer example. CI make protocol-check + git diff --exit-code guarantees no drift — generated artifacts are committed to the repo; any hand edit gets overwritten by the next generation and caught by the check.
• API surface snapshots: sdk-api/snapshots/{web,dart,kotlin,swift}.api is the reviewable public-API baseline (--write regenerates; CI runs --check); default-behavior.json separately records default behavior — any change must come with a docs/sdk-migrations/<version>.md migration report; source-digests.json validates source integrity.
• Platform adaptation layer: the TS SDK has an isomorphic core + three platform entries — Electron (credentials in the main process; preload exposes only a restricted transport, so the renderer never touches ipcRenderer), React Native (AsyncStorage + AppState/NetInfo lifecycle), WeChat Mini Program (wx.connectSocket + async storage + wx.uploadFile); Dart/Swift/Kotlin each have a full implementation + UIKit.
• Compatibility policy: the SDK version line is independent of the server (clients/SDK_VERSION); the protocol is additive forward-compatible (any server ≥ v1.2.0).

Topic 17: Workflow Orchestration of the AI Reply Pipeline

Location: lib/aiagent/worker.go, worker_external.go, hooks.go, hooks_worker.go, handover_sweeper.go.

Problem Background

After a customer-service DM reaches ai_worker it passes through a dozen stages — rate limiting, transcription, rules, memory, retrieval, generation, tools, persistence — with a fork for “local pipeline vs external workflow platform” in the middle. “Orchestration” must answer three questions: how stage order is decided, where the fork happens, and how background cron jobs coexist. In this project these are not a freely composable DAG engine but three orchestration mechanisms that each converge.

Overall Pipeline

user DM → [ws] dispatchAIBotDM ──publish──► ai.request (workqueue: exactly one ai_worker consumes)
                                              │
                 ┌────────────────────────────┘
                 │  processRequest pipeline (fixed stage order)
                 ▼
   ①rate limit(Lua sliding window) ②speech transcription ③beforeChat rule chain ──stop/handover──► terminate
                 │
           ④ fork point (isExternalWorkflow?)
           ├─ yes → external workflow (Dify/Coze/webhook): the platform owns memory/RAG/tools; only query+session id are sent
           └─ no → native pipeline: ⑤memory assembly (budget degradation) ⑥KB retrieval (pgvector) ⑦history window (role attribution)
                 ▼
           ⑧ provider.Stream (SSE→chunk chan)
                 ▼
           ⑨ tool loop ≤3 rounds ──finish_reason=tool_calls──► ExecuteTool
                 │     (audit first → idempotent claim → execute → possibly suspend awaiting human approval; see Topic 18)
                 ▼
           ⑩ persist dm_messages → publish chat.stream.done (push: fan out per instance → local frame delivery)

   [ai_worker background] a sweeper with one ticker and two phases: session close (5min) → memory write-back (30min)
   [ai_worker background] routing-queue dispatcher (5s): cs_routing_queue → RouteCSConversation (Topic 19)

17.1 Fixed Stage Order and an Explicit Fork Point

processRequest is a fixed-order stage chain, with each relative position having an explicit reason:

• rate limit → transcription → hooks come first: the gates related to money and external calls pass first;
• the fork point sits after hooks and before local assembly (the isExternalWorkflow check, worker_external.go:29): the external workflow platform (Dify/Coze/self-hosted webhook) owns conversation memory, RAG, and tools, so local assembly ⑤⑥⑦ is skipped entirely and only one user query + the platform’s previous-round session id is sent (ai_bots_external_sessions stores the mapping keyed by (bot, conversation)); the reply still flows through the same chat.stream.* protocol — the WS layer and the frontend are completely unaware of who the backend is;
• the persistence boundary precedes publishing done: when chat.stream.done reaches the client, this reply is guaranteed to already be in dm_messages (insertSyncedDMMessage runs before the publish).

17.2 Declarative Rule Chain: beforeChat Hooks

The part closest to a traditional workflow engine — each bot can configure a set of when <condition> then <action> rules (ai_hooks table), evaluated at pipeline position ③. The core engine is the pure function evaluateHooks (lib/aiagent/hooks.go:120-146), with three semantic essentials:

1. stable priority ordering: sort.SliceStable ascending by priority (same priority keeps configuration order);
2. actions split into accumulating and terminating: a webhook action, when hit, accumulates and evaluation continues with the later rules; stop_reply / handover return immediately and terminate the whole chain (handover implies stop_reply), mirroring RongCloud Agent Hooks’ priority + stopReply semantics;
3. fail-safe down to the byte level: condition-JSON parse failures, unknown condition types, and unknown comparison operators all return false (comment at hooks.go:70-72: “a broken rule never fires rather than firing unexpectedly”) — broken configs don’t fire and don’t hurt the main flow.

Two condition kinds today (keyword substring match / turns conversation turns, supporting > >= < <= ==), deliberately kept deterministic — intent recognition is a documented future enhancement. Suppressed replies send a terminal chat.stream.error(code="blocked", detail=<hook name>) and are audited with status=blocked; webhook side effects execute in hooks_worker.go: SSRF guard + fail-open (a broken hook does not block the main flow).

17.3 Delegation to External Workflow Platforms

Providers split into two interfaces: LLMProvider.Stream (OpenAI-compatible, runs the full native pipeline) and WorkflowProvider.StreamWorkflow (dify / coze / webhook, forked after isExternalWorkflow). Both yield StreamChunk (the SessionID field exists specifically for platform session continuation); API keys resolve through the ai_worker.api_keys ref; dialing passes the SSRF guard first. Context isolation on the external platform is delegated to the platform (local history is never sent); handover/auditing/dm_messages persistence match the native path.

17.4 Multi-Phase Background Scheduler: One Ticker, Two Phases, Halved Budgets

ai_worker has no ad-hoc “one timer per job”; instead it uses a shared ticker + single-flight Redis lock + per-phase budgets (handover_sweeper.go):

• phase 1 = session close (idle 5 minutes, must be fast — summaries and satisfaction take effect ASAP); phase 2 = memory write-back (idle 30 minutes, must be accurate — no drawing conclusions mid-conversation). The two thresholds are deliberately different: shared ticker, separate configs;
• phase 1 gets a total budget/2 = 120s sub-budget — otherwise 20 due sessions × ~12s of LLM summary each eat the whole 240s budget, phase 2 fails its first budget check, and memory write-back is permanently stalled under that load;
• worst-case round ≤ 300s, which must be < the 600s lock TTL (the lock is Deled at the end of each round; TTL is only a crash fallback).

The routing-queue dispatcher (Topic 19) is another independent loop in the same process, ticking every 5 seconds without interference.

Topic 18: Streaming Accumulation and Controlled Side-Effect Execution for Tool Calling

Location: lib/aiagent/tool_loop.go, tool_executor.go, tool_builtin.go, lib/toolcatalog/, lib/toolaudit/.

Problem Background

LLM function calling lets the model trigger real side effects (HTTP calls into customer business systems: refunds, orders, status changes), while tool_calls arrive fragmented in the SSE stream. Two problems to solve: how streaming fragments reassemble into an executable complete call; and how to safely execute LLM-decided side effects under an untrusted decision source (redelivery/replay/crash — three kinds of duplication + suspension for human confirmation).

18.1 Schema Delivery and Streaming Fragment Accumulation

• toolsToSchema (tool_loop.go:49-69): ai_tools rows → OpenAI tools schema; an empty parameters_json is replaced with a permissive default schema; disabled tools are skipped;
• streaming accumulation is the first technical hurdle: the same call’s arguments JSON string is split across multiple deltas. accumulateToolCalls (tool_loop.go:14-34) accumulates into a map keyed by Index: ID/Type/Name overwrite only when non-empty, Arguments appends as a string; after the stream ends, sortedToolCalls outputs the complete call list sorted by index — without this step, half a JSON cannot be parsed or executed.

18.2 Convergence Guarantees of the Tool Loop

The loop in worker.go: finish_reason=tool_calls → execute each one → append the results to messages as role=tool → re-issue the streaming request. Three hard constraints:

1. at most 3 rounds (maxToolLoops); the final round’s request omits the tools field entirely — removing the model’s ability to keep calling tools, forcing plain-text output and guaranteeing termination;
2. intermediate tool messages stay in memory: only the final assistant text lands in dm_messages (history replay contains no tool-protocol details);
3. one ai_tool_call_audit row per call.

18.3 Defense in Depth for Execution

The step order of ExecuteTool (tool_executor.go:43-177) is itself the security design:

① unified audit first (beginAgentToolAudit) ── attempted-event write failure ⇒ fail-closed, refuse outright
   (no audit, no side effect; parameters stored only as SHA-256 digests, never plaintext)
② tool exists + enabled checks, arguments JSON parsing
③ permission scopes: hasToolPermission, supporting * / module:* wildcards
④ content-addressed idempotent claim:
   key = sha256(bot + userMessageID + toolID + name + arguments)
   ClaimAIToolExecution already-terminal ⇒ replay the persisted result directly, no re-execution
⑤ execute: builtin registry (handover_to_human) / HTTP tools
   (SSRF guard, 4KB response cap, per-tool timeout; on failure fall back to the fallback tool and mark degraded)
⑥ dual-audit wrap-up: SideEffect tri-state not_started/committed/unknown
   (timeout, or fallback succeeded while the main call's outcome is unknown ⇒ explicitly mark unknown; never pretend to know)

The content-addressed idempotency key is the key decision: a redelivered message and an LLM-regenerated identical call hit the same key — “replay returns the previous result” instead of a second refund. The same claim-gate pattern also covers kb.ingest (TryClaimKBDocumentForIngest) and call.analyze (TryClaimCallRecordForAnalysis) — one DB claim gate per paid/non-idempotent subject is the project’s fixed pattern; ai.request itself has no claimable row and relies on the AckWait budget as a fallback (the redelivery cost is documented: the customer receives a second reply + a second paid completion).

18.4 Human-Confirmation State Machine (Suspend → Approve → Resume)

Tools with RequiresConfirmation in the catalog (high risk): claim returns pending_confirmation, and the LLM, holding “awaiting human confirmation”, continues talking with the user; after the admin calls ApproveAIToolExecution on the admin side, ExecuteApprovedExecution (tool_executor.go:204-275) resumes from the same execution row — bypassing the confirmation gate while keeping the full audit chain. This is a suspend state machine spanning processes (worker ↔ admin) and minute-scale delays.

18.5 The Tool Catalog Contract: Declared Once, Projected Everywhere

lib/toolcatalog defines the stable authorization contract for 20 reviewed tools (Key / Version / RequiredScope / five-level Risk / RequiresConfirmation / AllowedActors). HTTP tools, MCP, the AI Agent, and OpenAPI inbound writes all project this single catalog instead of each maintaining its own permission metadata; OpenAPI writes and AI tool execution share the same append-only tool_call_audits lifecycle evidence table (attempted → completed, with digests as the parameter envelope).

Topic 19: Task Flow Chain: From Broker Subjects to the Queue-Routing State Machine

Location: lib/broker/aisubjects, lib/dblib/cs_routing_runtime.go, lib/aiagent/cs_routing_queue.go, lib/cschannel/, lib/dblib/cs_channels.go.

Problem Background

Cross-process tasks (AI requests, KB ingestion, CS assignment) must answer: who does the work (exactly one vs everyone), what if it is done twice (idempotency), and what if it cannot be done (retry/defer/cancel). The project has two coexisting flow infrastructures: the broker subject topology (cross-process message tasks) and the database queue (CS queueing and assignment) — the latter does not depend on the broker and implements a persistent queue + lease coordination in pure SQL.

19.1 Subject Registry and Dual Streams: The “Type System” of Task Flow

The subject→channel mapping for all AI/CS tasks is a closed set, with the source of truth in lib/broker/aisubjects (a drift test intercepts literals that bypass registration — unregistered subjects publish in NATS with no error and no log, a silent black hole):

work uses a shared durable pull consumer (not push + deliver group: that piles messages in the memory of the server-chosen member, a slow worker privately hoarding paid tasks); RabbitMQ/Redis express the same semantics with fixed queue/group names (the SubscribeWork single entry point; see Topic 3). Decision discipline: every new subject must pass a work/fanout decision — anything with paid side effects or non-idempotent writes (e.g. ai.survey.ready increments unread counts and sends offline pushes) must go to the work side; otherwise on NATS it will be charged/written once per replica.

19.2 Consumption Semantics: Always-Ack + Idempotency Pushed Down

Worker handlers uniformly defer ack() + defer recover() — always acknowledge; failures surface through chat.stream.error events rather than broker retries (preventing retry storms); the duplicates of at-least-once are absorbed by the claim gates at each business point. This turns “will the message be redelivered” from an ops question into a code-review question: “does every side-effect point have an idempotency gate?”

19.3 The Queue-Routing State Machine: A Pure-Database Task Queue

CS assignment is the most complete instance of task flow — cs_routing_queue (created by migrations/{mysql,postgres}/20260826_complete_cs_routing_runtime.sql, logic in lib/dblib/cs_routing_runtime.go):

                    ┌─────────────────────────────────────────────┐
                    │          cs_routing_queue                    │
                    │  PK = conversation_id (one row per          │
                    │  conversation, naturally idempotent)        │
                    └─────────────────────────────────────────────┘
  enqueue (upsertRoutingQueueTx, attempts+1):
    · RouteCSConversation found no eligible agent ──► waiting (next_attempt_at=now+15s, "no eligible agent")
    · outside business hours & offline_policy=queue ──► waiting ("outside business hours")
    · DM team transfer with no candidate ✗ does not enter this queue (returns team_unavailable directly)

  dispatch loop (ai_worker, interval_sec=5s, batch 100):
    SELECT ... WHERE status='waiting' AND next_attempt_at <= now
    ORDER BY priority DESC, enqueued_at ASC
    FOR UPDATE SKIP LOCKED                ← multiple workers never grab the same row
    then next_attempt_at = now+30s        ← picking the item is the lease
    │
    ├─ session already closed       → cancelled("session no longer open")
    ├─ widget disabled / switched to manual → cancelled
    ├─ outside hours & policy≠queue → cancelled; policy=queue → defer to now+5min
    ├─ RouteCSConversation error    → defer to now+5s
    └─ assignment succeeded → assigned + SnapshotOpenGroupCSSessionClaim (writes first response/ownership)

  also: group CS release → assigned→waiting with next_attempt_at=now (immediate re-dispatch)
        session closer / handover release → cancelled

Three designs worth calling out:

1. next_attempt_at doubles as two things: both the retry-due time and the multi-worker lease — there is no separate lease column. No other worker can take the row within 30 seconds; if a worker crashes, the batch auto-expires after 30 seconds and is taken over — no cleanup job needed. The cross-worker lease semantics are guarded by an integration test (TestListDueCSRoutingQueue_LeasesRowsAcrossWorkers);
2. fixed backoff, not exponential: errors 5s, enqueue/retry 15s, out-of-hours 5min, lease 30s — each magnitude aligned to its own rhythm (scan interval / human-correction time / business hours); there is no workload here where “the more retries, the longer you should wait”;
3. cancellation is passively discovered: disabling a widget or switching to manual does not synchronously cancel queue items from the admin side; the dispatch loop cancels them within 5 seconds of scanning — zero admin-surface coupling, at the cost of at most 5 seconds of delay.

The idle closer deliberately skips waiting sessions (NOT EXISTS (queue.status='waiting') inside ListDueOpenCSSessions) — queued sessions must not be closed by the 5-minute idle timeout; this is an implicit contract between the two background loops.

19.4 The Capacity-Reservation State Machine: cs_agent_assignments

 reserved (reserved_until = DB clock+120s)  ──FinalizeCSAssignmentReservation──►  active
     │                                        (locks the unexpired reservation row, rewrites conversation_id)
     │ mid-process crash ⇒ no cleanup job: all capacity predicates simply ignore rows with reserved_until <= now
     ▼
  active ──session close (closeCSSession in the same transaction: optimistic lock prevents double close + releases capacity + cancels queue)──► released
  active ──DM transfer handover / reservation rollback (ReleaseCSConversationAssignment)──────────────────────────────────► released

Reservations solve multi-step DM handover (a transfer must validate both sides, write routing, and write the brief — capacity must not be snatched mid-way); the TTL uses the database clock, not the application clock (with multi-instance clock skew, the DB is authoritative). Expired reserved rows are not written to released and need no cleanup job — “logical expiry” via predicate ignorance eliminates leaks (TestReserveCSAgent_ExpiredReservationDoesNotLeakCapacity).

19.5 The Assignment Transaction: RouteCSConversation

cs_routing_runtime.go:373-453, completing “pick agent + take capacity + claim + write queue” in a single transaction, with concurrency correctness resting on two locks:

1. agent row locks: SELECT ... FROM cs_agents WHERE enabled AND ready ORDER BY user_id ASC FOR UPDATE — deterministically ordered locking by user_id, so two concurrent assigners take the locks in the same order: no deadlock, and no double-spending of the same last capacity slot. After locking, filter in memory: online (cross-instance presence), shift, skill/language, and countCapacityUsageTx counting actual usage one by one (active + unexpired reserved);
2. conversation-claim optimistic lock: claimConversationRoutingTx idempotent INSERT + conditional UPDATE (only when switched_by='' or self) + read-back of the holder — if you lost the race but the holder is a valid agent, you defer to the holder (when two assigners race for the same conversation, the later one accepts the earlier one’s result instead of erroring).

The selection strategy lives in chooseLockedRoutingAgent (:494-528): the widget’s primary team strictly precedes overflow_team_ids (comma/semicolon/newline separated, dedup preserving order); least_loaded sorts stably ascending by active count; round_robin looks up the widget’s most recent assignment to find the last agent and takes the next one circularly — the cursor lives in the DB, not memory, so it is multi-replica safe.

19.6 Channel Inbound: Unified Envelope + Event-Level Idempotency (and Current Boundaries)

Inbound flow for external channels (first adapter: Email): HTTP entry (X-IMCore-Channel-Secret constant-time SHA256 verification) → adapter.Decode normalizes into the unified Envelope (EventID / external conversation id / identity / attachments) → IngestCSChannelEnvelope persists in a single transaction: cs_channel_audit uses (channel_id, external_event_id) as a unique key with INSERT IGNORE for event-level idempotency (webhook resends return ErrDuplicateChannelEvent directly) → identity merge (ext_email_<sha256>) → create the external conversation. Current boundary (recorded honestly): channel inbound is not yet wired into the 19.3 queue routing — the only production call sites of RouteCSConversation are the group-CS hook and the dispatcher; RoutingMode on a channel is currently just a stored field; future WeChat-Work/WhatsApp adapters must first map into the Envelope and must not bypass the unified tables to persist separately.

19.7 Three Design Principles (Wrap-Up)

1. Idempotency always lives in the business layer — brokers/queues promise only at-least-once; every paid or non-idempotent point gets a DB claim gate (tool execution, KB ingestion, call analysis, channel events), and “duplicate delivery” is converted into “replay the previous result”;
2. state machines are explicit and convergence is guaranteed — the tool loop’s final round drops tools to force termination; the rule chain short-circuits on stop/handover; queue items end up either assigned or cancelled, with no dangling states; reserved capacity expires via predicates rather than cleanup jobs;
3. coordination mechanisms hug the data — whatever one SQL can solve (SKIP LOCKED pickup, optimistic-lock claim) gets no extra component; Redis is used only across processes (cancel signals, conversation leases, sweeper single-flight); what needs audit and durable evidence (queueing, capacity, tool execution) all goes into the database.

Appendix A: Key Documentation Index

Appendix B: Glossary