Self-hosted AI customer service: an on-prem selection guide

Three drivers dominate. First, data boundaries: support conversations carry customer identities, orders and complaint details; finance, healthcare, government and cross-border operations often require data to stay within a defined perimeter, and sending conversations to third-party cloud inference needs separate assessment under many frameworks. Second, model sovereignty: self-hosting means you can swap models and versions, or route different conversation types to different models — with a cloud product you accept their model strategy. Third, predictable cost: per-seat or per-resolution billing rises linearly with volume, while a buyout turns that curve into a one-time license plus your own inference spend.

Run every candidate — open-source assembly or commercial engine — through these eight checks; vagueness on any one becomes production risk:

• Bring-your-own model endpoint: OpenAI-compatible APIs and private inference endpoints, with replaceable, rollback-able models
• RAG grounding: answers must come from your knowledge base with citable sources, not model improvisation
• Human handover: seamless transfer to agents who see the full context
• Evaluation & red-teaming: batch quality evaluation before launch, plus simulation of prompt-injection attacks
• Streaming & voice: streamed replies, with a voice-transcription path you control
• Deployment footprint: start on one node, scale without touching clients
• No phone-home: no forced telemetry or cloud dependencies; runs fully on intranets
• Data governance: retention, export, anonymization and legal hold built in

Cloud AI support typically bills per seat monthly plus AI add-ons, or per resolution — the more conversations, the higher the bill, at a unit price the vendor controls. The self-hosted ledger is different: a one-time or yearly license plus your own server and model-usage costs (usage scales with conversations, but you can pick cheaper models, cache frequent questions and bound context length). The more agents and the longer the horizon, the wider the gap; run your seat count and conversation volume through a three-year total and the conclusion is usually vivid.

Using IMCore as the example: one command pulls up a local stack with the AI worker (PostgreSQL + pgvector + Redis + message bus); configure a bot and a model endpoint and you are conversing in the web support widget. Step two: load the knowledge base into RAG and set handover rules. Step three: gate answer quality with evaluation runs before production. Deployment scales from single-node Compose to a Helm cluster without client changes, and the model endpoint can switch at any time — including to an inference service that never leaves the intranet.