Multi-Service
Last verified: 2026-08-12
Kai supports 29 LLM providers (plus a built-in Free tier). Each provider uses one of three API formats: OpenAI-compatible (most services), Gemini native, or Anthropic native -- plus LiteRT on-device for local inference. Users can configure multiple service instances, reorder them, and Kai automatically falls back through the chain on failure.
Concepts
Service
A supported LLM provider. Each service is defined by:
- A unique name and ID
- Whether it requires an API key
- Which API format it uses (OpenAI-compatible, Gemini native, or Anthropic native)
- Links to the provider's API-key management page
Service Instance
A configured connection to a service. Users can add multiple instances of the same service (e.g. two OpenAI accounts with different keys). Each instance stores its own:
- API key
- Selected model
- Base URL (relevant for the OpenAI-Compatible API service)
Free Tier
A built-in service that requires no API key. Free is never shown in the service picker — it is used as:
- The sole service when no other services are configured
- A last-resort fallback when "Use as fallback" is enabled (default)
When Free is the only path and the user hits Free FAST/EXPERT rate or quota limits, chat shows a free-provider signup panel (Groq, Cerebras, Gemini, OpenRouter, Ollama Cloud) so they can open an API-key page and continue with a personal free tier. See chat.md.
Fallback Chain
- Configured instances are tried in the order the user arranged them
- Only instances with valid API keys are considered
- If no instances are configured, the Free tier is used as the only service
- If instances exist and "Use as fallback" is enabled (default), the Free tier is appended as the last resort
- Each individual API request retries up to 2 times with increasing delays before the service is considered failed. During a tool-use loop, each request inside the loop retries independently — a failure mid-loop never replays the loop (and its tool executions) from the start. On-device (Local Model) attempts are not retried, since their failures are deterministic rather than transient
- On failure, the next instance in the chain is tried; if all fail, the last error is shown
- If a fallback succeeds, the response indicates which service answered
- While the chain is being walked, the thinking indicator shows per-attempt status — the name of the service currently being tried, or the reason the previous one failed before moving on — so silent fallbacks are visible to the user
- Entries whose context window can't fit the current chat history are skipped during the walk
- On-device (Local Model) failures are not silently absorbed — they short-circuit the fallback chain so the user sees the actual error rather than being quietly bumped to a cloud service
- On-device entries are also never used as fallback targets: a local model is only tried when it is the primary (first) service in the chain. A cloud-service failure never silently starts a local model load
- Certain non-retryable errors (notably Anthropic's "insufficient credits" and quota-exhausted responses from OpenAI-compatible providers) skip further per-service retries and fail that service immediately; the fallback chain still continues to the next instance. Only on-device (Local Model) failures short-circuit the entire chain
API Formats
Most services use the OpenAI-compatible chat completions format. Gemini uses Google's native Generative Language API. Anthropic uses its own Messages API with x-api-key header authentication and a different request/response structure. LiteRT runs inference on-device using Google's LiteRT LM SDK -- no HTTP, no API key, fully offline.
The OpenAI-Compatible API service supports a custom base URL, defaulting to localhost:11434/v1 for local Ollama setups. The base URL should include the version path segment (e.g., http://localhost:11434/v1 or https://my-provider.com/api/v1), following the OpenAI SDK convention. Kai appends only /chat/completions or /models to this base URL.
Supported Services
| Service | id |
Requires API Key | API Type |
|---|---|---|---|
| Free | free |
No | OpenAI-compatible |
| Atlas Cloud | atlascloud |
Yes | OpenAI-compatible |
| Gemini | gemini |
Yes | Gemini native |
| Anthropic | anthropic |
Yes | Anthropic native |
| OpenAI | openai |
Yes | OpenAI-compatible |
| DeepSeek | deepseek |
Yes | OpenAI-compatible |
| Mistral | mistral |
Yes | OpenAI-compatible |
| xAI | xai |
Yes | OpenAI-compatible |
| OpenRouter | openrouter |
Yes | OpenAI-compatible |
| GroqCloud | groqcloud |
Yes | OpenAI-compatible |
| NVIDIA | nvidia |
Yes | OpenAI-compatible |
| Cerebras | cerebras |
Yes | OpenAI-compatible |
| Ollama Cloud | ollamacloud |
Yes | OpenAI-compatible |
| LongCat | longcat |
Yes | OpenAI-compatible (ships with a curated default model list; also exposes a /models endpoint used during validation) |
| Together AI | together |
Yes | OpenAI-compatible |
| Hugging Face | huggingface |
Yes | OpenAI-compatible |
| Venice AI | venice |
Yes | OpenAI-compatible |
| Moonshot AI | moonshot |
Yes | OpenAI-compatible |
| Z.AI | zai |
Yes | OpenAI-compatible |
| Z.AI Coding Plan | zai-coding-plan |
Yes | OpenAI-compatible |
| MiniMax | minimax |
Yes | OpenAI-compatible |
| AIHubMix | aihubmix |
Yes | OpenAI-compatible |
| Deep Infra | deepinfra |
Yes | OpenAI-compatible |
| Fireworks AI | fireworksai |
Yes | OpenAI-compatible |
| OpenCode | opencode |
Yes | OpenAI-compatible |
| Public AI | publicai |
Yes | OpenAI-compatible |
| AI Horde | aihorde |
Yes (anonymous key 0000000000 allowed at lowest priority) |
OpenAI-compatible (via oai.aihorde.net; model list is the set of text models with online volunteer workers — availability and latency vary) |
| Perplexity | perplexity |
Yes | OpenAI-compatible (Sonar; ships with a curated default model list — no authenticated /models endpoint for Sonar; connection validation probes the chat endpoint with an incomplete body to check the API key) |
| OpenAI-Compatible API | openai-compatible |
No (optional) | OpenAI-compatible |
| Local Model | litert |
No | On-device (LiteRT LM) |
Connection Validation
When the user enters or changes an API key (or base URL), the app validates the connection after an 800 ms debounce and shows a status indicator: checking, connected, invalid key, quota exhausted, rate limited, connection failed, or local network access denied. Validation also runs for all services when the settings screen opens. Services validate by fetching their model list — Gemini, Anthropic, and OpenAI-compatible services (including LongCat) each call their respective models endpoint. Perplexity is an exception: Sonar has no authenticated models list, so validation probes the chat endpoint with an incomplete body to verify the API key, then loads the curated Sonar model list. On a successful connection, the available model list is refreshed.
Local Network Servers (Android)
Android 17+ blocks all traffic to local network hosts unless the user grants the local network permission. When a service's base URL points at a LAN host (private IP range, .local name, or a bare hostname — loopback is exempt), the app requests the permission before validating the connection and before sending a chat message. If the user denies it, the connection status shows "local network access denied" with an "Open settings" button that jumps straight to the app's system settings page, and chat shows an actionable error instead of a silent failure. When the user returns to the app after granting the permission there, the denied connection re-validates automatically (without ever re-prompting). Other platforms don't gate local network access, so the check is a no-op there.
Model Selection
When a connection is validated and models are fetched, the app auto-selects a model if none is chosen — first checking for a per-service default model (e.g. LongCat defaults to "LongCat-Flash-Lite", Perplexity defaults to "sonar-pro"), then preferring "kimi-k2.5" if available, otherwise the first model in the list. Services filter their model lists:
- OpenAI shows only chat-oriented models (prefix filter)
- GroqCloud shows only models marked as active
- Together AI filters by type == "chat" to exclude non-chat models (embedding, code, etc.)
- Other services show all non-retired models
If a stored list-selection model id is not in the fetched list, it is kept and shown as selected rather than overwritten by auto-select.
Custom model (OpenAI-Compatible API)
The OpenAI-Compatible API config card keeps the normal model dropdown (when a list is available) and adds a Custom model checkbox below it. When checked, a free-text field appears for a model id the server accepts even if /models never returns it (e.g. a free tier omitted from the list). List selection and custom id are stored separately; chat uses the custom id only while the checkbox is on. Toggling the checkbox off restores the list selection without losing the typed id.
Model Cards
The model picker modal shows each candidate as a card with consistent metadata regardless of provider:
- Title (top left) — a human-readable display name from the curated catalog or the provider's API; falls back to the raw model id only when no display name is available
- Free badge — a green "Free" label next to the title when the model is on that provider's free tier according to the curated free-tier catalog (currently Ollama Cloud and OpenRouter only). Free-ness is per service, not global.
- Arena score (top right) — LMArena Elo rating as colored text, gradient from green (>= 1400) through lime/yellow to orange (< 1250)
- Detail line (below the title) — release date, parameter count, and context window joined into a single muted line separated by
·(e.g.Mar 2025 · 70B · 200K ctx); any missing field is simply omitted from the line
The card representing the currently selected model is highlighted with a filled accent background so users can identify their current choice at a glance when reopening the picker.
The modal includes sort chips (Date, Score, Ctx) below the search field. Tapping a chip switches which field is active; all sorts are descending (highest or most recent first), with no ascending option. Default sort is by score. When the current service list includes any free-tier model, a Free filter chip also appears: selected, it shows only free-tier models while the active sort still applies; if the filtered list is empty, a short empty message is shown. The Free chip is hidden for services with no free-tier catalog entries.
Context window and release date come from two sources, merged by the mapping layer: a bundled curated catalog of well-known models, and whatever the provider's own models endpoint returns (e.g. OpenAI-compat context_window and created, Anthropic created_at). The curated catalog wins; provider-supplied values are used only as a fallback when the catalog has no entry for that model. The catalog is hand-maintained to correct inconsistencies in what providers report. Models not present in the catalog still render — they just use whatever the API provided, and any unknown fields are hidden.
Free-tier membership is a separate curated list (not derived from live pricing APIs at runtime). Runtime ids live in the free-tier catalog in code; curation policy, provenance, and a mirrored snapshot live in the OKF knowledge bundle under docs/knowledge/free-tier/. Both are refreshed together with the update-free-tier-models skill.
Chat Screen Service Toggle
The chat screen service toggle lists two distinct Free entries — Free FAST and Free EXPERT — followed by every configured non-Free service (including on-device and models that do not support tools). When this list contains more than one entry, a circular service icon button appears to the right of the chat input, next to the send/stop button. Because the two Free modes alone already count as two entries, the toggle can appear even when no non-Free services are configured. The icon represents the current primary service (each service has its own simplified vector icon). Tapping it opens a dropdown listing those services with their icons, names, and model IDs; the current primary is highlighted with a primary container background. The dropdown is height-capped to the screen and becomes scrollable when the list is long enough to overflow, so every entry stays reachable.
In Interactive UI mode, the same control is filtered to services whose selected model supports tool use (agentic flows), and on-device services are excluded so the user cannot switch to a model that cannot produce kai-ui.
Selecting a non-Free entry reorders the configured list so the chosen service becomes first (primary). Selecting a Free entry (FAST or EXPERT) instead flips an "is Free primary" flag and records the chosen Free mode — Free can be promoted to primary independently of the configured fallback order, without rearranging the non-Free chain. The fallback chain picks up the new state automatically. The fallback walker also skips any entry whose context window can't fit the current chat history, so very long conversations may transparently move past services that would otherwise be eligible.
Attachments
Image attachments are broadly supported across cloud services, gated on two levels. The built-in Free tier is text-only at the service level — its proxy fans out to a chain that includes text-only fallback models, so it never accepts images regardless of the nominal model. Beyond that, image support is per-model: DeepSeek has no vision models, so images are dropped for it everywhere (including when it's reached through an aggregator), and services that host both text-only and vision models in one catalog (e.g. Z.AI, where the text GLM models sit next to the multimodal GLM-V variants) accept images only when the selected model is vision-capable. When the active service or model can't take images, images are dropped from the request — and the file picker stops offering image types — so a turn that includes an image still goes through as text. Unknown models are assumed image-capable, since most modern flagship models are multimodal. PDF attachments are advertised only by services with native document support — currently Anthropic, Gemini, OpenAI, and OpenRouter. Anthropic and Gemini encode PDFs natively in the request; the OpenAI-compatible wire path currently drops PDF binaries (images still go through as image_url), so a PDF chosen while OpenAI or OpenRouter is active is accepted by the UI but not sent to the model. The on-device Local Model hides file attachment affordances entirely; users running purely locally don't see attachment buttons.
Settings UI
Users manage services through the settings screen: - Add — pick from the list of available services (can add the same service multiple times); the OpenAI-Compatible API and the on-device Local Model are pinned to the top of the picker, followed by the highlighted featured provider Atlas Cloud, with the remaining providers sorted alphabetically - Remove — delete an instance and its stored credentials; deletion is deferred with a snackbar "Undo" option (~4 seconds) before the service is permanently removed - Reorder — drag to change priority (first = primary, rest = fallbacks) - Configure — per-instance API key, model selection, base URL (OpenAI-Compatible only; optional custom model id via checkbox) - Free fallback toggle — controls whether Free is appended as last resort - Sponsors — the Free tier card lists all GitHub sponsors in a single grid, with active sponsors first followed by past sponsors
Key Files
| File | Purpose |
|---|---|
composeApp/src/commonMain/.../data/Service.kt |
Service definitions, all provider metadata |
composeApp/src/commonMain/.../data/ModelCatalog.kt |
Curated context window / release date / Arena Elo for well-known models |
docs/knowledge/model-catalog/ |
OKF bundle: Arena Elo matching policy, attested snapshot, refresh playbook |
composeApp/src/commonMain/.../data/FreeTierModels.kt |
Runtime free-tier model ids per service (Ollama Cloud, OpenRouter) |
docs/knowledge/free-tier/ |
OKF bundle: free-tier policy, snapshot, sources, refresh playbook |
composeApp/src/commonMain/.../data/FreeProviderSuggestions.kt |
Providers recommended in chat when Free is rate-limited with no services configured |
composeApp/src/commonMain/.../data/ModelTransformations.kt |
Maps provider model DTOs to SettingsModel, merges with catalog and free-tier flags |
composeApp/src/commonMain/.../data/AppSettings.kt |
Service instance storage, credential persistence, migration |
composeApp/src/commonMain/.../data/RemoteDataRepository.kt |
Fallback chain, request orchestration |
composeApp/src/commonMain/.../network/Requests.kt |
HTTP clients for all three API formats |
composeApp/src/commonMain/.../network/dtos/anthropic/ |
Anthropic Messages API DTOs |
composeApp/src/commonMain/.../ui/settings/SettingsViewModel.kt |
Connection validation, service management UI logic |
composeApp/src/commonMain/.../tools/PermissionController.kt |
Runtime permission requests, including the local network gate for LAN server URLs (Android 17+) |
composeApp/src/commonMain/.../tools/LocalNetworkUrl.kt |
Detects LAN base URLs that need the local network permission |
composeApp/src/commonMain/.../ui/chat/ChatScreen.kt |
Chat screen, renders ServiceSelector |
composeApp/src/commonMain/.../ui/chat/composables/ServiceSelector.kt |
Compact service toggle dropdown |
composeApp/src/commonMain/.../ui/chat/ChatViewModel.kt |
Wires service selection and reordering |