Real-Time Support

• Main Page
• Documentation
• Career Opportunities
• Get Assistance
• Contact Sales Team

• Key Products
• AI and ML Solutions
• Compute and Infrastructure
• Databases and Storage
• Security and Networking
• Developer Tools
• All Products and Solutions

• Login Options
• Community Forum
• DigitalOcean Account

• Sign Up Options
• Community Forum
• DigitalOcean Account

• Login Options
• Community Forum
• DigitalOcean Account

• Sign Up Options
• Community Forum
• DigitalOcean Account

• Tutorial Guides
• Frequently Asked Questions
• Product Documentation
• Community Search

Table of Contents Overview

Authors: Adrien Payong and Shaoni Mukherjee

Customer support is a powerful application of AI, involving repetitive queries, time pressure, and the need for accurate responses. Millions of businesses receive similar questions daily, such as order status, refunds, and feature explanations.

A properly trained AI support agent can instantly respond to many of these requests, reducing expenses and delighting customers. However, not all customer requests have the same complexity level, with some being simple and low-risk, while others are sensitive, ambiguous, or business-critical.

This poses a challenge for businesses developing AI-driven support agents. While a fast, cheap model may suffice for simple queries, it may struggle with unclear customer intent, emotionally sensitive issues, or technically complex problems. Using a more powerful model for every message can be costly and introduce latency, which can be problematic as conversation volume grows. A better solution is to design a customer support agent with fallback routing.

This article explains how to build such a system, covering the architecture of a support agent, the role of the router, confidence-based fallback, and defining priority and cost constraints in the router configuration.

• Fallback routing balances speed, cost, and answer quality by sending routine questions to fast models and complex cases to stronger models.
• Confidence scoring is essential for deciding when the primary model's answer is safe and complete enough to send.
• Priority classes improve business decision-making by giving higher-quality handling to sensitive, high-value, or escalated customer issues.
• Cost controls prevent overuse of expensive models by setting limits per message, per conversation, per customer tier, and across the platform.

Understanding Fallback Routing

Fallback routing is an AI orchestration pattern where a request is initially routed to a primary model, and the answer quality is assessed. If the answer reaches established confidence, safety, and policy thresholds, it is returned to the customer. Otherwise, the request is routed to a stronger model, a specialist model, a human agent, or another workflow.

Fallback routing is useful for many customer support use cases because customer support workloads tend to be very imbalanced. A large percentage of tickets can be routed to a lightweight model because they're completely repetitive. A smaller percentage of tickets need more advanced reasoning skills, better instruction-following, a more careful tone, or access to advanced tools.

• Low confidence score
• Missing or incomplete answer
• Customer frustration
• Sensitive topic
• High-value customer
• Complex technical issue
• Policy uncertainty
• Failed knowledge-base retrieval
• Contradictory information
• Repeated user dissatisfaction

The fallback router acts as the traffic controller of the AI support system, deciding which model should handle each request, when escalation is necessary, and how much cost the system is allowed to spend.

Core Architecture of a Real-Time Fallback Support Agent

A real-time customer support agent with fallback routing usually contains several components.

The customer inputs their message using various customer support channels, such as live chat, email, or a website chat widget.

An intent classifier categorizes the type of request, such as billing, refund, or technical support.

A retrieval layer queries the company's knowledge bases, such as an FAQ repository or product documentation.

The model needs a reliable context before producing an answer.

The router inspects the request and decides which model should fulfill the request, typically pointing to the cheapest and fastest model by default.

If allowed to answer the request, the primary model generates its answer, which may then be evaluated by a confidence checker.

The system either returns the answer, escalates to a stronger model, asks the customer for clarification, or transfers the conversation to a human support agent.

Primary Route: Fast and Cheap Model

The primary model refers to the initial model the customer support agent uses, which should be fast, cheap to run, and of acceptable quality for answering routine queries.

• Answering frequently asked questions
• Summarizing simple policies
• Providing order tracking guidance
• Explaining basic account steps

The primary model shouldn't have to handle every situation, but it should be able to confidently resolve most simple requests.

If a customer asks how to reset their password, the primary model should be able to retrieve the password policy and generate a concise response without escalation.

However, if a customer asks about a complex issue, such as being double-charged, the primary model may not be equipped to handle the complexity, and the request can be escalated to a stronger model.

Fallback Route: Stronger Model for Low-Confidence Cases

The fallback model should be more capable, but often comes at the cost of being slower and/or more expensive.

Enabling the fallback model allows for better reasoning, stronger instruction-following capabilities, careful policy interpretation, and better handling of ambiguity.

• Complex troubleshooting
• Cases requiring multi-step reasoning
• Customer refund disputes
• Contract or pricing interpretation

The fallback process should not be noticeable to the customer, and the conversation should flow as normal.

A good fallback system can also work by passing the primary model's draft answer to the stronger model for editing, validation, or replacement.

Confidence Scoring in the Router

Confidence scoring is crucial for fallback routing, as the system needs to know when to trust the primary answer and when not to.

• Retrieval confidence
• Model self-assessment
• Answer validation
• Customer sentiment

It's possible to create a practical confidence score that considers each of these aspects before making a routing decision.

The answer must pass several checks before being sent to the customer, and if one or more checks fail, the system uses the fallback route.

confidence_policy: minimum_confidence: 0.78 fallback_if: retrieval_score_below: 0.70 answer_completeness_below: 0.75 policy_risk_above: 0.40 customer_sentiment: "angry" contains_sensitive_topic: true

In this configuration, the answer is evaluated based on multiple factors, including retrieval confidence, model self-assessment, answer validation, and customer sentiment.

Defining Priority in the Router Config

Priority should define which requests get better models, faster responses, or human escalation.

priority_classes: low: examples: - general_faq - product_information - password_reset primary_model: fast_support_model fallback_model: standard_reasoning_model max_latency_ms: 2500 max_cost_per_request_usd: 0.01 medium: examples: - billing_question - subscription_change - technical_issue primary_model: fast_support_model fallback_model: advanced_support_model max_latency_ms: 5000 max_cost_per_request_usd: 0.05 high: examples: - refund_dispute - escalated_customer - enterprise_account - legal_or_compliance_topic primary_model: advanced_support_model escalation_target: human_agent max_latency_ms: 8000 max_cost_per_request_usd: 0.20

In a realistic support environment, not all tickets have equal business value, and the router configuration should make priority classes obvious.

The goal is not simply to reduce AI cost but to spend intelligently, investing more resources where the business risk is higher.

Handling Latency in Real Time

Real-time support requires low latency, and customers expect immediate responses, especially in live chat.

• Pre-routing
• Parallel evaluation
• Streaming
• Cache common responses
• Clarification questions before escalating

Defining Cost Constraints

Cost constraints allow you to keep your AI support system sustainable by applying limits at several levels, including per request, per conversation, per day, and per customer tier.

• Per request to limit spending on an individual customer message
• Per conversation to manage the overall expense of lengthy support threads

A practical router configuration might look like this, with cost controls specifying how the support platform spends AI credits per message and conversation.

cost_controls: default: max_cost_per_message_usd: 0.02 max_cost_per_conversation_usd: 0.15 max_fallbacks_per_conversation: 2 customer_tiers: free: max_cost_per_conversation_usd: 0.05 allow_human_escalation: false fallback_mode: cheapest pro: max_cost_per_conversation_usd: 0.20 allow_human_escalation: true fallback_mode: balanced enterprise: max_cost_per_conversation_usd: 1.00 allow_human_escalation: true fallback_mode: quality_first prefer_advanced_model: true global_budget: daily_limit_usd: 500 action_when_limit_reached: degrade_to_primary_model output_length_policy: shorten escalation_policy: restrict_to_high_priority

The configuration includes tiered limits for free, pro, and enterprise users, with varying budgets, escalation options, and fallback modes.

The daily budget for the entire platform is also set, with measures in place to prevent excessive spending, such as using only the primary model, shortening output, and disallowing escalation except for high-priority cases.

Finally, the system is designed to handle a wide range of scenarios, from simple queries to complex issues, while minimizing costs and maximizing customer satisfaction.

Example End-to-End Scenario

Imagine a customer writes about a billing issue, and the intent classifier recognizes the topic as billing or account access.

The sentiment detector recognizes the comment as negative, specifically frustration, and the customer tier is currently free, but the message mentions a pro payment.

The retrieval system finds help articles about billing troubleshooting, and the router sees that this is not a simple FAQ but a potentially complex issue with explicit annoyance and a failed payment.

The primary model generates an answer, but the confidence checker shows that it is overly generic and doesn't explain what the user can do when their payment goes through but account activation doesn't occur.

The router escalates to the stronger model, which takes the retrieved policy into account and produces a far more helpful answer, apologizing, explaining likely causes, and offering to open a support ticket for manual verification.

If the account lookup tool confirms the payment, this can automatically be escalated to a human billing agent to process activation.

• What is fallback routing in AI customer support?

Fallback routing is an AI orchestration method where a request first goes to a fast, low-cost model and is escalated to a stronger model or human agent when confidence, safety, or policy checks fail.

• Why not send every customer request to the strongest model?

Using the strongest model for every request improves quality but increases cost and latency, and fallback routing helps reserve expensive models for complex, sensitive, or high-risk cases.

• When should a support request fall back to a stronger model?

Fallback should occur when the primary model shows low confidence, gives an incomplete answer, detects customer frustration, faces policy uncertainty, or handles sensitive topics such as billing, refunds, or enterprise accounts.

• How does the router decide which model to use?

The router evaluates factors such as customer intent, retrieval quality, confidence score, customer sentiment, business priority, latency budget, and cost constraints.

• Can fallback routing reduce customer support costs?

Yes, it allows simple requests to be handled by cheaper models while reserving stronger models or human agents for cases where quality and risk management matter most.

Fallback routing enables AI customer support to be practical, reliable, and cost-effective, routing intelligently based on confidence, risk thresholds, case complexity, customer priority, and other business rules.

• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Learn more about our products and solutions for AI and machine learning.

About the Authors

The authors are experienced AI consultants and technical writers with a strong background in data science and AI.

They specialize in creating in-depth content on AI, machine learning, and GPU computing, and have written innovative articles that provide developers and researchers with actionable insights.

This text area defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation, and marketplace offerings and insert the link.

Join the many businesses that use our cloud platform to accelerate growth and innovation.

Featured Tutorials and Guides

• All Tutorials
• All Topic Tags

Please complete your information to access our resources and support.

• Table of Contents
• Introduction
• Key Takeaways
• What Is Fallback Routing?

• Join the many businesses that use our cloud platform to accelerate growth and innovation.

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Documentation and Resources

Full documentation for every product and solution.

Resources for Startups and AI-Native Businesses

The Wave has everything you need to know about building a business, from raising funding to marketing your product.

The Developer Cloud

Scale up as you grow, whether you're running one virtual machine or ten thousand.

Start Building Today

From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.

• About Us
• Leadership
• Blog
• Careers
• Customers
• Partners

• Knowledge Bases
• GPU Droplets
• Bare Metal GPUs
• Inference Engine

• Community Tutorials
• Community Q&A
• CSS-Tricks
• Currents Research

• AI Training GPU
• GPU Inference
• VPS Hosting
• Website Hosting

• Support
• Sales
• Report Abuse
• System Status

• About Us
• Leadership
• Blog
• Careers

• Knowledge Bases
• GPU Droplets
• Bare Metal GPUs

• Community Tutorials
• Community Q&A
• CSS-Tricks

• AI Training GPU
• GPU Inference
• VPS Hosting

• Support
• Sales
• Report Abuse