Qwen 3 Inference

Blog Posts
Documentation
Career Opportunities
Support Options
Sales Contact

Featured Products
AI and Machine Learning Solutions
Developer Tools
Gaming and Media Services
Security and Networking Solutions
Startup and SMB Solutions
Web and App Platforms

Login Options
Community Access
DigitalOcean Account

Signup Options
Community Membership
DigitalOcean Account

Login Options
Community Access
DigitalOcean Account

Signup Options
Community Membership
DigitalOcean Account

Tutorial Guides
FAQs
Product Documentation
Community Search

Authored by Adrien Payong and Shaoni Mukherjee

Selecting a suitable Qwen model is just the initial step in production decisions. The next crucial question is where and how to deploy it.

The ideal provider for an experimental chatbot may not be suitable for a regulated enterprise application, a latency-sensitive coding assistant, or an agent processing millions of tokens daily. Production teams must consider more than just the advertised cost per million tokens, including time to first token, output speed, concurrency limits, context window lengths, geographic availability, security, compliance, observability, model version consistency, and operational control.

Qwen is a family of large language models developed by Alibaba, featuring dense and mixture-of-experts models, hybrid reasoning modes, multilingual support, tool use, and varying model sizes from 0.6 billion to 235 billion parameters.

Alibaba Cloud has expanded the Qwen family with Qwen 3.5, introducing models like Qwen3.5-27B, Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, and Qwen3.5-397B-A17B, with the model size indicating the number of parameters and the -A number denoting the active parameters per token.

This guide compares managed APIs, routing platforms, dedicated inference, and self-hosted GPU infrastructure for Qwen 3 deployment.

Choosing the Right Provider

Compare popular Qwen inference providers based on recommended use cases, relative pricing, speed, and infrastructure control to determine the best fit for your workload.

There is no one-size-fits-all solution; the choice depends on the specific workload requirements.

Opt for a managed serverless API for unpredictable traffic and minimal operational overhead.
Choose a speed-focused provider for low-latency applications like interactive chat or coding.
Select a router for model portability and provider fallback.
Use dedicated inference for sustained traffic with predictable capacity.
Self-host for infrastructure control, custom weights, or private networking.

Key Metrics for Provider Selection

The cheapest advertised price per token may not always result in the lowest production cost; evaluate providers using identical prompts, model versions, output lengths, and concurrency levels.

Critical metrics to consider include time to first token, output speed, end-to-end latency, tail latency, and throughput.

Time to first token impacts the perceived responsiveness of applications.
Output speed is crucial for long answers or code generation.
End-to-end latency encompasses the full request lifecycle.
Tail latency captures slow edge-case requests, often measured as p95 or p99 latency.
Throughput measures overall tokens or requests processed over time.

Additionally, test support for structured outputs, tool calling, prompt caching, long-context, regional latency differences, rate limits, and failure behavior, as some providers may only support a subset of the OpenAI parameter set.

OpenAI-compatible APIs accept requests with similar endpoints and JSON structures, allowing for easier migration between providers but not guaranteeing identical behavior.

Cheapest Provider per Token

DeepInfra and Novita are often cost-effective starting points for Qwen inference, but comparisons should be based on equivalent model versions and service tiers.

For instance, DeepInfra listed Qwen3.5-27B at ~$0.26/million input tokens and ~$2.60/million output tokens on the regular tier, with the Qwen3.5-397B-A17B listing priced at ~$0.45 for input and ~$3.00 for output.

Smaller models can be significantly cheaper and may suffice for many classification, retrieval, summarization, and routing tasks, with well-evaluated 4B, 9B, or 27B models potentially outperforming larger models economically.

Calculate cost using the expected input-to-output token ratio, as the ratio of tokens used by the workload can affect which provider is cheaper.

For an agent request consuming 10,000 input tokens and producing 2,000 output tokens, the cost would be approximately $7,800 at 1 million requests, excluding retries, embeddings, storage, routing fees, or other service uses.

Avoid making price comparisons based on an arbitrary 50:50 token split, as the actual ratio can significantly impact costs.

OpenAI-Compatible Options for Qwen

DeepInfra, Together AI, Fireworks AI, Groq, Novita, OpenRouter, and DigitalOcean offer APIs supporting OpenAI-compatible endpoints to varying degrees, allowing for easier migration between providers.

Obtain the provider-specific model ID and base URL from the current provider’s documentation, as the model ID may vary.

import os import time from openai import OpenAI client = OpenAI ( api_key = os . environ [ "INFERENCE_API_KEY" ] , base_url = os . environ [ "INFERENCE_BASE_URL" ] , ) start = time . perf_counter ( ) first_token_time = None output = [ ] stream = client . chat . completions . create ( model = os . environ [ "QWEN_MODEL_ID" ] , messages = [ { "role" : "system" , "content" : ( "You are a production reliability assistant. " "Return concise, technically accurate recommendations." ) , } , { "role" : "user" , "content" : ( "A Qwen inference endpoint has rising p99 latency while " "average latency remains stable. List likely causes." ) , } , ] , temperature = 0.2 , max_tokens = 400 , stream = True , ) for event in stream : if event . choices and event . choices [ 0 ] . delta . content : if first_token_time is None : first_token_time = time . perf_counter ( ) token_text = event . choices [ 0 ] . delta . content output . append ( token_text ) print ( token_text , end = "" , flush = True ) end = time . perf_counter ( ) print ( "\n" ) print ( f"TTFT: { first_token_time - start : .3f } seconds" ) print ( f"End-to-end latency: { end - start : .3f } seconds" ) # Configure it through environment variables rather than placing credentials in source code: export INFERENCE_API_KEY = "your-api-key" export INFERENCE_BASE_URL = "https://provider.example.com/v1" export QWEN_MODEL_ID = "provider-specific-qwen-model-id"

python app . py

This abstraction simplifies initial migration but may require additional work for portable production code, as models can differ in tool-call schema, reasoning controls, JSON enforcement, token usage tracking, context window size, safety filtering, and error codes.

Consider building a custom model gateway and provider-neutral evaluation suite if you expect to switch providers frequently.

OpenRouter: Host or Aggregator?

OpenRouter is best considered an aggregator and routing layer, providing a single API for accessing models served by various underlying providers, which is helpful for applications requiring automatic fallback, centralized billing, or rapid model comparison.

For example, OpenRouter listed Qwen3.5 Plus with a one-million-token context window and separately priced inputs/outputs, with the current models available in its Qwen model catalog.

When interacting with an aggregator, there is less visibility and control over the infrastructure serving the request, requiring examination of data handling requirements at both the routing layer and the underlying provider.

import os from openai import OpenAI client = OpenAI ( api_key = os . environ [ "OPENROUTER_API_KEY" ] , base_url = "https://openrouter.ai/api/v1" , ) response = client . chat . completions . create ( model = "qwen/qwen3.5-plus-20260420" , messages = [ { "role" : "user" , "content" : "Explain continuous batching in three sentences." } ] , max_tokens = 200 , extra_body = { "provider" : { "sort" : "price" , "allow_fallbacks" : True , "data_collection" : "deny" } } ) print ( response . choices [ 0 ] . message . content )

Pinning to a provider can improve predictability, while dynamic routing can offer higher availability or better pricing.

Best Option for Agentic Qwen Workloads

For agentic systems making repeated requests to a model, low time to first token, reliable tool calling, structured output support, large contexts, prompt caching, and stable rate limits are crucial factors to consider when selecting providers.

Together AI and Fireworks may be compelling for agentic development teams seeking to own their production infrastructure and tailor models, while DeepInfra could be attractive if cost is the primary concern and agents generate high token volumes.

Qwen3.5 models are particularly promising for tool-oriented and multimodal agents, but the model’s actual capability with tools should be tested with the application’s specific tools, as benchmark scores may not reliably indicate valid argument production for internal functions.

Build an evaluation set of conversations including failed tool calls, ambiguous instructions, long conversation histories, prompt injection, and faulty tool outputs, and benchmark both completion rate and cost per completed task rather than cost per token.

Serverless or Dedicated Inference?

Serverless inference is typically ideal for prototypes, variable traffic patterns, and teams that don’t want to manage GPUs, with pricing based on tokens used and the vendor handling batching, scaling, and model serving.

However, serverless inference may have downsides such as cold starts, shared-capacity queueing, rate limits, limited customization, and less predictable tail latency, with the magnitude of these effects varying by provider and service tier.

Dedicated inference allocates capacity for a single organization, suitable for sustained traffic, strict latency requirements, or predictable throughput needs, and may offer more control over networking, scaling, observability, and model configuration.

The break-even point depends on utilization, as a dedicated GPU can be economical at high utilization but expensive when idle, while serverless shifts utilization risk to the provider.

DigitalOcean provides both serverless and dedicated inference, with dedicated inference priced per GPU-hour and offering various GPU options.

DigitalOcean’s current inference pricing page lists configurations for dedicated inference.

When to Self-Host Qwen

Self-hosting is suitable when custom weights, adapters, or quantization are required, or when private networking, data residency, or fixed model versions are necessary, offering direct control over the inference engine.

RunPod and GPU Droplets are infrastructure solutions, not hosted model endpoints, requiring the team to choose a GPU, deploy an inference server, configure autoscaling, and manage monitoring and upgrades.

A basic deployment can expose Qwen through an OpenAI-compatible endpoint, but this example is for learning purposes and not meant for production use without additional features like TLS, authentication, and health checks.

# 1) Install vLLM (pin a version to avoid surprises) pip install "vllm>=0.8.4" # 2) Run vLLM with Qwen3-8B over an OpenAI-compatible HTTP API export LOCAL_API_KEY = "your-local-key" vllm serve Qwen / Qwen3 - 8B \ - - host 0.0 .0 .0 \ - - port 8000 \ - - api - key "$LOCAL_API_KEY" \ - - max - model - len 32768 \ - - gpu - memory - utilization 0.90 \ - - trust - remote - code # The application can then call it with the same OpenAI client: # --------- # Python: call Qwen3-8B via OpenAI client # --------- import os from openai import OpenAI # Read the same API key used by vLLM; fall back to a default for demos api_key = os . getenv ( "LOCAL_API_KEY" , "your-local-key" ) client = OpenAI ( api_key = api_key , base_url = "http://localhost:8000/v1" , ) response = client . chat . completions . create ( model = "Qwen/Qwen3-8B" , messages = [ { "role" : "user" , "content" : "Explain continuous batching simply." } ] , temperature = 0.2 , max_tokens = 300 , ) print ( response . choices [ 0 ] . message . content )

DigitalOcean’s GPU Droplets support self-managed approaches, with lower-cost GPUs suitable for smaller quantized Qwen models and multiple high-memory GPU accelerators required for large MoE variants.

DigitalOcean recently launched 1-Click Models for quick deployment of supported open models with an OpenAI-compatible endpoint, and provides a guide on hosting models on GPUs.

Compliance Considerations

Don’t assume a provider is compliant with regulated workloads based on a SOC 2 badge; compliance depends on the service, contract terms, region, data flow, and configuration.

Ask each provider about their SOC 2 report inclusion, business associate agreements for HIPAA workloads, data retention, customer data use, request and data processing locations, private networking and customer-managed keys, subprocessor access, audit logs, and deletion controls.

SOC 2 report inclusion
Business associate agreements for HIPAA
Data retention policies
Customer data use
Request and data processing locations
Private networking and customer-managed keys
Subprocessor access
Audit logs and deletion controls

If using an aggregator, ask additional questions, as another company will be running the underlying inference, and self-hosting provides more control over architecture but requires owning security, patching, and audits.

The best provider for Qwen 3 or Qwen 3.5 depends on the specific workload, not the provider’s overall reputation.

DeepInfra and Novita stand out for cost-sensitive inference, while Groq offers compelling performance for interactive applications, and Together AI and Fireworks provide more full-fledged production environments with advanced serving optimization and customization options.

The safest production strategy is to avoid permanent dependence on untested claims, instead choosing a specific model version, preparing a realistic evaluation dataset, benchmarking a few providers under practical concurrency levels, and estimating costs using the actual input-output token ratio, while keeping the application behind an OpenAI-compatible internal interface.

A reproducible evaluation process is more valuable than any static provider ranking, as inference catalogs will continue to change with new Qwen generations.

Qwen models on Hugging Face
Qwen3.5-27B API reference and pricing
Inference overview
Available serverless models
Querying text models
Supported models
Provider routing and selection

Learn more about available products and services.

About the Authors

The authors are skilled AI consultants and technical writers with extensive experience in AI, data science, and related technologies.

With strong backgrounds in data science and AI, the authors specialize in creating in-depth content on AI, machine learning, and GPU computing, focusing on topics like deep learning frameworks and optimizing GPU-based workloads.

Join the many businesses using DigitalOcean’s Gradient AI Agentic Cloud to accelerate growth, and reach out to the team for assistance with GPU Droplets, 1-click LLM models, AI agents, and bare metal GPUs.

Featured Tutorials and Guides

All tutorials
All topic tags

Please complete your information to access more resources.

Table of contents
Provider selection
Key metrics
Cheapest provider
OpenAI-compatible options
OpenRouter
Agentic workloads
Serverless or dedicated inference
Self-hosting
Compliance
Conclusion
References

Join the community
Popular topics
All tutorials
Talk to an expert
Featured tutorials
All topic tags

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

DigitalOcean Documentation

Full documentation for every DigitalOcean product.

Resources for startups and AI-native businesses

The Wave has everything you need to know about building a business, from raising funding to marketing your product.

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Start building today

From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.

About
Leadership
Blog
Careers
Customers
Partners
Referral Program
Press
Legal
Privacy Policy
Security
Investor Relations

Knowledge Bases
GPU Droplets
Bare Metal GPUs
Inference Engine
Data & Learning
Evaluations
Model Library
Droplets
Kubernetes
Functions
App Platform
Load Balancers
Managed Databases
Spaces
Block Storage
Network File Storage
API
Uptime
Cloud Security Posture Management (CSPM)
Identity and Access Management (IAM)
Cloudways
View all Products

Community Tutorials
Community Q&A
CSS-Tricks
Currents Research
DigitalOcean Startups
Wavemakers Program
Compass Council
Open Source
Marketplace
Pricing
Pricing Calculator
Documentation
Release Notes
Code of Conduct
Shop Swag

AI Training GPU
GPU Inference
VPS Hosting
Website Hosting
VPN
Docker Hosting
Node.js Hosting
Web Mobile Apps
WordPress Hosting
Virtual Machines
View all Solutions

Support
Sales
Report Abuse
System Status
Share your ideas

About
Leadership
Blog
Careers
Customers
Partners
Referral Program
Press
Legal
Privacy Policy
Security
Investor Relations

Knowledge Bases
GPU Droplets
Bare Metal GPUs
Inference Engine
Data & Learning
Evaluations
Model Library
Droplets
Kubernetes
Functions
App Platform
Load Balancers
Managed Databases
Spaces
Block Storage
Network File Storage
API
Uptime
Cloud Security Posture Management (CSPM)
Identity and Access Management (IAM)
Cloudways
View all Products

Community Tutorials
Community Q&A
CSS-Tricks
Currents Research
DigitalOcean Startups
Wavemakers Program
Compass Council
Open Source
Marketplace
Pricing
Pricing Calculator
Documentation
Release Notes
Code of Conduct
Shop Swag

AI Training GPU
GPU Inference
VPS Hosting
Website Hosting
VPN
Docker Hosting
Node.js Hosting
Web Mobile Apps
WordPress Hosting
Virtual Machines
View all Solutions

Support
Sales
Report Abuse
System Status
Share your ideas

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!