GPU Surcharges

When it comes to GPU bills, the per-hour rate is just the beginning. There are four hidden surcharges that can significantly increase your costs: egress, idle time, the noisy-neighbor tax, and cold-start latency. In this article, we'll explore each of these surcharges and provide a step-by-step guide on how to optimize your costs.

The workload we’ll price: a real-time 70B inference API on four H100s

We'll use a real-time inference API serving an open 70B-class model on four H100s as our example workload. This setup is common and gets hit by all four surcharges at once. The on-demand pricing for H100s has fallen to around $3.93 per GPU-hour, making the total compute cost for four H100s running around the clock approximately $11,300 a month.

Surcharge 1: Egress, plus the NAT and cross-AZ taxes stacked on top

Uploading data into a cloud is free, but getting it back out is not. The headline rate for internet egress is around $0.09/GB on AWS, $0.087 on Azure, and $0.12 on GCP Premium. However, two line items quietly stack on top of it: the NAT Gateway and cross-AZ traffic.

7,500 GB × $0.09 ≈ $675
NAT Gateway: 7,500 GB × $0.045 ≈ $338
cross-AZ chatter: assume replicas are split across two availability zones behind a load balancer, so roughly half the request/response volume about 4 TB, crosses a zone boundary. At $0.01/GB in each direction that’s $0.02 round-trip, or ≈ $80

This results in roughly $1,090 a month in data movement, about 10% on top of compute. For data-heavy workloads, transfer routinely lands in the 10-15% range, and on distributed multi-AZ architectures, higher still.

Surcharge 2: Idle time - you’re paying for GPUs doing nothing

The average GPU utilization is around 5%, meaning organizations are provisioning roughly 20 times the GPU capacity their workloads use at any given moment. This results in a significant idle tax, with 70% of the four GPUs sitting idle and only about $3,400 doing useful work.

Surcharge 3: The noisy-neighbor tax that never appears on the bill

In a multi-tenant GPU environment, you share physical silicon, and a GPU has two things that are easy to starve: VRAM and memory bandwidth. This can result in a noisy-neighbor tax, where your service needs more hardware to compensate for the interference from co-tenants.

Surcharge 4: Cold starts turn scale-to-zero into the idle tax

Scale-to-zero is the obvious answer to the idle tax, but it comes with a catch: the cold start. When a request hits a cold endpoint, the platform creates the container, initializes the ML runtime and CUDA context, fetches weights from object storage, loads them into VRAM, and warms up CUDA graphs and the KV cache. This can take 30 to 90 seconds, dominated by fetching weights.

The total bill for our example workload on a hyperscaler is around $18,050 a month, with only about 30% of the base compute doing useful work. In contrast, single-tenant dedicated capacity can delete the surcharges, resulting in a total bill of around $12,800 a month.

Why the usual fixes: serverless and reserved commitments fail

The usual fixes, such as going serverless or buying bigger reserved commitments, aren't helpful. Serverless moves the cost into cold-start latency, while reserved commitments lower the fare but do nothing about egress, the neighbor, and the idle tax.

Technology teams are watching gpu surcharges closely because changes in this space often arrive faster than internal policies can adapt.

For product and engineering leaders, the practical question is how this could reshape roadmaps, vendor choices, and security reviews over the next few quarters.

Organizations that document lessons early tend to respond more calmly when similar patterns appear again.

In many companies, the first impact shows up in planning meetings: teams reassess priorities, revisit risk registers, and check whether existing tooling still fits.

Smaller businesses feel these shifts too. A single platform change or market move can affect customer trust, delivery timelines, and hiring plans.

The most resilient teams treat stories like this as input for quarterly reviews rather than one-day headlines.

If your business depends on modern software, ERP, VoIP, or customer-facing apps, staying informed helps you separate noise from decisions that require action.

Looking ahead, disciplined follow-through matters: assign owners, set review dates, and measure whether your response improved outcomes.

Security and compliance stakeholders should ask whether current controls still match the pace of change described in this update.

Operations leaders can reduce friction by translating the headline into a short internal brief with clear next steps for each department.

Customer support teams may see early signals through tickets, outages, or policy questions long before leadership reviews are scheduled.

Finance and procurement groups should note whether licensing, vendor risk, or implementation costs need revisiting after this development.

Training programs benefit from timely updates so staff understand what changed, what did not change, and what requires escalation.

To optimize your GPU costs, you need to address the architecture problems, not just the optimization problems. This includes keeping traffic on a private path, stopping silicon sharing, and using dedicated capacity to eliminate the surcharges.