Your AI Bill Isn't the Model. It's the Plumbing.

Sixty-two percent of senior IT leaders say they are paying an inference tax.

That is Google Cloud's term, from a survey of more than 1,400 of them, and it covers three specific costs: data egress fees, storage bloat, and idle specialized hardware.

Worth saying plainly before we go further. Google Cloud sells infrastructure, and a report finding that enterprises have infrastructure problems is a report that sells infrastructure. Read the number with that in mind. Then notice that all three mechanisms are things you can go verify in your own billing console this afternoon, which is more than most vendor research gives you.

The Bill You Are Watching Is Not the Bill

Two weeks ago I wrote that cheaper tokens do not mean a cheaper bill. That post was about token spend: context payloads, routing, chatty tools.

This is a different problem, and in some ways a worse one.

The costs in that survey are not token costs at all. They do not appear on your model provider's invoice. They land under cloud infrastructure, storage, and networking, on a different line item, usually owned by a different person.

So you get a specific and very common failure. The person who owns the AI budget watches token spend and reports that it is under control. The person who owns the cloud bill sees egress and storage climbing and has no idea it is AI. Both are telling the truth. Nobody is looking at the actual number, because the actual number does not exist anywhere as a single figure.

That is not a technology problem. That is an accounting boundary problem, and it is the reason this cost category grows quietly for a year before anybody names it.

Three Places It Leaks

Data egress. Every time your retrieval pipeline pulls documents across a cloud boundary or a region boundary, you pay per gigabyte to move them. If your vector store lives in one place and your model endpoint lives in another, you are paying a toll on every single query. That cost scales with how often people use the thing, which means it grows exactly when the project starts working.

Storage bloat. Embeddings. Chunked copies of documents you already store elsewhere. Cached responses. Evaluation datasets. Model checkpoints nobody will load again. Agent execution traces, which I wrote about eleven days ago for entirely different reasons.

Every one of those is small. None of them has a retention policy. All of them only grow. Ask your team when anything in the AI stack last got deleted and watch the pause.

Idle specialized hardware. You reserved GPU capacity for a workload with spiky demand, because the alternative was capacity you could not get when you needed it. So you provisioned for the peak and you pay for the trough. That is often a defensible decision. It is only a bad one when nobody has looked at the utilization number since the day it was provisioned.

The Pilot Ran on Borrowed Capacity

Here is the finding in that survey that explains why this keeps surprising people.

Eighty-three percent of organizations say they need infrastructure upgrades before they can support production-grade agentic AI. Eighty-one percent call operational complexity a hidden cost of scaling.

Read those together. The pilot worked. Of course it did. A pilot runs on spare capacity, in one region, with ten users, against a small document set. None of the three mechanisms above are visible at that size, because at that size they cost almost nothing.

Then it goes to production and every one of them turns on at once. The architecture that was fine for a demo is now moving real volume across boundaries that charge per gigabyte.

The pilot did not lie to you. It just was not measuring the thing that eventually costs money.

What I'd Do This Week

Get one number for total AI spend. Tokens plus the infrastructure attributable to AI workloads. If nobody can produce that in an afternoon, you have found the actual problem, and it is not technical.

Tag AI infrastructure at the resource level. Storage buckets, endpoints, reserved capacity, egress-heavy paths. You cannot attribute what you never labeled, and retrofitting tags is far worse than applying them now.

Look at where your data sits versus where inference happens. If those are different regions or different clouds, you are paying a per-query toll. Co-locating them is often a one-week change with a permanent effect on unit economics.

Set a retention policy for AI artifacts. Embeddings, traces, caches, eval sets, checkpoints. Pick a number of days for each. Anything with no owner and no expiry will still be there in three years.

Check GPU utilization against what you reserved. Not once. Monthly, with a name attached. Reserved capacity is a bet, and bets should be reviewed.

The Part That Matters

The model is the part everyone watches because it is the part with a price list.

The plumbing is where the money actually goes, and it hides in the cloud bill under a category nobody associates with AI.

You do not have a cost problem. You have a visibility problem that turns into a cost problem.

Go find the real number first.