The money behind AI infrastructure is moving from building models to running them. As you take AI out of development and into customer-facing and operational systems, your models have to work in real time, all the time. Gartner expects inference spending to overtake training on AI-optimized infrastructure as a service (IaaS), at $23.3 billion against $19 billion.
It also expects total worldwide spending in this category to reach US$42 billion, up 96% on the year before. Inference’s share of it is set to keep growing, from 55% to 59% the following year.
This means that instead of experiments, companies are now putting more of their AI budgets into models that read data on every request. That said, moving from pilot to production is essential and it changes what your data layer has to handle.
The companies that are seeing results from AI have already prepared for this.
A Gartner survey of 353 data, analytics and AI leaders found that 39% leaders are confident that their AI investments will improve their financial performance. In fact, the organizations with successful AI initiatives invest up to four times more, as a share of revenue, in foundations such as data quality, governance and change management.
Your AI can only scale as far as your data architecture allows. Often, that limit comes from decisions made before AI was even part of the plan. Let’s understand it in a bit more detail, and what decisions you need to make today to ensure your AI scalability tomorrow.
What changes when AI moves from pilot to production
Your data layer goes from being queried now and then to being read on every request.
Here’s how it works. In a pilot, a few people query your data once in a while. In production, AI models query it all the time for every user and every request because now your AI models are running inside customer-facing and operational systems. Those systems can’t wait for data.
So the speed, data freshness and cost per query start to matter in a way they never did during the pilot. It translates to checking:
- Concurrency: How many requests hit your data at once
- Latency: How fast each request gets an answer
- Data freshness: How current the data is
- Billing: How much are you getting charged per call
If your architecture can only handle these four after a complete rebuild, it has a scalability ceiling, however much spare capacity it reports.
So before you scale any use case, work out the load it will create at full volume in a sandbox. Not every use case needs real-time data. The real question is whether one architecture can serve both kinds of use cases without a rebuild.
Five data architecture decisions that set the AI scalability ceiling
Here are the five decisions required to set the foundation.
1. Building a semantic layer
The semantic layer holds one agreed definition of each business term, so everyone reads the data the same way. Without one, every new model or agent may define “active customers,” for instance, in its own way – one may define the purchases made in the last 30 days to be active, and the other might go with 90.
Eventually, the versions will drift apart, and each agent will end up working from its own reading of the same data. As a result, you may get two different answers for the same question based on their definition of the business term related to the query, which will impact the final decision made using that data.
According to Gartner, by 2030, companies will rank the semantic layer alongside their data platforms and cybersecurity. So, you need to set it up before your next AI use case goes live.
2. Whether data lineage is recorded now or rebuilt later
Data lineage is a record of where each piece of data originated, what changed along the way and where it ended up. If it’s not recorded, you can’t rebuild it when an auditor asks which data produced an AI answer.
AI needs this at a finer level. You have to know which version of a document your AI read and which parts of it shaped the answer. Regulators want this too. EU rules for high-risk AI require activity logs that make results traceable, and they’re due to apply on businesses from December 2027 under a provisional agreement.
A data lineage that your pipelines can record automatically becomes a vital element.
3. How will your data move
This step defines whether your data feeds can switch to real-time when it’s in production, without breaking the systems that read them.
Now, there are two methods designed to handle data: Batch data and streaming data.
Batch data is only as fresh as its last run and still suits reports and trend analysis. Plus, it’s simpler and cheaper to run. Streaming data is processed as it’s created, so systems can react within milliseconds. Streaming matters when timing decides the outcome, for instance, to catch fraud right when it happens.
Many organizations use both.
The risk here is locking yourself into a batch. We’ve covered this and 5 more warning signs in depth here: Six signs your data architecture is holding back AI adoption.
So, what you need to do instead is put a data contract between each feed and the systems that use it. It spells out what they receive, so you can change how the data arrives without breaking anything downstream.
4. Where will AI cost be attributed
AI is billed in pay-per-token services that charge for what it’s used for. Reserved capacity charges by the hour, even when it’s idle.
Either way, if you’re using a shared bill, you’ll have difficulty finding which use case is worth running. So tag every AI workload to its use case when you set it up to track the billing.
5. Where your data is allowed to be stored and used
This is called data residency. You’ll need to decide which country your data or data centres will be physically located. Data protection laws of different countries may vary with some restricting where certain data can go.
In India, for example, the Reserve Bank requires the payment system data to stay in the country. A few years back, when Mastercard violated this rule, they were prohibited from issuing any new debit or credit card.
Why these data architecture decisions determine AI scalability tomorrow
Because every AI tool you add is built on top of them. Each new AI agent, model or dashboard reads your data using the definitions, structure and feeds you’ve already set up. So, if you change any of those, everything reading them has to change too.
- Semantic layer: Adding a semantic layer means agreeing on one version of definitions, and if any change is required, you’ll just need to make one. And every AI agent, dashboard or model will get it right.
- Data lineage: A record that was never kept can’t be recreated. Any AI answer given before you start recording can’t be traced back to its data.
- Data movement: If systems read a batch feed directly, switching it to real time later can mean reworking each of them. A data contract in between avoids that, because those systems only depend on the contract.
- AI cost: If you don’t tag the AI spendings by use cases from the start, months of AI costs will pile up on one shared bill. Splitting which is already hard, and without these tags, you can’t find out which use case the spend belongs to in the overall bill.
- Data residency: If an AI use case turns out to need the data that must stay in one country, you may have to move where this AI will run, along with the prompts, answers and logs it has already created. Something much costlier to do later than getting it right from the start.
Besides that, the number of AI tools reading your data is rising fast. Among companies with more than $1 billion in annual revenue, the share of respondents scaling AI agents has risen from 27% to 40% in a year, according to a McKinsey survey.
Every new agent is one more thing to update when you change something. That’s why deciding early pays.
Decide your AI ceiling now, while it’s still cheap to change
The limit on your AI is being set right now, and every AI workload you add makes those decisions harder to undo. So, it’s better to agree on one set from the get go.
Most importantly, do it before your next AI use case goes live.
