Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.
This one is a little different.
Regular readers will know that I’ve recently been exploring AI hardware from a few different angles. I’ve explored AI accelerator architectures in a series of posts with Avik. I looked back at ENIAC and compared it with today’s infrastructure buildout. And my most recent post was a deep dive on LLM inference execution.
Another key aspect of this whole story is the economics: what happens to a datacenter GPU after it’s deployed, how its value changes over time, and under what conditions older hardware can remain economically useful long after newer chips arrive. This part of the story is a little far from my world, but I had the opportunity to ask someone who is better equipped than me to answer a few questions about what’s really happening in the AI infrastructure world today…
Bharath: For readers who don’t already know, could you introduce yourself and what you do?
Carmen: I’m the CEO of Silicon Data and Compute Exchange. My background is in financial markets, including roles at Bloomberg, Citi, and DRW, and that shaped how I initially approached the compute market.
What interested me was that compute was becoming one of the most economically important inputs into AI, but the market lacked many of the things we take for granted in mature markets: transparent pricing, standardized benchmarks, forward curves, secondary liquidity, and tools for understanding risk.
At Silicon Data, we focus on measuring and benchmarking the economics and performance of compute. At Compute Exchange, we operate a neutral marketplace connecting buyers with more than 100 compute providers. The two give me an interesting view into both how compute is measured and how it actually trades in the market.
Bharath: Can you walk us through the lifecycle of a GPU, from being purchased and deployed in a datacenter to eventually being resold, refurbished, or retired?
Carmen: A GPU can have a much longer and more complicated economic life than people assume. The first owner might deploy it for training or inference in a large cluster and rent that capacity under multi-year contracts. When newer hardware arrives, that doesn’t necessarily mean the older GPU becomes obsolete. It can move to different workloads, another operator, another geography, or eventually into the secondary hardware market.
The A100 is a good example. It launched in 2020, but years later there is still demand for A100 capacity, and we’ve seen the market support multi-year contracts for it. The economics change over time, but the hardware can continue generating value.
Eventually, an operator might remove the GPU from its original environment, test and refurbish it, and sell or redeploy it. So rather than thinking about a GPU as having one useful life, I think about it as moving through different economic lives as its price, performance relative to newer hardware, and ideal workloads change.
Bharath: When a new GPU generation launches, what happens to the value and demand for the previous generation?
Carmen: New hardware doesn’t immediately make the previous generation worthless. What changes is the price at which that older compute makes sense.
The newest GPUs are extremely attractive for workloads where performance, memory capacity, or power efficiency justify the premium. But there are plenty of workloads where you don’t need the newest architecture. If an older GPU can perform the job effectively at a sufficiently lower price, it can remain economically competitive.
That’s why the lifecycle of these assets is interesting. A GPU can move from being cutting-edge infrastructure to lower-cost general compute without becoming useless. The market essentially keeps repricing the hardware as newer alternatives become available.
Here are two Silicon Data pricing charts that demonstrate this.
The A100 index tracks rental pricing from September 2024 through October 2026, showing an 8.43% decline to $1.52/GPU-hour.
In contrast, the newer B300 index has risen 45.34% since April 2026 to $7.02/GPU-hour.
Bharath: Why are GPUs like the A100 still economically useful years after newer architectures have entered the market?
Carmen: Because buyers ultimately need compute, not the newest chip.
If you’re training the largest frontier model, the economics may strongly favor the latest architecture. But there are many smaller models, enterprise workloads, fine-tuning jobs, research workloads, and inference applications where an A100 can still perform perfectly well.
The relevant question isn’t simply, “Which GPU is fastest?” It’s, “What is the most economical way to complete this workload?”
That distinction becomes increasingly important as the market matures. Not every workload should run on a B200 just because a B200 exists. In many cases, older hardware at the right price can be the more rational choice.
Bharath: If two clusters use the exact same GPU model, why can they still deliver meaningfully different real-world performance?
Carmen: This is one of the biggest misconceptions in the compute market. An H100 is a standardized chip, but an H100 cluster is not a standardized product.
You have the GPU itself, but then you have CPUs, memory, storage, networking, interconnects, software configuration, cooling, and the topology of the overall cluster. All of those variables can affect what the customer actually experiences.
At Compute Exchange, where we see capacity across more than 100 providers, this becomes very apparent. Two providers can both tell you they’re selling the same GPU model, but that doesn’t necessarily mean you’re buying equivalent compute.
This is why benchmarking and verification become important. Knowing the chip model is the beginning of understanding what you’re buying, not the end.
Bharath: How much do networking, CPU and RAM configuration, interconnects, cooling, and cluster architecture affect the performance and economic value of a GPU?
Carmen: Potentially a lot, and the importance of each component depends on the workload.
For distributed training, networking and interconnect performance can be critical because the GPUs need to communicate efficiently. Other workloads may be more sensitive to memory, storage, CPU configuration, or data movement.
There are physical constraints as well. A datacenter designed around an older generation of hardware may not necessarily be able to replace those systems with the newest GPUs. Power density and cooling requirements can be completely different.
So the GPU shouldn’t really be evaluated in isolation. As clusters get larger, you’re increasingly evaluating the system around the GPU.
Bharath: Is $/GPU-hour actually a good way to compare compute, or should buyers be looking at the cost of completing a workload instead?
Carmen: $/GPU-hour is useful because it’s simple, but it can be misleading if you treat it as the entire answer.
Imagine one provider offers an H100 at a lower hourly price, but the workload takes materially longer to complete because the cluster performs worse. The cheaper GPU-hour could produce a more expensive workload.
What ultimately matters is some version of price-performance: how much useful work am I getting for what I’m paying?
That’s why I think the market will become more sophisticated about benchmarking. Buyers will increasingly want to understand not just the cost of accessing a chip, but throughput, utilization, reliability, and the total cost of completing their workload.
Bharath: How should buyers think about choosing between A100s, H100s, H200s, B200s, and alternative accelerators based on real-world economics rather than spec sheets?
Carmen: Start with the workload rather than the GPU.
What are you trying to run? How much memory do you need? How sensitive are you to latency? How much does time-to-completion matter? Do you need a large interconnected cluster? How long do you need the capacity? And how much are you willing to pay for incremental performance?
Then you can look at what’s actually available in the market. One of the things we’ve learned building Compute Exchange is that availability matters more than people sometimes realize. A theoretically optimal GPU isn’t very useful if you can’t get enough of them in the configuration and location you need.
The market is moving toward matching the right compute to the right workload, rather than assuming everyone should always migrate to the newest accelerator.
Bharath: Speaking of matching compute to the workload, what’s the state of benchmarking when it comes to datacenter computing? Are current benchmarks reliable? If not, what would make a good benchmark to compare different compute clusters?
Carmen: I don’t think there is one benchmark today that tells you everything you need to know about a compute cluster. The challenge is that a benchmark can accurately measure one dimension of performance while still failing to tell you how the system will perform on your actual workload.
An ideal benchmarking framework needs multiple layers. At the hardware level, you want to understand things like compute throughput, memory bandwidth, host-to-device bandwidth, networking, kernel overhead, power consumption, and thermal behavior. At the cluster level, you need to understand how effectively GPUs communicate and scale together. Then you ultimately want workload-level measurements such as training throughput, inference throughput, tokens per second, step time, reliability, and performance consistency.
There are widely used benchmarks today, including MLPerf, as well as more targeted hardware and workload benchmarks. They are valuable, but buyers should understand what each benchmark actually measures. A strong result under a standardized benchmark doesn’t necessarily mean that cluster will be the most economical option for your particular model and workload.
Bharath: As GPUs move between owners and datacenters, what needs to be measured or verified for a secondary GPU market to function efficiently?
Carmen: Buyers need confidence in both what the hardware is and what condition it’s in.
That can include verifying the chip identifier and system configuration, but also looking at network health, measured performance, thermal behavior, and other indicators of how the system has been operating. With used hardware, condition becomes another dimension of price discovery.
This is where the GPU market starts looking more like other mature equipment markets. A six-year-old aircraft isn’t valued simply by saying, “This is this model of aircraft.” Maintenance history, condition, configuration, utilization, and remaining economic life all matter.
Compute doesn’t yet have decades of residual-value history like aircraft or other mature equipment markets. We’re building that history now.
The broader opportunity is to create enough measurement, verification, price discovery, and secondary liquidity that a buyer can understand what a GPU is worth throughout its economic life. That’s what ultimately allows these assets to move efficiently between workloads and owners rather than becoming stranded every time a new architecture arrives.
Bharath: In summary, what do you see happening to datacenter GPU demand in the next 3-5 years? What fundamental trends are you tracking to support this view?
Carmen:
I expect aggregate demand for AI compute to continue growing over the next three to five years, but I also expect the market to become much more segmented.
Today there is a tendency to talk about GPU demand as though every buyer wants the same thing. In reality, frontier training, inference, fine-tuning, research, enterprise AI, and smaller models can have very different infrastructure requirements.
The fundamental trends I’m watching are inference growth, model efficiency, token consumption, utilization, power availability, alternative accelerators, and the price-performance improvements of each new hardware generation. I’m also watching how quickly workloads migrate from general-purpose GPUs toward more specialized infrastructure.
The result may seem counterintuitive: demand for the newest accelerators can grow dramatically while older generations continue to have substantial economic lives. As the market matures, I think we’ll increasingly stop asking, “What is the best GPU?” and start asking, “What is the most economical compute for this particular workload?”
That’s the end of this Q&A. Kudos to Carmen and team for helping me put this together. You can follow Carmen’s work via her socials (Linkedin, X). And if you’re new to Chip Insights, subscribe and stay tuned for more content like this.