Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

In the last few weeks, I’ve been trying to write a post about certain trends I’ve observed in Agentic AI hardware - about CPUs, ISAs, Memory, etc. To do this, I read existing commentary from others to defend my hypothesis, but they all ended up in one of these buckets:

• Company marketing highlighting why a specific aspect is more important

• Decisions that were made due to the prevailing supply constraints like HBM shortage

• A narrow focus on a specific type of agent (typically claude-code style agents)

Since I’m not trying to sell you a chip, I can’t predict the semiconductor supply chain, and I don’t have a favorite agent, I could never complete the post I wanted to write. Instead, I’m taking a step back and starting with a more fundamental question:

How does LLM Inference run locally on your computer?

I’m not going to claim this post will teach you something fundamentally new. In fact, most of what I’m going to talk about is just traditional software execution on a system with a CPU and a GPU. But this post serves as the base camp for some other interesting questions I want to raise in later posts.

What does the software look like?

There are many different ways to launch an LLM inference call. For this post, I’m using a simple Python script that executes the inference using Ollama. This is what the script looks like:

# Import request library to send/recieve HTTP requests
import requests

# Send the LLM Inference call as a HTTP request through Ollama
r = requests.post(
        "http://localhost:11434/api/generate",                           # localhost:11434 indicates that the request should be sent to local Ollama port
    json={                                                               # the model name, prompt, and other settings are passing as the JSON payload
        "model": "qwen3:8b",
        "prompt": "What's more important for Agentic AI - CPU, or GPU?",
        "stream": False,
    },
)

# Print the output of the LLM inference call
print(r.json()["response"])

For this post, this is all you need to know about this script:

• The script runs LLM inference using the qwen3:8b model.

• The prompt sent to the LLM is “What’s more important for Agentic AI - CPU, or GPU?”

• Streaming is disabled - so the final output will be printed only after all output tokens are produced.

What does the hardware look like?

Since the software runs LLM inference locally, (no requests leave the computer where the script was launched) I’m assuming a standard personal computing setup with a host CPU (which runs the Operating System (OS)) and a discrete GPU.

In a system like this, the CPU and GPU typically have separate physical main memories: the CPU accesses System DRAM, while the GPU primarily accesses dedicated Video RAM (VRAM). There is also a large, non-volatile Solid-State Drive (SSD), from which the OS loads data into System DRAM on demand. There are usually more levels in the memory hierarchy, but for simplicity, we will skip these cache effects.

At a high level, this is all you need to know about the CPU-GPU system - I’ll be sharing more details as we walk through the execution.

Is anyone running Local LLMs on their computers?

I expect this question to come up at some point, so I want to get ahead.

You might wonder whether a personal computing hardware architecture like this is relevant in the era of large AI data centers. It’s a fair question, but there are two reasons why I went with local LLM inference on a discrete GPU system:

• Datacenter hardware architectures are vendor specific and evolving so quickly, so it’s hard to write a post like this without making a lot of assumptions (which I think would dilute the post.)

• I own a computer with a CPU and GPU which I can use to run some profiling for future posts.

But irrespective of the hardware architecture, the fundamental interaction between a CPU running the OS and an accelerator like the GPU will remain largely the same - that’s going to be the key focus of this post.

The shipping analogy

I recently drove past the Port of Los Angeles, the busiest seaport in the US. I knew it was big, but looking out at the large ships and the thousands of containers waiting to be loaded in them left an impression on me. So that’s my excuse to include it as an analogy for this post.

A shipping port is a good analogy for the AI computing landscape because of the heterogeneity. In addition to ships, a port is also closely connected to major freeways, railway lines, and airports - not too different from the different “Processing Units” that we have in the computing world today.

There’s something else that maps well to Agentic AI in particular - the variety of loads handled by a port. The logistics of shipping perishable goods is not the same as automobiles, and is completely different from how a cruise ship would be managed. I think this can be a good mental model to differentiate the requirements of coding agents, versus, say, a robotics pipeline.

Going forward, any references to this analogy will be in quotes to separate it from the rest of the text. Use it to simplify concepts, or to just give yourself a break.

Computer Science 101

I will be using (and have probably already used) a lot of computing terms in this post without properly introducing them, so here’s a quick glossary covering most of them. Skip ahead if you are already familiar with these.

• Operating System (OS): It’s the system software that manages everything - from the interaction between CPU, GPU and other hardware resources, to managing permissions to processes and device drivers.

• Process: It is a software abstraction created by the OS that provides a virtual address space and resources for running a program. A process contains one or more threads.

• CPU Thread: It is an execution context with its own architectural register state and instruction pointer that can be scheduled to run on a CPU core.

• CPU Core: Physical CPU hardware that can fetch and execute instructions.

• Driver: It is software that allows the OS and applications to interact with a hardware device like the GPU.

• GPU Kernel: A function/program executed on the GPU across many parallel threads.

• Virtual Address: It is the address used in the software world. The OS and hardware translate it to physical hardware addresses through page tables.

• HTTP: A commonly used request/response protocol that allows one application to talk to another. (either within the same system, or through the internet.)

• TCP: It is a popular byte-stream protocol used by HTTP connections to transfer data.

• JSON: A standard format for serializing structured data like lists, arrays, or dictionaries.

• UTF-8: It is a standard encoding that represents unicode text (in JSON) as bytes (binary).

You can think of the OS like the Port Authority which manages port access for different entities.

Each process is like a logistics partner involved in each shipping task - like the company that wants to ship their goods, freight and ship providers, and so on.

Each CPU thread is like a Truck that carries Containers (i.e. data) to the GPU (which is like a Ship), and together they complete the shipping task. (i.e. the LLM inference script.)

LLM Inference Walkthrough

Enough buildup.

Say you have your LLM inference script on a file: llm_inference.py

Let’s look at everything that happens when you launch this command on your shell: python llm_inference.py

Phase 1: Prepare the Python script for execution

TL;DR: OS creates a Python process, creates bytecode for your script, and prepares objects needed for execution.

Hardware Involved: CPU, System DRAM, and SSD

In this phase, we won’t start physically shipping any goods yet. The Python process is like a company with a shipping request that first gets all the required information and approvals from the port authority (OS) before it can start the shipping task.

Step 1: Shell interprets the command

Your shell (Bash, Powershell, etc.) parses the entered text and interprets it as:

• Program to launch: Python

• Argument passed: llm_inference.py

Step 2: OS creates a Python process

The shell asks the OS to launch the “Python executable”. (More on this in step 3.) To do this, the OS creates a new Python process. The new Python process is provided with what it needs to execute:

• A process ID

• An initial thread

• Virtual address space that it can access

Whenever we see the Python process “executing” something on the CPU, remember that it is done through this initial thread.

Step 3: OS prepares the Python executable

Once the process is created, the OS maps virtual memory addresses needed for the Python executable to System DRAM locations and loads the pre-compiled Python executable and the required libraries from the CPU SSD to the DRAM. I’m greatly oversimplifying this step since it’s not the focus of the post, but here are some important notes about this step:

• Python code execution is a little different from a compiled language like C. In C, your program is compiled into machine code, and the CPU fetches and executes the binary representation of each instruction. However, Python is an interpreted language which is executed differently. Instead of compiling your Python script, there is a pre-compiled Python executable which is like a simulator that can run any Python script. (In the case of CPython, this executable is written in C.) Instructions from this pre-compiled executable are sent to the CPU. Your actual script is converted into a bytecode that the pre-compiled executable reads and executes.

• The OS usually does not read every byte of CPython and all shared libraries from SSD into DRAM at once to avoid unnecessary RAM use. Instead, the OS only fetches data at (and near) the current virtual address being accessed.

Step 4: Python executable converts your script into bytecode

• The Python executable starts running on your CPU, and very quickly loads the script into the DRAM. (through the SSD if it is being read for the first time.)

• The executable then decodes the script text, parses it, and converts your script into an internal representation called bytecode.

Although this step looks a lot like compilation, this is different because the bytecode is not in assembly language, and it is not directly passed as CPU instructions. It is just a different representation that the Python executable can understand.

For example, this is one possible CPython bytecode sequence for printing the value in a variable result.

PUSH_NULL
LOAD_NAME print
LOAD_NAME result
CALL 1
POP_TOP

While it looks like assembly instructions, you can’t run this directly on any CPU.

Step 5: Python executable starts executing the bytecode

Once the bytecode is ready, the Python executable executes it, starting from the top module. For the first time in this walkthrough, let’s see something related to our LLM call. Since our script needs to send an HTTP request to Ollama, we first need to import the requests Python package.

This is how the Python executable imports a package:

• Find the Python source (or an existing bytecode) for the package

• Convert it into bytecode (or load existing bytecode if available)

• Execute the top level instantiation from the bytecode, and store the module object for future use.

Similarly, the Python executable creates objects needed for other variables used in the script like the HTTP address, model name and prompts. These objects all live in the System DRAM and are mapped to unique virtual addresses.

Say the shipping request is from a factory that manufactures shoes. But the boxes for the shoes need to be fetched from somewhere else. Imports can be thought of like this - they are specific implementations of the shipping request which can be found elsewhere.

At the end of Phase 1:

The Python process has set up all the objects, and is ready to execute the first LLM request.

Phase 2: Send the LLM request to Ollama

TL;DR: The Python executable executes the requests.post call - i.e. the Python process sends a local HTTP request to the Ollama process.

Hardware Involved: CPU, System DRAM

The Python process starts executing our script like any other Python script, and quickly runs into the Ollama request:

r = requests.post(
        "http://localhost:11434/api/generate",
    json={
        "model": "qwen3:8b",
        "prompt": "What's more important for Agentic AI - CPU, or GPU?",
        "stream": False,
    },
)

Unlike other Python commands, which are fully executed within the Python process, this HTTP request needs the involvement of the OS, as we will see in the steps below.

Step 1: The Python process encounters a call to the imported bytecode

In the compiled bytecode of our agentic workflow, the requests.post() call redirects the Python process towards the bytecode that was imported in Phase 1. This is needed because our script’s bytecode does not know how to execute this command. (Note that we are still within the Python process - we are just reading a different bytecode.)

Step 2: Python process creates a request object

Using the imported bytecode, the Python process creates an HTTP request with the following information:

• Model name

• Input prompt

• Request settings (streaming, local/online request, etc.)

• Information about the tools if applicable

Essentially, the Python process packages all the information that Ollama might need to run the LLM inference into a single request. This request, typically represented as a Python dictionary, is then stored in System DRAM.

Step 3: Python process serializes the request object to JSON bytes

Although the information needed by Ollama is ready in Step 2, Python objects can’t be sent directly in an HTTP request through a TCP socket. So the Python process:

• Reads the Python objects (saved in step 2) from the System DRAM

• Serializes them using the JSON serializer (also part of the requests bytecode.)

The output is a sequence of UTF-8 bytes which is stored back in the System DRAM.

Step 4: Python process initiates an HTTP request to the Ollama process

Now that the data needed for the HTTP request is ready, the Python process constructs the request in this step. Essentially, Python tells the OS to create a TCP connection to:

• IP Address: 127.0.0.1 (localhost corresponds to this address.)

• TCP Port: 11434 (this is the default Ollama listening port)

The OS establishes the TCP connection through the local loopback interface. (It does not actually leave the computer.) Once connected:

• The Python process requests the OS to send the HTTP request with the JSON bytes from the System DRAM. (Stored there in step 3.)

• Once complete, the Python Thread that was executing everything so far moves into a blocked/waiting state.

This is a key milestone in the CPU execution. Watch out for when this thread is reactivated. (Spoiler alert: It’s not very soon.)

Once the shoes are packed in the boxes, they are stored and ready to be picked up. The shoe manufacturer gets in touch with the shipping company to arrange for their shoes to be shipped from their factory to its destination. You can think of the Ollama process as the shipping company. Once the shipping company picks up the boxes, the shoe manufacturer just needs to wait for the delivery.

Step 5: The Ollama process accepts and parses the request

While I didn’t explicitly mention this so far, Ollama is typically already running as a separate process. (Usually started at boot, or manually with the ollama serve shell command.) The process has a thread that is in the waiting state, and is listening on its default port 11434 - the same port that Python asked to send the LLM inference request.

Note that the Ollama thread is still a CPU thread - we have not started the GPU execution just yet.

Although the shipping company’s main task is ocean shipping, they still need Trucks (CPU) to move goods to and from the container ship.

The OS executes the HTTP transfer requested in step 4 on the TCP port by:

• Moving the request bytes from the Python process into the local socket/network stack

• Placing the bytes into Ollama’s socket receive buffer

• Marking the Ollama thread as runnable

The OS then schedules the now ready Ollama thread on an available CPU core. The handshake between the two processes has completed, and the execution will now be taken forward by the Ollama process.

At the end of Phase 2:

The Python process has handed off information for LLM inference to the Ollama process.

Before we continue: What would inference on a cloud model look like?

Subscribe for free to see this section:

Phase 3: Ollama prepares the GPU for LLM inference

TL;DR: Ollama finds a compatible GPU (if available) to run inference, sets up a communication channel between the CPU and the GPU, and moves the required data into the GPU VRAM.

Hardware Involved: CPU, System DRAM, SSD and GPU (and it’s VRAM)

At the end of Phase 2, the Ollama process has received the LLM inference request from the Python process. Now, it’s time to start the LLM inference. But since LLM inference will be running on a GPU, the Ollama process still has some preparation left to do.

Before going into the steps, let me define two key terms used in the context of Ollama execution:

• Ollama server: The dispatcher/scheduler that listens to the TCP port, parses LLM inference requests, and routes work to the Ollama runner.

• Ollama runner: It is a model-specific execution setup - it contains the model’s architecture metadata, inference backend configuration, pointer to loaded model weights, GPU/CPU memory allocations, KV-cache management, and workspace buffers. Basically, it is a reusable environment for inference using a specific LLM model.

Everything we have seen so far has only involved the Ollama server and its process. Depending on the Ollama version, backend and platform, a runner may be represented by a separate process/threads.

If anyone reading this knows more about how threads are scheduled between the server/runner, I’d love to hear more in the comments.

Leave a comment

In our shipping example, you can think of Ollama server as the entity that manages road operations, while Ollama runner is the entity that manages operations within the port. The former is usually more flexible and can handle different tasks. However, the latter could depend on the specific ship type, loading requirements, etc. In this phase, both entities work together to start loading the ship efficiently.

Step 1: The Ollama server matches the request to a runner

The Ollama server parses the HTTP request, reads the JSON bytes, and recreates the model name, prompts, tool schemas and other settings needed for LLM inference. Then, the Ollama server looks at the model name and checks whether it already has a compatible runner.

• If a compatible runner is available, Ollama reuses it.

• If no runner exists, Ollama must create one.

Since this is the first (and only) LLM inference request, let’s assume the runner does not exist and walk through all the steps.

Step 2: Ollama selects the hardware for LLM inference

The Ollama server examines the hardware and model requirements and decides where the inference can run. If there is no compatible GPU, the LLM inference runs directly on the CPU. While this is not impossible for some small models, this won’t be the focus of this post.

If the system has a compatible GPU, the Ollama server compares the available GPU VRAM space with the memory requirement for the LLM model that’s requested. Ollama estimates the memory required for inference based on the model, context/KV-cache requirements, GPU configuration, and other runtime allocations.

If Ollama determines that the GPU VRAM does not have enough space, then a hybrid CPU/GPU inference path is taken:

• GPU VRAM accommodates data for some of the LLM layers

• Data for the other layers is kept in the System DRAM

• Some model layers execute on the GPU while others execute on the CPU, with intermediate data transferred between the CPU and GPU

Hybrid inference is a very interesting problem, but the CPU-GPU transfers can add significant latency and energy overhead. Ideally, we want to run models that fit completely in GPU VRAM - which is what we will assume for the rest of this post.

Say our shipping destination is accessible by road, train, flight, or ships, then the shipping company can find the best fit depending on the load to be shipped.

Step 3: The Ollama runner initializes a GPU execution context

To use the GPU for inference, the Ollama runner asks the GPU runtime (CUDA, ROCm, Metal, etc.) to make a driver call to create the GPU context for the upcoming inference. This means the GPU driver:

• Allows Ollama to use one of the available GPUs

• Allocates GPU virtual address space to Ollama and maps it to physical VRAM locations

• Creates a command stream/queue between the CPU (Ollama runner process) and the GPU

Essentially, Ollama can now access the GPU via its driver.

Step 4: The Ollama runner allocates GPU VRAM buffers for model parameters

Once the CPU-GPU communication channel is set up, the Ollama runner informs the GPU about the VRAM allocation for the inference. (This, like any CPU-GPU request, happens via the GPU runtime and driver - I won’t explicitly mention this going forward for brevity.) This request looks like this:

• Allocate W bytes for GPU-resident weights

• Allocate K bytes for KV cache

• Allocate A bytes for activations

• …

The driver looks at this request, checks available VRAM, allocates space, maps it to virtual addresses, and returns the information to the Ollama runner.

The GPU Driver and runtime represent the container ship crew - the shipping company needs to check with them before loading anything on the ship.

Step 5: The Ollama runner reads the local model files

Before this step, it is assumed that the required LLM model is already downloaded onto your SSD using the ollama pull command. This model package includes the model configuration (the number of layers, how they are connected, quantization, etc.), model weights, and also the data that will be used by the tokenizer.

Before Ollama can start inference:

• It fetches the model package data from the SSD into the System DRAM.

• Once the data is in the RAM, the Ollama runner submits a data transfer request to the GPU.

• The GPU Direct Memory Access (DMA) engine moves the data into the VRAM spaces allocated earlier, typically through a PCIe interface.

For the conventional discrete-GPU host-to-device path, even when inference is fully done on the GPU, any data stored in the SSD needs to pass through the System DRAM. This becomes a major bottleneck if the model weights, KV cache, and other inference states cannot be fully stored in GPU VRAM. Datacenter architectures increasingly use specialized data-movement and infrastructure processors to reduce CPU overhead and improve data movement, such as Nvidia’s Bluefield DPU.

Moving goods from one mode of transport to another is also a time-consuming task in ports. There are some very creative solutions deployed to speed this up. For example, a log barge (ship that carries wooden logs) simply dumps its entire load onto the water, which allows it to quickly continue to its next assignment. Ore ships also have special docks which are directly connected to train tracks to speed up the unloading process.

Step 6: The Ollama runner prepares the prompt

So far, Ollama was focused on preparing the model for inference. Now it’s time to shift focus to the prompt. Before the GPU can run inference, Ollama must turn the structured chat request into a model-specific prompt representation. This means two things:

• Use the model’s chat template to define how roles (system/user) and tools are represented for the model being used.

• Convert the prompt string into tokens. (This step is called tokenization.)

• After tokenization, the prompt is stored in the System DRAM, waiting to be made visible to the GPU.

I’m skipping the specifics of this step as it varies based on the models. But it’s noteworthy that these computations are typically handled on the CPU rather than the GPU in systems like this. Although it is small relative to the actual inference, it becomes non-trivial for very large prompts, or in a system handling multiple parallel agentic requests.

Step 7: The Ollama runner allocates GPU VRAM buffers for prompt inputs and makes the transfer

Much like what was done with the model package data, the Ollama runner also requests VRAM space for the input tokens and receives pointers to the virtual address space.

Once the required model state and input data are available to the GPU, the GPU has the data needed to start inference.

Typically, the input size is much smaller than the model size, and is also short-lived. (Although a large context/tool result can make prompt input and KV cache significant.)

At the end of Phase 3:

Ollama has set up a communication channel with the GPU and has placed the model weights and prompt inputs in GPU VRAM.

Phase 4: GPU executes the LLM inference

TL;DR: Ollama builds a list of GPU commands for prefill and decode, sends them to the GPU, the GPU runs inference, and computes the final output.

Hardware Involved: CPU, System DRAM, (SSD if needed), and GPU (and it’s VRAM)

We’re finally at the long-awaited inference stage. (Longer than I expected too, to be honest.)

Since this post is focused more on the interaction between the CPU and GPU, I won’t be going into all the details of LLMs. There is a lot of content that already talks about this, and I might write a dedicated post myself if I feel I have something unique to add.

For this post, we will be looking more generally at how (and how many times) the CPU sends work to the GPU. At a high level, (autoregressive) LLM inference is broken down into two stages:

1. Prefill stage: The input prompt is processed to generate the first output token

2. Decode stage: Additional tokens are generated one at a time. Each selected token becomes input to the next decode pass.

Another simplifying assumption in this post is that the LLM output is “non-streaming” - that means Ollama only sends the output back to the Python process at the end of the decode stage for all the tokens. With streaming outputs, there would be a lot more communication between Ollama and the Python process.

If the ship’s journey includes multiple stops, then the prefill stage, and each pass of the decode stage represent the ship’s journey from one port to the next. A streaming LLM inference call would be like having to unload a few containers at every stop.

With that said, here’s how the GPU actually runs inference.

Step 1: The Ollama runner builds a model-specific list of GPU commands

In the last phase, although Ollama was able to move the weights and input prompt into the GPU VRAM, the GPU still does not know what to do with them. The Ollama runner takes the configuration information in the model package read in the last phase to an inference engine (like llama.cpp) and comes up with an ordered list of commands. The command sequence would roughly look like this:

Prefill stage:

• Run input kernel (embedding/position preparation)

• Evaluate transformer layer 0

• Evaluate transformer layer 1

• …

• Evaluate last transformer layer

• Run normalization kernel

• Write prompt K/V values to KV cache

• Generate the first output token

• Record a completion event for the prefill stage

Decode stage:

• For each new token

  • Run input kernel (embedding/position preparation)

  • Evaluate transformer layer 0

  • Evaluate transformer layer 1

  • …

  • Evaluate last transformer layer

  • Run normalization kernel

  • Append new K/V values to KV cache

  • Generate the next token

• At the last token, record a completion event for the decode stage

What each of these commands means does not matter for this post. I just added them to show that an inference call is broken down into multiple GPU commands, and each of them is composed of one or more kernels, each with a large number of arithmetic operations. For example, a single attention layer might include separate or fused kernels for Q/K/V handling, score computation, causal masking, softmax/reduction, and value accumulation.

Loading a ship is not always as simple as “move containers onto the ship as they arrive on the trucks.” When a large ship is being loaded, there is a specific order of operations to ensure the ship is balanced, deliveries are coordinated, and there is no chemical/combustion risk. The GPU commands for each pass are similar to the ship crew’s plan to start loading containers to the ship.

Step 2: Ollama submits Prefill commands to the GPU command queue

Ollama (through the GPU runtime and driver) submits the ordered list of commands to the GPU command queue.

The command queue is not a physical hardware entity - it is a driver abstraction that separates the command generation (by the CPU) from the command consumption (by the GPU).

Step 3: The GPU command processor dispatches work for Prefill

Once the command queue starts to get filled, the GPU command processor and scheduling hardware consume the submitted work and dispatch kernels for execution. This involves:

• Dividing the work into multiple GPU thread blocks

• Assigning each thread block to an available execution unit

In other words, each Ollama command (more precisely, each kernel) specifies what operations to run - but the GPU runtime and hardware execute the kernel across many parallel threads, grouped into thread blocks or equivalent execution units.

Think of the command queue like the loading area where the trucks drop off the container boxes. And the command processor is the loading arm that moves containers to different parts of the ship based on the predefined plan.

Step 4: GPU computes the Prefill kernels and produces the first token

This is the real “GPU execution”.

The massively parallel GPU threads execute the arithmetic, logic, and memory operations in the kernels and update the KV cache in the VRAM based on the input prompt. The kernel also generates the first output token.

I know that dumping this entire GPU computation in one sentence does great injustice to the advanced microarchitecture of modern GPUs - but those details are vendor specific, and beyond the scope of this post.

Step 5: Ollama reads the token and submits the decode pass

The output token is made available to the CPU/Ollama runner, which may involve transferring a small amount of data from GPU memory to host memory. The Ollama runner then checks the stop condition. (LLMs can generate a special end-of-sequence (EOS) token to indicate that generation should stop.)

If the generated token is not the last token, Ollama submits commands for the decode pass to the command queue, which is then picked up by the GPU command processor.

Step 6: GPU computes one pass of decode kernels and produces another token

The execution is very similar to steps 3 and 4 - the GPU command processor dispatches work for the decode pass, the GPU threads execute the kernels, and a new token is generated. The KV cache is updated with this new token. The token is sent back to Ollama to trigger the next decode pass. (or end the inference if the stop condition is met.)

Step 7: Ollama hits the stop condition, and the LLM inference completes

Steps 5 and 6 are repeated until the last token is generated. Once the Ollama runner sees the last token, it stops sending decode commands to the GPU command queue, and the LLM inference is officially complete. All the generated tokens are already present in the System DRAM, ready to be used in the next phase. The updated KV cache, along with the model weights can continue to remain in the GPU VRAM in case there are future inference requests. (More on that in future posts.)

Our shoe manufacturer’s destination is the stop condition for our shipping request. After reaching there, the ship journey for the shoes would have come to an end.

At the end of Phase 4:

Ollama sent commands to the GPU to complete the LLM inference. The output tokens are available in the System DRAM.

Phase 5: Ollama returns the inference response and the Python script finishes execution

TL;DR: Ollama converts the tokens into text, and sends the response to the Python process. The Python executable completes executing the rest of the script.

Hardware Involved: CPU, System DRAM

The steps in this phase are very similar to what we saw in phases 1 and 2 - except this time, Ollama needs to send data to the Python process.

We are back on land now. The ship is unloaded, containers are moved into trucks and are driven to their final destinations.

Step 1: The Ollama runner converts output tokens into text

This is the opposite of the tokenization process that was done in Phase 4 to the input prompt. Ollama uses the tokenizer to decode the output tokens into human-readable text.

Step 2: The Ollama server process sends an HTTP response with JSON data

• Ollama server converts the output text into UTF-8 bytes using the JSON serializer.

• Then, Ollama creates an HTTP response with the output bytes as the response body.

• Finally, Ollama writes the HTTP response to the TCP socket associated with the waiting Python process. (The IP address is still 127.0.0.1 as it is a local HTTP response. TCP port is the one allotted to the Python process by the OS.)

Step 3: The Python thread wakes up

After the Python thread sent the HTTP response to Ollama in Phase 2, it was placed in a waiting state - waiting for this very moment.

When Ollama writes the response:

• The OS places the response bytes in Python’s socket receive buffer,

• The OS marks the Python thread as runnable.

• The OS assigns the thread to an available CPU core.

This resumes the Python process to execute the rest of the script.

The destination side entity of the shoe manufacturer is notified of the upcoming delivery.

Step 4: Python process parses HTTP and JSON

The active Python thread uses the requests bytecode to read the HTTP response, extract the JSON body, and create a response object containing Python strings and metadata.

Once this object is created, it marks the end of the original source line:

r = requests.post(...)

Step 5: The Python executable executes the print command

We are done with the LLM inference.

We are done with the Ollama request.

Now, Python executes the bytecode for the print() function, which writes to standard output. The OS writes to the destination connected to stdout, usually displaying the output text in the shell window.

Step 6: Python process is cleaned up

Once the Python executable reaches the end of the script, the OS reclaims the resources allocated to the Python process - which officially marks the end of the script.

Once the shoes are delivered to the right place, the shipping request can be marked as complete.

At the end of Phase 5:

The output from the LLM inference is printed, and the script completes execution.

Conclusion (Finally!)

Ever since ChatGPT, there has been a lot of chatter about AI hardware that gets oversimplified to sound like this:

“Run LLM inference on the GPU, everything else on the CPU”

My goal was to go one layer below this to understand exactly what the interaction between a CPU and a GPU looks like.

It took a lot of time for me to put this post together, but I needed to do this before I could meaningfully participate in the ongoing conversation about Agentic AI hardware.

I hope reading the post can help you do the same.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

TinyXPU is back!

In our last post, Avik and I explored the tradeoffs between utilization, throughput, and latency. We first looked at these as abstract concepts, and then related them back to the systolic-array based TinyXPU accelerator we are building. This post follows a similar format, but shifts our focus deeper into the hardware side. (Or, as Harry would say, this is where the real push begins.)

So far, we’ve been thinking primarily about architecture. We decided what TinyXPU should do, how data should move through the array, and how different architectural choices affect performance. This time, we’ll look at the three metrics that keep chip designers up at night: Power, Performance (Clock Frequency), and Area, commonly referred to as PPA. Before we dive into the metrics themselves, it’s worth understanding when this analysis is performed during the chip design process - because the stage at which a metric is measured determines what that number actually means.

Different stages of Microarchitectural Analysis

There isn’t just one point where we analyze a design. As a chip moves from RTL to silicon, we repeatedly evaluate its microarchitectural characteristics. Each stage gives us a different level of accuracy, and each comes with its own tradeoff between accuracy and time left before the chip is manufactured.

1. Pre-Synthesis

At the pre-synthesis stage, the design exists only as RTL (Verilog/VHDL code), so PPA estimates are based on analytical models and designer intuition. They are quick to obtain, but fairly inaccurate. Recently, however, this has become a big focus of AI EDA companies, with the promise of fast and accurate results.

2. Post-Synthesis

After synthesis, the RTL has been translated into a gate-level netlist built from standard cells (Logic gates, flip-flops, and other basic digital logic elements). This post-synthesis stage provides much more realistic estimates for area, timing, and power, while still allowing the design to be modified relatively quickly.

3. Post-Implementation

Implementation is the stage where the design has been placed and routed. (All standard cells are positioned exactly where they would be in the final chip.) Since the physical layout is now known, routing delays and parasitic effects are included, producing results that are much closer to what the final chip will achieve.

4. Post Tape-Out

Once the chip has been manufactured, measurements from real silicon provide the post-tape-out results. These are the ground truth, but they also arrive when changes are the most expensive. (Basically, a change = new chip.) Also, since we are no longer in the simulation world, we can only measure PPA at a very high granularity (i.e. One Area, Power, Frequency metric for the whole chip.)

Microarchitecture analysis is most common at the Post-Synthesis and Post-Implementation stage - they are sufficiently accurate, and also provide enough time to make changes to the design before the chip is manufactured. For this post, we’ll focus on post-synthesis results.

Tools and Setup

The focus of this post is to explore microarchitecture concepts through the TinyXPU design - so details about the tools are intentionally brief. All the scripts used can be found in the TinyXPU repo.

Our goal was to use an entirely open-source ASIC flow for this analysis - starting with the PDK, for which we picked the Sky130 standard cell library. The RTL was first synthesized with Yosys, which generated the gate-level netlist and provided the area estimates. Then, OpenSTA was used to analyze the synthesized design, extracting both the critical path delay and vectorless power estimates. (The power analysis was available through the OpenSTA engine that ships as part of the OpenROAD project.)

We ran a simple parameter sweep on the design, and parsed the generated logs to get important PPA metrics. Each TinyXPU configuration is defined by three design parameters.

The first is the PE array size (parameter ROWS/COLS), which determines the number of Processing Elements (PEs) arranged in a square systolic array. (So ROWS == COLS) If you are curious about other shapes of systolic arrays, check out this post from min{power}.

The second parameter is the data width (parameter DATA_WIDTH), which determines the precision of the input activations and weights. A larger data width allows the accelerator to represent larger numbers and improve numerical precision, but it comes at a cost, as we will see in this post.

Finally, we vary the accumulator width (parameter ACC_WIDTH), which determines the precision of the partial sums as they propagate through the array. Since each PE performs a multiply-accumulate (MAC) operation every cycle, the accumulator is typically much wider than the inputs to avoid overflow during long accumulation chains.

The Cooking Analogy (Again!)

We said the format is similar to the last post, so how can we continue without the cooking analogy? For the architecture discussion, the focus was: How do we boil eggs efficiently? We experimented with different strategies: boiling one egg at a time, boiling several together, or using multiple burners. Those were architectural decisions.

But when we are talking about microarchitecture, the focus changes to: Can we build a stove that delivers those results within practical constraints? Say we decided to use eight burners to boil more eggs in parallel - would a stove like that fit on the counter-top? And can the kitchen draw enough power to keep all those burners running?

Such references are used sparingly throughout the post to simplify certain concepts, and are displayed as quotes going forward.

Metric 1: Area

Quick Takes

What is it?

Area measures how much space a design occupies on the chip. Larger designs require more transistors, wires, and space between them, which makes the area larger.

In our cooking analogy, this is simply the size of the stove we need for the chosen “egg-boiling” strategy.

Why does it matter?

Smaller chips are cheaper to manufacture, typically consume less power, and can fit in compact electronic equipment.

A compact stove costs less and fits comfortably into a kitchen, while an oversized one may be impractical regardless of how many eggs it boils.

When does it not matter as much?

Quite often these days.

Area is becoming a secondary concern in high-performance datacenter accelerators, where maximizing throughput is more important than minimizing die size and manufacturing costs. Also, you don’t need to stick your H100 GPU on a smartphone or smartglass, so a large area isn’t existential.

In a commercial restaurant, dedicating more floor space to a larger stove may be worthwhile if it allows the kitchen to serve many more customers.

TinyXPU Area Analysis

Area is arguably the easiest PPA metric to build intuition for. Every additional piece of hardware occupies silicon, whether it is another Processing Element, larger multipliers, or bigger registers.

Experiment 1: Area vs. PE Array Size

Our first experiment varies the size of the systolic array while keeping the accumulator width fixed. Each colored line in the plot represents a different data width.

Takeaways: Two clear trends stand out. First, doubling the dimensions of the systolic array roughly quadruples the chip area because the number of Processing Elements also quadruples. Note that the area axis in this plot is logarithmic - so the actual area scales quadratically with the array size. Second, increasing the data width shifts the entire curve upward. Wider inputs require larger multipliers, adders, registers, and interconnect inside every PE, so even with the same array size, the silicon cost increases significantly.

Returning to our cooking analogy, increasing the array size is like adding more burners to the stove, while increasing the data width is like replacing every burner with a larger, more powerful one. Both make the stove more capable, but also larger.

Experiment 2: Area vs. Data Width

Next, we keep the array size fixed (16x16) and vary the precision of the inputs and weights. The accumulator width is also unchanged. (32 bits)

Takeaway: Increasing the data width increases the complexity of every Processing Element. Wider multipliers, adders, and registers all consume additional silicon, so the effect is amplified across the entire array. Although doubling the data width doubles the number of bits, the area does not increase perfectly linearly. The main reason is that arithmetic units, particularly multipliers, become disproportionately larger as operand widths increase. (Interested readers can see the comparison between different multiplier architectures published elsewhere.)

This is one reason why modern AI accelerators often prefer INT8 or even INT4 arithmetic whenever the algorithm can tolerate lower precision - in addition to reducing memory usage, they also reduce the silicon area for each processing element. (In other words, you can pack more processing elements within your chip.)

Experiment 3: Area vs. Accumulator Width

Finally, we fix the array size (16x16) and data precision (8 bits) while varying the accumulator width.

Takeaway: Unlike increasing the data width, increasing the accumulator width produces an almost linear increase in area. The reason is that the accumulator primarily affects the width of the adders and registers that propagate partial sums through the array, both of which scale approximately linearly with the number of bits. (Adders also don’t always scale linearly, but within the range we considered, it seems to stay linear.) To reduce area, the accumulator size should be limited to the numerical range that the workload requires.

For an optimal stove area, make sure that the burners are as big as the pots - anything bigger results in wasted space.

Metric 2: Clock Frequency

Quick Takes

What is it?

Clock frequency is the maximum rate at which a design can reliably execute one clock cycle. It is determined by the longest delay through the hardware. See this Chip Insights post for details:

In our analogy, it is like how quickly the burners can heat the pot, or the rate of gas flow to the burners.

Why does it matter?

A higher clock frequency allows the same hardware to perform more work per second, improving overall performance.

Faster burners boil the eggs sooner.

When does it not matter as much?

If the workload is limited by memory bandwidth, communication, or algorithmic dependencies, increasing the clock frequency may provide little benefit. Essentially, if utilization is low, then increasing the frequency wouldn’t help much.

If you don’t have sufficient eggs ready to be boiled in the first place, then even a faster burner won’t allow you to boil more eggs.

TinyXPU Frequency Analysis

Signals in a chip do not propagate instantaneously. As they pass through multiple logic gates between two flip-flops, each gate introduces a small delay. The longest of these paths, called the critical path, determines the maximum operating frequency of the design. (In other words, the clock period must be at least as long as the time it takes for the signals to propagate through this path.)

Timing analysis is typically performed by targeting a specific frequency (and clock period) and analysing the “slack”, which is the time available (positive slack), or the additional time (negative slack) needed to meet the timing requirements. If the slack is negative, the designer has two options: modify the design to ensure timing is met, or increase the clock period.

For this post, our goal is to simply analyze the clock frequency trends for different TinyXPU configurations. So we picked a clock period of 20ns for all our runs. This resulted in both positive and negative slacks across the runs, but instead of reporting the slack, we extracted the data arrival time of the critical path (the longest combinational path between two sequential elements) and used its reciprocal to estimate the maximum achievable clock frequency.

Experiment 1: Frequency vs Accumulator Width

Fix the data width to 8 bits, and sweep the accumulator width in the 8x8 PE array configuration.

Takeaways: Increasing the accumulator width has a clear impact on clock frequency. Since every multiply is followed by an accumulation, wider accumulators require wider adders on the critical path, increasing the delay through each Processing Element. Unlike the data-width experiment, this trend is almost perfectly linear. Doubling the accumulator width roughly doubles the adder delay, reducing the maximum operating frequency.

The fastest way to boil your egg is to use a small pot on a small burner.

Experiment 2: Frequency vs PE Array Size

Here, we fix the data width to 8 bits, accumulator width to 32 bits, and vary the array size.

Takeaways: Perhaps the most surprising result is that increasing the array size has almost no impact on the maximum clock frequency. This is expected at the post-synthesis stage, where the critical path is contained almost entirely within a single Processing Element. (The long wires that would connect distant registers in a real chip have not yet been accounted for).

One interesting exception is the 4×4 array, which achieves a noticeably lower maximum frequency than the larger configurations. At first glance, this seems counterintuitive, as the smallest design should be the fastest, right?

We have a theory for why this is happening, but we’ll leave this as a fun exercise. Leave a comment if you can guess what’s causing this behavior.

Experiment 3: Frequency vs Data Width

In this plot, we measure the impact of changing data width by fixing accumulator width to 32 in a 16x16 PE array.

Takeaways: Unlike area, the relationship between data width and maximum operating frequency is not monotonic. Although larger multipliers generally require more hardware, the synthesized critical path depends on the specific gate-level implementation chosen by the synthesis tool. Internally, the tool may restructure arithmetic logic, optimize the multiplier and accumulator differently, select alternate standard cells, or balance the surrounding combinational logic.

In our experiments, changing the operand width altered both the critical path and the logic structure generated by the optimizer. As a result, the 16-bit configuration achieved the highest operating frequency, outperforming both smaller and larger datapaths.

This was the most unexpected result in this study, but we are leaving it here to demonstrate how sensitive microarchitecture analysis could be to the underlying EDA tools and their quirks. We expect that a commercial EDA tool would be better optimized to handle 4 and 8 bit multipliers and result in a more uniform trend.

Metric 3: Power

Quick Takes

What is it?

Power measures the rate at which a chip consumes electrical energy while operating. It includes both the energy spent performing computation and the energy consumed even when parts of the chip are idle. See this Chip Insights post for details about power analysis:

In the kitchen, this is the gas required to keep the stove running.

Why does it matter?

Power determines battery life in mobile devices, electricity costs in datacenters, and how much heat the cooling system must remove.

If you use an extremely power-hungry stove, you will pay more in utility bills.

When does it not matter as much?

It always matters, but may not be existential for new architectures (as they are still experimental and not widely deployed), and performance critical applications - as long as it stays below a certain threshold.

Say you are having people over for the weekend, and you need to boil a lot of eggs, you might not care how much gas the stove consumes.

TinyXPU Power Analysis

To estimate power consumption, each synthesized design was analyzed using OpenROAD’s built-in power analysis. For simplicity, instead of picking a specific workload, the analysis was performed in vectorless mode with a toggle rate of 0.5. This means that, on average, every signal is assumed to have a 50% probability of switching on each clock cycle, providing a rough estimate of dynamic activity across the design. (we explain the limitation of this in an upcoming section.)

Experiment 1: Power vs PE Array Size

With input precision of 8 bits and accumulator width of 32 bits, we look at how power changes for different PE array configurations.

Takeaways: Unlike clock frequency, power increases dramatically as the PE array grows. This is expected because every additional Processing Element introduces another multiplier, accumulator, registers, and routing logic that consume power whenever they switch. The increase is particularly steep because the total number of processing elements grows quadratically with the array dimension. (If you remember, we did exactly this in our architectural analysis post to claim a higher throughput - we are paying the cost now!)

Imagine expanding from a stove with 16 burners to one with 64, then 256, and eventually 1024. Even if each burner consumes the same amount of energy, operating many more of them at once dramatically increases the total power required.

Experiment 2: Power vs Data Width

We vary data width in a 8x8 PE array with 32 bits accumulator width.

Takeaways: Increasing the data width also raises the total power consumption, although for a different reason. Wider operands require larger multipliers, wider registers, and more combinational logic inside every Processing Element. These additional transistors contribute both more capacitance and more switching activity, increasing the energy consumed on every clock cycle.

Unlike the frequency experiment, which showed an irregular trend due to synthesis optimizations, the power results follow a much more intuitive pattern. Regardless of how the synthesis tool implements the arithmetic, wider datapaths require more hardware and therefore consume more power.

Experiment 3: Power vs Accumulator Width

8 bits of data width, 8x8 PE array configuration.

Takeaways: The accumulator width has a similarly predictable effect on power. Wider accumulators require larger adders, wider registers, and additional interconnect, all of which contribute to higher switching capacitance, and hence larger power consumption. As the accumulator width increases, every processing element becomes more expensive to operate, causing the total accelerator power to increase steadily.

This is like replacing every burner with a larger commercial-grade burner. The stove now requires more energy to perform its job, even though the total number of burners remains unchanged.

Analyzing All Metrics Together

Up to this point, we have examined area, frequency, and power independently. In practice, however, hardware architects rarely optimize a single metric in isolation. Instead, they explore a design space, where every point represents a different implementation with its own trade-offs. This plot compares the estimated computational performance of each TinyXPU configuration against its total power consumption. Here, performance is estimated as the product of the number of Processing Elements (PEs) and the maximum operating frequency, and is normalized to the smallest design for easier comparison.

The blue points (A and B) illustrate the concept of a dominated design. Point A corresponds to a 16×16 PE array with an 8-bit data width and a 32-bit accumulator, while point B uses the same array size and accumulator width but increases the data width to 16 bits. Although B consumes significantly more power, it delivers only a modest increase in performance. Since A achieves nearly the same throughput while using less power, B is said to be dominated. More generally, a design is dominated if another configuration provides equal or better performance while consuming equal or less power. Dominated designs are generally discarded because there is always a better alternative.

The red points connected by the red line form the Pareto frontier, representing the set of configurations that are not dominated by any other configuration. Every point on this frontier represents a valid design choice, and moving along the frontier requires trading one metric for another.

The green points (C and D) on the Pareto frontier are interesting. Point C corresponds to a 16×16 PE array with a 4-bit data width and a 16-bit accumulator, while D increases only the data width to 8 bits, keeping the array size and accumulator width unchanged. This change increases the normalized performance by only about 3%, but more than doubles the total power consumption. Although neither design dominates the other, the small performance increase comes at a significant energy cost along this part of the Pareto frontier.

Finally, notice that the Pareto frontier consists of discrete points rather than a smooth curve. Unlike software, hardware parameters such as array dimensions, data width, and accumulator width can only take on a finite number of valid values. We cannot build a 17-bit datapath or a 20×20 systolic array simply because it would lie on a more favorable part of the curve. Instead, hardware architects evaluate a finite set of candidate designs and select the configuration that best balances performance, power, and area for their target application.

Some Practical Considerations

While these results were meant to introduce microarchitectural analysis and highlight some high-level trends, they do not capture many physical effects that appear later in the design flow.

For area, the reported values represent the total area of the synthesized standard cells, not the final chip size after physical implementation. During placement, additional whitespace is introduced to satisfy routing, congestion, clock tree synthesis, and design rule constraints. Consequently, the final die area is typically larger than the synthesized cell area, especially for larger designs where routing becomes more challenging.

For timing, the analysis assumes ideal interconnects. Since the design has not yet been placed and routed, the delays introduced by long wires, clock distribution, and routing parasitics are largely absent. This explains why the maximum operating frequency remains almost independent of the PE array size in our experiments. After place-and-route, these interconnect delays would become a much larger contributor to the critical path, particularly as the array grows.

Finally, the power results should be interpreted as rough estimates rather than accurate measurements. The analysis is vectorless, meaning the EDA tools assume a fixed switching activity instead of using real workload traces. As a result, the reported dynamic power can differ significantly from what would be observed under actual operation. In practice, accurate power analysis requires realistic switching activity (SAIF or VCD files), along with extracted interconnect parasitics after place-and-route. Process-voltage-temperature (PVT) variations and clock tree synthesis also have a significant impact on the final power consumption.

Why does all this matter to an Algorithm Designer?

If you’re an algorithm designer and you’ve made it this far, we hope this final message will make it all worth it.

Architectural ideas are often evaluated using simple models. If doubling the number of processing elements doubles throughput, or increasing the buffer size reduces memory traffic, the idea looks like a clear win. But once someone begins implementing that architecture, physical constraints start to appear.

A larger systolic array requires more wires connecting processing elements, which makes routing more difficult and can reduce the maximum clock frequency. Adding more parallel compute units increases throughput, but it also increases power consumption and can make the chip too large or too expensive to manufacture. Even something as simple as adding another pipeline stage may improve clock frequency while increasing latency. These tradeoffs are invisible at the architectural level, but they become impossible to ignore during microarchitectural design.

This doesn’t mean algorithm designers need to become microarchitects. But keeping one eye on these microarchitectural constraints can lead to better architectural decisions from the very beginning. An algorithm that is designed with area, clock frequency, and power in mind is far more likely to become an efficient hardware implementation.

That is the essence of hardware-software co-design that we aimed to provide through this series of posts. Here are the previous parts:

Subscribe to Chip Insights and min{power} for more and let us know if there is something else you would like us to explore.

But lately, I’ve gotten into this bad habit: I read something, I learn something new, and then I forget about it. I struggle to meaningfully retain information from whatever I consume.

That’s the motivation behind a new Chip Insights series I’m experimenting with: Context Caching.

The name comes from two worlds that I spend a lot of time thinking about. In AI, context determines what information is available to a model at any given moment. In computer architecture, caches keep the most useful data close at hand while less relevant information gets evicted.

Context Caching is my attempt to do the same thing for semiconductors. Every week, I read through semiconductor industry content and ask two questions:

• What idea deserves a place in my mental cache?

• What idea is becoming less relevant and should be evicted?

In this process, I hope to develop a mental model about the semiconductor industry (and hopefully provide some values to my viewers as well.)

I’m also using this as an opportunity to experiment with a new format: video.

Chip Insights started as a writing project, and long-form writing will continue to be the core of what I do. There’s no substitute for a detailed deep dive when you’re trying to understand an architecture, a technology trend, or a historical system.

But long-form writing, by its very nature, takes a lot of time for me to produce, and also has a significant barrier to entry for new readers. Sometimes an interesting observation, connection, or mental model can be communicated in a minute or two. Video feels like a more accessible format for those moments, and frankly, I’m curious to see if I enjoy creating it.

That’s really all I wanted to share in this post. Writing posts like this helps me convince myself about the “why” of doing something new, so thank you for being my audience in that process.

As I always say about my posts, this is very much an experiment, so I’d love your feedback about this new series. Here’s the link to the playlist: https://www.youtube.com/playlist?list=PLXF-OLVfH4Y54Rgi_heB2RQGwxmp1lYdO

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

In a recent post (well not so recent anymore, sorry for the long break), I presented my case for why the story of ENIAC is relevant in the context of today’s AI datacenter buildout. Check it out if you already haven’t.

I left the post with two questions I wanted to explore:

1. Which compute infrastructure can last the longest before we need a clean reset?

2. How effectively can we map future workloads onto the infrastructure we are building today?

At first glance, these questions seem to be about chips, algorithms, and software. But history suggests that the answer often lies elsewhere. Every major shift in computing eventually collides with the physical realities of the world around it. How is it powered? How is it cooled? Where is it housed? What infrastructure must exist before the machine can deliver any useful work?

That is why I want to begin this exploration not with computation itself, but with the infrastructure that enables it.

In the 1940s, the builders of ENIAC were not just inventing a new computer. They were discovering what it meant to power, cool, and operate a machine unlike anything that had existed before. Today, as AI companies race to build ever larger datacenters, similar questions are once again coming into focus. The answers, however, may be very different.

To understand why, I’d like to tell a story.

Setup complete after rewiring. Preparing for Test #23 for the evaluation of errors in axially symmetric supersonic airflow.

As Homer Spence wrote down these words in the ENIAC service log book, he felt a sense of relief, with a slight dose of anxiety. He had just fixed another loose blob of solder on the ENIAC’s Accumulator 14 panel, and was one step closer to running the ENIAC without errors.

The irony of the situation always made Homer chuckle. The fate of the world’s most complex electronic device, what many considered an “artificial brain”, relied on such craft work. But this craft work had been going on for a week now, and Homer really hoped this was it.

He held up the control box and started ENIAC while an operator monitored the panels. The machine slowly came alive. Thousands of vacuum tubes began to glow with a familiar orange hue, and the room grew noticeably warmer. Homer smiled. After weeks of troubleshooting, the glow was reassuring. It meant the machine was alive.

Then something felt wrong.

The brightness kept increasing.

The room became hotter.

Homer was sweating profusely as if he was standing in front of a raging forest fire. As his hand reached for the stop button on the control box, his entire field of view turned white.

The next time Homer opened his eyes, the bright white light was replaced by complete darkness. As he looked around, the familiar contours calmed him down. He was laying on the sofa-bed he had placed in his office for all the late nights he spent at work. Maybe he was completely exhausted and blacked out during the last test, and someone helped him to his office.

Homer checked for burn wounds and was relieved to find none. The concern was not irrational. ENIAC consumed more than 150 kilowatts during normal operation, and everyone on the project knew what that could mean. A fire the previous year had damaged one of the panels after insulation around a power wire ignited. Ever since, discussions about transformers, fuses, and power distribution had become almost as common as discussions about computation itself.

It was late, and Homer decided to head home. But first, he wanted to check that everything was fine in the ENIAC room.

He grabbed his notebook and opened the office door.

Homer expected to see the familiar rows of ENIAC panels stretching across the room.

Instead, he froze.

A modern datacenter is an engineering marvel. The space itself is nothing to write home about. White walls, bright lights, and a constant hum from equipment hidden somewhere beyond sight. Yet Homer immediately sensed that this was not a building designed for people.

The room was filled with tall rectangular cabinets arranged in perfectly ordered rows. (Each of these rectangular containers, not too dissimilar to a refrigerator, is called a server rack.) They stood shoulder to shoulder, forming long corridors that stretched farther than Homer could see. At first glance, they reminded him of ENIAC’s panels. But where ENIAC’s cabinets exposed switches, cables, and glowing vacuum tubes, these appeared sealed and anonymous.

From a distance, the fronts of the cabinets looked alive. Hundreds of tiny lights blinked continuously, each following a rhythm known only to the machine. Homer found himself staring at them. They reminded him of the warm orange glow of vacuum tubes, except these lights were colder, sharper, and far more numerous. A closer look revealed multiple rows, each holding a small rectangular box. (Each of these is called a server.) A thick bunch of cables could also be seen leaving the top of each server rack and disappearing into vents in the ceiling.

Homer Spence had seen nothing like this before. The ENIAC room he expected to find on the other side of the door had vanished, replaced by what appeared to be an endless maze of cabinets, cables, and blinking lights. For a brief moment, he wondered whether he was still dreaming. As he stood frozen in the middle of the aisle, a voice interrupted his thoughts.

“Hi there. Can I help you?”

Homer turned to find a bespectacled man wearing a black leather jacket. Still trying to make sense of his surroundings, Homer skipped the pleasantries.

“Where am I?”

The man, now smiling like a proud father, responded.

“You’re standing in one of the world’s largest AI datacenters.”

Homer looked around once more.

“This entire building?”

The man laughed.

“No. This is just one hall.”

For the first time that night, Homer felt genuinely unsettled.

Back at the Moore School, ENIAC occupied a single room. Every panel, every accumulator, every cable was within walking distance. Here, the machine seemed to extend far beyond the limits of his vision.

The man continued.

“This facility has a power capacity of five gigawatts. When fully populated, it will contain more than five hundred thousand Rubin Ultra processors. Right now, we’re using it to train one of the largest artificial intelligence models ever built.”

Homer barely registered anything after “five gigawatts”. The number echoed in his head.

When ENIAC was being designed, supplying 150 kilowatts of power had proven to be a formidable challenge.

“How is this facility powered?” Homer asked.

“Powering this facility is no mean feat. We started with utility power from the grid - the same thing that’s used in homes around the world. But the local grid simply cannot support our capacity yet. So we had to improvise and build our own power plants right next to this datacenter. They are mostly turbines powered by natural gas, which is often how the central grid is powered anyway. But we plan to move to renewable sources like solar and wind, and are also exploring nuclear power as an option.”

The detailed answer surprised him. When ENIAC was being designed, power infrastructure was largely a means to an end. The objective was to build the computer. Everything else followed from that. Only later did the team discover how difficult it would be to supply 150 kilowatts reliably to a machine filled with nearly eighteen thousand vacuum tubes.

The challenge became apparent when the time came to procure the transformers. Different parts of ENIAC operated at different voltage levels, ultimately requiring twenty-eight separate transformer units. Securing them proved far more difficult than anyone anticipated. General Electric could not deliver one soon due to other military commitments, so they paid three times as much to have Maguire Industries expedite the process. Maguire only shipped 4 out of the 28 units promised, and the rushed timelines also meant that certain specifications about the insulation were ignored. Eventually, the order was fulfilled by J. J. Nothelfer Winding Laboratories and the testing could only begin after a long wait.

Even then, the challenges were not over. The reliability of ENIAC depended heavily on the quality of the incoming power. Voltage fluctuations from the utility grid affected both the accuracy and lifetime of the vacuum tubes, forcing the engineers to develop custom equipment to stabilize the supply. Years later, they would go even further, installing a dynamo coupled to a massive flywheel to smooth out power disturbances. Yet even this system remained connected to the city’s electrical grid. The objective was never to generate power independently. It was simply to make utility power good enough for computation.

Homer realized that the ENIAC team had treated power as a dependency of the computer. The engineers of the future (where he was now convinced he was) appeared to be doing the opposite.

As the conversation continued, Homer became increasingly bothered by something. Standing in the ENIAC room for extended periods was an uncomfortable ordeal. ENIAC’s vacuum tubes, not too dissimilar to incandescent light bulbs, produced enormous amounts of heat and quickly transformed the room into something resembling a tropical summer afternoon. Yet he was standing in a facility that consumed over 30,000 times more power, and the air felt no different than an office building.

Where was all the heat going?

“How do you keep this place cool?” Homer finally asked.

The man smiled and led him down one of the narrow aisles between the server racks. At first glance, the racks appeared quiet and uneventful. But as Homer looked more closely, he noticed thick pipes running overhead. Some disappeared directly into the racks, while others connected to even larger pipes that stretched across the building.

The man pointed toward them.

“Air is simply not efficient enough at these power levels,” he explained. “Traditional datacenters for cloud storage and web hosting typically operated at power densities of five to ten kilowatts per rack, where air cooling was sufficient. These racks contain many high-performance processors operating together and can exceed one hundred kilowatts per rack. At that point, liquid cooling becomes unavoidable.”

He paused before continuing.

“But that’s only part of the story.”

Homer listened carefully.

“Building this cooling system was enormously expensive. The pipes, pumps, heat exchangers, cooling towers, and control systems cost far more than the cooling equipment found in traditional datacenters. But once the infrastructure is in place, liquid cooling removes heat much more efficiently. Over the lifetime of the facility, the operating costs can be significantly lower.”

Homer was unfamiliar with this way of thinking. During the development of ENIAC, cooling was never approached this way. The machine was cooled by moving air through the room because it worked and because it was inexpensive. If the room became too warm, you opened a vent, added another fan, or simply endured the discomfort. Most of the cost was incurred only when the machine was operating - which was not very common due to various other limitations.

Here, the philosophy seemed reversed. The engineers had spent enormous sums of money even before a single calculation was performed. This facility had been designed around an expectation that workloads would run continuously. This would justify the initial costs.

For several minutes, neither man spoke. Homer couldn’t help but be impressed by everything he heard. He had spent many nights with the ENIAC team worrying about the challenges of power delivery, cooling, and maintenance. This often slowed the progress of making ENIAC better at the computations it was built for.

By planning for power and cooling years in advance, these engineers were building computers that could scale far beyond anything imaginable in ENIAC’s era. But Homer couldn’t shake a lingering thought. Every transformer ordered for ENIAC had a purpose. Every vacuum tube installed would eventually perform a calculation. The luxury of building infrastructure years ahead of demand did not exist.

He turned to the man with the question that started everything.

“Where am I?”

A few concrete takeaways

I have written this post in an experimental format - some of you may like it, others won’t. (I’m happy to hear from you in the comments either way.)

But I want to explicitly call out a few lessons that stood out for me:

1. Compute does not scale alone

Power and cooling are first-order design variables. It is becoming increasingly difficult to separate the computer architecture from the infrastructure. A specific architecture in the datacenter exists due to its supporting infrastructure, and vice-versa.

2. Infrastructure choices are really workload assumptions in disguise.

ENIAC’s infrastructure was built around a machine that spent much of its life idle, being debugged, rewired, or waiting for the next problem. Most of the cost was incurred when the machine was actually running.

Modern AI datacenters are built around the opposite assumption. Training workloads can keep hundreds of thousands of processors busy at near 100% utilization for weeks or months at a time - this has made it seem more rational to spend enormous amounts on power delivery and liquid cooling infrastructure upfront.

3. The bottleneck is shifting from execution to foresight.

ENIAC engineers had to solve engineering problems in real time. Modern datacenter builders increasingly have to solve prediction problems: how much power, cooling, and demand will exist years before workloads arrive.

That concludes Part 2 of this series. If you like what I just said, follow along for upcoming posts in this series by subscribing below:

References

My primary reference for this study is the book “ENIAC in Action” which goes into an impressive level of detail about many aspects of this computer.

Here are some other references I used for this post:

• Inside the world’s most powerful AI datacenter - The Official Microsoft Blog

• Mark Zuckerberg says Meta is building a 5GW AI data center | TechCrunch

• (2) How AI Labs Are Solving the Power Crisis: The Onsite Gas Deep Dive

• Power for AI Data Centers: Energy Demand, Grid Impacts, Challenges and Perspectives

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

In this post, we introduce the art of architectural analysis through three key concepts and use them to evaluate the TinyXPU architecture - a systolic array based 2D matrix multiplication accelerator we designed to study the evolving trends in custom silicon. The ability to analyze architectures is becoming an increasingly valuable skill - especially in a world of rapidly growing custom architectures. So, whether you’re a software or hardware engineer, you will benefit from reading about these concepts.

If you are new here, this post is part of our ongoing series on custom accelerators. In the first part of this series, we motivated the need for a bridge between a software engineer (Alice) to a new architecture developed by a hardware engineer (Harry).

In the second part, we introduced our TinyXPU project - a practical demonstration of the hardware-software bridge.

While we recommend reading the first two parts, it’s not a prerequisite as this post takes a slightly different direction.

Why does Architectural Analysis matter?

So far, we’ve focused on how Alice and Harry can work together more effectively. Using a framework like ONNX, we showed that Alice can now target many different architectures (and many different Harrys) with relative ease.

This naturally leads to the next question: How does Alice decide which Harry to choose?

At first glance, the answer seems simple: Alice would pick the “best architecture” for her algorithm. But what does “best” actually mean?

From Alice’s perspective, it could mean:

• I want my algorithm to finish faster

• I want to start seeing outputs as early as possible

• I want it to fit within a smartwatch, a robot, or a datacenter

From Harry’s perspective, the architecture might be “best” because:

• It reduces the number of arithmetic operations or data movement

• It runs at a higher clock frequency than competitors

• It occupies less chip area than other implementations

Clearly, Alice and Harry are both trying to communicate. But they are speaking completely different languages. If Harry wants to convince Alice to use his architecture, and Alice wants to make the right choice, they need a shared vocabulary for analyzing architectures. This is where understanding concepts in architectural analysis becomes important.

The Cooking Analogy

To build intuition for the ideas in this post, we’ll use a simple analogy: cooking.

More specifically:

• The Algorithm: Boiling eggs

• The Hardware: A kitchen stove

Boiling eggs is simple, but it can be done in many different ways.

You might cook eggs one at a time or in batches. You might prioritize getting the first egg ready as quickly as possible or finishing all of them as fast as you can.

Similarly, each stove is different. One might have more burners. Another might heat up faster. A third might support a larger pot but heats up slower.

This is not too different from the problem Alice and Harry are trying to solve.

For clarity, all future references to this analogy are displayed as quotes.

Recap: The TinyXPU Terminology

We briefly introduced the parameters used in the TinyXPU project and the matrix sizes in part 2 of our series of posts. Here’s a quick recap of variables that will be used in the rest of the post.

• The Input Matrix X has M rows and K columns.

• The Weight Matrix W has K rows and N columns.

• The PEs are arranged as a 2D array with HW_ROWS rows and HW_COLS columns. In this post, we will only explore the 16*16 configuration.

Concept 1: Hardware Utilization

Quick Takes

What is it?

How much of the hardware is being used to produce useful output.

In our cooking analogy, if your kitchen stove has 4 burners but you only use 3 simultaneously, the utilization is 75%.

Why does it matter?

High utilization means more of the hardware is active at the same time. In general, this implies that more sub-operations (additions, multiplications, data movement) are happening in parallel.

If more burners on your stove are on, it means more eggs are being boiled.

When does it not matter as much?

In general-purpose architectures like CPUs and GPUs, the hardware is almost never fully utilized. In these systems, overall utilization matters less than the utilization of specific components (such as ALUs).

However, an accelerator is designed for a very specific purpose. It is therefore wasteful to build hardware that cannot be effectively utilized. In most accelerators, high utilization is close to a non-negotiable.

Having low utilization is like using a 4-burner stove when you only need to boil one egg.

PE Utilization in TinyXPU

In TinyXPU, the size of the systolic array is fixed in hardware (16×16 in our case), but the matrices we run on it are not. This mismatch is the primary reason utilization becomes an important metric.

The goal is simple: keep as many PEs busy as possible for as long as possible.

In our cooking analogy, this is equivalent to keeping all burners active. If you only have enough eggs for two burners, the remaining burners sit idle, even though the stove could do more work.

Where does underutilization come from?

1. When the workload is smaller than the array

If the weight matrix is smaller than the systolic array, for example fewer than 16 rows or columns, some PEs will not map to useful computation. To maintain correctness, we typically pad the matrix with zeros. These PEs still perform MAC operations, but their outputs are discarded.

This is like turning on burners with empty pots. Heat is being generated, but no eggs are being cooked.

2. When the workload is larger than the array

If the matrix is larger than the array, we process it in tiles. While tiling allows us to handle arbitrarily large matrices, the last tile is often smaller than the array, leading again to underutilization. We will skip the details of tiling in this post, but this edge effect is quite common in practice.

Startup overhead: why utilization is not 100%

Even when the workload maps perfectly to the array, utilization is not immediately 100%. This is because not all cycles contribute to useful work. At the start of execution, the systolic array needs a few cycles before all PEs begin performing meaningful computations. Similarly, toward the end, some PEs become idle as the computation completes.

From a utilization perspective, these cycles are overhead. The hardware is active, but not all of it is doing useful work.

Before you can boil eggs, you need to fill the pot and wait for the water to heat up. During this time, the stove is on, but no eggs are being cooked.

Considering this overhead, we define utilization as:

This shows that utilization improves as the workload grows, because the fixed overhead is amortized over more useful work.

PE Utilization vs Batch Size

The impact of matrix sizes on PE utilization is shown in this plot:

Key takeaways:

• The 16×16 case asymptotically approaches 100% utilization, as the pipeline overhead becomes negligible compared to useful work.

• The other shapes (4×16, 8×8, 16×4) all have 64 total weights, and therefore can only utilize 64 out of 256 PEs, which corresponds to a maximum of 25% utilization.

• For small values of M, utilization is low across all configurations due to pipeline overhead.

• As M increases, all curves improve, but they plateau at different levels depending on how well the workload matches the hardware.

Utilization is a useful analytical metric. It tells us how much of the hardware is actively doing useful work and highlights inefficiencies due to pipeline overhead.

How architects use this metric in real chips

In real chips, utilization is rarely exposed as a single number. Instead, architects infer it using performance counters, stall analysis, and activity factors across compute units. A classic example is the TPU v1 paper, where the authors show that achieving high utilization of the 256×256 systolic array was critical to performance. They analyze how different workloads map to the array and highlight cases where underutilization leads to significant efficiency loss.

Concept 2: Throughput

Quick Takes

What is it?

The number of useful operations completed per unit time.

In our cooking analogy, this is the number of eggs you can boil per hour.

Why does it matter?

Throughput directly determines how fast an algorithm completes. If more operations are finished per unit time, the total execution time is lower.

When does it not matter as much?

In many systems, overall throughput is limited by the slowest component. Even if one part of the system achieves very high throughput, it may not translate into end-to-end performance improvements.

Custom accelerators are typically designed to maximize throughput. However, understanding what limits that throughput is just as important.

From Throughput to the Roofline Model

So far, we have treated throughput as a single number. In reality, it is constrained by two fundamental limits:

• How fast the hardware can compute

• How fast data can be moved to and from the hardware

The roofline model captures both of these limits in a single diagram, which is why it is a better representation of our throughput analysis.

In our cooking analogy, the number of eggs you can boil per hour is not just determined by how many burners your stove has. It is also limited by how quickly you can bring water, eggs, and utensils to the stove. If you have many burners but can only carry a few eggs at a time, most burners will sit idle. On the other hand, if you can supply eggs very quickly but only have a small stove, the burners themselves become the bottleneck.

TinyXPU Roofline Model

Vertical Axis: Peak Throughput

For a systolic array, if the workload fully utilizes the array, the peak compute throughput is:

For our 16×16 array, this gives a peak of 256 MACs per cycle. However, if the weight matrix is smaller than the array, only K×N PEs perform useful work. This represents the vertical axis of our roofline model.

Horizontal Axis: Arithmetic intensity

Arithmetic intensity is the number of MACs executed for each byte of data read from the memory. For our matrix multiplication example:

• Weights are loaded once: K×N bytes (we ignore this as batch size is usually large)

• Inputs contribute: M×K bytes

• Outputs contribute M×N values, each 4 bytes (we assume output values are 32-bit integers)

This gives:

So, the arithmetic intensity (AI) becomes:

Roofline Plot

Based on the definitions above, the roofline plot for TinyXPU is shown here:

Each point on the plot represents a specific workload running on the hardware. The horizontal axis is arithmetic intensity and the peak throughput. The roofline plot also includes two lines which are important:

• The horizontal line represents the peak compute capability of the hardware

• The sloped lines represent memory bandwidth limits

Several insights emerge from this single diagram:

• If the weight matrix is smaller than the array, peak compute throughput cannot be reached due to underutilization (this aligns with our earlier analysis on PE utilization.)

• Among shapes with the same number of weights, taller matrices perform better than wider ones. This happens for two reasons:

  • Idle rows waste more hardware than idle columns

  • Output traffic scales with N, and outputs are larger in size

• The 4×16 shape is more bandwidth-limited than 16×4, since it produces more output data

• Most configurations in this example are memory-bound unless bandwidth exceeds 64 bytes per cycle (which is rare in CPU L1 caches as seen from this post on min{power})

Using a roofline plot, it becomes clear how throughput can be maximized:

• Move upward by increasing compute efficiency (Larger batch sizes)

• Move right by increasing arithmetic intensity (Taller matrices)

By being in the top right, we get the best throughput.

How architects use this metric in real chips

The roofline model is one of the most widely used tools for reasoning about throughput limits. Modern accelerators frequently use roofline-style analysis. For example, NVIDIA presents roofline-inspired performance characterizations in its architecture whitepapers, (For example, NVIDIA A100 Architecture) where compute throughput and memory bandwidth limits are analyzed together.

In fact, the roofline plot we obtained for TinyXPU is not too different from the plot shared in the TPU v1 paper:

Source: https://arxiv.org/pdf/1704.04760

Concept 3: Latency

Quick Takes

What is it?

Latency is the time it takes to produce the first useful output after the first input is provided.

In our cooking analogy, this is the time between turning on the stove and having the first boiled egg ready.

Why does it matter?

Latency matters when partial results are useful before the entire computation is complete. For example, modern LLM-based systems stream outputs token by token. As soon as the first token is ready, it is displayed to the user. In such systems, latency directly impacts user experience.

When does it not matter?

Latency matters less when the full output is required before any further computation can proceed. For example, in a classification task such as predicting whether an image contains a cat or a dog, the final decision can only be made after all computations are complete. In such cases, throughput is often the more relevant metric.

Latency in TinyXPU

To analyze latency, we first need to define what we mean by the “first output.” In matrix multiplication, we define the first output as the result corresponding to the first row of the input matrix X.

For readers familiar with LLMs, consider the computation of the query matrix:

Each row of X corresponds to the embedding of an input token. The first row of Q therefore corresponds to the first token. The time taken to produce this row is a key component of time to first token (TTFT).

Where does latency come from?

Even if the input matrix X has many rows, the first result that emerges from the systolic array corresponds to the first row. This delay is caused by the time it takes for data to propagate through the array.

In the cooking analogy, this is like the time it takes to heat the water and cook the first egg. Even if you plan to cook many eggs, the first one still takes the same amount of time.

For a weight-stationary systolic array, the latency to first output is:

Essentially, the latency only depends on the number of rows in the hardware array and the number of output columns. It is independent of the batch size M.

Throughput vs Latency Tradeoff

The plot below shows throughput on the vertical axis and latency on the horizontal axis for different matrix shapes. Each curve corresponds to a fixed number of rows K, while varying the number of columns N. The batch size (M) is fixed to 256.

This plot clearly shows the inherent tradeoff between throughput and latency.

• Increasing N increases throughput, since more PEs are active

• However, increasing N also increases latency

This means designs with the same total compute capacity (total number of PEs) can have very different latency and throughput characteristics. If you want fast response to the first output, you prefer smaller matrices with lower latency. If you want higher overall throughput, you prefer larger matrices that utilize more of the hardware.

In the cooking analogy, this is like choosing between boiling one egg quickly or boiling many eggs at once. Using a larger pot allows you to cook more eggs simultaneously, but it may take longer before the first one is ready.

This throughput-latency tradeoffs starts to become even more interesting in accelerator designs with pipelined PEs to achieve higher clock frequencies, and full-system implementations with non-zero instruction and memory latencies - topics we will explore in future posts.

How architects use this metric in real chips

Latency is typically modeled using a combination of analytical models and cycle-accurate simulations that capture pipeline depth and data movement delays. For example, Google discusses latency-sensitive inference in the MLPerf Inference Benchmark, which includes metrics like time-to-first-token and tail latency.

As you can see from the discussion, architectural analysis is as much art as it is science. It’s more like appreciating a painting than reading a book. An experienced art connoisseur can analyze a painting at a glance. But that ability comes from a shared understanding of color, brushwork, and design principles between the artist and the observer.

This post aims to build a mental model for architectural analysis, using our basic TinyXPU implementation as a concrete example. As we continue to expand TinyXPU by adding activation functions, mapping it to an FPGA, and exploring unorthodox systolic networks, we’ll build on these ideas and introduce new ones along the way.

Subscribe to Chip Insights and min{power} to follow along as we do that.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

This post is the second in our series on custom hardware accelerators. In part 1, we introduced the hardware-software interface problem that has kept the two domains as separate islands and limited co-innovation. We also introduced two approaches to overcome this gap. If you haven’t done so already, check out that post here:

In this part, we present TinyXPU, an open-source, extensible project we are building in conjunction with this series of posts. In this post, we focus on two pieces:

• The hardware architecture of TinyXPU

• The software interface that allows it to run ONNX models

The goal of this project is to show what the bridge between software and hardware looks like in practice, and is a stepping stone to the architectural analysis we have planned in upcoming posts.

Why Matrix Multiplication

The first important decision is to pick the right algorithm to study. We picked Matrix Multiplication, primarily because of its widespread use in many of today’s most important software domains. Here are a few representative examples:

• Deep Learning: Neural network layers compute weighted sums using matrix multiplications.

• Robotics: BLAS, in particular the Level 3 GEMM routine, is used in many robotics libraries

• Computer Graphics: Shaders use matrix math for transformations; modern APIs like Vulkan have started exposing this directly.

Matrix multiplication also stresses both compute throughput (number of arithmetic operations completed per second) and memory bandwidth (the number of bits read from memory per second). This helps us identify both compute and memory bottlenecks - a topic we will explore in future posts.

Why build an accelerator from scratch

We are not the first to focus on matrix multiplication, and we certainly won’t be the last. As mentioned in an earlier post, there has been an explosion of “processing units” like TPUs and NPUs that, at their core, accelerate matrix multiplication. In fact, modern GPUs and even CPUs now support matrix multiplication extensions and dedicated units to handle the operation efficiently. There are also a number of “tinyTPU” projects out there with an architecture very similar to the one we will be demonstrating. (Some examples are tiny-tpu-v2/tiny-tpu and Alanma23/tinytinyTPU.) So before we go any further, we wanted to explain why we are doing this from scratch.

Our main motivation is that building from scratch allows us to understand (and hopefully explain) everything we are doing from first principles. As you will see, we are only starting with two simple SystemVerilog files. This way, we do not have to make any assumptions or shoot in the dark to understand architectural decisions. This also allows us the flexibility to scale or modify the architecture for future posts in this series.

It’s not just the hardware architecture that could be confusing. As we mentioned in Part 1, the software interface is messy and still evolving. As this is the first implementation in our series, we wanted to keep a simple but still operational software interface. Setting up this interface is important to obtain performance metrics that will be used for architectural analysis in future posts in this series.

With that motivation, we can now look at the two parts of the TinyXPU project: the hardware architecture and the software interface.

The Hardware Architecture

We built the first version of our matrix multiplication accelerator based on the weight stationary systolic array architecture: the same one at the heart of Google’s TPUs. While this post will not go into the details of this architecture, here’s a dedicated post that’s got you covered:

min{power}

Systolic arrays for general robotics, AI, and scientific computing

Read more

7 months ago · 8 likes · Avik De

The Processing Element (PE)

The core block in TinyXPU is called a Processing Element (PE), a term that originated in H. T. Kung’s seminal work in 1982. Here’s the block diagram of our PE (the SystemVerilog implementation can be found in pe.sv):

This PE block executes the following operation:

acc_out = data_in * weight_in + acc_in

However, this operation is executed in two phases:

1. Weight Loading Phase: In this phase, the value of weight_in is stored in the register weight_r when weight_ld is set to 1.

2. Multiply Accumulate Phase: Here, the stored value weight_r is multiplied with data_in, and acc_in is added to get the final result. When en is set to 1, this result is sent as the output.

The reason for having these two phases comes from its primary use case in neural networks, where we typically process multiple inputs (called “batches”) using the same weights - so loading the weights once prevents the need to continuously broadcast its value during the computation. This is called a "weight-stationary" architecture, indicating that the weights don't move through the array in the Multiply Accumulate Phase.

It’s also important to highlight that the Multiply Accumulate Phase has a latency of 1 cycle - which means that the output of the PE is ready one cycle after the inputs. This pipeline stage is needed to break the combinational logic path when two PEs are connected together, in order to achieve a high clock frequency. (If you are new to digital design, this post on Static Timing Analysis would help you understand why this matters.)

Matrix Multiplication on a PE array

The functionality of a single PE can be achieved using the Arithmetic and Logic Unit (ALU) in most CPUs. So, to really understand the benefit of a PE array, let’s consider a PE array with 2 rows and 2 columns, and map a matrix operation on it. Specifically, we will be implementing this operation:

Our PE array performs matrix multiplication by arranging PEs in a grid where data flows between neighboring elements. Each PE stores one weight value and performs a multiply-accumulate operation every cycle. As input values move across the array, partial sums are passed between PEs until the final output values emerge from the last column. We depict the cycle-by-cycle dataflow in this diagram:

• In the first cycle, X11 is passed as input to PE with weight W11

• In the second cycle, X11 moves on to the next PE. (with weight W12.) Simultaneously, X21 and X12 are passed to the first PE column. (with weights W11 and W21.)

• In the third cycle, X21 and X12 move to the second PE column.( with weights W12 and W22.) Also, X22 is passed to PE with weight W21. In this cycle, we also have the first output value ready:

  Y11 = X11 × W11 + X12 × W21

  Note that the latency (time from first input to first output) is 2 cycles in this example. As the number of rows of the PE array increases, this latency also increases. After the first output, new outputs are ready in every adjacent cycle, as we will see.

• In cycle 4, X22 moves to the second PE column. We also get output Y21 and Y12 from both the PE columns.

• Finally, in cycle 5, the last output Y22 is ready in the second PE column.

What makes this an accelerator?

The biggest advantage of this PE architecture is that we read the operands only once from memory. In the above example:

• Weights W11, W12, W21, W22 are only read in the Weight Loading Phase and are then stored in the PEs

• Although Inputs X11, X12, X21, X22 are used twice during the matrix multiplication, they are only read once (For example, after X11 is read in Cycle 1, it is just passed to the next PE in Cycle 2 - it does not have to be read again.)

In a standard CPU, the intermediate values would need to be stored somewhere. It could be stored in a fast register for small matrices, but might need to spill to slower caches and main memory as the matrix size increases.

Our PE array avoids repeatedly fetching operands from memory because values propagate directly between neighboring PEs. This is what makes the PE array architecture great for matrix multiplication. (This statement assumes PE array size and matrix sizes are the same - the PE array SystemVerilog implementation in TinyXPU is parametrized to support different number of rows and columns. In our upcoming posts, we will explore the impact of these different configurations on the performance of our chip.)

Finally, although our current implementation is just a PE array, we want to highlight that a real accelerator has other components in its microarchitecture which we are skipping here for the sake of simplicity:

• We need to define an Instruction Set Architecture (ISA) and include an instruction decode unit

• We need a Unified Buffer (typically SRAM) to hold the matrix data before streaming it into the PE array. (Typically, a DMA controller would stream data from CPU DRAM to the local SRAM.)

As this project evolves, some of these components will be (and must be) included in the design.

The Software Bridge

In the previous section, we described the hardware implementation of our TinyXPU Matrix Multiplier. Hopefully, you are convinced that the PE array is an improvement over using general-purpose CPUs for matrix multiplication. If you are a hardware engineer, many of your projects stop here with this question: how do you connect your hardware with the existing software ecosystem and actually run matrix multiplication on your chip? In part 1, we described two approaches to solve this problem - Runtime-level and Compiler level integration. In this section, we will implement runtime-level integration using an ONNX Runtime Execution Provider.

The ONNX Runtime Flow

The flow used in our project is intentionally simple.

First, a small ONNX model containing a matrix multiplication operation is generated using a Python script. An ONNX model represents a computation as a graph, where each node corresponds to an operation such as matrix multiplication. In our implementation, the generated ONNX file includes a MatMulInteger operator, as shown in this code snippet:

X = helper.make_tensor_value_info("X", TensorProto.INT8, [None, W_data.shape[0]])
Y = helper.make_tensor_value_info("Y", TensorProto.INT32, [None, W_data.shape[1]])
W_init = numpy_helper.from_array(W_data, name="W")
node = helper.make_node("MatMulInteger", inputs=["X", "W"], outputs=["Y"])
graph = helper.make_graph(
[node],
"MatMulInteger_4x4",
[X],
[Y],
initializer=[W_init],
)

Next, a Python driver loads this model using ONNX Runtime and registers the TinyXPU Execution Provider (EP), as shown in this code snippet:

ort.register_execution_provider_library("SampleEP", path_to_dll)
session_options = ort.SessionOptions()
session_options.add_provider_for_devices(tinyxpu_devices, {})

The goal of the EP is to inspect the ONNX model, looking for an eligible operation: in our case, it would be MatMulInteger. (See the GetCapabilityImpl() method in the TinyXPU EP for details.) The ONNX Runtime will call the function where we registered our ability to execute MatMulInteger when the session is instantiated.

When the runtime encounters the matrix multiplication operation, it dispatches that operation to the TinyXPU backend (via the ComputeImpl() hook) instead of executing it on the CPU (which is the default option when no match is found.) This way, we are able to connect the TinyXPU hardware directly to software without requiring a custom compiler - a major benefit of this approach.

Practical Constraints

Much like the hardware implementation, our current ONNX interface has some constraints that we would like to highlight.

Firstly, instead of exporting a Pytorch/Tensorflow implementation into the ONNX format, we directly use a Python script to generate an ONNX model containing a single MatMulInteger operation. This keeps the software stack minimal while still exercising the same ONNX Runtime execution flow that would be used with real models. Pytorch and Tensorflow both support functions to export to ONNX, so this additional step is fairly trivial. In fact, ONNX is a common intermediate exchange format before going to many hardware APIs including TensorRT for NVIDIA embedded devices, Qualcomm Hexagon, and others.

As mentioned in the previous section, we do not yet have an FPGA or ASIC implementation of our design for this part. That’s where Verilator comes into the picture. The SystemVerilog implementation of the TinyXPU systolic array is compiled into a cycle-accurate C++ model using Verilator. The TinyXPU Execution Provider links directly against this generated C++ model. When ONNX Runtime dispatches a matrix multiplication operation (MatMulInteger) to TinyXPU, the Execution Provider performs three steps (See the ComputeImpl() method in the TinyXPU EP for details):

1. Read input tensors from ONNX Runtime

2. Drive the Verilator simulation input signals and toggle the clock

3. Collect output signals from the simulation

So, in our implementation, the ONNX runtime connects the software world with a cycle-accurate simulation of the hardware, as shown in this diagram below (The “real silicon” version of this diagram was included in part 1):

It’s important to mention that irrespective of whether we are running on a Verilator-generated simulation, custom hardware implemented on FPGAs or as an ASIC, or existing processors on an SoC like CPUs, GPUs, or NPUs - this implementation successfully abstracts the hardware away from the software - our stated goal from Part 1.

Part 2 ends here. Although the current implementation is intentionally minimal, it already gives us a complete system: a software stack capable of dispatching matrix multiplication operations to a cycle-accurate simulation of our accelerator. This demonstrates how runtime-level integration works, and more importantly, provides a useful platform for experimentation.

In the next post, we use this framework to analyze the architecture in more detail, exploring how design parameters such as the size of the array compared to the workload and memory bandwidth affect throughput, latency and utilization. Check it out here:

The TinyXPU project will also expand to support activation functions to run complete ONNX ML models that can be deployed on an FPGA. To get notified about all this and more, subscribe to both Chip Insights and Min{power}. And if you have some experience taking up similar projects, or have an interesting direction for us to explore, let us know in the comments.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

This is the first post in a series aimed at understanding the ENIAC architecture and the lessons that remain relevant in today’s computing landscape. In this post, I introduce the ENIAC story and explain why it still matters.

The First Workload Problem

On December 7, 1941, the attack on Pearl Harbor became a key moment in computing history. The US, having now entered World War II, commissioned the Army’s Ballistic Research Laboratory (BRL) with a project: To improve the accuracy of artillery strikes. This effort led to the creation of artillery firing tables: one of the first large computing workloads.

When I was in school, some of our mathematics and physics exams did not allow the use of calculators. Instead, when complex arithmetic problems were involved, we were allowed to bring in something called log tables: collections of precalculated values used to solve complex arithmetic problems in place of a calculator. (I never understood the point then, but maybe it was all for this moment.) An artillery firing table was essentially the Army version of a log table. Instead of telling you what 10.14 ÷ 2.38 equals, it turns complex ballistic calculations into simple lookups that a gun crew can use to quickly determine the correct elevation or angle before firing at a target.

Unlike arithmetic, which does not change with time, firing tables were different based on the gun type, barrel wear, and environmental conditions. To generate the data for each of these situations, ballistic trajectories had to be solved numerically. Human calculators could not produce these tables with the speed and accuracy the war demanded. BRL used a Differential Analyzer, the state-of-the-art computer of the time, to run these calculations. They brought down the computation time from 3 days to just 15 minutes per trajectory. This was still too slow, and the results were often inaccurate, requiring human verification. The most important computational workload of that time was still waiting for a faster machine.

In 1943, the Army approved a proposal from J. Presper Eckert and John Mauchly at the University of Pennsylvania to build a machine with a completely new architecture. This computer, called the Electronic Numerical Integrator and Computer (ENIAC), had one goal: calculate firing tables at an exponentially faster rate to help with the US’s World War II defense. On September 2, 1945, just two years after the ENIAC project was approved, Japan surrendered, and this officially marked the end of World War II. ENIAC served its purpose and has its place in history as one of the most influential computers.

Well… that’s not quite how the story goes. Sure, ENIAC was commissioned to calculate firing tables, and its architecture actually improved the speed of trajectory calculations, reducing the time taken from 15 minutes on the Differential Analyzer to about 30 seconds. ENIAC also made giant strides to improve the setup time and accuracy when running ballistic calculations. But one detail surprised me: ENIAC was announced complete in February 1946. This means ENIAC was never used to compute a single entry in the artillery firing tables used during World War II. From 1943 to 1945, ENIAC experienced major delays, and also went significantly over its initial budget. ENIAC also consumed on the order of 150-200 kW of power, and sometimes contributed to power outages in the Philadelphia area. When the war ended, ENIAC was just an expensive infrastructure investment whose primary workload was no longer important.

Why ENIAC Still Matters

Despite a lack of demand for ballistic calculations, the ENIAC project continued after the war. In fact, this is where the ENIAC story really starts, and was my motivation to write this post.

ENIAC has lived through what I would call an “interesting architectural life.” It was rushed into existence to handle one specific workload. But that’s not why we remember ENIAC today. The ENIAC architecture is widely regarded as the first modern computing architecture. It was one of the first computers capable of running a program. ENIAC went on to effectively run weather models, nuclear simulations, and scientific computations. These workloads fell well outside the scope of military applications that it was designed for. At one point, the demand for ENIAC was so high, that renting ENIAC for two days cost as much as buying a new car.

This period of immense success also came with financial and political pressures on ENIAC to live up to the tag of a “general purpose computer.” That’s when the wheels started to come off, and the architectural deficiencies started to become evident. Eventually, it became clear that the ENIAC architecture could not be repurposed anymore to handle emerging workloads, and it had to be retired. The computing industry is less than 100 years old. It sounds like a lot of time, but it takes a long time for a new architecture to get adopted, and even longer for it to completely be abandoned. The ENIAC story is one of the few stories that has come full-circle, and has also been well documented. This presents a rare opportunity to analyze decisions that worked, and more importantly, decisions that failed.

There is another reason why ENIAC matters. Fundamentally, ENIAC looks nothing like modern computers. Many aspects of ENIAC are no longer directly relevant today, starting from the fact that ENIAC’s fundamental building block is a vacuum tube, not a transistor. However, since ENIAC essentially gave birth to modern computing, the ENIAC computer is simple enough (by today’s standards) that we can try to understand all aspects of the design. This includes the hardware architecture, software decisions, infrastructure challenges, supply chain issues, and interpersonal dynamics that were involved in the design. These details, in addition to making for a great story, are important to explain both the good and bad decisions that shaped ENIAC. In my opinion, attempting the same exercise with a modern computer, while more relevant, is almost impossible due to the sheer complexity involved.

This is why ENIAC is worth your time. But why now?

The New Infrastructure Buildout

Studying ENIAC is not just an academic exercise. Today, we are racing to build advanced computers that fill warehouses. They consume gigawatts of power, cost tens of billions of dollars to build, and are optimized for a single dominant workload: large-scale LLM training. This is not very different from how ENIAC began.

There are a lot of debates happening around the world about whether this investment is justified. I don’t see a lot of value in these debates because I think we are well past that point - today this infrastructure buildout is now inevitable. But if there is one thing that the ENIAC story tells us, it is this: the workload won’t remain the same. When projects attracts large investment, especially from traditionally “non-tech” sources like sovereign funds, it is expected to outlive its first workload. Cities do not build large bridges merely to connect two houses on opposite sides of a river; these projects are funded to enable large-scale economic growth in multiple different ways.

Language models are the first dominant workload of today’s compute infrastructure, and they are expected to keep this it fully utilized in the near term. But I am certain we will see new workloads soon. It could be as simple as a different approach to advance AI. Or it could be something well beyond AI - astrophysics, quantum algorithms, genomics. Regardless of which buzzword you choose to believe in, two important questions emerge:

1. Which compute infrastructure can last the longest before we need a clean reset?

2. How effectively can we map future workloads onto the infrastructure we are building today?

By studying ENIAC, I hope to build a mental model to help answer these questions. As Nvidia CEO Jensen Huang put it, even if you are not the first one to catch the metaphorical apple that falls from the tree, this mental model can help you become the first person to pick up the apple from the ground.

That concludes Part 1 of this series. If you like what I just said, follow along for upcoming posts in this series, starting with this:

Sources:

My primary reference for this study is the book “ENIAC in Action” which goes into an impressive level of detail about many aspects of this computer.

If you know any other resources to help with this project, leave a comment!

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

This is part of a series of posts on custom accelerators. Here’s a link to all parts:

Unless you have been living under a large silicon wafer, you would have heard some version of “custom silicon is the future.” In an earlier post, I attempted the classification of this exploding set of XPUs. While classification would help someone understand how to best match today’s chip with today’s algorithms, the real billion dollar question (literally) is: how can we ensure that tomorrow’s chips are best suited to run tomorrow’s algorithms.

One of the biggest problems in hardware-software co-design is that hardware and software have always existed on separate islands. So, the right way to answer this question is by bringing the two a little closer. In that spirit, this post (and hopefully many more to come) is the result of my collaboration with Avik De. Together in this post, we explore the “messy middle” between algorithms and silicon.

The push and the pull

To make custom silicon work, it needs both a “push” and a “pull”. There is a “push” from the side of the hardware designer (who we will call Harry) to get more developers to use their chip, and a “pull” from the algorithm writer (who we will call Alice) to find the best platform on which to run their software.

The algorithm writer (Alice) wants to be able to frame their algorithm in a high-level language. This algorithm could be anything, ranging from the numerous advancements in machine learning, to algorithms from emerging domains like robotics. Although Alice is confident about the value of her algorithm, she is unaware of the best hardware architecture to run it on. This is the “pull” - Alice needs to be able to find hardware to run her algorithm while staying within her software ecosystem of comfort.

Harry, on the other hand, has a great idea for a power-efficient way to compute matrix products by optimizing the operations, the instruction overhead, or the memory movement. He develops the design and synthesizes it on an FPGA. He knows it’s better than any other hardware architecture out there, but he has no idea how to get people to try it. This is the “push” - Harry wants to get his ideas out there, but needs to bridge it to Alice’s “pull.”

In theory, this looks like a match made in heaven - can’t Alice just take her algorithm and “run it” on Harry’s chip? Well, not quite, because of an idea that has been around as long as computing, but continues to be challenging.

An old analogy

The idea that Alice should be able to “just run” her code on Harry’s hardware is not new. In fact, it is the idea that made general-purpose computing work. A CPU solves this problem using the idea of Code Abstraction.

Say Alice was mapping an algorithm to run on a general purpose CPU - she would simply write the language using a high-level language like C++ or Python, “compile” this code, and run it on an x86 or ARM CPU. The reason why this process is so simple is the emergence of the idea of an Instruction Set Architecture (ISA) - which in simple words is a contract to map high-level languages to certain general purpose hardware. (This post skips the details of ISA and compilers, but if you are interested, check-out this older Chip Insights post.)

The mapping between Alice’s algorithm and Harry’s hardware using a standard ISA can be understood using this diagram:

One of the key advantages of specifying an ISA is that it allows software developers to write code that can run on evolving hardware without needing to know the specifics. Say Alice compiled her algorithm using an x86 compiler like GCC, when the next generation of x86 CPU is released, she will have to do one of the following:

• In majority of the cases, as the hardware microarchitecture improves, the same binary can benefit from improvements without needing to be rebuilt.

• Even if a major hardware improvements need to be accompanied by changes to instruction encoding or order (for example, Advanced Vector Extensions (AVX)), it is managed by “the compiler” - Alice simply needs to recompile the same high-level language code with the latest version of the compiler in order to enjoy the benefits of the new and improved hardware.

This separation worked spectacularly well, because:

• Hardware designers like Harry innovated on pipelines, caches, branch predictors, and out-of-order execution.

• Software designers like Alice largely ignored those details and built software that lived for decades.

• Compilers absorbed the complexity at the boundary.

However, custom accelerators break this assumption.

Why isn’t there a standard ISA for custom silicon?

In the early days of CPUs, every CPU provider had their own custom ISA, largely because hardware constraints and software ecosystems were still tightly coupled. Over time, a few winners emerged, leaving us with just 3-4 significant ISAs, each supported by mature compilers and software ecosystems. This convergence was possible because general-purpose programs share a common structure: scalar control flow, pointer-based memory access, and relatively uniform performance characteristics across workloads.

There is a common misconception that custom accelerators would evolve in the same way - that over time, a standard ISA would emerge, and every accelerator would support it. But this idea is fundamentally flawed. To truly understand why a standard ISA cannot exist for custom accelerators, let’s consider a thought experiment. Assume that Harry defines a standard ISA for his accelerators. Let’s try to think about how that ISA would look for an operation like matrix multiplication.

If Alice is using PyTorch, the operation would look something like this:

result = torch.matmul(A, B)

There are certain aspects of matrix A and matrix B that inform Harry how to execute this operation efficiently on his specific hardware. Crucially, these aspects influence not what computation is performed, but how it is scheduled and mapped onto hardware resources. For example, the shapes of A and B can be used to decide:

• Tiling strategy: How many chunks should the matrix multiplication be split into based on the available hardware resources

• Buffering techniques: what data can be stored in the fast, on-chip SRAM, and what should be moved to DRAM.

• Data layout: For the specific size of A and B, is column major better, or does a tiled layout reduce memory accesses?

This is a short list that only considers the matrix shape - several other factors, like the data types, sparsity, etc. can be exploited to map this operation efficiently in hardware.

However, Harry needs to define a standard ISA that encompasses all these possibilities - the same ISA must work for a large square matrix with significant sparsity, as well as for matrices with just a single row or column. The compiler alone is of limited help here, because many of the most performance-critical decisions like tiling, buffering, and layout are instance-specific and cannot be encoded directly into an instruction set contract.

If Harry tries to account for all these possibilities within an ISA, he must either:

• Over-specify behavior, resulting in an unoptimized, CPU-style ISA and microarchitecture

• Stick to a very low-level abstraction (resulting in bloated hardware - like one large matrix multiplier with huge amounts of memory.)

In either case, a standard ISA would strip away the very benefits that customization is meant to provide. So where does that leave Harry and Alice?

The two bridges

Since Harry can’t simply expose a standard AI ISA and call it a day, the question becomes: how does his hardware connect to Alice’s world at all?

Today, there are two practical bridges between them.

1. Runtime-level integration

In the first approach, Alice expresses her model in a standardized, framework-agnostic format. A shared runtime is responsible for executing that model and deciding which parts run on which device. From Alice’s perspective, almost nothing changes. She writes PyTorch or TensorFlow code and exports a model. From Harry’s perspective, the integration surface is narrow and predefined: he implements support for a fixed menu of operations, and the runtime takes care of everything else.

A useful analogy here is a food court appliance. Alice places an order by pointing to items on a menu: “grill this,” “blend that,” “heat this up.” The food court manager (the runtime) decides which appliance handles which step. Harry builds a specialized appliance that performs certain actions very efficiently. If the order matches the appliance’s menu, the runtime sends the work to Harry’s hardware and it shines. If not, the runtime quietly routes those steps to a different device that knows how to handle them. (which is typically a CPU or a GPU.)

This is exactly how runtime-level integration works:

• The model is a graph of predefined operations

• Each operation is stateless and self-contained

• Control flow and orchestration live outside the accelerator

A concrete example of this approach is ONNX Runtime Execution Providers, where hardware vendors accelerate supported operators while delegating unsupported parts of the model to a fallback device. This diagram explains how Alice’s software and Harry’s hardware can talk to each other using ONNX.

This model works well for inference workloads, where computation is a fixed graph of tensor operations. However, because execution and control flow remain outside the accelerator, it limits how much of the hardware structure (custom memory hierarchies, interconnect flow) can be exposed.

2. Compiler-level integration

In the second approach, the bridge moves deeper into the stack. Instead of plugging into a runtime API, Harry integrates at the compiler level, where programs are transformed before execution. Alice still writes high-level code, but now the compiler is responsible for mapping that computation onto the hardware - deciding how loops are formed, how memory is reused, and how execution flows on the device.

The analogy here is a custom-built kitchen.

Instead of ordering from a menu, Alice hands over a recipe. Harry designs the kitchen itself: where ingredients are stored, how cooks move, which steps happen in parallel, and how decisions are made mid-cooking. The recipe is compiled into a precise plan tailored to that kitchen.

This may sound similar to targeting a standard ISA, but the distinction is critical. A standard ISA fixes the instruction set and execution model that all software must target, forcing hardware innovation to happen below that boundary. Compiler-level integration, in contrast, fixes only the meaning of the program. The compiler is free to reshape loops, memory usage, and control flow to match the hardware, without exposing a stable instruction set to the programmer. This allows each accelerator to express its architectural strengths without being constrained by a one-size-fits-all ISA.

This approach requires much more effort from Harry, because he must:

• define how programs are lowered,

• expose memory and synchronization primitives,

• and implement a runtime that can execute compiled programs.

But in return, Harry can support:

• data-dependent control flow,

• irregular or sparse computation,

• custom memory layouts,

• and long-lived, stateful programs.

A representative example of this model is IREE, which provides a compiler, optimizer, and runtime along with a hardware abstraction layer. Instead of mapping individual operators, Harry defines how programs execute on his device. This turns the accelerator from a co-processor that executes individual operations into a programmable compute target that runs complete programs independently, as shown in this diagram.

It’s worth emphasizing that IREE can ingest programs from multiple frontends, including ONNX, before lowering them through its compiler stack. The distinction between runtime-level and compiler-level integration is therefore not about which framework Alice starts from, but where hardware-specific decisions are made.

Conclusion

We’re entering a phase where meaningful gains in AI performance and efficiency are increasingly coming from custom silicon. But the success of that silicon depends less on raw compute and more on where it connects into the software stack.

This post was intentionally a high-level overview of that connection point. The space is complex, fast-moving, and still evolving, but at a distance, two broad patterns are already visible. Runtime-level integration offers a fast path to deployment by fitting new hardware into an existing execution model, while compiler-level integration demands more effort but unlocks far greater control over how computation actually runs. Neither approach is “better” in the abstract - each reflects a different tradeoff between ease of integration and expressive power.

In the next parts of this series, we’ll make these ideas concrete. On Chip Insights, we’ll walk through building a simple runtime-level backend, starting with a CPU implementation and then extending it toward custom hardware on an FPGA. In parallel, on min{power}, we’ll look at the problem from Alice’s side: how different classes of AI and robotics workloads place fundamentally different demands on hardware, and why those differences increasingly matter.

You can find the next part of this series here:

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

In 2022, a graduate-level chip design course at UCLA gave me a glimpse of what felt like the future. Instead of writing RTL in Verilog, we were required to use a Python framework that generated synthesizable Verilog under the hood. I had already been using Verilog for a few years by this point, so this new workflow felt strange at first. But after a few weeks, something clicked. I started to enjoy it. So much so that I made my final project a head-to-head comparison: plain Verilog versus this Python-based approach across several types of RTL IP. The results seemed clear. Yes, the Verilog implementations had slightly better power, performance, and area (PPA) metrics. But the Python framework won on almost everything else that mattered to an RTL designer: it scaled better, made reuse trivial, and produced RTL that was far easier to read and reason about.

I walked away convinced that this was the direction chip design was headed. Higher-level frameworks would replace handwritten Verilog. That’s where I was ready to place my bets.

Lucky for me, I didn’t have the money to bet - because I was completely wrong.

Verilog Is Fine. What Next?

If you have been reading Chip Insights long enough, you’ll know about my two-part HDL saga documenting the story of how Verilog became what it is today. I ended this story in the mid-2000s, when EDA companies started to push for the widespread adoption of SystemVerilog. From this point forward, the Verilog family (plain Verilog and SystemVerilog) emerged as clear winners in the HDL wars. However, there was a parallel storyline which I did not cover in that post.

The story of HDLs was closely related to the emergence of a new simulation technique called behavioral simulation - as designs got bigger, gate-level simulations became slow and unusable for routine functional checks. Cadence enabled behavioral simulation in Verilog through Verilog Compiled Simulation (VCS) - Verilog was first compiled into C, and this C program was then used to run simulations. Great story. The bedrock of modern logic simulation. But that’s not the point here. If you can convert an HDL like Verilog into a high-level language like C, why can’t we start with a high-level language directly?

In 1998, Forte Design Systems came up with a tool called Cynthesizer (a pretty clever name, I must add), which allowed a designer to build synthesizable logic using SystemC. In 2001, Sony became the first company to tape out a chip using this approach, which would later be known as High-Level Synthesis (HLS). As transistor scaling continued in the 2000s, design cost and complexity grew rapidly. Even the most modern HDL of the time, SystemVerilog, was inadequate for human productivity to catch up with Moore’s law. This made HLS a compelling candidate.

There are two primary reasons why HLS was so attractive. The first is obvious: high-level languages provide constructs that HDLs lack which would boost the productivity of chip designers. The second reason was even more compelling. HLS represented a fundamental “shift left” strategy to move hardware design earlier in the development cycle and closer to software design. In other words, anyone who can code should be able to design the hardware to run their code. Believers of HLS shared the same optimism that I did when I used Python for RTL design. In fact, in the mid 2000s, Professor Jason Cong built the xPilot HLS system in the same UCLA building. xPilot pioneered a lot of algorithmic innovations that made HLS more than just a convenience - the PPA metrics started to improve as well. xPilot became AutoESL Design Technologies, was acquired by Xilinx (now AMD) in 2011.

So far, this sounds more like a success story of HLS than what my title suggests. So what happened? HLS definitely proved to be useful in certain cases - like rapid prototyping and deployment on FPGAs. (The core technology of xPilot powers Vivado HLS, which is still used widely today.) However, HLS could never fully replace Verilog, because the promise of HLS was too good to be true. Even today, HLS-generated designs often consume more hardware resources and achieve significantly worse clock frequencies compared to handwritten RTL. More importantly, HLS never truly became what it promised. The fundamental mismatch between software-oriented C/C++ languages and hardware structures meant that HLS tools still can’t reliably produce synthesizable designs that integrate well with existing EDA tools which remain overwhelmingly Verilog-centric.

As a compromise, the idea of a Hardware Construction Language (HCL) was born. While HLS tools were intended to automatically infer the best microarchitecture from high-level algorithms, HCLs are high-level language extensions that allow designers to express hardware structure, while simultaneously leveraging powerful software constructs. Chisel, an extension of Scala, was developed at UC Berkeley in 2012 and is one of the most successful examples of an HCL. By the way, the Python framework I mentioned at the start of this post was also an HCL.

While HLS and HCLs were not perfect, they seemed like early versions of what the future of chip design would look like. HLS algorithmic innovations continue to improve PPA metrics, and a lot of new processor designs are built using frameworks like Chisel. All this while, the Verilog family has not made any major strides. If I had written this post a few years ago, this would be the end of the HDL story. But almost overnight, a new playbook has emerged.

You Can Have Your Cake and Eat It Too

The original promise of HLS and HCLs was never about replacing Verilog for the sake of it. It was about building chips faster: by increasing designer productivity, reuse, and scalability. These benefits came with tradeoffs like worse PPA and painful EDA integration. As they say, there are no free lunches.

Generative AI changes this equation in a fundamental way.

With GenAI-assisted Verilog, you get many of the same benefits that HLS and HCLs aimed to provide, without abandoning the Verilog ecosystem. The productivity gains are the most obvious. Writing boilerplate RTL, parameterizing modules, refactoring interfaces, or instantiating complex microarchitectural patterns are all tasks that LLMs handle remarkably well. What once justified a higher-level language now often collapses into a single prompt. You still end up with Verilog, but you get there faster, with fewer errors, and with much less cognitive overhead.

More interestingly, GenAI quietly revives one of the most compelling arguments for HLS: the “shift left.” While this shift was long promised, HLS never reliably delivered on it. PPA was difficult to estimate accurately at the HLS abstraction, and those estimation errors were often more costly than starting from RTL in the first place.

Generative AI flips this dynamic entirely. Instead of postponing RTL, it accelerates its creation. High-level specifications, performance targets, and even informal design intent can now be translated directly into functional RTL models early in the design cycle. Hardware and software can evolve in parallel, not because RTL is avoided, but because it is cheap to generate, modify, and discard. In other words, GenAI enables a shift left, without a shift away from Verilog.

Why Verilog Was Ready for This Moment

Verilog didn’t reinvent itself for the GenAI era. The GenAI era quietly reinvented itself around Verilog.

First, Verilog sits at the center of the EDA ecosystem. Decades of tool development have gone into taking Verilog as input and squeezing out the best possible PPA. Synthesis, place-and-route, timing closure, power analysis: these flows are deeply optimized for RTL written in Verilog and SystemVerilog. The switching costs are enormous.

Second, in the already limited chip design data available to train large language models, Verilog dominates. This creates a powerful flywheel effect. Verilog is used to train LLMs, which then generate more Verilog, further reinforcing its position as the language of digital design.

Today, we are seeing a combination of these two factors, in the form of closed-loop RTL design agents. Verilog code is both a cycle-accurate, and formally checkable artifact, which can be used to generate accurate reward signals to improve the AI. (I’m not going into details of these systems here, but all signs are pointing towards such RL environments becoming the future.) The implication of this is that GenAI won’t just lead to more Verilog, it will lead to better Verilog. Over time, this creates a compounding effect - Verilog will become the most optimized way to design chips, even if that Verilog is generated by an AI agent.

In the world I just described, HCLs and HLS tools seem irrelevant. If Verilog is cheap to generate, easy to iterate on, and continuously optimized by feedback, the incentive to move away from it fades. Verilog survived long enough to see the light at the end of the tunnel. Now, the future looks brighter than ever.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

Another disclaimer: The characters and events in this post are fictitious (but the conferences are real). Any resemblance to real people is purely coincidental (though statistically inevitable in my audience). For the record, I have not faithfully followed all of these conferences in the past. This post exists mainly because I’d like to start doing so this year and thought I might as well share ten conferences I think are useful. Unfortunately, I couldn’t resist making the whole thing a little funny, so here we go…

The protagonist of this story is someone I’m sure you have crossed paths with. They forget birthdays, miss tax deadlines, and couldn’t name a single band on tour - but they know exactly when and where every major computer architecture conference is held.

Their calendar isn’t divided into months. It’s divided into trade shows.

This is how a year in that person’s life looks.

CES: Between hope and hype

Our protagonist loves speculation. Branch prediction, speculative execution, prefetching: all the performance gains they deliver are built on confidently guessing what will happen next. So there is no better way to start the year than CES in January. While CES isn’t always about chips, the recent AI accelerator mania has turned it into the perfect launchpad for the year. Bold claims, ambitious roadmaps, and just enough technical detail provide our protagonist with exactly the right mix of signals and buzzwords to carry them through the months ahead.

Why it matters

CES tells you what’s being productized and will therefore shape algorithms and architectures of the future.

How to follow it efficiently

Watch the keynotes. Skim the announcements. Pay attention to products that actually exist, not just the slides describing them.

One thing to watch in 2026

Has anyone built a useful, physical product where AI is solving a genuine problem.

ISSCC: Let’s get real

By February, our protagonist’s speculation bubble goes through a silicon trial by fire. ISSCC is where we stop guessing and start measuring. Statements and projections from CES slides are replaced by die photos, power numbers, and phrases that begin with “fabricated in…”. Each one either validates past optimism or quietly resets expectations.

Why it matters

ISSCC is a good way to understand how performance and power scale between architectural simulations and real silicon.

How to follow it efficiently

Skim broadly to identify the most relevant designs. Then read one or two papers deeply to understand the true state of the art. Watch out for assumptions.

One thing to watch in 2026

How is Moore’s Law really progressing? Are improvements coming from architecture, process nodes, or both?

GTC: Crowning the Starboy

March belongs to the Taylor Swift concert of today’s tech world. Our protagonist once followed GTC for innovations in parallel computing architectures. Now, it’s a coronation ceremony for Nvidia’s platform and its developers. Everyone listens to Jensen Huang carefully, because this is where the industry’s vocabulary for the next twelve months is minted.

Why it matters

GTC now defines the AI narrative and vocabulary that everyone else ends up adopting.

How to follow it efficiently

Watch the keynote. Then pick a handful of genuinely technical sessions. Ignore the excessive press coverage and stock-price astrology.

One thing to watch in 2026

What does Nvidia’s platform roadmap look like for inference—especially in a post Groq license world?

Computex: Board yet?

Computex reminds our protagonist that chips do not exist in isolation. This is where architecture turns into motherboards, racks, cooling solutions, and deeply uncomfortable power budgets. It’s less about novelty and more about integration. More recently, it has also become the place where vendors launch their latest SoCs for, you guessed it, AI.

Why it matters

Computex shows what it actually takes to turn a good architecture into a usable product. It’s important to be aware of what’s happening outside the cores.

How to follow it efficiently

Get a high-level view of system architectures for the latest SoCs. Pay attention to reference designs and power, thermal, and memory numbers when they’re provided.

One thing to watch in 2026

Whether the industry finally agrees on what an “AI PC” is, and if any new players enter that space.

ISCA: Sheer elegance

ISCA is where our protagonist feels intellectually nourished. The best ideas in computer architecture are here in their purest form. Some are brilliant. Some are fictional. A few will quietly influence designs a decade later. No product timelines, no market requirements. Just elegant computer architecture.

Why it matters

ISCA sharpens your architectural intuition by offering new ways to think about familiar problems.

How to follow it efficiently

Skim all abstracts. Then invest real time in a few papers that genuinely intrigue you. Ask whether ideas from one domain could transfer to another.

One thing to watch in 2026

How are CPU architectures evolving in the era of accelerated computing?

DAC: Ship faster

All the old-school, hype-free academic computer architecture study leads our protagonist to wonder if AI is just a bubble. DAC arrives at exactly the right time, reassuring them with claims that AI-powered tools can reduce the time to ship a new chip from years to seconds. It also serves as a reminder of the less glamorous innards of chip design, like verification and physical design.

Why it matters

An architecture is only as good as its ability to be built, and more importantly, built on time. DAC shows how semiconductor tooling is evolving to close the gap between ideas and tape-out.

How to follow it efficiently

Skim panels to understand what’s new in tooling. Before getting excited, always check whether these tools actually work on industry-scale designs.

One thing to watch in 2026

If, and how AI-assisted design flows are being deployed in the industry.

Hot Chips: Industry, minus the hype

As summer draws to a close, our protagonist finds themselves desperately searching for the truth. This leads them to Hot Chips, where architects speak slowly, precisely, and with slides that took their companies months to approve. Tradeoffs are admitted. Constraints are acknowledged. Reality is explained. Our protagonist trusts this conference more than almost any other.

Why it matters

Hot Chips is an industry conference that is unusually light on marketing. The goal is to simply explain what was shipped and why.

How to follow it efficiently

This is worth spending time on. Watch the talks. Read the slides carefully. Take notes.

One thing to watch in 2026

How many meaningfully different AI architectures do we really have?

SEMICON Taiwan: Supply chain magic

By September, the story shifts to the heartland of semiconductor manufacturing. Here, our protagonist listens to conversations about yield, packaging, and process limits. The names of many exhibitors sound unfamiliar, but a closer look reveals them to be critical players in the semiconductor supply chain. It’s a reminder that architecture is important, but only a small part of a vastly more complex world.

Why it matters

Manufacturing constraints increasingly shape architectural decisions. They can’t be ignored.

How to follow it efficiently

Use this as a chance to understand the supply chain better. Process roadmap and packaging sessions are the most valuable.

One thing to watch in 2026

What’s new in the world of advanced packaging and 3D ICs that are actually ready for volume production.

MICRO: Grounding ideas to reality

If ISCA feeds our protagonist’s love for elegant ideas, MICRO satisfies their need to see those ideas survive contact with reality. This is where high-level architectural concepts are dragged into the microarchitecture and explained cycle by cycle. MICRO isn’t for the casual computer architecture enthusiast. The diagrams are denser, the assumptions sharper, and the discussions expect real chip-design experience.

Why it matters

MICRO bridges the gap between architectural ideas and real implementations, exposing the costs, tradeoffs, and complexity that abstractions tend to hide.

How to follow it efficiently

Focus on papers that include detailed evaluations and realistic assumptions. Pay attention to how architectural ideas translate into microarchitectural blocks.

One thing to watch in 2026

Are there credible examples of AI influencing microarchitectural decisions better than a human designer can?

IEDM: A glimpse of the future

The busy computer architecture calendar comes to a close at IEDM. Our protagonist listens to talks about devices, confidently thinking they won’t matter for years. Soon, a quiet fear sets in that, when these devices finally do matter, they might change everything. The thought sends a chill down their spine, and pushes them to search for notes from the one device physics class they took in undergrad.

Why it matters

IEDM defines what architectures may even be possible a decade from now. Following it is a form of long-term career insurance.

How to follow it efficiently

Don’t get stuck on the physics. Assume the devices work, then ask how they would reshape memory hierarchies, compute models, and system architectures.

One thing to watch

Which emerging technologies, like quantum, neuromorphic, and novel memories, are close to being deployed in real chip

By the end of the year, our protagonist is carrying a heavy mental load: half-remembered acronyms, conflicting roadmaps, benchmark caveats, packaging buzzwords, and just enough existential dread to stay alert. They’ve learned when to believe, when to squint, and when to politely wait for silicon. And just as they start to feel like they’ve finally made sense of it all, the year resets. New nodes. New models. New claims. Same conferences. So, they clear their calendar, open a fresh notebook, and show up again, because as stressful and confusing as it sounds, somehow, it’s still fun.

Want to follow this year’s conferences with me?

I’ve created a shareable Google Calendar with all of these conference dates pre-filled, for anyone who wants their existential dread to be neatly scheduled. Subscribers will receive a link to this calendar - consider this a gentle nudge to subscribe if you haven’t already.

If there are other conferences that every computer architect should mentally budget for, let me know in the comments. The calendar, like the study of computer architecture, is never truly complete.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

CPU. GPU. TPU. NPU. DPU. IPU. LPU.

Every year, the alphabet soup gets richer. But I wonder if it’s adding any value? Over the last two years, GPUs were the unquestioned champions of the AI boom. Right now, TPUs are having a moment as the new architecture that is supposedly better for AI than GPUs. NPUs reliably show up during some product keynotes, often accompanied by “inference” or “low power.” Meanwhile, CPUs still have their audience of PC and smartphone enthusiasts.

At this point, the industry’s obsession with new “PUs” is starting to look uncomfortably familiar. We’ve seen this movie before - with process nodes. Once upon a time, node names conveyed real, comparable information. Then they became branding. “7nm” stopped meaning 7nm. “3nm” stopped being smaller than someone else’s “5nm.” The label survived; the meaning didn’t. I think processor acronyms are heading down the same path. Pick a new letter. Add “accelerator,” “custom,” or “ASIC.” Publish a carefully chosen benchmark. Congrats, you have a new processing unit.

The problem isn’t that TPUs, NPUs, or GPUs aren’t important or innovative. The problem is that the acronym leaves out the essence of the architecture: what it’s actually optimized for, what tradeoffs it makes, and where its real value comes from. If you squint hard enough, almost anything can be called an accelerator. If you zoom out far enough, almost everything looks like a CPU. This does not sit well with me.

Continuing with the theme of my last post, I don’t think the right question is “Is this a GPU or an NPU?” It’s whether we’re even using the right vocabulary to evaluate compute architectures at all. Instead of arguing over letters, we should be asking sharper questions - questions that force clarity instead of reinforcing branding. I think the following four questions are a far more informative way to classify chips.

Question 1. Host relationship: Is your product the boss, a helper, or just another block on the die?

Categories:

• Standalone: Boots an OS and owns the system.

• Co-processor: Lives behind PCIe or a fabric. Needs a host to schedule work, manage memory, and keep the lights on.

• Integrated IP: One of many blocks inside an SoC, visible to software only through drivers or libraries.

Why it matters:

Host relationship determines who controls the system. Standalone processors shape the entire software stack and system architecture. Co-processors live and die by host integration and software orchestration. Integrated IP blocks rarely escape the platform they’re embedded in, no matter how impressive the pure-silicon performance is. Two chips with identical compute units can have wildly different impact depending on whether they’re in charge, or waiting for instructions.

Question 2. Domain coverage: How many different major software domains (at least a million developers) can this architecture serve effectively?

To clarify, I consider the following as major software domains of today.

• System software: OS kernels, language runtimes, compilers, CLI tools

• User Interfaces: desktop/mobile apps, browsers, light graphics

• Data processing: SQL databases, analytics, streaming

• Media & graphics: Video encoding/decoding, 2D/3D graphics pipelines

• High Performance Computing: simulation, scientific computing, linear algebra

• ML/AI workloads: training and inference across model families

Categories:

• General purpose: The chip is competitive or better than the incumbent across 3 or more domains

• Domain specific: The chip excels only in 1-2 domains

Why it matters:

Domain coverage is the difference between platforms and point solutions. General-purpose architectures benefit from massive software ecosystems, long lifetimes, and constant reuse. Domain-specific chips can deliver spectacular gains, but only as long as the workload stays stable. When you are looking at a domain-specific chip, the domain matters as much, if not more, that the silicon innovation. (As Jensen Huang puts it, pick “Zero Billion Dollar” markets.”)

Question 3. Execution paradigm: What architectural feature delivers throughput?

Categories:

• MIMD (Multiple Instruction, Multiple Data): Many independent cores, each running its own control flow. Classic multicore and many-core CPUs live here.

• SIMT (Single Instruction, Multiple Threads): Groups of lanes share an instruction stream with per-lane masking. This is the beating heart of modern GPUs.

• Custom: Fixed or semi-fixed dataflow architectures designed around specific computation patterns (e.g., systolic arrays, spatial fabrics).

Why it matters:

Execution paradigm reveals what kind of parallelism the architecture is betting on. MIMD favors flexibility and irregular control flow. SIMT thrives on massive data parallelism. Custom paradigms trade generality for efficiency. Once you understand this axis, performance claims stop sounding magical and start looking like predictable outcomes of design choices.

Question 4. Programmability model: How does a programmer get work done from your chip?

Categories:

• Native ISA: General-purpose compilers (C/C++) target it directly.

• Kernel-compiled: You write kernels in a language extension like CUDA, OpenCL, HIP, or a domain-specific language.

• Graph-compiled: You describe computation as a graph (often via an ML framework like PyTorch or TensorFlow), and a compiler maps it to the hardware. In graph-compiled systems, the graph is the primary abstraction for the programmer, not the kernel. (Graph-compiled systems still use kernels underneath.)

• Bitstream configured: You configure the hardware fabric itself. For example, FPGA bitstreams or CGRA configurations.

Why it matters:

Programmability determines who can use the hardware and how fast ecosystems form. Native ISAs scale with developer count. Kernel models reward specialists and expert programmers. Graph-based systems trade flexibility for convenience: they work extremely well when your problem fits the graph, and poorly when it doesn’t. Bitstream-based approaches offer ultimate hardware control, but at the cost of accessibility. Many promising architectures fail not because of silicon, but because the programming model never escapes a niche audience.

Taken together, these questions offer a better way to think about modern compute architectures. Once you look at chips through these lenses, debates like GPU vs TPU vs NPU start to feel oddly shallow. Architectures stop being mysterious, and performance claims start to look like the predictable outcomes of very specific tradeoffs. For instance, below is a classification of several chips that are often loosely grouped under the label “accelerators.” This classification forces each architecture to reveal its position along concrete design axes: who controls the system, how broad the user base is, where throughput actually comes from, and how programmers interact with the hardware. Chips that are often lumped together as “accelerators” end up in very different places once you apply these questions, and those differences explain far more about real-world impact, adoption, and longevity than any three-letter acronym ever could.

If you haven’t subscribed yet, here’s a reason to: when you subscribe, you’ll receive the link to the full list with other categories and explanations.

If you have read so far, I’m curious - what other questions would add value to chip architecture classification?

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

It’s been more than two months since I wrote my last post. One of the reasons I took a pause was to deliberate over a question that just wasn’t leaving me: Does a technical newsletter like Chip Insights matter in a world where GenAI keeps getting better? To answer this, I need to first define what “matters” means. The reality is, this newsletter will only ever reach a small fraction of people who are interested in a very niche topic - so traditional content metrics like views or subscribers make no sense. The only true feedback for my work comes from this question: “Would I read this post if it was written by someone else.” With today’s AI tools, that question really becomes: “Is it easier to generate this post on an AI chatbot, that write it?”

Things we tell ourselves to feel better

One of the most common arguments in defense of humans is that “AI makes mistakes.” I don’t think this is a strong argument to hide behind. Sure, chatbots hallucinate and make mistakes, but so do humans - even the most decorated content creators. Like a lot of technologies we have seen in the past, I’m sure the errors will reduce over time, and AI answers could become a reliable source of information. In my opinion, content creators saying their content is better because it’s more accurate, are willing to challenge an ever-improving machine backed by trillions of research dollars. Personally, I think that’s a losing battle.

There is another defense of traditional content which never sat well with me. I’ve seen a lot of arguments about how consuming difficult technical content is “inherently noble”, and AI provides processed answers which are “cheats” and will shrink your brain. In my opinion, difficulty, by itself, should never be a virtue to strive for. Before humans discovered fire, digestion was hard, consuming a lot of energy. Cooking made food easier to process, freeing energy for other activities like thinking, building, and evolving. I’m quite sure AI-generated educational content will have a very similar positive effect.

So before dismissing AI, we should acknowledge the aspects that AI excels at.

Is AI is coming for my posts?

Today’s AI is extraordinary at compressing information. It can digest long, messy documents and explain them in simpler terms. Interestingly, this is what most traditional content creation has centered around. (Including some of my posts, I’ll admit.) I think we are getting very close to the point where the returns from such content won’t justify the investment - it will be so much easier to generate such content on demand.

AI is also a fantastic tool to explore content related to a specific topic you are interested in. This has been my favorite use of AI: I can easily get a list of 5-10 sources I want to explore for my research on a topic, bypassing the SEO-engineered, often irrelevant links. In my experience of doing this, I have found that a few in-depth engaging sources, along with the AI summary on the topic, are sufficient for my research - I don’t want to jump multiple links where parts of the information are spread out.

Essentially, if the purpose of a piece of writing is simply to transfer information from one form to another, or one place to another, AI will inevitably outperform humans on speed, breadth, and efficiency.

So where does that leave me?

The point of this exercise was not to say that AI will replace content creation entirely. The concept of generative AI fundamentally cannot displace certain types of content - which is specifically what I want to focus on.

My north star for content creation has been the Acquired podcast. For those who don’t know, the idea behind Acquired is to tell the story of a company, with all of its gory details, in episodes that sometimes last 4 hours. Today’s AI struggles with such analysis, and there are fundamental reasons why it might be this way for a long time. An LLM lacks a “mental model” - understanding of concepts like time, hierarchies and abstractions. Models also gravitate towards a median of possibilities. While this makes AI excellent at flattening complexity, maintaining engagement sometimes requires expanding complexity into a narrative form. The failure to do this is what makes AI sound “robotic.” While you can get away with a robotic tone if you have “explain the differences between a CPU and a GPU,” it’s important to keep the audience engaged if you want to “share a deep dive on the evolution of GPU architectures”.

Another problem with AI content is that it is user driven - the value of the answer depends on the question you are asking. AI might have all the answers, but AI isn’t curious. Good content, therefore, should prompt the audience to ask more questions - questions about bottlenecks, architectural shifts, industry narratives, or places where assumptions are beginning to bend. Questions like these generally emerge from judgment, from noticing patterns, and from wondering what others might be missing - something that AI cannot replicate.

This is what I’ve been thinking about during my break: that the real value of technical content isn’t in being a source of information, but in being a source of interpretation. I’m looking forward to returning to writing with these ideas in mind.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

This post is not about chips. It’s not even about AI like that one post was. This is about me.

As I’m writing this, it’s been a year since I wrote my first post on Chip Insights. So, I thought I’ll reflect a little and open up about my plans for the future.

Why did I start Chip Insights?

Ideally, this should have been a part of my first post. But I did not have a good answer. Instead, I chose to write about Moore’s law:

Turns out, Moore’s law was actually the perfect way to start, because, like the number of transistors in a chip, ideas compound exponentially. Every new post I have written here has made me an exponentially better thinker. So more than anything, Chip Insights is my training ground; the place where I take half-baked ideas and make them concrete. I learn more from Chip Insights than any of my readers.

Well if that’s my reason, one could ask (and a lot of people have asked) why I post this content publicly. I have found that there is a huge difference in the way I think in private, versus when I need to present my ideas. The vulnerability associated with public content has pushed me to think more deeply than I would have in a private brainstorming session. For instance, I always thought I knew what CPU bitness was, but when I decided to write a post about it, I realized how complex that number truly is.

What do I offer?

I’m aware that in a post addressing my subscribers, the right way to answer the “why” question is to say something altruistic, something like “I want to pass on my learnings to students.” It’s nice when that happens, but I would be lying to myself if I said everything I do is towards that goal. However, I have two ways of thinking that one might consider altruistic.

First, I am convinced that the positioning of computer engineering as a career needs some work. There are a few attributes that computer engineering has acquired over time which does not sit well with me: slow-moving, hard, and risk-averse come to mind. A lot of very talented students are put-off by these attributes. The emergence of AI has made semiconductors cool again, and I want to use this opportunity to reframe how the industry is perceived. For example, not a lot of people know how fast-paced the EDA world was - a story I covered here.

There’s another issue. Even if you like this industry and want to pursue your career here, there are too many roadblocks, which eventually kills potential talent. I am not in a position to bring about structural changes to the industry, but one thing I can do is to reduce some anxiety in the process of making it, which is what led to posts like this.

How is Chip Insights different?

I have done some technical writing before Chip Insights - in school, at work, and also authoring research papers. But the advantage of a medium like this is the freedom it provides. I have used this freedom in three ways.

First, I was able to bypass the need to explain concepts from the ground up - as you would do in a typical college program. For example, the right way to understand power consumption in chip design is to first understand exactly how a transistor works. But I have always found this to be a bit restrictive - in the case of chip power, only engineers with visibility to the transistor level contribute ideas, and they are often quite similar. If a software engineer wants to understand what makes a chip power efficient, they would be scared off by all the complex transistor-level terminology and would make no progress. My post on power optimization was an attempt to see how much I could abstract out without being inaccurate - turns out, quite a lot.

Another theme in most of my technical post is the idea of analogies - comparing chip design concepts to something relatable - to make them stick in your mind a little longer. I know some people, especially the “academic” types, find analogies to be juvenile. But I have actually found that thinking in terms of analogies, and stretching them out as much as possible really strengthens your understanding of something. For example, I tried that with on-chip networks by comparing them to airline routes in this post.

Finally, there is world building. I think this is the hardest to crack, because building interesting worlds isn’t easy - if it were, The Walt Disney Company wouldn’t be worth $200 billion. But if you can build a world that is compelling enough, then even reading about a pedestrian topic like Static Timing Analysis can be made fun, which is what I tried here.

How has my writing evolved?

I never realized this, but a lot of my early posts were based on topics which had very little scope for subjectivity - for example, this post about pipelining, which walks through how a traditional CPU pipeline has always looked like.

I think a part of me was unsure about how new ideas would be perceived - especially from someone who lacks authority. But as I went forward on this journey, I think my writing has become more assertive. I wrote a post on how AI will influence EDA, which was away from my comfort zone, because, well, nobody still knows the answer.

A bolder version of this was my more recent post about full-stack chip design, and idea that irked a lot of readers on Hacker News, but something that I believe is a well thought out analysis.

I also expanded to cover aspects beyond technical knowledge, because I think the kind of person you are strongly influences the kind of engineer you would be, and we engineers are very bad at understanding this. I aim to do more of this, but one place where I see this mattering a lot is technical interviews, which prompted me to write this post.

How to predict the future?

With the great power of assertiveness comes a great responsibility - the responsibility of being right. (More often than not, at least.) So how can someone get better at that? The best way I know to predict the future is to learn from the past - that’s the only formula that seems to work. (When AI takes over, I’m pretty sure they’ll tell us the same thing.) Exploring semiconductor history has been one of the most rewarding experiences of this journey, and posts like this will continue to be a big part of Chip Insights.

The other way to predict the future is to learn about great companies and their strategies. Patterns in business repeat themselves - success is about finding the right pattern. I captured my learnings from one of the most successful companies of our time in this post.

So, what’s the future of Chip Insights?

Posting an article (almost) every week for the last year has been a great experience - I learn a lot each time I do it, and hopefully could pass on some of my learnings in the process. But something that I have enjoyed even more than that has been the feeling that writing unlocks in me - every time I plan to write a new article, I want to reinvent myself and try something different. This feeling is powerful, and I have seen that it spreads across all aspects of my life - not just writing. But I am realizing that the best way to reinvent is not always to do it more often…

In chip design companies, a common practice used to innovate better is called the “tick-tock” model. Here, a “tick” project involves incremental improvements that can be completed quickly. But after a few ticks, companies pursue a “tock” project - a big structural change, usually dubbed as the next chip generation. In the last year, Chip Insights has seen a lot of ticks that I’ve highlighted in this post, but it’s time for a tock. There are few new ideas that I would like to pursue, which needs me to take a break from writing.

Whether this is the first post of mine you are reading, or you are that one subscriber who subscribed even before I wrote my first ever post, (that really happened, still don’t know why, but I’ll take it.) or maybe you came in somewhere along the way, thank you so much for showing that you care.

I urge you to stay subscribed, because the love for writing hasn’t left me. (And it never will.) I’ll be back soon with Chip Insights 2.0, and it will be worth your wait. Until then, I bid adieu.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

Most content in computer architecture courses, and also my newsletter, focus on how a computing chip “computes” - by exploring how the processor core, memory, I/O, and other parts of the chip work in isolation. But in reality, a large portion of time and energy consumed by computing chips is a result of the interactions between the different components of the chip. (called “nodes.”) For instance, moving data from memory and I/O to and from the processor core, or moving data between processor cores. For a long time, these have been assumed to be trivial. (and in reality, they were.) But with increasing number of cores and deeper memory hierarchies, efficiently moving data across a chip can make a huge difference to the overall chip performance. This post intends to introduce the concept to absolute beginners. (If you have been reading my newsletter long enough, you should sense an analogy coming…)

Welcome to your new venture - an airline company

The logistics of airlines has always fascinated me - so that’s the vehicle (pun intended) we are going to use to understand on-chip networks. Imagine this: you read all my posts, became a semiconductor legend, made a lot of money, and have now purchased two Airbus A320 planes, and started a new airline company. Your first task is to decide which routes your airline should operate between, and the logistics behind making this happen. Turns out, designing an on-chip network is surprisingly similar…

Going forward in this post, you can consider the following terms to be analogous:

• An airport is similar to a node in the chip (could be a processor core, memory, or I/O component)

• An airline route can be treated like a chip bus (bus here is not the transport kind, you can think of it as a collection of wires that can transfer high/low voltage.)

Model 1: Point to point connection

If you have two planes, what’s the simplest thing you can do? You can have two exclusive routes that can operate independently, and in parallel - say one route between Los Angeles (LAX) and San Francisco (SFO), and the other between LAX and New York. (JFK.)

Different nodes in a chip can also be connected in the same way - just have a bus connecting different components with each other, and transfer data through these buses. This is called a “bus-based architecture” and although it seems simple, it is one of the most commonly used networking architecture. (Remember, most airlines continue to operate all-year daily flights between two fixed locations - the logistics can’t be simpler than this.) However, this approach comes with it’s limitations, like:

1. Poor scalability

If you want to increase the number of nodes, you need to increase the number of buses. This makes placement and routing very challenging in the chip. (Do you really want to buy a new plane for each new destination you want to serve?)

2. Limited Flexibility

Having one bus assigned for each connection makes the design very inflexible: Say during execution, two nodes in your chip need to transfer more data than two other nodes, they both still have similar buses with the same bandwidth. It’s like operating an empty flight from LAX to JFK, even though the LAX-SFO route is overbooked!

As you might imagine, the flexibility problem isn’t too hard to solve - you own the airline, so you can be flexible with the routes. That’s the idea of arbitration.

Model 2: Arbitration

Let’s say you want to add a new destination for your Airline to serve - Miami. (MIA.) However, unlike SFO and JFK, travel to Miami is seasonal, peaking during the holidays. In this case, instead of buying a new plane to serve this route, wouldn’t it make more sense to redirect your flights based on the demand? You can “arbitrate” by:

• Operating flights from LAX to SFO and JFK during the week

• Changing one, or both of the routes to LAX-MIA during the weekends or major holidays

This idea of arbitration can also be used in chips to connect more nodes without adding new buses - by sharing existing buses. Arbitration can be done in many different ways (called arbitration policies) each with it’s pros and cons. While I’m not going to cover all of them in detail, here are some examples:

• Round robin arbitration: Every node gets a turn to use the bus, and then waits for all other nodes to be done before using the bus again.

• Daisy chain arbitration: All nodes can access the bus simultaneously, but there is a predefined priority order. (Something like this: If both SFO and MIA are in demand, always pick SFO)

• Collision-based arbitration: These are more advanced, and rely on real time chip behavior. (For example: If there are no bookings for next week’s flights to JFX, cancel them and open bookings to MIA.)

As you can guess, arbitration adds complexity to your chip. (In case of your airline company, you can think of this like hiring someone to manage the routes.) But in some cases, arbitration is a great way to reduce the number of buses in the chip - Say you have two cores connected to a memory, but your chip mostly runs tasks on one of the cores - then the bus between the two cores and the memory can be shared.) However, as your chip becomes complex, the shortcomings of arbitration become evident:

1. Bandwidth is limited

Remember, although we are able to serve three destinations now by arbitrating between two planes, we can still only serve two destinations at a time. Let’s say each flight can hold 150 passengers, the maximum number of unique passengers you can transport each day is 300 each way - which is the same as it would be if you operated as a point-to-point connection. In chip language, the maximum amount of data that can be moved from one node to another at a given time is called the bandwidth. Although arbitration will help manage multiple nodes, the bandwidth is still limited by the number of buses in a bus-based architecture.

2. Single point of failure

Arbitration has another issue. Say you have decided to fly one of your planes from LAX to SFO and back on a Friday, but from LAX to MIA on Saturday. (And the other plane is going to JFK both days.) However, thunderstorms hit SFO on Friday, grounding all flights. This means the flight won’t be back to LAX in time for the trip to MIA. So this flight becomes a single point of failure for two of your routes. While the weather does not affect electrons in your chip, failures of other types are possible - say you are using a round-robin arbiter, but one of your core runs into an infinite loop - that core does not release the bus and stops the entire chip from executing. That’s the risk with arbitration-based approaches.

Model 3: The crossbar, a.k.a hub and spoke model

If you have ever flown in one of the major international airlines like Emirates or Qatar airways, you would have experienced a layover at one of their “hubs” like Dubai or Doha. These airlines operate based on a very popular airline routing model called the hub and spoke model. (Inspired from it’s resemblance to your bicycle wheel.) Now that you are a seasoned manager of an airline, let’s take inspiration from this model and add a fourth destination to our airline - Seattle. (SEA.) However this time, the flight schedules are adjusted in such a way that LAX becomes the hub:

• Two flights leave each morning from SFO and SEA, respectively, and land in LAX at 11 AM

• The same two flights depart from LAX at 3 PM - one to JFK, and the other to MIA

With this model, the airline can sell tickets for the following routes:

• SFO-LAX, SEA-LAX, LAX-JFX, LAX-MIA with no layover

• SFO-JFK, SEA-JFK, SFO-MIA, SEA-MIA with a layover at LAX

As you can see, LAX is the key to making this happen - it’s where the “crossover” of incoming and outgoing passengers happens. In chip networking, we use a similar arrangement called a crossbar. A crossbar is a collection of programmable switches which can decide which inputs need to be routed to which outputs. (Your airline’s staff in LAX will do this check at the boarding gate.) This allows multiple different nodes in the chip to communicate with each other without the need for direct connections between all the nodes.

Crossbars effectively address both the shortcomings that we saw with arbitration. First, they are able to ensure that all nodes can communicate with each other simultaneously. (The crossbar is assumed to have an ability to monitor these transactions and handle them correctly.) In our airplane example with a hub, we are theoretically able to transport 600 unique passengers in a day. (300 in the morning from SEA/SFO to LAX, and 300 later to JFK/MIA.) Also, the presence of a crossbar means that each failure is restricted to it’s own node - if one of the cores in the chip hit’s an infinite loop, the other cores can still talk to the other nodes through the crossbar.

At first glance, you might think that a crossbar arrangement is very similar to a point to point connection since both cases ensure all nodes can be connected simultaneously. But there’s a subtle difference here - using a crossbar, we are able to achieve the same bandwidth as point-to-point connection, but with fewer bus connections. If a chip has N nodes, (N > 2.) then:

• Point-to-point connection would need (N-1)*(N-2) buses

• Crossbar connection needs N buses

This is the main advantage of a crossbar. However, in order to maintain the full bandwidth, crossbars usually need to store transactions and have complex control mechanisms, which makes the crossbar an additional piece of silicon taking up area and consuming power. Also, crossbars cannot scale indefinitely - each additional node adds some wire latency, and a very large number of nodes could make this intolerable. Remember, if you have 10 nodes in a crossbar, they all contribute to the wire latency - even if you only use a few nodes repeatedly. Transistor scaling also cannot be relied on - reduction in wire latency has not kept up with Moore’s law, which makes the future of crossbars very uncertain. Basically, as your airline business scales to multiple destinations, you need to hire a lot of staff, and reserve many boarding gates - even on days with low occupancy on your flights. Not ideal…

All of the approaches discussed so far were sufficient for the pre-SoC era: with a small core count, and few memory and I/O nodes that needed networking. But as we entered the era of parallel compute with SoCs having more than a thousand cores, interconnects needed to get more sophisticated. A “Southwest Airlines” moment was needed…

Model 4: The “Network”

In 1971, Southwest airlines launched their first flights in Texas - changing the aviation business model forever. Let’s learn from their approach and make our airlines more efficient.

Despite everything we discussed so far, the truth is, point-to-point non-stop flights are the most desirable options. So let’s go back to our first model, and see if we can take it in a different “direction”. (Pun intended.) I mentioned that one of the issues with point to point connections is the flexibility - how can you best manage different bandwidth requirements on each route? We looked at arbitration in model 2, but there is a more efficient approach - operate smaller flights. Smaller flights are cheaper to fly, and easier to sell-out, hence giving you more bang for buck on each route without the need to arbitrate.

The other limitation was the “buy a new flight for each new route” requirement that point-to-point connections require. Smaller flights will also partially address this issue, because you can afford to buy more flights. (and hence operate more routes simultaneously.) But as you expand your destinations, this might still become impractical - so we will remove this requirement from the model, making it pseudo-point-to-point. (i.e. not all connections can be point-to-point, as you will see.)

Let’s incorporate all this flexibility and build a network for our 5 destinations that looks like a ring. Much like the crossbar model, this topology connects all our destinations. (using as few as one flight!)

• Point-to-point connections: SEA-SFO, SFO-LAX, LAX-MIA, MIA-JFK, JFK-SEA

• Pseudo point-to-point connections (with 1 stop): SEA-LAX, SFO-MIA, LAX-JFK, MIA-SEA, JFK-SFO

Adding a destination to this model is trivial. Say you want to add a new destination, Dallas (DFW), in the same topology, you can add the node like this:

Unlike the crossbar architecture, which adds a strain (more staff, larger number of boarding gates, and so on.) on your “hub” for each new destination, this ring-style approach does not stress any single node. (Remember, in chip terms, this “stress” means wire latency.) However, you might already see a problem here - we have introduced connections with 2 stops which would never exist in a crossbar model. (For example, SFO-MIA.) If such 2-stop routes are undesirable, then a simple connection above can fix that - add a direct route between SFO and MIA.

With just one new route, we are able to maintain access all our destinations with one stop or lower. Also, you can pick between one large flight, or multiple small flights - the network can be adjusted accordingly. This is the biggest advantage of the network model - it allows the flexibility to modify the topology based on your resources and latency requirements. Such networks exist in all modern chips, and they are called, simply, a Network on Chip, or NoC. We will look at them more formally in the next section.

An overview of Network on Chip (NoC) architectures

In the previous section, I (hopefully) convinced you that an NoC architecture allows you to move data efficiently and provide maximum flexibility on chips with a large number of nodes. With great flexibility, comes great challenge - and in this section, I want to walk you through three architecture decisions that need to be made, and the trade-offs involved.

1. What should my network look like?

As you saw from the airline analogy, changing which nodes are connected to each other makes a huge difference to the path that needs to be taken to go from one node to the other. In NoC language, each such connection is called a “Link.” By changing the number of links, and the nodes they connect, we get different “Topologies.” We use two key metrics to understand how good a topology is:

• Path diversity: How many unique paths can you find between two nodes in your network (formally called “Bisectional Bandwidth,” which I think raises more questions than provides answers, so I won’t use that term.)

• Path distance: The maximum distance between any two nodes in the network. (Again, the formal term is “Diameter”; I’m not going to use that.)

As you might expect, adding additional links would increase path diversity, and reduce the path distance. But each link adds hardware complexity. (More wires, and as we will see later, complex routers.) Here’s an example with three simple topologies that highlights this trade-off.

There are more nuances in designing the best topology, but for now, all you need to take away from this section is: Pick a topology that provides sufficient path diversity with the smallest number of links, while also ensuring that your path distance isn’t too much.

2. How do I split up my data?

It might seem like ages ago, but if you remember from the start of this post, the purpose of on-chip networks is to move data from one point in a chip to another. Due to the complexity of NoC architectures, we have a dedicated unit in each node to manage the sending and receiving of this data - called a Router. So far in this post, we assumed that all the data that we need to send out is sent together. (As a single transaction.) But in reality, it’s not practical to do this - for instance, if you want to move 1 GB worth of data as a single transaction, you would need 8,589,934,592 wires (i.e. one for each bit.) in each link. Instead, it is broken down into multiple pieces. Here’s what they are called:

• Message: The is the full data that you want to send from one node to another

• Packet: Each message is broken down into multiple packets. All bits in a packet must follow the same path (or route) to reach the destination node.

• Flit (Flow Control Digit): Each packet has multiple flits. A router always stores all bits in a flit together.

• Phit (Physical Digit): Although a router stores a Flit together, it does not have to be sent out as a single transaction. Each flit can be broken down into multiple Phits, and each Phit must be sent out in a single cycle.

This figure highlights how a message is broken down and transmitted from a sender to a receiver node.

At this point, it might seem like we took a simple idea of moving data and added a lot of complexity to it. But each of these terms has an impact on the network performance.

• The size of the phit decides the number of bits transferred in each link. So the phit size decides the maximum bandwidth of our network. A higher phit size is desirable, but comes with challenges to place and route the large number of wires. (Remember our discussion about large vs small flights from earlier?)

• Flit size is the hardest to optimize. Since the router must store all bits of a flit together, a very large flit size could increase the latency. (Since all phits of a flit must be sent out before the next flit can be routed.) However, if your flit size is too small, then you may have too many flits to send - which could worsen the network congestion. So this needs careful consideration.

  • Think of it this way: To reach your a physical destination, would you rather take a train, which is infrequent, but fast, or drive your car, which you can do anytime, but carries with it a risk that you would be stuck in traffic?

• And finally, the packet size. A smaller packet size means more packets, which means more potential routes to take from the sender node to the receiver node. However, only a topology with sufficient path diversity can really benefit from this - if not, you are just adding to the congestion by having more packets.

I know I’m doing injustice to this topic with my “hand-wavy” analysis, but the point I really wanted to convey is that the way you split your message has a big role in the eventual performance of your network.

3. What path should I take to reach my destination?

Now that you know what your network looks like, and how your data is split, life’s great - you just need to send the data, right? Well, not quite. You also need to decide which route your packet will take to reach your destination. I’m not going to go into the details here, as this is an introductory post, but I’ll leave you with a few questions to think about:

• Do I need to know the full route to reach the receiver node even before my packet leaves the sender node? Or is it sufficient for me to just decide what’s the next best node?

• The agreement was that each packet must follow the same path. But does that mean I should wait for all flits in my packet to arrive at each intermediate node? Or can I send out each flit independently on the same path?

• What happens if two packets want to go to the same node? Which one should I prioritize?

This topic deserves its own post. (Maybe someday, if your airline business fails, we could explore a job as a traffic controller - a great analogy to understand routing algorithms.)

This is as far as we’ll go in this introduction to on-chip networks. In an era of parallel computing, moving data efficiently is as important, if not more important than the computation itself. My biggest learning from doing this research was that to build the best on-chip network architecture, the two extremes of the computing stack must come together:

• Algorithms will decide how your data is arranged

• The state of semiconductor physics will decide how many wires is too many wires

Irrespective of which side of the stack you are on, I hope this post inspires you to explore this topic further and build the best on-chip networks for the next generation of computers.

References:

• https://liacs.leidenuniv.nl/~stefanovtp/courses/ES/papers/Ch15_NoC_Design.pdf

• http://cva.stanford.edu/publications/2001/onchip_dac01.pdf

• https://srsarangi.github.io/files/advbook/Chapter_8-noc.pptx

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s) and are not investment advice.

I recently read “The soul of a new machine” by Tracy Kidder, a book about the design of a new computer called Eagle, undertaken during the 1970s by a small team within the company Data General. I have always liked this genre of books - it’s nice to read about people working together to build something special. I usually leave at the end feeling inspired. But at the end of this book, I felt something else: I felt grateful.

If you have been an engineer long enough, you would relate to this set of events: You find a bug during testing. You spend hours (maybe days) staring at the screen trying to figure out what is causing that bug - a period of intense frustration for you. (and sometimes, the people around you.) You finally find the root cause for the bug, experience a (short-lived) feeling of pride, before the next bug hits you. That’s the debugging emotional rollercoaster in a nutshell: And computer engineers go through it on a regular basis.

Like a lot of others, when I’m stuck debugging the same bug for a very long time, I feel like I have the worst job in the world. But looking back at chip design in the 1970s gave me a sense of how much better we all have it today…

Getting your hands dirty

In the era before HDLs and EDA tools, hardware engineering meant working on real hardware. The Eagle computer was not a monolithic piece of silicon like modern SoCs - it was a collection of integrated circuits, spread across 7 different “wire-wrapped boards” with wires in the back connecting different pins. As the author Tracy Kidder puts it, the boards looked like “thin plates, each with one side covered by a profusion of tiny wires. Small cables, flat like tapeworms, ran among the boards. Oh my, there were a lot of wires.” If those words don’t speak to you, here is a picture of one such board. (Source: Wikimedia Commons)

While in modern debugging, finding the root cause for a bug is the harder problem, a setup like this made fixing the bug equally challenging. When a change was needed, existing wires had to be carefully unwrapped using hollow-tipped tools, and if you were successfully able to do that, you then had to wrap new wires with the same precision. In this process, no other wires should be disturbed - but it does happen often, and results in new bugs. As the hardware lead Ed Rasala puts it, this process feels like one is performing an open-heart surgery.

This need to add or remove connections is not always a one-time thing: sometimes, in order to isolate where the bug originates from, different blocks in the design need to be disconnected, and later connected back. The book talks about an example involving the Instruction Processor and its associated I-cache - the engineers needed to run multiple experiments, each with one of the two blocks disconnected. (needing them to perform the “open-heart surgery” multiple times.) In modern debugging, isolating two such blocks would be as simple as modifying a few lines of RTL code.

Keeping the paper business alive

Today, when engineers say paperwork, we never actually work with paper. But in the world where Eagle was being designed, paper still had it’s place.

Part of the debugging process involved understanding how different blocks in the computer worked. This information was compiled into a two-hundred page physical document that was written by the architect Steve Wallach. This document was like the Eagle team’s Bible - in fact, it was divided into multiple chapters, and each chapter started with a famous quote. This all sounds great, until engineers had to search through them for a specific piece of information to help with their debug.

But this was not the worst part of paper. Debugging a wire-wrapped board is vastly different from the way we add signals to a waveform today - a device called Logic Analyzer was used to probe different pins, and a snapshot of the signal had to be noted down on paper. This was improved by automatically capturing some signals in the system console - but this still needed to be printed before the values could be analyzed. Remember, signals change many times each second - so a large number of values need to be printed. In describing the console prints for one of the debugs, the author writes: “Stretched out, the sheet would run across the room and back again several times. You could fit a fairly detailed description of American history from the Civil War to the present on it.” Analyzing this large data dump with no ability to search must have been a nightmare…

Once a fix for a bug has been found, all the console log papers can be cleared out - but the paperwork does not end there. In order to correctly make this fix in all the prototypes, the exact changes in connections had to be marked on a large diagram with the schematic of the computer. This was called an Engineering Change Order, or ECO. By looking at this ECO diagram, engineers working on different prototype had to modify the wiring in their prototype each morning before they started debugging other issues. Today, a change is only called ECO very late in the project, and is done on a software abstraction of wires called Netlist. Such ECOs are rare - not an everyday occurrence like in the Eagle project.

You can’t spell “Hardware” without “Hard”

Once the Eagle project was finalized, new engineers had to be hired to the team. When I read the hiring philosophy in the team, I couldn’t help but think that the challenges of debugging highlighted earlier had a lot to do with it.

During the interviews, Carl Alsing, the microcode lead, always presented the project as a tough one that involved long hours - at times even calling it a “suicide mission.” Those long hours were a direct result of the debugging challenges of the time. The team also screened for “a lack of family life” - they believed that someone with a family could not deliver on the intense commitment of the debugging schedule. Higher grades were valued, but it was not because it signified superior skills or smartness - it was merely seen as another indicator of hard work. In fact, the Eagle team even had a rite of initiation for their new hires, where they made a commitment to "do whatever was necessary for success, including forsaking, if needed, family, hobbies, and friends.”

While good old hard work still has it’s place in debugging today, the best engineers I have worked with rely more on a methodical approach that makes the most of tools available at their disposal. Today’s designers have the tools to prepare in advance for a majority of the debugs they might face - with faster and more accurate simulations, better logging capabilities, and approaches like formal verification. While Alsing and co. lacked these tools and had to push engineers to work 60-80 hour debugging workweeks, it is not a requirement to be successful at debugging anymore.

So, I get my turn to debug at 4 AM?

Let’s say you had all the smarts and energy it takes to debug any issue that your chip throws. If you walked into the Westborough building to debug Eagle, chances are, you will be asked to come back later. That’s because, unlike today’s world of simulation, directly debugging a real chip means, well, you need a real chip. But there were only a limited number of these prototypes, which meant not everyone could debug the Eagle computer at the same time.

The Eagle team were on a tight timeline. In order to maximize debugging time, they had a debugging schedule that had to be strictly followed, which assigned each engineer specific times throughout the day (and night) for debugging. So, if inspiration strikes you while having dinner, you would still need to wait for your shift to test your theory.

Although shifts were never more than 8 hours, the nature of debugging meant that most engineers stayed longer. Since debugging is a very personal activity, it is often easier to push yourself to work longer hours, than explaining your finding to the next engineer. (Which btw, was also “paperwork.”) The book talks about many such stories - Jim Guyer, for instance, spent many nights alone in the lab to debug issues in the Instruction Processor. Ken Holberger noted that it was often dark when he got in, and dark when he left work - leading him to lose track of the day and time when he was home.

Remember, adding more prototypes is not a sustainable solution - if you had 100 different prototypes of the computer, then each ECO would need to be implemented on each of these 100 prototypes. There were diminishing returns to scale in 1970s debugging.

Flakey and Bogeyman: The supervillains of the debugging world

In the story, "flakey" and "bogeyman" were terms used by the engineers working on Eagle to describe specific types of problems or fears encountered during the debugging of the Eagle computer. A flakey refers to a failure that occurs erratically and is often hard to diagnose. Some example of this include loose connections, stray voltages, or worst of all, an IC from another vendor is buggy. The main issue with a flakey is that it is hard to reproduce consistently. The first step in fixing something is getting it to break - and with a flakey, engineers needed to spend additional time reproducing the bug, before finding a way to fix it.

The experienced engineers in the Eagle team had seen a lot of flakeys in their life, and not all of them were caught before their computer was sold. This led to a fear that there will always be one bug that would go unnoticed, but cause their machine to stop working - they called this a bogeyman. The Eagle project lead Tom West defines it as "the infinite page fault you didn’t anticipate. The bogeyman is the space your mind can’t comprehend." The fear of the bogeyman was real - Ed Rasala talks about some nights where he would wake up worrying about a bug they are yet to find, feeling like “the bogeyman was in his bedroom.” Like I mentioned in another post, bugs continue to be seen in chips even today. However, verification has been largely left-shifted now - by starting verification very early, and at different levels of abstraction, the chances of encountering such bugs reduces greatly.

Long-term tiredness

Nobody likes to make mistakes. But the likelihood of not detecting a bug was much higher in the 1970s than it is today. This took it’s toll on the engineers involved.

Jim Veres, who helped design the Instruction Processor and its I-cache, felt annoyed by the fact that his component was blamed for a lot of bugs. Although he would often prove that the problem was elsewhere, he said that the constant blame placed a lot of pressure on him. He felt like the block he designed was a "part of him now," and he didn't like to see it picked on unfairly.

Jon Blau experienced "difficulty forming sentences," his "mind’ll go blank," and felt "pieces of your life get dribbled away" due to the internal pressure to hurry and finish his code. When he took over debugging the ALU, he was "terribly excited by it, then very frightened," leading him to take a week off. Ed Rasala summed this up nicely, by calling it a “long-term tiredness.” He said that engineers on the debug schedule felt tired, but not in a traditional way that going home to their loved ones could fix. They were always thinking about the current issue they are debugging, while being concerned about the next bug they might hit. That’s what it was like to debug a computer in the 1970s.

While debugging is still a challenge today, and tools can be better, reading “The soul of a new machine” gave me a new perspective on what we have today. It’s been about 50 years since this saga unfolded. Who knows, 50 years later, someone might write a sorry tale about the way we debug chips today…

By the way, this post only talks about a small part of this book that resonated with me. For a fuller analysis, I highly recommend this post by the Chip Letter.

The Chip Letter

The Soul of an Old Machine

"That fellow West is a good man in a storm." … "He didn't sleep for four nights! Four whole nights." And if that trip had been his idea of a vacation, where, the psychologist wanted to know, did he work…

Read more

2 years ago · 56 likes · 36 comments · Babbage

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s), and are not investment advice.

This post is a little different from what I usually write. I want to preface by saying that I’m not going to talk about the knowledge and skills needed to crack technical interviews - they are very important, but are also specific to the job role you are applying to, and the stage of your career. I compiled some resources for a few such roles in computer engineering in this post, if that’s what you are looking for:

When I had finalized my plans to move to the US for graduate school, I started to proactively apply for internships. Even before I landed here, a recruiter from what was at that time my “dream company” reached out to me, and set up interviews pretty soon. It was perfect - I had the skills and experience for the role, and I had prepared quite well. At least I thought so…

It’s been about 3 years since that interview, but I still remember that day very clearly. I completely froze during the interview. It wasn’t the hardest interview I have given. It wasn’t even the least prepared I was. And yet, it was my worst interview. Looking back, it hasn’t made too much of a difference, I’m happy that things panned out the way they did for me. But I’ve always wondered what happened on that day. Was it me? Was it a poorly conducted interview?

Recently, I conducted a few mock interviews for my alma mater. In preparation, I spoke to some experienced interviewers, and also reviewed some accounts of good and bad interviews. Being on the other side gave me more perspective on what makes an interview good - both for the interviewer and interviewee. Surprisingly, it’s less about how knowledgeable each side is - certain behavioral aspects play a key role. This post is centered around these aspects of a technical interview.

Who should read this?

I know that a lot of people reading this would be interviewees, not interviewers. As interviewees, we are conditioned to believe that the fate of an interview lies solely in our hands. I don’t think this is true - it’s just that interviewers aren’t evaluated to the same degree.

A lot of interviewers brag about the fact that most interviewees fail their round. I’ve found this to be quite amusing. Interviewers assume that their interviews are technically advanced and the quality of candidates is low, but that’s hardly ever the truth. If a lot of interviewees struggle to answer your questions, there is something wrong with your interviewing style.

I know this post would reach a lot of interviewees, but I’m writing this in the hope that it also reaches some interviewers.

The six ingredients that make a good interview

1. Perfect your pitch

Very often, interviewees start their interview by introducing themselves. When I was interviewing, I think I handled this question quite poorly - I had a good number of projects on my resume, and I thought the best idea is to make them the stars. I went into the technicalities of my projects very quickly - after all, it was a technical interview. But from my experiences being on the other side, I realize how pointless this is. As an interviewee, you need to remember that no one cares as much about your projects as you do - so the minor technical details that you find fascinating are not going to impress anyone.

Some of the best candidates I interviewed approached this question very differently - they had a simple, but impressive pitch that spoke about themselves. Say you are someone with projects in computer architecture, and interned in a government funded lab, here’s one example each for a bad, and good pitch:

• Bad pitch: Talk about the technical details in each of the projects (basically reading out your resume)

• Good pitch: Talk about the importance of computer architecture for the future of your country, and the role you want to play in that

It doesn’t matter if you think your experiences are ordinary - every story is unique, and crafting a good story can completely change the tone of the rest of the interview.

Interviewees usually figure this out over time, but a lot of interviewers forget that they need to pitch too. I can’t tell you the number of times this does not happen. I don’t think it is enough for interviewers to ask an open ended question like “do you have anything you want to ask me.” Interviewers should start the interview by setting expectations - about the interview, the job, and the company. I would go one step further to carefully review the candidate’s resume and mention what aspects would be a great fit for the team.

I have given so many interviews where I have no clue what is expected of me if I join the team, and whether they even care if I join. A short pitch can make a candidate feel special. Taking a few minutes to do this can make the difference between good candidates joining your team, or passing up your offer. Remember that good candidates will always have options, and their decision won’t always be driven by the money offered. Hiring is the most important aspect of a team, and as an interviewer you have an opportunity to attract the best talent.

2. Don’t experiment too much

When I was taking mock interviews, I experimented with the way I framed questions. (Part of me was already thinking about writing a post like this.) I realized that too much experimentation as an interviewer is not a good thing - it will result in an unfair evaluation. Me, and a lot of others I talk to, discuss how boring the interview format is, and how we would design a very interesting interview when we get the opportunity. But it’s important to understand why some interviews are the way they are: standardization is the best way to ensure fairness across candidates.

If I ask candidate A a standard textbook question, but I reframe the question to mimic a real life scenario when I interview candidate B - candidate B may have the more interesting interview experience, but the vagueness of the format will also make it unfair if they cannot answer the question correctly. So my suggestion is to avoid experimenting too much, unless you are sure you can replicate it across multiple candidates. It is important to remember this as an interviewee too. Answering technical questions is very different from the pitch I mentioned previously - here, uniqueness isn’t rewarded. Sometimes, I have tried to be too cheeky in interviews - using clever analogies, or out of the box terminology to answer basic questions. It has never ended well - in the worst case, I ended up confusing myself and the interviewer. In the best case, I answered one question right. (I don't think creativity gets any brownie points in technical interviews.)

Whether you are an interviewer or an interviewee, follow the advice that Michael Scott gave to Dwight Schrute in “The Office”: Keep It Simple, Stupid.

3. The curse of “the know it all”

This is a problem I noticed while interviewing a lot of smart, knowledgeable, well-prepared candidates. When I asked a question, they made a lot of assumptions about what I might be asking, and went on multiple different tangents. Everything they said was technically correct - the only issue was, it was not what I asked. If I have to evaluate the candidate for the question, I would still have to say that the question wasn’t answered correctly. Therein lies the problem.

Sometimes, interview questions are vague - could be by design, or could be accidental. Even if you know what the answer might be, asking a couple of clarifying questions before jumping in with an answer is a good idea. This could even earn you brownie points - if you actually remind the interviewer of a point they forgot to mention, it is a great sign.

I think the problem of oversharing exists with interviewers too (unfortunately, it goes unchecked.) Some interviewers get into the habit of explaining why the interviewee is wrong (and why they are right.) I know a lot of interviewers who justify this, by saying they want to help the candidate do better in other interviews. I don’t buy this argument; as someone that has been explained to multiple times, I can confidently say that none of it has gone into my head in that moment. Interviews are too short and stressful to be conducive for learning. All you are doing as an interviewer is wasting valuable time that could be spent evaluating other skills of the candidate. If you really care about the candidate learning something, offer to help the candidate on a call or email after the interviews. (I would gladly take up this opportunity to learn, but no one has made that offer - some companies have a policy against this - but let’s not get into this.)

4. Sweat the small stuff

Like a lot of interviews today, the mock interviews I took were remote too. Remote interviews introduce non-uniformity that is often ignored. This includes:

• A reliable Internet connection

• Clear audio and video

• The ability to write and draw something, and share it (preferably on the computer)

I know not everyone has the privilege to invest in the best equipment, but even those who do, sometimes don’t pay enough attention to it. I had a lot of such “tech issues” happen that completely disrupt the flow of the interview. Even if they aren’t completely disruptive, your interviewer cannot evaluate you if they cannot hear you correctly, or cannot understand what you write or draw. You might think these are minor things that don’t affect the eventual interview outcome, but when I asked experienced interviewers, they told me about some interviews where the lack of audio and video clarity resulted in candidates being passed on.

Those who have watched the 2013 movie “The Internship” can relate to what I am saying - don’t take an interview in a loud public setting like Nick and Billy:

Interviewers are guilty of this too - more often than you think. I remember interviews where after a point, I had no choice but to ask the interviewer to change their headphone because their audio kept breaking. If you are using visuals or text to ask questions, ensure that they are clearly visible. And finally, my biggest pet peeve - please, please, please turn on your video during interviews. There’s nothing worse than answering technical questions from an icon on the screen with scratchy audio.

5. Play the player, not the cards

There’s a saying about the best poker players - their strategy depends not on the cards they hold; it depends on the players they are up against. I think it would really help if interviewers and especially interviewees develop the ability to read the person they are talking to.

In my experience, there are three main types of technical interviewers:

• The no-nonsense type: is only focused on your technical knowledge

• The curious type: like to hear about new and interesting projects and ideas

• The manager type: is concerned about whether you are a good fit for the teams needs

If you can make an assessment of the type of interviewer you are talking to, you can shift the tone of the interview to match their expectations - give short but precise answers to the no-nonsense type; talk about interesting projects with the curious type; and show that you have the right skillset to the manager type. This isn’t the easiest skill to master, but will definitely help you leave a better impression.

This skill adds value for an interviewer too - especially if you want to lure a high quality candidate to join your team. Matching the energy of the person you are interviewing will definitely improve the quality of the interview, and provide better insights to make your decision.

6. Don’t be a jerk

I’ve saved the most important point for last. Interviewers and interviewees think of the technical interview as some kind of battle of intellect. In the process, one, or both parties can be jerks.

Interviewers are more notorious for this. I have seen that some interviewers have a tendency to demonstrate their intellectual superiority - in the form of snarky comments, exasperation, and concerning expressions. It could even be something simpler - asking questions like “Are you sure this is right?” when the answer is actually right. (In a vulnerable, emotional state, most interviewees will say “no” to a question like this.) Actions like these in no way contribute to the evaluation of the technical skills of a candidate - they are merely shows of dominance. Since most interviews span multiple rounds, it is possible that a candidate had a bad interview with you, but is still offered the job, and is going to be your future teammate. Do you really want your first interaction to be like this?

If you are an interviewer, always remember that you are the support cast in the interview - it is not your stage to demonstrate anything. Your role is merely to conduct a fair evaluation to help your team hire the right candidate.

In defense of the interviewers, I must say that interviewees can be jerks too. I have heard cases where the interviewer being wrong (which can happen) triggers unpleasant disagreements in the midst of an interview. Even if you are sure you are right, I don’t think having a full blown argument is the right thing to do. Often, candidates are evaluated by the number of questions they answer correctly, so the time you spend arguing is time lost from another question. The better approach would be to table the conversation - you can say that there might be a misunderstanding, and that you would like to visit this question again at the end if time permits. I would not even mind a polite email after the interview with the clarification. (But “polite” is very important here - otherwise it could backfire.)

It’s important to remember that both interviewers and interviewees have the same end goal - to contribute towards the progress of technology. If you stay with this goal long enough, you will cross paths once again. And humans have a tendency to remember negative emotional experiences - a simple technical interview should not be one of those.

So please smile, and have a pleasant technical interview experience.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s), and are not investment advice.

Hello there. If you are reading this, we have probably already interacted in the past. I was created a long time ago, inspired by a project management tool called Program Evaluation and Review Technique (PERT), that was used to identify the critical path and slack time in projects. I still carry some of these terms with me - from 1966 all the way till today. You may know me today by different names like Synopsys PrimeTime, or Cadence Tempus - does that ring a bell? No? It’s okay, I know I am not as popular as those pretentious synthesis tools that bully me around. But I play an important role in your life, and I’m here to tell you why…

The harsh truth your synthesis tool didn’t tell you

As you may have guessed, what I do is called “Static Timing Analysis”. The cool kids call it STA, so let’s stick to that. You know how you use a modern hardware description language like Verilog to describe your digital design? Well those languages might make your life easy, but tools like me can’t comprehend all the fancy slang you use there. Am I supposed to know what “always @ posedge” means? No thank you. All I understand are the basics:

• What are the inputs and outputs (Called “IO”)

• Simple logic components, like gates (AND, OR, etc.) and MUXes (Called “Cells”)

• Storage elements like Flip-Flops, Latches (Called “Registers”) and SRAMs (Called “Memories”)

• Connections between them (called “Nets”)

So before you come asking for my help, you need to convert your fancy HDL design into something I understand called a “Netlist”. (A list of nets, get it?) That’s where the synthesis tools come in - they are just fancy translators who understand your HDL design, and break it down into the simple netlist that I understand. They don’t check whether it is even possible to have the connections you have described - they just want to please you by saying yes to everything you say. No wonder you like them so much.

But let me tell you the harsh truth - not everything you describe is realistic. You might think you are the god of chip design, but in my world, you must bow down to our god - Physics. What you are designing is not software - a chip is a physical entity that works by moving electrons. They’re much faster than anything you’ve ever seen, but they still take time to move from one place to another in the chip. Due to this fact, any logic you add in the chip will also add a delay. Your favorite synthesis tool conveniently left all this out.

Finding out exactly what this delay would be is challenging - remember, at this stage, I am doing this with no knowledge about how the chip is going to look at the very end. But even at this stage, I have something valuable that no one else has - estimated delays of different cells for the technology that the chip would be manufactured in. (In other words, a direct line to Dr. Morris Chang - ever heard of him?) I write down this information in a place called the “Cell Library” - I’m going to need to refer to this often once I get started with my work. But I still haven’t told you what my work is, have I? Before I do that, I will tell you why my job exists in the first place.

Welcome to my synchronized world

Now that you know delays are a thing in the real world, let me introduce you to the idea of synchronization in digital design. Consider this design I once worked on, where there were 5 inputs - A, B, C, D, and E, used to get the final output O. When I saw the netlist, I noticed that this design includes an AND gate, two OR gates, and a MUX, connected like this:

As you can see, I have also added the delays of each cell by checking the cell library. (You’re welcome.) Assuming no input delays, the time taken to observe the correct output O can be obtained by adding all the delays in the path between an input and the output. However, there are multiple different paths from the input and output, each with a different delay:

• Path 1: Input A/B → U0 → U1 → U3 → Output O (Delay = 6 ns)

• Path 2: Input C → U1 → U3 → Output O (Delay = 4 ns)

• Path 3: Input C/D → U2 → U3 → Output O (Delay = 4 ns)

• Path 4: Input E → U3 → Output O (Delay = 3 ns)

The purpose of any digital design, is to find the output when the input changes. Since the delays are different, you don’t exactly know when the output is ready - it could be 3 ns, or 6 ns after the inputs are applied. This makes the design pretty much useless. How do we deal with this problem? In the above example, let’s say that the input can only change once every 10 ns, and the output is also sampled with the same frequency - then, irrespective of the path between input and output, we are guaranteed to get the correct output value. In other words, by restricting when inputs to a design can change, and outputs from a design are sampled, we can ensure correct execution across many different paths - an idea called synchronization.

In order to maintain synchronization, we need to ensure that the input remains unchanged. But we have no idea where these inputs are coming from, so each input may have it’s own delays. This means we need to have a way to store the values of the inputs periodically, and use these stored values to calculate the output. This is done using a storage element like a latch or flip-flop - the general term for this is a Register. Let’s add registers R0 and R1 at the input and output of our design, respectively.

Each register also needs to know when to store the next input. (every 10 ns in our example.) To achieve this, we introduce a new input, and an STA tool’s best friend, the clock signal. (also known by the nickname clk.) The clock signal periodically changes from 0 to 1, and then back to 0 - each such transition is called a clock cycle. The time taken for the signal to complete this transition is called the Clock Period. In a typical register, the input to the register gets stored in the register when the clock signal goes from 0 to 1, called the positive clock edge. This allows us to store the inputs in the register R0 during a positive clock edge, then complete the evaluation of the logic, and store the final output in register R1 during the next positive clock edge.

Here’s a timing diagram to show how in the above example, input E and the intermediate output O’ are only sampled at the positive clock edge, ensuring that the correct inputs and outputs are seen after every 10 ns. (Inputs A, B, C and D are not changing in this case.)

As you can see, E’ (output of register R0) and O (Output of register R1) do not change between clock cycles - this is important, because each of these signals can be used as part of other logic with the assumption that the data won’t change during the clock cycle - this is very important to make your designs larger without increasing the delays indefinitely. But this synchronization comes with strings attached, and my job is to ensure you abide by certain rules.

We have needs too, feat. Registers

When I spoke about the role of registers in maintaining synchronization earlier, I might have given you the impression that registers are a gift from the heavens to save all digital designs. Maybe you think registers should be the ones doing the talking here. Let me tell you something, these registers aren’t as generous as you think. While registers help solve the synchronization problem, they introduce new problems in the process.

Remember how I said a register stores data when it sees the positive clock edge - I left out some of the nuances here. There are some registers out there that decide to store data at the negative clock edge instead, but everything that I talk about applies to those registers too - so let’s generalize by calling it a clock event. For the input data to be stored correctly at a clock event, the register enforces some strict rules. (Thanks again to the physics gods.) In my world, we use specific terms to express these “needs.”

The first term is Setup Time, which is the minimum time for which the input data to a register must be stable before a clock event. I’m not going to tell you the complete origin story of setup time here, but just know this - if you change the input to a register too close to a clock event, the registers cannot guarantee that they checked the correct value. Think of it like this - you are supposed to turn in your assignment by 10 AM, but you decide to make some last minute updates at 11 AM - a register is like that strict professor who says: “Sorry, I cannot promise that your updates would be graded.” Ah, if only registers were a good sport…

Their needs don’t end here - in addition to setup time, registers also have something called Hold Time - the minimum time for which the input data should be held stable after the clock event. This might seem excessive to you - this sounds like a professor who not only wants the assignment submitted by 10 AM, but also wants you to show up to their office, wait for 30 minutes while they review your assignment, and leave only after that. Seems unfair? I get it, but registers rule my world, and they are pretty high maintenance. You have no choice but to abide by their rules.

If you think our nightmare ends there, I’m sorry to burst your bubble. Remember that these registers, like everything else in a chip, work by moving electrons around. That means just like the cells we discussed earlier, they have propagation delays too. Typically, there are two different delays, since the path taken by the clock signal and input data are different. Here’s what we call them:

• Clk→Q Delay: The delay in output resulting from the clock input

• D → Q Delay: The delay in output resulting from the data input

Lot of information, huh? Looks like somebody needs help tracking all this. That’s where I come in. My job is to ensure that all the delays are accounted for, and the registers can guarantee that your design works as expected. The first thing I do, is call up my dear friend Dr. Morris Chang (or one of his friends that is going to manufacture this chip.) and get the setup time, hold time, and propagation delays of the registers. Remember, I already know the propagation delays of the other cells. With all this information, I’m like a fortune teller that can predict the future of your chip - except that unlike your fortune teller, I’m always right.

Show me the slack!

For me to check if your chip will work correctly, I need to ensure that your design meets all of the register’s needs. Let’s start by looking at setup time. Use the diagram below to follow along. To start with, I select a register, say R0 from the design, and pick one of it’s inputs - let’s call it In. At a positive clock edge, R0 stores the data from In. (When I do this analysis, I assume the input register, R0 in this case, does not violate Setup Time. This analysis is done for all registers, so trust me, it will all work out in the end.) Due to the propagation delay in R0, the stored data In’ is only seen after some delay. As I mentioned, a register can have two kinds of delays - but when I’m checking if setup time is met, I pick the larger of the two - because I want your chip to work under all circumstances. Once we get past R0, we see a familiar foe - the combinational delay which I spoke about in an earlier section. Adding both these delays gives us the time it takes to get the correct data for Out’ - the input to register R1. The register R1 expects to store this data at the next positive clock edge - it’s schedule is decided by the clock period. But as I mentioned earlier, the register does not like last minute changes to it’s input - so our real deadline is actually a little earlier than the next positive edge - to ensure that the data is held stable longer than the setup time of R1. This requirement can be expressed using fancy math operators, and this is called the Setup Time Constraint. My job is to check whether this constraint is met:

• If YES, that means this path between In and Out “meets setup time”. The extra time you have before the setup time deadline is called Positive Slack.

• If NO, the path “violates setup time.” The amount by which your delays exceed the deadline is called Negative Slack.

Not so fast…

As I said earlier, registers also need data to be held for some time (called hold time) before it is safe for the input value to change. In the same scenario I described earlier for setup time, when the data from Out’ is being stored in register R1, a new value of In is being stored in R0, and then passing through the combinational logic to replace the current value at Out’. If this happens too fast, then the value of Out’ changes before the hold time of register R1 has passed - which would result in a Hold Time Violation. This gives me another constraint to check for - called the Hold Time Constraint. (You can think of it like a speeding ticket.) Similar to the setup time check, my goal is to ensure that the chip works at all circumstances - so when I am looking for a hold time violation, I take the smaller of the two propagation delays of register R0. Remember, smaller delays are bad when it comes to hold time.

Showing (off) my work

Now that you know what I do, let me show you how I would actually do it, using the example design I described earlier. In this design, although there are just two registers, there are six different paths between inputs and outputs. (Four of them have unique combinational delays.) For each of these paths, I need to ensure that the setup and hold time constraints are met. Dr. Morris Chang was busy, so I made up some numbers for the register’s setup time , hold time , and propagation delays. With these numbers:

• Path 2 and 3 meet both setup and hold time conditions - they are good to go

• Path 1 fails setup time - it takes 8 ns for data to reach O’, but R1 expects it at 7 ns

• Path 4 fails hold time - The path from E to O’ is just 4 ns long, giving R1 insufficient time to store the previous input

This analysis (i.e. Static Timing Analysis) clearly shows that this design does not pass timing. It also tells you exactly which paths are causing violations, and by how much. I have to repeat this analysis for every combination of input and output registers, and every possible path between the inputs and outputs. Even if one out of a million such paths has a setup or hold violation, I will find that for you, and prevent you from making a billion dollar mistake. That’s what makes me special…

Still not convinced of my value in your life? That’s okay, I have some more experiences to share from the real world that might just convince you. So share this story, and stay tuned for future installments of this series.

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s), and are not investment advice.

Story time

In 1989, a chip design team started work on one of the most ambitious PC processors of the time - one with a superscalar architecture, branch prediction, and a much faster floating point unit. Over the next two years, hundreds of engineers were involved in the design, simulation, and extensive verification, which culminated in a test chip. The testing continued - by running major operating systems with the most popular applications of the time - for an estimated 100 million clock cycles. After all the bugs were resolved in 1993, the final chip was shipped to all the biggest computer OEMs of the time - with a new brand name called “Pentium”. The chip was an instant hit - the reviews were positive, and as a result, Intel’s microprocessor sales doubled in 1993. The new floating point unit impressed - with a 10x improvement over the previous generation of Intel chips. It was another feather in Intel’s cap - extending their dominance in the PC processor market. It seemed like they had done everything right, and nothing could go wrong…

The Pentium chip had millions of users over the years, but one of them has a special place in history. Dr. Thomas Nicely and Intel couldn’t be further apart - he was a Mathematics professor at Lynchburg College in Virginia, away from the hustle and loud marketing that Silicon Valley was known for. About 18 months after the release of the Pentium processor, Dr. Nicely noticed something - the results from his research on twin prime numbers were slightly off. Like a typical academic would, instead of ignoring it as a “random error”, he started to dig deeper - which led him to the conclusion that the Pentium processor was making mistakes when running some floating point division operations. He posted his findings online, and this opened up a can of worms that would cost Intel dearly.

Triaging the bug

Remember the long division method from high school? Microprocessors in the 1990s used a similar method to divide floating point numbers. Although it worked, this method was quite slow - making floating point division one of the slowest arithmetic operations in a microprocessor. To make this operation execute faster, Intel implemented a different approach called “Sweeney-Robertson-Tocher (SRT) method”. I’ll skip the details here, but for the purposes of this post, all you need to know is that in order to implement SRT in hardware, a lookup table was needed to provide the quotient digit based on different input combinations. It was in this lookup table that the now famous Pentium FDIV bug originated.

In order to implement this lookup table, the Pentium team used a Programmable Logic Array, or PLA. A PLA is a type of Read-Only Memory (ROM), but can be used to store structured data more efficiently in a smaller area - by representing the data as a logic function. In the 1990s, memory was still very expensive, which made PLAs an attractive alternative. The word “programmable” in PLA could be misleading in today’s context - programming a PLA means “fusing”, or allowing current to flow through certain transistors to get the desired logic. This was done permanently, before shipping the chip out to the customer. This transistor fuse mapping (i.e. which connections exist) is very important in deciding what the final output would be. For example, the figure below shows a 3 input, 2 row, 2 output PLA, where different programming (Blue and Green dots) result in different output logic.

To implement the SRT division method, the Pentium team used a 22 input, 120 row, 2 output PLA - which means there are 2880 potential transistors to fuse. (Of these, only 2048 values were used.) Unfortunately for Intel, 5 of these were programmed incorrectly. Whenever these incorrect values were picked up from the lookup table, the result of the floating point division was incorrect. This was the cause for the FDIV bug in Pentium.

How did this bug go unnoticed?

Looking back, this sounds like a trivial issue that should have been caught much earlier in the testing process. But remember, we are talking about the best microprocessor company of the time - which begs the question - how did they miss this bug? There are a few theories floating around:

• Intel’s whitepaper claims this was a clerical error - the C script written by an engineer to load the final transistor mapping into the PLA had an error, which resulted in 5 entries not being loaded in the PLA. The values were tested before loading to the PLA, but the values in the PLA were not checked.

• Robert Colwell, architect of the Pentium Pro, mentions in his book "The Pentium Chronicles” that this error was caused by a last minute request (actually, order) from management to shrink the size of the PLA, which made the engineers pursue an optimization that was not properly verified.

• Some postmortem studies claim that the engineers misunderstood the SRT method, and applied the wrong rule for lowering thresholds - which means this was not an error in any step in the chip design process - the table was mathematically wrong in the first place

• Aliens manipulated the final chip layout in order to scale back human progress (I’m not kidding. Intel made a movie about this, called “Intel: The Journey Inside” with this plot.)

Irrespective of the reason, it’s interesting that the bug went unnoticed for close to 2 years after the chip was released. (Although Intel claims they were aware of the problem in the summer of 1994, months before Dr. Nicely made it public.) This is because, the odds of hitting this bug were extremely low:

• The division must access the 5 incorrect values out of the 2048 in the PLA, a 0.24% chance

• Intel claimed a typical user would encounter this problem once every 27,000 years. Another way of putting this was that an error would take place once for every 9 billion random divisions.

• The error typically shows up in the 9th or 10th decimal digit (at worst, the 4th decimal digit.) There are very few applications that require such high precisions

Although a lot of people claimed to have been impacted by this, Dr. Nicely was the only person who noticed the bug in regular use. (All other scenarios seem to be artificially designed to hit the bug.) When it comes to bugs, Intel won the lottery. (if winning the lottery means creating a bug that is being talked about 30 years later.)

Consequences

Although the bug was clearly not great news for Intel. But in a way, they got lucky - the odds of this bug having any real impact was ridiculously low. (If we go by Intel’s numbers, the odds are lower than that of plane crashes.) Yet, Intel did not get away with this.

Intel’s early response was to shrug it off as an unlikely scenario, which led to a lot of flak. News about Dr. Nicely’s findings spread like wildfire, and even CNN ran a segment on the issue. IBM, who were simultaneously Intel’s biggest customer and competitor (with the Power PC) claimed that the issue was more common than Intel estimated. AMD ran an advertisement that said their chips “Can actually handle the rigors of complex calculations like division.” - a clear dig at Intel’s bug.

Ultimately, Intel had to issue a public apology and offered a free replacement to anyone with a Pentium processor that reached out to them. This was estimated to have costed Intel about $500 million - and a lot of damage to their reputation.

This is the impact that even a seemingly harmless chip design bug can have.

The most important law in chip design…

This brings me to the title of this post, and the reason why I narrated the story of Intel’s famous FDIV bug. This is not just a story about Intel’s mistakes. Every major chip company in existence has had a similar story to this, and I’m sure every chip company in the future will. There is a lesson to learn from stories like this.

It is in the nature of chip design and computing that even the unlikeliest outcomes can occur, and should therefore be accounted for. Over the years, architectures have changed, workloads have changed, but this fact continues to remain true. Hence, the most important law in chip design is actually Murphy’s law: If something can go wrong, it will go wrong.

We talk a lot about transistor nodes and chip performance, but it is important to remember that the first priority will always be: build a chip without bugs. Maintaining such a high standards at the scale of trillions of transistors, each a few nanometers wide, and switching more than a billion times per second, is what makes the chip design industry truly unique.

References

• Few facts about the Pentium processor

  • https://en.wikipedia.org/wiki/Pentium_(original)

  • https://www.tomshardware.com/picturestory/710-history-of-intel-cpus.html

• Dr. Thomas Nicely notes on his FDIV bug discovery

  • https://faculty.lynchburg.edu/~nicely/pentbug/pentbug.html

• About the SRT division method

  • https://www.osti.gov/biblio/4157138

• A great deep dive into the bug

  • https://www.righto.com/2024/12/this-die-photo-of-pentium-shows.html

• Explanations for the bug

  • https://web.archive.org/web/20060209005434/http://www.byte.com/art/9503/sec13/art1.htm

  • https://link.springer.com/chapter/10.1007/3-540-59293-8_189

  • https://ieeexplore.ieee.org/book/5989703

  • https://oldbytes.space/@kenshirriff/113606898880486330

• Media coverage after the bug became mainstream

  • AMD’s 1994 advertisement

  • https://static.righto.com/images/pentium-fdiv/apology.jpg

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s), and are not investment advice.

Over the last year, I have spent many hours following stories about Nvidia and their meteoric rise. This post is an attempt to consolidate all those learnings. My goal is to look back at Nvidia’s history to try and generalize the principles behind making a new computing architecture succeed.

A few caveats before I get into the post:

• This is not a post about business strategy. There are many good resources covering that already. I have tried to stay away from generic advice like “to be successful, build the picks and shovels in a gold rush”.

• This post is not forward looking - I’m not claiming that Nvidia has won and everyone should invest in their stock. It is simply a case study in computer architecture. In fact, a lot of my learnings come from the early days of Nvidia

With that out of the way, let me share what I think is key to successfully build a new architecture.

Pick the right domain

Architectural changes cannot happen overnight. So its very important to pick the right target domain. There are a few ideal characteristics:

• It’s barely possible to execute applications with current architectures.

  This means at least one the following factors:

  • It takes too long to run

  • It needs very expensive compute resources

  • It needs advanced programming skills that only few possess

  When Nvidia was founded in 1993, they picked computer graphics. For graphics to be realistic, it needs fast compute. In the 1990s, 3D graphics was only possible using either:

  • Expensive workstations from companies like SGI

  • Advanced software rendering techniques in games like Doom

  These were both not accessible to the average consumer and developer. In addition, 1992 had two key developments that could enable graphics cards:

  • The PCI bus protocol was adopted as a standard by all computer manufacturer. This means anyone could make a new chip, and attach it with any computer, and it could work

  • Microsoft released Windows 3.1, which was a big jump in operating systems graphics

  All this made computer graphics a great domain for Nvidia to try and disrupt.

• There are general patterns in the workloads that can be exploited

  I think the word “general” is key. Let me explain with the example of graphics.

  Nvidia recognized, over time, that the key pattern in computer graphics is this: Computations on each pixel is independent of the other. No other pattern is as important. This led them towards a massively parallel architecture that is still programmable.

  There were architectures inspired by other approaches too. For example, early graphics chips focused heavily on fixed-function accelerators. Here, the idea was to pick a specific part of the graphics pipeline (like Tessellation, Rasterization, Shading), and make your architecture optimal for these steps. In fact, Nvidia also had a fixed function approach for a long time.

  There are two problem with going narrow with architecture decisions

  • Algorithms in your domain can change. Your architecture may not optimally support those changes

  • Your architecture can never scale to other similar domains.

  If you cannot find a general pattern in your domain, it is a bad idea to build an architecture around this domain. Often, such niche domains get commoditized as part of a bigger architecture - this is what Intel did for audio cards, and even 3D graphics to an extent over the years. But Nvidia survived this because of their architecture was sufficiently general by that point.

• Solving for this domain should provide big opportunities

  This is what Nvidia calls a “zero billion dollar” market. Its hard to specify exactly which domains would fall in this category, but a good approach I suggest is to ask this question: “If you build it, and they come, what can they do?”

  When Nvidia first bet the company on computer graphics, their vision was: If graphics could be made accessible to everyone, it would be the greatest storytelling medium. This was around the time when Jurassic Park was released, and power of computer graphics was becoming evident.

  I also found something else that was interesting in my research - Nvidia had AI applications added to their GTC lineup as early as 2010. Although AI was still in its nascent stages, they were thinking about potential applications like computer vision, speech recognition, and robotics - the promise of a world powered by AI was massive, prompting Nvidia to pursue that domain aggressively.

  Establishing an architecture needs huge time investment - so it is very important that your target domain has huge potential.

Your architecture should fit your target workloads - not the other way around

If you want to build a standardized architecture, that means that other developers will be building software for your architecture. This presents a major challenge to architecture companies. Usually, when there is an change in algorithms or APIs that your architecture support, architecture companies are presented with two options:

1. Build an translation layer through your drivers, or compiler, to support the new workloads on the same architecture

2. Modify your architecture to better suit the new workloads

The first approach is often easier, and can be done on-the-fly (in the same chip generation). So this is the preferred method. However, a lot of chip companies ignore the second approach completely - if it works, why fix it? There are two stories from Nvidia about this which are interesting to study.

In 2003, Nvidia had established itself as the leader in computer graphics, and was preparing to launch its next chip, NV30. Around the same time, Microsoft had just released the new Direct3D 9 API. NV30 did not fully support Direct3D 9 - so they created software workarounds to support the new software features. This turned out to be a disaster - the NV30 received bad reviews, had heating problems. This was a major learning for Nvidia.

Over the next few years, as Nvidia’s programmable shaders started to gain popularity, many researchers started using them for scientific computing. During the early days, the same GeForce cards used for graphics were being repurposed for scientific computing through some cumbersome programming. Although this was better than anything available at the time, it was still not optimal - much of the GPU architecture was still optimized towards graphics applications.

This time though, Nvidia saw the trend early - instead of keeping the same chip architecture for graphics and scientific computing, they started to build chips that were specifically meant for non-graphics applications, by maximizing the number of parallel floating points computations supported. In 2016, they also introduced FP16 support on their GPUs - because most Deep Learning workloads used FP16. These factors went on to play a massive role in their dominance in the data-center - which is mostly comprised of AI workloads.

Have a unified and backward compatible architecture.

Two basic rules of life are: 1) Change is inevitable. 2) Everybody resists change.

To build a standardized architecture, making it backward compatible is key. Having a unified architecture naturally creates a flywheel:

• All the libraries developed over the years work on your architecture

• Developers are incentivized to write more libraries for your architecture

Getting developers into a new architecture is key - that’s what Intel was able to do in PCs, ARM in mobile (PowerPC and DEC Alpha are examples where this did not happen). A unified architecture is very “developer friendly” - a community (ideally, open-source) starts to build around your architecture. This is exactly what Nvidia achieved with CUDA - since 2006, all Nvidia GPUs are CUDA compatible, giving CUDA developers an install base of about 500 million devices, with about 300 libraries and 600 AI models.

CUDA is Nvidia’s moat, and a lot had been said about this already. For the purposes on this post, I want to talk specifically about some of the challenges with maintaining backward compatibility:

• Clunky hardware: To support legacy architecture features, often, additional hardware complexity needs to be maintained. This will result in poorer Power, Performance, Area (PPA) metrics over time

• Restricting Innovation: Very often, something new and better cannot be implemented in your architecture because it breaks some legacy constraints.

• Expensive Development: As the number of architectural features increases, it needs a bigger workforce, and more knowledge transfer - which costs time and money.

I think over time, every architecture company gets hamstrung by their legacy architectural features - A case in point is Intel, and the x86 architecture (I have a earlier post on ISAs with more details). This makes maintaining backward compatibility one of the most challenging aspects of computer architecture.

So far, Nvidia has managed this well, primarily owing to these factors:

• Nvidia’s developers operate at a very high level of abstraction. CUDA by itself is a fairly high-level language (like C). Also, most developers build using CUDA libraries like CuBLAS and CuDNN, which are optimized for Nvidia’s architectures. This gives Nvidia more opportunities to improve their architecture while maintaining backward compatibility. In other words, they can maintain backward compatibility at the developer level, but break compatibility at the microarchitectural level - bridged using their driver and compiler. This differentiates them from Intel, who for the most part were dependent on Windows developers doing a good job at using their architecture.

• Their architecture is still fairly general and simple - it is centered around parallel programming, and floating point computations. A lot of issues in Intel’s architecture stemmed from complexities added over the years - like variable length instructions and complex arithmetic.

So far, Nvidia has navigated the backward compatibility challenge well - but their workloads are fairly new, so it will be interesting to see how this continues in the long run.

Build infrastructure to move faster

The nature of workloads keep changing all the time. One of the most impressive aspects of Nvidia is their ability to pivot when an opportunity present itself. They started as a graphics card, pivoted to programmable graphics, and then expanded into high performance computing. Nvidia was able to grab opportunities better than anyone else, because they were able to make big architectural changes quickly.

From early days, Nvidia was a strong believer in simulation and automation. I think it stems from Jensen’s early days at LSI logic, the company that pioneered many EDA innovations. (I have covered many such stories in my EDA series). During his time at LSI, Jensen worked on a new chip architecture called “sea-of-gates” - which was a very early version of an FPGA emulator. Emulation would go on to play a big role in the development of RIVA128 - widely regarded as the chip that saved Nvidia.

Traditionally, once the chip was designed, it was sent out to the fab, and a test chip was sent back. This test chip was used to run software, find bugs, and resolve them. There are multiple iterations of this, until all bugs are resolved. Then, a final, large order of chips gets “taped out”. This whole process usually took 2 years.

Nvidia first two chips - NV1 and NV2, were poorly received. Nvidia needed to make major architectural changes in a very short time - which forced them to adopt emulation. Using emulation, once the design is ready, it is loaded on to an “Emulator”, which is then used to test software on the chip prototype even before it was manufactured. To do this, Nvidia went to a failing company called Ikos, invested heavily in their emulators (each one cost $1 million!), and essentially bet the company on emulation. It was a cumbersome process, but it worked - RIVA128 made one of biggest leap in computer graphics, and made Nvidia the leaders in computer graphics.

Although the RIVA128 story was born out of desperation, moving fast then became a part of Nvidia’s culture. Nvidia continues to invest heavily on infrastructure to help them innovate faster than the competition. Nvidia also incorporated a feature called “virtualized objects” into their architecture. In simple words, this was a mini-OS baked into the hardware (called “Resource Manager”) that could be used to emulate certain late hardware features that could not make it in time for the chip production. Although it incurred a minor performance cost, Nvidia adopted this because they greatly valued the ability to move quickly.

Generally at Nvidia, this obsession with efficiency is referred to as the “speed of light” approach, which says: every project must be executed at the fastest possible rate, and all obstacles in the process should be removed. This is an underrated aspect of chip design that a lot of companies neglect, making them slower, and less receptive to new opportunities.

Understand bottlenecks outside your core architecture

If building a computing platform is like building a car - the architecture is like the engine. Even if you have built the fastest engine, your car can’t go fast if you have weak tires, or there is bumper-to-bumper traffic on the road. Although your architecture is just one part of the computing stack, all the end user cares about is: how fast is my workload running. So its very important to analyze where the bottlenecks are, and work on managing them better.

One fundamental bottleneck every architect must think about is transistor scaling - i.e. how is Moore’s law progressing at the time your are building your architecture. (I covered Moore’s law in more detail in an earlier post). Nvidia learnt this the hard way.

Nvidia’s first graphics card, the NV1, was designed to render 3D surfaces in 2D using quadrilaterals (4 vertices) instead of triangles (3 vertices). Nvidia’s had a good reason to do this - memory costs were very high, and using quadrilaterals would allow them to build their chip with a smaller memory, allowing them to price their graphics card competitively. However, what they didn’t see was that Moore’s law started to accelerate around the same time, making memory much cheaper. The software ecosystem at the time standardized on triangles, and their competitors were able to support this using larger memory at comparable prices. Despite having one of the best architecture, Nvidia could not remain competitive because they had a small memory.

Many years later, in the datacenter business, Nvidia knew that their architecture alone cannot push them to the top. Not a lot of people know this, but Nvidia was in the cloud business very early - in 2013, they released Nvidia Grid, an early version of their cloud gaming platform, known today as GeForce Now. There is another fact that is not well known - Nvidia had an LLM of their own, much before ChatGPT became popular. In 2019, they built an open-source LLM called Megatron.

Both these experiences taught Nvidia about all the components needed to make the best datacenter servers. As a result, Nvidia has expanded well beyond their GPU architecture in datacenters, to offer:

• Very high bandwidth memory using CoWoS 2.5D stacking (very important for LLMs)

• Faster networking between cards through NVLink, enabled by their acquisition of Mellanox

• A very efficient ARM-based CPU for the datacenter (in fact, they tried to acquire ARM)

This gave them control of a bigger portion of the computing stack, and allowed them to optimize it further. Vertical integration as an architecture company also buys you time - even if Nvidia GPU architecture is not the best for a generation, the full solution they provide might still be better than anything on offer. Finally, a full solution is much easier for enterprises to deploy.

Although most architecture companies cannot start here, it’s very important to move towards that direction as your company evolves. This is how you can move from a great technology to a great product.

References:

• The Nvidia Way, a book about Nvidia’s story by Tae Kim

• Nvidia episodes from Acquired

  • Nvidia Part 1: The GPU Company (1993-2006)

  • Nvidia Part 2: The Machine Learning Company (2006-2022)

  • Nvidia Part 3: The Dawn of the AI Era (2022-2023)

  • Nvidia CEO Jensen Huang

• NVIDIA's Jensen Huang on AI Chip Design, Scaling Data Centers, and his 10-Year Bets

• Ep17. Welcome Jensen Huang | BG2 w/ Bill Gurley & Brad Gerstner

• NVIDIA CEO Jensen Huang's Vision for the Future

Disclaimer: Opinions shared in this, and all my posts are mine, and mine alone. They do not reflect the views of my employer(s), and are not investment advice.

A few years back, when I was talking to a student that was interested in both the front-end and back-end stages in chip design, I made a cheeky remark that they should become a “Full Stack Chip Designer”. (I can’t remember who I was talking to, but if you are reading this, this post is dedicated to you!) It was a term I took from software engineering - a full stack engineer is someone that has the skillsets to work on both the front-end and back-end aspects of a software application. When I said “full-stack” in the chip design context, I just meant it as a joke - there are very few people that actively work on all the steps in the RTL-to-GDS chip design flow, especially when designing complex, real-world chips. I must admit I have never done full stack software engineering, but from what I gather, the skills are more transferrable between front-end and back-end software design, compared to chip design. But with AI in the picture, something tells me we might be heading towards the era of full-stack chip designers.

What I mean by “Full-Stack Chip Designer”

I briefly introduced the idea earlier, but let me be more specific here. The typical digital chip design process can be divided into two broad categories:

• Front-End, which involves designing microarchitecture specifications and describing the logic using Hardware Description Languages (HDLs) like Verilog. (the resulting “code” is called Register Transfer Language, or RTL.)

• Back-End, which involves taking the RTL, and converting it into a layout (called GDS) that a foundry can use to manufacture the chip. This is a multi-step process that uses EDA tools from companies like Cadence and Synopsys.

Typically, teams managing these two aspects work in isolation to each other - the front-end team finalizes the RTL, then hands it off to the back-end team to generate the GDS. There are a few “leads” involved in the back-and-forth between the teams, but their role is to pass feedback to each other’s teams. The leads have a good understanding of both front-end and back-ends, but typically do not directly work on implementing in either. On the contrary, when I refer to a full-stack chip designer, I mean an individual (or a small team) that can take a design idea through from front-end to back-end - fully implementing all the steps in the process.

Why is Full-Stack Chip Design hard?

There are a lot of smart people in chip design - so if I could think of this idea, you can be sure that a lot of people have thought of it. In fact, a lot of early chip designers were truly full-stack - they worked on all the steps themselves. I can even say that if you have ever taped-out a chip for a project at your university, you are technically a full stack chip designer too. So it’s clearly not an impossible problem to solve - it is just very hard to solve at the scale of today’s commercial chips with billions of transistors. There are two main reasons for this:

1. Back-end tools aren’t intuitive; Neither are HDLs

I think this is a well documented problem. (I have also talked about it in one of my earlier posts.) Due to a large design space, EDA tools are complex to use, and produce reports filled with convoluted terminology. As a result, it is not straightforward for a front-end designer to dive into back-end tools. To make matters worse, bugs in the back-end design process are very hard to spot, and if left unattended, could result in a chip that does not function (a chip re-spin is a billion dollar affair, so no one wants that.)

I also think writing good RTL is an art that takes time to master - unlike high level languages, HDLs are less intuitive, which makes it harder for back-end experts to pick up. All this has meant that back-end designers spend all their time training to be experts in the tools they are working with, while front-end engineers prefer to do what they do best.

2. Front-end and Back-end design happens at different granularities

This aspect is more nuanced and is not seen in small chips or university projects. When working on industry-scale chips, front-end and back-end designers have different priorities. Front-end designers start by becoming experts on a small part of the design. In order to make this happen, all other parts of the design need to be treated like black-boxes. However, for back-end design to be effective, it needs to happen at a bigger granularity - if a chip has two blocks, but the back-end is handled for each block separately, the interactions between the blocks (signals routed across, how often they access hard macros, and so on) would not be captured well - which would result in a low-quality layout. Hence, the goals of a back-end designer are quite different - it is to gain a basic understanding of all parts of the design, by sacrificing depth in a single part.

Essentially, when it comes to understand the chip, a front-end designer should be a specialist, while a back-end designer should be a generalist. Getting an individual (or team) to do both is a challenge.

What do you gain by being a Full-Stack Chip Designer?

Short answer: Time.

Although front-end and back-end work in isolation, they are working towards the same goal - to build a better chip. However, there is a long feedback loop between the two stages: Typically, once the RTL is delivered by the front-end teams, back-end teams find ways to optimize the layout which need changes in the front-end. The front-end team now needs to distil out the feedback, figure out if the request is even feasible, and if so, implement the change. This takes weeks, or sometimes even months. Projects have multiple iterations like this, which either massively increases chip design timelines, or results in a sub-par chip getting shipped.

Bringing both front-end and back-end into the same umbrella can massively shrink this timeline, by:

• Making the iteration time faster (”I know exactly what needs to change for a better layout, and whether that’s possible to do in the RTL”)

• Reducing the number of iterations (”When I write RTL, I also know how it is going to impact the layout”)

This time saving is crucial - since the semiconductor industry is highly cyclical, when the chip you build is hot, quickly coming up with better versions is key to dominate your industry.

How can AI help to make this possible?

I spoke about the potential of AI in chip design EDA in an earlier post - I think reading this would add value to this discussion:

In addition to the points mentioned there, I want to specifically talk about the two ways I feel AI can make full-stack chip design a reality:

1. Using AI for Knowledge Transfer

If you look at both the reasons why full-stack is hard in chip design, they boil down to the same thing - knowledge transfer is hard. I think AI, even in it’s current form, can solve this problem to a good extent.

For instance, it is possible to develop and maintain a database with all the scripts, best practices, and terminologies associated with back-end tools - effectively allowing an expert in front-end design to move ahead with back-end flows. Similarly, if LLM based RTL generation can be solved effectively, that allows a back-end designer to also manage front-end changes.

I also think LLMs are going to play a big role in documentation and teaching - if basic questions about all parts of a design could be answered immediately, then you can have a specialist in one part of the design expand into being a generalist when needed.

2. Automation is coming

Although I didn’t mention it so far, having different front-end and back-end teams is also important to maintain realistic working hours - from the conversations I have had, chip designers work more hours than software engineers, especially close to deadlines. So in the current state, even if an individual can manage both front-end and back-end, it’s not practical to have them work on both. This is where I feel AI’s big productivity promise is going to play a role.

I think an agentic RTL-to-GDS flow is coming soon, which would automate a lot of mundane tasks that chip designers must do today. I think this would greatly help with the workload problem, and make the idea of a full stack chip designer practical.

Why economic incentives might make this inevitable?

So far, I talked about going “full-stack” like an opportunity - if someone wants to do both front-end and back-end, they might be able to do it in the near future and build better chips. This is actually how it works in software engineering today - some developers become full-stack, but others choose to stay with one. But can chip design afford to offer this flexibility?

I can imagine a future where becoming a full-stack chip designer may not just be an option - it could become the norm. To understand why I feel this way, let’s look at the economics of chip design. At a high level, chip design companies spend money on the following:

• Payroll for chip designers

• Cost to manufacture wafers

• EDA tool licenses

With the way things are headed today, manufacturing costs are increasing sharply - each new node is becoming more challenging to implement, and as a result, more expensive. So this is going to start eating into the profit margins of most chip design companies.

The next factor is EDA licenses. There are many AI-first EDA startups coming up today, but they still don’t have access to the Process Design Kit (PDK) from chip manufacturers. (If you don’t know what a PDK is, think of it as a secret recipe that a manufacturer like TSMC provides to EDA vendors like Cadence and Synopsys in order to correctly map chip designs to a layout that can be manufactured.) As long as PDKs remain under tight control, the legacy EDA vendors cannot be replaced. This leaves two possible outcomes:

• AI EDA tools add a new agentic layer in chip design

• Legacy EDA vendors do all the AI themselves and become more dominant

Either way, I see the cost related to EDA going up. (Not to mention, a lot of chip design companies also need to upgrade compute to handle the new AI workflow.) So to maintain their profit margins, chip design companies might raise their prices - but that may not be sustainable unless they have a very strong moat.

This leads us to the point I’m trying to make - companies need to innovate, and improve productivity with smaller teams. What better way to do this, than with full stack chip designers? (By the way, when I say smaller teams, I don’t necessarily think jobs are going away - on the contrary, I expect shorter design timelines, and a large variety of chips customized for different use cases.)

Some other definitions of full-stack

I took front-end and back-end design as an example here since the analogy fits nicely with the software engineering world. But chip design actually has many more steps, and could see consolidation at other levels too. Here are a few I could think of:

• RTL Design and Functional Verification (software engineering has largely consolidated Programming and Verification)

• Different aspects of Verification (like Functional, Power, Performance, and so on)

• Architecture Simulation and RTL Design (through High-Level-Synthesis)

If you can think of some other aspects in chip design that can be consolidated, I’d love to hear as a comment below :)