A great first-principles walkthrough of what actually happens when you make an inference call. The interesting part isn’t simply “GPU does AI, CPU does everything else,” but the choreography between software, memory, CPU, and GPU that makes inference possible.
Thanks for all the detail, I learned a few new things. In particular the inference engine’s task is quite interesting… is llama.cpp made to target different GPU architectures with separate work for each, or is there some abstraction layer allowing it to reach different ones?
Thanks Avik. That’s a good question, and I must admit I’m not an expert, but I do think that llama.cpp completely supports different GPU targets like CUDA, Metal, OpenCL, etc. Each of these targets would then be executed by the vendor specific runtimes. I found some info here if you are interested: https://deepwiki.com/ggml-org/llama.cpp/5-gpu-and-accelerator-backends
Since llama.cpp can be run standalone, I don’t think there’s any other abstraction layer that would be needed. However, in the post, I describe the LLM inference using Ollama - so there is an Ollama runner which runs an LLM inference engine (llama.cpp). I still think it would be llama.cpp’s backends which get used, but we may need to dig deeper to see exactly what’s happening.
Very detailed post which made me understand lots of missing pieces. Thanks for this first principles breakdown..
Thanks Shreyas!
A great first-principles walkthrough of what actually happens when you make an inference call. The interesting part isn’t simply “GPU does AI, CPU does everything else,” but the choreography between software, memory, CPU, and GPU that makes inference possible.
Thanks for all the detail, I learned a few new things. In particular the inference engine’s task is quite interesting… is llama.cpp made to target different GPU architectures with separate work for each, or is there some abstraction layer allowing it to reach different ones?
Thanks Avik. That’s a good question, and I must admit I’m not an expert, but I do think that llama.cpp completely supports different GPU targets like CUDA, Metal, OpenCL, etc. Each of these targets would then be executed by the vendor specific runtimes. I found some info here if you are interested: https://deepwiki.com/ggml-org/llama.cpp/5-gpu-and-accelerator-backends
Since llama.cpp can be run standalone, I don’t think there’s any other abstraction layer that would be needed. However, in the post, I describe the LLM inference using Ollama - so there is an Ollama runner which runs an LLM inference engine (llama.cpp). I still think it would be llama.cpp’s backends which get used, but we may need to dig deeper to see exactly what’s happening.
Excellent post
Thank you!