Thinking in Python Runtime: How an Enterprise AI Application Occupies a Multi-Processor, Multi-Core System

Publication Date: October 05, 2026 Last Updated: October 05, 2026
Hardware and Runtime Fundamentals
Before examining how a Python AI application occupies a multi-processor, multi-core system, it is useful to establish a few foundational concepts. Terms such as CPU socket, physical core, logical CPU, NUMA node, cache, memory bandwidth, PCIe, and GPU are often used interchangeably in architectural discussions, but they describe different layers of the system.
This section establishes the terminology used throughout the article. The goal is not to provide a comprehensive treatment of computer architecture, but to build the minimum mental model required to understand the execution diagrams that follow.
A CPU socket is the physical slot on a server's motherboard into which one processor chip is installed. "Two CPU sockets" means the server has two separate processor chips, each plugged into its own slot. (It has nothing to do with network sockets in programming, which is an unrelated use of the same word.)
If you say "64 cores across two sockets", it means each chip contributes 32 cores. The schematic below zooms in from the server down to a single core.

Next is the physical view of those two sockets on the motherboard. Each socket has its own memory and its own PCIe lanes to the GPUs, and the two are joined by a link between the chips.

The two sockets are not one pool of 64 equal cores. Each chip reaches its own memory and GPUs quickly, but anything on the other socket has to cross the inter-socket link, which is slower. A service tuned for ten requests per minute on a laptop therefore will not scale just because it lands on a bigger machine. How the workload is placed across the sockets decides the result, and that is the point Figure 1 (Section: Start with the Hardware: Topology Determines Placement) of this article builds on.
PCIe (PCI Express) is the high-speed connection between a CPU and the devices attached to it, such as GPUs, network cards and NVMe SSDs. A lane is its basic building block. One lane is two pairs of wires, one for sending and one for receiving, so data flows in both directions at the same time. A device connects through a bundle of lanes, written x1, x4, x8 or x16, and more lanes means more bandwidth.

Lanes are a fixed budget per socket and cannot be borrowed from the other socket. Each socket has its own memory and its own PCIe lanes to the GPUs, and the two are joined by a link between the chips. This means GPU 0 and GPU 1 talk to the CPU directly through socket 0's lanes, and GPU 2 and GPU 3 do the same through socket 1's lanes. If a process running on socket 1 needs data in a GPU attached to socket 0, the traffic also has to cross the inter-socket link. That adds latency and competes for the link's bandwidth. This is why pinning each worker to the socket that owns its GPU matters.

The lane counts in the above diagram are illustrative. Real server CPUs offer anywhere from several dozen to over a hundred lanes per socket, depending on the model.
A CPU core is an independent processing unit inside a CPU chip. It can fetch instructions, decode them, execute them and store the results, all on its own. A chip with 32 cores is like a workshop with 32 separate workers, each with a private workbench (registers and small caches), who share a common storeroom (L3 cache and memory). Each core can run a different program, or a different thread of the same program, at the same time as the others.
The diagram below opens up one core and shows how it relates to the rest of the chip.

The terms nest as follows: a server contains sockets, each socket holds one CPU chip, each chip holds cores, and with SMT each core presents two logical CPUs. With SMT enabled, the operating system would list 128 logical CPUs, and tools such as lscpu report both numbers. SMT typically improves throughput by a modest margin for mixed workloads, but it does not make a core twice as fast, so capacity planning should count physical cores.
For the Python runtime, the CPU core is where execution happens. Each Python thread is an OS thread that the scheduler places on one logical CPU at a time, and the GIL limits how many of those threads can run bytecode at once. A multi-process design uses several cores in parallel, one interpreter per core, which is why pinning workers to specific cores works so well.

SMT stands for Simultaneous Multithreading. It is a CPU feature that lets one physical core work on two threads at the same time. Intel markets its version as Hyper-Threading, while AMD simply calls it SMT. Each hardware thread keeps its own registers and program counter, but the two share the core's execution units and caches. The operating system then sees one physical core as two logical CPUs, which is what the sentence means by "each core presents two logical CPUs."
The motivation is that a single thread rarely keeps a core fully busy. It stalls while waiting for data from memory, after a mispredicted branch, or on instructions that depend on earlier results. SMT lets a second thread use the execution slots that would otherwise sit idle.

The busy percentages in the diagram are illustrative. In practice SMT raises a core's total throughput by a modest margin that depends heavily on the workload, often somewhere around 10 to 30 percent. It never doubles the speed, because the two threads share the same execution units.
For the Python and AI workloads in this article, this has three practical consequences:
- Count physical cores when sizing. A server with 64 physical cores shows 128 logical CPUs, but the extra 64 are not equivalent to additional cores.
- Heavy numerical work may gain little. Native libraries such as BLAS and PyTorch's CPU kernels already keep a core's vector units busy, so a second thread has few gaps to fill and mostly competes for the same resources.
- Latency-sensitive services often pin workers to physical cores. Placing one worker per physical core avoids two busy threads sharing one core's resources, and some operators turn SMT off for that reason.
NUMA stands for Non-Uniform Memory Access. A NUMA node is a group of CPU cores together with the memory that is directly attached to them. In most two-socket servers each socket forms one NUMA node, although some processors can split a single socket into several nodes. Cores reach memory in their own node quickly. Reaching memory that belongs to another node takes longer because the request has to travel across the link between the chips.
The diagram below contrasts this with the older uniform design, where every core reached one shared pool at the same speed.

The diagram below shows the same thread reading two pieces of data. Data A sits in its own node's memory. Data B sits in the other node's memory.

A server might have 512 GB of RAM in total, but that memory is split into pieces, for example 256 GB per node. Two services with identical memory footprints can perform very differently depending on whether their data sits in the same node as the cores that use it. Remote access typically costs noticeably more latency and offers less bandwidth than local access. The exact gap varies by hardware, and commonly lands around 1.5 to 2 times the local latency.
Important Point: Linux usually places a memory page on the node of the thread that first writes to it. If a master process loads a model while running on node 0, and workers are later scheduled on node 1, those workers read the weights remotely on every access. You can inspect the layout on a Linux server with numactl --hardware or lscpu. You can fix placement by pinning workers and their memory together, for example with numactl --cpunodebind=1 --membind=1.
A GPU (Graphics Processing Unit) is a processor built to run a very large number of simple calculations at the same time. It was originally designed to render images, where millions of pixels can be computed independently. The same property turned out to suit neural networks, which are mostly large matrix multiplications. That is why GPUs now power AI training and inference. A CPU is the opposite kind of processor: it has a small number of powerful cores that handle complex, varied work one step at a time and finish each step quickly.
The diagram below compares how the two chips spend their silicon.

The CPU decides what to do; the GPU does the heavy arithmetic. The CPU side runs Python, so the interpreter, the GIL and NUMA placement shape how fast work reaches the GPU. The PCIe lanes carry the data and launch commands across, and the GPU's own memory holds the model weights while it computes. A fast GPU can sit idle if the CPU side feeds it too slowly, which is a common bottleneck in production AI serving.

HBM stands for High Bandwidth Memory. It is a type of DRAM designed to move data to a processor much faster than ordinary memory, and it is what most AI GPUs use as their main memory.
Ordinary memory modules sit in slots on the motherboard, far from the processor and connected by a relatively narrow bus. HBM takes a different approach. Several DRAM chips are stacked vertically, and the stack is mounted right beside the GPU die on a shared piece of silicon, so the connection can be extremely wide and extremely short.


The figures above are approximate peak values and vary by product. The GPU's HBM delivers several times the bandwidth of the whole server's CPU memory, and orders of magnitude more than the PCIe link feeding it.
This matters for AI inference because generating each token requires reading essentially all of the model's weights from memory. That makes LLM inference limited largely by memory bandwidth rather than by raw arithmetic speed, so HBM bandwidth often sets the ceiling on tokens per second.
HBM has trade-offs too:
- Capacity is limited. A single GPU carries tens to a couple of hundred GB, such as 80 GB on an H100 and up to about 192 GB on a B200. That is far less than server DRAM, which can reach terabytes.
- It is expensive. Stacking dies and packaging them on an interposer is costly.
- It is not upgradeable. The memory is fixed inside the GPU package.
This is why the memory diagram in Figure 6 (Section. Memory: fork(), Copy-on-Write, and the Page-Sharing Trap) shows weights being copied from host RAM into HBM once per worker. The model must fit in HBM to run at full speed, and when it does not, techniques like tensor parallelism split it across several GPUs and use NVLink to keep them in step.
NVLink is NVIDIA's high-speed interconnect for connecting GPUs directly to each other. Without it, two GPUs in the same server can only exchange data over PCIe, usually by going through the CPU's PCIe root. NVLink gives them a dedicated, much faster path of their own.


NVLink matters most when a model is too large for one GPU. Techniques such as tensor parallelism split each layer across several GPUs, and those GPUs must exchange intermediate results many times per token. Libraries such as NCCL perform these exchanges, and they run far faster over NVLink and NVSwitch than over PCIe. NVLink only works inside a node, so communication between servers uses a network such as InfiniBand or Ethernet instead.
In the hardware topology from this article, the two roles are separate. PCIe carries the CPU-to-GPU traffic, meaning data in and kernel launches. NVLink carries the GPU-to-GPU traffic, which is why it appears as its own link between the GPUs in Figure 1 (Section. Start with the Hardware: Topology Determines Placement).
WSGI stands for Web Server Gateway Interface. It is the traditional Python standard defined in PEP 3333 that allows web servers (like Gunicorn or Nginx) to communicate seamlessly with Python web applications and frameworks (like Django or Flask). It handles synchronous, one-way request-and-respo.
ASGI stands for Asynchronous Server Gateway Interface. It is the spiritual successor to WSGI, designed to provide a standard interface between async-capable Python web servers (like Uvicorn) and web frameworks (like FastAPI or modern Django). Unlike WSGI, it natively supports asynchronous features like long-polling, WebSockets, HTTP/2, and background tasks.

Gunicorn ("Green Unicorn") is a lightweight, production-ready Python WSGI HTTP Server for UNIX that manages and serves synchronous Python web applications. The master process acts as a manager that oversees worker processes, tracks their health, and restarts them automatically if they crash or time out. Worker processes execute isolated copies of the application code in memory to translate raw incoming HTTP requests into readable WSGI operations. It employs a pre-fork design where a single master process initializes the application and forks worker processes to handle high traffic concurrently. It supports synchronous workers by default, but allows alternative async configurations like Gevent or Eventlet for I/O-heavy workloads.
Uvicorn is a lightning-fast ASGI web server implementation for Python that bridges the gap between low-level network connections and high-performance asynchronous frameworks like FastAPI and Starlette. It handles thousands of concurrent client connections efficiently by executing non-blocking I/O operations through an event-driven loop. It works with any Python web framework that implements the ASGI standard, such as FastAPI, Starlette, and Django. It supports a built-in auto-reload flag for instant code refreshing during local development, and integrates seamlessly with Gunicorn worker configurations for stable production deployments.
An ASGI deployment means running a Python web application (typically a service that exposes an API over HTTP or WebSockets) written to the ASGI interface in production through an ASGI-capable server, usually with a process manager and a proxy around it. The schematic below shows the typical ASGI deployment stack.

- Client Request: A browser or external application sends a request.
- Reverse Proxy (Nginx / Cloud LB): Handles encryption (TLS/SSL), serves static files, and routes incoming traffic to the application layer.
- Process Manager (Gunicorn Master): Instead of handling web traffic directly, the Gunicorn master process runs on a single thread. Its only job is to monitor system health, manage process lifecycles, and automatically restart crashed workers.
- Uvicorn Workers: Gunicorn forks multiple independent worker processes using Uvicorn's ASGI worker class. Each worker runs its own asynchronous event loop (uvloop), loads the FastAPI application, and manages incoming requests concurrently.
- GPU Execution: The FastAPI app inside the worker triggers heavy computations (like PyTorch model inferences), which are offloaded onto the system's GPU kernels.
An ASGI application is an async function that receives three arguments: a scope describing the connection, a receive channel for incoming events and a send channel for outgoing ones. Frameworks like FastAPI hide this contract, but every ASGI server and framework agrees on it.
async def app(scope, receive, send):
... # read request events from receive(), write response events with send()
The caveat for AI workloads is that the event loop runs on a single thread under the GIL. While it is waiting, it serves other requests well. But if model code blocks it, for example a CPU-heavy tokenization step or a synchronous inference call that holds the thread, every other request on that worker stalls. That is why Figure 7 of this article (Section: Following One Request Through the Runtime) moves such work to thread pools, process pools and a dynamic batcher, and keeps the event loop free for accepting and streaming.
Introduction
Python is the working language of enterprise AI, yet it is often treated as a black box once an application leaves the developer's laptop. A service that handles ten requests per minute in development may be deployed onto a server with two CPU sockets, 64 cores, and four GPUs, with the expectation that the workload will scale with the available hardware. Sometimes it does. Often it does not. The reasons are rarely obvious from the application code alone.
The explanation lies in the runtime.
CPython is not an isolated execution environment. It is one layer in a larger system. The Python process runs under an operating system scheduler, which places runnable threads and processes onto CPU cores. Those cores operate within a memory hierarchy whose access costs vary significantly by locality. On multi-socket systems, memory may also be physically distributed across NUMA nodes, making the location of data as important as the amount of data. Within the interpreter itself, the execution model, memory-management mechanisms, and native extensions determine how effectively Python code can exploit the underlying machine.
For AI workloads, the picture becomes even more interesting. Frameworks such as PyTorch can move computation from Python into highly optimized native code and onto GPUs. A single Python request may therefore involve interpreter execution on one or more CPU cores, native computation in compiled libraries, memory movement across CPU and GPU boundaries, and parallel execution across thousands of GPU threads. The Python code is only the visible tip of a much larger execution system.
This article develops a runtime mental model for that system through seven schematic views, progressing from silicon and operating-system scheduling to the execution of a single inference request. Each view isolates one architectural idea: processor topology, core utilization, process and thread placement, memory locality, interpreter execution, native parallelism, and GPU execution. Taken together, they provide a practical way to reason about where a Python-based AI application actually runs, how much of the underlying hardware it can use, and why adding CPUs, cores, processes, or GPUs does not automatically produce linear performance gains.
The discussion assumes CPython 3.12 through 3.14, an ASGI deployment using a server such as Uvicorn or Gunicorn, an AI framework such as PyTorch, and GPU-accelerated inference. The objective is not to explain every implementation detail of CPython or every feature of a particular framework. It is to build the systems-level mental model required to make sound architecture, capacity-planning, deployment, and performance-engineering decisions for enterprise AI workloads.