OpenVINO: Running AI Models Fast on Intel CPUs, iGPUs, and NPUs
Hey everyone, in this tutorial I want to introduce OpenVINO, Intel's toolkit for making AI models run fast on the hardware you already own, without buying an NVIDIA GPU. Yes, you read that right. A great many real world AI use cases do not need an expensive GPU. What they need is fast, cheap inference on an ordinary CPU, on a laptop's integrated GPU, or on the NPU that ships in newer processors.
I run into this constantly with clients: they want object detection across 10 factory CCTV cameras, or OCR for documents on office machines, or an offline AI assistant on employee laptops because the data is sensitive. Buying a GPU for every one of those endpoints makes no economic sense. That is exactly where OpenVINO becomes very practical.
In this tutorial we cover the core concepts, installation, converting models from PyTorch and ONNX, inference in Python, INT8 quantization with NNCF, running local LLMs, and finally how to choose a device and measure performance correctly.
Introduction
What Is OpenVINO
OpenVINO stands for Open Visual Inference and Neural Network Optimization. Despite the "visual" in the name, it now supports nearly every model type, including LLMs and audio models.
The idea works like this. Your PyTorch or TensorFlow model is converted into OpenVINO's intermediate representation, called IR: an .xml file describing the graph structure and a .bin file holding the weights. The OpenVINO runtime then executes that IR with kernels tuned for Intel hardware, using AVX-512 and AMX instructions on CPUs, dedicated kernels on integrated GPUs, and specialized accelerators on NPUs.
There are four main benefits. First, one API for many devices, where you literally change the string "CPU" to "GPU" or "NPU". Second, serious CPU optimization that often yields a two to four times speedup over running the PyTorch model as is. Third, mature INT8 quantization through NNCF. Fourth, a small runtime footprint that suits edge deployment.
When OpenVINO Makes Sense
OpenVINO makes sense when you deploy on CPUs or Intel edge devices, when you need many small to medium model instances running cheaply in parallel, when data must never leave the device so inference has to be local, or when you want to use the integrated GPUs sitting idle in thousands of office laptops.
It is a poor fit when you are training models (it is inference only), when you need very large LLMs at high throughput (that is dedicated GPU territory), or when your stack is deeply tied to the NVIDIA ecosystem.
Installation
python -m venv venv
source venv/bin/activate # Linux / Mac
venv\Scripts\activate # Windows
pip install openvino openvino-dev
pip install "optimum[openvino]" # for HuggingFace models
pip install nncf # for quantization
Check which devices your machine exposes:
import openvino as ov
core = ov.Core()
print("available devices:", core.availabledevices)
for d in core.availabledevices:
print(d, "->", core.getproperty(d, "FULLDEVICENAME"))
You will normally see CPU, plus GPU if your processor has integrated graphics. On newer Core Ultra processors, NPU appears as well.
Converting a PyTorch Model
The most direct route is converting a PyTorch model you already have.
import torch
import openvino as ov
import torchvision
1. Prepare the PyTorch model
model = torchvision.models.resnet50(weights="IMAGENET1KV2")
model.eval()
2. Convert to OpenVINO
exampleinput = torch.randn(1, 3, 224, 224)
ovmodel = ov.convertmodel(model, exampleinput=exampleinput)
3. Save as IR
ov.savemodel(ovmodel, "resnet50.xml")
You now have resnet50.xml and resnet50.bin. Those two files are what you ship to production, with no need to install PyTorch there. That point is often underestimated: the OpenVINO runtime is far lighter than PyTorch, which means your Docker image can shrink dramatically.
For ONNX models it is even simpler, because OpenVINO reads ONNX directly:
import openvino as ov
core = ov.Core()
model = core.readmodel("model.onnx")
compiled = core.compilemodel(model, "CPU")
First Inference
This is the base pattern you will reuse constantly.
import numpy as np
import openvino as ov
from PIL import Image
core = ov.Core()
1. Read and compile the model for a target device
model = core.readmodel("resnet50.xml")
compiled = core.compilemodel(model, "CPU")
2. Prepare the input
img = Image.open("image.jpg").convert("RGB").resize((224, 224))
x = np.array(img).astype(np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406])
std = np.array([0.229, 0.224, 0.225])
x = (x - mean) / std
x = x.transpose(2, 0, 1)[None].astype(np.float32) # NCHW
3. Run
result = compiled(x)[compiled.output(0)]
4. Top-5
top5 = np.argsort(-result[0])[:5]
print("top class indices:", top5)
print("scores:", result[0][top5])
Note that compilemodel is the expensive step, because that is where OpenVINO selects kernels and optimizes the graph for the target device. Do it once at application startup, never inside a request handler.
Asynchronous Inference
For throughput, do not use the synchronous pattern above. Use an async inference queue so the CPU never idles while waiting.
import openvino as ov
import numpy as np
core = ov.Core()
model = core.readmodel("resnet50.xml")
compiled = core.compilemodel(model, "CPU", {"PERFORMANCEHINT": "THROUGHPUT"})
allresults = []
def store(request, userdata):
allresults.append((userdata, request.getoutputtensor(0).data.copy()))
queue = ov.AsyncInferQueue(compiled)
queue.setcallback(store)
images = [np.random.randn(1, 3, 224, 224).astype(np.float32) for in range(100)]
for i, image in enumerate(images):
queue.startasync({0: image}, userdata=i)
queue.waitall()
print("processed", len(allresults), "images")
PERFORMANCEHINT has two main values. LATENCY optimizes the response time of a single request, right for interactive applications. THROUGHPUT optimizes total requests per second by running several parallel streams, right for batch processing or servers. Pick according to your workload, since this is one of the highest impact knobs available.
INT8 Quantization with NNCF
Quantization converts weights and activations from fp32 to int8. The model becomes roughly four times smaller and inference is typically two to three times faster on modern CPUs, with an accuracy drop usually under one percent when done properly.
import nncf
import openvino as ov
import numpy as np
from torch.utils.data import DataLoader
core = ov.Core()
model = core.readmodel("resnet50.xml")
Calibration data: 100 to 300 representative samples is plenty
def transformfn(item):
image, = item
return image.numpy()
calibrationloader = DataLoader(validationdataset, batchsize=1, shuffle=True)
calibrationdataset = nncf.Dataset(calibrationloader, transformfn)
modelint8 = nncf.quantize(
model,
calibrationdataset,
preset=nncf.QuantizationPreset.MIXED,
subsetsize=300,
)
ov.savemodel(modelint8, "resnet50int8.xml")
An important point about calibration data: draw it from your real distribution, not random noise. If the model will see dark, shadowy factory CCTV frames, do not calibrate on bright stock photos. Quantization quality depends heavily on this.
And always re-measure accuracy after quantization on your own validation data. If the drop exceeds what you can accept, switch to a more conservative preset or exclude a few sensitive layers from quantization.
Running HuggingFace Models
Through Optimum Intel, nearly any HuggingFace model runs on OpenVINO without manual conversion.
from optimum.intel import OVModelForSequenceClassification
from transformers import AutoTokenizer
modelid = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.frompretrained(modelid)
model = OVModelForSequenceClassification.frompretrained(modelid, export=True)
model.savepretrained("distilbert-ov")
inputs = tokenizer("The movie was great, I loved it.", returntensors="pt")
logits = model(inputs).logits
print(logits.argmax(-1))
The export=True argument triggers automatic conversion from PyTorch weights to OpenVINO IR. Once saved, later runs just load from the local directory with no conversion.
Running Local LLMs
This is the use case people ask about most these days: running a small LLM on a laptop, fully offline.
from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer
modelid = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tokenizer = AutoTokenizer.frompretrained(modelid)
model = OVModelForCausalLM.frompretrained(modelid, export=True)
prompt = "Explain what edge computing is in three sentences."
inputs = tokenizer(prompt, returntensors="pt")
out = model.generate(inputs, maxnewtokens=200, dosample=True, temperature=0.7)
print(tokenizer.decode(out[0], skipspecialtokens=True))
For decent CPU performance you almost certainly want weight quantization. The easiest path is the CLI:
optimum-cli export openvino \
--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
--weight-format int4 \
--group-size 128 \
--ratio 0.8 \
tinyllama-ov-int4
Then load it:
model = OVModelForCausalLM.frompretrained("tinyllama-ov-int4")
With INT4, a 1 billion parameter model shrinks to roughly 700 MB and generates at a comfortable reading speed even on a laptop without a dedicated GPU. For a 7B model you need about 5 to 6 GB of RAM and it will be slower, but still usable for non-interactive tasks.
Choosing a Device
Switching devices is a string change, but there are trade-offs worth knowing.
CPU is the most universal and most mature target. For small and medium models it is frequently more than sufficient, especially on server processors with AMX.
GPU here means an Intel integrated GPU. It is usually more power efficient and faster than the CPU for vision models, but transferring data to GPU memory has overhead, so very small models sometimes run faster on CPU.
NPU is available on Core Ultra processors and is extremely power efficient. It suits models running continuously in the background, such as face detection or background blur. Its operator coverage is still narrower than CPU.
AUTO lets OpenVINO pick the best device, and it can even start on CPU while the GPU compilation finishes.
compiled = core.compilemodel(model, "AUTO")
MULTI and HETERO let you spread work across several devices at once.
compiled = core.compilemodel(model, "MULTI:GPU,CPU")
Measuring Performance Properly
OpenVINO ships a benchmark tool you should prefer over hand-rolled time.time() measurements.
benchmarkapp -m resnet50.xml -d CPU -hint throughput -t 30
benchmarkapp -m resnet50int8.xml -d CPU -hint throughput -t 30
Watch two numbers in the output: average latency and throughput in FPS. Compare your fp32 and int8 builds here. If the speedup is smaller than expected, it is usually because the model is so small that framework overhead dominates, or because you benchmarked with the wrong hint.
Tips and Best Practices
Compile the model once at startup, never per request. This is the number one performance mistake I see.
Enable the model cache so subsequent compilations are faster, especially for GPU and NPU.
core.setproperty({"CACHEDIR": "./ovcache"})
Choose PERFORMANCEHINT to match your workload and test both. The difference between LATENCY and THROUGHPUT can be a factor of two in opposite directions.
For video applications, do not process frames one at a time synchronously. Use AsyncInferQueue and keep the pipeline full.
Store converted IR files in your artifact repository rather than reconverting on every deployment. Conversion is deterministic and does not need repeating.
Always compare accuracy before and after quantization on your own data, not on a public benchmark.
Mind your thread counts. If you run several inference processes on one machine, cap threads per process so they do not fight over cores.
core.setproperty("CPU", {"INFERENCENUMTHREADS": 4})
Conclusion
OpenVINO makes AI substantially more affordable, because it extracts real performance from hardware you already own. The key takeaways:
Models are converted to IR once, then executed by a light runtime that does not require PyTorch in production.
The same API serves CPU, integrated GPU, and NPU with only a device string change.
INT8 quantization through NNCF delivers large speedups with small accuracy loss, provided the calibration data is representative.
For HuggingFace models and local LLMs, Optimum Intel reduces conversion to a single line, and INT4 makes small LLMs comfortable on laptops.
Use AsyncInferQueue and the right PERFORMANCEHINT to get real throughput, then verify with benchmark_app.
Take a classification or detection model you already have, convert it to OpenVINO, quantize it to INT8, and compare the numbers against PyTorch on the same CPU. Many people are surprised to discover they never needed a GPU for the use case they are working on. Happy optimizing.