1. System Prerequisites
Ensure your system meets the following requirements:
Ubuntu 20.04 / 22.04 / 24.04 NVIDIA GPU installed NVIDIA drivers installed on the host Sudo accessCheck GPU:
lspci | grep -i nvidia
Check drivers:
nvidia-smi
If nvidia-smi fails, install the driver first:
sudo ubuntu-drivers devices
sudo ubuntu-drivers autoinstall
sudo reboot
2. Install Docker Engine
Install dependencies
sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release
Add Docker GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg\
| sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
Add Docker repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(lsbrelease -cs) stable" \
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Install Docker
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io
Run Docker without sudo (optional)
sudo usermod -aG docker $USER
newgrp docker
Docker Verification:
docker run hello-world
3. Install NVIDIA Container Toolkit
Add NVIDIA GPG key
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey\
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
Add NVIDIA repository
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list\
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'\
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
Install toolkit
sudo apt update
sudo apt install -y nvidia-container-toolkit
4. Configure Docker Runtime NVIDIA
Configure Docker to recognize the NVIDIA runtime:
sudo nvidia-ctk runtime configure --runtime=docker
Restart Docker:
sudo systemctl restart docker
5. Verify GPU in Container
Run the official CUDA container:
docker run --rm --gpus all \
nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
If nvidia-smi appears Inside the container, the installation was successful.
6. Example of Using PyTorch
docker run --rm -it --gpus all \
pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime \
pytorch -c "import torch; print(torch.cuda.isavailable())"
Correct output:
True
7. Verifying the Docker Runtime
docker info | grep -i runtime
Make sure nvidia appears in the runtime list.
8. General Troubleshooting
Error: could not select device driver "nvidia"
Solution:
sudo systemctl restart docker
sudo reboot
CUDA not detected in container
Check the CUDA library on the host:
ls /usr/lib/x86_64-linux-gnu/libcuda.so
If it's not there, the NVIDIA driver isn't installed correctly.
Note for Jetson (ARM)
For Jetson Nano, Xavier, and Orin:
NVIDIA Docker is included in JetPack
- Do not install NVIDIA Container Toolkit manually
9. Installation Flow (Short)
NVIDIA Driver (host)
↓
Docker Engine
↓
NVIDIA Container Toolkit
↓
docker run --gpus all
↓
CUDA available in containers