Complete JAX Tutorial: Google's High-Performance Machine Learning Framework
JAX is an open-source library from Google that combines NumPy with automatic differentiation capabilities and hardware acceleration through XLA (Accelerated Linear Algebra). JAX allows you to write numerical code with familiar NumPy syntax while achieving significantly higher performance through JIT compilation and native support for GPUs and TPUs.
In this tutorial, we will learn JAX from the ground up through advanced usage, including function transformations, neural network training, and performance optimization for large-scale machine learning workloads.
Why JAX?
JAX has become the framework of choice for many ML research and production teams for several reasons:
- NumPy Compatibility: The
jax.numpyAPI is nearly identical to NumPy, making the learning curve extremely low for Python users already familiar with the scientific computing ecosystem. - Automatic Differentiation: The
jax.gradfunction enables automatic differentiation of regular Python functions, supporting both forward-mode and reverse-mode differentiation. - JIT Compilation: Through
jax.jit, Python functions are compiled into XLA-optimized code for the target hardware (CPU, GPU, or TPU). - Automatic Vectorization:
jax.vmapenables automatic vectorization of functions without writing manual loops. - Parallelization:
jax.pmapprovides transparent multi-device parallelization. - Rich Ecosystem: Libraries like Flax, Optax, and Orbax provide high-level abstractions for neural networks, optimizers, and checkpointing.
Installation
Basic Installation (CPU)
pip install jax
Installation with GPU Support (CUDA 12)
pip install jax[cuda12]
Installation with TPU Support
pip install jax[tpu] -f https://storage.googleapis.com/jax-releases/libtpureleases.html
Verify Installation
import jax
import jax.numpy as jnp
print(f"JAX version: {jax.version}")
print(f"Available devices: {jax.devices()}")
print(f"Default backend: {jax.defaultbackend()}")
Simple test
x = jnp.array([1.0, 2.0, 3.0])
print(f"Array: {x}")
print(f"Device: {x.devices()}")
Expected output:
JAX version: 0.4.x
Available devices: [CpuDevice(id=0)]
Default backend: cpu
Array: [1. 2. 3.]
Basic Usage
JAX NumPy: Arrays and Basic Operations
JAX provides jax.numpy as a drop-in replacement for NumPy with a nearly identical API:
import jax.numpy as jnp
Creating arrays
a = jnp.array([1.0, 2.0, 3.0, 4.0])
b = jnp.ones((3, 3))
c = jnp.zeros((2, 4))
d = jnp.linspace(0, 1, 5)
Mathematical operations
result = jnp.dot(a, a)
print(f"Dot product: {result}")
Matrix operations
matrix = jnp.array([[1, 2], [3, 4]])
eigenvalues = jnp.linalg.eigvalsh(matrix)
print(f"Eigenvalues: {eigenvalues}")
Statistics
data = jnp.array([1.0, 2.0, 3.0, 4.0, 5.0])
print(f"Mean: {jnp.mean(data)}")
print(f"Std: {jnp.std(data)}")
Key Difference: Immutability
Unlike NumPy, JAX arrays are immutable. You cannot modify elements in-place:
import jax.numpy as jnp
x = jnp.array([1, 2, 3, 4, 5])
NumPy style (NOT possible in JAX):
x[0] = 10 # Error!
JAX style using .at[].set():
xupdated = x.at[0].set(10)
print(f"Original: {x}") # [1, 2, 3, 4, 5]
print(f"Updated: {xupdated}") # [10, 2, 3, 4, 5]
Other .at operations:
xadd = x.at[1].add(100) # Add 100 to index 1
xmul = x.at[2].multiply(3) # Multiply index 2 by 3
print(f"After add: {xadd}")
print(f"After multiply: {xmul}")
Random Number Generation
JAX uses an explicit and reproducible PRNG (Pseudo-Random Number Generator) system:
import jax
import jax.numpy as jnp
Create PRNG key
key = jax.random.PRNGKey(42)
Each random operation requires a separate key
key1, key2, key3 = jax.random.split(key, 3)
Generate random numbers
uniform = jax.random.uniform(key1, shape=(3,))
normal = jax.random.normal(key2, shape=(3,))
integers = jax.random.randint(key3, shape=(5,), minval=0, maxval=10)
print(f"Uniform: {uniform}")
print(f"Normal: {normal}")
print(f"Integers: {integers}")
Common pattern: split key at each iteration
key = jax.random.PRNGKey(0)
for i in range(3):
key, subkey = jax.random.split(key)
sample = jax.random.normal(subkey, shape=(2,))
print(f"Iteration {i}: {sample}")
Function Transformations
The core power of JAX lies in its ability to transform regular Python functions. There are four main transformations: grad, jit, vmap, and pmap.
Automatic Differentiation with jax.grad
import jax
import jax.numpy as jnp
Simple function
def f(x):
return jnp.sin(x) 2 + jnp.cos(x)
Gradient (first derivative)
df = jax.grad(f)
print(f"f(1.0) = {f(1.0):.4f}")
print(f"f'(1.0) = {df(1.0):.4f}")
Second derivative
d2f = jax.grad(jax.grad(f))
print(f"f''(1.0) = {d2f(1.0):.4f}")
Gradient for multi-variable functions
def g(x, y):
return x 2 y + jnp.exp(y)
Gradient with respect to first argument (default)
dgdx = jax.grad(g, argnums=0)
print(f"dg/dx(2.0, 1.0) = {dgdx(2.0, 1.0):.4f}")
Gradient with respect to both arguments
dgboth = jax.grad(g, argnums=(0, 1))
grads = dgboth(2.0, 1.0)
print(f"dg/dx = {grads[0]:.4f}, dg/dy = {grads[1]:.4f}")
Gradients for ML Loss Functions
import jax
import jax.numpy as jnp
def mseloss(params, X, y):
"""Mean Squared Error loss for linear regression."""
w, b = params
predictions = X @ w + b
return jnp.mean((predictions - y) 2)
Dummy data
key = jax.random.PRNGKey(42)
X = jax.random.normal(key, (100, 3))
truew = jnp.array([1.0, -2.0, 0.5])
trueb = 1.0
y = X @ truew + trueb
Initialize parameters
key, subkey = jax.random.split(key)
w = jax.random.normal(subkey, (3,))
b = jnp.array(0.0)
params = (w, b)
Compute gradients
gradfn = jax.grad(mseloss)
grads = gradfn(params, X, y)
print(f"Gradient w: {grads[0]}")
print(f"Gradient b: {grads[1]}")
Value and gradient simultaneously
lossandgrad = jax.valueandgrad(mseloss)
lossval, grads = lossandgrad(params, X, y)
print(f"Loss: {lossval:.4f}")
Just-In-Time Compilation with jax.jit
jax.jit compiles Python functions into optimized XLA code:
import jax
import jax.numpy as jnp
import time
def slowfunction(x):
"""Function with many operations."""
for in range(100):
x = jnp.sin(x) + jnp.cos(x) 0.1
return x
JIT-compiled version
fastfunction = jax.jit(slowfunction)
x = jnp.ones((1000, 1000))
Warmup (first compilation)
= fastfunction(x).blockuntilready()
Benchmark without JIT
start = time.time()
= slowfunction(x).blockuntilready()
nojittime = time.time() - start
Benchmark with JIT
start = time.time()
= fastfunction(x).blockuntilready()
jittime = time.time() - start
print(f"Without JIT: {nojittime:.4f}s")
print(f"With JIT: {jittime:.4f}s")
print(f"Speedup: {nojittime / jittime:.1f}x")
Automatic Vectorization with jax.vmap
jax.vmap lets you write functions for a single sample and automatically vectorize them for batches:
import jax
import jax.numpy as jnp
Function for a single sample
def predictsingle(params, x):
w, b = params
return jnp.dot(w, x) + b
Data
key = jax.random.PRNGKey(42)
w = jax.random.normal(key, (5,))
b = jnp.array(0.0)
params = (w, b)
Single input
xsingle = jnp.ones(5)
print(f"Single prediction: {predictsingle(params, xsingle)}")
Batch input using vmap
predictbatch = jax.vmap(predictsingle, inaxes=(None, 0))
Xbatch = jax.random.normal(key, (10, 5))
predictions = predictbatch(params, Xbatch)
print(f"Batch predictions shape: {predictions.shape}")
print(f"Batch predictions: {predictions}")
vmap for computing Jacobians
def f(x):
return jnp.array([x[0] * 2 + x[1], jnp.sin(x[0]) x[1]])
Jacobian using vmap + grad
jacobian = jax.jacobian(f)
x = jnp.array([1.0, 2.0])
J = jacobian(x)
print(f"Jacobian:\n{J}")
Advanced Usage
Training a Neural Network from Scratch
Here is a complete example of training a simple neural network using pure JAX:
import jax
import jax.numpy as jnp
from typing import List, Tuple
def initmlp(layersizes: List[int], key):
"""Initialize MLP parameters with Xavier initialization."""
params = []
for i in range(len(layersizes) - 1):
key, wkey, bkey = jax.random.split(key, 3)
scale = jnp.sqrt(2.0 / layersizes[i])
w = jax.random.normal(wkey, (layersizes[i], layersizes[i + 1])) scale
b = jnp.zeros(layersizes[i + 1])
params.append((w, b))
return params
def forward(params, x):
"""MLP forward pass with ReLU activation."""
for w, b in params[:-1]:
x = jnp.maximum(0, x @ w + b) # ReLU
w, b = params[-1]
return x @ w + b # Output layer without activation
def crossentropyloss(params, X, y):
"""Cross-entropy loss for classification."""
logits = forward(params, X)
logprobs = jax.nn.logsoftmax(logits)
onehot = jax.nn.onehot(y, logits.shape[-1])
return -jnp.mean(jnp.sum(onehot logprobs, axis=-1))
def accuracy(params, X, y):
"""Compute prediction accuracy."""
logits = forward(params, X)
predicted = jnp.argmax(logits, axis=-1)
return jnp.mean(predicted == y)
@jax.jit
def trainstep(params, X, y, lr):
"""Single training step with gradient descent."""
loss, grads = jax.valueandgrad(crossentropyloss)(params, X, y)
params = jax.tree.map(lambda p, g: p - lr g, params, grads)
return params, loss
Generate synthetic dataset
key = jax.random.PRNGKey(42)
nsamples = 1000
nfeatures = 20
nclasses = 3
key, datakey = jax.random.split(key)
X = jax.random.normal(datakey, (nsamples, nfeatures))
truew = jax.random.normal(key, (nfeatures, nclasses))
logits = X @ truew
y = jnp.argmax(logits, axis=-1)
Split data
Xtrain, Xtest = X[:800], X[800:]
ytrain, ytest = y[:800], y[800:]
Initialize model
key = jax.random.PRNGKey(0)
params = initmlp([nfeatures, 64, 32, nclasses], key)
Training loop
lr = 0.01
for epoch in range(100):
params, loss = trainstep(params, Xtrain, ytrain, lr)
if (epoch + 1) % 20 == 0:
trainacc = accuracy(params, Xtrain, ytrain)
testacc = accuracy(params, Xtest, ytest)
print(f"Epoch {epoch+1}: loss={loss:.4f}, "
f"trainacc={trainacc:.4f}, testacc={testacc:.4f}")
Using JAX with Flax and Optax
For larger projects, use Flax (neural network library) and Optax (optimizer library):
import jax
import jax.numpy as jnp
import flax.linen as nn
import optax
from flax.training import trainstate
class CNN(nn.Module):
"""Simple Convolutional Neural Network."""
numclasses: int = 10
@nn.compact
def call(self, x, training: bool = True):
x = nn.Conv(features=32, kernelsize=(3, 3))(x)
x = nn.relu(x)
x = nn.maxpool(x, windowshape=(2, 2), strides=(2, 2))
x = nn.Conv(features=64, kernelsize=(3, 3))(x)
x = nn.relu(x)
x = nn.maxpool(x, windowshape=(2, 2), strides=(2, 2))
x = x.reshape((x.shape[0], -1)) # Flatten
x = nn.Dense(features=256)(x)
x = nn.relu(x)
x = nn.Dropout(rate=0.5)(x, deterministic=not training)
x = nn.Dense(features=self.numclasses)(x)
return x
def createtrainstate(key, learningrate, model):
"""Initialize training state."""
dummyinput = jnp.ones([1, 28, 28, 1])
variables = model.init(key, dummyinput)
tx = optax.adam(learningrate)
return trainstate.TrainState.create(
applyfn=model.apply,
params=variables['params'],
tx=tx
)
@jax.jit
def trainstep(state, batch, dropoutkey):
"""Training step with Flax and Optax."""
images, labels = batch
def lossfn(params):
logits = state.applyfn(
{'params': params},
images,
training=True,
rngs={'dropout': dropoutkey}
)
onehot = jax.nn.onehot(labels, 10)
loss = optax.softmaxcrossentropy(logits, onehot).mean()
return loss, logits
gradfn = jax.valueandgrad(lossfn, hasaux=True)
(loss, logits), grads = gradfn(state.params)
state = state.applygradients(grads=grads)
accuracy = jnp.mean(jnp.argmax(logits, axis=-1) == labels)
return state, loss, accuracy
@jax.jit
def evalstep(state, batch):
"""Evaluation step."""
images, labels = batch
logits = state.applyfn(
{'params': state.params},
images,
training=False
)
accuracy = jnp.mean(jnp.argmax(logits, axis=-1) == labels)
return accuracy
Setup
model = CNN(numclasses=10)
key = jax.random.PRNGKey(42)
state = createtrainstate(key, learningrate=1e-3, model=model)
print(f"Model parameters: {sum(p.size for p in jax.tree.leaves(state.params)):,}")
Custom Gradients and Checkpointing
For very large models, you can use gradient checkpointing to save memory:
import jax
import jax.numpy as jnp
from jax import checkpoint
def heavycomputation(x):
"""Function with many intermediate values."""
for in range(10):
x = jnp.tanh(x @ jnp.ones((x.shape[-1], x.shape[-1])) 0.1)
return jnp.sum(x * 2)
With checkpointing (saves memory, slightly slower)
checkpointedfn = checkpoint(heavycomputation)
key = jax.random.PRNGKey(0)
x = jax.random.normal(key, (100, 100))
gradnormal = jax.grad(heavycomputation)(x)
gradcheckpointed = jax.grad(checkpointedfn)(x)
print(f"Gradients match: {jnp.allclose(gradnormal, gradcheckpointed)}")
Custom JVP and VJP
JAX allows you to define custom forward-mode (JVP) and reverse-mode (VJP) differentiation rules:
import jax
import jax.numpy as jnp
from jax import customjvp
@customjvp
def safelog(x):
"""Numerically safe logarithm."""
return jnp.log(x)
@safelog.defjvp
def safelogjvp(primals, tangents):
x, = primals
t, = tangents
# Clamp gradient to avoid exploding gradients
safex = jnp.maximum(x, 1e-7)
return jnp.log(safex), t / safex
Test
x = jnp.array(0.0001)
print(f"safelog(x) = {safelog(x)}")
print(f"grad safelog(x) = {jax.grad(safelog)(x)}")
PyTrees and Nested Structures
JAX works natively with nested data structures (pytrees):
import jax
import jax.numpy as jnp
Model parameters as nested dict
params = {
'layer1': {
'weights': jnp.ones((3, 4)),
'bias': jnp.zeros(4)
},
'layer2': {
'weights': jnp.ones((4, 2)),
'bias': jnp.zeros(2)
}
}
treemap: apply function to all leaves
scaledparams = jax.tree.map(lambda x: x 0.1, params)
print(f"Scaled weights: {scaledparams['layer1']['weights'][0]}")
treemap with multiple trees
grads = jax.tree.map(jnp.oneslike, params) # Dummy gradients
lr = 0.01
updatedparams = jax.tree.map(
lambda p, g: p - lr g,
params, grads
)
Tree structure information
leaves = jax.tree.leaves(params)
print(f"Number of parameter arrays: {len(leaves)}")
totalparams = sum(l.size for l in leaves)
print(f"Total parameters: {totalparams}")
Scan: Efficient Loops
jax.lax.scan replaces Python loops with JIT-compilable operations:
import jax
import jax.numpy as jnp
RNN-style computation using scan
def rnnstep(carry, x):
"""Single RNN step."""
h = carry
hnew = jnp.tanh(h 0.5 + x 0.3 + 0.1)
return hnew, hnew # (newcarry, output)
Initialization
h0 = jnp.zeros(5)
xs = jax.random.normal(jax.random.PRNGKey(0), (20, 5)) # Sequence length 20
Run scan
finalh, allh = jax.lax.scan(rnnstep, h0, xs)
print(f"Final hidden state shape: {finalh.shape}")
print(f"All hidden states shape: {allh.shape}")
Cumulative sum using scan
def cumsumstep(carry, x):
newcarry = carry + x
return newcarry, newcarry
, cumulative = jax.lax.scan(cumsumstep, 0.0, jnp.array([1., 2., 3., 4., 5.]))
print(f"Cumulative sum: {cumulative}")
Best Practices
1. Use jax.jit for Functions Called Repeatedly
# Good: JIT at the right level
@jax.jit
def trainstep(params, batch):
loss, grads = jax.valueandgrad(lossfn)(params, batch)
return updateparams(params, grads), loss
Less optimal: JIT too granular
def trainstep(params, batch):
loss = jax.jit(lossfn)(params, batch) # Compilation overhead each call
grads = jax.jit(jax.grad(lossfn))(params, batch)
return updateparams(params, grads), loss
2. Avoid Python Control Flow Inside JIT
# Bad: Python if inside jit
@jax.jit
def badfn(x):
if x > 0: # Error! x is a traced value
return x
return -x
Good: Use jnp.where
@jax.jit
def goodfn(x):
return jnp.where(x > 0, x, -x)
Or use lax.cond for complex conditionals
@jax.jit
def complexcond(x):
return jax.lax.cond(
x > 0,
lambda x: jnp.sin(x),
lambda x: jnp.cos(x),
x
)
3. Manage PRNG Keys Properly
# Bad: Reusing the same key
key = jax.random.PRNGKey(0)
a = jax.random.normal(key, (3,))
b = jax.random.normal(key, (3,)) # Same as a!
Good: Split key each time
key = jax.random.PRNGKey(0)
key, k1, k2 = jax.random.split(key, 3)
a = jax.random.normal(k1, (3,))
b = jax.random.normal(k2, (3,)) # Different from a
4. Use donateargnums to Save Memory
@jax.jit
def update(params, grads):
return jax.tree.map(lambda p, g: p - 0.01 g, params, grads)
With donate: input buffers are reused for output
updatedonated = jax.jit(update, donateargnums=(0,))
5. Profile and Debug
# Check if a function is compiled
from jax import makejaxpr
def myfn(x, y):
return jnp.dot(x, y) + jnp.sin(x)
View intermediate representation (jaxpr)
x = jnp.ones((3, 3))
y = jnp.ones((3, 3))
print(jax.makejaxpr(myfn)(x, y))
Profiling with jax.profiler
jax.profiler.starttrace("/tmp/jax-trace")
... run code ...
jax.profiler.stoptrace()
6. Batch Data with jax.vmap
# Instead of manual loops:
def processone(x):
return jnp.sum(x 2)
vmap is more efficient and JIT-compatible
processbatch = jax.vmap(processone)
batch = jax.random.normal(jax.random.PRNGKey(0), (100, 10))
results = processbatch(batch)
Performance Tips
.blockuntilready() when benchmarking because JAX uses asynchronous dispatch.jnp.float32 as the default. Use jnp.bfloat16 for faster training on TPUs.jax.lax.scan instead of Python loops for sequence processing.JAX vs PyTorch vs TensorFlow Comparison
| Aspect | JAX | PyTorch | TensorFlow |
|--------|-----|---------|------------|
| Paradigm | Functional | Imperative (OOP) | Graph-based / Eager |
| Auto-diff | jax.grad (functional) | loss.backward() | tf.GradientTape |
| Compilation | jax.jit (XLA) | torch.compile | tf.function (XLA) |
| GPU Support | Native via XLA | Native CUDA | Native via XLA |
| TPU Support | Native | Limited | Native |
| Learning Curve | Medium | Low | Medium-High |
| Ecosystem | Flax, Optax, Orbax | torchvision, HuggingFace | Keras, TF Hub |
Conclusion
JAX provides a unique combination of ease of use (familiar NumPy API), high performance (XLA compilation), and flexibility (composable function transformations). Key takeaways to remember:
jax.numpy as a NumPy replacement to get free hardware acceleration.jax.grad for clean and composable automatic differentiation.jax.jit to frequently called functions for significant speedups.jax.vmap to avoid manual loops and vectorize computations.JAX is an excellent choice for ML research, large model training, and scientific computing that demands high performance. With a continuously growing ecosystem, JAX has become one of the most powerful ML framework choices in the industry today.