Complete JAX Tutorial: Google's High-Performance Machine Learning Framework

# Tutorial Lengkap JAX: Framework Machine Learning Berkinerja Tinggi dari Google JAX adalah library open-source dari Google yang menggabungkan NumPy dengan kemampuan diferensiasi otomatis (automatic...

By Ruby Abdullah · · tutorial
JAXMachine LearningGoogleXLADeep Learning

Complete JAX Tutorial: Google's High-Performance Machine Learning Framework

JAX is an open-source library from Google that combines NumPy with automatic differentiation capabilities and hardware acceleration through XLA (Accelerated Linear Algebra). JAX allows you to write numerical code with familiar NumPy syntax while achieving significantly higher performance through JIT compilation and native support for GPUs and TPUs.

In this tutorial, we will learn JAX from the ground up through advanced usage, including function transformations, neural network training, and performance optimization for large-scale machine learning workloads.

Why JAX?

JAX has become the framework of choice for many ML research and production teams for several reasons:

  • NumPy Compatibility: The jax.numpy API is nearly identical to NumPy, making the learning curve extremely low for Python users already familiar with the scientific computing ecosystem.
  • Automatic Differentiation: The jax.grad function enables automatic differentiation of regular Python functions, supporting both forward-mode and reverse-mode differentiation.
  • JIT Compilation: Through jax.jit, Python functions are compiled into XLA-optimized code for the target hardware (CPU, GPU, or TPU).
  • Automatic Vectorization: jax.vmap enables automatic vectorization of functions without writing manual loops.
  • Parallelization: jax.pmap provides transparent multi-device parallelization.
  • Rich Ecosystem: Libraries like Flax, Optax, and Orbax provide high-level abstractions for neural networks, optimizers, and checkpointing.

Installation

Basic Installation (CPU)

pip install jax

Installation with GPU Support (CUDA 12)

pip install jax[cuda12]

Installation with TPU Support

pip install jax[tpu] -f https://storage.googleapis.com/jax-releases/libtpureleases.html

Verify Installation

import jax

import jax.numpy as jnp

print(f"JAX version: {jax.version}")

print(f"Available devices: {jax.devices()}")

print(f"Default backend: {jax.defaultbackend()}")

Simple test

x = jnp.array([1.0, 2.0, 3.0])

print(f"Array: {x}")

print(f"Device: {x.devices()}")

Expected output:

JAX version: 0.4.x

Available devices: [CpuDevice(id=0)]

Default backend: cpu

Array: [1. 2. 3.]

Basic Usage

JAX NumPy: Arrays and Basic Operations

JAX provides jax.numpy as a drop-in replacement for NumPy with a nearly identical API:

import jax.numpy as jnp

Creating arrays

a = jnp.array([1.0, 2.0, 3.0, 4.0])

b = jnp.ones((3, 3))

c = jnp.zeros((2, 4))

d = jnp.linspace(0, 1, 5)

Mathematical operations

result = jnp.dot(a, a)

print(f"Dot product: {result}")

Matrix operations

matrix = jnp.array([[1, 2], [3, 4]])

eigenvalues = jnp.linalg.eigvalsh(matrix)

print(f"Eigenvalues: {eigenvalues}")

Statistics

data = jnp.array([1.0, 2.0, 3.0, 4.0, 5.0])

print(f"Mean: {jnp.mean(data)}")

print(f"Std: {jnp.std(data)}")

Key Difference: Immutability

Unlike NumPy, JAX arrays are immutable. You cannot modify elements in-place:

import jax.numpy as jnp

x = jnp.array([1, 2, 3, 4, 5])

NumPy style (NOT possible in JAX):

x[0] = 10 # Error!

JAX style using .at[].set():

xupdated = x.at[0].set(10)

print(f"Original: {x}") # [1, 2, 3, 4, 5]

print(f"Updated: {xupdated}") # [10, 2, 3, 4, 5]

Other .at operations:

xadd = x.at[1].add(100) # Add 100 to index 1

xmul = x.at[2].multiply(3) # Multiply index 2 by 3

print(f"After add: {xadd}")

print(f"After multiply: {xmul}")

Random Number Generation

JAX uses an explicit and reproducible PRNG (Pseudo-Random Number Generator) system:

import jax

import jax.numpy as jnp

Create PRNG key

key = jax.random.PRNGKey(42)

Each random operation requires a separate key

key1, key2, key3 = jax.random.split(key, 3)

Generate random numbers

uniform = jax.random.uniform(key1, shape=(3,))

normal = jax.random.normal(key2, shape=(3,))

integers = jax.random.randint(key3, shape=(5,), minval=0, maxval=10)

print(f"Uniform: {uniform}")

print(f"Normal: {normal}")

print(f"Integers: {integers}")

Common pattern: split key at each iteration

key = jax.random.PRNGKey(0)

for i in range(3):

key, subkey = jax.random.split(key)

sample = jax.random.normal(subkey, shape=(2,))

print(f"Iteration {i}: {sample}")

Function Transformations

The core power of JAX lies in its ability to transform regular Python functions. There are four main transformations: grad, jit, vmap, and pmap.

Automatic Differentiation with jax.grad

import jax

import jax.numpy as jnp

Simple function

def f(x):

return jnp.sin(x) 2 + jnp.cos(x)

Gradient (first derivative)

df = jax.grad(f)

print(f"f(1.0) = {f(1.0):.4f}")

print(f"f'(1.0) = {df(1.0):.4f}")

Second derivative

d2f = jax.grad(jax.grad(f))

print(f"f''(1.0) = {d2f(1.0):.4f}")

Gradient for multi-variable functions

def g(x, y):

return x 2 y + jnp.exp(y)

Gradient with respect to first argument (default)

dgdx = jax.grad(g, argnums=0)

print(f"dg/dx(2.0, 1.0) = {dgdx(2.0, 1.0):.4f}")

Gradient with respect to both arguments

dgboth = jax.grad(g, argnums=(0, 1))

grads = dgboth(2.0, 1.0)

print(f"dg/dx = {grads[0]:.4f}, dg/dy = {grads[1]:.4f}")

Gradients for ML Loss Functions

import jax

import jax.numpy as jnp

def mseloss(params, X, y):

"""Mean Squared Error loss for linear regression."""

w, b = params

predictions = X @ w + b

return jnp.mean((predictions - y) 2)

Dummy data

key = jax.random.PRNGKey(42)

X = jax.random.normal(key, (100, 3))

truew = jnp.array([1.0, -2.0, 0.5])

trueb = 1.0

y = X @ truew + trueb

Initialize parameters

key, subkey = jax.random.split(key)

w = jax.random.normal(subkey, (3,))

b = jnp.array(0.0)

params = (w, b)

Compute gradients

gradfn = jax.grad(mseloss)

grads = gradfn(params, X, y)

print(f"Gradient w: {grads[0]}")

print(f"Gradient b: {grads[1]}")

Value and gradient simultaneously

lossandgrad = jax.valueandgrad(mseloss)

lossval, grads = lossandgrad(params, X, y)

print(f"Loss: {lossval:.4f}")

Just-In-Time Compilation with jax.jit

jax.jit compiles Python functions into optimized XLA code:
import jax

import jax.numpy as jnp

import time

def slowfunction(x):

"""Function with many operations."""

for in range(100):

x = jnp.sin(x) + jnp.cos(x) 0.1

return x

JIT-compiled version

fastfunction = jax.jit(slowfunction)

x = jnp.ones((1000, 1000))

Warmup (first compilation)

= fastfunction(x).blockuntilready()

Benchmark without JIT

start = time.time()

= slowfunction(x).blockuntilready()

nojittime = time.time() - start

Benchmark with JIT

start = time.time()

= fastfunction(x).blockuntilready()

jittime = time.time() - start

print(f"Without JIT: {nojittime:.4f}s")

print(f"With JIT: {jittime:.4f}s")

print(f"Speedup: {nojittime / jittime:.1f}x")

Automatic Vectorization with jax.vmap

jax.vmap lets you write functions for a single sample and automatically vectorize them for batches:
import jax

import jax.numpy as jnp

Function for a single sample

def predictsingle(params, x):

w, b = params

return jnp.dot(w, x) + b

Data

key = jax.random.PRNGKey(42)

w = jax.random.normal(key, (5,))

b = jnp.array(0.0)

params = (w, b)

Single input

xsingle = jnp.ones(5)

print(f"Single prediction: {predictsingle(params, xsingle)}")

Batch input using vmap

predictbatch = jax.vmap(predictsingle, inaxes=(None, 0))

Xbatch = jax.random.normal(key, (10, 5))

predictions = predictbatch(params, Xbatch)

print(f"Batch predictions shape: {predictions.shape}")

print(f"Batch predictions: {predictions}")

vmap for computing Jacobians

def f(x):

return jnp.array([x[0] * 2 + x[1], jnp.sin(x[0]) x[1]])

Jacobian using vmap + grad

jacobian = jax.jacobian(f)

x = jnp.array([1.0, 2.0])

J = jacobian(x)

print(f"Jacobian:\n{J}")

Advanced Usage

Training a Neural Network from Scratch

Here is a complete example of training a simple neural network using pure JAX:

import jax

import jax.numpy as jnp

from typing import List, Tuple

def initmlp(layersizes: List[int], key):

"""Initialize MLP parameters with Xavier initialization."""

params = []

for i in range(len(layersizes) - 1):

key, wkey, bkey = jax.random.split(key, 3)

scale = jnp.sqrt(2.0 / layersizes[i])

w = jax.random.normal(wkey, (layersizes[i], layersizes[i + 1])) scale

b = jnp.zeros(layersizes[i + 1])

params.append((w, b))

return params

def forward(params, x):

"""MLP forward pass with ReLU activation."""

for w, b in params[:-1]:

x = jnp.maximum(0, x @ w + b) # ReLU

w, b = params[-1]

return x @ w + b # Output layer without activation

def crossentropyloss(params, X, y):

"""Cross-entropy loss for classification."""

logits = forward(params, X)

logprobs = jax.nn.logsoftmax(logits)

onehot = jax.nn.onehot(y, logits.shape[-1])

return -jnp.mean(jnp.sum(onehot logprobs, axis=-1))

def accuracy(params, X, y):

"""Compute prediction accuracy."""

logits = forward(params, X)

predicted = jnp.argmax(logits, axis=-1)

return jnp.mean(predicted == y)

@jax.jit

def trainstep(params, X, y, lr):

"""Single training step with gradient descent."""

loss, grads = jax.valueandgrad(crossentropyloss)(params, X, y)

params = jax.tree.map(lambda p, g: p - lr g, params, grads)

return params, loss

Generate synthetic dataset

key = jax.random.PRNGKey(42)

nsamples = 1000

nfeatures = 20

nclasses = 3

key, datakey = jax.random.split(key)

X = jax.random.normal(datakey, (nsamples, nfeatures))

truew = jax.random.normal(key, (nfeatures, nclasses))

logits = X @ truew

y = jnp.argmax(logits, axis=-1)

Split data

Xtrain, Xtest = X[:800], X[800:]

ytrain, ytest = y[:800], y[800:]

Initialize model

key = jax.random.PRNGKey(0)

params = initmlp([nfeatures, 64, 32, nclasses], key)

Training loop

lr = 0.01

for epoch in range(100):

params, loss = trainstep(params, Xtrain, ytrain, lr)

if (epoch + 1) % 20 == 0:

trainacc = accuracy(params, Xtrain, ytrain)

testacc = accuracy(params, Xtest, ytest)

print(f"Epoch {epoch+1}: loss={loss:.4f}, "

f"trainacc={trainacc:.4f}, testacc={testacc:.4f}")

Using JAX with Flax and Optax

For larger projects, use Flax (neural network library) and Optax (optimizer library):

import jax

import jax.numpy as jnp

import flax.linen as nn

import optax

from flax.training import trainstate

class CNN(nn.Module):

"""Simple Convolutional Neural Network."""

numclasses: int = 10

@nn.compact

def call(self, x, training: bool = True):

x = nn.Conv(features=32, kernelsize=(3, 3))(x)

x = nn.relu(x)

x = nn.maxpool(x, windowshape=(2, 2), strides=(2, 2))

x = nn.Conv(features=64, kernelsize=(3, 3))(x)

x = nn.relu(x)

x = nn.maxpool(x, windowshape=(2, 2), strides=(2, 2))

x = x.reshape((x.shape[0], -1)) # Flatten

x = nn.Dense(features=256)(x)

x = nn.relu(x)

x = nn.Dropout(rate=0.5)(x, deterministic=not training)

x = nn.Dense(features=self.numclasses)(x)

return x

def createtrainstate(key, learningrate, model):

"""Initialize training state."""

dummyinput = jnp.ones([1, 28, 28, 1])

variables = model.init(key, dummyinput)

tx = optax.adam(learningrate)

return trainstate.TrainState.create(

applyfn=model.apply,

params=variables['params'],

tx=tx

)

@jax.jit

def trainstep(state, batch, dropoutkey):

"""Training step with Flax and Optax."""

images, labels = batch

def lossfn(params):

logits = state.applyfn(

{'params': params},

images,

training=True,

rngs={'dropout': dropoutkey}

)

onehot = jax.nn.onehot(labels, 10)

loss = optax.softmaxcrossentropy(logits, onehot).mean()

return loss, logits

gradfn = jax.valueandgrad(lossfn, hasaux=True)

(loss, logits), grads = gradfn(state.params)

state = state.applygradients(grads=grads)

accuracy = jnp.mean(jnp.argmax(logits, axis=-1) == labels)

return state, loss, accuracy

@jax.jit

def evalstep(state, batch):

"""Evaluation step."""

images, labels = batch

logits = state.applyfn(

{'params': state.params},

images,

training=False

)

accuracy = jnp.mean(jnp.argmax(logits, axis=-1) == labels)

return accuracy

Setup

model = CNN(numclasses=10)

key = jax.random.PRNGKey(42)

state = createtrainstate(key, learningrate=1e-3, model=model)

print(f"Model parameters: {sum(p.size for p in jax.tree.leaves(state.params)):,}")

Custom Gradients and Checkpointing

For very large models, you can use gradient checkpointing to save memory:

import jax

import jax.numpy as jnp

from jax import checkpoint

def heavycomputation(x):

"""Function with many intermediate values."""

for in range(10):

x = jnp.tanh(x @ jnp.ones((x.shape[-1], x.shape[-1])) 0.1)

return jnp.sum(x * 2)

With checkpointing (saves memory, slightly slower)

checkpointedfn = checkpoint(heavycomputation)

key = jax.random.PRNGKey(0)

x = jax.random.normal(key, (100, 100))

gradnormal = jax.grad(heavycomputation)(x)

gradcheckpointed = jax.grad(checkpointedfn)(x)

print(f"Gradients match: {jnp.allclose(gradnormal, gradcheckpointed)}")

Custom JVP and VJP

JAX allows you to define custom forward-mode (JVP) and reverse-mode (VJP) differentiation rules:

import jax

import jax.numpy as jnp

from jax import customjvp

@customjvp

def safelog(x):

"""Numerically safe logarithm."""

return jnp.log(x)

@safelog.defjvp

def safelogjvp(primals, tangents):

x, = primals

t, = tangents

# Clamp gradient to avoid exploding gradients

safex = jnp.maximum(x, 1e-7)

return jnp.log(safex), t / safex

Test

x = jnp.array(0.0001)

print(f"safelog(x) = {safelog(x)}")

print(f"grad safelog(x) = {jax.grad(safelog)(x)}")

PyTrees and Nested Structures

JAX works natively with nested data structures (pytrees):

import jax

import jax.numpy as jnp

Model parameters as nested dict

params = {

'layer1': {

'weights': jnp.ones((3, 4)),

'bias': jnp.zeros(4)

},

'layer2': {

'weights': jnp.ones((4, 2)),

'bias': jnp.zeros(2)

}

}

treemap: apply function to all leaves

scaledparams = jax.tree.map(lambda x: x 0.1, params)

print(f"Scaled weights: {scaledparams['layer1']['weights'][0]}")

treemap with multiple trees

grads = jax.tree.map(jnp.oneslike, params) # Dummy gradients

lr = 0.01

updatedparams = jax.tree.map(

lambda p, g: p - lr g,

params, grads

)

Tree structure information

leaves = jax.tree.leaves(params)

print(f"Number of parameter arrays: {len(leaves)}")

totalparams = sum(l.size for l in leaves)

print(f"Total parameters: {totalparams}")

Scan: Efficient Loops

jax.lax.scan replaces Python loops with JIT-compilable operations:
import jax

import jax.numpy as jnp

RNN-style computation using scan

def rnnstep(carry, x):

"""Single RNN step."""

h = carry

hnew = jnp.tanh(h 0.5 + x 0.3 + 0.1)

return hnew, hnew # (newcarry, output)

Initialization

h0 = jnp.zeros(5)

xs = jax.random.normal(jax.random.PRNGKey(0), (20, 5)) # Sequence length 20

Run scan

finalh, allh = jax.lax.scan(rnnstep, h0, xs)

print(f"Final hidden state shape: {finalh.shape}")

print(f"All hidden states shape: {allh.shape}")

Cumulative sum using scan

def cumsumstep(carry, x):

newcarry = carry + x

return newcarry, newcarry

, cumulative = jax.lax.scan(cumsumstep, 0.0, jnp.array([1., 2., 3., 4., 5.]))

print(f"Cumulative sum: {cumulative}")

Best Practices

1. Use jax.jit for Functions Called Repeatedly

# Good: JIT at the right level

@jax.jit

def trainstep(params, batch):

loss, grads = jax.valueandgrad(lossfn)(params, batch)

return updateparams(params, grads), loss

Less optimal: JIT too granular

def trainstep(params, batch):

loss = jax.jit(lossfn)(params, batch) # Compilation overhead each call

grads = jax.jit(jax.grad(lossfn))(params, batch)

return updateparams(params, grads), loss

2. Avoid Python Control Flow Inside JIT

# Bad: Python if inside jit

@jax.jit

def badfn(x):

if x > 0: # Error! x is a traced value

return x

return -x

Good: Use jnp.where

@jax.jit

def goodfn(x):

return jnp.where(x > 0, x, -x)

Or use lax.cond for complex conditionals

@jax.jit

def complexcond(x):

return jax.lax.cond(

x > 0,

lambda x: jnp.sin(x),

lambda x: jnp.cos(x),

x

)

3. Manage PRNG Keys Properly

# Bad: Reusing the same key

key = jax.random.PRNGKey(0)

a = jax.random.normal(key, (3,))

b = jax.random.normal(key, (3,)) # Same as a!

Good: Split key each time

key = jax.random.PRNGKey(0)

key, k1, k2 = jax.random.split(key, 3)

a = jax.random.normal(k1, (3,))

b = jax.random.normal(k2, (3,)) # Different from a

4. Use donateargnums to Save Memory

@jax.jit

def update(params, grads):

return jax.tree.map(lambda p, g: p - 0.01 g, params, grads)

With donate: input buffers are reused for output

updatedonated = jax.jit(update, donateargnums=(0,))

5. Profile and Debug

# Check if a function is compiled

from jax import makejaxpr

def myfn(x, y):

return jnp.dot(x, y) + jnp.sin(x)

View intermediate representation (jaxpr)

x = jnp.ones((3, 3))

y = jnp.ones((3, 3))

print(jax.makejaxpr(myfn)(x, y))

Profiling with jax.profiler

jax.profiler.starttrace("/tmp/jax-trace")

... run code ...

jax.profiler.stoptrace()

6. Batch Data with jax.vmap

# Instead of manual loops:

def processone(x):

return jnp.sum(x 2)

vmap is more efficient and JIT-compatible

processbatch = jax.vmap(processone)

batch = jax.random.normal(jax.random.PRNGKey(0), (100, 10))

results = processbatch(batch)

Performance Tips

  • Always call .blockuntilready() when benchmarking because JAX uses asynchronous dispatch.
  • Avoid repeated data transfers between host (CPU) and device (GPU/TPU). Keep all computation on-device.
  • Use jnp.float32 as the default. Use jnp.bfloat16 for faster training on TPUs.
  • Batch small operations into larger ones to maximize GPU/TPU throughput.
  • Use jax.lax.scan instead of Python loops for sequence processing.
  • JAX vs PyTorch vs TensorFlow Comparison

    | Aspect | JAX | PyTorch | TensorFlow |

    |--------|-----|---------|------------|

    | Paradigm | Functional | Imperative (OOP) | Graph-based / Eager |

    | Auto-diff | jax.grad (functional) | loss.backward() | tf.GradientTape |

    | Compilation | jax.jit (XLA) | torch.compile | tf.function (XLA) |

    | GPU Support | Native via XLA | Native CUDA | Native via XLA |

    | TPU Support | Native | Limited | Native |

    | Learning Curve | Medium | Low | Medium-High |

    | Ecosystem | Flax, Optax, Orbax | torchvision, HuggingFace | Keras, TF Hub |

    Conclusion

    JAX provides a unique combination of ease of use (familiar NumPy API), high performance (XLA compilation), and flexibility (composable function transformations). Key takeaways to remember:

  • Start with jax.numpy as a NumPy replacement to get free hardware acceleration.
  • Use jax.grad for clean and composable automatic differentiation.
  • Apply jax.jit to frequently called functions for significant speedups.
  • Leverage jax.vmap to avoid manual loops and vectorize computations.
  • Use Flax + Optax for more structured neural network projects.
  • Be mindful of immutability** and PRNG key management as the two main differences from NumPy.
  • JAX is an excellent choice for ML research, large model training, and scientific computing that demands high performance. With a continuously growing ecosystem, JAX has become one of the most powerful ML framework choices in the industry today.

    Related Articles

    Complete Keras 3 Tutorial: Multi-Backend Deep Learning Framework for TensorFlow, JAX, and PyTorch

    Tutorial Lengkap Keras 3: Framework Deep Learning Multi-Backend untuk TensorFlow, JAX, dan PyTorch Keras 3 adalah evolus...

    Complete Guide to PyTorch Lightning: Deep Learning Made Simple

    Panduan Lengkap PyTorch Lightning: Deep Learning Made Simple PyTorch Lightning adalah framework high-level untuk PyTorch...

    AutoGluon: A Complete Guide to AWS's AutoML Library for Real-World Machine Learning

    Tutorial Lengkap AutoGluon: Library AutoML dari AWS untuk Machine Learning Otomatis AutoGluon adalah library open-source...

    Complete LM Evaluation Harness Tutorial: The Standard Framework for Evaluating Large Language Models

    Tutorial Lengkap LM Evaluation Harness: Framework Standar untuk Evaluasi Model Bahasa Besar Evaluasi model bahasa besar ...