Complete Google Gemma Tutorial: Open-Source Models from Google DeepMind
Google Gemma is a family of open-source language models developed by Google DeepMind. Built with the same technology and research behind the Gemini models, Gemma offers highly competitive performance in lightweight and efficient packages. This tutorial covers how to use Gemma for various purposes, from simple inference to fine-tuning for specific needs.
Why Gemma?
Gemma comes in several size variants (2B, 7B, 9B, 27B parameters) designed to run efficiently on diverse hardware. Key advantages of Gemma compared to other open-source models:
- High performance per parameter: Gemma consistently outperforms models of similar size across various benchmarks
- Open license: Can be used for both commercial and research purposes
- Broad ecosystem support: Compatible with Hugging Face Transformers, vLLM, Ollama, llama.cpp, and many other frameworks
- Specialized variants: CodeGemma for coding, PaliGemma for vision, and RecurrentGemma for memory efficiency
- Hardware optimization: Optimized for NVIDIA GPUs, Google TPUs, and even CPUs
Installation and Setup
Prerequisites
Make sure you have Python 3.9 or newer and pip installed.
python --version
pip --version
Library Installation
pip install torch transformers accelerate bitsandbytes
pip install huggingfacehub
To access Gemma models, you need a Hugging Face account and must accept the Gemma license agreement:
pip install huggingfacehub
huggingface-cli login
Enter your Hugging Face token when prompted. You can get your token from Settings > Access Tokens on the Hugging Face website.
GPU Verification (Optional but Recommended)
import torch
print(f"CUDA available: {torch.cuda.isavailable()}")
if torch.cuda.isavailable():
print(f"GPU: {torch.cuda.getdevicename(0)}")
print(f"Memory: {torch.cuda.getdeviceproperties(0).totalmem / 1e9:.1f} GB")
Basic Usage: Text Generation
Using Pipeline (Easiest Approach)
from transformers import pipeline
generator = pipeline(
"text-generation",
model="google/gemma-2-9b-it",
devicemap="auto",
torchdtype="auto"
)
prompt = "Explain the concept of machine learning in 3 paragraphs:"
output = generator(
prompt,
maxnewtokens=512,
temperature=0.7,
topp=0.9,
dosample=True
)
print(output[0]["generatedtext"])
Using Model and Tokenizer Directly
For more granular control, you can load the model and tokenizer separately:
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
modelid = "google/gemma-2-9b-it"
tokenizer = AutoTokenizer.frompretrained(modelid)
model = AutoModelForCausalLM.frompretrained(
modelid,
devicemap="auto",
torchdtype=torch.bfloat16
)
prompt = "What is the difference between supervised and unsupervised learning?"
inputs = tokenizer(prompt, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
inputs,
maxnewtokens=256,
temperature=0.7,
topp=0.9,
dosample=True
)
response = tokenizer.decode(outputs[0], skipspecialtokens=True)
print(response)
Chat Template Format
Gemma instruction-tuned models (the -it variants) use a specific chat format. Use the built-in chat template for optimal results:
messages = [
{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"},
]
inputtext = tokenizer.applychattemplate(
messages,
tokenize=False,
addgenerationprompt=True
)
inputs = tokenizer(inputtext, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
inputs,
maxnewtokens=512,
temperature=0.3
)
response = tokenizer.decode(outputs[0][inputs["inputids"].shape[-1]:], skipspecialtokens=True)
print(response)
Multi-turn Conversation
conversation = [
{"role": "user", "content": "What is a neural network?"},
{"role": "assistant", "content": "A neural network is a computational model inspired by how the human brain works. It consists of layers of artificial neurons that are interconnected and can learn patterns from data."},
{"role": "user", "content": "How do you train one?"},
]
inputtext = tokenizer.applychattemplate(
conversation,
tokenize=False,
addgenerationprompt=True
)
inputs = tokenizer(inputtext, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(inputs, maxnewtokens=512, temperature=0.7)
response = tokenizer.decode(outputs[0][inputs["inputids"].shape[-1]:], skipspecialtokens=True)
print(response)
Quantization: Running Gemma on Limited Hardware
4-bit Quantization with BitsAndBytes
Quantization allows you to run large models on GPUs with limited VRAM:
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch
quantizationconfig = BitsAndBytesConfig(
loadin4bit=True,
bnb4bitcomputedtype=torch.bfloat16,
bnb4bitusedoublequant=True,
bnb4bitquanttype="nf4"
)
modelid = "google/gemma-2-27b-it"
tokenizer = AutoTokenizer.frompretrained(modelid)
model = AutoModelForCausalLM.frompretrained(
modelid,
quantizationconfig=quantizationconfig,
devicemap="auto"
)
print(f"Model loaded. Memory usage: {model.getmemoryfootprint() / 1e9:.2f} GB")
With 4-bit quantization, Gemma 27B which normally requires around 54GB of VRAM can run with only about 16GB of VRAM.
8-bit Quantization
If you want a balance between quality and memory efficiency:
quantizationconfig = BitsAndBytesConfig(
load
in8bit=True
)
model = AutoModelForCausalLM.from
pretrained(
"google/gemma-2-9b-it",
quantizationconfig=quantizationconfig,
devicemap="auto"
)
VRAM Requirements Comparison
| Model | FP16 | 8-bit | 4-bit |
|-------|------|-------|-------|
| Gemma 2B | ~4 GB | ~2.5 GB | ~1.5 GB |
| Gemma 7B | ~14 GB | ~8 GB | ~5 GB |
| Gemma 2 9B | ~18 GB | ~10 GB | ~6 GB |
| Gemma 2 27B | ~54 GB | ~28 GB | ~16 GB |
Using Gemma with Ollama
The easiest way to run Gemma locally:
# Install Ollama (Linux)
curl -fsSL https://ollama.com/install.sh | sh
Download and run Gemma
ollama run gemma2:9b
Or a smaller version
ollama run gemma2:2b
Accessing Ollama from Python
import requests
import json
def chatwithgemma(prompt, model="gemma2:9b"):
response = requests.post(
"http://localhost:11434/api/generate",
json={
"model": model,
"prompt": prompt,
"stream": False,
"options": {
"temperature": 0.7,
"topp": 0.9,
"numpredict": 512
}
}
)
return response.json()["response"]
result = chatwithgemma("Explain how transformer architecture works")
print(result)
Ollama with OpenAI-compatible API
from openai import OpenAI
client = OpenAI(
baseurl="http://localhost:11434/v1",
apikey="ollama"
)
response = client.chat.completions.create(
model="gemma2:9b",
messages=[
{"role": "user", "content": "Write a SQL query to calculate average monthly sales"}
],
temperature=0.3
)
print(response.choices[0].message.content)
Advanced Usage: Fine-tuning Gemma with LoRA
Fine-tuning allows you to customize Gemma for specific tasks. Using LoRA (Low-Rank Adaptation) makes this process efficient and affordable.
Additional Dependencies
pip install peft trl datasets
Preparing the Dataset
from datasets import Dataset
data = {
"instruction": [
"Classify the sentiment of the following review",
"Classify the sentiment of the following review",
"Classify the sentiment of the following review",
"Extract entities from the following text",
],
"input": [
"This product is amazing, fast shipping!",
"Poor quality, doesn't match the description.",
"It's okay, but could be better.",
"John bought a laptop from Amazon yesterday.",
],
"output": [
"Positive",
"Negative",
"Neutral",
"Entities: John (Person), Amazon (Organization), yesterday (Time)",
]
}
dataset = Dataset.fromdict(data)
def formatinstruction(example):
if example["input"]:
text = f"""ofturn>user
{example['instruction']}
{example['input']}ofturn>
ofturn>model
{example['output']}ofturn>"""
else:
text = f"""ofturn>user
{example['instruction']}ofturn>
ofturn>model
{example['output']}ofturn>"""
return {"text": text}
dataset = dataset.map(format
instruction)
print(dataset[0]["text"])
LoRA Configuration and Training
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, getpeftmodel, preparemodelforkbittraining
from trl import SFTTrainer, SFTConfig
import torch
modelid = "google/gemma-2-9b-it"
bnbconfig = BitsAndBytesConfig(
loadin4bit=True,
bnb4bitcomputedtype=torch.bfloat16,
bnb4bitusedoublequant=True,
bnb4bitquanttype="nf4"
)
tokenizer = AutoTokenizer.frompretrained(modelid)
tokenizer.padtoken = tokenizer.eostoken
model = AutoModelForCausalLM.frompretrained(
modelid,
quantizationconfig=bnbconfig,
devicemap="auto"
)
model = preparemodelforkbittraining(model)
loraconfig = LoraConfig(
r=16,
loraalpha=32,
targetmodules=["qproj", "kproj", "vproj", "oproj"],
loradropout=0.05,
bias="none",
tasktype="CAUSALLM"
)
model = getpeftmodel(model, loraconfig)
model.printtrainableparameters()
trainingconfig = SFTConfig(
outputdir="./gemma-finetuned",
numtrainepochs=3,
perdevicetrainbatchsize=4,
gradientaccumulationsteps=4,
learningrate=2e-4,
weightdecay=0.01,
warmupratio=0.1,
loggingsteps=10,
savestrategy="epoch",
bf16=True,
maxseqlength=512,
datasettextfield="text",
)
trainer = SFTTrainer(
model=model,
traindataset=dataset,
args=trainingconfig,
tokenizer=tokenizer,
)
trainer.train()
trainer.savemodel("./gemma-finetuned-final")
tokenizer.savepretrained("./gemma-finetuned-final")
Using the Fine-tuned Model
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
basemodel = AutoModelForCausalLM.frompretrained(
"google/gemma-2-9b-it",
devicemap="auto",
torchdtype=torch.bfloat16
)
model = PeftModel.frompretrained(basemodel, "./gemma-finetuned-final")
tokenizer = AutoTokenizer.frompretrained("./gemma-finetuned-final")
model = model.mergeandunload()
prompt = """ofturn>user
Classify the sentiment of the following review
The customer service at this store was exceptional, I'll definitely come back!ofturn>
ofturn>model
"""
inputs = tokenizer(prompt, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(inputs, maxnewtokens=50, temperature=0.1)
response = tokenizer.decode(outputs[0][inputs["inputids"].shape[-1]:], skipspecialtokens=True)
print(response)
Using Gemma for Specific Tasks
Text Summarization
def summarize(text, maxlength=256):
messages = [
{
"role": "user",
"content": f"Provide a concise summary of the following text:\n\n{text}"
}
]
inputtext = tokenizer.applychattemplate(
messages, tokenize=False, addgenerationprompt=True
)
inputs = tokenizer(inputtext, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
inputs,
maxnewtokens=maxlength,
temperature=0.3,
repetitionpenalty=1.1
)
return tokenizer.decode(
outputs[0][inputs["inputids"].shape[-1]:],
skipspecialtokens=True
)
article = """
The World Health Organization reported a significant breakthrough in malaria
vaccine development. The new vaccine, which completed phase 3 trials across
15 African countries, showed 78% efficacy in preventing severe malaria cases
in children under 5. Health officials project that widespread deployment could
save an estimated 500,000 lives annually.
"""
summary = summarize(article)
print(summary)
Code Generation
def generatecode(taskdescription):
messages = [
{
"role": "user",
"content": f"""Write Python code for: {task
description}
Provide clean code with type hints and usage examples."""
}
]
inputtext = tokenizer.applychattemplate(
messages, tokenize=False, addgenerationprompt=True
)
inputs = tokenizer(inputtext, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
inputs,
maxnewtokens=1024,
temperature=0.2,
topp=0.95
)
return tokenizer.decode(
outputs[0][inputs["inputids"].shape[-1]:],
skipspecialtokens=True
)
code = generatecode("A REST API endpoint for CRUD operations using FastAPI and SQLAlchemy")
print(code)
Structured Data Extraction
import json
def extractentities(text):
messages = [
{
"role": "user",
"content": f"""Extract entities from the following text in JSON format.
Entity categories: person, organization, location, date, amount.
Text: {text}
Provide output only in JSON format like this:
{{"entities": [{{"text": "...", "type": "..."}}]}}"""
}
]
inputtext = tokenizer.applychattemplate(
messages, tokenize=False, addgenerationprompt=True
)
inputs = tokenizer(inputtext, returntensors="pt").to(model.device)
with torch.nograd():
outputs = model.generate(
inputs,
maxnewtokens=512,
temperature=0.1
)
response = tokenizer.decode(
outputs[0][inputs["inputids"].shape[-1]:],
skipspecialtokens=True
)
try:
return json.loads(response)
except json.JSONDecodeError:
return {"rawresponse": response}
text = "Apple Inc. reported $94.8 billion in revenue for Q3 2026 at their headquarters in Cupertino, California."
entities = extractentities(text)
print(json.dumps(entities, indent=2))
Serving Gemma with vLLM
For production deployment, vLLM offers extremely fast inference:
pip install vllm
Running vLLM Server
python -m vllm.entrypoints.openai.apiserver \
--model google/gemma-2-9b-it \
--dtype bfloat16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.9 \
--port 8000
Client for vLLM Server
from openai import OpenAI
client = OpenAI(
baseurl="http://localhost:8000/v1",
apikey="not-needed"
)
response = client.chat.completions.create(
model="google/gemma-2-9b-it",
messages=[
{"role": "user", "content": "Design a microservices architecture for an e-commerce platform"}
],
temperature=0.7,
maxtokens=1024
)
print(response.choices[0].message.content)
Batch Inference with vLLM
from vllm import LLM, SamplingParams
llm = LLM(
model="google/gemma-2-9b-it",
dtype="bfloat16",
maxmodellen=4096
)
samplingparams = SamplingParams(
temperature=0.7,
topp=0.9,
maxtokens=256
)
prompts = [
"What is containerization?",
"Explain the difference between SQL and NoSQL",
"How do you optimize database queries?",
]
outputs = llm.generate(prompts, samplingparams)
for output in outputs:
print(f"Prompt: {output.prompt[:50]}...")
print(f"Response: {output.outputs[0].text}\n")
Best Practices
1. Choose the Right Model Size
- Gemma 2B: Suitable for simple tasks, classification, or edge device deployment
- Gemma 7B/9B: Good balance between performance and efficiency, suitable for most use cases
- Gemma 27B: For complex tasks requiring deep reasoning
2. Optimize Your Prompts
# Less optimal
prompt = "tell me about python"
More optimal - specific and structured
prompt = """Explain the top 3 advantages of Python for data science.
For each advantage, provide an example of a relevant library.
Format: bullet points."""
3. Temperature Settings
- 0.0 - 0.3: For tasks requiring consistency (classification, extraction, code generation)
- 0.5 - 0.7: For creative tasks with constraints (summarization, rewriting)
- 0.8 - 1.0: For free creative tasks (brainstorming, creative writing)
4. Use System Prompts Effectively
messages = [
{
"role": "user",
"content": """You are a data analyst assistant expert in SQL and Python.
Answer the following question with code that can be run directly.
Question: How to calculate customer churn rate from a transactions table?"""
}
]
5. Limit Output with maxnewtokens
Always set maxnewtokens according to your needs to avoid overly long outputs and save inference time.
6. Use Repetition Penalty
To avoid unnecessary text repetition:
outputs = model.generate(
inputs,
maxnewtokens=512,
temperature=0.7,
repetitionpenalty=1.15
)
7. Memory Management
import gc
import torch
After you're done with the model
del model
del tokenizer
gc.collect()
torch.cuda.emptycache()
Gemma Variants Comparison
| Feature | Gemma 1 | Gemma 2 | CodeGemma | PaliGemma |
|---------|---------|---------|-----------|-----------|
| Sizes | 2B, 7B | 2B, 9B, 27B | 2B, 7B | 3B |
| Focus | General | General (improved) | Code | Vision + Language |
| Context Length | 8K | 8K | 8K | Varies |
| Use Case | Chat, text gen | Chat, reasoning | Coding, debugging | Image understanding |
Troubleshooting
OutOfMemoryError
If you get a CUDA out of memory error:
# Use quantization
Or reduce maxseqlength
Or use a smaller model
Check memory usage
print(f"Allocated: {torch.cuda.memoryallocated() / 1e9:.2f} GB")
print(f"Reserved: {torch.cuda.memoryreserved() / 1e9:.2f} GB")
Slow Generation
# Use torch.compile to speed up inference
model = torch.compile(model)
Or use Flash Attention
model = AutoModelForCausalLM.frompretrained(
modelid,
attnimplementation="flashattention2",
torchdtype=torch.bfloat16,
devicemap="auto"
)
Conclusion
Google Gemma is an excellent choice for anyone looking to use high-quality open-source language models. With various size variants and broad ecosystem support, Gemma can be used from laptop experiments to large-scale production deployments.
Key takeaways:
- Choose the model variant based on your needs and available resources
- Use quantization to run large models on limited hardware
- Leverage chat templates for optimal results with instruction-tuned models
- Fine-tuning with LoRA is efficient and affordable
- For production, consider vLLM or Ollama for optimal serving
With mastery of Gemma, you have a strong foundation for building powerful and efficient AI applications.