Complete Guidance Tutorial: Constrained Generation and Structured Output from LLMs
Hey everyone, in this tutorial I want to introduce you to a library that I think gets overlooked way too often even though it is incredibly powerful for working with LLMs. It is called Guidance. This is a library built by Microsoft that focuses on one thing that makes our lives so much easier: controlling and steering the output of language models so it genuinely follows the structure we want. It is not about politely asking through a prompt and then praying the model complies, it is about actually forcing the output to obey the rules we define.
If you have ever been frustrated because your LLM sometimes returns answers in a messy format, sometimes adds words you never asked for, sometimes picks an option outside the list you provided, then Guidance is going to be a really elegant solution for you. I am going to cover this from the very basics, starting with why constrained generation matters, how to install it, basic usage of gen() and select(), all the way to advanced features like regex constraints, grammar or CFG, token healing, and building reusable functions with the @guidance decorator. I will give you Python code examples for everything that you can actually run yourself. Let us get started.
Introduction
Before we dive into the code, let me tell you about the problem that gave birth to this library. Imagine you are building an application that needs an LLM to classify the sentiment of a review as "positive", "negative", or "neutral". The most naive approach is to write a prompt like "Classify the sentiment of this review, answer with a single word only: positive, negative, or neutral." Then we send it to the model and hope that what comes back is just one word from those three options.
The problem is that an LLM is fundamentally a text generator that picks the next token based on probabilities. It has no guarantee that it will obey our instructions. Sometimes it answers "The sentiment of this review is positive." Sometimes it answers "Positive" with a capital letter and extra punctuation. Sometimes it answers "slightly positive" which is not in our list of options at all. And every time the output goes off the rails, our code that parses the result becomes fragile and prone to errors.
The traditional approach to solving this usually involves adding longer and more detailed instructions to the prompt, or parsing manually with regex, or wrapping everything in giant try-except blocks. But all of this is fragile. The moment the model produces slightly different output, our code breaks.
The idea behind Guidance is different from mere prompting. Guidance works at the token generation level. So instead of just asking the model through text, Guidance actually constrains which tokens the model is allowed to produce at each step. If we say the output must be one of three options, then technically Guidance only allows tokens that lead to those three options. The model has no way to produce output outside of that, because invalid tokens have their probability zeroed out.
Besides constrained generation, Guidance also has a concept I really love: we can interleave our program's control flow with the generation process. So inside a single template, we can define which part is static text, which part is generated by the model, which part is constrained to choices, and we can even have if-else branches and loops. This is very different from how we usually call an LLM in a one-shot request-response model. With Guidance, we get to compose programs that are more deterministic and controlled.
Guidance is also token-efficient and often faster, because the parts we have defined as static text do not need to be regenerated by the model. The model only focuses on filling in the parts that actually need to be generated. This is different from a regular chat approach where the model has to regenerate the entire JSON structure or format every single time.
Alright, enough theory, let us jump straight into practice.
Installation
Installation is super easy, just one line with pip.
pip install guidance
Guidance automatically brings the core dependencies you need. But depending on which model you want to use, you might need extra libraries. If you want to use a model from OpenAI, install the openai library too.
pip install guidance openai
If you want to run a local model using transformers from HuggingFace, install torch and transformers.
pip install guidance transformers torch
And if you want to use a model in GGUF format via llama.cpp, which is my personal favorite for local models because it is lightweight and can run on CPU, install llama-cpp-python.
pip install guidance llama-cpp-python
My advice, always use a virtual environment so your project dependencies do not get mixed up with other projects. This is a pattern I follow every time I start a new project.
python -m venv venv
source venv/bin/activate # on Windows: venv\Scripts\activate
pip install guidance llama-cpp-python
One important thing you need to understand from the start: real constrained generation, like regex and grammar constraints, works most optimally on local models where we have access to the token level. API-based models like OpenAI are also supported, but since we do not have full access to their token logits, some constraint features become more limited. So to learn the full feature set, I recommend trying a local model first.
Basic Usage
Alright let us start from the very basics: loading a model. Guidance has a guidance.models module that serves as the entry point for all kinds of models. We will start with a local model using llama.cpp because it is the easiest to experiment with.
from guidance import models
Load a local GGUF model via llama.cpp
lm = models.LlamaCpp(
"models/llama-2-7b.Q4KM.gguf",
nctx=2048,
echo=False,
)
If you want to use a model from HuggingFace transformers, here is how.
from guidance import models
lm = models.Transformers("microsoft/Phi-3-mini-4k-instruct")
And if you want to use OpenAI, just set your API key in the OPENAIAPIKEY environment variable and load it like this.
from guidance import models
lm = models.OpenAI("gpt-4o-mini")
The interesting thing about Guidance is that the lm object is immutable and we can "add to" it using the + operator. Every time we add something, we get a new object with updated state. This is a really important concept to understand. We can add plain text to the model.
from guidance import models
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
Add static text to the model's context
lm = lm + "The capital of Indonesia is "
print(lm)
Static text like that is not generated by the model, it just gets added to the context. Now, to actually ask the model to generate something, we use the gen() function.
The gen() Function
The gen() function is the heart of Guidance. It tells the model to generate tokens. We can give it a name so the result is easy to retrieve later, and we can limit how many tokens get generated at most.
from guidance import models, gen
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
lm = lm + "The capital of Indonesia is " + gen("capital", maxtokens=10, stop="\n")
Retrieve the generation result via the name we gave it
print(lm["capital"])
Notice how we gave the name "capital" to gen(). After the generation is done, we can access the result via lm["capital"]. This is one of the things I love about Guidance, the result of each generation piece can be retrieved separately using a name, so there is no need to manually parse from a big output blob.
The stop parameter determines when generation stops. In the example above, the model stops generating as soon as it hits a newline. We can also pass a list of several stop strings.
from guidance import models, gen
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
prompt = "Write one short motivational sentence: "
lm = lm + prompt + gen("motivation", maxtokens=50, stop=[".", "\n"])
print(lm["motivation"])
The select() Function
Now this is the one I use most often and the one that gave me the first "wow" moment. The select() function forces the model to pick from a list of options we define. The model cannot escape that list, period. This is a genuine solution to the classification problem I described at the beginning.
from guidance import models, select
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
review = "The product is really great, fast shipping, I am satisfied!"
lm = lm + f"""Classify the sentiment of the following review.
Review: {review}
Sentiment: """ + select(["positive", "negative", "neutral"], name="sentiment")
print(lm["sentiment"]) # guaranteed to be one of the three options
Let this sink in for a moment. With select(), we no longer need to give a long-winded instruction like "answer with a single word only, choose from positive negative neutral, do not add anything else". We just provide the options, and technically the model has no ability to produce output outside of those three options. Because at the token level, Guidance only allows token paths that lead to one of the valid options.
This is the difference from naive prompting that I mentioned at the start. Naive prompting is us hoping the model complies. Constrained generation is us making it so the model has no choice but to comply. The difference is in the guarantee. One is probabilistic, the other is deterministic on the structural side.
Interleaving Text and Generation
The real power of Guidance shows up when we start mixing static text, generation, and choices in a single flow. For example, say we want to extract structured data from a piece of text.
from guidance import models, gen, select
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
email = "Hi, my name is Budi Santoso, I want to complain because my order has not arrived even though it has been 2 weeks."
lm = lm + f"""Extract information from the following email.
Email: {email}
Name: """ + gen("name", stop="\n") + """
Category: """ + select(["complaint", "question", "praise", "other"], name="category") + """
Urgency: """ + select(["low", "medium", "high"], name="urgency")
print("Name:", lm["name"])
print("Category:", lm["category"])
print("Urgency:", lm["urgency"])
Look at how clean that is. We compose a template that mixes static parts (the labels "Name:", "Category:", "Urgency:") with parts that are generated or selected by the model. Each result can be retrieved directly by name. No JSON parsing, no regex, no try-except. The output structure is guaranteed by design.
What is also cool is that since those static labels are not generated by the model, we save tokens and time. The model only focuses on filling in the important parts.
Advanced Usage
Alright, now we level up. This section is what makes Guidance truly different from other libraries.
Regex Constraint
Sometimes we need output with a very specific format, like a phone number, a date, or a number in a certain format. Guidance lets us constrain output using a regular expression. So the model is forced to produce tokens that match our regex pattern.
from guidance import models, gen
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
Force the output to be a 4-digit number
lm = lm + "The year Indonesia declared independence is " + gen("year", regex=r"\d{4}")
print(lm["year"]) # guaranteed to be a 4-digit number
Another example, say we want to extract a price in a numeric format.
from guidance import models, gen
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
text = "This laptop is sold for three million five hundred thousand rupiah."
lm = lm + f"""Text: {text}
Price as a number (Rp format): Rp""" + gen("price", regex=r"[\d\.]+")
print("Rp" + lm["price"])
With a regex constraint, we get a format guarantee at the character level. The model cannot produce a letter where there should be a digit, because tokens that do not match the pattern are immediately rejected. This is incredibly powerful for cases where the output format is critical, for example to be inserted into a database or parsed by another system.
Grammar and CFG (Context-Free Grammar)
This is the most advanced feature and the one that impressed me the most. Guidance lets us define a grammar or context-free grammar to control complex and recursive output structures. A grammar is like a set of grammatical rules that defines valid structure.
We can compose a grammar from small components using operators. For example, say we want to build a generator that only produces valid simple math expressions.
from guidance import models, gen, select, oneormore
lm = models.LlamaCpp("models/llama-2-7b.Q4
KM.gguf")
Build grammar components
def operator():
return select(["+", "-", "*", "/"])
def number():
return gen(regex=r"\d+")
Grammar: number (operator number)+
def expression():
return number() + one
ormore(operator() + number())
lm = lm + "Example math expression: " + expression()
print(lm)
With a grammar, we can define output structures far more complex than mere choices or regex. We can build a grammar for valid JSON, for a particular programming language's syntax, or any custom data format. The important thing is that the generated output is guaranteed to be structurally valid according to the grammar we defined.
Guidance also provides built-in building blocks for composing grammars, like oneormore, zeroormore, and combining operators. This lets us compose complex grammars out of small reusable pieces.
For the JSON case, the concept is that we define a grammar describing the object structure: opening brace, key-value pairs, comma separators, closing brace. The model is forced to follow this structure token by token, so it is impossible for it to produce invalid JSON.
Token Healing
This is a subtle but really important concept, everyone. It is called token healing. To understand it, we need to understand a bit about how language models process text. Models do not read text character by character, they read it token by token. A token can be a word, a fragment of a word, or even a combination of a word and punctuation.
The problem arises when our prompt ends in the middle of an unnatural token boundary. For example, we write a prompt that ends with "http:" and then ask the model to continue. Now, the model usually sees "http://" as a single whole token. Because our prompt has "forced" a stop at "http:", the model gets confused and might produce a weird continuation, because the tokenization becomes unnatural.
Token healing solves this by "backing up" one token at the end of the prompt, then regenerating with the constraint that the result must remain consistent with the original text. So the token boundary that was forcibly cut off gets "healed" so the tokenization becomes natural again.
from guidance import models, gen
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
No token boundary problem because Guidance handles token healing automatically
lm = lm + "Visit our website at http:" + gen("url", regex=r"//[\w\./]+", maxtokens=20)
print(lm["url"])
The good news is that Guidance does this token healing automatically behind the scenes for local models. So you do not need to think about the technical details, but it is important to know why your output ends up nicer than with other approaches. This is one of the reasons constrained generation in Guidance produces cleaner results.
Building Reusable Functions with @guidance
Now this is the feature that makes Guidance truly scalable for large projects. We can build reusable functions using the @guidance decorator. So a generation pattern we use often can be wrapped into a function and called repeatedly like a regular function.
from guidance import models, gen, select, guidance
@guidance
def classifysentiment(lm, text):
lm += f"""Classify the sentiment of the following text.
Text: {text}
Sentiment: """ + select(["positive", "negative", "neutral"], name="sentiment")
return lm
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
Call the reusable function
lm = lm + classifysentiment("The item is bad, I am really disappointed.")
print(lm["sentiment"])
Notice the structure. A function decorated with @guidance always takes lm as its first argument, then any other arguments you want. Inside the function, we add to lm using +=, then we return the lm. After that, this function can be called like a regular building block and combined into the model with +.
We can build these functions for various tasks: data extraction, classification, structured generation, and so on. Then we can compose these functions into a larger flow. This makes our code modular, easy to read, and easy to test.
from guidance import models, gen, select, guidance
@guidance
def extractperson(lm, text):
lm += f"""Extract person data from the text: {text}
Name: """ + gen("name", stop="\n") + """
Age: """ + gen("age", regex=r"\d+") + """
City: """ + gen("city", stop="\n")
return lm
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
lm = lm + extractperson("Andi is 28 years old and lives in Bandung.")
print("Name:", lm["name"])
print("Age:", lm["age"])
print("City:", lm["city"])
Combining Control Flow with Generation
Since Guidance is just plain Python, we can mix program logic like if-else and loops with generation. For example, say we want to generate several items in a single loop.
from guidance import models, gen, select, guidance
@guidance
def generatetodo(lm, count):
lm += "Daily task list:\n"
for i in range(count):
lm += f"{i+1}. " + gen(f"task{i}", stop="\n") + "\n"
return lm
lm = models.LlamaCpp("models/llama-2-7b.Q4KM.gguf")
lm = lm + generatetodo(3)
for i in range(3):
print(lm[f"task{i}"])
This is what I meant at the start about interleaving control flow with generation. We have full control from the program side, but still leverage the model's generative capability. This is very different from a regular chat pattern of one request, one big response.
Best Practices
After using Guidance for quite a while, there are a few things I have learned that I want to share so you do not fall into the same pits I did.
First, pick the right tool for the problem. If the output only needs to pick from a few closed options, use select(). If the format is specific like a number, date, or ID, use regex. If the structure is complex and recursive like JSON or an expression, then use a grammar or CFG. Do not use a heavy grammar for a problem that a simple select() would solve. Match the level of constraint to the need.
Second, always give a clear name to every gen() and select() using the name parameter. This makes it easy for you to retrieve each part's result separately via lm["name"]. Your code becomes much cleaner and easier to maintain than having to parse a big output.
Third, for real constrained generation (regex, grammar, token healing), use a local model via llama.cpp or transformers. API models like OpenAI have more limited support because we do not have access to their token logits. So if you need a strict structure guarantee, a local model is the safer choice.
Fourth, leverage @guidance to build reusable components. If you find a generation pattern used repeatedly, wrap it into a function. This makes your codebase modular and easy to test. I personally usually keep a sort of library of my own Guidance functions for common tasks like extraction, classification, and formatting.
Fifth, limit max_tokens and set an appropriate stop on each gen(). This is important for controlling cost and speed, and also prevents the model from rambling on and generating text you do not need. If you know the output should be short, cap it firmly.
Sixth, remember the token efficiency principle. One of Guidance's advantages is that static text parts are not regenerated by the model. So leverage this by writing templates that clearly separate the static parts from the generative parts. The more structure you define as static text, the fewer tokens need to be generated, the faster and cheaper it is.
Seventh, test your Guidance functions like regular code. Since the output is already structured and can be retrieved by name, you can write unit tests that check the type and format of the results. This is far easier than testing free-form LLM output.
And finally, remember that constrained generation is not magic. We force the structure to be correct, but the content or quality of the content still depends on the model's capability. If the model picks the wrong category, select() only ensures it picks from a valid list, not that its choice is semantically correct. So still pick a model capable enough for your task.
Conclusion
Alright everyone, we have traveled quite far through Guidance. I hope you now have a clear picture of why I think this library is one of the most underrated tools for working with LLMs.
The bottom line is that Guidance changes how we think about controlling an LLM. It is no longer about writing longer and more pleading prompts to get the model to comply, but about technically constraining what the model is and is not allowed to produce. With gen() for free generation, select() for closed choices, regex for specific formats, grammar for complex structures, token healing for cleaner output, and @guidance for reusable components, we have a complete toolkit for building structured and reliable LLM output.
Why does this beat naive prompting? Because we get guarantees. Naive prompting is probabilistic, we hope. Constrained generation is deterministic on the structural side, we force. For production applications where the output must feed into other systems, this structural guarantee is the difference between a reliable system and one that randomly errors every few requests.
My advice, just start small. Try using select() for your classification cases that have been giving you headaches with inconsistent output. Feel the difference for yourself. After that, move up to regex, then grammar if you actually need it. And once you find a repeating pattern, wrap it with @guidance to make it reusable.
Happy experimenting, everyone, I hope this tutorial helps you build LLM applications that are more controlled and reliable. If you have questions or want me to cover another topic, do not hesitate to let me know. See you in the next tutorial!