Complete Guidance Tutorial: Constrained Generation and Structured Output from LLMs

Hey everyone, in this tutorial I want to introduce you to a library that I think gets overlooked way too often even though it is incredibly powerful for working with LLMs. It is called Guidance. This is a library built by Microsoft that focuses on one thing that makes our lives so much easier: controlling and steering the output of language models so it genuinely follows the structure we want. It is not about politely asking through a prompt and then praying the model complies, it is about actually forcing the output to obey the rules we define.

If you have ever been frustrated because your LLM sometimes returns answers in a messy format, sometimes adds words you never asked for, sometimes picks an option outside the list you provided, then Guidance is going to be a really elegant solution for you. I am going to cover this from the very basics, starting with why constrained generation matters, how to install it, basic usage of gen() and select(), all the way to advanced features like regex constraints, grammar or CFG, token healing, and building reusable functions with the @guidance decorator. I will give you Python code examples for everything that you can actually run yourself. Let us get started.

Introduction

Before we dive into the code, let me tell you about the problem that gave birth to this library. Imagine you are building an application that needs an LLM to classify the sentiment of a review as "positive", "negative", or "neutral". The most naive approach is to write a prompt like "Classify the sentiment of this review, answer with a single word only: positive, negative, or neutral." Then we send it to the model and hope that what comes back is just one word from those three options.

The problem is that an LLM is fundamentally a text generator that picks the next token based on probabilities. It has no guarantee that it will obey our instructions. Sometimes it answers "The sentiment of this review is positive." Sometimes it answers "Positive" with a capital letter and extra punctuation. Sometimes it answers "slightly positive" which is not in our list of options at all. And every time the output goes off the rails, our code that parses the result becomes fragile and prone to errors.

The traditional approach to solving this usually involves adding longer and more detailed instructions to the prompt, or parsing manually with regex, or wrapping everything in giant try-except blocks. But all of this is fragile. The moment the model produces slightly different output, our code breaks.

The idea behind Guidance is different from mere prompting. Guidance works at the token generation level. So instead of just asking the model through text, Guidance actually constrains which tokens the model is allowed to produce at each step. If we say the output must be one of three options, then technically Guidance only allows tokens that lead to those three options. The model has no way to produce output outside of that, because invalid tokens have their probability zeroed out.

Besides constrained generation, Guidance also has a concept I really love: we can interleave our program's control flow with the generation process. So inside a single template, we can define which part is static text, which part is generated by the model, which part is constrained to choices, and we can even have if-else branches and loops. This is very different from how we usually call an LLM in a one-shot request-response model. With Guidance, we get to compose programs that are more deterministic and controlled.

Guidance is also token-efficient and often faster, because the parts we have defined as static text do not need to be regenerated by the model. The model only focuses on filling in the parts that actually need to be generated. This is different from a regular chat approach where the model has to regenerate the entire JSON structure or format every single time.

Alright, enough theory, let us jump straight into practice.

Installation

Installation is super easy, just one line with pip.

pip install guidance

Guidance automatically brings the core dependencies you need. But depending on which model you want to use, you might need extra libraries. If you want to use a model from OpenAI, install the openai library too.

pip install guidance openai

If you want to run a local model using transformers from HuggingFace, install torch and transformers.

pip install guidance transformers torch