Semantic Router Tutorial: Smart Routing for LLM Applications

Semantic Router is a Python library that enables you to make semantic-based routing decisions quickly and efficiently. Instead of relying on an LLM for every classification decision, Semantic Router uses embedding vectors to determine user intent in milliseconds.

This library is invaluable for building chatbots, virtual assistants, and AI applications that require fast responses to various types of user queries. With Semantic Router, you can direct queries to the appropriate handler without the overhead of calling an LLM every time.

Why Semantic Router?

In LLM-based application development, one common challenge is determining how to respond to user input. Traditional approaches use rigid keyword matching or send all queries to expensive and slow LLMs.

Semantic Router offers an optimal middle ground:

• Speed: Routing is performed in milliseconds, not seconds
• Accuracy: Uses semantic similarity, not keyword matching
• Cost efficiency: Reduces unnecessary LLM calls
• Flexibility: Supports both static routing and function-calling routes

Example use cases:

• Filtering irrelevant questions before sending them to an LLM
• Directing technical questions to specific knowledge bases
• Activating guardrails for sensitive topics
• Triggering specific functions based on user intent

Installation

Semantic Router can be installed using pip. Several installation variants are available depending on which encoder you want to use.

# Basic installation
pip install semantic-router

With OpenAI encoder support
pip install "semantic-router[openai]"

With Cohere encoder support
pip install "semantic-router[cohere]"

With local encoder support (HuggingFace)
pip install "semantic-router[local]"

Full installation
pip install "semantic-router[all]"

For this tutorial, we will use the local encoder so it can run without an API key:

pip install "semantic-router[local]"

Verify the installation:

import semanticrouter
print(semanticrouter.version)

Core Concepts

Before diving into code, it is important to understand the core concepts in Semantic Router.

Route

A Route is a rule that defines an intent or category. Each route consists of a name and several example utterances that represent that intent.

from semanticrouter import Route

Define a route for weather-related questions
weatherroute = Route(
    name="weather",
    utterances=[
        "what is the weather like today?",
        "will it rain tomorrow?",
        "what is the temperature in New York?",
        "weather forecast for this week",
        "is it going to be sunny or cloudy today?",
    ],
)

Encoder

The Encoder is responsible for converting text into embedding vectors. Semantic Router supports various encoders:

# Local encoder (free, no API key required)
from semanticrouter.encoders import HuggingFaceEncoder

encoder = HuggingFaceEncoder()

OpenAI encoder
from semanticrouter.encoders import OpenAIEncoder
encoder = OpenAIEncoder(apikey="sk-...")


Cohere encoder
from semanticrouter.encoders import CohereEncoder
encoder = CohereEncoder(apikey="...")

RouteLayer

The RouteLayer combines routes and an encoder into a single routing system ready for use.

from semanticrouter.layer import RouteLayer

routelayer = RouteLayer(

    encoder=encoder,
    routes=[weatherroute, greetingroute],

)

Basic Usage

Let us build a simple routing system that can distinguish between several types of user intent.

from semanticrouter import Route, RouteLayer
from semanticrouter.encoders import HuggingFaceEncoder


1. Define routes
greetingroute = Route(
    name="greeting",
    utterances=[
        "hello",
        "hi, how are you?",
        "good morning",
        "hey there, nice to meet you",
        "hello, pleased to meet you",
    ],
)

farewellroute = Route(

    name="farewell",
    utterances=[
        "goodbye",
        "see you later",
        "bye bye",
        "take care, see you soon",
        "I have to go now",
    ],
)

technicalroute = Route(
    name="technicalsupport",

    utterances=[
        "my application is throwing an error",
        "how do I reset my password?",
        "I cannot log into my account",
        "the website is loading very slowly",
        "there is a bug in the payment feature",
    ],
)

billingroute = Route(
    name="billing",
    utterances=[
        "what is my bill this month?",
        "I want to pay my invoice",
        "what is the status of my payment?",
        "how do I upgrade my subscription plan?",
        "I want to cancel my subscription",
    ],
)

2. Initialize encoder
encoder = HuggingFaceEncoder()

3. Create RouteLayer
routes = [greetingroute, farewellroute, technicalroute, billingroute]
routelayer = RouteLayer(encoder=encoder, routes=routes)


4. Test routing
testqueries = [
    "hi, I need some help",
    "I cannot access my account",
    "I have a question about my bill",
    "ok thanks, bye",
    "what is the weather today?",  # does not match any route
]

for query in testqueries:

    result = routelayer(query)
    print(f"Query: '{query}'")
    print(f"Route: {result.name}")
    print(f"Similarity: {result.similarityscore:.4f}")

    print()

Expected output:

Query: 'hi, I need some help'
Route: greeting
Similarity: 0.8234

Query: 'I cannot access my account'
Route: technicalsupport
Similarity: 0.8567

Query: 'I have a question about my bill'
Route: billing
Similarity: 0.8901

Query: 'ok thanks, bye'
Route: farewell
Similarity: 0.8123

Query: 'what is the weather today?'
Route: None
Similarity: 0.0000

Notice that the last query about weather does not match any route and returns None. This is an important feature because Semantic Router can recognize when a query does not fall into any defined category.

Advanced Usage

Dynamic Routes with Function Calling

Semantic Router supports dynamic routes that can call Python functions when a specific route is triggered. This is very useful for building agents that can perform specific actions.

from semanticrouter import Route, RouteLayer
from semanticrouter.encoders import HuggingFaceEncoder
from datetime import datetime

Define functions to be called
def getcurrenttime(timezone: str = "UTC"):
    """Get the current time."""
    from zoneinfo import ZoneInfo
    now = datetime.now(ZoneInfo(timezone))
    return f"Current time in {timezone}: {now.strftime('%H:%M:%S %d-%m-%Y')}"

def calculateprice(item: str, quantity: int = 1):

    """Calculate item price."""
    prices = {
        "coffee": 5.00,
        "tea": 3.00,
        "sandwich": 8.50,
        "salad": 7.00,
    }
    price = prices.get(item.lower(), 0)
    total = price * quantity
    return f"{item} x{quantity} = ${total:.2f}"

Define routes with function schemas
timeroute = Route(
    name="gettime",

    utterances=[
        "what time is it?",
        "current time please",
        "what is the time right now?",
        "tell me the current time",
    ],
    functionschemas=[
        {
            "name": "getcurrenttime",
            "description": "Get the current time",
            "parameters": {
                "type": "object",
                "properties": {
                    "timezone": {
                        "type": "string",
                        "description": "Timezone, default UTC",
                    }
                },
                "required": [],
            },
        }
    ],
)

priceroute = Route(

    name="checkprice",
    utterances=[
        "how much is a coffee?",
        "calculate my total order",
        "I want to order 3 sandwiches, how much is that?",
        "menu price list",
    ],
    functionschemas=[

        {
            "name": "calculateprice",
            "description": "Calculate item price",
            "parameters": {
                "type": "object",
                "properties": {
                    "item": {
                        "type": "string",
                        "description": "Item name",
                    },
                    "quantity": {
                        "type": "integer",
                        "description": "Number of items",
                        "default": 1,
                    },
                },
                "required": ["item"],
            },
        }
    ],
)

encoder = HuggingFaceEncoder()
routelayer = RouteLayer(

    encoder=encoder,
    routes=[timeroute, priceroute],

)

Test dynamic routing
result = routelayer("what time is it in Tokyo?")
print(f"Route: {result.name}")
print(f"Function call: {result.functioncall}")

Threshold Tuning

You can set the threshold for each route to be more strict or lenient in matching queries.

# Route with high threshold - only matches very similar queries
strictroute = Route(
    name="passwordreset",

    utterances=[
        "reset my password",
        "forgot password",
        "change account password",
        "I do not remember my password",
    ],
    scorethreshold=0.85,  # high threshold
)

Route with low threshold - more lenient
generalroute = Route(

    name="generalinquiry",
    utterances=[
        "I have a question",
        "can you help me?",
        "need some info",
        "I want to ask something",
    ],
    scorethreshold=0.5,  # low threshold

)

Using Multiple Encoders

For specific use cases, you might need different encoders for different purposes.

from semanticrouter.encoders import HuggingFaceEncoder

Multilingual encoder for international support
multilingualencoder = HuggingFaceEncoder(

    name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
)

Larger model encoder for higher accuracy
largeencoder = HuggingFaceEncoder(
    name="sentence-transformers/all-mpnet-base-v2"
)

Hybrid Route Layer

Semantic Router also supports a hybrid approach that combines semantic routing with sparse retrieval (BM25).

from semanticrouter import Route, HybridRouteLayer
from semanticrouter.encoders import HuggingFaceEncoder

encoder = HuggingFaceEncoder()

routes = [greetingroute, technicalroute, billingroute]


Hybrid layer combines dense and sparse matching
hybridlayer = HybridRouteLayer(
    encoder=encoder,
    routes=routes,
    alpha=0.5,  # weight between dense (1.0) and sparse (0.0)
)

result = hybridlayer("I want to pay my internet bill from last month")

print(f"Route: {result.name}")

Guardrails with Semantic Router

One of the most popular use cases is using Semantic Router as guardrails to filter unwanted input before sending it to an LLM.

from semanticrouter import Route, RouteLayer
from semanticrouter.encoders import HuggingFaceEncoder


Route for content that should be blocked
harmfulcontent = Route(
    name="harmful",
    utterances=[
        "how to hack a website",
        "how to create a computer virus",
        "how to bypass security",
        "illegal hacking tutorial",
        "how to steal someone's data",
    ],
    scorethreshold=0.75,

)

offtopic = Route(
    name="offtopic",

    utterances=[
        "who is the president of the US?",
        "how far is the earth from the sun?",
        "recipe for chocolate cake",
        "football league schedule",
        "recommend a good movie",
    ],
    scorethreshold=0.70,
)

Route for allowed content
allowedtopics = Route(

    name="allowed",
    utterances=[
        "how do I use this API?",
        "explain the product features",
        "help with account settings",
        "how to integrate with other systems",
        "REST API endpoint documentation",
    ],
    scorethreshold=0.60,
)

encoder = HuggingFaceEncoder()
guardrail = RouteLayer(
    encoder=encoder,
    routes=[harmfulcontent, offtopic, allowedtopics],

)

def processquery(query: str):
    """Process query with guardrails."""
    result = guardrail(query)

    if result.name == "harmful":
        return "Sorry, I cannot assist with that request."
    elif result.name == "offtopic":

        return "This question is outside the scope of our service. Please ask about our products."
    elif result.name == "allowed":
        return f"[Forwarded to LLM] Processing: {query}"
    else:
        return "Sorry, I did not understand your question. Could you clarify?"

Test guardrails
testinputs = [
    "how to use the payment gateway API",
    "how to hack someone's account",
    "who won the world cup in 2022?",
    "how to set up webhooks for notifications",
]

for q in testinputs:

    response = processquery(q)
    print(f"Input: {q}")
    print(f"Response: {response}\n")

Integration with LangChain

Semantic Router can be integrated with popular frameworks like LangChain to build more complex applications.

from semanticrouter import Route, RouteLayer
from semanticrouter.encoders import HuggingFaceEncoder
from langchainopenai import ChatOpenAI

from langchaincore.prompts import ChatPromptTemplate

Setup routes
coderoute = Route(

    name="codehelp",
    utterances=[
        "help me debug this Python code",
        "how to write a unit test?",
        "explain this error",
        "review my code",
    ],
)

dataroute = Route(

    name="dataanalysis",
    utterances=[
        "analyze sales data",
        "create a visualization from this dataset",
        "SQL query for monthly report",
        "effective data cleaning techniques",
    ],
)

encoder = HuggingFaceEncoder()
router = RouteLayer(encoder=encoder, routes=[coderoute, dataroute])

Setup LLM chains per route
codellm = ChatOpenAI(model="gpt-4o", temperature=0)

datallm = ChatOpenAI(model="gpt-4o", temperature=0.3)

codeprompt = ChatPromptTemplate.frommessages([
    ("system", "You are an expert Python developer. Help the user with their code."),
    ("human", "{query}"),
])

dataprompt = ChatPromptTemplate.frommessages([
    ("system", "You are an expert data analyst. Help the user analyze data."),
    ("human", "{query}"),
])

codechain = codeprompt | codellm

datachain = dataprompt | datallm

def smartroute(query: str):

    """Route query to the appropriate chain."""
    result = router(query)

    if result.name == "codehelp":
        return codechain.invoke({"query": query})

    elif result.name == "dataanalysis":
        return datachain.invoke({"query": query})

    else:
        return "Question is out of scope. Please ask about coding or data analysis."

response = smartroute("please review my sorting algorithm code")
print(response.content)

Saving and Loading Routes

For production use, you can save route configurations and load them back without needing to redefine everything.

import json

Save route configuration to JSON
def saveroutes(routes, filepath):

    config = []
    for route in routes:
        config.append({
            "name": route.name,
            "utterances": route.utterances,
            "scorethreshold": route.scorethreshold,

        })
    with open(filepath, "w") as f:
        json.dump(config, f, indent=2)
    print(f"Routes saved to {filepath}")

Load routes from JSON
def loadroutes(filepath):
    with open(filepath) as f:
        config = json.load(f)
    routes = []
    for item in config:
        routes.append(Route(
            name=item["name"],
            utterances=item["utterances"],
            scorethreshold=item.get("scorethreshold", 0.5),
        ))
    return routes

Save
saveroutes(routes, "routesconfig.json")

Load back
loadedroutes = loadroutes("routesconfig.json")

routelayer = RouteLayer(encoder=encoder, routes=loadedroutes)

Best Practices

1. Provide Diverse Utterances

Each route should have at least 5-10 diverse utterances. The more diverse the examples, the better the routing accuracy.

# Poor - too similar
badroute = Route(
    name="greeting",
    utterances=["hello", "hi", "hey"],  # too few and similar
)

Better - diverse and representative
goodroute = Route(

    name="greeting",
    utterances=[
        "hello, how are you?",
        "good morning, can you help me?",
        "hi, I am a new user",
        "hey there, nice to meet you",
        "greetings, I need some assistance",
        "excuse me, I would like to ask something",
        "hi, could you point me in the right direction?",
    ],
)

2. Use Appropriate Thresholds

A threshold that is too high causes many queries to go unrouted. A threshold that is too low causes false positives. Start at 0.5 and adjust based on testing.

# Start with default threshold, then tune
route = Route(
    name="specificintent",
    utterances=[...],
    scorethreshold=0.5,  # start here

)

Test with various queries and adjust
Too many false positives: raise to 0.6-0.7
Too many misses: lower to 0.4-0.45

3. Separate Routes by Action

Do not create routes that are too broad. Separate them based on different actions or responses.

# Poor - one route for everything
supportroute = Route(
    name="support",
    utterances=[
        "reset password",
        "what is my bill",
        "how to upgrade my plan",
    ],
)

Better - separate routes per action
passwordroute = Route(name="passwordreset", utterances=[...])
billingroute = Route(name="billinginquiry", utterances=[...])
upgraderoute = Route(name="planupgrade", utterances=[...])

4. Monitor and Iterate

Implement logging to monitor routing performance in production.

import logging
from datetime import datetime

logger = logging.getLogger("semanticrouter")


def routewithlogging(routelayer, query: str):
    """Route query with logging for monitoring."""
    result = routelayer(query)


    logentry = {
        "timestamp": datetime.now().isoformat(),
        "query": query,
        "route": result.name,
        "score": result.similarityscore,

    }

    if result.name is None:
        logger.warning(f"Unrouted query: {query}")
    elif result.similarityscore < 0.6:
        logger.info(f"Low confidence route: {logentry}")


    return result

5. Choose the Right Encoder

For production with high traffic, consider the trade-off between speed and accuracy:

• Local HuggingFace encoder: Free, fast, suitable for prototyping and low-latency requirements
• OpenAI encoder: High accuracy, requires API calls, has per-request costs
• Cohere encoder: Alternative cloud encoder with good multilingual performance

6. Handling Edge Cases

Always handle the case when no route matches:

def handlequery(routelayer, query):
    result = routelayer(query)

    if result.name is None:
        # Fallback: send to general LLM
        return fallbackllm(query)


    return routehandlersresult.name

Comparison with Other Approaches

| Aspect | Semantic Router | Keyword Matching | LLM Classification |

|--------|----------------|------------------|---------------------|

| Speed | ~5ms | ~1ms | ~500ms-2s |

| Accuracy | High | Low | Very High |

| Cost | Low/Free | Free | Expensive |

| Setup | Easy | Easy | Complex |

| Flexibility | High | Low | Very High |

| Scalability | Good | Good | Limited by rate limits |

Conclusion

Semantic Router is an elegant solution for routing challenges in LLM applications. By combining the speed of embedding-based classification with the flexibility of function calling, this library bridges the gap between rigid keyword matching and expensive LLM classification.

Key takeaways:

Use for fast decisions: Semantic Router is ideal for intent classification that needs instant responses

Provide diverse examples: Routing quality heavily depends on utterance quality

Tune thresholds: Start from defaults and adjust based on real data

Guardrails: Leverage it as a security layer before queries reach the LLM

Combine with LLMs: Use Semantic Router for fast routing, LLMs for deep processing

Semantic Router continues to evolve with an active community. For the latest information, visit the official GitHub repository from Aurelio AI.