Arize Phoenix Tutorial: Comprehensive LLM Observability and Evaluation

Arize Phoenix is an open-source platform for observability, evaluation, and tracing of LLM applications. With Phoenix, you can trace every step of your AI pipeline execution, evaluate the quality of model responses, and detect performance issues in real-time. This tutorial will guide you from installation to advanced usage of Phoenix for building reliable AI applications.

Why Arize Phoenix?

Building LLM-powered applications goes far beyond calling a model API. The real challenges emerge after deployment: ensuring response quality remains high, identifying failing queries, and optimizing token costs. Phoenix provides comprehensive tooling to address these challenges:

• Tracing: Track every execution step from input to output, including retrieval, reranking, and generation
• Evaluation: Assess response quality using LLM-as-judge or custom metrics
• Dataset Management: Manage evaluation datasets for regression testing
• Experiment Tracking: Compare performance across prompt or model versions

Phoenix integrates with OpenTelemetry, so the traces it produces follow industry standards and can be exported to other observability backends.

Installation

Basic Installation

pip install arize-phoenix

Installation with Evaluation Dependencies

pip install "arize-phoenix[evals]"

Installation with OpenAI Instrumentation

pip install arize-phoenix openinference-instrumentation-openai openai

Full Installation for This Tutorial

pip install "arize-phoenix[evals]" \
    openinference-instrumentation-openai \
    openinference-instrumentation-langchain \
    openinference-instrumentation-llamaindex \
    openai langchain langchain-openai

Starting the Phoenix Server

Phoenix provides a UI dashboard that runs as a local server:

import phoenix as px

session = px.launchapp()

print(f"Phoenix UI: {session.url}")

phoenix serve

import phoenix as px
from openinference.instrumentation.openai import OpenAIInstrumentor
from phoenix.otel import register

session = px.launchapp()

tracerprovider = register(projectname="my-llm-app")

OpenAIInstrumentor().instrument(tracerprovider=tracerprovider)

Making Traced Calls

Once instrumentation is active, every OpenAI call is automatically traced:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what machine learning is in 3 sentences."}
    ],
    temperature=0.7,
    maxtokens=200

)

print(response.choices[0].message.content)

from opentelemetry import trace

tracer = trace.gettracer(name)

with tracer.startascurrentspan("customer-support-query") as span:

    span.setattribute("user.id", "user-123")
    span.setattribute("session.id", "session-456")

    span.setattribute("query.category", "billing")

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a customer support agent."},