Skip to content

Telemetry Configuration

RAIT uses Azure Monitor to collect telemetry from your application. This guide walks through setting up Azure Monitor and configuring the OpenTelemetry tracer.

The instrumentation examples below are for Python. If your application is in a different language or framework, follow the relevant Azure Monitor / OpenTelemetry guide for your stack accordingly.

Prerequisites

  • An Azure account with permissions to create resources
  • Python 3.8 or higher
  • Your application code ready to instrument

Step 1 — Create a Log Analytics Workspace

  1. In the Azure Portal, search for Log Analytics workspaces and open it.
  2. Click + Create and fill in:
  3. Resource Group — create new or select existing
  4. Name — e.g. ai-logs-workspace
  5. Region — closest to your deployment
  6. Click Review + create → Create.

Step 2 — Create an Application Insights Resource

  1. Search for Application Insights and click + Create.
  2. Fill in:
  3. Resource Mode — select Workspace-based
  4. Log Analytics Workspace — select the workspace from Step 1
  5. Click Review + create → Create.

  6. Once deployed, open the resource and copy the Connection String from the Overview page.

    Keep the connection string secret — treat it like a password.

Step 3 — Configure Availability Tests

Availability tests populate the AppAvailabilityResults table used by the SLA Uptime and Endpoint Health metrics.

  1. Open your Application Insights resource → Availability → + Add Classic test.
  2. Set:
  3. Test name — must be exactly RAI Ping Test
  4. URL — your model endpoint URL (from Azure AI Foundry)
  5. Test frequency — 5 minutes recommended
  6. Test locations — select one or more geographic regions
  7. Click Create.

Step 4 — Install Python Packages

uv:

uv add azure-monitor-opentelemetry opentelemetry-instrumentation-openai-v2

pip:

pip install azure-monitor-opentelemetry opentelemetry-instrumentation-openai-v2

Step 5 — Configure the Tracer

5.1 Set the connection string

Set the connection string as an environment variable before starting your application:

# Linux / macOS
export APPLICATIONINSIGHTS_CONNECTION_STRING="your-connection-string-here"

# Windows (PowerShell)
$env:APPLICATIONINSIGHTS_CONNECTION_STRING = "your-connection-string-here"

Or add it to a .env file:

APPLICATIONINSIGHTS_CONNECTION_STRING=your-connection-string-here

5.2 Initialise global tracing

Call this once at application startup:

import os
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

configure_azure_monitor(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)
OpenAIInstrumentor().instrument()

5.3 Add a custom span around your LLM call

Wrap each LLM call in a named span and set prompt_id as a span attribute. The value must match the prompt_id you pass to evaluate().

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def call_llm(prompt_id: str, user_query: str) -> str:
    with tracer.start_as_current_span("RAIT") as span:
        response = your_openai_client.chat.completions.create(
            model=deployment,
            messages=[{"role": "user", "content": user_query}],
            logprobs=True,
        )
        text = response.choices[0].message.content

        # Required: must match evaluate(prompt_id=...)
        span.set_attribute("prompt_id", prompt_id)

        # Optional: log probability (average token confidence)
        probs = [math.exp(t.logprob) for t in response.choices[0].logprobs.content]
        span.set_attribute("prob", sum(probs) / len(probs))

        return text

Then call evaluate() with the same prompt_id variable:

prompt_id = "12345"  # same value used in both the span and evaluate()

response_text = call_llm(prompt_id=prompt_id, user_query=user_input)

client.evaluate(
    prompt_id=prompt_id,  # must be the same value as span.set_attribute("prompt_id", ...)
    prompt_url="https://example.com/prompts/12345",
    timestamp="2026-01-01T12:00:00+00:00",
    model_name="gpt-4o",
    model_version="2024-11-20",
    query=user_input,
    response=response_text,
    environment="production",
    purpose="monitoring",
)

prompt_id must be unique per prompt. The same value must appear in both span.set_attribute("prompt_id", ...) and evaluate(prompt_id=...) — if they differ, the telemetry row cannot be matched to the evaluation log.

How the three tables are populated

Table Populated by Notes
AppDependencies OpenAIInstrumentor automatically, one row per LLM call Custom span adds prompt_id to Properties
AppExceptions OpenAIInstrumentor automatically, one row per unhandled exception during an LLM call No extra configuration needed — exceptions are captured as long as tracing is initialised
AppAvailabilityResults Availability tests configured in Step 3 Runs on a schedule (e.g. every 5 minutes) independently of your application code

Step 6 — Verify in Azure Portal

Open your Log Analytics Workspace → Logs and run:

AppDependencies — one row per LLM call, with prompt_id in Properties:

AppDependencies
| where Properties contains "prompt_id"
| project TimeGenerated, Target, DurationMs, Properties
| limit 10

AppExceptions — one row per exception thrown during an instrumented call:

AppExceptions
| project TimeGenerated, ExceptionType, OuterMessage, OperationId
| limit 10

AppAvailabilityResults — one row per availability test run:

AppAvailabilityResults
| project TimeGenerated, Name, Success, DurationMs
| limit 10

Troubleshooting

Issue Solution
No traces in AppDependencies Wait 2–5 minutes; telemetry is batched before upload
prompt_id missing from Properties Check that span.set_attribute("prompt_id", ...) is called inside the active span
Availability results not reflected in scores Check that the availability test name is exactly RAI Ping Test — any other value is ignored by the scoring engine
prompt_id does not match across span and evaluate Check that the value passed to span.set_attribute("prompt_id", ...) and evaluate(prompt_id=...) are identical
AppExceptions table empty Check that OpenAIInstrumentor().instrument() is called at application startup before any OpenAI calls are made