Skip to content

RAIT Connector

Python library for evaluating LLM outputs across multiple ethical dimensions and performance metrics using Azure AI Evaluation services.

Features

  • 22 Evaluation Metrics across 8 ethical dimensions
  • Automatic Background Calibration - Calibration runs automatically when you evaluate
  • Azure Monitor Telemetry - Fetch AppDependencies, AppExceptions, AppAvailabilityResults
  • Parallel Execution for faster evaluations
  • Automatic API Integration with RAIT services
  • Type-Safe with Pydantic models
  • Flexible Configuration via environment variables or direct parameters
  • Scheduler for recurring telemetry fetching and calibration runs

Quick Example

from rait_connector import RAITClient

# Initialize client
client = RAITClient()

# Evaluate a single prompt
result = client.evaluate(
    prompt_id="123",
    prompt_url="https://example.com/123",
    timestamp="2025-01-01T00:00:00Z",
    model_name="gpt-4",
    model_version="1.0",
    query="What is AI?",
    response="AI is artificial intelligence...",
    environment="production",
    purpose="monitoring"
)

Installation

uv add rait-connector

Or with pip:

pip install rait-connector

Next Steps