Short-lived Processes

When evaluate() or evaluate_batch() is called, the connector may spawn a background thread to run calibration against the current model. That thread is a daemon — Python will not wait for it when the main thread exits, so a process that exits promptly can silently discard calibration results.

This matters for:

• Scheduled batch jobs that run and exit
• One-shot evaluation scripts
• Containers with a short lifespan (e.g. a Kubernetes Job or AWS Lambda-style runner)

It does not matter for long-running processes such as web servers or persistent workers — the process stays alive well beyond the calibration window.

Waiting for calibration before exit

Call client.wait_for_calibration() before the process exits. It blocks until all in-flight calibration threads finish and returns True, or returns False if the timeout is reached first.

from rait_connector import RAITClient

client = RAITClient()

results = client.evaluate_batch(prompts)

completed = client.wait_for_calibration(timeout=300)  # 5-minute default
if not completed:
    print("WARNING: calibration did not finish within the timeout, some results may be lost")

The same applies after a single evaluate() call:

client.evaluate(
    prompt_id="abc-123",
    ...
)

client.wait_for_calibration()

Timeout

The default timeout is 300 seconds (5 minutes). Adjust it based on the expected number of calibration prompts and the latency of your evaluators:

client.wait_for_calibration(timeout=60)   # tight SLA — fail fast
client.wait_for_calibration(timeout=600)  # large calibration set

If the timeout is reached, a warning is logged and False is returned. The process can still exit — the return value lets you decide whether to surface it as an error or continue regardless.

When calibration is triggered

Background calibration only runs under all three of these conditions:

1. The evaluate() call is a regular evaluation (for_calibration=False).
2. No calibration is already running for the same model name, version, and environment combination.
3. The RAIT API returns a non-empty set of calibration prompts for the model.

If none of those conditions are met, no background thread is spawned and wait_for_calibration() returns immediately.

Batch jobs — recommended pattern

import sys
from rait_connector import RAITClient

def run():
    client = RAITClient()

    prompts = load_prompts()  # your data source
    summary = client.evaluate_batch(prompts)

    completed = client.wait_for_calibration(timeout=300)
    if not completed:
        print(f"WARNING: calibration timed out — {summary['total']} evaluations posted but calibration incomplete")
        sys.exit(1)

    print(f"Done. {summary['successful']}/{summary['total']} evaluations posted.")

if __name__ == "__main__":
    run()