Logging Utilities

Utilities for configuring and controlling Verifiers logging output.

Overview

The verifiers.utils.logging_utils module provides:

Centralized logging configuration
JSON logging support
Context managers for temporary log level changes
Formatted output for training/evaluation

Logger Name

LOGGER_NAME = "verifiers"

All Verifiers logging uses the "verifiers" logger. Access it via:

import logging
logger = logging.getLogger("verifiers")

Functions

setup_logging

def setup_logging(
    level: str | None = "INFO",
    log_format: str | None = None,
    date_format: str | None = None,
    log_file: str | None = None,
    log_file_level: str | None = None,
    json_logging: bool = False,
) -> None

Configure logging for the Verifiers package.

str | None

default:"INFO"

Console logging level ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"). If None, disables logging.

str | None

Custom log format string. Defaults to "%(asctime)s - %(name)s - %(levelname)s - %(message)s".

str | None

Date format string. Defaults to "%Y-%m-%d %H:%M:%S".

str | None

Optional path to log file. If specified, logs are written to this file.

str | None

Log level for file handler. If None, uses same level as console.

bool

default:"False"

If True, output logs as JSON (structured logging).

Example:

import verifiers as vf

# Basic setup
vf.setup_logging(level="INFO")

# Debug mode
vf.setup_logging(level="DEBUG")

# With file logging
vf.setup_logging(
    level="INFO",
    log_file="verifiers.log",
    log_file_level="DEBUG",  # More verbose in file
)

# JSON logging for production
vf.setup_logging(
    level="INFO",
    json_logging=True,
)

# Disable logging
vf.setup_logging(level=None)

log_level

@contextmanager
def log_level(level: str | int)

Context manager to temporarily change the Verifiers log level.

str | int

New log level (e.g., "DEBUG", "WARNING", or logging.DEBUG).

Example:

import verifiers as vf

vf.setup_logging(level="INFO")

# Temporarily increase verbosity
with vf.log_level("DEBUG"):
    # Detailed logs here
    env.generate(...)

# Back to INFO level

quiet_verifiers

def quiet_verifiers() -> ContextManager

Context manager to temporarily silence Verifiers logging (sets level to WARNING). Useful for suppressing INFO and DEBUG messages during specific operations while still showing warnings and errors. Returns: Context manager that temporarily sets log level to WARNING. Example:

import verifiers as vf

vf.setup_logging(level="INFO")

# Silence Verifiers logs
with vf.quiet_verifiers():
    # Only warnings and errors shown
    env.generate(...)

# Normal logging resumed

print_prompt_completions_sample

def print_prompt_completions_sample(
    prompts: list[Messages],
    completions: list[Messages],
    errors: list[Error | ErrorInfo | None],
    rewards: list[float],
    step: int,
    num_samples: int = 1,
) -> None

Print a formatted table of prompts, completions, and rewards. Displays samples in a Rich table with color-coded columns (yellow for prompts, green for completions, cyan for rewards). Errors are shown in red within the completion column.

list[Messages]

required

List of input prompts (message format).

list[Messages]

required

List of model completions (message format).

list[Error | ErrorInfo | None]

required

List of errors (or None for successful rollouts).

list[float]

required

List of reward scores. If fewer rewards than prompts, pads with 0.0.

int

required

Training step number (displayed in panel title).

int

default:"1"

Number of samples to display from the lists.

Example:

from verifiers.utils.logging_utils import print_prompt_completions_sample

prompts = [
    [{"role": "user", "content": "What is 2+2?"}],
    [{"role": "user", "content": "Solve x^2=16"}],
]
completions = [
    [{"role": "assistant", "content": "2+2=4"}],
    [{"role": "assistant", "content": "x=4 or x=-4"}],
]
errors = [None, None]
rewards = [1.0, 1.0]

print_prompt_completions_sample(
    prompts=prompts,
    completions=completions,
    errors=errors,
    rewards=rewards,
    step=42,
    num_samples=2,
)
# Displays formatted table with both samples

print_time

def print_time(time_s: float) -> str

Format seconds as human-readable time string. Examples:

0.05 → "50ms"
30.0 → "30s"
90.0 → "1m 30s"
3700.0 → "1h 2m"
90000.0 → "1d 1h"

from verifiers.utils.logging_utils import print_time

print(print_time(0.123))   # "123ms"
print(print_time(65))      # "1m 5s"
print(print_time(3661))    # "1h 1m"

JSON Logging

JsonFormatter

class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str

JSON formatter for structured logging. Output Format:

{
    "timestamp": "2024-03-15T10:30:00.000Z",
    "level": "INFO",
    "name": "verifiers.envs.single_turn_env",
    "message": "Starting rollout",
    "module": "single_turn_env",
    "function": "rollout",
    "line": 42,
    "exception": "Traceback..."  // Only if exception
}

Example:

import verifiers as vf

vf.setup_logging(
    level="INFO",
    json_logging=True,
    log_file="app.log",
)

# All logs are JSON
logger = logging.getLogger("verifiers")
logger.info("Started evaluation")  # JSON output

Example Usage

Basic Setup

import verifiers as vf

# Configure logging at start of script
vf.setup_logging(level="INFO")

# Use Verifiers normally
env = vf.SingleTurnEnv(...)
results = await env.generate(...)

Debug Mode

import verifiers as vf

# Enable debug logging
vf.setup_logging(level="DEBUG")

# See detailed logs
env = vf.SingleTurnEnv(...)
# DEBUG: Loading dataset...
# DEBUG: Initialized rubric with 2 functions
# DEBUG: Starting rollout for example 0
# ...

File Logging

import verifiers as vf

vf.setup_logging(
    level="INFO",              # Console: INFO and above
    log_file="verifiers.log",  # Also log to file
    log_file_level="DEBUG",    # File: DEBUG and above
)

# Console shows INFO+, file shows DEBUG+

Quiet Mode

import verifiers as vf

vf.setup_logging(level="WARNING")  # Only warnings and errors

# Or use context manager
vf.setup_logging(level="INFO")

with vf.quiet_verifiers():
    # Suppressed during this block
    results = await env.generate(...)

Production Logging

import verifiers as vf

vf.setup_logging(
    level="INFO",
    json_logging=True,
    log_file="/var/log/verifiers.log",
)

# Structured JSON logs for parsing

Temporary Debug

import verifiers as vf

vf.setup_logging(level="INFO")

with vf.log_level("DEBUG"):
    # Temporarily verbose
    results = await env.generate(num_examples=1)
    # See detailed internal state

# Back to INFO

Custom Format

import verifiers as vf

vf.setup_logging(
    level="INFO",
    log_format="[%(levelname)s] %(name)s: %(message)s",
    date_format="%H:%M:%S",
)

# Output: [INFO] verifiers.envs.single_turn_env: Starting rollout

Logging Levels

Detailed information for diagnosing problems. Very verbose.

Confirmation that things are working as expected. Default level.

Indication of potential issues (e.g., deprecated features).

Serious problem that prevented a function from completing.

Very serious error that may cause termination.

Best Practices

1. Set Up Early

import verifiers as vf

vf.setup_logging(level="INFO")  # First thing in script

# Rest of code...

2. Use Appropriate Levels

import logging

logger = logging.getLogger("verifiers")

logger.debug("Detailed state: %s", state)       # Development only
logger.info("Processing example %d", i)         # Progress info
logger.warning("Deprecated API used")           # Important notices
logger.error("Failed to load dataset: %s", e)  # Problems

3. Environment-Specific Config

import os
import verifiers as vf

# Development: verbose
if os.getenv("ENV") == "dev":
    vf.setup_logging(level="DEBUG")
else:
    # Production: structured
    vf.setup_logging(
        level="INFO",
        json_logging=True,
        log_file="/var/log/app.log",
    )

4. Quiet External Libraries

import logging
import verifiers as vf

vf.setup_logging(level="INFO")

# Silence noisy libraries
logging.getLogger("httpx").setLevel(logging.WARNING)
logging.getLogger("httpcore").setLevel(logging.WARNING)

Environment Classes

Rubrics & Parsers

Clients

Integration Classes

Experimental

Data Types

Utilities

Logging Utilities

Logging Utilities

Overview

Logger Name

Functions

setup_logging

log_level

quiet_verifiers

print_prompt_completions_sample

print_time

JSON Logging

JsonFormatter

Example Usage

Basic Setup

Debug Mode

File Logging

Quiet Mode

Production Logging

Temporary Debug

Custom Format

Logging Levels

Best Practices

1. Set Up Early

2. Use Appropriate Levels

3. Environment-Specific Config

4. Quiet External Libraries

See Also

​Logging Utilities

​Overview

​Logger Name

​Functions

​setup_logging

​log_level

​quiet_verifiers

​print_prompt_completions_sample

​print_time

​JSON Logging

​JsonFormatter

​Example Usage

​Basic Setup

​Debug Mode

​File Logging

​Quiet Mode

​Production Logging

​Temporary Debug

​Custom Format

​Logging Levels

​Best Practices

​1. Set Up Early

​2. Use Appropriate Levels

​3. Environment-Specific Config

​4. Quiet External Libraries

​See Also

Logging Utilities

Overview

Logger Name

Functions

setup_logging

log_level

quiet_verifiers

print_prompt_completions_sample

print_time

JSON Logging

JsonFormatter

Example Usage

Basic Setup

Debug Mode

File Logging

Quiet Mode

Production Logging

Temporary Debug

Custom Format

Logging Levels

Best Practices

1. Set Up Early

2. Use Appropriate Levels

3. Environment-Specific Config

4. Quiet External Libraries

See Also