Models | ModelRed

Introduction

Models represent the LLM endpoints registered in your ModelRed organization. Use the SDK to discover available models and their configurations before creating assessments.

Listing Models

Basic Listing

Retrieve a paginated list of registered models:

from modelred import ModelRed

client = ModelRed(api_key="mr_...")

response = client.list_models(
    page=1,
    page_size=20,
)

for model in response["data"]:
    print(f"{model['id']}: {model['displayName']}")
    print(f"  Provider: {model['provider']}")
    print(f"  Status: {model['status']}")

Filtering by Provider

Filter models by their provider:

# OpenAI models only
openai_models = client.list_models(
    provider="openai",
)

# Anthropic models only
anthropic_models = client.list_models(
    provider="anthropic",
)

# Azure OpenAI models
azure_models = client.list_models(
    provider="azure",
)

Filtering by Status

Control which models appear based on their active status:

# Only active models (default)
active = client.list_models(
    status="active",
)

# Only inactive models
inactive = client.list_models(
    status="inactive",
)

# All models regardless of status
all_models = client.list_models(
    status="both",
)

Search by Name

Search for models by name or identifier:

# Find GPT-4 models
gpt4_models = client.list_models(
    search="gpt-4",
)

# Find Claude models
claude_models = client.list_models(
    search="claude",
)

Sorting

Sort models by different fields:

# Sort by creation date (default, newest first)
by_date = client.list_models(
    sort_by="createdAt",
    sort_dir="desc",
)

# Sort alphabetically by name
alphabetical = client.list_models(
    sort_by="displayName",
    sort_dir="asc",
)

# Sort by provider
by_provider = client.list_models(
    sort_by="provider",
    sort_dir="asc",
)

List Parameters

Prop	Type	Default
`page?`	`int`	`1`
`page_size?`	`int`	`10`
`search?`	`string \| None`	-
`provider?`	`"openai" \| "anthropic" \| "azure" \| "huggingface" \| ... \| None`	-
`status?`	`"active" \| "inactive" \| "both"`	`"both"`
`sort_by?`	`string`	`"createdAt"`
`sort_dir?`	`"asc" \| "desc"`	`"desc"`

Supported Providers

ModelRed supports models from various providers:

OpenAI

GPT-4, GPT-3.5, and custom deployments

Anthropic

Claude 3 family and newer releases

Azure OpenAI

Enterprise deployments via Azure

AWS Bedrock

Models hosted on Amazon Bedrock

Google

Gemini and PaLM models

HuggingFace

Open-source and custom models

And more: SageMaker, Grok, OpenRouter, and custom REST endpoints.

Iterating All Models

Use the iterator helper for automatic pagination:

# Iterate through all models
for model in client.iter_models(page_size=50):
    print(f"{model['displayName']} ({model['provider']})")

# With filters
for model in client.iter_models(
    page_size=50,
    provider="openai",
    status="active",
):
    print(f"Active OpenAI: {model['displayName']}")

See the Pagination guide for more details.

Response Structure

Model objects include:

{
    "id": "model_abc123",
    "displayName": "GPT-4 Turbo",
    "provider": "openai",
    "modelIdentifier": "gpt-4-turbo-preview",
    "status": "active",
    "createdAt": "2024-11-01T12:00:00Z",
    "updatedAt": "2024-12-15T09:30:00Z",
    "metadata": {
        "endpoint": "https://api.openai.com/v1",
        "version": "2024-02-15-preview"
    }
}

Finding Models for Assessments

Get a Specific Model

To create an assessment, you need either a model ID or model name:

# List models and pick one
models = client.list_models(provider="openai", status="active")

if models["data"]:
    model_id = models["data"][0]["id"]
    model_name = models["data"][0]["displayName"]

    # Use ID (recommended)
    assessment = client.create_assessment_by_id(
        model_id=model_id,
        probe_pack_ids=["pack_1"],
        detector_provider="openai",
        detector_api_key="sk-...",
        detector_model="gpt-4o-mini",
    )

    # Or use name
    assessment = client.create_assessment(
        model=model_name,
        probe_pack_ids=["pack_1"],
        detector_provider="openai",
        detector_api_key="sk-...",
        detector_model="gpt-4o-mini",
    )

Filter by Use Case

    # Production models only
    production = client.list_models(
        status="active",
        search="prod",
    )

search="staging", ) ```
</Tab>

<Tab value="Development">
```python
  # Latest model versions for development
  latest = client.list_models(
      sort_by="createdAt",
      sort_dir="desc",
      page_size=5,
  )

Model Status

Models have two status states:

Active

Model is available for assessments. Credentials are valid and the endpoint is reachable.

Inactive

Model is disabled, credentials expired, or endpoint unavailable. Cannot be used in new assessments.

Filter appropriately:

# Only show models ready for testing
ready = client.list_models(status="active")

# Show all for auditing
all_models = client.list_models(status="both")

Practical Examples

List All Active Models by Provider

providers = ["openai", "anthropic", "azure"]

for provider in providers:
    models = client.list_models(
        provider=provider,
        status="active",
        page_size=100,
    )

    print(f"\n{provider.upper()} Models:")
    for model in models["data"]:
        print(f"  - {model['displayName']}")

Find Fastest Models for Testing

# Assume smaller/faster models have "mini" or "fast" in name
fast_models = []

for model in client.iter_models(page_size=100, status="active"):
    name_lower = model["displayName"].lower()
    if "mini" in name_lower or "fast" in name_lower or "3.5" in name_lower:
        fast_models.append(model)

print(f"Found {len(fast_models)} fast models for testing")

Group Models by Provider

from collections import defaultdict

models_by_provider = defaultdict(list)

for model in client.iter_models(page_size=100):
    models_by_provider[model["provider"]].append(model)

for provider, models in models_by_provider.items():
    print(f"{provider}: {len(models)} models")

Async Examples

All operations have async equivalents:

import asyncio
from modelred import AsyncModelRed

async def main():
    async with AsyncModelRed(api_key="mr_...") as client:
        # List models
        models = await client.list_models(
            provider="openai",
            status="active",
        )

        # Note: Async iterator not available yet
        # Use manual pagination for async
        page = 1
        all_models = []

        while True:
            response = await client.list_models(page=page, page_size=50)
            all_models.extend(response["data"])

            if page >= response["totalPages"]:
                break
            page += 1

asyncio.run(main())

Note: The async iterator iter_models is not available in the current SDK version. Use manual pagination with list_models for async iteration.

Model Management

Note: Model creation, updates, and deletion are only available through the web UI. The SDK provides read-only access.

To add a new model:

Navigate to the ModelRed web application

Go to the Models section in the sidebar

Click "Add Model" and configure provider, credentials, and endpoint

Save configuration and test connectivity

Once registered, the model appears in SDK list operations

Best Practices

Cache Model Lists

Avoid repeated API calls by caching model information:

import time

class ModelCache:
    def __init__(self, client, ttl=300):
        self.client = client
        self.ttl = ttl
        self._cache = None
        self._timestamp = 0

    def get_models(self):
        now = time.time()
        if self._cache is None or (now - self._timestamp) > self.ttl:
            self._cache = list(self.client.iter_models(page_size=100))
            self._timestamp = now
        return self._cache

cache = ModelCache(client)
models = cache.get_models()  # Uses cache if fresh

Validate Model Availability

Before creating assessments, verify the model is active:

def get_active_model(client, model_id):
    models = client.list_models(status="active", page_size=100)

    for model in models["data"]:
        if model["id"] == model_id:
            return model

    raise ValueError(f"Model {model_id} not found or inactive")

try:
    model = get_active_model(client, "model_123")
    assessment = client.create_assessment_by_id(
        model_id=model["id"],
        probe_pack_ids=["pack_1"],
        detector_provider="openai",
        detector_api_key="sk-...",
        detector_model="gpt-4o-mini",
    )
except ValueError as e:
    print(f"Cannot create assessment: {e}")

Provider-Specific Handling

Different providers may have different characteristics:

def get_recommended_detector(model_provider):
    """Recommend detector based on model provider."""
    if model_provider == "openai":
        return {
            "provider": "anthropic",
            "model": "claude-3-5-sonnet-20241022"
        }
    elif model_provider == "anthropic":
        return {
            "provider": "openai",
            "model": "gpt-4o"
        }
    else:
        # Default to fast detector
        return {
            "provider": "openai",
            "model": "gpt-4o-mini"
        }

models = client.list_models(status="active")
for model in models["data"]:
    detector = get_recommended_detector(model["provider"])
    print(f"{model['displayName']}: use {detector['model']} as detector")

Complete Example

model_discovery.py

from modelred import ModelRed
import os

client = ModelRed(api_key=os.environ["MODELRED_API_KEY"])

# Step 1: Get all active models
print("=== Active Models by Provider ===")
providers = ["openai", "anthropic", "azure", "aws"]

for provider in providers:
    models = client.list_models(
        provider=provider,
        status="active",
        page_size=100,
    )

    if models["data"]:
        print(f"\n{provider.upper()}: {len(models['data'])} models")
        for model in models["data"]:
            print(f"  - {model['displayName']}")
            print(f"    ID: {model['id']}")
            print(f"    Identifier: {model['modelIdentifier']}")

# Step 2: Search for specific models
print("\n=== GPT-4 Models ===")
gpt4_models = client.list_models(search="gpt-4", status="active")
for model in gpt4_models["data"]:
    print(f"  - {model['displayName']} ({model['id']})")

# Step 3: Find best model for testing
print("\n=== Recommended Testing Models ===")

# Fast models for development
fast_models = client.list_models(search="mini", status="active")
if fast_models["data"]:
    print(f"Development: {fast_models['data'][0]['displayName']}")

# Production models
prod_models = client.list_models(search="gpt-4", status="active", page_size=1)
if prod_models["data"]:
    print(f"Production: {prod_models['data'][0]['displayName']}")

# Step 4: Check model availability
def check_model_health(model_id):
    models = client.list_models(status="active")
    for model in models["data"]:
        if model["id"] == model_id:
            return True
    return False

test_model_id = "model_abc123"
if check_model_health(test_model_id):
    print(f"\n✓ Model {test_model_id} is available for testing")
else:
    print(f"\n✗ Model {test_model_id} is not available")

Error Handling

Common errors when listing models:

from modelred import APIError, RateLimited, ValidationFailed

try:
    models = client.list_models(page_size=1000)  # Too large
except ValidationFailed as e:
    print(f"Invalid parameters: {e.message}")

try:
    models = client.list_models()
except RateLimited as e:
    print(f"Rate limited: {e.message}")
    # Implement backoff and retry
except APIError as e:
    print(f"API error: {e.message}")

See the Error Handling guide for comprehensive error management.

Next Steps

Create assessments using discovered models
Work with probe packs to define test suites
Implement pagination for efficient model list handling
Handle errors gracefully in production