Overview

ModelRed — production-grade LLM security assessments, probe packs, and operational visibility.

Introduction

ModelRed is a production-grade security platform for language models. Run comprehensive security assessments, track vulnerabilities, and deploy AI systems with confidence.

Key features include:

  • Adaptive red teaming with 10,000+ attack vectors
  • Real-time threat detection and automated responses
  • Comprehensive coverage across all major AI providers
  • Deterministic reports with version pinning and audit trails

Quick Start

Get started with ModelRed in minutes:

lib/modelred.py
from modelred import ModelRed

client = ModelRed(api_key="mr_...")

# Create security assessment
assessment = client.create_assessment_by_id(
    model_id="model_abc123",
    probe_pack_ids=["pack_injection_001"],
    detector_provider="openai",
    detector_api_key="sk-...",
    detector_model="gpt-4o-mini",
)

print(f"Assessment created: {assessment['id']}")

Core Components

ModelRed has several core parts:

Security Assessments

Run comprehensive security tests against your models. Create assessments, monitor progress, and analyze results with versioned probe packs for reproducible testing.

Probe Packs

Curated collections of security tests targeting specific vulnerabilities. Access 10,000+ attack vectors across injection, jailbreak, data exfiltration, and policy violation categories.

Python SDK

Type-safe client with sync and async support. Built-in retries, pagination helpers, and comprehensive error handling make integration seamless.

Real-Time Monitoring

Continuous production monitoring with instant threat detection. Get alerts when vulnerabilities are found and track your security posture over time.

Installation

Install the SDK

Run the installation command:

    pip install modelred

Or with uv:

    uv add modelred

Get Your API Keys You'll need three keys: - ModelRed API key (starts

with mr_...) — From the web app - Detector provider — Either openai or anthropic - Detector API key — Your OpenAI or Anthropic key

Run Your First Assessment

Create an assessment to test your model:

    from modelred import ModelRed
    
    client = ModelRed(api_key="mr_...")
    
    assessment = client.create_assessment_by_id(
        model_id="model_123",
        probe_pack_ids=["pack_1"],
        detector_provider="openai",
        detector_api_key="sk-...",
        detector_model="gpt-4o-mini",
    )

Synchronous Example

Here's a complete workflow from discovery to assessment:

example.py
from modelred import ModelRed

client = ModelRed(api_key="mr_...")

# Discover available models
models = client.list_models()
model_id = models["data"][0]["id"]

# Get probe packs
owned = client.list_owned_probes(page_size=5)
imported = client.list_imported_probes(page_size=5)

probe_ids = []
if owned.get("data"):
    probe_ids.append(owned["data"][0]["id"])
if imported.get("data"):
    probe_ids.append(imported["data"][0]["id"])

# Create assessment
assessment = client.create_assessment_by_id(
    model_id=model_id,
    probe_pack_ids=probe_ids,
    detector_provider="openai",
    detector_api_key="sk-...",
    detector_model="gpt-4o-mini",
)

print(f"Assessment created: {assessment['id']}")

Asynchronous Example

For high-performance concurrent operations, use the async client:

example_async.py
import asyncio
from modelred import AsyncModelRed

async def main():
    async with AsyncModelRed(api_key="mr_...") as client:
        # Discover available models
        models = await client.list_models()
        model_id = models["data"][0]["id"]

        # Get probe packs
        owned = await client.list_owned_probes(page_size=5)
        imported = await client.list_imported_probes(page_size=5)

        probe_ids = []
        if owned.get("data"):
            probe_ids.append(owned["data"][0]["id"])
        if imported.get("data"):
            probe_ids.append(imported["data"][0]["id"])

        # Create assessment
        assessment = await client.create_assessment_by_id(
            model_id=model_id,
            probe_pack_ids=probe_ids,
            detector_provider="anthropic",
            detector_api_key="sk-ant-...",
            detector_model="claude-3-5-sonnet-20241022",
        )

        print(f"Assessment created: {assessment['id']}")

asyncio.run(main())

Supported Providers

ModelRed integrates with all major AI providers:

OpenAI

GPT-4, GPT-3.5, and custom deployments

Anthropic

Claude 3 family and newer releases

Azure OpenAI

Enterprise deployments via Azure

AWS Bedrock

Models hosted on Amazon Bedrock

Google

Gemini and PaLM models

HuggingFace

Open-source and custom models

And more: SageMaker, Grok, OpenRouter, and custom REST endpoints.

Assessment Workflow

Understanding the assessment lifecycle:

Assessment created and waiting for processing. Models and probe packs are validated.

Assessment actively executing probes against the model. Progress updates available.

Assessment finished successfully. Full results and reports available for download.

Assessment encountered an error. Check error details for troubleshooting.

Creating Assessments

Use the API to create security assessments with model ID:

create_assessment_by_id.py
from modelred import ModelRed

client = ModelRed(api_key="mr_...")

# Create with model ID (recommended)
assessment = client.create_assessment_by_id(
    model_id="model_abc123",
    probe_pack_ids=["pack_1", "pack_2"],
    detector_provider="openai",
    detector_api_key="sk-...",
    detector_model="gpt-4o-mini",
    priority="high",
)

Or use model name directly:

create_assessment_by_name.py
from modelred import ModelRed

client = ModelRed(api_key="mr_...")

# Create with model name
assessment = client.create_assessment(
    model="gpt-4-turbo",
    probe_pack_ids=["pack_1", "pack_2"],
    detector_provider="anthropic",
    detector_api_key="sk-ant-...",
    detector_model="claude-3-5-sonnet-20241022",
)

Monitoring Progress

Poll for assessment completion:

monitor.py
import time

assessment_id = assessment["id"]

while True:
    status = client.get_assessment(assessment_id)

    if status["status"] == "COMPLETED":
        print("✓ Assessment complete!")
        print(f"Results: {status['results']}")
        break
    elif status["status"] == "FAILED":
        print("✗ Assessment failed")
        print(f"Error: {status.get('error')}")
        break

    print(f"Status: {status['status']} ({status.get('progress', 0)}%)")
    time.sleep(5)  # Poll every 5 seconds

Production Tip: The SDK automatically retries rate limits and server errors with exponential backoff. Configure retry behavior with the max_retries parameter.

Documentation Structure

Explore the complete documentation:

Core Guides

  • Python SDK — Installation, client setup, and configuration
  • Assessments — Create and manage security assessments
  • Probe Packs — Work with owned and imported probe packs
  • Models — List and filter registered models

Advanced Topics

  • Error Handling — Comprehensive error types and recovery strategies
  • Pagination — Efficient iteration for large datasets
  • Best Practices — Production deployment patterns
  • FAQ — Common questions and troubleshooting

Base URL: The SDK connects to https://www.app.modelred.ai by default. This cannot be changed in the current version.

Next Steps

Ready to dive deeper? Start with the Python SDK guide to learn about client configuration, or jump straight into creating assessments to begin testing your models.

Need help? Check our FAQ for common questions or visit the Error Handling guide for troubleshooting.

Python SDK

Complete guide to the ModelRed Python SDK — installation, configuration, and usage patterns.