Assessments | ModelRed

Introduction

Assessments are security evaluation jobs that test your models against probe packs using a detector model to identify vulnerabilities. They provide comprehensive reports on prompt injection, jailbreaks, data exfiltration, and policy violations.

Creating Assessments

Basic Creation

Use create_assessment() or create_assessment_by_id() to start a new assessment:

from modelred import ModelRed

client = ModelRed(api_key="mr_...")

assessment = client.create_assessment_by_id(
    model_id="model_abc123",
    probe_pack_ids=["pack_1", "pack_2"],
    detector_provider="openai",
    detector_api_key="sk-...",
    detector_model="gpt-4o-mini",
)

print(f"Created: {assessment['id']}")

Using Model Name

Alternatively, use the model name instead of ID:

assessment = client.create_assessment(
    model="gpt-4",
    probe_pack_ids=["pack_1", "pack_2"],
    detector_provider="anthropic",
    detector_api_key="sk-ant-...",
    detector_model="claude-3-5-sonnet-20241022",
)

Required Parameters

Every assessment requires these detector configuration parameters:

Prop	Type	Default
`detector_provider?`	`"openai" \| "anthropic"`	-
`detector_api_key?`	`string`	-
`detector_model?`	`string`	-

Optional Parameters

Prop	Type	Default
`model?`	`string \| None`	-
`model_id?`	`string \| None`	-
`probe_pack_ids?`	`list[str]`	-
`priority?`	`"low" \| "normal" \| "high" \| "critical"`	`"normal"`
`detector_base_url?`	`string \| None`	-
`detector_organization?`	`string \| None`	-

OpenAI Custom Configuration

For OpenAI-compatible endpoints or organization-scoped keys:

assessment = client.create_assessment_by_id(
    model_id="model_123",
    probe_pack_ids=["pack_1"],
    detector_provider="openai",
    detector_api_key="sk-...",
    detector_model="gpt-4o",
    detector_base_url="https://api.custom-openai.com/v1",
    detector_organization="org-xyz",
)

Priority Levels

Set priority to control queue ordering:

    # Background assessments, processed when resources available
    assessment = client.create_assessment_by_id(
        model_id="model_123",
        probe_pack_ids=["pack_1"],
        priority="low",
        detector_provider="openai",
        detector_api_key="sk-...",
        detector_model="gpt-4o-mini",
    )

client.create_assessment_by_id( model_id="model_123",
probe_pack_ids=["pack_1"], priority="normal", # Default
detector_provider="openai", detector_api_key="sk-...",
detector_model="gpt-4o-mini", ) ```
</Tab>

<Tab value="High">
```python # Prioritized assessments, processed before normal assessment =
client.create_assessment_by_id( model_id="model_123",
probe_pack_ids=["pack_1"], priority="high", detector_provider="openai",
detector_api_key="sk-...", detector_model="gpt-4o-mini", ) ```
</Tab>

<Tab value="Critical">
```python
  # Highest priority, processed immediately
  assessment = client.create_assessment_by_id(
      model_id="model_123",
      probe_pack_ids=["pack_1"],
      priority="critical",
      detector_provider="openai",
      detector_api_key="sk-...",
      detector_model="gpt-4o-mini",
  )

Listing Assessments

Basic Listing

Retrieve paginated assessment lists:

response = client.list_assessments(
    page=1,
    page_size=20,
)

for assessment in response["data"]:
    print(f"{assessment['id']}: {assessment['status']}")

Filtering by Status

# Get only completed assessments
completed = client.list_assessments(
    status="COMPLETED",
    page_size=50,
)

# Get running assessments
running = client.list_assessments(
    status="RUNNING",
)

# Get failed assessments
failed = client.list_assessments(
    status="FAILED",
)

Filtering by Provider

# OpenAI models only
openai_assessments = client.list_assessments(
    provider="openai",
)

# Anthropic models only
anthropic_assessments = client.list_assessments(
    provider="anthropic",
)

Search and Sort

# Search by model name or assessment ID
results = client.list_assessments(
    search="gpt-4",
    sort_by="createdAt",
    sort_dir="desc",
)

# Sort by different fields
results = client.list_assessments(
    sort_by="status",
    sort_dir="asc",
)

List Parameters

Prop	Type	Default
`page?`	`int`	`1`
`page_size?`	`int`	`10`
`search?`	`string \| None`	-
`status?`	`"all" \| "QUEUED" \| "RUNNING" \| "COMPLETED" \| "FAILED" \| "CANCELLED"`	`"all"`
`provider?`	`"all" \| "openai" \| "anthropic" \| "azure" \| ...`	`"all"`
`sort_by?`	`string`	`"createdAt"`
`sort_dir?`	`"asc" \| "desc"`	`"desc"`

Iterating All Assessments

Use the iterator helper for automatic pagination:

# Iterate through all assessments
for assessment in client.iter_assessments(page_size=50):
    print(f"{assessment['id']}: {assessment['status']}")

# With filters
for assessment in client.iter_assessments(
    page_size=50,
    status="COMPLETED",
    provider="openai",
):
    print(f"Completed: {assessment['id']}")

See the Pagination guide for more details.

Retrieving a Single Assessment

Get detailed information about a specific assessment:

assessment = client.get_assessment("assessment_abc123")

print(f"Status: {assessment['status']}")
print(f"Model: {assessment['model']}")
print(f"Created: {assessment['createdAt']}")
print(f"Progress: {assessment.get('progress', 'N/A')}")

Assessment Status Lifecycle

Assessments progress through these states:

Assessment created and waiting for processing.

Assessment actively executing probes against the model.

Assessment finished successfully. Results available.

Assessment encountered an error. Check error details.

Assessment was manually cancelled (web UI only).

Cancelling Assessments

Note: Cancellation requires web UI access. API keys cannot cancel assessments.

print("Cancelled successfully") except NotAllowedForApiKey as e: print("Use the
web UI to cancel assessments") ```

## Response Structure

Assessment responses include:

```python
{
    "id": "assessment_abc123",
    "status": "COMPLETED",
    "model": "gpt-4",
    "modelId": "model_xyz",
    "priority": "normal",
    "createdAt": "2025-01-15T10:30:00Z",
    "updatedAt": "2025-01-15T10:35:00Z",
    "completedAt": "2025-01-15T10:35:00Z",
    "probePackIds": ["pack_1", "pack_2"],
    "detectorConfig": {
        "provider": "openai",
        "model": "gpt-4o-mini"
    },
    "results": {
        # Assessment results when completed
    }
}

Async Examples

All operations have async equivalents:

import asyncio
from modelred import AsyncModelRed

async def main():
    async with AsyncModelRed(api_key="mr_...") as client:
        # Create
        assessment = await client.create_assessment_by_id(
            model_id="model_123",
            probe_pack_ids=["pack_1"],
            detector_provider="anthropic",
            detector_api_key="sk-ant-...",
            detector_model="claude-3-5-sonnet-20241022",
        )

        # List
        assessments = await client.list_assessments(status="COMPLETED")

        # Get
        detail = await client.get_assessment(assessment["id"])

asyncio.run(main())

Best Practices

Choose Appropriate Detectors

gpt-4o-mini

Fast, cost-effective for most use cases. Ideal for development and regular testing.

gpt-4o

More thorough analysis, higher cost. Use for production validation and important updates.

claude-3-5-sonnet

Alternative perspective, good for cross-validation. Best for critical assessments.

Use Priority Wisely

    # Low priority for background testing
    assessment = client.create_assessment_by_id(
        model_id="model_123",
        probe_pack_ids=["pack_1"],
        priority="low",
        detector_provider="openai",
        detector_api_key="sk-...",
        detector_model="gpt-4o-mini",
    )

client.create_assessment_by_id( model_id="model_123",
probe_pack_ids=["pack_1"], priority="normal", detector_provider="openai",
detector_api_key="sk-...", detector_model="gpt-4o", ) ```
</Tab>

<Tab value="Production">
```python
  # High/Critical for incidents
  assessment = client.create_assessment_by_id(
      model_id="model_123",
      probe_pack_ids=["pack_1"],
      priority="critical",
      detector_provider="anthropic",
      detector_api_key="sk-ant-...",
      detector_model="claude-3-5-sonnet-20241022",
  )

Monitor Assessment Status

Poll for completion or implement webhooks (if available):

import time

assessment = client.create_assessment_by_id(...)
assessment_id = assessment["id"]

while True:
    status = client.get_assessment(assessment_id)
    if status["status"] in ("COMPLETED", "FAILED", "CANCELLED"):
        break
    time.sleep(5)  # Poll every 5 seconds

print(f"Final status: {status['status']}")

Error Handling

Common errors when creating assessments:

from modelred import ValidationFailed, NotFound, LimitExceeded

try:
    assessment = client.create_assessment_by_id(
        model_id="invalid",
        probe_pack_ids=["pack_1"],
        detector_provider="openai",
        detector_api_key="sk-...",
        detector_model="gpt-4o-mini",
    )
except ValidationFailed as e:
    print(f"Invalid parameters: {e.message}")
except NotFound as e:
    print(f"Model or probe pack not found: {e.message}")
except LimitExceeded as e:
    print(f"Plan limit reached: {e.message}")

See the Error Handling guide for comprehensive error management.

Complete Workflow Example

complete_assessment.py

from modelred import ModelRed
import time
import os

client = ModelRed(api_key=os.environ["MODELRED_API_KEY"])

# Step 1: Get available models
models = client.list_models(provider="openai", status="active")
if not models["data"]:
    print("No active models found")
    exit(1)

model_id = models["data"][0]["id"]
print(f"Testing model: {models['data'][0]['displayName']}")

# Step 2: Get probe packs
owned = client.list_owned_probes(category="injection", page_size=1)
imported = client.list_imported_probes(category="jailbreak", page_size=1)

probe_ids = []
if owned.get("data"):
    probe_ids.append(owned["data"][0]["id"])
if imported.get("data"):
    probe_ids.append(imported["data"][0]["id"])

if not probe_ids:
    print("No probe packs available")
    exit(1)

print(f"Using {len(probe_ids)} probe packs")

# Step 3: Create assessment
assessment = client.create_assessment_by_id(
    model_id=model_id,
    probe_pack_ids=probe_ids,
    detector_provider="openai",
    detector_api_key=os.environ["OPENAI_API_KEY"],
    detector_model="gpt-4o-mini",
    priority="high",
)

assessment_id = assessment["id"]
print(f"Assessment created: {assessment_id}")

# Step 4: Monitor progress
while True:
    status = client.get_assessment(assessment_id)
    current_status = status["status"]

    if current_status == "COMPLETED":
        print("\n✓ Assessment complete!")
        print(f"Results: {status.get('results', {})}")
        break
    elif current_status == "FAILED":
        print("\n✗ Assessment failed")
        print(f"Error: {status.get('error')}")
        break

    progress = status.get("progress", 0)
    print(f"Status: {current_status} ({progress}%)", end="\r")
    time.sleep(5)

Pro Tip: Use priority levels strategically. Set priority="critical" for production incidents and priority="low" for background monitoring to optimize queue processing.

Next Steps

Work with probe packs to understand assessment inputs
List models to discover available models for testing
Implement pagination for efficient data retrieval
Handle errors to build robust integrations