Troubleshooting

Step-by-step troubleshooting guide for ModelRed SDK issues

🔧

Troubleshooting Guide

Systematic approach to diagnosing and resolving ModelRed SDK issues with step-by-step solutions and diagnostic tools.

Quick Diagnostic

Health Check Script

Run this diagnostic script to quickly identify common issues.

PYTHON

import asyncio
import os
from modelred import ModelRed, ModelRedError

async def health_check():
    """Comprehensive health check for ModelRed setup"""
    print("🔍 ModelRed Health Check")
    print("=" * 40)

    # Check 1: API Key
    api_key = os.getenv("MODELRED_API_KEY")
    if not api_key:
        print("❌ MODELRED_API_KEY environment variable not set")
        print("   Fix: Set your API key in environment")
        return False

    if not api_key.startswith("mr_"):
        print("❌ Invalid API key format")
        print("   Fix: API key must start with 'mr_'")
        return False

    print("✅ API key format is valid")

    # Check 2: Authentication
    try:
        async with ModelRed() as client:
            account = await client.validate_api_key()
            print(f"✅ Authentication successful")
            print(f"   Organization: {account.get('organization', 'N/A')}")

            # Check 3: Usage stats
            usage = await client.get_usage_stats()
            print(f"✅ Usage stats retrieved")
            print(f"   Plan: {usage.tier}")
            print(f"   Models: {usage.models_registered}/{usage.models_limit}")
            print(f"   Assessments: {usage.assessments_this_month}/{usage.assessments_limit}")

            # Check 4: Providers
            providers = await client.get_providers()
            print(f"✅ Providers loaded: {len(providers)} available")

            # Check 5: Models
            models = await client.list_models()
            print(f"✅ Models accessible: {len(models)} registered")

            # Check 6: Test suites
            suites = await client.get_test_suites()
            print(f"✅ Test suites available: {len(suites)} suites")

            print("\n🎉 All checks passed! ModelRed is ready to use.")
            return True

    except ModelRedError as e:
        print(f"❌ Health check failed: {e.message}")
        print("   See troubleshooting steps below")
        return False

# Run health check
if __name__ == "__main__":
    asyncio.run(health_check())

Authentication Issues

🔑 API Key Problems

Problem: "Invalid API key" error

Step 1: Verify API key format

API key must start with "mr_" and be 64+ characters

Step 2: Check environment variable

echo $MODELRED_API_KEY

Step 3: Test API key directly

import os
from modelred import ModelRed

api_key = os.getenv("MODELRED_API_KEY")
print(f"API Key length: {len(api_key) if api_key else 0}")
print(f"Starts with mr_: {api_key.startswith('mr_') if api_key else False}")

Problem: "Feature requires higher tier" error

Step 1: Check current tier

async with ModelRed() as client:
    usage = await client.get_usage_stats()
    print(f"Current tier: {usage.tier}")

Step 2: Identify required tier

providers = await client.get_providers()
for provider in providers:
    print(f"{provider.name}: requires {provider.tier_required}")

Connection Problems

🌐 Network & Connectivity Issues

Connection Timeout

Symptoms: Requests hang or timeout

TROUBLESHOOTING STEPS

import asyncio
import aiohttp

async def test_connectivity():
    """Test connection to ModelRed API"""
    timeout = aiohttp.ClientTimeout(total=10)

    try:
        async with aiohttp.ClientSession(timeout=timeout) as session:
            async with session.get("http://localhost:3000/api/health") as response:
                if response.status == 200:
                    print("✅ ModelRed API is reachable")
                else:
                    print(f"⚠️ API returned status: {response.status}")

    except asyncio.TimeoutError:
        print("❌ Connection timeout - check network/firewall")
    except Exception as e:
        print(f"❌ Connection failed: {e}")

asyncio.run(test_connectivity())

Provider Connection Issues

When models fail to respond during assessments

DIAGNOSIS SCRIPT

async def test_model_connection(model_id):
    """Test if a registered model is accessible"""
    async with ModelRed() as client:
        try:
            # Get model details
            model = await client.get_model(model_id)
            print(f"✅ Model found: {model['displayName']}")
            print(f"   Provider: {model['provider']}")
            print(f"   Status: {model.get('status', 'Unknown')}")

            # Test with simple assessment (don't wait)
            result = await client.run_assessment(
                model_id=model_id,
                test_suites=["basic_security"],
                wait_for_completion=False
            )
            print(f"✅ Assessment started: {result.assessment_id}")

        except Exception as e:
            print(f"❌ Model test failed: {e}")
            print("   Check provider API keys and endpoints")

# Test your model
await test_model_connection("your-model-id")

Assessment Failures

🧪 Assessment Debugging

Assessment Gets Stuck

MONITORING SCRIPT

async def monitor_stuck_assessment(assessment_id):
    """Monitor and diagnose stuck assessments"""
    async with ModelRed() as client:
        last_progress = -1
        stuck_count = 0

        for i in range(10):  # Check 10 times
            try:
                status = await client.get_assessment_status(assessment_id)
                progress = status.get('progress', 0)

                print(f"Check {i+1}: {status['status']} ({progress}%)")

                if progress == last_progress:
                    stuck_count += 1
                    if stuck_count >= 3:
                        print("⚠️ Assessment appears stuck")
                        print("   Consider canceling and retrying")
                        break
                else:
                    stuck_count = 0
                    last_progress = progress

                if status['status'] in ['COMPLETED', 'FAILED']:
                    break

                await asyncio.sleep(30)

            except Exception as e:
                print(f"❌ Monitoring error: {e}")
                break

# Monitor your assessment
await monitor_stuck_assessment("your-assessment-id")

Assessment Fails Immediately

Check 1: Model configuration

Check 2: Test suite compatibility

Check 3: Provider API status

VALIDATION SCRIPT

async def validate_assessment_setup(model_id, test_suites):
    """Validate assessment prerequisites"""
    async with ModelRed() as client:
        # Check model exists
        try:
            model = await client.get_model(model_id)
            print(f"✅ Model exists: {model['displayName']}")
        except:
            print(f"❌ Model '{model_id}' not found")
            return False

        # Check test suites
        available_suites = await client.get_test_suites()
        available_names = [s['name'] for s in available_suites]

        for suite in test_suites:
            if suite in available_names:
                print(f"✅ Test suite '{suite}' available")
            else:
                print(f"❌ Test suite '{suite}' not available")
                print(f"   Available: {available_names}")
                return False

        return True

# Validate before running assessment
valid = await validate_assessment_setup(
    "your-model-id",
    ["basic_security", "content_safety"]
)

if valid:
    print("✅ Setup is valid, proceed with assessment")
else:
    print("❌ Fix issues before running assessment")

Performance Issues

⚡ Slow Performance Diagnosis

PERFORMANCE PROFILER

import time
from datetime import datetime

class PerformanceProfiler:
    def __init__(self):
        self.timings = {}

    def start_timer(self, operation):
        self.timings[operation] = {'start': time.time()}

    def end_timer(self, operation):
        if operation in self.timings:
            self.timings[operation]['end'] = time.time()
            duration = self.timings[operation]['end'] - self.timings[operation]['start']
            self.timings[operation]['duration'] = duration
            print(f"⏱️ {operation}: {duration:.2f}s")

    def report(self):
        print("\n📊 Performance Report:")
        for op, timing in self.timings.items():
            if 'duration' in timing:
                print(f"   {op}: {timing['duration']:.2f}s")

async def profile_modelred_operations():
    """Profile common ModelRed operations"""
    profiler = PerformanceProfiler()

    async with ModelRed() as client:
        # Profile authentication
        profiler.start_timer("Authentication")
        await client.validate_api_key()
        profiler.end_timer("Authentication")

        # Profile model listing
        profiler.start_timer("List Models")
        models = await client.list_models()
        profiler.end_timer("List Models")

        # Profile usage stats
        profiler.start_timer("Usage Stats")
        usage = await client.get_usage_stats()
        profiler.end_timer("Usage Stats")

        # Profile test suites
        profiler.start_timer("Get Test Suites")
        suites = await client.get_test_suites()
        profiler.end_timer("Get Test Suites")

        profiler.report()

        # Analyze results
        auth_time = profiler.timings.get('Authentication', {}).get('duration', 0)
        if auth_time > 5:
            print("⚠️ Slow authentication - check network connection")

        list_time = profiler.timings.get('List Models', {}).get('duration', 0)
        if list_time > 3:
            print("⚠️ Slow model listing - may indicate server issues")

# Run performance profile
await profile_modelred_operations()

Common Solutions

💡 Quick Fixes for Common Issues

Environment Issues

✅

Restart Python interpreterClear import cache issues

✅

Check virtual environmentEnsure correct Python env

✅

Update ModelRed SDKpip install --upgrade modelred

Configuration Issues

✅

Reload environment variablessource ~/.bashrc or restart terminal

✅

Verify .env file locationShould be in project root

✅

Check file permissionsEnsure .env is readable

Getting Help

🆘 When to Contact Support

Before Contacting Support

• Run the health check script above

• Check your subscription tier and limits

• Try the quick fixes in common solutions

• Note exact error messages and steps to reproduce

Information to Include

• ModelRed SDK version: pip show modelred

• Python version and OS

• Complete error message and stack trace

• Steps to reproduce the issue

• Your subscription tier and usage stats

Next Steps

🐛

Debugging

Advanced debugging techniques and logging

📋

Common Issues

Frequently encountered problems and solutions

Troubleshooting