Test Schedules

Schedules let you run your test scenarios automatically on a recurring basis. Think of them as your agent’s automated quality assurance system that works while you sleep.

Why Automate Testing?

Manual testing works for initial validation, but you need automation to:

Catch regressions - Know immediately when code changes break existing functionality
Monitor production - Run daily tests against live agents to detect quality degradation
Save time - Test continuously without manual effort
Build confidence - Deploy knowing your agents are consistently tested

Example: Schedule a nightly test that runs 20 scenarios against your production agent. Wake up to a report showing if everything still works perfectly.

How Schedules Work

When you create a scenario, you can optionally add a schedule to run it automatically:

Scenario + Schedule = Automatic Testing

"Refund Request" scenario
+ Daily at 2 AM
= Automatic quality checks every night

Each scheduled run creates new simulations that you can review in your dashboard or via API.

Creating a Schedule

Schedules are configured directly in your scenarios:

Via UI
Via API

When creating or editing a scenario:

Navigate to the scenario settings
Toggle “Enable Schedule”
Choose frequency (Once, Daily, Weekly, Monthly)
Set end condition (Never, End Date, After N Runs)
Save the scenario

curl -X POST https://api.chanl.ai/v1/scenarios \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Daily Production Test",
    "prompt": "Customer requests refund for defective product",
    "personas": ["frustrated", "analytical"],
    "agents": ["prod-agent"],
    "scorecard": "customer-service",
    "schedule": {
      "frequency": "daily",
      "time": "02:00",
      "timezone": "America/New_York",
      "endCondition": "never"
    }
  }'

Scheduling Options

Frequency Types

Once

Use for: One-time validationRuns immediately when created. Can be manually triggered again later.

{
  "frequency": "once"
}

Daily

Use for: Continuous monitoringRuns every day at specified time. Perfect for regression testing.

{
  "frequency": "daily",
  "time": "02:00",
  "timezone": "UTC"
}

Weekly

Use for: Regular checkpointsRuns once per week on specified day.

{
  "frequency": "weekly",
  "day": "monday",
  "time": "08:00",
  "timezone": "UTC"
}

Monthly

Use for: Quarterly auditsRuns once per month on specified date.

{
  "frequency": "monthly",
  "dayOfMonth": 1,
  "time": "00:00",
  "timezone": "UTC"
}

End Conditions

Control when your schedule stops running:

Never

Schedule runs indefinitely until manually stopped.Use for: Production monitoring that should run continuously

{
  "endCondition": "never"
}

End Date

Schedule stops after a specific date.Use for: Time-limited testing periods or trials

{
  "endCondition": "end_date",
  "endDate": "2024-12-31"
}

After N Runs

Schedule stops after specified number of executions.Use for: Fixed testing cycles (e.g., 30 days of daily tests)

{
  "endCondition": "after_runs",
  "maxRuns": 30
}

Common Scheduling Patterns

Continuous Production Monitoring

Run critical scenarios every night to catch issues early:

{
  "name": "Nightly Production Health Check",
  "scenarios": [
    "refund-request",
    "angry-customer",
    "complex-technical-issue"
  ],
  "schedule": {
    "frequency": "daily",
    "time": "02:00",
    "timezone": "America/New_York",
    "endCondition": "never"
  },
  "notifications": {
    "onFailure": true,
    "recipients": ["ops@company.com"]
  }
}

Pre-Deployment Validation

Test before each deployment:

{
  "name": "Pre-Deploy Validation",
  "scenarios": ["smoke-tests"],
  "schedule": {
    "frequency": "once"
  },
  "notifications": {
    "onCompletion": true,
    "slack": "#deployments"
  }
}

Weekly Regression Suite

Comprehensive testing every weekend:

{
  "name": "Weekend Regression Suite",
  "scenarios": [
    "all-customer-service-scenarios",
    "all-sales-scenarios",
    "edge-cases"
  ],
  "schedule": {
    "frequency": "weekly",
    "day": "saturday",
    "time": "00:00",
    "timezone": "UTC",
    "endCondition": "never"
  }
}

Managing Schedules

Viewing Active Schedules

Via UI
Via API

Navigate to Schedules in the sidebar to see:

Active schedules
Next run time
Last run date and status
Total number of runs

curl https://api.chanl.ai/v1/schedules \
  -H "Authorization: Bearer YOUR_API_KEY"

Response:

{
  "schedules": [
    {
      "id": "sched_123",
      "name": "Nightly Production Test",
      "frequency": "daily",
      "active": true,
      "lastRun": "2024-01-15T02:00:00Z",
      "nextRun": "2024-01-16T02:00:00Z",
      "totalRuns": 47,
      "avgScore": 87.3
    }
  ]
}

Pausing a Schedule

Temporarily stop a schedule without deleting it:

curl -X PATCH https://api.chanl.ai/v1/schedules/sched_123 \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "active": false
  }'

When to pause:

During maintenance windows
When testing major agent changes
Temporarily reducing API usage

Reactivating a Schedule

Resume a paused schedule:

curl -X PATCH https://api.chanl.ai/v1/schedules/sched_123 \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "active": true
  }'

Schedule Notifications

Get alerted when tests complete or fail:

Email Notifications

{
  "notifications": {
    "email": {
      "onFailure": true,
      "onSuccess": false,
      "recipients": [
        "team@company.com",
        "oncall@company.com"
      ]
    }
  }
}

Slack Integration

{
  "notifications": {
    "slack": {
      "webhook": "https://hooks.slack.com/...",
      "channel": "#ai-agent-alerts",
      "onFailure": true,
      "scoreThreshold": 80
    }
  }
}

Webhook Integration

Send results to your own systems:

{
  "notifications": {
    "webhook": {
      "url": "https://your-system.com/chanl-results",
      "method": "POST",
      "headers": {
        "Authorization": "Bearer YOUR_TOKEN"
      },
      "onCompletion": true
    }
  }
}

Monitoring Schedule Performance

Key Metrics to Track

Success Rate

Percentage of scheduled runs that complete successfully

Average Score

Mean quality score across all scheduled runs

Score Trends

Whether quality is improving, stable, or degrading over time

Run Duration

How long tests take to complete

Analyzing Trends

curl https://api.chanl.ai/v1/schedules/sched_123/analytics?days=30 \
  -H "Authorization: Bearer YOUR_API_KEY"

Response shows trends over time:

{
  "schedule": "Nightly Production Test",
  "period": "30 days",
  "totalRuns": 30,
  "successRate": 96.7,
  "avgScore": 87.3,
  "trend": "stable",
  "alerts": [
    {
      "date": "2024-01-10",
      "issue": "Score dropped to 72",
      "cause": "Agent update deployed"
    }
  ]
}

Best Practices

Start with Critical Scenarios

Schedule your most important test scenarios first. Don’t try to automate everything at once.

Run During Low-Traffic Periods

Schedule tests for nights or weekends to avoid impacting production systems during peak hours.

Set Up Alerts

Configure notifications so you know immediately when quality drops below acceptable levels.

Review Trends Weekly

Don’t just look at individual runs. Monitor trends over time to catch gradual degradation.

Adjust Based on Results

If scenarios consistently pass, reduce frequency. If they catch issues, increase frequency.

Programmatic Schedule Management

Create and manage schedules via API:

// create-validation-schedule.js
const chanl = require('@chanl/sdk');

async function createValidationSchedule(agentId) {
  // Create one-time validation schedule
  const schedule = await chanl.schedules.create({
    name: `Agent Validation - ${new Date().toISOString()}`,
    scenarios: [
      'smoke-test-scenario',
      'critical-paths-scenario'
    ],
    frequency: 'once',
    notifications: {
      webhook: {
        url: process.env.WEBHOOK_URL,
        onCompletion: true
      }
    }
  });

  // Wait for completion
  const results = await chanl.schedules.waitForCompletion(schedule.id, {
    timeout: 600000 // 10 minutes
  });

  // Check results
  if (results.avgScore < 80) {
    throw new Error(`Validation failed. Score: ${results.avgScore}`);
  }

  console.log(`✅ Validation passed. Score: ${results.avgScore}`);
  return results;
}

createValidationSchedule(process.env.AGENT_ID);

Troubleshooting

Schedule not running

Problem: Schedule shows as active but isn’t executingCheck:

Verify timezone is correct
Ensure end condition hasn’t been reached
Check if schedule was manually paused
Verify API key has necessary permissions

Tests failing consistently

Problem: Scheduled tests suddenly showing low scoresInvestigate:

Review recent agent configuration changes
Check if test scenarios need updating
Verify scorecard criteria haven’t changed
Look for patterns (specific personas failing?)

Missing notifications

Problem: Not receiving schedule completion alertsSolutions:

Verify notification settings in schedule configuration
Check spam/junk folders for emails
Test webhook URLs are accessible
Confirm Slack integration is properly configured

High API usage

Problem: Schedules consuming too many API callsSolutions:

Reduce frequency of less critical schedules
Decrease number of personas/agents in scenarios
Pause schedules during testing periods
Contact support to discuss quota limits

What’s Next?

Create Scenarios

Build scenarios to schedule for automated testing

Review Simulations

Analyze results from scheduled test runs

Set Up Alerts

Configure notifications for test failures

API Reference

Complete API documentation for schedules

Overview

Test

Observe

Optimize

​Test Schedules

​Why Automate Testing?

​How Schedules Work

​Creating a Schedule

​Scheduling Options

​Frequency Types

Once

Daily

Weekly

Monthly

​End Conditions

​Common Scheduling Patterns

​Continuous Production Monitoring

​Pre-Deployment Validation

​Weekly Regression Suite

​Managing Schedules

​Viewing Active Schedules

​Pausing a Schedule

​Reactivating a Schedule

​Schedule Notifications

​Email Notifications

​Slack Integration

​Webhook Integration

​Monitoring Schedule Performance

​Key Metrics to Track

Success Rate

Average Score

Score Trends

Run Duration

​Analyzing Trends

​Best Practices

​Programmatic Schedule Management

​Troubleshooting

​What’s Next?

Create Scenarios

Review Simulations

Set Up Alerts

API Reference

Test Schedules

Why Automate Testing?

How Schedules Work

Creating a Schedule

Scheduling Options

Frequency Types

End Conditions

Common Scheduling Patterns

Continuous Production Monitoring

Pre-Deployment Validation

Weekly Regression Suite

Managing Schedules

Viewing Active Schedules

Pausing a Schedule

Reactivating a Schedule

Schedule Notifications

Email Notifications

Slack Integration

Webhook Integration

Monitoring Schedule Performance

Key Metrics to Track

Analyzing Trends

Best Practices

Programmatic Schedule Management

Troubleshooting

What’s Next?