Orchestration

This guide covers setting up and using multi-region orchestration as a standalone HTTP server.

Overview

The orchestrator allows you to:

Run the same tests across multiple regions
Execute tests in parallel
Aggregate results from distributed runs
Coordinate execution across multiple runner endpoints

Architecture

Client
  ↓
Orchestrator (single request)
  ↓
fanoutJobs() - Groups runners by URL/region
  ↓
Jobs Created (one per URL-region combination)
  ↓
Parallel Execution
  ├─→ Remote Runner (us-east-1)
  ├─→ Remote Runner (eu-west-1)
  └─→ Remote Runner (ap-southeast-1)
  ↓
Results Aggregated
  ↓
Single Run Summary

Setup

Installation

pnpm add runners

Create Orchestrator Server

Create a standalone orchestrator server:

// server.ts
import { createOrchestratorHandler } from "runners/orchestrator";

export default createOrchestratorHandler();

The handler provides orchestrator endpoints that can be integrated with any HTTP server framework.

Configure Runner Endpoints

Set the PLAYWRIGHT_RUNNERS environment variable:

export PLAYWRIGHT_RUNNERS='{
  "us-east-1": "https://runner-us-east-1.example.com/api/runner",
  "eu-west-1": "https://runner-eu-west-1.example.com/api/runner",
  "ap-southeast-1": "https://runner-ap-southeast-1.example.com/api/runner"
}'

Usage

Submit a Run Request

curl -X POST http://orchestrator.example.com/api/orchestrator \
  -H "Content-Type: application/json" \
  -d '{
    "runners": [
      {
        "name": "cookieBannerTest",
        "region": "us-east-1",
        "input": {
          "url": "https://example.com"
        }
      },
      {
        "name": "cookieBannerTest",
        "region": "eu-west-1",
        "input": {
          "url": "https://example.com"
        }
      }
    ],
    "mode": "remote"
  }'

Response:

{
  "runId": "run_abc123"
}

Check Run Status

curl http://orchestrator.example.com/api/orchestrator/run_abc123/status

Response:

{
  "runId": "run_abc123",
  "state": "running",
  "totalJobs": 2,
  "completedJobs": 1,
  "failedJobs": 0,
  "createdAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-01T00:01:00Z"
}

Get Run Results

curl http://orchestrator.example.com/api/orchestrator/run_abc123

Response:

{
  "runId": "run_abc123",
  "state": "completed",
  "jobs": [
    {
      "jobId": "job_123",
      "region": "us-east-1",
      "state": "completed",
      "results": [
        {
          "name": "cookieBannerTest",
          "status": "pass",
          "durationMs": 1234
        }
      ],
      "startedAt": "2024-01-01T00:00:00Z",
      "completedAt": "2024-01-01T00:01:00Z",
      "durationMs": 60000
    }
  ],
  "summary": {
    "total": 2,
    "passed": 2,
    "failed": 0,
    "errored": 0
  },
  "createdAt": "2024-01-01T00:00:00Z",
  "completedAt": "2024-01-01T00:01:00Z",
  "durationMs": 62000
}

API Endpoints

The orchestrator provides:

POST /api/orchestrator - Submit a new run request
GET /api/orchestrator/{runId}/status - Get run status
GET /api/orchestrator/{runId} - Get run results
GET /api/orchestrator/docs - Interactive API documentation
GET /api/orchestrator/spec.json - OpenAPI specification

Run Modes

Remote Mode

Execute runners on remote endpoints:

{
  "runners": [
    {
      "name": "cookieBannerTest",
      "region": "us-east-1",
      "input": { "url": "https://example.com" }
    }
  ],
  "mode": "remote"
}

Requirements:

Each runner must specify a region
PLAYWRIGHT_RUNNERS must be configured
Runner endpoints must be accessible

Local Mode

Execute runners locally (for testing):

{
  "runners": [
    {
      "name": "cookieBannerTest",
      "input": { "url": "https://example.com" }
    }
  ],
  "mode": "local"
}

Requirements:

Runners must be available locally
Each runner must specify a url in input

Concurrency Control

Control how many jobs run in parallel:

{
  "runners": [ /* ... */ ],
  "mode": "remote",
  "concurrency": 3
}

Default: Unlimited (all jobs run in parallel)
Limited: Set concurrency to limit parallel jobs
Sequential: Set concurrency: 1 to run one at a time

Timeouts

Set timeout for entire run:

{
  "runners": [ /* ... */ ],
  "mode": "remote",
  "timeout": 60000
}

Timeout applies to entire run (all jobs)
Individual job timeouts are handled by runner endpoints
Run status becomes "timed_out" if timeout exceeded

Best Practices

1. Use Consistent URLs

Group runners by URL to minimize job count:

{
  "runners": [
    { "name": "test1", "region": "us-east-1", "input": { "url": "https://example.com" } },
    { "name": "test2", "region": "us-east-1", "input": { "url": "https://example.com" } }
  ]
}

2. Set Appropriate Timeouts

{
  "timeout": 300000  // 5 minutes for slow tests
}

3. Monitor Run Status

Poll status endpoint for long-running runs:

async function waitForCompletion(runId: string) {
  while (true) {
    const status = await getRunStatus(runId);
    if (status.state === "completed" || status.state === "failed") {
      return status;
    }
    await sleep(5000); // Poll every 5 seconds
  }
}

4. Handle Failures Gracefully

const summary = await getRunResults(runId);

if (summary.state === "failed") {
  const failedJobs = summary.jobs.filter(job => job.state === "failed");
  console.error(`Failed jobs: ${failedJobs.length}`);
  
  for (const job of failedJobs) {
    console.error(`Region ${job.region}: ${job.error}`);
  }
}

Overview

The orchestrator allows you to:

Run the same tests across multiple regions
Execute tests in parallel
Aggregate results from distributed runs
Coordinate execution across multiple runner endpoints

Architecture

Client
  ↓
Orchestrator (single request)
  ↓
fanoutJobs() - Groups runners by URL/region
  ↓
Jobs Created (one per URL-region combination)
  ↓
Parallel Execution
  ├─→ Remote Runner (us-east-1)
  ├─→ Remote Runner (eu-west-1)
  └─→ Remote Runner (ap-southeast-1)
  ↓
Results Aggregated
  ↓
Single Run Summary

Setup

Installation

pnpm add runners

Create Orchestrator Server

Create a standalone orchestrator server:

// server.ts
import { createOrchestratorHandler } from "runners/orchestrator";

export default createOrchestratorHandler();

The handler provides orchestrator endpoints that can be integrated with any HTTP server framework.

Configure Runner Endpoints

Set the PLAYWRIGHT_RUNNERS environment variable:

export PLAYWRIGHT_RUNNERS='{
  "us-east-1": "https://runner-us-east-1.example.com/api/runner",
  "eu-west-1": "https://runner-eu-west-1.example.com/api/runner",
  "ap-southeast-1": "https://runner-ap-southeast-1.example.com/api/runner"
}'

Usage

Submit a Run Request

curl -X POST http://orchestrator.example.com/api/orchestrator \
  -H "Content-Type: application/json" \
  -d '{
    "runners": [
      {
        "name": "cookieBannerTest",
        "region": "us-east-1",
        "input": {
          "url": "https://example.com"
        }
      },
      {
        "name": "cookieBannerTest",
        "region": "eu-west-1",
        "input": {
          "url": "https://example.com"
        }
      }
    ],
    "mode": "remote"
  }'

Response:

{
  "runId": "run_abc123"
}

Check Run Status

curl http://orchestrator.example.com/api/orchestrator/run_abc123/status

Response:

{
  "runId": "run_abc123",
  "state": "running",
  "totalJobs": 2,
  "completedJobs": 1,
  "failedJobs": 0,
  "createdAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-01T00:01:00Z"
}

Get Run Results

curl http://orchestrator.example.com/api/orchestrator/run_abc123

Response:

{
  "runId": "run_abc123",
  "state": "completed",
  "jobs": [
    {
      "jobId": "job_123",
      "region": "us-east-1",
      "state": "completed",
      "results": [
        {
          "name": "cookieBannerTest",
          "status": "pass",
          "durationMs": 1234
        }
      ],
      "startedAt": "2024-01-01T00:00:00Z",
      "completedAt": "2024-01-01T00:01:00Z",
      "durationMs": 60000
    }
  ],
  "summary": {
    "total": 2,
    "passed": 2,
    "failed": 0,
    "errored": 0
  },
  "createdAt": "2024-01-01T00:00:00Z",
  "completedAt": "2024-01-01T00:01:00Z",
  "durationMs": 62000
}

API Endpoints

The orchestrator provides:

POST /api/orchestrator - Submit a new run request
GET /api/orchestrator/{runId}/status - Get run status
GET /api/orchestrator/{runId} - Get run results
GET /api/orchestrator/docs - Interactive API documentation
GET /api/orchestrator/spec.json - OpenAPI specification

Run Modes

Remote Mode

Execute runners on remote endpoints:

{
  "runners": [
    {
      "name": "cookieBannerTest",
      "region": "us-east-1",
      "input": { "url": "https://example.com" }
    }
  ],
  "mode": "remote"
}

Requirements:

Each runner must specify a region
PLAYWRIGHT_RUNNERS must be configured
Runner endpoints must be accessible

Local Mode

Execute runners locally (for testing):

{
  "runners": [
    {
      "name": "cookieBannerTest",
      "input": { "url": "https://example.com" }
    }
  ],
  "mode": "local"
}

Requirements:

Runners must be available locally
Each runner must specify a url in input

Concurrency Control

Control how many jobs run in parallel:

{
  "runners": [ /* ... */ ],
  "mode": "remote",
  "concurrency": 3
}

Default: Unlimited (all jobs run in parallel)
Limited: Set concurrency to limit parallel jobs
Sequential: Set concurrency: 1 to run one at a time

Timeouts

Set timeout for entire run:

{
  "runners": [ /* ... */ ],
  "mode": "remote",
  "timeout": 60000
}

Timeout applies to entire run (all jobs)
Individual job timeouts are handled by runner endpoints
Run status becomes "timed_out" if timeout exceeded

Best Practices

1. Use Consistent URLs

Group runners by URL to minimize job count:

{
  "runners": [
    { "name": "test1", "region": "us-east-1", "input": { "url": "https://example.com" } },
    { "name": "test2", "region": "us-east-1", "input": { "url": "https://example.com" } }
  ]
}

2. Set Appropriate Timeouts

{
  "timeout": 300000  // 5 minutes for slow tests
}

3. Monitor Run Status

Poll status endpoint for long-running runs:

async function waitForCompletion(runId: string) {
  while (true) {
    const status = await getRunStatus(runId);
    if (status.state === "completed" || status.state === "failed") {
      return status;
    }
    await sleep(5000); // Poll every 5 seconds
  }
}

4. Handle Failures Gracefully

const summary = await getRunResults(runId);

if (summary.state === "failed") {
  const failedJobs = summary.jobs.filter(job => job.state === "failed");
  console.error(`Failed jobs: ${failedJobs.length}`);
  
  for (const job of failedJobs) {
    console.error(`Region ${job.region}: ${job.error}`);
  }
}

Orchestration

Overview

Architecture

Setup

Installation

Create Orchestrator Server

Configure Runner Endpoints

Usage

Submit a Run Request

Check Run Status

Get Run Results

API Endpoints

Run Modes

Remote Mode

Local Mode

Concurrency Control

Timeouts

Best Practices

1. Use Consistent URLs

2. Set Appropriate Timeouts

3. Monitor Run Status

4. Handle Failures Gracefully

See Also

Orchestration

Overview

Architecture

Setup

Installation

Create Orchestrator Server

Configure Runner Endpoints

Usage

Submit a Run Request

Check Run Status

Get Run Results

API Endpoints

Run Modes

Remote Mode

Local Mode

Concurrency Control

Timeouts

Best Practices

1. Use Consistent URLs

2. Set Appropriate Timeouts

3. Monitor Run Status

4. Handle Failures Gracefully

See Also