Error Handling and Retries

When using the Firecrawl API, you may encounter errors due to rate limiting, temporary service issues, or other conditions. This guide explains how to properly handle these errors and implement robust retry logic.

Understanding HTTP Status Codes

The Firecrawl API returns standard HTTP status codes to indicate the outcome of requests:

Success

200: Request was successful

Client Errors

400: Invalid parameters - verify your request
401: Missing or invalid API key
402: Payment required - check your account balance or plan limits
404: Resource not found
429: Rate limit exceeded - see Rate Limiting below

Server Errors

5xx: Server-side errors - these are temporary and should be retried

Rate Limiting (429 Errors)

When you exceed your plan’s rate limits, the API returns a 429 (Too Many Requests) response. Each plan has specific rate limits for different endpoints:

Check your plan’s limits in the Rate Limits documentation
When you hit a 429 error, your request is rejected and you should retry after a delay
The Retry-After header (if present) indicates how long to wait in seconds

Recommended Approach

Rather than manually implementing retries, use exponential backoff to:

Automatically retry failed requests
Gradually increase the delay between retries
Avoid overwhelming the API during recovery

Exponential Backoff Explained

Exponential backoff is a retry strategy where:

First retry waits 1 second
Second retry waits 2 seconds
Third retry waits 4 seconds
And so on, up to a maximum delay

This prevents all clients from hammering the API simultaneously during outages. Formula: delay = min(initialDelay * (backoffFactor ^ attemptNumber), maxDelay)

Implementation Examples

Node.js with Exponential Backoff

import Firecrawl from '@mendable/firecrawl-js';

const firecrawl = new Firecrawl({
  apiKey: process.env.FIRECRAWL_API_KEY
});

// Configuration for retries
const retryConfig = {
  maxRetries: 3,
  initialDelay: 1000, // milliseconds
  maxDelay: 10000,
  backoffFactor: 2
};

async function scrapeWithRetry(url, options = {}) {
  let lastError;

  for (let attempt = 1; attempt <= retryConfig.maxRetries + 1; attempt++) {
    try {
      const result = await firecrawl.scrape(url, options);
      return result;
    } catch (error) {
      lastError = error;

      // Check if error is retryable (429, 5xx)
      const status = error.statusCode || error.status;
      const isRetryable = status === 429 || (status && status >= 500);

      if (!isRetryable || attempt > retryConfig.maxRetries) {
        throw error;
      }

      // Calculate exponential backoff delay
      const delay = Math.min(
        retryConfig.initialDelay * Math.pow(retryConfig.backoffFactor, attempt - 1),
        retryConfig.maxDelay
      );

      console.log(`Attempt ${attempt} failed (${status}), retrying in ${delay}ms...`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }

  throw lastError;
}

// Usage
try {
  const result = await scrapeWithRetry('https://example.com', {
    formats: ['markdown', 'html']
  });
  console.log('Success:', result.markdown.substring(0, 200));
} catch (error) {
  console.error('Failed after retries:', error.message);
}

Python with Exponential Backoff

import time
from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="your-api-key")

# Configuration for retries
retry_config = {
    'max_retries': 3,
    'initial_delay': 1,  # seconds
    'max_delay': 10,
    'backoff_factor': 2
}

def scrape_with_retry(url, **options):
    last_error = None

    for attempt in range(retry_config['max_retries'] + 1):
        try:
            result = firecrawl.scrape(url, **options)
            return result
        except Exception as error:
            last_error = error

            # Check if error is retryable (429, 5xx)
            status = getattr(error, 'status_code', None)
            is_retryable = status == 429 or (status and status >= 500)

            if not is_retryable or attempt >= retry_config['max_retries']:
                raise

            # Calculate exponential backoff delay
            delay = min(
                retry_config['initial_delay'] * (retry_config['backoff_factor'] ** attempt),
                retry_config['max_delay']
            )

            print(f"Attempt {attempt + 1} failed ({status}), retrying in {delay}s...")
            time.sleep(delay)

    raise last_error

# Usage
try:
    result = scrape_with_retry(
        'https://example.com',
        formats=['markdown', 'html']
    )
    print(f"Success: {result['markdown'][:200]}")
except Exception as error:
    print(f"Failed after retries: {str(error)}")

Handling Specific Error Cases

429 Rate Limit Error

try {
  const result = await firecrawl.scrape(url);
} catch (error) {
  if (error.status === 429) {
    // Extract retry delay from header if available
    const retryAfter = error.headers?.['retry-after'];
    const delay = retryAfter ? parseInt(retryAfter) * 1000 : 5000;

    console.log(`Rate limited. Retrying after ${delay}ms`);
    await new Promise(resolve => setTimeout(resolve, delay));

    // Retry the request
    const result = await firecrawl.scrape(url);
  }
}

401/403 Authentication Errors

These are not retryable and indicate your API key is invalid or missing:

try {
  const result = await firecrawl.scrape(url);
} catch (error) {
  if (error.status === 401) {
    console.error('Invalid API key. Check FIRECRAWL_API_KEY environment variable.');
    process.exit(1);
  }
}

402 Payment Required

This means your plan doesn’t have enough credits. Upgrade your plan or wait for monthly credits to reset:

try {
  const result = await firecrawl.scrape(url);
} catch (error) {
  if (error.status === 402) {
    console.error('Insufficient credits. Please upgrade your plan.');
  }
}

5xx Server Errors

Server errors are temporary and should be retried with exponential backoff:

const isServerError = (status) => status >= 500;

try {
  const result = await firecrawl.scrape(url);
} catch (error) {
  if (isServerError(error.status)) {
    console.log('Temporary server error, will retry...');
    // Use exponential backoff retry strategy
  }
}

Best Practices

Use Exponential Backoff

Don’t retry immediately on failure. Use increasing delays to allow the API to recover and prevent overwhelming the service.

Set Maximum Retries

Limit retry attempts (typically 3-5) to avoid retrying forever. Eventually fail fast if the API is down.

Respect Rate Limits

Monitor your request rate and stay within your plan’s limits. Use the rate limit information to adjust your request frequency.

Log Failures

Log retry attempts and final failures for debugging. Include status codes, error messages, and retry attempts.

Only Retry Transient Errors

Don’t retry on 400, 401, 402, or 404 errors. These indicate permanent issues that retrying won’t fix.

Use Async/Await

Use async/await syntax for cleaner error handling compared to callbacks or promises.

Queue-Based Approach for Bulk Operations

For large-scale operations, use a queue to manage requests and respect rate limits:

import pQueue from 'p-queue';

const queue = new pQueue({
  concurrency: 1, // Process one request at a time
  interval: 60000, // Per minute
  intervalCap: 10  // Max 10 requests per minute (adjust to your plan)
});

async function processUrls(urls) {
  const results = [];

  for (const url of urls) {
    queue.add(async () => {
      try {
        return await scrapeWithRetry(url);
      } catch (error) {
        console.error(`Failed to scrape ${url}:`, error.message);
        return null;
      }
    });
  }

  results.push(await queue.onIdle());
}

Monitoring and Debugging

Check Your Usage

Monitor your API usage to understand your rate limit consumption:

Visit firecrawl.dev/app/dashboard
Check your plan’s rate limits in Rate Limits
Monitor successful vs failed requests

Enable Debug Logging

// Enable verbose logging to see what's happening
const firecrawl = new Firecrawl({
  apiKey: process.env.FIRECRAWL_API_KEY,
  debug: true // Not all SDKs support this - check documentation
});

Troubleshooting

Getting frequent 429 errors

Causes:

Your request rate exceeds your plan’s limit
All concurrent browser slots are in use

Solutions:

Reduce request frequency
Upgrade to a plan with higher limits
Increase delays between requests
Use queue-based rate limiting

Retries aren't working

Check:

Are you retrying non-retryable errors (400, 401, 402)?
Is your maximum delay long enough for the API to recover?
Are you catching all error types (network, timeout, HTTP)?

API keeps returning 5xx errors

What to do:

Check Firecrawl Status
Wait a few minutes before retrying
Contact support at help@firecrawl.com if errors persist

Rate limiting in production

Best practices:

Use a queue to manage throughput
Monitor API usage dashboard
Set up alerts for approaching rate limits
Test retry logic with load testing

SDK Support

All Firecrawl SDKs support error handling via try/catch blocks. However, automatic retry logic may vary:

Node.js SDK: Manual retry implementation recommended (see examples above)
Python SDK: Manual retry implementation recommended (see examples above)
Go SDK: Implement retry logic in your application
Raw HTTP/cURL: Implement retry logic in your client

Check your SDK’s documentation for any built-in retry mechanisms.

Get Started

New Features

Standard Features

Webhooks

Developer Guides

Use Cases

Contributing

Error Handling and Retries

Understanding HTTP Status Codes

Success

Client Errors

Server Errors

Rate Limiting (429 Errors)

Recommended Approach

Exponential Backoff Explained

Implementation Examples

Node.js with Exponential Backoff

Python with Exponential Backoff

Handling Specific Error Cases

429 Rate Limit Error

401/403 Authentication Errors

402 Payment Required

5xx Server Errors

Best Practices

Use Exponential Backoff

Set Maximum Retries

Respect Rate Limits

Log Failures

Only Retry Transient Errors

Use Async/Await

Queue-Based Approach for Bulk Operations

Monitoring and Debugging

Check Your Usage

Enable Debug Logging

Troubleshooting

SDK Support

Get Started

New Features

Standard Features

Webhooks

Developer Guides

Use Cases

Contributing

​Understanding HTTP Status Codes

​Success

​Client Errors

​Server Errors

​Rate Limiting (429 Errors)

​Recommended Approach

​Exponential Backoff Explained

​Implementation Examples

​Node.js with Exponential Backoff

​Python with Exponential Backoff

​Handling Specific Error Cases

​429 Rate Limit Error

​401/403 Authentication Errors

​402 Payment Required

​5xx Server Errors

​Best Practices

Use Exponential Backoff

Set Maximum Retries

Respect Rate Limits

Log Failures

Only Retry Transient Errors

Use Async/Await

​Queue-Based Approach for Bulk Operations

​Monitoring and Debugging

​Check Your Usage

​Enable Debug Logging

​Troubleshooting

​SDK Support

Understanding HTTP Status Codes

Success

Client Errors

Server Errors

Rate Limiting (429 Errors)

Recommended Approach

Exponential Backoff Explained

Implementation Examples

Node.js with Exponential Backoff

Python with Exponential Backoff

Handling Specific Error Cases

429 Rate Limit Error

401/403 Authentication Errors

402 Payment Required

5xx Server Errors

Best Practices

Queue-Based Approach for Bulk Operations

Monitoring and Debugging

Check Your Usage

Enable Debug Logging

Troubleshooting

SDK Support