Reliability & Resilience

Production-grade reliability features for handling failures gracefully under load.

Overview

Go Bananas! includes built-in reliability patterns to handle:

• Transient failures - Network glitches, temporary API unavailability
• Rate limiting - Gemini API quota limits (429 errors)
• Cascade failures - Preventing one failure from overwhelming the system
• Queue management - Fair timeout allocation under concurrent load

Retry Logic

Automatic retry with exponential backoff for transient failures.

How It Works

Request fails → Is it retryable? → Wait with backoff → Retry
                      ↓ No
                   Throw error

Configuration

import { withRetry, GEMINI_RETRY_CONFIG } from '../utils/retry';

// Default configuration
const GEMINI_RETRY_CONFIG = {
  maxRetries: 3,           // 3 retry attempts
  initialDelayMs: 1000,    // Start with 1s delay
  maxDelayMs: 10000,       // Cap at 10s delay
  retryablePatterns: [     // Error messages to retry
    'rate limit', '429', '503', '502',
    'service unavailable', 'timeout',
    'ECONNRESET', 'ETIMEDOUT', 'network', 'fetch failed'
  ]
};

// Usage
const result = await withRetry(
  () => geminiClient.generateImage(params),
  GEMINI_RETRY_CONFIG
);

Backoff Schedule

Attempt	Base Delay	With Jitter (0-1s)	Max Delay
1	1s	1-2s	10s
2	2s	2-3s	10s
3	4s	4-5s	10s

Retryable vs Non-Retryable Errors

Retried automatically:

• Rate limit exceeded (429)
• Service unavailable (503, 502)
• Network errors (ECONNRESET, ETIMEDOUT)
• Timeout errors
• Temporary unavailability

Not retried (fail immediately):

• Invalid prompt format (400)
• Authentication errors (401)
• Invalid API key
• Quota exceeded (different from rate limit)

Monitoring Retries

Check logs for retry activity:

npm run tail | grep "Retrying"

Log format:

[GeminiClient] Retrying (attempt 2): rate limit exceeded

Circuit Breaker

Prevents cascade failures when the Gemini API is degraded.

How It Works

┌─────────┐     Failures ≥ threshold     ┌─────────┐
│ CLOSED  │ ─────────────────────────────▶│  OPEN   │
│ (normal)│                               │(failing)│
└─────────┘                               └─────────┘
     ▲                                         │
     │    Success threshold met                │ Reset timeout
     │                                         ▼
     └──────────────────────────────────┌───────────┐
                                        │ HALF-OPEN │
                                        │ (testing) │
                                        └───────────┘

States

State	Behavior	Transitions
Closed	Normal operation, all requests pass through	Opens after 5 consecutive failures
Open	Fails fast immediately without calling API	Transitions to half-open after 30s
Half-Open	Allows limited requests to test recovery	Closes after 2 successes, reopens on failure

Configuration

import { CircuitBreaker } from '../utils/circuit-breaker';

const breaker = new CircuitBreaker({
  failureThreshold: 5,     // Open after 5 failures
  resetTimeoutMs: 30000,   // Try recovery after 30s
  successThreshold: 2,     // Close after 2 successes in half-open
  onStateChange: (from, to, reason) => {
    console.log(`Circuit breaker: ${from} → ${to} (${reason})`);
  }
});

Monitoring

Check circuit breaker state:

const stats = breaker.getStats();
// {
//   state: 'closed',
//   failures: 0,
//   lastFailureTime: 0,
//   timeUntilReset: undefined
// }

When open, timeUntilReset shows milliseconds until half-open transition.

Error Handling

import { CircuitOpenError } from '../utils/circuit-breaker';

try {
  await breaker.execute(() => apiCall());
} catch (error) {
  if (error instanceof CircuitOpenError) {
    // Service temporarily unavailable
    // Show user-friendly message, suggest retry later
  }
}

Queue Management

Fair timeout allocation for concurrent image generation requests.

Split Timeout Budgets

// src/mcp/agent.ts
private readonly QUEUE_WAIT_TIMEOUT_MS = 90000;   // 90s max wait
private readonly EXECUTION_TIMEOUT_MS = 120000;   // 120s for execution

Why Split Timeouts?

Problem with combined timeout:

Combined 120s timeout
├── Position 1: 30s execution → OK (90s remaining)
├── Position 2: 30s wait + 30s execution → OK (60s remaining)
├── Position 3: 60s wait + 30s execution → OK (30s remaining)
├── Position 4: 90s wait + 30s execution → TIMEOUT (needed 120s)
└── Position 5: 120s wait → TIMEOUT (never starts)

With split timeouts:

90s wait + 120s execution = 210s total budget
├── Position 1: 0s wait + 30s execution → OK
├── Position 2: 30s wait + 30s execution → OK
├── Position 3: 60s wait + 30s execution → OK
├── Position 4: 90s wait + 30s execution → OK (still within 90s wait limit)
└── Position 5: Depends on previous task times

Queue Depth Limits

private readonly MAX_QUEUE_DEPTH = 5;  // Max concurrent pending requests

When queue is full, new requests receive immediate error:

Generation queue is full (5 pending requests)

Position Tracking

Timeout errors include queue position for debugging:

Request timed out after waiting 95s in queue. Queue position was 3.

Error Handling Best Practices

In Tool Handlers

import { withRetry, GEMINI_RETRY_CONFIG } from '../utils/retry';
import { isAbortError } from './agent-tools';

try {
  const result = await withRetry(
    () => geminiClient.generateImage(params),
    GEMINI_RETRY_CONFIG
  );
  return result;
} catch (error) {
  if (isAbortError(error)) {
    // Client disconnected - clean up gracefully
    await logger.fail('Operation cancelled - client disconnected');
    return;
  }

  // Re-throw for upstream handling
  throw error;
}

Error Logging

Errors are logged with context instead of swallowed:

// In queue chain
this.generationQueue = currentTask.catch((error) => {
  this.logger.error(
    `[${tenantId}] [Queue] Task failed:`,
    error instanceof Error ? error.message : 'Unknown error'
  );
});

Monitoring Checklist

Metric	Where to Check	Alert Threshold
Retry rate	Logs: "Retrying"	>10% of requests
Circuit breaker opens	Logs: "Circuit breaker: closed → open"	Any occurrence
Queue timeouts	Logs: "timed out after waiting"	>5% of requests
Queue full rejections	Logs: "queue is full"	Any occurrence

Log Filtering

# Retry activity
npm run tail | grep -E "Retrying|retry"

# Circuit breaker state changes
npm run tail | grep "Circuit breaker"

# Queue issues
npm run tail | grep -E "queue|Queue|timeout"

# All reliability events
npm run tail | grep -E "Retrying|Circuit|queue|timeout"

Testing Reliability

Unit Tests

# Run reliability tests
npm test -- tests/utils/retry.test.ts
npm test -- tests/utils/circuit-breaker.test.ts
npm test -- tests/mcp/queue-timeout.test.ts

Manual Testing

1. Test retry logic: Temporarily set maxRetries: 0 and observe immediate failures
2. Test circuit breaker: Lower failureThreshold to 2 and trigger failures
3. Test queue limits: Send 6+ concurrent requests

Configuration Reference

Setting	Default	Description
maxRetries	3	Number of retry attempts
initialDelayMs	1000	First retry delay
maxDelayMs	10000	Maximum retry delay
failureThreshold	5	Failures before circuit opens
resetTimeoutMs	30000	Time before recovery attempt
successThreshold	2	Successes to close circuit
QUEUE_WAIT_TIMEOUT_MS	90000	Max queue wait time
EXECUTION_TIMEOUT_MS	120000	Max execution time
MAX_QUEUE_DEPTH	5	Max pending requests

Next Steps

• Operations → - Monitoring and maintenance
• Troubleshooting → - Common issues
• Scaling → - Handling higher load