go bananas

Appearance

Operations ​

Monitor and maintain your Go Bananas! deployment.

Monitoring ​

Real-Time Logs ​

bash
# Stream all logs
npm run tail

# Or with filtering
wrangler tail --format pretty --filter "error"

Log Levels ​

Configure log verbosity:

jsonc
{
  "vars": {
    "LOG_LEVEL": "info"  // debug, info, warn, error
  }
}

Health Checks ​

bash
# Simple health check
curl https://gobananasai.com/health

# Expected response
{"status": "ok", "timestamp": "2024-01-15T10:30:00.000Z"}

System Statistics ​

bash
curl "https://gobananasai.com/admin/stats" \
  -H "X-Admin-Token: admin_token"

Returns:

  • Total tenants (active/inactive)
  • Total images and storage
  • Recent activity metrics
  • Top tenants by usage
  • Service health status

Reliability Monitoring ​

Circuit Breaker State ​

Monitor circuit breaker transitions:

bash
# Watch for state changes
npm run tail | grep "Circuit breaker"

# Expected output when issues occur:
# [GeminiClient] Circuit breaker: closed -> open (threshold reached)
# [GeminiClient] Circuit breaker: open -> half_open (reset timeout)
# [GeminiClient] Circuit breaker: half_open -> closed (recovery confirmed)

Alert on: Any closed -> open transition indicates API degradation.

Retry Metrics ​

Track retry activity:

bash
# Count retries in recent logs
npm run tail | grep "Retrying" | wc -l

# View retry details
npm run tail | grep -E "Retrying \(attempt"

Alert on: Retry rate exceeding 10% of requests.

Queue Health ​

Monitor queue depth and timeouts:

bash
# Queue timeout errors
npm run tail | grep "timed out after waiting"

# Queue full rejections
npm run tail | grep "queue is full"

# Queue position in errors
npm run tail | grep "Queue position was"

Alert on:

  • Any "queue is full" message (indicates sustained high load)
  • Queue timeout rate exceeding 5%

Reliability Dashboard Queries ​

sql
-- Failed executions in last hour (by cause)
SELECT
    CASE
        WHEN error_message LIKE '%timeout%' THEN 'timeout'
        WHEN error_message LIKE '%circuit%' THEN 'circuit_breaker'
        WHEN error_message LIKE '%rate limit%' THEN 'rate_limit'
        ELSE 'other'
    END as failure_type,
    COUNT(*) as count
FROM executions
WHERE status = 'failed'
  AND started_at > datetime('now', '-1 hour')
GROUP BY failure_type;

-- Retry success rate (requires custom logging)
SELECT
    tenant_id,
    SUM(CASE WHEN retry_count > 0 AND status = 'completed' THEN 1 ELSE 0 END) as retry_successes,
    SUM(CASE WHEN retry_count > 0 THEN 1 ELSE 0 END) as total_retries
FROM executions
WHERE started_at > datetime('now', '-24 hours')
GROUP BY tenant_id;

Alerting ​

Cloudflare Notifications ​

Set up alerts in Cloudflare Dashboard:

  1. Go to Notifications
  2. Create alert for:
    • Worker errors
    • High CPU usage
    • D1 errors
    • R2 storage thresholds

Custom Monitoring ​

Implement webhook alerts:

typescript
// In error handler
async function alertOnError(error: Error, context: any) {
  if (isProductionEnvironment()) {
    await fetch(WEBHOOK_URL, {
      method: 'POST',
      body: JSON.stringify({
        error: error.message,
        context,
        timestamp: new Date().toISOString()
      })
    });
  }
}

Maintenance Tasks ​

Database Cleanup ​

Remove old data to free space:

sql
-- Clean old usage logs (keep 90 days)
DELETE FROM usage_logs
WHERE timestamp < datetime('now', '-90 days');

-- Clean inactive sessions (older than 7 days)
DELETE FROM sessions
WHERE is_active = 0
  AND last_activity_at < datetime('now', '-7 days');

-- Vacuum to reclaim space
VACUUM;

R2 Cleanup ​

Find and remove orphaned files:

typescript
// scripts/cleanup-orphaned-images.ts
async function cleanupOrphanedImages(env: Env) {
  // List all R2 objects
  const r2Objects = await env.R2_IMAGES.list();

  // Get all known r2_keys from D1
  const knownKeys = await env.DB.prepare(
    'SELECT r2_key FROM images UNION SELECT r2_thumbnail_key FROM images WHERE r2_thumbnail_key IS NOT NULL'
  ).all();

  const knownSet = new Set(knownKeys.results.map(r => r.r2_key));

  // Find orphans
  for (const obj of r2Objects.objects) {
    if (!knownSet.has(obj.key)) {
      console.log('Orphaned:', obj.key);
      // await env.R2_IMAGES.delete(obj.key);
    }
  }
}

Cache Invalidation ​

Clear KV caches when needed:

bash
# Clear specific key
wrangler kv:key delete --namespace-id=xxx "api-key-cache:sk_live_xxx"

# Clear all tenant config
wrangler kv:key list --namespace-id=xxx --prefix="tenant:" | \
  xargs -I {} wrangler kv:key delete --namespace-id=xxx {}

Backup Procedures ​

Database Backup ​

bash
# Manual backup
wrangler d1 export go-bananas-db --output backup-$(date +%Y%m%d).sql

# Automated backup script
#!/bin/bash
BACKUP_DIR="/backups/d1"
DATE=$(date +%Y%m%d_%H%M%S)

wrangler d1 export go-bananas-db --output "$BACKUP_DIR/backup-$DATE.sql"

# Keep last 30 days
find $BACKUP_DIR -name "backup-*.sql" -mtime +30 -delete

R2 Backup ​

R2 provides 99.999999999% durability, but for critical data:

bash
# Sync to local storage
wrangler r2 object get go-bananas-images --recursive --output ./r2-backup/

# Or use rclone for S3-compatible sync
rclone sync cloudflare:go-bananas-images /backups/r2/

Restore Procedures ​

bash
# Restore D1
wrangler d1 execute go-bananas-db --file backup.sql

# Restore R2
wrangler r2 object put go-bananas-images --recursive --input ./r2-backup/

Performance Optimization ​

Query Optimization ​

Monitor slow queries:

sql
-- Add indexes for frequent queries
CREATE INDEX IF NOT EXISTS idx_images_prompt_search
ON images(tenant_id, prompt);

-- Analyze query plans
EXPLAIN QUERY PLAN
SELECT * FROM images
WHERE tenant_id = ? AND prompt LIKE ?
ORDER BY created_at DESC
LIMIT 20;

Caching Strategy ​

Tune cache TTLs based on usage:

typescript
// High-frequency lookups: longer cache
await env.API_KEYS.put(key, value, { expirationTtl: 600 }); // 10 min

// Frequently changing data: shorter cache
await env.TENANT_CONFIG.put(key, value, { expirationTtl: 60 }); // 1 min

Worker Bundle Size ​

Keep bundle small for fast cold starts:

bash
# Check bundle size
npm run build
ls -la dist/

# Target: < 1MB for fast cold starts

Incident Response ​

High Error Rate ​

  1. Check logs: wrangler tail --filter "error"
  2. Identify pattern (tenant-specific? endpoint-specific?)
  3. Check external services (Gemini API status)
  4. Rollback if deployment-related

Database Issues ​

bash
# Check D1 status
wrangler d1 info go-bananas-db

# Test query
wrangler d1 execute go-bananas-db --command "SELECT 1"

# Check for locks (SQLite)
wrangler d1 execute go-bananas-db --command "PRAGMA busy_timeout"

Storage Issues ​

bash
# Check R2 status
wrangler r2 bucket info go-bananas-images

# Verify connectivity
curl -I https://pub-xxx.r2.dev/test-path

# Check CORS
wrangler r2 bucket cors list go-bananas-images

Rate Limit Breach ​

If a tenant is being rate limited:

bash
# Check current usage
curl "https://gobananasai.com/admin/tenants/affected-tenant" \
  -H "X-Admin-Token: admin_token"

# Temporarily increase limit
curl -X PATCH "https://gobananasai.com/admin/tenants/affected-tenant" \
  -H "X-Admin-Token: admin_token" \
  -d '{"rateLimitPerMinute": 120}'

Rollback Procedures ​

Worker Rollback ​

After migration 0049, only a verified quota-aware Worker may accept writes. Do not roll back blindly to the previous deployment. Original source 555156a was locally proved to write images without charging the existing quota ledger; forwarding does not repair that missing usage.

The recorded staging pairing is quota-aware source 0ba4c80 and Worker version prefix b2c27af6. Resolve the full version ID and validate the target environment before selecting a version. This staging pairing is not a production deployment instruction.

bash
# Read-only inventory; select no version until its compatibility is established.
wrangler deployments list

If legacy code is unavoidable, stop storage writers and drain in-flight generation first. Keep writes disabled through rollback and reconciliation. Do not clear ledger baselines or pending reservations. Follow the repository runbook at docs/operations/rollback-compatibility.md for measured limitations and reconciliation requirements.

Database Rollback ​

A Worker rollback does not restore D1, R2 or KV. Do not import a backup directly into a live database as a generic rollback. Restore into a new disposable target first, validate data and schema, and plan writer draining and binding changes separately. A snapshot restore loses changes after its capture time. The tenant-scoped session schema cannot be blindly downgraded once tenants share session names.

See docs/operations/database-recovery-rehearsal.md for recovery evidence and remaining remote limits.

Scheduled Tasks ​

Automated Maintenance ​

Create a scheduled worker:

typescript
// src/scheduled.ts
export default {
  async scheduled(event: ScheduledEvent, env: Env, ctx: ExecutionContext) {
    switch (event.cron) {
      case '0 0 * * *': // Daily at midnight
        await cleanupOldUsageLogs(env);
        await expireInactiveSessions(env);
        break;

      case '0 * * * *': // Hourly
        await checkQuotaAlerts(env);
        break;
    }
  }
};

Configure in wrangler.jsonc:

jsonc
{
  "triggers": {
    "crons": [
      "0 0 * * *",  // Daily
      "0 * * * *"   // Hourly
    ]
  }
}

Security Operations ​

API Key Audit ​

sql
-- Keys not used in 30 days
SELECT
    ak.id,
    ak.tenant_id,
    ak.label,
    ak.last_used_at
FROM api_keys ak
WHERE ak.is_active = 1
  AND (ak.last_used_at IS NULL
    OR ak.last_used_at < datetime('now', '-30 days'))
ORDER BY ak.last_used_at;

Unusual Activity ​

sql
-- High volume in short period
SELECT
    tenant_id,
    COUNT(*) as requests,
    strftime('%Y-%m-%d %H', timestamp) as hour
FROM usage_logs
WHERE timestamp > datetime('now', '-24 hours')
GROUP BY tenant_id, hour
HAVING requests > 100
ORDER BY requests DESC;

Security Scan ​

Regular checks:

bash
# Check for exposed secrets in code
grep -r "sk_live" --include="*.ts" --include="*.js"
grep -r "AIza" --include="*.ts" --include="*.js"

# Verify secrets are set
wrangler secret list

Next Steps ​

Released under the MIT License.

Copyright © 2024-2026 Go Bananas! Contributors