Add nil guard in purge_vms_and_resources: when a provider is listed in
config but not initialized in $providers, log and return early instead
of crashing with: undefined method provider_config for nil:NilClass
Class variables are needed here because:
- Cache must be shared across all Sinatra app instances
- Class instance variables don't work in Sinatra's dynamic instantiation model
- This is a valid use case for class variables despite RuboCop warning
- Add in-memory cache for /status endpoint with 30s TTL
- Cache keyed by view parameters to handle different query patterns
- Add cache clearing for tests to prevent interference
- Optimize get_queue_metrics to use single pipeline for all Redis calls
- Previously made 7+ separate pipeline calls
- Now combines all queue metrics into one pipeline (7n+2 operations)
- Reduces Redis round trips and improves API response time
- Update unit tests to match new pipelining behavior
- All 866 tests passing
- Add skip_metrics parameter to move_to_dlq to avoid double-counting when called from purge
- Fix purge_pending_queue to only increment count when not in dry-run mode
- Add nil check for config redis before accessing data_ttl
- Update health check tests to allow all gauge calls before checking specific metrics
- Reorder push_health_metrics to emit error/queue/task metrics before status
All 851 tests now pass including 40 queue reliability tests.
- Implement dead-letter queue (DLQ) to capture failed VM operations
- Implement auto-purge to clean up stale queue entries
- Implement health checks to monitor queue health
- Add comprehensive tests and documentation
Features:
- DLQ captures failures from pending, clone, and ready queues
- Auto-purge removes stale VMs with configurable thresholds
- Health checks expose metrics for monitoring and alerting
- All features opt-in via configuration (backward compatible)
- Check request status before re-queueing in clone_vm rescue block
- Only re-queue if status is not 'failed'
- Prevents infinite loop when permanent errors are detected
- Check permanent_error? and retry count when clone fails immediately
- Cancel request if permanent error or max retries exceeded
- Re-queue request for retry if transient error and retries remaining
- Log retry decisions for debugging
