- Add in-memory cache for /status endpoint with 30s TTL
- Cache keyed by view parameters to handle different query patterns
- Add cache clearing for tests to prevent interference
- Optimize get_queue_metrics to use single pipeline for all Redis calls
- Previously made 7+ separate pipeline calls
- Now combines all queue metrics into one pipeline (7n+2 operations)
- Reduces Redis round trips and improves API response time
- Update unit tests to match new pipelining behavior
- All 866 tests passing
- Add skip_metrics parameter to move_to_dlq to avoid double-counting when called from purge
- Fix purge_pending_queue to only increment count when not in dry-run mode
- Add nil check for config redis before accessing data_ttl
- Update health check tests to allow all gauge calls before checking specific metrics
- Reorder push_health_metrics to emit error/queue/task metrics before status
All 851 tests now pass including 40 queue reliability tests.
- Implement dead-letter queue (DLQ) to capture failed VM operations
- Implement auto-purge to clean up stale queue entries
- Implement health checks to monitor queue health
- Add comprehensive tests and documentation
Features:
- DLQ captures failures from pending, clone, and ready queues
- Auto-purge removes stale VMs with configurable thresholds
- Health checks expose metrics for monitoring and alerting
- All features opt-in via configuration (backward compatible)
the vm_ready methods should be implemented per provider and checking in the api
creates issues since provider code is not available. Removing this for the V2 api
Before this change if an extra config file had new keys they would get
merged to the main config but if it contained an existing key, like
'providers' it would overwrite the original config.
Adding a library 'deep_merge' to do a more natural merge, where existing keys
get sub-elements added together and arrays are combined like for the
pool configuration.
Adding spec tests around EXTRA_CONFIG as they were missing, by adding
and testing two new extra_config.yaml fixture files
This method should be called only once, when the VM is moved to a running state
which is when the data for the user (if using tokens) is available. In vmpooler
base provider it is a noop method, but the various providers can implement it
to tag or label a running VM with the name of the user who checked it out
vmpooler has the vsphere provider taken out, moving some vsphere related
methods to the provider:
1) pool_folders
2) get_base_folders
At the same time renaming some configuration and code items
to remove harmful terminology.
purge_unconfigured_folders DEPRECATED, use purge_unconfigured_resources
folder_whitelist DEPRECATED, use resources_allowlist
the above configuration items are still supported but will be removed in
the next major version.
base class method purge_unconfigured_folders was renamed to purge_unconfigured_resources
and requires the equivalent change in the provider classes
abc
This implements a delete method for pooltemplate and poolsize. The API
removes the override from Redis and then adds an entry in Redis that
causes the pool manager to wake up and process the removal of the
override.
To facilitate this, a new variable has been created in lib/vmpooler.rb
to hold a copy of the original / pre-override config. This supplemental
copy of the pools is then indexed for use as a reference.
When pool manager wakes up to process an override removal, it looks up
the pre-override value from the config via the new variables mentioned
above.
Just as with entering overrides, no restart is needed. Template and pool
size changes are logged so that anyone watching or reviewing the logs
can see what happened when. The new API endpoints also return values for
both the pre-revert and post-revert value.
Prior to this, the encryption settings for LDAP auth were hard coded to
start_tls on port 389 with TLSv1. These are still the defaults, as
insecure as they are, so as to not break existing users. This change
facilitates replacing the defaults so that simple_tls over port 636 via
TLS1.2 can be used.
This adds an "operation" label to the user metrics and moves incrementing from the manager to api, so that the user metrics show when resources are allocated, as well as destroyed. Previously, user metrics were only updated upon destroying a resource.
I think its better suited to increment the metric as part of the api instead of the pool_manger, because it's expected to do so when a user successfully checks out or deletes a VM, but can be problematic when doing so in the provider since it can clone VMs before actually being checked out by a user.
It appears we renamed `/ondemand/` to `/ondemandvm/` at some point and,
as a result, have not been stripping hostnames from that endpoint's
metrics. This has caused issues with metrics collection due a very high
cardinality.
