Break down the usage stats into smaller groups so as to manage the
number of stat lines collected for Prometheus.
This may need some further revision to filter out Litmus stats, or
otherwise collect litmus usage information.
The redis pooler connection metric used "metric_prefix" which is
misleading, so split this into connpool_type and connpool_provider.
Also remove some earlier jruby compatibility code to reduce
rebase conflicts when this is rebased on top of Matt's changes.
This is a re-architect of the vmpooler initialisation code to:
1. Allow an API service for both manager and the api
2. Add the Prometheus endpoints to the web service.
Needed to change the way the Rack Service is started as instantiating
using ".New" leads to a failure to initialise the http Stats
collection.
3. Selectively load the pooler api and/or Prometheus endpoints.
4. Rework API Spec tests for revised API loading. Needed to tidy up the
initialisation and perform a reset! after each test to avoid "leaks"
and dependencies between the tests.
* (POOLER-174) Reduce duplicate of on demand code introduced in POOLER-158
refactored every parsing of request of type 'pool_alias:pool:count' into a
utility class, that is used by pool_manager and the api v1 class
* add some metrics to the od request generation
* fix rubocop offenses, we are now friends
This change adds a capability to vmpooler to provision instances on
demand. Without this change vmpooler only supports retrieving machines
from pre-provisioned pools.
Additionally, this change refactors redis interactions to reduce round
trips to redis. Specifically, multi and pipelined redis commands are
added where possible to reduce the number of times we are calling redis.
To support the redis refactor the redis interaction has changed to
leveraging a connection pool. In addition to offering multiple
connections for pool manager to use, the redis interactions in pool
manager are now thread safe.
Ready TTL is now a global parameter that can be set as a default for all
pools. A default of 0 has been removed, because this is an unreasonable
default behavior, which would leave a provisioned instance in the pool
indefinitely.
Pool empty messages have been removed when the pool size is set to 0.
Without this change, when a pool was set to a size of 0 the API and pool
manager would both show that a pool is empty.
This commit fixes the purge_unconfigured_folders feature to ensure that it can successfully identify folders and instances that are no longer used. Without this change the feature does not work as advertised.
This commit adds detection for redis connection failures to pool_manager. When a connection fails the error will be raised to executeforcing the connection to be re-established. Without this change, when a redis connection fails, it generates a redis connection error, which is swallowed by a rescue for StandardError, preventing the manager application component from recovering in the case of a redis connection failure.
This commit adds a capability to vmpooler to reset a pool, deleting its ready and pending instances and replacing them with fresh ones. Without this change vmpooler does not offer a mechanism to reset a pool without also changing its template.
Prior to this commit the pooler had no awareness of the complete set of
hostnames that are currently in use. This meant that it was possible to
allocate the same hostname twice, which would result in the original
host with that hostname becoming unreachable.
This commit adds a check for the existence of the
`vmpooler__vm__<hostname>` key before attempting to clone the vm.
This should prevent duplicate hostnames.
If the hostname is already taken, `_clone_vm` will retry with a new
random hostname multiple times before raising an exception.
This allows the user to change the cluster in which the targeted pool
will clone to. Upon configuration change, the thread will wake up and
execute the change within 1 second.
This commit updates vm usage stats collection to replace all instances of '.' characters within node strings. Without this change the node string containing a '.' character causes the metric to be interpreted as containing another node.
This commit updates vmpooler to ship VM usage stats when a VM is destroyed. The stats are gathered for jobs based on user and pool name. If a jenkins build URL is present then this is broken down by user, instance, value stream, branch and project. Additionally, if present then the RMM_COMPONENT_TO_TEST_NAME will be listed after project. Without this change we do not collect stats on per VM usage and its correlation to users and pools.
This commit fixes checking of a VM that has already been identified as ready. Without this change a ready VM that has failed will be identified as having failed, but will not successfully be removed from the ready queue. Additionally, the default vm_checktime value has been reduced from 15 to 1 to ensure that ready VMs are checked within one minute of the time they have reached the ready state by default.
Lastly, the docker-compose files are updated to specify that the redis
instance used is a local redis instance.
This commit updates delta disk creation to reduce the likelihood of this being run more than once for any given template. Without this change an error can be generated with vsphere 6.5 or later when a template is updated, and then the update is reverted. The error prevents the image from being used because the template is never marked as prepared. To address this any failure is now logged, and the template is marked as prepared regardless of whether this was successful, or not, which allows the image to be used despite the error. This failure mode is more graceful and allows the pool to continue to function.
This commit refactorss the check_pool method in pool_manager.
Specifically, each commented section describing a stage of check_pool is
broken out into a separate method and check_pool is simplified by
calling these methods. Without this change it is difficult to follow the
intent for or make changes to check_pool.
Additionally, a docker-compose file is added to make it simple to launch
an all-in-one vmpooler instance along with a separate redis server with
docker.
This commit updates destroy_vm to remove references to its mutex tracking object when destroyed. Without this change a VM that is destroyed will leave its mutex tracking object forever causing the pool manager memory footprint to increase.
Prior to this commit the docs and examples used 'company.com'
(a real domain). This commit changes those occurrences to
'example.com', which is a IANA-managed reserved domain.
This commit enables optional purging for vms and folders when they are
not configured in the vmpooler provided configuration. Base folders are
determined from folders specified in the pool configuration. Then,
anything not configured in that folder for that provider and is not a
whitelisted folder title will be destroyed. Without this change vmpooler
will leave unconfigured vms and folders behind and any vms will be left
running forever without manual intervention. Additionally, any
associated redis data will never be expired.
* Minimize duplicate checking of VMs
This commit updates check_pool pending, running and ready methods to greatly reduce instances in which the VM object is retrieved. Without this change get_vm is run for each of these check_pool steps even though the VM is already validated as being in inventory being running the check. This is left for checking running VMs when the VM is no longer ready. Without this change there is an unnecessarily large volume of VM object checks.
* Make hostname mismatch checking optional
This commit makes hostname mismatch checking optional on a pool and global config level. The default behavior of checking this is preserved. When disabled _check_ready_vm will not run get_vm, which allows for ready VMs to be checked without retrieving an object from vsphere and waiting for a free provider connection. Without this change it is not possible to disable VM object retrieval in _check_vm_ready.
* Check if a hostname return is empty string
This commit checks whether a hostname returned is an empty string.
Without this change a VM that returns a hostname with a empty string
will report as having a hostname mismatch, which may happen before all
VM data is updated.
* Only check hostname for mismatch after time since ready has past
Configure hostname checking so it is not checked until a VM has been
ready for greater than one minute. Without this change hostname checking
will often fail because the configured value has not propogated to the
VM on some platforms before this check is run.
* (POOLER-34) Ship clone request to ready time to metrics
Before, we were already capturing this metric but we failed to ship
it anywhere. This ships the appropriate metric as `time_to_ready_state`
Dashboards can be found in grafana.
* Add spec test to ensure metric is being shipped properly on
move_pending_vm_to_ready call.
This commit updates how template delta disk creation is evaluated. Without this change template deltas are created for every template on each applicatoin startup. This change updates this behavior to instead run template delta disk creation only once per template configured for a pool. Without this change it is possible to get a template to a state where the XML depth is too great to be read with default settings and the template requires a new clone to resolve.
This commit updates check_pool inventory check to prevent multiple instances from running at once. Without this change the inventory check may run in multiple threads simultaneously.
This commit updates pool_manager to expire a redis VM key when a clone fails. Without this change VMs that fail to clone have their metadata left forever.
This commit implements a vm_mutex hash to allow synchronizing VM operations that should only happen once across threads. Without this change pool_manager will try to evaluate or destroy a VM multiple times, which results in an error being thrown by one of the destroy attempts as only one can succeed and a duplication of resources unnecessarily when there are no errors.
This commit adds a configuration endpoint to the vmpooler API. Pool
size, and pool template, can be adjusted for pools that are configured
at vmpooler application start time. Pool template changes trigger a pool
refresh, and the new template has delta disks created automatically by
vmpooler.
Additionally, the capability to create template delta disks is added to
the vsphere provider, and this is implemented to ensure that templates
have delta disks created at application start time.
The mechanism used to find template VM objects is simplified to make the flow of logic easier to understand. As an additional benefit, performance of this lookup is improved by using FindByInventoryPath.
A table of contents is added to API.md to ease navigation. Without this change API.md has no table of contents and is difficult to navigate.
Add mutex object for managing pool configuration updates
This commit adds a mutex object for ensuring that pool configuration changes are synchronized across multiple running threads, removing the possibility of two threads attempting to update something at once, without relying on redis data. Without this change this is managed crudely by specifying in redis that a configuration update is taking place. This redis data is left so the REPOPULATE section of _check_pool can still identify when a configuration change is in progress, and prevent a pool from repopulating at that time.
Add wake up event for pool template changes
This commit adds a wake up event to detect pool template changes.
Additionally, GET /config has a template_ready section added to the
output for each pool, which makes clear when a pool is ready to populate
itself.
This commit add a redis hash where there is one key per pool, and the
stored value is the last time a VM was booted e.g. the last time
a VM went from 'pending' to 'ready'. This is also displayed in the
API as lastBoot:'2018-03-23 17:43:39 +0000'. The data can then be
used by any external system, in this case our alarming system.
This commit moves the migrate_vm logic to the vsphere provider. Without
this change migrate_vm has lots of vsphere specific logic in
pool_manager migrate_vm method.
* Fix no implicit conversion to rational from nil
Before this change if the boottime was nil, the check_ready
loop would exit on Time.now - host['boottime'] with a TypeError
in jruby. The boottime is nil when the power is Off so moving that check
earlier should catch that bug.
* set test data properly
Previously in commit 9b0e55f959 the looping period was changed from a static
number to a dynamic one depending on load, however this meant that the operation
to refill a pool was slowed down somewhat. While not a problem under normal
loads, when a pool was quickly consumed, the pool manager may not respond
quickly enough to refill the pool. This commit:
- Changes the sleep method, to us a helper sleep method that will wakeup
periodically and evaluate other wakeup events. This could be used later to
exist sleep loops when pooler is shutting down to stop blocking threads
- By default the wakeup_period is set to the minimum pool check loop time, thus
emulating the behaviour prior to commit 9b0e55f959
- Adds tests for the behaviour
Previously, if inventory failed for some reason, it would return an incomplete
set of VMs which could then cause the pool to perform off behaviours such as
fill the pool high than it should, or remove VMs which exist. Also, if the
redis cache of VMs in a pool had a VM but it did not actually exist in the
inventory it would never be removed.
This commit:
- Immediately exits the check_pool if an error occurs during inventory
collection
- Will mark a VM as completed if it exists in Redis, but does not exist in
inventory
- Adds tests for these behaviours
Sometimes this test would fail if the computer running the tests was under a
bit of load. This commit changes the expected output to be up to 1.99 seconds
instead of the previous 0.99 seconds.
Previously the check_pool would always check the pool every 5 seconds, however
with a large number of pools, this can cause resource issues inside the
providers. This commit:
- Introduces a dynamic check_pool period which increases during stability and
decreases when the pool is being change in an important way
- Surfaces the settings as global config defaults but can also be set on a per
pool basis
- Adds defaults to emulate the current behaviour
- Unit tests for the new behaviour
This commit updates find_least_used_compatible_host method to specify
the pool name when evaluating a VM for migration. Without this change VM
migration fails with a wrong number of arguments error. Pool_manager
test references are updated to reflect the change.
Refactoring the vmpooler.yaml format to support multiple providers.
The second level key under :providers: is a unique key name that
represents a provider that can be refered in the pool's parameter
called provider. The code is still backward compatible to support
the :vsphere: and :dummy: keys but in reality if you have more than
one vsphere configuration you would give them a different name. For
example :vsphere-pdx: and :vsphere-bfs: and the actual provider
class would be specified as a parameter called 'provider_class'.
See tests and examples for more information.
This commit modifies execute! to create the VM Providers on VMPooler startup
instead of check_pool creating a provider per pool. This commit also adds
legacy support for old configuration files:
- Setting the default provider for pools to be vsphere
- Copying VSphere connection settings in the configuration file from the legacy
location in the root, to under :providers/:vsphere which is new location for
all provider configuration
This commit adds a public function to access the internal variable holding the
VMPooler configuration. This is required for later commits for the execute!
function testing.
