This commit adds a capability to vmpooler to reset a pool, deleting its ready and pending instances and replacing them with fresh ones. Without this change vmpooler does not offer a mechanism to reset a pool without also changing its template.
* (POOLER-123) Implement a max TTL
Before this change, we could checkout a vm and set the lifetime to a
very high number which would esssentially keep the vm running forever.
Now implementing a config setting max_lifetime_upper_limit which enforces
a maximum lifetime in hours both for initial checkout and extending a
running vm
* (POOLER-123) Improve PUT vm endpoint error messaging
Prior to this commit the PUT vm endpoint didn't give any useful
information about why a user's request failed.
This commit updates PUT to output a more helpful set of error messages
in the `failure` key that gets returned in the JSON response.
* (POOLER-123) Update max_lifetime_upper_limit key
This commit switches the max_lifetime_upper_limit key from being a
symbol to being a string, which is what the config hash seems to contain.
* (maint) Add option to disable Redis persistence in docker-compose
This commit is just a handy little command override to the redis
container to prevent persistence.
Prior to this commit the pooler had no awareness of the complete set of
hostnames that are currently in use. This meant that it was possible to
allocate the same hostname twice, which would result in the original
host with that hostname becoming unreachable.
This commit adds a check for the existence of the
`vmpooler__vm__<hostname>` key before attempting to clone the vm.
This should prevent duplicate hostnames.
If the hostname is already taken, `_clone_vm` will retry with a new
random hostname multiple times before raising an exception.
Prior to this commit the hostname_shorten regex wouldn't match the
updated human readable hostnames because they contain dashes.
This commit updates the regex to capture dashes in the hostname, and
adds a few specs to verify that behavior.
This commit adds a shared mutex to vmpooler API so that checkout requests can be synchronized across threads. Without this change it is possible in some scenarios for vmpooler to allocate the same SUT to different API requests for a VM.
This commit updates the create_linked_clone pool option to correctly detect when linked clones have been set at a pool level. Without this change a pool setting create_linked_clone to false is not interpreted correctly, and a linked clone is created if possible.
This change adds the running host for a VM to the API data available via /vm/hostname. Without this change the running host would be logged to vmpooler log, but not available any other way. Additionally, the data will specify if a machine has been migrated. Without this change parent host data for a vmpooler machine is not available via the vmpooler API.
This commit adds a new configuration parameter to allow setting whether to create linked clones on a global, or per pool basis. Without this change vmpooler would always attempt to create linked clones. The default behavior of creating linked clones is preserved.
This allows the user to change the cluster in which the targeted pool
will clone to. Upon configuration change, the thread will wake up and
execute the change within 1 second.
This commit duplicates the vm_ready? check to the API layer to allow for API to validate that a VM is alive at checkout. Without this change API relies upon the checks in pool_manager validating pools. This change should allow for additional insight into whether a machine is in a ready state and resopnding at checkout time.
Before this change looping over many pools would query the redis backend
for each pool, leading in slow response from the backend for configurations
with many pools (60+)
Changed the requests to use redis pipelines https://redis.io/topics/pipelining
This is supported since the beginning, so will not force any redis update for
users. The pipeline method runs the queries in batches and we need to loop
over the result and reduces the number of requests to redis by N=number of
pools in the configuration.
This commit updates how a VM is checked out to ensure that there is no window where the VM could be considered discovered, and therefore destroyed. Without this change the VM is retrieved by calling 'spop' on the ready queue, and then adding it to the running queue. This change moves to selecting the VM by retrieving the last member of the set, and moving it with 'smove' from ready to running. As a result of this change vmpooler moves from retrieving the VMs from the ready state randomly, to instead retrieve the oldest VM in the queue. This change should reduce churn where it would otherwise not be required to satisfy demand.
This change updates handling of pool aliases to allow for more than a
single pool to be configured as an alias pool. Without this change if
multiple pools are configured as an alias the last one to configure it
is set as the alias for that pool.
Additionally, redis testing requirements are removed in favor of
mock_redis. Without this change a redis server is required to run
vmpooler tests.
This commit updates vmpooler to set configuration values received via environment variables to integer values when an integer value is expected. Without this change vmpooler does not support setting integer values via environment variables. Additionally, testing is added for configuring vmpooler via environment variables.
To support this testing the gem climate_control is added, which allows
for testing environment variables without those set variables leaking to
other tests.
This commit updates vm usage stats collection to replace all instances of '.' characters within node strings. Without this change the node string containing a '.' character causes the metric to be interpreted as containing another node.
This commit updates providers_spec so the test ensures array content of the providers match. Without this change the provider_spec test will fail when comparing providers if the order is not exactly the same in each array.
This commit updates vmpooler to ship VM usage stats when a VM is destroyed. The stats are gathered for jobs based on user and pool name. If a jenkins build URL is present then this is broken down by user, instance, value stream, branch and project. Additionally, if present then the RMM_COMPONENT_TO_TEST_NAME will be listed after project. Without this change we do not collect stats on per VM usage and its correlation to users and pools.
This commit fixes checking of a VM that has already been identified as ready. Without this change a ready VM that has failed will be identified as having failed, but will not successfully be removed from the ready queue. Additionally, the default vm_checktime value has been reduced from 15 to 1 to ensure that ready VMs are checked within one minute of the time they have reached the ready state by default.
Lastly, the docker-compose files are updated to specify that the redis
instance used is a local redis instance.
This commit updates fetch_single_vm to return the name of the template that was requested, instead of the name of the pool providing the VM to meet the request. Without this change, when an alias is used for fetching a VM, a different pool title may be returned containing the requested VMs than the user initially requested.
This commit updates delta disk creation to reduce the likelihood of this being run more than once for any given template. Without this change an error can be generated with vsphere 6.5 or later when a template is updated, and then the update is reverted. The error prevents the image from being used because the template is never marked as prepared. To address this any failure is now logged, and the template is marked as prepared regardless of whether this was successful, or not, which allows the image to be used despite the error. This failure mode is more graceful and allows the pool to continue to function.
This commit refactorss the check_pool method in pool_manager.
Specifically, each commented section describing a stage of check_pool is
broken out into a separate method and check_pool is simplified by
calling these methods. Without this change it is difficult to follow the
intent for or make changes to check_pool.
Additionally, a docker-compose file is added to make it simple to launch
an all-in-one vmpooler instance along with a separate redis server with
docker.
This commit updates destroy_vm to remove references to its mutex tracking object when destroyed. Without this change a VM that is destroyed will leave its mutex tracking object forever causing the pool manager memory footprint to increase.
Prior to this commit the docs and examples used 'company.com'
(a real domain). This commit changes those occurrences to
'example.com', which is a IANA-managed reserved domain.
This commit removes find_pool, which is no longer used. Additionally, tests which are listed as pending and have been resolved are removed from pending tests. Lastly, a folder check that doesn't make sense given the change in method for finding objects is removed.
This commit enables optional purging for vms and folders when they are
not configured in the vmpooler provided configuration. Base folders are
determined from folders specified in the pool configuration. Then,
anything not configured in that folder for that provider and is not a
whitelisted folder title will be destroyed. Without this change vmpooler
will leave unconfigured vms and folders behind and any vms will be left
running forever without manual intervention. Additionally, any
associated redis data will never be expired.
* Minimize duplicate checking of VMs
This commit updates check_pool pending, running and ready methods to greatly reduce instances in which the VM object is retrieved. Without this change get_vm is run for each of these check_pool steps even though the VM is already validated as being in inventory being running the check. This is left for checking running VMs when the VM is no longer ready. Without this change there is an unnecessarily large volume of VM object checks.
* Make hostname mismatch checking optional
This commit makes hostname mismatch checking optional on a pool and global config level. The default behavior of checking this is preserved. When disabled _check_ready_vm will not run get_vm, which allows for ready VMs to be checked without retrieving an object from vsphere and waiting for a free provider connection. Without this change it is not possible to disable VM object retrieval in _check_vm_ready.
* Check if a hostname return is empty string
This commit checks whether a hostname returned is an empty string.
Without this change a VM that returns a hostname with a empty string
will report as having a hostname mismatch, which may happen before all
VM data is updated.
* Only check hostname for mismatch after time since ready has past
Configure hostname checking so it is not checked until a VM has been
ready for greater than one minute. Without this change hostname checking
will often fail because the configured value has not propogated to the
VM on some platforms before this check is run.
This commit updates vmpooler to allow the API component and dashboard to
run separately from pool_manager. Without this change vmpooler does not
offer a mechanism to run only the API, or pool_manager components.
Two instances of hardcoded puma environment settings are removed. This
is still set in the init script explicitly as well as via an environment
variable in the dockerfile.
To extend the mechanism of running the API or pool_manager components to
instances running in docker an entrypoint is added in the dockerfile.
The entrypoint allows a user to specify whether to run the API or
pool_manager components when running the application. The default
behavior is preserved where both components are run.
To support these changes vmpooler.rb is updated to allow more of the
configuration to be specified via individual environment variables. It
was already possible to specify the entire config block as an
environment variable, but this is more difficult to manage and less of a
standard implementation than specifying individual parameters. Where
specified environment variable options will override a value configured
via the configuration file or environment.
The running pool configuration when starting pool_manager is loaded to
redis at pool_manager start time. This allows the API to load the
running pool configuration from redis and be able to run without
requiring the pool configuration.
Lastly, the dockerfile leveraging entrypoint will no longer start
vmpooler with the init script or write logs to a file. Instead, LOGFILE
is set to /dev/stdout and the vmpooler application is started directly.
This behavior is preferred because the log file writes to disk are an
unnecessary overhead. Without this change the docker installation will
attempt to daemonize the vmpooler application and always requires puma.
* (POOLER-81) Add time_remaining information
Before, the only time calculation displayed for a given VM was the
lifetime parameter. Added the remaining parameter which will
display time until the VM is destroyed as a float.
Additionally, start_time and end_time were added to api to return
as UTC based times (e.g. 2018-07-10 11:01:03 -0700).
* Remove abs eval from GET, rework spec tests to check each field.
This allows us to account for "flakiness" of the remaining return.
* Change datetime to RFC3339 for start_time and end_time
* Revert "(POOLER-34) Ship clone request to ready time to metrics (#277)"
This reverts commit a865e6bd2f.
* Revert "(POOLER-81) Add time_remaining information (#276)"
This reverts commit 1910cffaf7.
* (POOLER-34) Ship clone request to ready time to metrics
Before, we were already capturing this metric but we failed to ship
it anywhere. This ships the appropriate metric as `time_to_ready_state`
Dashboards can be found in grafana.
* Add spec test to ensure metric is being shipped properly on
move_pending_vm_to_ready call.
* (POOLER-81) Add time_remaining information
Before, the only time calculation displayed for a given VM was the
lifetime parameter. Added the time_remaining parameter which will
display time until the VM is destroyed in hours, minutes, seconds.
Additionally, updated the running parameter to display in a similar
fashion as time_remaining.
* Add spec testing for testing time_remaining stat
This commit updates a test reference to find_vm. This method has been renamed find_vm_folder, so this commit updates usage to reflect the new title and reduced arguments required. Without this change tests fail on master.
This commit updates usage of max_tries to ensure that the increment of the try value happens after evaluation of whether max_tries has been exceeded. Without this change max_tries doesn't work as advertised. Additionally, checking looks to see if try is greater than or equal to so the try count exceeding max_tries would also be detected.
This commit updates get_pool_vms to only return if the object is a VirtualMachine. Without this change a folder wtihin a VM target folder can be discovered and destroyed as though it were a VM.
