Before this change if an extra config file had new keys they would get
merged to the main config but if it contained an existing key, like
'providers' it would overwrite the original config.
Adding a library 'deep_merge' to do a more natural merge, where existing keys
get sub-elements added together and arrays are combined like for the
pool configuration.
Adding spec tests around EXTRA_CONFIG as they were missing, by adding
and testing two new extra_config.yaml fixture files
This method should be called only once, when the VM is moved to a running state
which is when the data for the user (if using tokens) is available. In vmpooler
base provider it is a noop method, but the various providers can implement it
to tag or label a running VM with the name of the user who checked it out
vmpooler has the vsphere provider taken out, moving some vsphere related
methods to the provider:
1) pool_folders
2) get_base_folders
At the same time renaming some configuration and code items
to remove harmful terminology.
purge_unconfigured_folders DEPRECATED, use purge_unconfigured_resources
folder_whitelist DEPRECATED, use resources_allowlist
the above configuration items are still supported but will be removed in
the next major version.
base class method purge_unconfigured_folders was renamed to purge_unconfigured_resources
and requires the equivalent change in the provider classes
abc
This implements a delete method for pooltemplate and poolsize. The API
removes the override from Redis and then adds an entry in Redis that
causes the pool manager to wake up and process the removal of the
override.
To facilitate this, a new variable has been created in lib/vmpooler.rb
to hold a copy of the original / pre-override config. This supplemental
copy of the pools is then indexed for use as a reference.
When pool manager wakes up to process an override removal, it looks up
the pre-override value from the config via the new variables mentioned
above.
Just as with entering overrides, no restart is needed. Template and pool
size changes are logged so that anyone watching or reviewing the logs
can see what happened when. The new API endpoints also return values for
both the pre-revert and post-revert value.
Prior to this, the encryption settings for LDAP auth were hard coded to
start_tls on port 389 with TLSv1. These are still the defaults, as
insecure as they are, so as to not break existing users. This change
facilitates replacing the defaults so that simple_tls over port 636 via
TLS1.2 can be used.
This adds an "operation" label to the user metrics and moves incrementing from the manager to api, so that the user metrics show when resources are allocated, as well as destroyed. Previously, user metrics were only updated upon destroying a resource.
I think its better suited to increment the metric as part of the api instead of the pool_manger, because it's expected to do so when a user successfully checks out or deletes a VM, but can be problematic when doing so in the provider since it can clone VMs before actually being checked out by a user.
It appears we renamed `/ondemand/` to `/ondemandvm/` at some point and,
as a result, have not been stripping hostnames from that endpoint's
metrics. This has caused issues with metrics collection due a very high
cardinality.
This change adds detection of running instances that are in a running
queue, but have no data in a active queue for the same pool. When this
happens a machine will live forever, impacting the running count, and
preventing the machine from being killed. Without this change running
instances that are not marked as active will live forever.
Introducing the Prometheus Stats code into ABS showed that the Clarity
could be improved a bit with better variable naming, some refactoring
to reduce repitition and documenting the Metrics table itself.
Filtering these changes back to the vmpooler code base.
This commit updates folder purging references to ensure that provider
name references are referring to the named provider, rather than the
provider type. Without this change folder purging fails because it
cannot identify target folders.
Ensure that the correct stats are registered for the Manager and the api
respectively. E.g. all checkout counters are for the api only, whereas
clone times belong to the manager.
Also new ondemand functionality stats weren't registered, so add these
along with missing delete stats.
Review changes suggested to revise the Metrics related files into a more
logical class structure.
Also fixup grammar typos in docs strings and any trailing metrics that
have been recently added to vmpooler.
Use the example provided in the Ruby Client to provide a customised
collector appropriate to log all calls to the API. The customised
filtering is used to replace individual node names and templates
for the /vm and request ID's for the /ondemand endpoints.
This module was failing our rubocop checks so have updated it since
it now forms part of vmpooler.
Separate trapping for litmus jobs is also included so that they don't
interfere with stats from the jenkins pipelines.
Break down the usage stats into smaller groups so as to manage the
number of stat lines collected for Prometheus.
This may need some further revision to filter out Litmus stats, or
otherwise collect litmus usage information.
The redis pooler connection metric used "metric_prefix" which is
misleading, so split this into connpool_type and connpool_provider.
Also remove some earlier jruby compatibility code to reduce
rebase conflicts when this is rebased on top of Matt's changes.
This is a re-architect of the vmpooler initialisation code to:
1. Allow an API service for both manager and the api
2. Add the Prometheus endpoints to the web service.
Needed to change the way the Rack Service is started as instantiating
using ".New" leads to a failure to initialise the http Stats
collection.
3. Selectively load the pooler api and/or Prometheus endpoints.
4. Rework API Spec tests for revised API loading. Needed to tidy up the
initialisation and perform a reset! after each test to avoid "leaks"
and dependencies between the tests.
Add a new Prometheus class as an additional stats feed along with the
existing feeds.
Move the metrics initialisation code into its own class and sub-class
the individual metrics implementations under this.
* (POOLER-174) Reduce duplicate of on demand code introduced in POOLER-158
refactored every parsing of request of type 'pool_alias:pool:count' into a
utility class, that is used by pool_manager and the api v1 class
* add some metrics to the od request generation
* fix rubocop offenses, we are now friends
This change adds a capability to vmpooler to provision instances on
demand. Without this change vmpooler only supports retrieving machines
from pre-provisioned pools.
Additionally, this change refactors redis interactions to reduce round
trips to redis. Specifically, multi and pipelined redis commands are
added where possible to reduce the number of times we are calling redis.
To support the redis refactor the redis interaction has changed to
leveraging a connection pool. In addition to offering multiple
connections for pool manager to use, the redis interactions in pool
manager are now thread safe.
Ready TTL is now a global parameter that can be set as a default for all
pools. A default of 0 has been removed, because this is an unreasonable
default behavior, which would leave a provisioned instance in the pool
indefinitely.
Pool empty messages have been removed when the pool size is set to 0.
Without this change, when a pool was set to a size of 0 the API and pool
manager would both show that a pool is empty.
This commit fixes the purge_unconfigured_folders feature to ensure that it can successfully identify folders and instances that are no longer used. Without this change the feature does not work as advertised.
This commit adds detection for redis connection failures to pool_manager. When a connection fails the error will be raised to executeforcing the connection to be re-established. Without this change, when a redis connection fails, it generates a redis connection error, which is swallowed by a rescue for StandardError, preventing the manager application component from recovering in the case of a redis connection failure.
This commit adds the extra_config option to vmpooler to allow specifying additional configuration files to load from. Without this change vmpooler does not offer a mechanism to provide additional configuration files for the application.
This commit adds a capability to vmpooler to reset a pool, deleting its ready and pending instances and replacing them with fresh ones. Without this change vmpooler does not offer a mechanism to reset a pool without also changing its template.
Prior to this commit the pooler had no awareness of the complete set of
hostnames that are currently in use. This meant that it was possible to
allocate the same hostname twice, which would result in the original
host with that hostname becoming unreachable.
This commit adds a check for the existence of the
`vmpooler__vm__<hostname>` key before attempting to clone the vm.
This should prevent duplicate hostnames.
If the hostname is already taken, `_clone_vm` will retry with a new
random hostname multiple times before raising an exception.
Prior to this commit the hostname_shorten regex wouldn't match the
updated human readable hostnames because they contain dashes.
This commit updates the regex to capture dashes in the hostname, and
adds a few specs to verify that behavior.
This commit updates the create_linked_clone pool option to correctly detect when linked clones have been set at a pool level. Without this change a pool setting create_linked_clone to false is not interpreted correctly, and a linked clone is created if possible.
This change adds the running host for a VM to the API data available via /vm/hostname. Without this change the running host would be logged to vmpooler log, but not available any other way. Additionally, the data will specify if a machine has been migrated. Without this change parent host data for a vmpooler machine is not available via the vmpooler API.
This commit adds a new configuration parameter to allow setting whether to create linked clones on a global, or per pool basis. Without this change vmpooler would always attempt to create linked clones. The default behavior of creating linked clones is preserved.
This allows the user to change the cluster in which the targeted pool
will clone to. Upon configuration change, the thread will wake up and
execute the change within 1 second.
Before this change looping over many pools would query the redis backend
for each pool, leading in slow response from the backend for configurations
with many pools (60+)
Changed the requests to use redis pipelines https://redis.io/topics/pipelining
This is supported since the beginning, so will not force any redis update for
users. The pipeline method runs the queries in batches and we need to loop
over the result and reduces the number of requests to redis by N=number of
pools in the configuration.
