vmpooler has the vsphere provider taken out, moving some vsphere related
methods to the provider:
1) pool_folders
2) get_base_folders
At the same time renaming some configuration and code items
to remove harmful terminology.
purge_unconfigured_folders DEPRECATED, use purge_unconfigured_resources
folder_whitelist DEPRECATED, use resources_allowlist
the above configuration items are still supported but will be removed in
the next major version.
base class method purge_unconfigured_folders was renamed to purge_unconfigured_resources
and requires the equivalent change in the provider classes
abc
This implements a delete method for pooltemplate and poolsize. The API
removes the override from Redis and then adds an entry in Redis that
causes the pool manager to wake up and process the removal of the
override.
To facilitate this, a new variable has been created in lib/vmpooler.rb
to hold a copy of the original / pre-override config. This supplemental
copy of the pools is then indexed for use as a reference.
When pool manager wakes up to process an override removal, it looks up
the pre-override value from the config via the new variables mentioned
above.
Just as with entering overrides, no restart is needed. Template and pool
size changes are logged so that anyone watching or reviewing the logs
can see what happened when. The new API endpoints also return values for
both the pre-revert and post-revert value.
Prior to this, the encryption settings for LDAP auth were hard coded to
start_tls on port 389 with TLSv1. These are still the defaults, as
insecure as they are, so as to not break existing users. This change
facilitates replacing the defaults so that simple_tls over port 636 via
TLS1.2 can be used.
This update includes two key benefits:
1. Spans will be named based on their route instead of the full path
info thanks to https://github.com/open-telemetry/opentelemetry-ruby/pull/415
2. Helper methods were added to the configurator to simplify setting
service.name and service.version
This change utilizes OpenTelemetry's automatic instrumentation to add
distributed tracing capabilities to VMPooler. This is a non-breaking
change as traces are processed in noop mode by default.
Use the example provided in the Ruby Client to provide a customised
collector appropriate to log all calls to the API. The customised
filtering is used to replace individual node names and templates
for the /vm and request ID's for the /ondemand endpoints.
This module was failing our rubocop checks so have updated it since
it now forms part of vmpooler.
Separate trapping for litmus jobs is also included so that they don't
interfere with stats from the jenkins pipelines.
The redis pooler connection metric used "metric_prefix" which is
misleading, so split this into connpool_type and connpool_provider.
Also remove some earlier jruby compatibility code to reduce
rebase conflicts when this is rebased on top of Matt's changes.
Need the logger code in promstats.rb so move logger initialisation to
the top of the vmpooler script, remove the class method from vmpooler.rb
and make appropriate downstream changes to metrics and pooler manger
handling.
This also means we can start decent logging in the API if we wish do.
Fixed up to rebase on top of Matts POOLER-158 changes
Add a new Prometheus class as an additional stats feed along with the
existing feeds.
Move the metrics initialisation code into its own class and sub-class
the individual metrics implementations under this.
This change adds a capability to vmpooler to provision instances on
demand. Without this change vmpooler only supports retrieving machines
from pre-provisioned pools.
Additionally, this change refactors redis interactions to reduce round
trips to redis. Specifically, multi and pipelined redis commands are
added where possible to reduce the number of times we are calling redis.
To support the redis refactor the redis interaction has changed to
leveraging a connection pool. In addition to offering multiple
connections for pool manager to use, the redis interactions in pool
manager are now thread safe.
Ready TTL is now a global parameter that can be set as a default for all
pools. A default of 0 has been removed, because this is an unreasonable
default behavior, which would leave a provisioned instance in the pool
indefinitely.
Pool empty messages have been removed when the pool size is set to 0.
Without this change, when a pool was set to a size of 0 the API and pool
manager would both show that a pool is empty.
This commit adds the extra_config option to vmpooler to allow specifying additional configuration files to load from. Without this change vmpooler does not offer a mechanism to provide additional configuration files for the application.
This commit adds a new configuration parameter to allow setting whether to create linked clones on a global, or per pool basis. Without this change vmpooler would always attempt to create linked clones. The default behavior of creating linked clones is preserved.
This change updates handling of pool aliases to allow for more than a
single pool to be configured as an alias pool. Without this change if
multiple pools are configured as an alias the last one to configure it
is set as the alias for that pool.
Additionally, redis testing requirements are removed in favor of
mock_redis. Without this change a redis server is required to run
vmpooler tests.
This commit updates vmpooler to set configuration values received via environment variables to integer values when an integer value is expected. Without this change vmpooler does not support setting integer values via environment variables. Additionally, testing is added for configuring vmpooler via environment variables.
To support this testing the gem climate_control is added, which allows
for testing environment variables without those set variables leaking to
other tests.
This commit updates vmpooler to ship VM usage stats when a VM is destroyed. The stats are gathered for jobs based on user and pool name. If a jenkins build URL is present then this is broken down by user, instance, value stream, branch and project. Additionally, if present then the RMM_COMPONENT_TO_TEST_NAME will be listed after project. Without this change we do not collect stats on per VM usage and its correlation to users and pools.
This commit fixes checking of a VM that has already been identified as ready. Without this change a ready VM that has failed will be identified as having failed, but will not successfully be removed from the ready queue. Additionally, the default vm_checktime value has been reduced from 15 to 1 to ensure that ready VMs are checked within one minute of the time they have reached the ready state by default.
Lastly, the docker-compose files are updated to specify that the redis
instance used is a local redis instance.
This commit updates get_vm in the vmpooler API to allow for setting weights for backends. Additionally, when an alias for a pool exists, and the backend configured is not weighted, then the selection of the pool based on alias will be randomly sampled. Without this change any pool with the title of the alias is exhausted before an alternate pool with the configured alias is used, which results in an uneven distribution of VMs. When all backends involved are configured with weighted values the VM selection will be based on probability using those weights.
A bug is fixed when setting the default ttl for check_ready_vm.
Pickup is added to handle weighted VM selection.
A dockerfile is added that allows for building and installing vmpooler
from the current HEAD in docker to make for easy testing.
This commit adds documentation for specifying vmpooler configuration via environment variables. LDAP server is corrected to LDAP host, and the capability to specify graphite prefix and port is added. Vagrant documentation is moved into its own file to reduce noise in the main readme. Lastly, docker usage is updated to reflect that you no longer bake in a configuration file, and that API and manager can be run separately from docker via its entrypoint.
This commit updates vmpooler to allow the API component and dashboard to
run separately from pool_manager. Without this change vmpooler does not
offer a mechanism to run only the API, or pool_manager components.
Two instances of hardcoded puma environment settings are removed. This
is still set in the init script explicitly as well as via an environment
variable in the dockerfile.
To extend the mechanism of running the API or pool_manager components to
instances running in docker an entrypoint is added in the dockerfile.
The entrypoint allows a user to specify whether to run the API or
pool_manager components when running the application. The default
behavior is preserved where both components are run.
To support these changes vmpooler.rb is updated to allow more of the
configuration to be specified via individual environment variables. It
was already possible to specify the entire config block as an
environment variable, but this is more difficult to manage and less of a
standard implementation than specifying individual parameters. Where
specified environment variable options will override a value configured
via the configuration file or environment.
The running pool configuration when starting pool_manager is loaded to
redis at pool_manager start time. This allows the API to load the
running pool configuration from redis and be able to run without
requiring the pool configuration.
Lastly, the dockerfile leveraging entrypoint will no longer start
vmpooler with the init script or write logs to a file. Instead, LOGFILE
is set to /dev/stdout and the vmpooler application is started directly.
This behavior is preferred because the log file writes to disk are an
unnecessary overhead. Without this change the docker installation will
attempt to daemonize the vmpooler application and always requires puma.
This commit moves net/ldap require from authenticate_ldap in api/helpers to vmpooler.rb. Without this change net/ldap and rubygems are required again every time authenticate_ldap is run.
This commit updates the method used to load a configuration file to use
YAML.load_file, which is how it was configured previously. The
capability to specify an alternate configuration file via the
VMPOOLER_CONFIG_FILE is retained, and now works as expected.
variable
Previously, there were two ways to configure Vmpooler, either by
changing the contents of vmpooler.yaml or by assigning the raw YAML
to the VMPOOLER_CONFIG environment variable. This commit adds a new
environment variable called VMPOOLER_CONFIG_FILE that can be assigned
the name of a config file to use. Also fixes#240 by whitelisting the
Symbol class when calling YAML.safe_load in Vmpooler.config.
Previously VMPooler had no concept of a connection pooler. While there is an
up to date connection pooler Gem (connection_pool), that supports MRI and jRuby,
it lacked metrics which are useful to diagnose errors and judge pool size.
This commit:
- Brings in the connection_pool gem
- Creates a new class called generic_connection_pool which inherits from the
ConnectionPool class in the connection_pool gem.
- Extends the connection pool object with a new function called `with_metrics`
This copies the code from the original `with` method but emits metrics for
how long it took to get an object from the pool, and then how many objects
are left in the pool. This is sent using VMPooler's metrics object.
Extending the object was used instead of overriding as it was not possible to
inject into the existing function and monkey patching did not seem the correct
way.
In order use the metics, the GenericConnectionPool object modifies the
initialize method to use :metrics and :metrics_prefix options
- Also added tests for the GenericConnectionPool class to ensure the new
functions are tested. Note that the functionality that was not extended is
not tested in VMPooler.
In previous commits the code from vsphere_helper is now all moved to the vSphere
Provider. This commit removes the vsphere_helper.rb file, spec tests and from
being loaded by vmpooler itself.
This commit updates vmpooler.rb, api.rb and providers.rb with style changes as
per rubocop style violations.
This commit also updates the rubocop configuration to always use LF line endings
even on Windows as rubocop was expecting CRLF even though git is configured
for LF.
Previously all of the VM provisioning code was intertwined with the VM lifecycle
code e.g. The VSphere specific code is mixed with Redis code. This makes it
impossible to add aditional providers or disable VSphere integration. This
commit begins the process to refactor the VSphere code out of the lifecycle code
by introducing the concept of VM Providers. A Provider will contain the logic/
code to manage VMs i.e. create/destroy/inquire. Therefore the Pool Manager can
query a strict interface into one or more Providers. Initially only a VSphere
provider will be available.
This commit adds the base class for all providers and describes the API or
contract that the Pool Manager will use to manage VMs.
This commit updates VM Pooler to fix any existing rubocop offenses and also
fixes any offenses in the lib/vmpooler.rb file. This commit also regenerates
the Todo file.
If a user attempts to start vmpooler using dummy authentication
without setting the environment variable VMPOOLER_DEBUG, the vmpooler
will now refuse to start.
If `ENV['VMPOOLER_CONFIG']` is defined, it is read in as a YAML
configuration. This allows vmpooler to run in a docker daemon via
`docker run -e VMPOOLER_CONFIG -p 80:4567 -it vmpooler` rather than
embedding a YAML file within the container.
* [QENG-3919] Make vmpooler checkouts be all or nothing (#153)
* (QENG-3919) spike for implementation of all-or-nothing checkout
* Fix two botched variable references
* Aggregate API helper methods
* Add specs for failed multi-vm allocation API endpoints
* (QENG-3919) Add tests for multiple vm requests
* (QENG-3919) Add (failing) specs for POST /vm/pool1+pool2 usages
This exposes the old (bad) behavior on this other code path. Will fix this up next.
* (QENG-3919) Bring query params version in line with JSON post version
Not clear to me why these had to be implemented so differently.
* (QENG-3919) extract common method from both methods of VM allocation
* (QENG-3919) Naming fix, cosmetic cleanups
I mean, I presume all these commits are going to get squashed away on merge anyway.
* (QENG-3919) Update API docs
We consider it a bug that the actual behavior was not this behavior, but the
documentation was also silent on this point.
* (QENG-3919) minor readability tweak in refactored method
* (QENG-3919) Clean up interim comments re: status codes
* (QENG-3919) Drop now-orphaned `checkout_vm` method
We kept this up-to-date while we were upgrading and refactoring, but, turns out,
this method is no longer called anywhere. 💀🔥
* (QENG-3919) Return 503 status on failed allocation
Making sure we go back to the original functionality, which was:
- status 200 when vms successfully allocated
- status 404 when a pool name is unknown
- status 404 when no pool name is specified
- status 503 when vm allocation failed
* (QENG-3919) add net-ldap to Gemfile
Maybe we shouldn't foil-ball gems onto servers.
* (QENG-3919) Turns out, spush isn't a redis command
And hence we see once again the weakness of mockist tests.
* (QENG-3919) Pin the net-ldap gem to 0.11 for the jrubies, etc.
* (QENG-3919) Correct an old spelling error in spec descriptions
* (QENG-3919) Further tweak net-ldap version
* (QENG-3919) return_single_vm -> return_vm_to_ready_state
cc @shermdog
* (RE-7014) Add support for statsd
They way we were using graphite was incorrect for the type of data we were sending it. statsd is the appropriate mechanism for our needs.
statsd and graphite are mutually exclusive and configuring statsd will take precendence over Graphite. Example of configuration in vmpooler.yaml.example
* (RE-7014) Add tracking of vm gets via statsd
Add the tracking of successful, failed, invalid, and empty pool vm gets. It is possible we may want to tweak this, but have validated with spec tests and pcaps.
```
vmpooler-tmp-dev.ready.debian-7-x86_64:1|c
vmpooler-tmp-dev.running.debian-7-x86_64:1|c
vmpooler-tmp-dev.checkout.invalid:1|c
vmpooler-tmp-dev.checkout.success.debian-7-x86_64:1|c
vmpooler-tmp-dev.checkout.empty:1|c
vmpooler-tmp-dev.running.debian-7-x86_64:1|c
vmpooler-tmp-dev.clone.debian-7-x86_64:12.10|ms
vmpooler-tmp-dev.ready.debian-7-x86_64:1|c
```
* (RE-7014) statsd nitpicks and additional rspec
Cleaned up some code review nitpicks and added pool_manager_spec for empty pool.
* (RE-7014) update statsd to use gauge for running/ready
Previously was using increment which was incorrect for that particular application.
* Revert "Merge pull request #155 from shermdog/RE-7014-cinext"
This reverts commit cc03a86f6a, reversing
changes made to 5aaab7c5c2.
* (QENG-4070) Consistently return 503 if valid pool is empty
There were several problems with how the pooler checked out vms with
respect to empty pools, invalid pools, and aliases:
- If the vmpooler config did not contain any aliases and the caller
requested a vm from an empty pool or a non-existent one, the vmpooler
would error with:
NoMethodError - undefined method `[]' for nil:NilClass
If the config contained a non-nil alias section, then:
- If the caller requested a vm from an empty pool and either the vm
didn't have an alias or the aliased pool was empty or non-existent, then
the request for that vm would be silently ignored. The vmpooler would
return 200 if the caller asked for multiple vms and the vmpooler was
able to checkout at least one vm. Otherwise it would return 404.
- Similarly, if the caller requested a vm from a non-existent pool, then
the request was silently ignored.
This commit adds a `pool_names` Set to the config containing all valid
pool names including aliases. This is used to determine whether a
requested template name is valid or not. This is necessary because redis
does not distinguish between empty and non-existent sets, e.g. the
following returns false in both cases:
backend.exists('vmpooler__ready__' + key)
If the caller requests a vm (single or multiple), and any vm references
an invalid pool name, we immediately return 404. Otherwise, we know the
request is for valid pool names, since the vmpooler requires a restart
to change pool names and counts.
We then attempt to acquire each vm, trying to match on pool name or
failing back to aliased pool name, as was the previous behavior.
The resulting behavior is:
- If the caller asks for at least one vm from an unknown pool, then
don't try to checkout any vms and respond with 404.
- If the caller asks for a vm, and at least one pool is empty, then
respond with 503, returning checked out vms back to the pool.
- Otherwise return 200 with the list of checked out vms.
This commit also makes `alias` optional again.
This commit also re-enables tests that were merged in from master, but
originally commented out due to the bugs described above..
* (maint) Add json pessimistic pin
json 2.0.x was released on July 1 and is not compatible with ruby < 2.0.
Since we still support that version, add a pessimistic pin, which is
what we were using prior to July 1.
* [QENG-4070] Make json version conditional on RUBY_VERSION
* Drop extraneous mocks from updated test
* Revert "Revert "Merge pull request #155 from shermdog/RE-7014-cinext""
This reverts commit 0fd6fff934.
* Fix some spec errors
These were caused in part by dropping changes from the original PR when we
dropped the v1_spec.rb master test file (in favor of the updated and separated
versions).
* [QENG-4075] Fix bug with template name on allocation failure
We're returning [nil,nil] in this case, meaning that name will not be set. This
means we'll get an error trying to concatenate the stats string. Use the
requested template name here instead.
* [QENG-4075] Refactor statsd methods / classes
Prior to this we could easily run into situations where `statds_prefix` would
be `nil` (and possibly the `statsd` handle itself). There was some significant
complexity and brittleness in how statsd was set up.
Refactored so that:
- `statsd_prefix` is no longer exposed to any callers of statsd methods
- there is now a `Vmpooler::DummyStatsd` class which can be returned when we are not actually going to publish stats, but would like to keep the calling interface consistent
- setup of the statsd handle is via just passing in `config[:statsd]`, if `nil`, this will result in a dummy handle being return
- defaulting of `server` values was fixed -- this did not actually work in the previous implementation. `config[:statsd][:server]` is now required.
- tests use a `DummyStatsd` instance instead of an rspec double.
- calls to `statsd.increment` were taking incorrect arguments (some our fault, some part of the prior implementation), and were not collecting data on which pools were "invalid" or "empty". Fixed this and are now explicitly tracking the invalid/empty pool names.
* [QENG-4075] Drop now-superfluous :statsd config defaulting
* [QENG-4075] Unify graphite and statsd for the pool manager
Prior to this, the `pool_manager.rb` library could take handles for both
graphite and statsd endpoints (which were considered mutually exclusive),
and then would use one. There was a bevy of conditional logic around sending
metrics to the graphite/statsd handles (and actually at least one bug of
omission).
Here we refactor more, building on earlier work:
- Our graphite class comes into line with the API of our Statsd and DummyStatsd classes
- In `pool_manager.rb` we now accept a single "metrics" handle, and we drop all the conditional logic around statsd vs. graphite
- We move the inconsistent error handling out of the calling classes and into our metrics classes, actually logging to `$stderr` when we can't publish metrics
- We unify the setup code to use `config` to determine whether statsd, graphite, or a dummy metrics handle should be used, and make that happen.
- Cleaned up some tests. We could probably stand to do a bit more work in this area.
* [QENG-4075] Clean up pool manager, specs
Prior to this, `pool_manager.rb` allowed the `metrics` argument to be optional,
but at this point it will be an instance of `Vmpooler::Statsd`,
'Vmpooler::Graphite', or `Vmpooler::DummyStatsd`, so making this non-optional.
Cleaned up that file's tests, cosmetically, as well as recognizing that the
behavioral difference between graphite and statsd does not depend on the pool
manager.
* [QENG-4075] update example vmpooler.yaml file
This documents the changes to :server being mandatory for all metrics
endpoints, as well as the graphite endpoint supporting an optional :port
configuration value.
* [QENG-4075] Rename usages of statsd -> metrics
Really, let's just support a generic metrics interface.
* (maint) move statsd-ruby require into Vmpooler::Statsd class
We've managed to move mentions of this out of the calling code, so let's
move the require.
* (maint) metrics.log -> metrics.timing
We missed this during the refactoring. Bringing this up to date.
* [QENG-4075] Allow specifying 'graphs:' for dashboard
Prior to this the dashboard front-end would use the configuration settings
for `graphite[:server]`/`graphite[:prefix]` to locate a graphite server
to use for rendering graphs.
Now that we have multiple possible metrics backends, the front-end graph
host for the dashboard could be entirely different from the back-end metrics
server that we publish to (if any).
This decouples those settings:
- use `graphs[:server]` / `graphs[:prefix]` for the graphite-compatible web front-end to use for dashboard display graphs
- fall back to `graphite[:server]`/`graphite[:prefix]` if `graphs` is not specified, in order to support legacy `vmpooler.yaml` configurations.
Note that since `statsd` takes precedence over `graphite`, it's possible to specify both `statsd` (for publishing) and `graphite` (for reading). We still prefer `graphs` over `graphite`.
Updated the example `vmpooler.yaml` config file.
* (maint) fix variable reference in new_metrics
This was referencing config directly, when what we want is for a
hash to be passed in (derived from config).
* (maint) Fix typo in updated graph link call
* (maint) default :graphs prefix to 'vmpooler'
* (maint) Fix parse error in vmpooler script
The things you find through manual QA 🧌
* (maint) use strings instead of symbols in config
Nested hash data comes back with string keys, not symbols. Be consistent.
* [QENG-4075] Factor out Vmpooler::DummyStatsd
This makes it visible to lib/vmpooler.rb, as well as putting this dummy
metrics endpoint in its own file for easier discovery.
* (maint) clean up statsd inclusion and require lines
The library is actually required as 'statsd' and not 'ruby-statsd', best I can tell.
* (maint) construct ::Statsd instead of Statsd
Because it's ambiguous in this scope, and, well, it doesn't
actually work in production.
* [QENG-4075] Also track completely invalid requests
When we don't even get a pool name we still want metrics to be recorded.
The following pool configuration would allow a pool to be aliased in POST
requests as 'centos-6-x86_64', 'centos-6-amd64', or 'centos-6-64':
````yaml
- name: 'centos-6-x86_64'
alias: [ 'centos-6-amd64', 'centos-6-64' ]
template: 'templates/centos-6-x86_64'
folder: 'vmpooler/centos-6-x86_64'
datastore: 'instance1'
size: 5
````
The 'alias' configuration can be either a string or an array.
Note that even when requesting an alias, the pool's 'name' is returned in
the JSON response:
````
$ curl -d '{"centos-6-64":"1"}' --url vmpooler/api/v1/vm
````
````json
{
"ok": true,
"centos-6-x86_64": {
"hostname": "cuna2qeahwlzji7"
},
"domain": "company.com"
}
````
Prior to this commit, several pieces of vmpooler performed configuration
and initialization steps within 'initialize'. This made it difficult to
pass in mock objects for testing purposes.
This commit performs a single configuration and passes the results to
the various pieces of vmpooler.
