The status endpoint provides a lot of statistics. This commit extends it
by supporting a query parameter called 'view' which may contain one or
multiple comma separated names for the top-level statistics returned
in the JSON response. status is always returned.
Optional elements are capacity,queue,clone,boot,pools
Everything is returned when 'view' is not specified, which is
backwards compatible with the current behavior.
Before this change if a pool had an alias configured, the information would not be
made public in the API. This commit adds the alias key in the pool object for each
pool if configured. The alias key can be abscent, a string or an one or multiple
array of strings. The value of the alias is copied from the configuration and can
represent another name for the pool, or another configured pool.
* Fix no implicit conversion to rational from nil
Before this change if the boottime was nil, the check_ready
loop would exit on Time.now - host['boottime'] with a TypeError
in jruby. The boottime is nil when the power is Off so moving that check
earlier should catch that bug.
* set test data properly
Nokogiri and Redis gems have had recent releases which are not compatible with
older ruby versions. This commit modifies the Gemfile to get the latest of
each of these gems on modern ruby versions and pin to the older gem versions for
older ruby engines.
Previously in commit 9b0e55f959 the looping period was changed from a static
number to a dynamic one depending on load, however this meant that the operation
to refill a pool was slowed down somewhat. While not a problem under normal
loads, when a pool was quickly consumed, the pool manager may not respond
quickly enough to refill the pool. This commit:
- Changes the sleep method, to us a helper sleep method that will wakeup
periodically and evaluate other wakeup events. This could be used later to
exist sleep loops when pooler is shutting down to stop blocking threads
- By default the wakeup_period is set to the minimum pool check loop time, thus
emulating the behaviour prior to commit 9b0e55f959
- Adds tests for the behaviour
This commit adds vmpooler inspection of configuration issues to host
selection. Specifically, configIssue is checked, which should allow an
issue like quickstats not being reported to be identified even when the
alarm will not trigger. Without this change a host will continue to be
used if quickstats are not reported when alarms are not triggered
because of this condition, which results in a single host being the
target for all deploys and migrations, overloading the host and causing
its VMs to have degraded performance.
The vmpooler.yaml file has been reorganized a bit and now nests the
vsphere credentials within a "providers" class. This tweak is needed
to keep the template delta script working.
Due to native extensions not working for JRuby on Windows easily, sometimes very
old versions of MRI Ruby are needed for debugging (1.9.3). This commit pins
nokogiri to a version which is compatible with ruby 1.9.x. It is expected this
is only required for debugging in edge cases. vmpooler should still be run on
modern ruby versions in production.
Previously, a static list was used to instantiate VM Pooler Provider objects.
This commit changes the loader to instead interrogate the available clases in
the Vmpooler::PoolManager::Provider namespace and then instantiate from there.
This means class names are not case sensitive and that VM Providers can now be
dynamically loaded from other sources such as gems in the LOADPATH. No tests
were added as this behaviour is exercised in the execute! tests already.
Previously, if inventory failed for some reason, it would return an incomplete
set of VMs which could then cause the pool to perform off behaviours such as
fill the pool high than it should, or remove VMs which exist. Also, if the
redis cache of VMs in a pool had a VM but it did not actually exist in the
inventory it would never be removed.
This commit:
- Immediately exits the check_pool if an error occurs during inventory
collection
- Will mark a VM as completed if it exists in Redis, but does not exist in
inventory
- Adds tests for these behaviours
Sometimes this test would fail if the computer running the tests was under a
bit of load. This commit changes the expected output to be up to 1.99 seconds
instead of the previous 0.99 seconds.
This commit fixes the many rubocop offenses. Also modifies the rubocop
settings:
- Set max method params higher than the default of 5
- Ignore Style/GuardClause. In some cases it's eaiser to read without the guard
- Renamed a cop
Previously the check_pool would always check the pool every 5 seconds, however
with a large number of pools, this can cause resource issues inside the
providers. This commit:
- Introduces a dynamic check_pool period which increases during stability and
decreases when the pool is being change in an important way
- Surfaces the settings as global config defaults but can also be set on a per
pool basis
- Adds defaults to emulate the current behaviour
- Unit tests for the new behaviour
This commit updates find_least_used_compatible_host method to specify
the pool name when evaluating a VM for migration. Without this change VM
migration fails with a wrong number of arguments error. Pool_manager
test references are updated to reflect the change.
Previously the vsphere provider assumed that there was one and only one
datacenter (DC) in the vsphere instance. However this is simply not true for
many vSphere installations. This commit:
- Adds the ability to define a vSphere datacenter at the Pool or Provider level
whereby the Pool setting takes precedence
- If no datacenter is specified the default behaviour of picking the first DC
in the vSphere instance
- Updated all tests for the new setting
- Update the vmpooler configuration file example with relevant setting name
and expected behaviour
- Fixed a bug in the rvmomi_helper whereby if no DC was found it would return
all DCs. This is opposite behaviour of the real RBVMOMI library as it returns
nil
This commit adds documentation for how to setup a development environment for
VM Pooler. This commit also moves the API under the new docs directory and
modifies the README appropriately.
Previosuly in find_vmdks in vsphere_helper it uses the call
vmdk_datastore._connection to get the underlying connection, however this is
already available as function parameter. This commit removes this bad code and
the associated test fixtures.
Set to use a newer JRuby
Added an environment variable for the log file location
Changed entrypoint to cmd to add flexibility
Added missing trailing slash to copy command path
Refactoring the vmpooler.yaml format to support multiple providers.
The second level key under :providers: is a unique key name that
represents a provider that can be refered in the pool's parameter
called provider. The code is still backward compatible to support
the :vsphere: and :dummy: keys but in reality if you have more than
one vsphere configuration you would give them a different name. For
example :vsphere-pdx: and :vsphere-bfs: and the actual provider
class would be specified as a parameter called 'provider_class'.
See tests and examples for more information.
The same method logic was being used in two places but only one was calling the method
get_cluster_host_utilization, so this refactors it to use the method for both.
The method could also return an empty array and the subsequent line would try
to .sort[0][1] which would return undefined method [] for nil:NilClass in that
case. The return value is now checked and an exception raised
The generic connection pooler is only responsible for managing the connection
objects, however the providers themselves are responsible for ensuring that the
connection is alive/healthy etc. Previously, the older vSphere helper would
reconnect however this was lost when the connection pooler was introduced. This
commit adds a method that checks the connection before use, and then reconnects
if the connection is in a bad state.
Previously the vSphere Provider would share a single vSphere connection for all
pools under management. This would cause issues in large environments as this
would cause errors to be thrown or operations to slow down. This commit
modifies the vSphere Provider to use a connection pool when communicating with
the vSphere API
- Uses the GenericConnectionPool object to manage the connection pool
- Uses a default connection pool size of:
Whatever is biggest from:
- How many pools this provider services
- Maximum number of cloning tasks allowed
- Need at least 2 connections so that a pool can have inventory functions
performed while cloning etc.
- A large connection_pool_timeout is used as a connection object is consumed
during a VM clone, which can take up to 2 minutes
- Removes the `get_connection` method as that is now obsolete due to the
connection pool
- Removes the `close` method as it is now obsolete
- Modified the spec tests slightly, to stop mocking get_connection as it no
longer exists, and set a super low pool timeout so that if a test fails, it
will fail quickly instead of taking the default time of 60+ seconds
Previously a connection pooler class was added. This commit modifies the Dummy
VM Provider to use a connection pooler. While the Dummy provider strictly
speaking does not use connections, this allows testing to see what happens when
connection pools are stressed or exhausted. This commit:
- Modifies functions to use a connection pool object for the public API
functions
- Modifies the VMPooler YAML with new settings for connection pool size and
timeout
Previously VMPooler had no concept of a connection pooler. While there is an
up to date connection pooler Gem (connection_pool), that supports MRI and jRuby,
it lacked metrics which are useful to diagnose errors and judge pool size.
This commit:
- Brings in the connection_pool gem
- Creates a new class called generic_connection_pool which inherits from the
ConnectionPool class in the connection_pool gem.
- Extends the connection pool object with a new function called `with_metrics`
This copies the code from the original `with` method but emits metrics for
how long it took to get an object from the pool, and then how many objects
are left in the pool. This is sent using VMPooler's metrics object.
Extending the object was used instead of overriding as it was not possible to
inject into the existing function and monkey patching did not seem the correct
way.
In order use the metics, the GenericConnectionPool object modifies the
initialize method to use :metrics and :metrics_prefix options
- Also added tests for the GenericConnectionPool class to ensure the new
functions are tested. Note that the functionality that was not extended is
not tested in VMPooler.
In previous commits the code from vsphere_helper is now all moved to the vSphere
Provider. This commit removes the vsphere_helper.rb file, spec tests and from
being loaded by vmpooler itself.
