This commit updates cpu_utilization_for and memory_utilization_for to detect when quickstats are not present. Without this change a nil result is transformed to 0, which is perceived as a host that has no utilization.
This commit removes an unnecessary rescue that results in duplicate clone error messages. Without this change clone failures due to unavailable host resources are logged twice. A log message is added to specify the host the VM is running on when migration_limit is not set and migration is disabled. Lastly, when a migration fails it reports the host the VM is running on in addition to the reason for the failed migration.
This change documents new vshpere specific parameters introduced related
to host selection and folder creation. Without this change these paremeters are not
documented.
This commit updates host selection to write check_time_finished whether
host retrieval was successful or not. Without this change when host
selection fails threads waiting for host selection to complete will
stuck waiting because check_time_finished doesn't update. Additionally,
because it would leave checking => true it would not attempt to inspect
and run host selection again.
Host selection for clones and migrations now make clear that no hosts
are available and fail logging a message. Without this change both
migrations and clones would fail with cryptic error messages logged
indicating clone and migrations failed.
Additionally, this change makes max_age configurable so a user can
specify that host selection should happen more or less frequently, as
required for migrations or clone operations when host selection is
enabled.
This commit updates the providers to move provider_hosts under the vsphere provider, which is the only place it's applicable. Methods where redis is passed through are updated to remove this pass through and use the globally available redis object, where applicable. Remove_vmpooler_migration_vm method is not needed and is removed.
This commit moves the migrate_vm logic to the vsphere provider. Without
this change migrate_vm has lots of vsphere specific logic in
pool_manager migrate_vm method.
This commit updates cluster host resource inspection to stop weighting memory as a part of the results. Additionally, instead of returning a single least used host a percentage of the eligible hosts with below average utilization are selected. Without this change host migration logic can cause cluster resource imbalances due to a single host being targeted for many migrations before the impacts of the migrations cause numbers to adjust.
This commit updates create_vm to target a cluster instead of an individual host for clone operations. Without this change cluster host utilization needs to be inspected for everyone clone operation.
This commit adds the capability to create folders within an existing target folder. Without this change folders to support platforms targets need to be created manually.
This commit adds a global provider_hosts concept in order to allow checking cluster utilization once per interval for a given cluster and retain the results, reusing them for an interval, and tracking the least used set of hosts. Without this change each migration and clone operation inspect host utilization and state for each host in the cluster, which is computationally expensive for vsphere.
The status endpoint provides a lot of statistics. This commit extends it
by supporting a query parameter called 'view' which may contain one or
multiple comma separated names for the top-level statistics returned
in the JSON response. status is always returned.
Optional elements are capacity,queue,clone,boot,pools
Everything is returned when 'view' is not specified, which is
backwards compatible with the current behavior.
Before this change if a pool had an alias configured, the information would not be
made public in the API. This commit adds the alias key in the pool object for each
pool if configured. The alias key can be abscent, a string or an one or multiple
array of strings. The value of the alias is copied from the configuration and can
represent another name for the pool, or another configured pool.
* Fix no implicit conversion to rational from nil
Before this change if the boottime was nil, the check_ready
loop would exit on Time.now - host['boottime'] with a TypeError
in jruby. The boottime is nil when the power is Off so moving that check
earlier should catch that bug.
* set test data properly
Previously in commit 9b0e55f959 the looping period was changed from a static
number to a dynamic one depending on load, however this meant that the operation
to refill a pool was slowed down somewhat. While not a problem under normal
loads, when a pool was quickly consumed, the pool manager may not respond
quickly enough to refill the pool. This commit:
- Changes the sleep method, to us a helper sleep method that will wakeup
periodically and evaluate other wakeup events. This could be used later to
exist sleep loops when pooler is shutting down to stop blocking threads
- By default the wakeup_period is set to the minimum pool check loop time, thus
emulating the behaviour prior to commit 9b0e55f959
- Adds tests for the behaviour
This commit adds vmpooler inspection of configuration issues to host
selection. Specifically, configIssue is checked, which should allow an
issue like quickstats not being reported to be identified even when the
alarm will not trigger. Without this change a host will continue to be
used if quickstats are not reported when alarms are not triggered
because of this condition, which results in a single host being the
target for all deploys and migrations, overloading the host and causing
its VMs to have degraded performance.
Previously, a static list was used to instantiate VM Pooler Provider objects.
This commit changes the loader to instead interrogate the available clases in
the Vmpooler::PoolManager::Provider namespace and then instantiate from there.
This means class names are not case sensitive and that VM Providers can now be
dynamically loaded from other sources such as gems in the LOADPATH. No tests
were added as this behaviour is exercised in the execute! tests already.
Previously, if inventory failed for some reason, it would return an incomplete
set of VMs which could then cause the pool to perform off behaviours such as
fill the pool high than it should, or remove VMs which exist. Also, if the
redis cache of VMs in a pool had a VM but it did not actually exist in the
inventory it would never be removed.
This commit:
- Immediately exits the check_pool if an error occurs during inventory
collection
- Will mark a VM as completed if it exists in Redis, but does not exist in
inventory
- Adds tests for these behaviours
This commit fixes the many rubocop offenses. Also modifies the rubocop
settings:
- Set max method params higher than the default of 5
- Ignore Style/GuardClause. In some cases it's eaiser to read without the guard
- Renamed a cop
Previously the check_pool would always check the pool every 5 seconds, however
with a large number of pools, this can cause resource issues inside the
providers. This commit:
- Introduces a dynamic check_pool period which increases during stability and
decreases when the pool is being change in an important way
- Surfaces the settings as global config defaults but can also be set on a per
pool basis
- Adds defaults to emulate the current behaviour
- Unit tests for the new behaviour
This commit updates find_least_used_compatible_host method to specify
the pool name when evaluating a VM for migration. Without this change VM
migration fails with a wrong number of arguments error. Pool_manager
test references are updated to reflect the change.
Previously the vsphere provider assumed that there was one and only one
datacenter (DC) in the vsphere instance. However this is simply not true for
many vSphere installations. This commit:
- Adds the ability to define a vSphere datacenter at the Pool or Provider level
whereby the Pool setting takes precedence
- If no datacenter is specified the default behaviour of picking the first DC
in the vSphere instance
- Updated all tests for the new setting
- Update the vmpooler configuration file example with relevant setting name
and expected behaviour
- Fixed a bug in the rvmomi_helper whereby if no DC was found it would return
all DCs. This is opposite behaviour of the real RBVMOMI library as it returns
nil
Previosuly in find_vmdks in vsphere_helper it uses the call
vmdk_datastore._connection to get the underlying connection, however this is
already available as function parameter. This commit removes this bad code and
the associated test fixtures.
Refactoring the vmpooler.yaml format to support multiple providers.
The second level key under :providers: is a unique key name that
represents a provider that can be refered in the pool's parameter
called provider. The code is still backward compatible to support
the :vsphere: and :dummy: keys but in reality if you have more than
one vsphere configuration you would give them a different name. For
example :vsphere-pdx: and :vsphere-bfs: and the actual provider
class would be specified as a parameter called 'provider_class'.
See tests and examples for more information.
The same method logic was being used in two places but only one was calling the method
get_cluster_host_utilization, so this refactors it to use the method for both.
The method could also return an empty array and the subsequent line would try
to .sort[0][1] which would return undefined method [] for nil:NilClass in that
case. The return value is now checked and an exception raised
The generic connection pooler is only responsible for managing the connection
objects, however the providers themselves are responsible for ensuring that the
connection is alive/healthy etc. Previously, the older vSphere helper would
reconnect however this was lost when the connection pooler was introduced. This
commit adds a method that checks the connection before use, and then reconnects
if the connection is in a bad state.
Previously the vSphere Provider would share a single vSphere connection for all
pools under management. This would cause issues in large environments as this
would cause errors to be thrown or operations to slow down. This commit
modifies the vSphere Provider to use a connection pool when communicating with
the vSphere API
- Uses the GenericConnectionPool object to manage the connection pool
- Uses a default connection pool size of:
Whatever is biggest from:
- How many pools this provider services
- Maximum number of cloning tasks allowed
- Need at least 2 connections so that a pool can have inventory functions
performed while cloning etc.
- A large connection_pool_timeout is used as a connection object is consumed
during a VM clone, which can take up to 2 minutes
- Removes the `get_connection` method as that is now obsolete due to the
connection pool
- Removes the `close` method as it is now obsolete
- Modified the spec tests slightly, to stop mocking get_connection as it no
longer exists, and set a super low pool timeout so that if a test fails, it
will fail quickly instead of taking the default time of 60+ seconds
Previously a connection pooler class was added. This commit modifies the Dummy
VM Provider to use a connection pooler. While the Dummy provider strictly
speaking does not use connections, this allows testing to see what happens when
connection pools are stressed or exhausted. This commit:
- Modifies functions to use a connection pool object for the public API
functions
- Modifies the VMPooler YAML with new settings for connection pool size and
timeout
Previously VMPooler had no concept of a connection pooler. While there is an
up to date connection pooler Gem (connection_pool), that supports MRI and jRuby,
it lacked metrics which are useful to diagnose errors and judge pool size.
This commit:
- Brings in the connection_pool gem
- Creates a new class called generic_connection_pool which inherits from the
ConnectionPool class in the connection_pool gem.
- Extends the connection pool object with a new function called `with_metrics`
This copies the code from the original `with` method but emits metrics for
how long it took to get an object from the pool, and then how many objects
are left in the pool. This is sent using VMPooler's metrics object.
Extending the object was used instead of overriding as it was not possible to
inject into the existing function and monkey patching did not seem the correct
way.
In order use the metics, the GenericConnectionPool object modifies the
initialize method to use :metrics and :metrics_prefix options
- Also added tests for the GenericConnectionPool class to ensure the new
functions are tested. Note that the functionality that was not extended is
not tested in VMPooler.
In previous commits the code from vsphere_helper is now all moved to the vSphere
Provider. This commit removes the vsphere_helper.rb file, spec tests and from
being loaded by vmpooler itself.
This commit updates vmpooler.rb, api.rb and providers.rb with style changes as
per rubocop style violations.
This commit also updates the rubocop configuration to always use LF line endings
even on Windows as rubocop was expecting CRLF even though git is configured
for LF.
Previously the vSphere based configuration was in the root of the configuration
YAML. As there is deprecation support to move the old configuration to the new
location, the vSphere provider can be updated. This commit updates the vSphere
Provider and tests to use the new configuration location under:
:providers:
:vsphere:
This commit modifies execute! to create the VM Providers on VMPooler startup
instead of check_pool creating a provider per pool. This commit also adds
legacy support for old configuration files:
- Setting the default provider for pools to be vsphere
- Copying VSphere connection settings in the configuration file from the legacy
location in the root, to under :providers/:vsphere which is new location for
all provider configuration
This commit adds a public function to access the internal variable holding the
VMPooler configuration. This is required for later commits for the execute!
function testing.
This commit adds a factory style function to create a VM provider object from
a provider name string. This is used during VMPooler initialization to create
the various VM providers on startup.
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the pool_manager to use the VM provider methods instead
- Removes the MockFindFolder class as it is no longer required
- Minor update for rubocop violations
Previously the Pool Manager would use a single VM provider per Pool. This
commit changes Pool Manager to use a single provider that services multiple
pools.
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the migrate_vm_and_record_timing method to use VM and Pool names
instead of VM and Pool objects.
