Previously VMPooler had no concept of a connection pooler. While there is an
up to date connection pooler Gem (connection_pool), that supports MRI and jRuby,
it lacked metrics which are useful to diagnose errors and judge pool size.
This commit:
- Brings in the connection_pool gem
- Creates a new class called generic_connection_pool which inherits from the
ConnectionPool class in the connection_pool gem.
- Extends the connection pool object with a new function called `with_metrics`
This copies the code from the original `with` method but emits metrics for
how long it took to get an object from the pool, and then how many objects
are left in the pool. This is sent using VMPooler's metrics object.
Extending the object was used instead of overriding as it was not possible to
inject into the existing function and monkey patching did not seem the correct
way.
In order use the metics, the GenericConnectionPool object modifies the
initialize method to use :metrics and :metrics_prefix options
- Also added tests for the GenericConnectionPool class to ensure the new
functions are tested. Note that the functionality that was not extended is
not tested in VMPooler.
In previous commits the code from vsphere_helper is now all moved to the vSphere
Provider. This commit removes the vsphere_helper.rb file, spec tests and from
being loaded by vmpooler itself.
Previously the vSphere based configuration was in the root of the configuration
YAML. As there is deprecation support to move the old configuration to the new
location, the vSphere provider can be updated. This commit updates the vSphere
Provider and tests to use the new configuration location under:
:providers:
:vsphere:
This commit modifies execute! to create the VM Providers on VMPooler startup
instead of check_pool creating a provider per pool. This commit also adds
legacy support for old configuration files:
- Setting the default provider for pools to be vsphere
- Copying VSphere connection settings in the configuration file from the legacy
location in the root, to under :providers/:vsphere which is new location for
all provider configuration
This commit adds a public function to access the internal variable holding the
VMPooler configuration. This is required for later commits for the execute!
function testing.
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the pool_manager to use the VM provider methods instead
- Removes the MockFindFolder class as it is no longer required
- Minor update for rubocop violations
Previously the Pool Manager would use a single VM provider per Pool. This
commit changes Pool Manager to use a single provider that services multiple
pools.
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the migrate_vm_and_record_timing method to use VM and Pool names
instead of VM and Pool objects.
Previously the Pool Manager would use vSphere objects directly. This commit
removes get_vm_host_info as this functionality is now in the vSphere VM
Provider.
Previously there was no simple way to calculate which pool a VM was a member of.
This commit adds a helper method which queries redis for the pool name for a
given VM.
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the pool_manager to use the VM provider methods instead
- Splits the destroy_vm function into two. One function spawns the thread
while the other actually does the work. This makes testing much easier.
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the pool_manager to use the VM provider methods instead
- Splits the check_ready_vm function into two. One function spawns the thread
while the other actually does the work. This makes testing much easier.
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the pool_manager to use the VM provider methods instead
- Modified to return true or false to indicate that the VM was failed
Previously the Pool Manager would use vSphere objects directly. This commit
- Modifies the pool_manager to use the VM provider methods instead
- Removes the open_socket method and tests as it is only required in the vSphere
VM provider
Previously the base VM provider class was added however it was missing various
functions from its definition. This commit:
- Modifies the VMPooler configuration to add an empty provider config. if the
provider config is missing
- Helper method to return all of the pools this provider is responsible for
This commit creates a VM Provider test fixture for spec tests that merely uses
the Base Provider class with a name of mock_provider. This will then be used
by unit tests in further commits.
Previously the only VM Provider was vSphere however this made testing and making
changes difficult as it required a functioning vSphere instance. This commit
adds a Dummy Provider which presents a VM provider to Pool Manager but manages
provisioned "VM"s in a hashtable. The Dummy Provider can also be configured to
randomly fail operations and take random amounts of time to perform operations,
such as cloning a VM, which is useful to see how the Pool Manager copes with
these events.
This commit also updates the configuration YAML documentation and adds
appropriate unit tests.
Previously the vSphere provider did not implement any of the required methods
from the base class. This commit modifies the vSphere provider so that in
can properly implement the following methods:
- name
- vms_in_pool
- get_vm_host
- find_least_used_compatible_host
- migrate_vm_to_host
- get_vm
- create_vm
- destroy_vm
- vm_ready?
- vm_exists?
- create_disk
- create_snapshot
- revert_snapshot
This commit also includes changes to syntax for rubocop violations.
Previously it was expected that the timeout setting should be passed when
determining whether a VM was ready. However this should be in the pool
configuration and is not required to be a method parameter.
Previously many of the methods did not have a pool name passed as a parameter.
While this may be ok for the vSphere provider, it may not for other providers.
This commit changes the base provider to consistently use (pool,vm,other..) as
method parameters. This commit also updates the base spec tests as well.
Additionally:
- Updated documentation around expected error states
- Updated documentation to be more consistent in format
- Added snapshot and disk manager functions and unit tests
- Update the initialization method to take in a more formal defintion with
required global objects for metrics, logging and configuration
- Added helper functions
- logger : Allows providers to log information as per Pool Manager
- metrics : Allows providers to submit metrics as per Pool Manager
- provider_options : Allows providers to access initialization options for a
provider
- pool_config : Get the configuration for a specific pool
- provider_config : Get the configuration of this specific provider
- global_config: Get the VMPooler global configuration
Previously, all calls to the vSphere API assumed an instance variable called
`@connection`. This commit prepares the provider for a connection pooler:
- Removes all references to `@connection` where needed and funnels all calls to
get a vSphere connection through the newly renamed method `get_connection`.
For the moment, this still uses `@connection` behind the scenes but will make
it easier in the future to migrate to a connection pooler
- Removes all references and tests for the ensure_connected method as it's no
longer required
- All methods that explicitly need a connection object will have this as part of
the method parameters
- The connect_to_vsphere method has been changed so that instead of setting the
instance level `@connection` object it just returns the newly created connection.
This can then be easily consumed by a connection pooler later.
Previously, the vSphere Provider had two methods called
`find_least_used_compatible_host`: one in the base class and one in the vSphere
helper methods. This commit renames the vSphere helper methods and tests to
`find_least_used_vsphere_compatible_host` to stop the conflict.
Previously there were no spec tests to document the behaviour of the
vsphere_helper. This commit adds the behavioural spec tests in prepration of
the refactoring work as part of POOLER-70, the connection pooling work in
POOLER-52 and various bugs found in vsphere_helper while these spec tests were
being created.
This commit fixes minor rubocopy violations in eleven source files. Minor
violations are those that include formatting, single quotes, and recently added
classes.
VM provisioning will be handled by VM Providers. This commit renames the use of
vsphere to provider where appropriate and changes the per-pool helper from
vsphere to providers to more accurately represent it's intended use.
Previously all of the VM provisioning code was intertwined with the VM lifecycle
code e.g. The VSphere specific code is mixed with Redis code. This makes it
impossible to add aditional providers or disable VSphere integration. This
commit begins the process to refactor the VSphere code out of the lifecycle code
by introducing the concept of VM Providers. A Provider will contain the logic/
code to manage VMs i.e. create/destroy/inquire. Therefore the Pool Manager can
query a strict interface into one or more Providers. Initially only a VSphere
provider will be available.
This commit adds the base class for all providers and describes the API or
contract that the Pool Manager will use to manage VMs.
Previously, the clone_vm method took various VSphere specific parameters e.g.
template folder. However in order make VMPooler less VSphere specific this
method should just take the pool configuration and then it can determine the
appropriate settings itself. This commit also moves the threading to a clone_vm
while the actual method which does the work is now _clone_vm as per all other
multithread worker methods in pool_manager. This commit also updates the spec
tests appropriately.
Previously in check_ready_vm, if the VM is powered off, the VM is moved in
redis however the function doesn't return there, and instead then checks if the
hostname is the same, and then if TCP socket 22 is open. This is unnecessary as
we already know the VM is turned off so of course the hostname is wrong and TCP
22 is unavailable. The same applies for the VM hostname.
This commit instead returns after it is found a VM is no longer ready. This
commit also amends the spec tests for the correct behaviour.
Add spec tests for check_snapshot_queue
Previously the check_snapshot_queue method would execute the loop indefinitely
as it did not have a terminating condition. This made it impossible to test.
This commit modifies the check_snapshot_queue method so that it can take a
maxloop and delay parameter so that it can be tested.
Add spec tests for check_disk_queue
Previously the check_disk_queue method would execute the loop indefinitely as it
did not have a terminating condition. This made it impossible to test. This
commit modifies the check_disk_queue method so that it can take a maxloop and
delay parameter so that it can be tested.
