Previously, all calls to the vSphere API assumed an instance variable called
`@connection`. This commit prepares the provider for a connection pooler:
- Removes all references to `@connection` where needed and funnels all calls to
get a vSphere connection through the newly renamed method `get_connection`.
For the moment, this still uses `@connection` behind the scenes but will make
it easier in the future to migrate to a connection pooler
- Removes all references and tests for the ensure_connected method as it's no
longer required
- All methods that explicitly need a connection object will have this as part of
the method parameters
- The connect_to_vsphere method has been changed so that instead of setting the
instance level `@connection` object it just returns the newly created connection.
This can then be easily consumed by a connection pooler later.
Previously, the vSphere Provider had two methods called
`find_least_used_compatible_host`: one in the base class and one in the vSphere
helper methods. This commit renames the vSphere helper methods and tests to
`find_least_used_vsphere_compatible_host` to stop the conflict.
Previously, in `find_vm` and `find_pool` if in an expected object was found the
entire pool manager, and probably API process, will terminate due to the use of
`abort`. This commit changes the use of abort to raise so that the error can be
trapped and handled instead of the entire process being shutdown. This is also
required so that the methods can be tested, otherwise rspec is shutdown
prematurely.
Previously, it was not able to mock objects to impersonate various RBVMOMI
objects. This commit changes the case statement to use `base.is_a?`
which can be mocked and allow mocked objects to mimic real objects.
This commit fixes minor rubocopy violations in eleven source files. Minor
violations are those that include formatting, single quotes, and recently added
classes.
Previously, it was not able to mock objects to impersonate a RbVmomi::VIM::Folder
object. This commit changes the case statement to an if statement using is_a?
which can be mocked and allow mocked objects to mimic real objects.
VM provisioning will be handled by VM Providers. This commit renames the use of
vsphere to provider where appropriate and changes the per-pool helper from
vsphere to providers to more accurately represent it's intended use.
Previously all of the VM provisioning code was intertwined with the VM lifecycle
code e.g. The VSphere specific code is mixed with Redis code. This makes it
impossible to add aditional providers or disable VSphere integration. This
commit begins the process to refactor the VSphere code out of the lifecycle code
by introducing the concept of VM Providers. A Provider will contain the logic/
code to manage VMs i.e. create/destroy/inquire. Therefore the Pool Manager can
query a strict interface into one or more Providers. Initially only a VSphere
provider will be available.
This commit adds the base class for all providers and describes the API or
contract that the Pool Manager will use to manage VMs.
In commit 03ad7bfa46 the global variables for credentials was change to an
instance variable however one reference was missed. This commit fixes this
omission.
This commit updates usage of global variables in vsphere_helper to be instance variables. They do not need to be global variables as brought up in issue #194. Without this change we are setting global variables when they are not needed.
This commit updates VM Pooler to fix any existing rubocop offenses and also
fixes any offenses in the lib/vmpooler.rb file. This commit also regenerates
the Todo file.
Previously, the clone_vm method took various VSphere specific parameters e.g.
template folder. However in order make VMPooler less VSphere specific this
method should just take the pool configuration and then it can determine the
appropriate settings itself. This commit also moves the threading to a clone_vm
while the actual method which does the work is now _clone_vm as per all other
multithread worker methods in pool_manager. This commit also updates the spec
tests appropriately.
Previously in check_ready_vm, if the VM is powered off, the VM is moved in
redis however the function doesn't return there, and instead then checks if the
hostname is the same, and then if TCP socket 22 is open. This is unnecessary as
we already know the VM is turned off so of course the hostname is wrong and TCP
22 is unavailable. The same applies for the VM hostname.
This commit instead returns after it is found a VM is no longer ready. This
commit also amends the spec tests for the correct behaviour.
Add spec tests for check_snapshot_queue
Previously the check_snapshot_queue method would execute the loop indefinitely
as it did not have a terminating condition. This made it impossible to test.
This commit modifies the check_snapshot_queue method so that it can take a
maxloop and delay parameter so that it can be tested.
Add spec tests for check_disk_queue
Previously the check_disk_queue method would execute the loop indefinitely as it
did not have a terminating condition. This made it impossible to test. This
commit modifies the check_disk_queue method so that it can take a maxloop and
delay parameter so that it can be tested.
Add spec tests for check_pool
Previously the check_pool method would execute the loop indefinitely as it did not
have a terminating condition. This made it impossible to test. This commit
modifies the check_pool method so that it can take a maxloop and delay parameter
so that it can be tested.
Add spec tests for execute!
Previously the execute! method would execute the loop indefinitely as it did not
have a terminating condition. This made it impossible to test. This commit
modifies the execute! method so that it can take a maxloop and delay parameter
so that it can be tested.
If a user attempts to start vmpooler using dummy authentication
without setting the environment variable VMPOOLER_DEBUG, the vmpooler
will now refuse to start.
Previously it was difficult to do local development as VMPooler requires an LDAP
service for authentication. This commit adds a dummy authentication provider.
The provider has passes authentication if the username and password are
different, and fails if the username and password are the same. This commit
also updates the documentation in the config YML file.
If `ENV['VMPOOLER_CONFIG']` is defined, it is read in as a YAML
configuration. This allows vmpooler to run in a docker daemon via
`docker run -e VMPOOLER_CONFIG -p 80:4567 -it vmpooler` rather than
embedding a YAML file within the container.
This commit updates vmpooler to ensure clone errors, and other pool manager errors are raised to the parent method. Without this change vmpooler gets stuck after a connection fails during clone operations and will not attempt to clone again.
This commit updates vmpooler to clear the migrations queue at application start time. When the application is shut down it is not considerate of any activities, like migrations, in flight. The result is that when the application is started again any stale entries in vmpooler__migration will be left until manually removed, which can prevent migrations from occurring.
This commit adds retry logic and configurable delays to vsphere helper.
Without this change vmpooler instances that have large numbers of pools
can create enough connections in a short period of time to cause vcenter
issues.
This commit updates vmpooler.migrate metric to send the pool name. Without this change the VM name is sent. Additionally, counts are added for migrate_from_#host and migrate_to_#host in order to allow tracking host migrations, and correlating them with CPU ready time changes.
This commit updates vmpooler to understand how to resolve a situation
where a pending VM does not exist. Without this change a pending VM that
does not exist in vmware inventory gets stuck in the pending state,
preventing the pool from ever reaching its target capacity.
As a part of this change the find_vm method is updated to perform a
light, then heavy search each time find_vm is called and all usage of
find_vm || find_vm_heavy is replaced. This makes find_vm usage
consistent across pool_manager.
Additionally, open_socket method is updated to resolve an incorrect
reference to the host name.
This commit updates pool manager to use a method for opening a socket
instead of opening it directly from check_pending_vm. Support is added
for specifying the domain of the VM to connect to, which lays the
groundwork for doing away with the assumption of having DNS search
domains set for vmpooler to move VMs to the ready state.
Additionally, this commit adds a block to ensure open_socket closes open connections. Without this change sockets are opened to each VM before moving to the ready state, and never explicitly closed.
Also, use open socket for check_ready_vm
Update migrate_vm to make clear when an if block is the end of the line by returning. Use scard instead of smembers.size() for determining migrations in progress.
A method is added to make more clear what's happening when checking if a socket can be opened to a pending VM on port 22. Additionally, the connection appends domain from the configuration, when present, to the VM name so DNS search is not required.
This commit adds a capability to pool_manager to migrate VMs placed in the migrating queue. When a VM is checked out an entry is created in vmpooler__migrating. The existing process for evaluating VM states executes the migrate_vm method for the provided VM, and removes it from the queue. The least used compatible host for the provided VM is selected and, if necessary, a migration to the lesser used host is performed. Migration time and time from the task being queued until completion are both tracked with the redis VM object in 'migration_time' and 'checkout_to_migration'. The migration time is logged in the vmpooler.log, or the VM is reported as not requiring migration. Without this change VMs are not evaluated for checkout at request time.
Add a method to wrap find_vm and find_vm_heavy in order to allow a
single operation to be performed that does both.
This commit also adds support for a configuration setting called
migration_limit that makes migration at checkout optional. Additionally,
logging is added to report a VM parent host when it is checked out. Without this
change vmpooler assumes that migration at checkout is always enabled.
If this setting is not present, or if the setting is 0, then migration
at checkout will be disabled. If the setting is greater than 0 then that
setting will be used to enforce a limit for the number of simultaneous
migrations that will be evaluated.
Documentation of this configuration option is added to the
vmpooler.yaml.example file.
Prior to this the only per-pool statistics that could be extracted from the API
were a list of empty pools in the "status" section of the returned results of
the `/status` endpoint.
This adds a new "pools" section to the '/status' results which lists, for each
pool, the following results:
- The number of ready vms in the pool
- The number of running vms in the pool
- The number of pending vms in the pool
- The maximum size of the pool (as specified in the vmpooler configuration)
Example:
```
{
"boot": {
"duration": {
"average": 163.6,
"min": 65.49,
"max": 830.07,
"total": 247744.71000000002
},
"count": {
"total": 1514
}
# ...
"pools": {
"pool1": {
"ready": 5,
"running": 2,
"pending": 1,
"max": 15
},
"pool2": {
"ready": 0,
"running": 10,
"pending": 0,
"max: 10
}
}
}
```
This includes spec coverage for this change (we could use more specs on `/status` in general); as well as a couple of general spec improvements.
