puppetlabs/vmpooler
2
0
Fork 0
mirror of https://github.com/puppetlabs/vmpooler.git synced 2026-01-26 01:58:41 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

(POOLER-107) Add configuration API endpoint

Browse source 
This commit adds a configuration endpoint to the vmpooler API. Pool
size, and pool template, can be adjusted for pools that are configured
at vmpooler application start time. Pool template changes trigger a pool
refresh, and the new template has delta disks created automatically by
vmpooler.

Additionally, the capability to create template delta disks is added to
the vsphere provider, and this is implemented to ensure that templates
have delta disks created at application start time.

The mechanism used to find template VM objects is simplified to make the flow of logic easier to understand. As an additional benefit, performance of this lookup is improved by using FindByInventoryPath.

A table of contents is added to API.md to ease navigation. Without this change API.md has no table of contents and is difficult to navigate.

Add mutex object for managing pool configuration updates

This commit adds a mutex object for ensuring that pool configuration changes are synchronized across multiple running threads, removing the possibility of two threads attempting to update something at once, without relying on redis data. Without this change this is managed crudely by specifying in redis that a configuration update is taking place. This redis data is left so the REPOPULATE section of _check_pool can still identify when a configuration change is in progress, and prevent a pool from repopulating at that time.

Add wake up event for pool template changes

This commit adds a wake up event to detect pool template changes.
Additionally, GET /config has a template_ready section added to the
output for each pool, which makes clear when a pool is ready to populate
itself.
This commit is contained in:
kirby@puppetlabs.com 2018-05-11 13:42:21 -07:00
parent e781ed258b
commit 9bb4df7d8e
11 changed files with 1393 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

115
docs/API.md
View file

@ -1,8 +1,17 @@
### API
# Table of contents
1. [API](#API)
2. [Token operations](#token)
3. [VM operations](#vmops)
4. [Add disks](#adddisks)
5. [VM snapshots](#vmsnapshots)
6. [Status and metrics](#statusmetrics)
7. [Pool configuration](#poolconfig)
### API <a name="API"></a>
vmpooler provides a REST API for VM management. The following examples use `curl` for communication.
#### Token operations
#### Token operations <a name="token"></a>
Token-based authentication can be used when requesting or modifying VMs. The `/token` route can be used to create, query, or delete tokens. See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), for information on configuring an authentication store to use when performing token operations.
@ -76,7 +85,7 @@ Enter host password for user 'jdoe':
}
```
#### VM operations
#### VM operations <a name="vmops"></a>
##### GET /vm
@ -230,7 +239,7 @@ $ curl -X DELETE --url vmpooler.company.com/api/v1/vm/fq6qlpjlsskycq6
}
```
#### Adding additional disk(s)
#### Adding additional disk(s) <a name="adddisks"></a>
##### POST /vm/&lt;hostname&gt;/disk/&lt;size&gt;
@ -270,7 +279,7 @@ $ curl --url vmpooler.company.com/api/v1/vm/fq6qlpjlsskycq6
````
#### VM snapshots
#### VM snapshots <a name="vmsnapshots"></a>
##### POST /vm/&lt;hostname&gt;/snapshot
@ -322,7 +331,7 @@ $ curl X POST -H X-AUTH-TOKEN:a9znth9dn01t416hrguu56ze37t790bl --url vmpooler.co
}
````
#### Status and metrics
#### Status and metrics <a name="statusmetrics"></a>
##### GET /status
@ -540,3 +549,97 @@ $ curl -G -d 'from=2015-03-10' -d 'to=2015-03-11' --url vmpooler.company.com/api
]
}
```
#### Managing pool configuration via API <a name="poolconfig"></a>
##### GET /config
Returns the running pool configuration
Responses:
* 200 - OK
* 404 - No configuration found
```
$ curl https://vmpooler.company.com/api/v1/config
```
```json
{
"pool_configuration": [
{
"name": "redhat-7-x86_64",
"template": "templates/redhat-7.2-x86_64-0.0.3",
"folder": "vmpooler/redhat-7-x86_64",
"datastore": "stor1",
"size": 1,
"datacenter": "dc1",
"provider": "vsphere",
"capacity": 1,
"major": "redhat",
"template_ready": true
}
],
"status": {
"ok": true
}
}
```
Note: to enable poolsize and pooltemplate config endpoints it is necessary to set 'experimental_features: true' in your vmpooler configuration. A 405 is returned when you attempt to interact with these endpoints when this configuration option is not set.
##### POST /config/poolsize
Change pool size without having to restart the service.
All pool template changes requested must be for pools that exist in the vmpooler configuration running, or a 404 code will be returned
When a pool size is changed due to the configuration posted a 201 status will be returned. When the pool configuration is valid, but will not result in any changes, 200 is returned.
Pool size configuration changes persist through application restarts, and take precedence over a pool size value configured in the pool configuration provided when the application starts. This persistence is dependent on redis. So, if the redis data is lost then the configuration updates revert to those provided at startup at the next application start.
An authentication token is required in order to change pool configuration when authentication is configured.
Responses:
* 200 - No changes required
* 201 - Changes made on at least one pool with changes requested
* 400 - An invalid configuration was provided causing requested changes to fail
* 404 - An unknown error occurred
* 405 - The endpoint is disabled because experimental features are disabled
```
$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"2","debian-7-x86_64":"1"}' --url https://vmpooler.company.com/api/v1/config/poolsize
```
```json
{
"ok": true
}
```
##### POST /config/pooltemplate
Change the template configured for a pool, and replenish the pool with instances built from the new template.
All pool template changes requested must be for pools that exist in the vmpooler configuration running, or a 404 code will be returned
When a pool template is changed due to the configuration posted a 201 status will be returned. When the pool configuration is valid, but will not result in any changes, 200 is returned.
A pool template being updated will cause the following actions, which are logged in vmpooler.log:
* Destroy all instances for the pool template being updated that are in the ready and pending state
* Halt repopulating the pool while creating template deltas for the newly configured template
* Unblock pool population and let the pool replenish with instances based on the newly configured template
Pool template changes persist through application restarts, and take precedence over a pool template configured in the pool configuration provided when the application starts. This persistence is dependent on redis. As a result, if the redis data is lost then the configuration values revert to those provided at startup at the next application start.
An authentication token is required in order to change pool configuration when authentication is configured.
Responses:
* 200 - No changes required
* 201 - Changes made on at least one pool with changes requested
* 400 - An invalid configuration was provided causing requested changes to fail
* 404 - An unknown error occurred
* 405 - The endpoint is disabled because experimental features are disabled
```
$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"templates/debian-7-i386"}' --url https://vmpooler.company.com/api/v1/config/pooltemplate
```
```json
{
"ok": true
}
```

23
lib/vmpooler/api/helpers.rb
View file

@ -378,6 +378,29 @@ module Vmpooler
result
end
def pool_index(pools)
pools_hash = {}
index = 0
for pool in pools
pools_hash[pool['name']] = index
index += 1
end
pools_hash
end
def template_ready?(pool, backend)
prepared_template = backend.hget('vmpooler__template__prepared', pool['name'])
return false if prepared_template.nil?
return true if pool['template'] == prepared_template
return false
end
def is_integer?(x)
Integer(x)
true
rescue
false
end
end
end
end

183
lib/vmpooler/api/v1.rb
View file

@ -120,6 +120,74 @@ module Vmpooler
result
end
def update_pool_size(payload)
result = { 'ok' => false }
pool_index = pool_index(pools)
pools_updated = 0
sync_pool_sizes
payload.each do |poolname, size|
unless pools[pool_index[poolname]]['size'] == size.to_i
pools[pool_index[poolname]]['size'] = size.to_i
backend.hset('vmpooler__config__poolsize', poolname, size)
pools_updated += 1
status 201
end
end
status 200 unless pools_updated > 0
result['ok'] = true
result
end
def update_pool_template(payload)
result = { 'ok' => false }
pool_index = pool_index(pools)
pools_updated = 0
sync_pool_templates
payload.each do |poolname, template|
unless pools[pool_index[poolname]]['template'] == template
pools[pool_index[poolname]]['template'] = template
backend.hset('vmpooler__config__template', poolname, template)
pools_updated += 1
status 201
end
end
status 200 unless pools_updated > 0
result['ok'] = true
result
end
def sync_pool_templates
pool_index = pool_index(pools)
template_configs = backend.hgetall('vmpooler__config__template')
unless template_configs.nil?
template_configs.each do |poolname, template|
if pool_index.include? poolname
unless pools[pool_index[poolname]]['template'] == template
pools[pool_index[poolname]]['template'] = template
end
end
end
end
end
def sync_pool_sizes
pool_index = pool_index(pools)
poolsize_configs = backend.hgetall('vmpooler__config__poolsize')
unless poolsize_configs.nil?
poolsize_configs.each do |poolname, size|
if pool_index.include? poolname
unless pools[pool_index[poolname]]['size'] == size.to_i
pools[pool_index[poolname]]['size'] == size.to_i
end
end
end
end
end
# Provide run-time statistics
#
# Example:
@ -196,6 +264,8 @@ module Vmpooler
}
}
sync_pool_sizes
result[:capacity] = get_capacity_metrics(pools, backend) unless views and not views.include?("capacity")
result[:queue] = get_queue_metrics(pools, backend) unless views and not views.include?("queue")
result[:clone] = get_task_metrics(backend, 'clone', Date.today.to_s) unless views and not views.include?("clone")
@ -502,6 +572,30 @@ module Vmpooler
invalid
end
def invalid_template_or_size(payload)
invalid = []
payload.each do |pool, size|
invalid << pool unless pool_exists?(pool)
unless is_integer?(size)
invalid << pool
next
end
invalid << pool unless Integer(size) >= 0
end
invalid
end
def invalid_template_or_path(payload)
invalid = []
payload.each do |pool, template|
invalid << pool unless pool_exists?(pool)
invalid << pool unless template.include? '/'
invalid << pool if template[0] == '/'
invalid << pool if template[-1] == '/'
end
invalid
end
post "#{api_prefix}/vm/:template/?" do
content_type :json
result = { 'ok' => false }
@ -747,6 +841,95 @@ module Vmpooler
JSON.pretty_generate(result)
end
post "#{api_prefix}/config/poolsize/?" do
content_type :json
result = { 'ok' => false }
if config['experimental_features']
need_token! if Vmpooler::API.settings.config[:auth]
payload = JSON.parse(request.body.read)
if payload
invalid = invalid_template_or_size(payload)
if invalid.empty?
result = update_pool_size(payload)
else
invalid.each do |bad_template|
metrics.increment("config.invalid.#{bad_template}")
end
result[:bad_templates] = invalid
status 400
end
else
metrics.increment('config.invalid.unknown')
status 404
end
else
status 405
end
JSON.pretty_generate(result)
end
post "#{api_prefix}/config/pooltemplate/?" do
content_type :json
result = { 'ok' => false }
if config['experimental_features']
need_token! if Vmpooler::API.settings.config[:auth]
payload = JSON.parse(request.body.read)
if payload
invalid = invalid_template_or_path(payload)
if invalid.empty?
result = update_pool_template(payload)
else
invalid.each do |bad_template|
metrics.increment("config.invalid.#{bad_template}")
end
result[:bad_templates] = invalid
status 400
end
else
metrics.increment('config.invalid.unknown')
status 404
end
else
status 405
end
JSON.pretty_generate(result)
end
get "#{api_prefix}/config/?" do
content_type :json
result = { 'ok' => false }
status 404
if pools
sync_pool_sizes
sync_pool_templates
pool_configuration = []
pools.each do |pool|
pool['template_ready'] = template_ready?(pool, backend)
pool_configuration << pool
end
result = {
pool_configuration: pool_configuration,
status: {
ok: true
}
}
status 200
end
JSON.pretty_generate(result)
end
end
end
end

140
lib/vmpooler/pool_manager.rb
View file

@ -21,6 +21,9 @@ module Vmpooler
# Our thread-tracker object
$threads = {}
# Pool mutex
@reconfigure_pool = {}
end
def config
@ -187,9 +190,9 @@ module Vmpooler
end
end
def move_vm_queue(pool, vm, queue_from, queue_to, msg)
def move_vm_queue(pool, vm, queue_from, queue_to, msg = nil)
$redis.smove("vmpooler__#{queue_from}__#{pool}", "vmpooler__#{queue_to}__#{pool}", vm)
$logger.log('d', "[!] [#{pool}] '#{vm}' #{msg}")
$logger.log('d', "[!] [#{pool}] '#{vm}' #{msg}") if msg
end
# Clone a VM
@ -482,6 +485,10 @@ module Vmpooler
# - Fires when the number of ready VMs changes due to being consumed.
# - Additional options
# :poolname
# :pool_template_change
# - Fires when a template configuration update is requested
# - Additional options
# :poolname
#
def sleep_with_wakeup_events(loop_delay, wakeup_period = 5, options = {})
exit_by = Time.now + loop_delay
@ -492,6 +499,10 @@ module Vmpooler
initial_ready_size = $redis.scard("vmpooler__ready__#{options[:poolname]}")
end
if options[:pool_template_change]
initial_template = $redis.hget('vmpooler__template__prepared', options[:poolname])
end
loop do
sleep(1)
break if time_passed?(:exit_by, exit_by)
@ -505,6 +516,14 @@ module Vmpooler
ready_size = $redis.scard("vmpooler__ready__#{options[:poolname]}")
break unless ready_size == initial_ready_size
end
if options[:pool_template_change]
configured_template = $redis.hget('vmpooler__config__template', options[:poolname])
if configured_template
break unless initial_template == configured_template
end
end
end
break if time_passed?(:exit_by, exit_by)
@ -532,6 +551,7 @@ module Vmpooler
loop_delay = loop_delay_min
provider = get_provider_for_pool(pool['name'])
raise("Could not find provider '#{pool['provider']}") if provider.nil?
sync_pool_template(pool)
loop do
result = _check_pool(pool, provider)
@ -541,7 +561,7 @@ module Vmpooler
loop_delay = (loop_delay * loop_delay_decay).to_i
loop_delay = loop_delay_max if loop_delay > loop_delay_max
end
sleep_with_wakeup_events(loop_delay, loop_delay_min, pool_size_change: true, poolname: pool['name'])
sleep_with_wakeup_events(loop_delay, loop_delay_min, pool_size_change: true, poolname: pool['name'], pool_template_change: true)
unless maxloop.zero?
break if loop_count >= maxloop
@ -555,6 +575,101 @@ module Vmpooler
end
end
def pool_mutex(poolname)
@reconfigure_pool[poolname] || @reconfigure_pool[poolname] = Mutex.new
end
def sync_pool_template(pool)
pool_template = $redis.hget('vmpooler__config__template', pool['name'])
if pool_template
unless pool['template'] == pool_template
pool['template'] = pool_template
end
end
end
def prepare_template(pool, provider)
provider.create_template_delta_disks(pool) if $config[:config]['create_template_delta_disks']
$redis.hset('vmpooler__template__prepared', pool['name'], pool['template'])
end
def evaluate_template(pool, provider)
mutex = pool_mutex(pool['name'])
prepared_template = $redis.hget('vmpooler__template__prepared', pool['name'])
configured_template = $redis.hget('vmpooler__config__template', pool['name'])
return if mutex.locked?
if prepared_template.nil?
mutex.synchronize do
prepare_template(pool, provider)
prepared_template = $redis.hget('vmpooler__template__prepared', pool['name'])
end
end
return if configured_template.nil?
return if configured_template == prepared_template
mutex.synchronize do
update_pool_template(pool, provider, configured_template, prepared_template)
end
end
def drain_pool(poolname)
# Clear a pool of ready and pending instances
if $redis.scard("vmpooler__ready__#{poolname}") > 0
$logger.log('s', "[*] [#{poolname}] removing ready instances")
$redis.smembers("vmpooler__ready__#{poolname}").each do |vm|
move_vm_queue(poolname, vm, 'ready', 'completed')
end
end
if $redis.scard("vmpooler__pending__#{poolname}") > 0
$logger.log('s', "[*] [#{poolname}] removing pending instances")
$redis.smembers("vmpooler__pending__#{poolname}").each do |vm|
move_vm_queue(poolname, vm, 'pending', 'completed')
end
end
end
def update_pool_template(pool, provider, configured_template, prepared_template)
pool['template'] = configured_template
$logger.log('s', "[*] [#{pool['name']}] template updated from #{prepared_template} to #{configured_template}")
# Remove all ready and pending VMs so new instances are created from the new template
drain_pool(pool['name'])
# Prepare template for deployment
$logger.log('s', "[*] [#{pool['name']}] preparing pool template for deployment")
prepare_template(pool, provider)
$logger.log('s', "[*] [#{pool['name']}] is ready for use")
end
def remove_excess_vms(pool, provider, ready, total)
return if total.nil?
return if total == 0
mutex = pool_mutex(pool['name'])
return if mutex.locked?
return unless ready > pool['size']
mutex.synchronize do
difference = ready - pool['size']
difference.times do
next_vm = $redis.spop("vmpooler__ready__#{pool['name']}")
move_vm_queue(pool['name'], next_vm, 'ready', 'completed')
end
if total > ready
$redis.smembers("vmpooler__pending__#{pool['name']}").each do |vm|
move_vm_queue(pool['name'], vm, 'pending', 'completed')
end
end
end
end
def update_pool_size(pool)
mutex = pool_mutex(pool['name'])
return if mutex.locked?
poolsize = $redis.hget('vmpooler__config__poolsize', pool['name'])
return if poolsize.nil?
poolsize = Integer(poolsize)
return if poolsize == pool['size']
mutex.synchronize do
pool['size'] = poolsize
end
end
def _check_pool(pool, provider)
pool_check_response = {
discovered_vms: 0,
@ -683,7 +798,15 @@ module Vmpooler
end
end
# UPDATE TEMPLATE
# Evaluates a pool template to ensure templates are prepared adequately for the configured provider
# If a pool template configuration change is detected then template preparation is repeated for the new template
# Additionally, a pool will drain ready and pending instances
evaluate_template(pool, provider)
# REPOPULATE
# Do not attempt to repopulate a pool while a template is updating
unless pool_mutex(pool['name']).locked?
ready = $redis.scard("vmpooler__ready__#{pool['name']}")
total = $redis.scard("vmpooler__pending__#{pool['name']}") + ready
@ -697,6 +820,11 @@ module Vmpooler
$logger.log('s', "[!] [#{pool['name']}] is empty")
end
# Check to see if a pool size change has been made via the configuration API
# Since check_pool runs in a loop it does not
# otherwise identify this change when running
update_pool_size(pool)
if total < pool['size']
(1..(pool['size'] - total)).each do |_i|
if $redis.get('vmpooler__tasks__clone').to_i < $config[:config]['task_limit'].to_i
@ -712,6 +840,10 @@ module Vmpooler
end
end
end
end
# Remove VMs in excess of the configured pool size
remove_excess_vms(pool, provider, ready, total)
pool_check_response
end
@ -739,6 +871,8 @@ module Vmpooler
$redis.set('vmpooler__tasks__clone', 0)
# Clear out vmpooler__migrations since stale entries may be left after a restart
$redis.del('vmpooler__migration')
# Ensure template deltas are created on each startup
$redis.del('vmpooler__template__prepared')
# Copy vSphere settings to correct location. This happens with older configuration files
if !$config[:vsphere].nil? && ($config[:providers].nil? || $config[:providers][:vsphere].nil?)

8
lib/vmpooler/providers/base.rb
View file

@ -217,6 +217,14 @@ module Vmpooler
def vm_exists?(pool_name, vm_name)
!get_vm(pool_name, vm_name).nil?
end
# inputs
# [Hash] pool : Configuration for the pool
# returns
# nil when successful. Raises error when encountered
def create_template_delta_disks(pool)
raise("#{self.class.name} does not implement create_template_delta_disks")
end
end
end
end

44
lib/vmpooler/providers/vsphere.rb
View file

@ -197,17 +197,10 @@ module Vmpooler
target_cluster_name = get_target_cluster_from_config(pool_name)
target_datacenter_name = get_target_datacenter_from_config(pool_name)
# Extract the template VM name from the full path
raise("Pool #{pool_name} did not specify a full path for the template for the provider #{name}") unless template_path =~ /\//
templatefolders = template_path.split('/')
template_name = templatefolders.pop
# Get the template VM object
raise("Pool #{pool_name} did not specify a full path for the template for the provider #{name}") unless valid_template_path? template_path
# Get the actual objects from vSphere
template_folder_object = find_folder(templatefolders.join('/'), connection, target_datacenter_name)
raise("Pool #{pool_name} specifies a template folder of #{templatefolders.join('/')} which does not exist for the provider #{name}") if template_folder_object.nil?
template_vm_object = template_folder_object.find(template_name)
raise("Pool #{pool_name} specifies a template VM of #{template_name} which does not exist for the provider #{name}") if template_vm_object.nil?
template_vm_object = find_template_vm(pool, connection)
# Annotate with creation time, origin template, etc.
# Add extraconfig options that can be queried by vmtools
@ -933,6 +926,37 @@ module Vmpooler
raise("Cannot create folder #{new_folder}") if folder_object.nil?
folder_object
end
def find_template_vm(pool, connection)
datacenter = get_target_datacenter_from_config(pool['name'])
raise('cannot find datacenter') if datacenter.nil?
propSpecs = {
:entity => self,
:inventoryPath => "#{datacenter}/vm/#{pool['template']}"
}
template_vm_object = connection.searchIndex.FindByInventoryPath(propSpecs)
raise("Pool #{pool['name']} specifies a template VM of #{pool['template']} which does not exist for the provider #{name}") if template_vm_object.nil?
template_vm_object
end
def create_template_delta_disks(pool)
@connection_pool.with_metrics do |pool_object|
connection = ensured_vsphere_connection(pool_object)
template_vm_object = find_template_vm(pool, connection)
template_vm_object.add_delta_disk_layer_on_all_disks
end
end
def valid_template_path?(template)
return false unless template.include?('/')
return false if template[0] == '/'
return false if template[-1] == '/'
return true
end
end
end
end

250
spec/integration/api/v1/config_spec.rb Normal file
View file

@ -0,0 +1,250 @@
require 'spec_helper'
require 'rack/test'
module Vmpooler
class API
module Helpers
def authenticate(auth, username_str, password_str)
username_str == 'admin' and password_str == 's3cr3t'
end
end
end
end
describe Vmpooler::API::V1 do
include Rack::Test::Methods
def app()
Vmpooler::API
end
let(:config) {
{
config: {
'site_name' => 'test pooler',
'vm_lifetime_auth' => 2,
'experimental_features' => true
},
pools: [
{'name' => 'pool1', 'size' => 5, 'template' => 'templates/pool1'},
{'name' => 'pool2', 'size' => 10}
],
statsd: { 'prefix' => 'stats_prefix'},
alias: { 'poolone' => 'pool1' },
pool_names: [ 'pool1', 'pool2', 'poolone' ]
}
}
describe '/config/pooltemplate' do
let(:prefix) { '/api/v1' }
let(:metrics) { Vmpooler::DummyStatsd.new }
let(:current_time) { Time.now }
before(:each) do
redis.flushdb
app.settings.set :config, config
app.settings.set :redis, redis
app.settings.set :metrics, metrics
app.settings.set :config, auth: false
create_token('abcdefghijklmnopqrstuvwxyz012345', 'jdoe', current_time)
end
describe 'POST /config/pooltemplate' do
it 'updates a pool template' do
post "#{prefix}/config/pooltemplate", '{"pool1":"templates/new_template"}'
expect_json(ok = true, http = 201)
expected = { ok: true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails on nonexistent pools' do
post "#{prefix}/config/pooltemplate", '{"poolpoolpool":"templates/newtemplate"}'
expect_json(ok = false, http = 400)
end
it 'updates multiple pools' do
post "#{prefix}/config/pooltemplate", '{"pool1":"templates/new_template","pool2":"templates/new_template2"}'
expect_json(ok = true, http = 201)
expected = { ok: true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails when not all pools exist' do
post "#{prefix}/config/pooltemplate", '{"pool1":"templates/new_template","pool3":"templates/new_template2"}'
expect_json(ok = false, http = 400)
expected = {
ok: false,
bad_templates: ['pool3']
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'returns no changes when the template does not change' do
post "#{prefix}/config/pooltemplate", '{"pool1":"templates/pool1"}'
expect_json(ok = true, http = 200)
expected = { ok: true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails when a invalid template parameter is provided' do
post "#{prefix}/config/pooltemplate", '{"pool1":"template1"}'
expect_json(ok = false, http = 400)
expected = {
ok: false,
bad_templates: ['pool1']
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails when a template starts with /' do
post "#{prefix}/config/pooltemplate", '{"pool1":"/template1"}'
expect_json(ok = false, http = 400)
expected = {
ok: false,
bad_templates: ['pool1']
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails when a template ends with /' do
post "#{prefix}/config/pooltemplate", '{"pool1":"template1/"}'
expect_json(ok = false, http = 400)
expected = {
ok: false,
bad_templates: ['pool1']
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
context 'with experimental features disabled' do
before(:each) do
config[:config]['experimental_features'] = false
end
it 'should return 405' do
post "#{prefix}/config/pooltemplate", '{"pool1":"template/template1"}'
expect_json(ok = false, http = 405)
expected = { ok: false }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
end
describe 'POST /config/poolsize' do
it 'changes a pool size' do
post "#{prefix}/config/poolsize", '{"pool1":"2"}'
expect_json(ok = true, http = 201)
expected = { ok: true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'changes a pool size for multiple pools' do
post "#{prefix}/config/poolsize", '{"pool1":"2","pool2":"2"}'
expect_json(ok = true, http = 201)
expected = { ok: true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails when a specified pool does not exist' do
post "#{prefix}/config/poolsize", '{"pool10":"2"}'
expect_json(ok = false, http = 400)
expected = {
ok: false,
bad_templates: ['pool10']
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'succeeds with 200 when no change is required' do
post "#{prefix}/config/poolsize", '{"pool1":"5"}'
expect_json(ok = true, http = 200)
expected = { ok: true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'succeeds with 201 when at least one pool changes' do
post "#{prefix}/config/poolsize", '{"pool1":"5","pool2":"5"}'
expect_json(ok = true, http = 201)
expected = { ok: true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails when a non-integer value is provided for size' do
post "#{prefix}/config/poolsize", '{"pool1":"four"}'
expect_json(ok = false, http = 400)
expected = {
ok: false,
bad_templates: ['pool1']
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'fails when a negative value is provided for size' do
post "#{prefix}/config/poolsize", '{"pool1":"-1"}'
expect_json(ok = false, http = 400)
expected = {
ok: false,
bad_templates: ['pool1']
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
context 'with experimental features disabled' do
before(:each) do
config[:config]['experimental_features'] = false
end
it 'should return 405' do
post "#{prefix}/config/poolsize", '{"pool1":"1"}'
expect_json(ok = false, http = 405)
expected = { ok: false }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
end
describe 'GET /config' do
let(:prefix) { '/api/v1' }
it 'returns pool configuration when set' do
get "#{prefix}/config"
expect(last_response.header['Content-Type']).to eq('application/json')
result = JSON.parse(last_response.body)
expect(result['pool_configuration']).to eq(config[:pools])
end
end
end
end

70
spec/unit/api/helpers_spec.rb
View file

@ -168,4 +168,74 @@ describe Vmpooler::API::Helpers do
end
end
describe '#pool_index' do
let(:pools) {
[
{
'name' => 'pool1'
},
{
'name' => 'pool2'
}
]
}
it 'should return a hash' do
pools_hash = subject.pool_index(pools)
expect(pools_hash).to be_a(Hash)
end
it 'should return the correct index for each pool' do
pools_hash = subject.pool_index(pools)
expect(pools[pools_hash['pool1']]['name']).to eq('pool1')
expect(pools[pools_hash['pool2']]['name']).to eq('pool2')
end
end
describe '#template_ready?' do
let(:redis) { double('redis') }
let(:template) { 'template/test1' }
let(:poolname) { 'pool1' }
let(:pool) {
{
'name' => poolname,
'template' => template
}
}
it 'returns false when there is no prepared template' do
expect(redis).to receive(:hget).with('vmpooler__template__prepared', poolname).and_return(nil)
expect(subject.template_ready?(pool, redis)).to be false
end
it 'returns true when configured and prepared templates match' do
expect(redis).to receive(:hget).with('vmpooler__template__prepared', poolname).and_return(template)
expect(subject.template_ready?(pool, redis)).to be true
end
it 'returns false when configured and prepared templates do not match' do
expect(redis).to receive(:hget).with('vmpooler__template__prepared', poolname).and_return('template3')
expect(subject.template_ready?(pool, redis)).to be false
end
end
describe '#is_integer?' do
it 'returns true when input is an integer' do
expect(subject.is_integer? 4).to be true
end
it 'returns true when input is a string containing an integer' do
expect(subject.is_integer? '4').to be true
end
it 'returns false when input is a string containing word characters' do
expect(subject.is_integer? 'four').to be false
end
end
end

467
spec/unit/pool_manager_spec.rb
View file

@ -1503,6 +1503,396 @@ EOT
end
end
describe 'sync_pool_template' do
let(:old_template) { 'templates/old-template' }
let(:new_template) { 'templates/new-template' }
let(:config) { YAML.load(<<-EOT
---
:pools:
- name: '#{pool}'
size: 1
template: old_template
EOT
)
}
it 'returns when a template is not set in redis' do
expect(subject.sync_pool_template(config[:pools][0])).to be_nil
end
it 'returns when a template is set and matches the configured template' do
redis.hset('vmpooler__config__template', pool, old_template)
subject.sync_pool_template(config[:pools][0])
expect(config[:pools][0]['template']).to eq(old_template)
end
it 'updates a pool template when the redis provided value is different' do
redis.hset('vmpooler__config__template', pool, new_template)
subject.sync_pool_template(config[:pools][0])
expect(config[:pools][0]['template']).to eq(new_template)
end
end
describe 'pool_mutex' do
it 'should return a mutex' do
expect(subject.pool_mutex(pool)).to be_a(Mutex)
end
it 'should return the same mutex when called twice' do
first = subject.pool_mutex(pool)
second = subject.pool_mutex(pool)
expect(first).to be(second)
end
end
describe 'update_pool_template' do
let(:current_template) { 'templates/pool_template' }
let(:new_template) { 'templates/new_pool_template' }
let(:config) {
YAML.load(<<-EOT
---
:config: {}
:pools:
- name: #{pool}
template: "#{current_template}"
EOT
)
}
let(:poolconfig) { config[:pools][0] }
before(:each) do
allow(logger).to receive(:log)
end
it 'should set the pool template to match the configured template' do
subject.update_pool_template(poolconfig, provider, new_template, current_template)
expect(poolconfig['template']).to eq(new_template)
end
it 'should log that the template is updated' do
expect(logger).to receive(:log).with('s', "[*] [#{pool}] template updated from #{current_template} to #{new_template}")
subject.update_pool_template(poolconfig, provider, new_template, current_template)
end
it 'should run drain_pool' do
expect(subject).to receive(:drain_pool).with(pool)
subject.update_pool_template(poolconfig, provider, new_template, current_template)
end
it 'should log that a template is being prepared' do
expect(logger).to receive(:log).with('s', "[*] [#{pool}] preparing pool template for deployment")
subject.update_pool_template(poolconfig, provider, new_template, current_template)
end
it 'should run prepare_template' do
expect(subject).to receive(:prepare_template).with(poolconfig, provider)
subject.update_pool_template(poolconfig, provider, new_template, current_template)
end
it 'should log that the pool is ready for use' do
expect(logger).to receive(:log).with('s', "[*] [#{pool}] is ready for use")
subject.update_pool_template(poolconfig, provider, new_template, current_template)
end
end
describe 'remove_excess_vms' do
let(:config) {
YAML.load(<<-EOT
---
:pools:
- name: #{pool}
size: 2
EOT
)
}
before(:each) do
expect(subject).not_to be_nil
end
context 'with a 0 total value' do
let(:ready) { 0 }
let(:total) { 0 }
it 'should return nil' do
expect(subject.remove_excess_vms(config[:pools][0], provider, ready, total)).to be_nil
end
end
context 'when the mutex is locked' do
let(:mutex) { Mutex.new }
let(:ready) { 2 }
let(:total) { 3 }
before(:each) do
mutex.lock
expect(subject).to receive(:pool_mutex).with(pool).and_return(mutex)
end
it 'should return nil' do
expect(subject.remove_excess_vms(config[:pools][0], provider, ready, total)).to be_nil
end
end
context 'with a total size less than the pool size' do
let(:ready) { 1 }
let(:total) { 2 }
it 'should return nil' do
expect(subject.remove_excess_vms(config[:pools][0], provider, ready, total)).to be_nil
end
end
context 'with a total size greater than the pool size' do
let(:ready) { 4 }
let(:total) { 4 }
it 'should remove excess ready vms' do
expect(subject).to receive(:move_vm_queue).exactly(2).times
subject.remove_excess_vms(config[:pools][0], provider, ready, total)
end
it 'should remove excess pending vms' do
create_pending_vm(pool,'vm1')
create_pending_vm(pool,'vm2')
create_ready_vm(pool, 'vm3')
create_ready_vm(pool, 'vm4')
create_ready_vm(pool, 'vm5')
expect(subject).to receive(:move_vm_queue).exactly(3).times
subject.remove_excess_vms(config[:pools][0], provider, 3, 5)
end
end
end
describe 'prepare_template' do
let(:config) { YAML.load(<<-EOT
---
:config:
create_template_delta_disks: true
:providers:
:mock:
:pools:
- name: '#{pool}'
size: 1
template: 'templates/pool1'
EOT
)
}
context 'when creating the template delta disks' do
before(:each) do
allow(redis).to receive(:hset)
allow(provider).to receive(:create_template_delta_disks)
end
it 'should run create template delta disks' do
expect(provider).to receive(:create_template_delta_disks).with(config[:pools][0])
subject.prepare_template(config[:pools][0], provider)
end
it 'should mark the template as prepared' do
expect(redis).to receive(:hset).with('vmpooler__template__prepared', pool, config[:pools][0]['template'])
subject.prepare_template(config[:pools][0], provider)
end
end
end
describe 'evaluate_template' do
let(:mutex) { Mutex.new }
let(:current_template) { 'templates/template1' }
let(:new_template) { 'templates/template2' }
let(:config) { YAML.load(<<-EOT
---
:config:
task_limit: 5
:providers:
:mock:
:pools:
- name: '#{pool}'
size: 1
template: '#{current_template}'
EOT
)
}
before(:each) do
allow(redis).to receive(:hget)
expect(subject).to receive(:pool_mutex).with(pool).and_return(mutex)
end
it 'should retreive the prepared template' do
expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).and_return(current_template)
subject.evaluate_template(config[:pools][0], provider)
end
it 'should retrieve the redis configured template' do
expect(redis).to receive(:hget).with('vmpooler__config__template', pool).and_return(new_template)
subject.evaluate_template(config[:pools][0], provider)
end
context 'when the mutex is locked' do
before(:each) do
mutex.lock
end
it 'should return' do
expect(subject.evaluate_template(config[:pools][0], provider)).to be_nil
end
end
context 'when prepared template is nil' do
before(:each) do
expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).and_return(nil)
end
it 'should prepare the template' do
expect(subject).to receive(:prepare_template).with(config[:pools][0], provider)
subject.evaluate_template(config[:pools][0], provider)
end
end
context 'when a new template is requested' do
before(:each) do
expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).and_return(current_template)
expect(redis).to receive(:hget).with('vmpooler__config__template', pool).and_return(new_template)
end
it 'should update the template' do
expect(subject).to receive(:update_pool_template).with(config[:pools][0], provider, new_template, current_template)
subject.evaluate_template(config[:pools][0], provider)
end
end
end
describe 'drain_pool' do
before(:each) do
allow(logger).to receive(:log)
end
context 'with no vms' do
it 'should return nil' do
expect(subject.drain_pool(pool)).to be_nil
end
it 'should not log any messages' do
expect(logger).to_not receive(:log)
subject.drain_pool(pool)
end
it 'should not try to move any vms' do
expect(subject).to_not receive(:move_vm_queue)
subject.drain_pool(pool)
end
end
context 'with ready vms' do
before(:each) do
create_ready_vm(pool, 'vm1')
create_ready_vm(pool, 'vm2')
end
it 'removes the ready instances' do
expect(subject).to receive(:move_vm_queue).twice
subject.drain_pool(pool)
end
it 'logs that ready instances are being removed' do
expect(logger).to receive(:log).with('s', "[*] [#{pool}] removing ready instances")
subject.drain_pool(pool)
end
end
context 'with pending instances' do
before(:each) do
create_pending_vm(pool, 'vm1')
create_pending_vm(pool, 'vm2')
end
it 'removes the pending instances' do
expect(subject).to receive(:move_vm_queue).twice
subject.drain_pool(pool)
end
it 'logs that pending instances are being removed' do
expect(logger).to receive(:log).with('s', "[*] [#{pool}] removing pending instances")
subject.drain_pool(pool)
end
end
end
describe 'update_pool_size' do
let(:newsize) { '3' }
let(:config) {
YAML.load(<<-EOT
---
:pools:
- name: #{pool}
size: 2
EOT
)
}
let(:poolconfig) { config[:pools][0] }
context 'with a locked mutex' do
let(:mutex) { Mutex.new }
before(:each) do
mutex.lock
expect(subject).to receive(:pool_mutex).with(pool).and_return(mutex)
end
it 'should return nil' do
expect(subject.update_pool_size(poolconfig)).to be_nil
end
end
it 'should get the pool size configuration from redis' do
expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool)
subject.update_pool_size(poolconfig)
end
it 'should return when poolsize is not set in redis' do
expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool).and_return(nil)
expect(subject.update_pool_size(poolconfig)).to be_nil
end
it 'should return when no change in configuration is required' do
expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool).and_return('2')
expect(subject.update_pool_size(poolconfig)).to be_nil
end
it 'should update the pool size' do
expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool).and_return(newsize)
subject.update_pool_size(poolconfig)
expect(poolconfig['size']).to eq(Integer(newsize))
end
end
describe "#execute!" do
let(:config) {
YAML.load(<<-EOT
@ -1824,6 +2214,7 @@ EOT
it 'should run startup tasks only once' do
expect(redis).to receive(:set).with('vmpooler__tasks__clone', 0).once
expect(redis).to receive(:del).with('vmpooler__migration').once
expect(redis).to receive(:del).with('vmpooler__template__prepared').once
subject.execute!(maxloop,0)
end
@ -1902,6 +2293,36 @@ EOT
subject.sleep_with_wakeup_events(loop_delay, wakeup_period, wakeup_option)
end
end
describe 'with the pool_template_change wakeup option' do
let(:wakeup_option) {{
:pool_template_change => true,
:poolname => pool
}}
let(:new_template) { 'templates/newtemplate' }
let(:wakeup_period) { -1 } # A negative number forces the wakeup evaluation to always occur
context 'with a template configured' do
before(:each) do
redis.hset('vmpooler__config__template', pool, new_template)
allow(redis).to receive(:hget)
end
it 'should check if a template is configured in redis' do
expect(subject).to receive(:time_passed?).with(:exit_by, Time).and_return(false, true)
expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).once
subject.sleep_with_wakeup_events(loop_delay, wakeup_period, wakeup_option)
end
it 'should sleep until a template change is detected' do
expect(subject).to receive(:sleep).exactly(3).times
expect(redis).to receive(:hget).with('vmpooler__config__template', pool).and_return(nil,nil,new_template)
subject.sleep_with_wakeup_events(loop_delay, wakeup_period, wakeup_option)
end
end
end
end
describe "#check_pool" do
@ -2785,6 +3206,52 @@ EOT
end
end
context 'when a pool size configuration change is detected' do
let(:poolsize) { 2 }
let(:newpoolsize) { 3 }
before(:each) do
config[:pools][0]['size'] = poolsize
redis.hset('vmpooler__config__poolsize', pool, newpoolsize)
expect(provider).to receive(:vms_in_pool).with(pool).and_return([])
end
it 'should change the pool size configuration' do
subject._check_pool(config[:pools][0],provider)
expect(config[:pools][0]['size']).to be(newpoolsize)
end
end
context 'when a pool template is updating' do
let(:poolsize) { 2 }
before(:each) do
redis.hset('vmpooler__config__updating', pool, 1)
expect(provider).to receive(:vms_in_pool).with(pool).and_return([])
end
it 'should not call clone_vm to populate the pool' do
expect(subject).to_not receive(:clone_vm)
subject._check_pool(config[:pools][0],provider)
end
end
context 'when an excess number of ready vms exist' do
before(:each) do
allow(redis).to receive(:scard)
expect(redis).to receive(:scard).with("vmpooler__ready__#{pool}").and_return(1)
expect(redis).to receive(:scard).with("vmpooler__pending__#{pool}").and_return(1)
expect(provider).to receive(:vms_in_pool).with(pool).and_return([])
end
it 'should call remove_excess_vms' do
expect(subject).to receive(:remove_excess_vms).with(config[:pools][0], provider, 1, 2)
subject._check_pool(config[:pools][0],provider)
end
end
context 'export metrics' do
it 'increments metrics for ready queue' do
create_ready_vm(pool,'vm1')

89
spec/unit/providers/vsphere_spec.rb
View file

@ -283,6 +283,7 @@ EOT
let(:clone_vm_task) { mock_RbVmomi_VIM_Task() }
let(:new_vm_object) { mock_RbVmomi_VIM_VirtualMachine({ :name => vmname }) }
let(:new_template_object) { mock_RbVmomi_VIM_VirtualMachine({ :name => vmname }) }
before(:each) do
allow(subject).to receive(:connect_to_vsphere).and_return(connection)
@ -305,19 +306,30 @@ EOT
end
end
context 'Given a template path that does not exist' do
context 'Given a template that starts with /' do
before(:each) do
config[:pools][0]['template'] = 'missing_Templates/pool1'
config[:pools][0]['template'] = '/bad_template'
end
it 'should raise an error' do
expect{ subject.create_vm(poolname, vmname) }.to raise_error(/specifies a template folder of .+ which does not exist/)
expect{ subject.create_vm(poolname, vmname) }.to raise_error(/did not specify a full path for the template/)
end
end
context 'Given a template that ends with /' do
before(:each) do
config[:pools][0]['template'] = 'bad_template/'
end
it 'should raise an error' do
expect{ subject.create_vm(poolname, vmname) }.to raise_error(/did not specify a full path for the template/)
end
end
context 'Given a template VM that does not exist' do
before(:each) do
config[:pools][0]['template'] = 'Templates/missing_template'
expect(subject).to receive(:find_template_vm).and_raise("specifies a template VM of #{vmname} which does not exist")
end
it 'should raise an error' do
@ -327,7 +339,8 @@ EOT
context 'Given a successful creation' do
before(:each) do
template_vm = subject.find_folder('Templates',connection,datacenter_name).find('pool1')
template_vm = new_template_object
allow(subject).to receive(:find_template_vm).and_return(new_template_object)
allow(template_vm).to receive(:CloneVM_Task).and_return(clone_vm_task)
allow(clone_vm_task).to receive(:wait_for_completion).and_return(new_vm_object)
end
@ -339,7 +352,7 @@ EOT
end
it 'should use the appropriate Create_VM spec' do
template_vm = subject.find_folder('Templates',connection,datacenter_name).find('pool1')
template_vm = new_template_object
expect(template_vm).to receive(:CloneVM_Task)
.with(create_vm_spec(vmname,'pool1','datastore0'))
.and_return(clone_vm_task)
@ -3461,5 +3474,71 @@ EOT
end
end
describe 'find_template_vm' do
let(:vm_object) { mock_RbVmomi_VIM_VirtualMachine() }
before(:each) do
allow(connection.searchIndex).to receive(:FindByInventoryPath)
end
it 'should raise an error when the datacenter cannot be found' do
config[:providers][:vsphere]['datacenter'] = nil
expect{ subject.find_template_vm(config[:pools][0],connection) }.to raise_error('cannot find datacenter')
end
it 'should raise an error when the template specified cannot be found' do
expect(connection.searchIndex).to receive(:FindByInventoryPath).and_return(nil)
expect{ subject.find_template_vm(config[:pools][0],connection) }.to raise_error("Pool #{poolname} specifies a template VM of #{config[:pools][0]['template']} which does not exist for the provider vsphere")
end
it 'should return the vm object' do
expect(connection.searchIndex).to receive(:FindByInventoryPath).and_return(vm_object)
subject.find_template_vm(config[:pools][0],connection)
end
end
describe 'valid_template_path?' do
it 'should return true with a valid template path' do
expect(subject.valid_template_path?('test/template')).to eq(true)
end
it 'should return false when no / is found' do
expect(subject.valid_template_path?('testtemplate')).to eq(false)
end
it 'should return false when template path begins with /' do
expect(subject.valid_template_path?('/testtemplate')).to eq(false)
end
it 'should return false when template path ends with /' do
expect(subject.valid_template_path?('testtemplate/')).to eq(false)
end
end
describe 'create_template_delta_disks' do
let(:template_object) { mock_RbVmomi_VIM_VirtualMachine({
:name => vmname,
})
}
before(:each) do
allow(subject).to receive(:connect_to_vsphere).and_return(connection)
end
context 'with a template VM found' do
before(:each) do
expect(subject).to receive(:find_template_vm).and_return(template_object)
end
it 'should reconfigure the VM creating delta disks' do
expect(template_object).to receive(:add_delta_disk_layer_on_all_disks)
subject.create_template_delta_disks(config[:pools][0])
end
end
end
end

6
vmpooler.yaml.example
View file

@ -443,6 +443,11 @@
# The value represents a percentage and applies to both memory and CPU
# (optional; default: 90)
#
# - experimental_features (Only affects API config endpoints)
# Enable experimental API capabilities such as changing pool template and size without application restart
# Expects a boolean value
# (optional; default: false)
#
# Example:
:config:
@ -458,6 +463,7 @@
- 'project'
domain: 'company.com'
prefix: 'poolvm-'
experimental_features: true
# :pools:
#