puppetlabs/vmpooler
2
0
Fork 0
mirror of https://github.com/puppetlabs/vmpooler.git synced 2026-01-26 01:58:41 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

(POOLER-158) Add capability to provision VMs on demand

Browse source 
This change adds a capability to vmpooler to provision instances on
demand. Without this change vmpooler only supports retrieving machines
from pre-provisioned pools.

Additionally, this change refactors redis interactions to reduce round
trips to redis. Specifically, multi and pipelined redis commands are
added where possible to reduce the number of times we are calling redis.

To support the redis refactor the redis interaction has changed to
leveraging a connection pool. In addition to offering multiple
connections for pool manager to use, the redis interactions in pool
manager are now thread safe.

Ready TTL is now a global parameter that can be set as a default for all
pools. A default of 0 has been removed, because this is an unreasonable
default behavior, which would leave a provisioned instance in the pool
indefinitely.

Pool empty messages have been removed when the pool size is set to 0.
Without this change, when a pool was set to a size of 0 the API and pool
manager would both show that a pool is empty.
This commit is contained in:
kirby@puppetlabs.com 2020-04-06 10:52:05 -07:00
parent e9a79cb6db
commit 86e92de4cf
34 changed files with 3247 additions and 1099 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Gemfile
View file

@ -13,6 +13,7 @@ gem 'statsd-ruby', '~> 1.4.0', :require => 'statsd'
gem 'connection_pool', '~> 2.2' gem 'connection_pool', '~> 2.2'
gem 'nokogiri', '~> 1.10' gem 'nokogiri', '~> 1.10'
gem 'spicy-proton', '~> 2.1' gem 'spicy-proton', '~> 2.1'
gem 'concurrent-ruby', '~> 1.1'
group :development do group :development do
gem 'pry' gem 'pry'

6
README.md
View file

@ -7,8 +7,7 @@ vmpooler provides configurable 'pools' of instantly-available (running) virtual
## Usage ## Usage
At [Puppet, Inc.](http://puppet.com) we run acceptance tests on thousands of disposable VMs every day. Dynamic cloning of VM templates initially worked fine for this, but added several seconds to each test run and was unable to account for failed clone tasks. By pushing these operations to a backend service, we were able to both speed up tests and eliminate test failures due to underlying infrastructure failures. At [Puppet, Inc.](http://puppet.com) we run acceptance tests on thousands of disposable VMs every day. Vmpooler manages the lifecycle of these VMs from request through deletion, with options available to pool ready instances, and provision on demand.
## Installation ## Installation
@ -85,7 +84,7 @@ docker run -it vmpooler manager
### docker-compose ### docker-compose
A docker-compose file is provided to support running vmpooler easily via docker-compose. A docker-compose file is provided to support running vmpooler easily via docker-compose. This is useful for development because your local code is used to build the gem used in the docker-compose environment.
``` ```
docker-compose -f docker/docker-compose.yml up docker-compose -f docker/docker-compose.yml up
@ -113,7 +112,6 @@ A dashboard is provided to offer real-time statistics and historical graphs. It
## Command-line Utility ## Command-line Utility
- The [vmpooler_client.py](https://github.com/puppetlabs/vmpooler-client) CLI utility provides easy access to the vmpooler service. The tool is cross-platform and written in Python.
- [vmfloaty](https://github.com/briancain/vmfloaty) is a ruby based CLI tool and scripting library written in ruby. - [vmfloaty](https://github.com/briancain/vmfloaty) is a ruby based CLI tool and scripting library written in ruby.
## Vagrant plugin ## Vagrant plugin

4
bin/vmpooler
View file

@ -7,6 +7,8 @@ config = Vmpooler.config
redis_host = config[:redis]['server'] redis_host = config[:redis]['server']
redis_port = config[:redis]['port'] redis_port = config[:redis]['port']
redis_password = config[:redis]['password'] redis_password = config[:redis]['password']
redis_connection_pool_size = config[:redis]['connection_pool_size']
redis_connection_pool_timeout = config[:redis]['connection_pool_timeout']
logger_file = config[:config]['logfile'] logger_file = config[:config]['logfile']
metrics = Vmpooler.new_metrics(config) metrics = Vmpooler.new_metrics(config)
@ -36,7 +38,7 @@ if torun.include? 'manager'
Vmpooler::PoolManager.new( Vmpooler::PoolManager.new(
config, config,
Vmpooler.new_logger(logger_file), Vmpooler.new_logger(logger_file),
Vmpooler.new_redis(redis_host, redis_port, redis_password), Vmpooler.redis_connection_pool(redis_host, redis_port, redis_password, redis_connection_pool_size, redis_connection_pool_timeout, metrics),
metrics metrics
).execute! ).execute!
end end

4
docker/Dockerfile
View file

@ -8,9 +8,9 @@
# RUN: # RUN:
# docker run -e VMPOOLER_CONFIG -p 80:4567 -it vmpooler # docker run -e VMPOOLER_CONFIG -p 80:4567 -it vmpooler
FROM jruby:9.2.9-jdk FROM jruby:9.2-jdk
ARG vmpooler_version=0.5.0 ARG vmpooler_version=0.11.3
COPY docker/docker-entrypoint.sh /usr/local/bin/ COPY docker/docker-entrypoint.sh /usr/local/bin/

2
docker/Dockerfile_local
View file

@ -8,7 +8,7 @@
# RUN: # RUN:
# docker run -e VMPOOLER_CONFIG -p 80:4567 -it vmpooler # docker run -e VMPOOLER_CONFIG -p 80:4567 -it vmpooler
FROM jruby:9.2.9-jdk FROM jruby:9.2-jdk
COPY docker/docker-entrypoint.sh /usr/local/bin/ COPY docker/docker-entrypoint.sh /usr/local/bin/
COPY ./ ./ COPY ./ ./

2
docker/docker-compose.yml
View file

@ -17,7 +17,7 @@ services:
- VMPOOLER_DEBUG=true # for use of dummy auth - VMPOOLER_DEBUG=true # for use of dummy auth
- VMPOOLER_CONFIG_FILE=/etc/vmpooler/vmpooler.yaml - VMPOOLER_CONFIG_FILE=/etc/vmpooler/vmpooler.yaml
- REDIS_SERVER=redislocal - REDIS_SERVER=redislocal
- LOGFILE=/dev/stdout - LOGFILE=/dev/null
image: vmpooler-local image: vmpooler-local
depends_on: depends_on:
- redislocal - redislocal

89
docs/API.md
View file

@ -6,6 +6,7 @@
5. [VM snapshots](#vmsnapshots) 5. [VM snapshots](#vmsnapshots)
6. [Status and metrics](#statusmetrics) 6. [Status and metrics](#statusmetrics)
7. [Pool configuration](#poolconfig) 7. [Pool configuration](#poolconfig)
8. [Ondemand VM provisioning](#ondemandvm)
### API <a name="API"></a> ### API <a name="API"></a>
@ -799,3 +800,91 @@ $ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"1"}' --
"ok": true "ok": true
} }
``` ```
#### Ondemand VM operations <a name="ondemandvm"></a>
Ondemand VM operations offer a user an option to directly request instances to be allocated for use. This can be very useful when supporting a wide range of images because idle instances can be eliminated.
##### POST /ondemandvm
All instance types requested must match a pool name or alias in the running application configuration, or a 404 code will be returned
When a provisioning request is accepted the API will return an indication that the request is successful. You may then poll /ondemandvm to monitor request status.
An authentication token is required in order to request instances on demand when authentication is configured.
Responses:
* 201 - Provisioning request accepted
* 400 - Payload contains invalid JSON and cannot be parsed
* 403 - Request exceeds the configured per pool maximum
* 404 - A pool was requested, which is not available in the running configuration, or an unknown error occurred.
* 409 - A request of the matching ID has already been created
```
$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"4"}' --url https://vmpooler.example.com/api/v1/ondemandvm
```
```json
{
"ok": true,
"request_id": "e3ff6271-d201-4f31-a315-d17f4e15863a"
}
```
##### GET /ondemandvm
Get the status of an ondemandvm request that has already been posted.
When the request is ready the ready status will change to 'true'.
The number of instances pending vs ready will be reflected in the API response.
Responses:
* 200 - The API request was successful and the status is ok
* 202 - The request is not ready yet
* 404 - The request can not be found, or an unknown error occurred
```
$ curl https://vmpooler.example.com/api/v1/ondemandvm/e3ff6271-d201-4f31-a315-d17f4e15863a
```
```json
{
"ok": true,
"request_id": "e3ff6271-d201-4f31-a315-d17f4e15863a",
"ready": false,
"debian-7-i386": {
"ready": "3",
"pending": "1"
}
}
```
```json
{
"ok": true,
"request_id": "e3ff6271-d201-4f31-a315-d17f4e15863a",
"ready": true,
"debian-7-i386": {
"hostname": [
"vm1",
"vm2",
"vm3",
"vm4"
]
}
}
```
##### DELETE /ondemandvm
Delete a ondemand request
Deleting a ondemand request will delete any instances created for the request and mark the backend data for expiration in two weeks. Any subsequent attempts to retrieve request data will indicate it has been deleted.
Responses:
* 200 - The API request was sucessful. A message will indicate if the request has already been deleted.
* 404 - The request can not be found, or an unknown error occurred.
```
$ curl -X DELETE https://vmpooler.example.com/api/v1/ondemandvm/e3ff6271-d201-4f31-a315-d17f4e15863a
```
```json
{
"ok": true
}
```

37
docs/configuration.md
View file

@ -74,6 +74,16 @@ The prefix to use while storing Graphite data.
The TCP port to communicate with the graphite server. The TCP port to communicate with the graphite server.
(optional; default: 2003) (optional; default: 2003)
### MAX\_ONDEMAND\_INSTANCES\_PER\_REQUEST
The maximum number of instances any individual ondemand request may contain per pool.
(default: 10)
### ONDEMAND\_REQUEST\_TTL
The amount of time (in minutes) to give for a ondemand request to be fulfilled before considering it to have failed.
(default: 5)
## Manager options <a name="manager"></a> ## Manager options <a name="manager"></a>
### TASK\_LIMIT ### TASK\_LIMIT
@ -123,6 +133,11 @@ The target cluster VMs are cloned into (host with least VMs chosen)
How long (in minutes) before marking a clone as 'failed' and retrying. How long (in minutes) before marking a clone as 'failed' and retrying.
(optional; default: 15) (optional; default: 15)
### READY\_TTL
How long (in minutes) a ready VM should stay in the ready queue.
(default: 60)
### MAX\_TRIES ### MAX\_TRIES
Set the max number of times a connection should retry in VM providers. This optional setting allows a user to dial in retry limits to suit your environment. Set the max number of times a connection should retry in VM providers. This optional setting allows a user to dial in retry limits to suit your environment.
@ -130,7 +145,7 @@ Set the max number of times a connection should retry in VM providers. This opti
### RETRY\_FACTOR ### RETRY\_FACTOR
When retrying, each attempt sleeps for the try count * retry_factor. When retrying, each attempt sleeps for the try count * retry\_factor.
Increase this number to lengthen the delay between retry attempts. Increase this number to lengthen the delay between retry attempts.
This is particularly useful for instances with a large number of pools This is particularly useful for instances with a large number of pools
to prevent a thundering herd when retrying connections. to prevent a thundering herd when retrying connections.
@ -183,6 +198,21 @@ The argument can accept a full path to a file, or multiple files comma separated
Expects a string value Expects a string value
(optional) (optional)
### ONDEMAND\_CLONE\_LIMIT
Maximum number of simultaneous clones to perform for ondemand provisioning requests.
(default: 10)
### REDIS\_CONNECTION\_POOL\_SIZE
Maximum number of connections to utilize for the redis connection pool.
(default: 10)
### REDIS\_CONNECTION\_POOL\_TIMEOUT
How long a task should wait (in seconds) for a redis connection when all connections are in use.
(default: 5)
## API options <a name="API"></a> ## API options <a name="API"></a>
### AUTH\_PROVIDER ### AUTH\_PROVIDER
@ -221,3 +251,8 @@ The name of your deployment.
Enable experimental API capabilities such as changing pool template and size without application restart Enable experimental API capabilities such as changing pool template and size without application restart
Expects a boolean value Expects a boolean value
(optional; default: false) (optional; default: false)
### MAX\_LIFETIME\_UPPER\_LIMIT
Specify a maximum lifetime that a VM may be extended to in hours.
(optional)

8
docs/dev-setup.md
View file

@ -1,13 +1,15 @@
# Setting up a vmpooler development environment # Setting up a vmpooler development environment
## Requirements ## Docker is the preferred development environment
The docker compose file is the easiest way to get vmpooler running with any local code changes. The docker compose file expects to find a vmpooler.yaml configuration file in the root vmpooler directory. The file is mapped into the running container for the vmpooler application. This file primarily contains the pools configuration. Nearly all other configuration can be supplied with environment variables.
## Requirements for local installation directly on your system (not recommended)
* Supported on OSX, Windows and Linux * Supported on OSX, Windows and Linux
* Ruby or JRuby * Ruby or JRuby
Note - Ruby 1.x support will be removed so it is best to use more modern ruby versions
Note - It is recommended to user Bundler instead of installing gems into the system repository Note - It is recommended to user Bundler instead of installing gems into the system repository
* A local Redis server * A local Redis server

22
lib/vmpooler.rb
View file

@ -1,6 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
module Vmpooler module Vmpooler
require 'concurrent'
require 'date' require 'date'
require 'json' require 'json'
require 'net/ldap' require 'net/ldap'
@ -58,9 +59,14 @@ module Vmpooler
# Set some configuration defaults # Set some configuration defaults
parsed_config[:config]['task_limit'] = string_to_int(ENV['TASK_LIMIT']) || parsed_config[:config]['task_limit'] || 10 parsed_config[:config]['task_limit'] = string_to_int(ENV['TASK_LIMIT']) || parsed_config[:config]['task_limit'] || 10
parsed_config[:config]['ondemand_clone_limit'] = string_to_int(ENV['ONDEMAND_CLONE_LIMIT']) || parsed_config[:config]['ondemand_clone_limit'] || 10
parsed_config[:config]['max_ondemand_instances_per_request'] = string_to_int(ENV['MAX_ONDEMAND_INSTANCES_PER_REQUEST']) || parsed_config[:config]['max_ondemand_instances_per_request'] || 10
parsed_config[:config]['migration_limit'] = string_to_int(ENV['MIGRATION_LIMIT']) if ENV['MIGRATION_LIMIT'] parsed_config[:config]['migration_limit'] = string_to_int(ENV['MIGRATION_LIMIT']) if ENV['MIGRATION_LIMIT']
parsed_config[:config]['vm_checktime'] = string_to_int(ENV['VM_CHECKTIME']) || parsed_config[:config]['vm_checktime'] || 1 parsed_config[:config]['vm_checktime'] = string_to_int(ENV['VM_CHECKTIME']) || parsed_config[:config]['vm_checktime'] || 1
parsed_config[:config]['vm_lifetime'] = string_to_int(ENV['VM_LIFETIME']) || parsed_config[:config]['vm_lifetime'] || 24 parsed_config[:config]['vm_lifetime'] = string_to_int(ENV['VM_LIFETIME']) || parsed_config[:config]['vm_lifetime'] || 24
parsed_config[:config]['max_lifetime_upper_limit'] = string_to_int(ENV['MAX_LIFETIME_UPPER_LIMIT']) || parsed_config[:config]['max_lifetime_upper_limit']
parsed_config[:config]['ready_ttl'] = string_to_int(ENV['READY_TTL']) || parsed_config[:config]['ready_ttl'] || 60
parsed_config[:config]['ondemand_request_ttl'] = string_to_int(ENV['ONDEMAND_REQUEST_TTL']) || parsed_config[:config]['ondemand_request_ttl'] || 5
parsed_config[:config]['prefix'] = ENV['PREFIX'] || parsed_config[:config]['prefix'] || '' parsed_config[:config]['prefix'] = ENV['PREFIX'] || parsed_config[:config]['prefix'] || ''
parsed_config[:config]['logfile'] = ENV['LOGFILE'] if ENV['LOGFILE'] parsed_config[:config]['logfile'] = ENV['LOGFILE'] if ENV['LOGFILE']
@ -84,6 +90,8 @@ module Vmpooler
parsed_config[:redis]['port'] = string_to_int(ENV['REDIS_PORT']) if ENV['REDIS_PORT'] parsed_config[:redis]['port'] = string_to_int(ENV['REDIS_PORT']) if ENV['REDIS_PORT']
parsed_config[:redis]['password'] = ENV['REDIS_PASSWORD'] if ENV['REDIS_PASSWORD'] parsed_config[:redis]['password'] = ENV['REDIS_PASSWORD'] if ENV['REDIS_PASSWORD']
parsed_config[:redis]['data_ttl'] = string_to_int(ENV['REDIS_DATA_TTL']) || parsed_config[:redis]['data_ttl'] || 168 parsed_config[:redis]['data_ttl'] = string_to_int(ENV['REDIS_DATA_TTL']) || parsed_config[:redis]['data_ttl'] || 168
parsed_config[:redis]['connection_pool_size'] = string_to_int(ENV['REDIS_CONNECTION_POOL_SIZE']) || parsed_config[:redis]['connection_pool_size'] || 10
parsed_config[:redis]['connection_pool_timeout'] = string_to_int(ENV['REDIS_CONNECTION_POOL_TIMEOUT']) || parsed_config[:redis]['connection_pool_timeout'] || 5
parsed_config[:statsd] = parsed_config[:statsd] || {} if ENV['STATSD_SERVER'] parsed_config[:statsd] = parsed_config[:statsd] || {} if ENV['STATSD_SERVER']
parsed_config[:statsd]['server'] = ENV['STATSD_SERVER'] if ENV['STATSD_SERVER'] parsed_config[:statsd]['server'] = ENV['STATSD_SERVER'] if ENV['STATSD_SERVER']
@ -117,6 +125,7 @@ module Vmpooler
parsed_config[:pools].each do |pool| parsed_config[:pools].each do |pool|
parsed_config[:pool_names] << pool['name'] parsed_config[:pool_names] << pool['name']
pool['ready_ttl'] ||= parsed_config[:config]['ready_ttl']
if pool['alias'] if pool['alias']
if pool['alias'].is_a?(Array) if pool['alias'].is_a?(Array)
pool['alias'].each do |pool_alias| pool['alias'].each do |pool_alias|
@ -154,6 +163,19 @@ module Vmpooler
pools pools
end end
def self.redis_connection_pool(host, port, password, size, timeout, metrics)
Vmpooler::PoolManager::GenericConnectionPool.new(
metrics: metrics,
metric_prefix: 'redis_connection_pool',
size: size,
timeout: timeout
) do
connection = Concurrent::Hash.new
redis = new_redis(host, port, password)
connection['connection'] = redis
end
end
def self.new_redis(host = 'localhost', port = nil, password = nil) def self.new_redis(host = 'localhost', port = nil, password = nil)
Redis.new(host: host, port: port, password: password) Redis.new(host: host, port: port, password: password)
end end

2
lib/vmpooler/api/helpers.rb
View file

@ -238,7 +238,7 @@ module Vmpooler
queue[:running] = get_total_across_pools_redis_scard(pools, 'vmpooler__running__', backend) queue[:running] = get_total_across_pools_redis_scard(pools, 'vmpooler__running__', backend)
queue[:completed] = get_total_across_pools_redis_scard(pools, 'vmpooler__completed__', backend) queue[:completed] = get_total_across_pools_redis_scard(pools, 'vmpooler__completed__', backend)
queue[:cloning] = backend.get('vmpooler__tasks__clone').to_i queue[:cloning] = backend.get('vmpooler__tasks__clone').to_i + backend.get('vmpooler__tasks__ondemandclone').to_i
queue[:booting] = queue[:pending].to_i - queue[:cloning].to_i queue[:booting] = queue[:pending].to_i - queue[:cloning].to_i
queue[:booting] = 0 if queue[:booting] < 0 queue[:booting] = 0 if queue[:booting] < 0
queue[:total] = queue[:pending].to_i + queue[:ready].to_i + queue[:running].to_i + queue[:completed].to_i queue[:total] = queue[:pending].to_i + queue[:ready].to_i + queue[:running].to_i + queue[:completed].to_i

269
lib/vmpooler/api/v1.rb
View file

@ -42,6 +42,68 @@ module Vmpooler
Vmpooler::API.settings.checkoutlock Vmpooler::API.settings.checkoutlock
end end
def get_template_aliases(template)
result = []
aliases = Vmpooler::API.settings.config[:alias]
if aliases
result += aliases[template] if aliases[template].is_a?(Array)
template_backends << aliases[template] if aliases[template].is_a?(String)
end
result
end
def get_pool_weights(template_backends)
pool_index = pool_index(pools)
weighted_pools = {}
template_backends.each do |t|
next unless pool_index.key? t
index = pool_index[t]
clone_target = pools[index]['clone_target'] || config['clone_target']
next unless config.key?('backend_weight')
weight = config['backend_weight'][clone_target]
if weight
weighted_pools[t] = weight
end
end
weighted_pools
end
def count_selection(selection)
result = {}
selection.uniq.each do |poolname|
result[poolname] = selection.count(poolname)
end
result
end
def evaluate_template_aliases(template, count)
template_backends = []
template_backends << template if backend.sismember('vmpooler__pools', template)
selection = []
aliases = get_template_aliases(template)
if aliases
template_backends += aliases
weighted_pools = get_pool_weights(template_backends)
pickup = Pickup.new(weighted_pools) if weighted_pools.count == template_backends.count
count.to_i.times do
if pickup
selection << pickup.pick
else
selection << template_backends.sample
end
end
else
count.to_i.times do
selection << template
end
end
count_selection(selection)
end
def fetch_single_vm(template) def fetch_single_vm(template)
template_backends = [template] template_backends = [template]
aliases = Vmpooler::API.settings.config[:alias] aliases = Vmpooler::API.settings.config[:alias]
@ -245,23 +307,19 @@ module Vmpooler
pool_index = pool_index(pools) pool_index = pool_index(pools)
template_configs = backend.hgetall('vmpooler__config__template') template_configs = backend.hgetall('vmpooler__config__template')
template_configs&.each do |poolname, template| template_configs&.each do |poolname, template|
if pool_index.include? poolname next unless pool_index.include? poolname
unless pools[pool_index[poolname]]['template'] == template
pools[pool_index[poolname]]['template'] = template pools[pool_index[poolname]]['template'] = template
end end
end end
end
end
def sync_pool_sizes def sync_pool_sizes
pool_index = pool_index(pools) pool_index = pool_index(pools)
poolsize_configs = backend.hgetall('vmpooler__config__poolsize') poolsize_configs = backend.hgetall('vmpooler__config__poolsize')
poolsize_configs&.each do |poolname, size| poolsize_configs&.each do |poolname, size|
if pool_index.include? poolname next unless pool_index.include? poolname
unless pools[pool_index[poolname]]['size'] == size.to_i
pools[pool_index[poolname]]['size'] == size.to_i pools[pool_index[poolname]]['size'] = size.to_i
end
end
end end
end end
@ -269,12 +327,67 @@ module Vmpooler
pool_index = pool_index(pools) pool_index = pool_index(pools)
clone_target_configs = backend.hgetall('vmpooler__config__clone_target') clone_target_configs = backend.hgetall('vmpooler__config__clone_target')
clone_target_configs&.each do |poolname, clone_target| clone_target_configs&.each do |poolname, clone_target|
if pool_index.include? poolname next unless pool_index.include? poolname
unless pools[pool_index[poolname]]['clone_target'] == clone_target
pools[pool_index[poolname]]['clone_target'] == clone_target pools[pool_index[poolname]]['clone_target'] = clone_target
end end
end end
def too_many_requested?(payload)
payload&.each do |_poolname, count|
next unless count.to_i > config['max_ondemand_instances_per_request']
return true
end end
false
end
def generate_ondemand_request(payload)
result = { 'ok': false }
requested_instances = payload.reject { |k, _v| k == 'request_id' }
if too_many_requested?(requested_instances)
result['message'] = "requested amount of instances exceeds the maximum #{config['max_ondemand_instances_per_request']}"
status 403
return result
end
score = Time.now.to_i
request_id = payload['request_id']
request_id ||= generate_request_id
result['request_id'] = request_id
if backend.exists("vmpooler__odrequest__#{request_id}")
result['message'] = "request_id '#{request_id}' has already been created"
status 409
return result
end
status 201
platforms_with_aliases = []
requested_instances.each do |poolname, count|
selection = evaluate_template_aliases(poolname, count)
selection.map { |selected_pool, selected_pool_count| platforms_with_aliases << "#{poolname}:#{selected_pool}:#{selected_pool_count}" }
end
platforms_string = platforms_with_aliases.join(',')
return result unless backend.zadd('vmpooler__provisioning__request', score, request_id)
backend.hset("vmpooler__odrequest__#{request_id}", 'requested', platforms_string)
if Vmpooler::API.settings.config[:auth] and has_token?
backend.hset("vmpooler__odrequest__#{request_id}", 'token:token', request.env['HTTP_X_AUTH_TOKEN'])
backend.hset("vmpooler__odrequest__#{request_id}", 'token:user',
backend.hget('vmpooler__token__' + request.env['HTTP_X_AUTH_TOKEN'], 'user'))
end
result['domain'] = config['domain'] if config['domain']
result[:ok] = true
result
end
def generate_request_id
SecureRandom.uuid
end end
get '/' do get '/' do
@ -395,7 +508,7 @@ module Vmpooler
end end
# for backwards compatibility, include separate "empty" stats in "status" block # for backwards compatibility, include separate "empty" stats in "status" block
if ready == 0 if ready == 0 && max != 0
result[:status][:empty] ||= [] result[:status][:empty] ||= []
result[:status][:empty].push(pool['name']) result[:status][:empty].push(pool['name'])
@ -689,6 +802,61 @@ module Vmpooler
JSON.pretty_generate(result) JSON.pretty_generate(result)
end end
post "#{api_prefix}/ondemandvm/?" do
content_type :json
need_token! if Vmpooler::API.settings.config[:auth]
result = { 'ok' => false }
begin
payload = JSON.parse(request.body.read)
if payload
invalid = invalid_templates(payload.reject { |k, _v| k == 'request_id' })
if invalid.empty?
result = generate_ondemand_request(payload)
else
result[:bad_templates] = invalid
invalid.each do |bad_template|
metrics.increment('ondemandrequest.invalid.' + bad_template)
end
status 404
end
else
metrics.increment('ondemandrequest.invalid.unknown')
status 404
end
rescue JSON::ParserError
status 400
result = {
'ok' => false,
'message' => 'JSON payload could not be parsed'
}
end
JSON.pretty_generate(result)
end
get "#{api_prefix}/ondemandvm/:requestid/?" do
content_type :json
status 404
result = check_ondemand_request(params[:requestid])
JSON.pretty_generate(result)
end
delete "#{api_prefix}/ondemandvm/:requestid/?" do
content_type :json
need_token! if Vmpooler::API.settings.config[:auth]
status 404
result = delete_ondemand_request(params[:requestid])
JSON.pretty_generate(result)
end
post "#{api_prefix}/vm/?" do post "#{api_prefix}/vm/?" do
content_type :json content_type :json
result = { 'ok' => false } result = { 'ok' => false }
@ -764,6 +932,78 @@ module Vmpooler
invalid invalid
end end
def check_ondemand_request(request_id)
result = { 'ok' => false }
request_hash = backend.hgetall("vmpooler__odrequest__#{request_id}")
if request_hash.empty?
result['message'] = "no request found for request_id '#{request_id}'"
return result
end
result['request_id'] = request_id
result['ready'] = false
result['ok'] = true
status 202
if request_hash['status'] == 'ready'
result['ready'] = true
platform_parts = request_hash['requested'].split(',')
platform_parts.each do |platform|
pool_alias, pool, _count = platform.split(':')
instances = backend.smembers("vmpooler__#{request_id}__#{pool_alias}__#{pool}")
result[pool_alias] = { 'hostname': instances }
end
result['domain'] = config['domain'] if config['domain']
status 200
elsif request_hash['status'] == 'failed'
result['message'] = "The request failed to provision instances within the configured ondemand_request_ttl '#{config['ondemand_request_ttl']}'"
status 200
elsif request_hash['status'] == 'deleted'
result['message'] = 'The request has been deleted'
status 200
else
platform_parts = request_hash['requested'].split(',')
platform_parts.each do |platform|
pool_alias, pool, count = platform.split(':')
instance_count = backend.scard("vmpooler__#{request_id}__#{pool_alias}__#{pool}")
result[pool_alias] = {
'ready': instance_count.to_s,
'pending': (count.to_i - instance_count.to_i).to_s
}
end
end
result
end
def delete_ondemand_request(request_id)
result = { 'ok' => false }
platforms = backend.hget("vmpooler__odrequest__#{request_id}", 'requested')
unless platforms
result['message'] = "no request found for request_id '#{request_id}'"
return result
end
if backend.hget("vmpooler__odrequest__#{request_id}", 'status') == 'deleted'
result['message'] = 'the request has already been deleted'
else
backend.hset("vmpooler__odrequest__#{request_id}", 'status', 'deleted')
platforms.split(',').each do |platform|
pool_alias, pool, _count = platform.split(':')
backend.smembers("vmpooler__#{request_id}__#{pool_alias}__#{pool}")&.each do |vm|
backend.smove("vmpooler__running__#{pool}", "vmpooler__completed__#{pool}", vm)
end
backend.del("vmpooler__#{request_id}__#{pool_alias}__#{pool}")
end
backend.expire("vmpooler__odrequest__#{request_id}", 129_600_0)
end
status 200
result['ok'] = true
result
end
post "#{api_prefix}/vm/:template/?" do post "#{api_prefix}/vm/:template/?" do
content_type :json content_type :json
result = { 'ok' => false } result = { 'ok' => false }
@ -923,6 +1163,7 @@ module Vmpooler
unless arg.to_i > 0 unless arg.to_i > 0
failure.push("You provided a lifetime (#{arg}) but you must provide a positive number.") failure.push("You provided a lifetime (#{arg}) but you must provide a positive number.")
end end
when 'tags' when 'tags'
unless arg.is_a?(Hash) unless arg.is_a?(Hash)
failure.push("You provided tags (#{arg}) as something other than a hash.") failure.push("You provided tags (#{arg}) as something other than a hash.")
@ -1047,7 +1288,7 @@ module Vmpooler
invalid.each do |bad_template| invalid.each do |bad_template|
metrics.increment("config.invalid.#{bad_template}") metrics.increment("config.invalid.#{bad_template}")
end end
result[:bad_templates] = invalid result[:not_configured] = invalid
status 400 status 400
end end
else else

18
lib/vmpooler/generic_connection_pool.rb
View file

@ -15,8 +15,6 @@ module Vmpooler
@metric_prefix = 'connectionpool' if @metric_prefix.nil? || @metric_prefix == '' @metric_prefix = 'connectionpool' if @metric_prefix.nil? || @metric_prefix == ''
end end
if Thread.respond_to?(:handle_interrupt)
# MRI
def with_metrics(options = {}) def with_metrics(options = {})
Thread.handle_interrupt(Exception => :never) do Thread.handle_interrupt(Exception => :never) do
start = Time.now start = Time.now
@ -34,22 +32,6 @@ module Vmpooler
end end
end end
end end
else
# jruby 1.7.x
def with_metrics(options = {})
start = Time.now
conn = checkout(options)
timespan_ms = ((Time.now - start) * 1000).to_i
@metrics&.gauge(@metric_prefix + '.available', @available.length)
@metrics&.timing(@metric_prefix + '.waited', timespan_ms)
begin
yield conn
ensure
checkin
@metrics&.gauge(@metric_prefix + '.available', @available.length)
end
end
end
end end
end end
end end

738
lib/vmpooler/pool_manager.rb
View file

File diff suppressed because it is too large Load diff

3
lib/vmpooler/providers/base.rb
View file

@ -14,10 +14,11 @@ module Vmpooler
# Provider options passed in during initialization # Provider options passed in during initialization
attr_reader :provider_options attr_reader :provider_options
def initialize(config, logger, metrics, name, options) def initialize(config, logger, metrics, redis_connection_pool, name, options)
@config = config @config = config
@logger = logger @logger = logger
@metrics = metrics @metrics = metrics
@redis = redis_connection_pool
@provider_name = name @provider_name = name
# Ensure that there is not a nil provider configuration # Ensure that there is not a nil provider configuration

4
lib/vmpooler/providers/dummy.rb
View file

@ -9,8 +9,8 @@ module Vmpooler
class Dummy < Vmpooler::PoolManager::Provider::Base class Dummy < Vmpooler::PoolManager::Provider::Base
# Fake VM Provider for testing # Fake VM Provider for testing
def initialize(config, logger, metrics, name, options) def initialize(config, logger, metrics, redis_connection_pool, name, options)
super(config, logger, metrics, name, options) super(config, logger, metrics, redis_connection_pool, name, options)
dummyfilename = provider_config['filename'] dummyfilename = provider_config['filename']
# This initial_state option is only intended to be used by spec tests # This initial_state option is only intended to be used by spec tests

49
lib/vmpooler/providers/vsphere.rb
View file

@ -9,8 +9,8 @@ module Vmpooler
# The connection_pool method is normally used only for testing # The connection_pool method is normally used only for testing
attr_reader :connection_pool attr_reader :connection_pool
def initialize(config, logger, metrics, name, options) def initialize(config, logger, metrics, redis_connection_pool, name, options)
super(config, logger, metrics, name, options) super(config, logger, metrics, redis_connection_pool, name, options)
task_limit = global_config[:config].nil? || global_config[:config]['task_limit'].nil? ? 10 : global_config[:config]['task_limit'].to_i task_limit = global_config[:config].nil? || global_config[:config]['task_limit'].nil? ? 10 : global_config[:config]['task_limit'].to_i
# The default connection pool size is: # The default connection pool size is:
@ -39,6 +39,7 @@ module Vmpooler
end end
@provider_hosts = {} @provider_hosts = {}
@provider_hosts_lock = Mutex.new @provider_hosts_lock = Mutex.new
@redis = redis_connection_pool
end end
# name of the provider class # name of the provider class
@ -59,12 +60,16 @@ module Vmpooler
def destroy_vm_and_log(vm_name, vm_object, pool, data_ttl) def destroy_vm_and_log(vm_name, vm_object, pool, data_ttl)
try = 0 if try.nil? try = 0 if try.nil?
max_tries = 3 max_tries = 3
$redis.srem("vmpooler__completed__#{pool}", vm_name) @redis.with_metrics do |redis|
$redis.hdel("vmpooler__active__#{pool}", vm_name) redis.multi
$redis.hset("vmpooler__vm__#{vm_name}", 'destroy', Time.now) redis.srem("vmpooler__completed__#{pool}", vm_name)
redis.hdel("vmpooler__active__#{pool}", vm_name)
redis.hset("vmpooler__vm__#{vm_name}", 'destroy', Time.now)
# Auto-expire metadata key # Auto-expire metadata key
$redis.expire('vmpooler__vm__' + vm_name, (data_ttl * 60 * 60)) redis.expire('vmpooler__vm__' + vm_name, (data_ttl * 60 * 60))
redis.exec
end
start = Time.now start = Time.now
@ -968,9 +973,10 @@ module Vmpooler
begin begin
connection = ensured_vsphere_connection(pool_object) connection = ensured_vsphere_connection(pool_object)
vm_hash = get_vm_details(pool_name, vm_name, connection) vm_hash = get_vm_details(pool_name, vm_name, connection)
$redis.hset("vmpooler__vm__#{vm_name}", 'host', vm_hash['host_name']) @redis.with_metrics do |redis|
redis.hset("vmpooler__vm__#{vm_name}", 'host', vm_hash['host_name'])
migration_count = redis.scard('vmpooler__migration')
migration_limit = @config[:config]['migration_limit'] if @config[:config].key?('migration_limit') migration_limit = @config[:config]['migration_limit'] if @config[:config].key?('migration_limit')
migration_count = $redis.scard('vmpooler__migration')
if migration_enabled? @config if migration_enabled? @config
if migration_count >= migration_limit if migration_count >= migration_limit
logger.log('s', "[ ] [#{pool_name}] '#{vm_name}' is running on #{vm_hash['host_name']}. No migration will be evaluated since the migration_limit has been reached") logger.log('s', "[ ] [#{pool_name}] '#{vm_name}' is running on #{vm_hash['host_name']}. No migration will be evaluated since the migration_limit has been reached")
@ -985,6 +991,7 @@ module Vmpooler
else else
logger.log('s', "[ ] [#{pool_name}] '#{vm_name}' is running on #{vm_hash['host_name']}") logger.log('s', "[ ] [#{pool_name}] '#{vm_name}' is running on #{vm_hash['host_name']}")
end end
end
rescue StandardError rescue StandardError
logger.log('s', "[!] [#{pool_name}] '#{vm_name}' is running on #{vm_hash['host_name']}") logger.log('s', "[!] [#{pool_name}] '#{vm_name}' is running on #{vm_hash['host_name']}")
raise raise
@ -993,15 +1000,23 @@ module Vmpooler
end end
def migrate_vm_to_new_host(pool_name, vm_name, vm_hash, connection) def migrate_vm_to_new_host(pool_name, vm_name, vm_hash, connection)
$redis.sadd('vmpooler__migration', vm_name) @redis.with_metrics do |redis|
redis.sadd('vmpooler__migration', vm_name)
end
target_host_name = select_next_host(pool_name, @provider_hosts, vm_hash['architecture']) target_host_name = select_next_host(pool_name, @provider_hosts, vm_hash['architecture'])
target_host_object = find_host_by_dnsname(connection, target_host_name) target_host_object = find_host_by_dnsname(connection, target_host_name)
finish = migrate_vm_and_record_timing(pool_name, vm_name, vm_hash, target_host_object, target_host_name) finish = migrate_vm_and_record_timing(pool_name, vm_name, vm_hash, target_host_object, target_host_name)
$redis.hset("vmpooler__vm__#{vm_name}", 'host', target_host_name) @redis.with_metrics do |redis|
$redis.hset("vmpooler__vm__#{vm_name}", 'migrated', true) redis.multi
redis.hset("vmpooler__vm__#{vm_name}", 'host', target_host_name)
redis.hset("vmpooler__vm__#{vm_name}", 'migrated', true)
redis.exec
end
logger.log('s', "[>] [#{pool_name}] '#{vm_name}' migrated from #{vm_hash['host_name']} to #{target_host_name} in #{finish} seconds") logger.log('s', "[>] [#{pool_name}] '#{vm_name}' migrated from #{vm_hash['host_name']} to #{target_host_name} in #{finish} seconds")
ensure ensure
$redis.srem('vmpooler__migration', vm_name) @redis.with_metrics do |redis|
redis.srem('vmpooler__migration', vm_name)
end
end end
def migrate_vm_and_record_timing(pool_name, vm_name, vm_hash, target_host_object, dest_host_name) def migrate_vm_and_record_timing(pool_name, vm_name, vm_hash, target_host_object, dest_host_name)
@ -1011,9 +1026,13 @@ module Vmpooler
metrics.timing("migrate.#{pool_name}", finish) metrics.timing("migrate.#{pool_name}", finish)
metrics.increment("migrate_from.#{vm_hash['host_name']}") metrics.increment("migrate_from.#{vm_hash['host_name']}")
metrics.increment("migrate_to.#{dest_host_name}") metrics.increment("migrate_to.#{dest_host_name}")
checkout_to_migration = format('%<time>.2f', time: Time.now - Time.parse($redis.hget("vmpooler__vm__#{vm_name}", 'checkout'))) @redis.with_metrics do |redis|
$redis.hset("vmpooler__vm__#{vm_name}", 'migration_time', finish) checkout_to_migration = format('%<time>.2f', time: Time.now - Time.parse(redis.hget("vmpooler__vm__#{vm_name}", 'checkout')))
$redis.hset("vmpooler__vm__#{vm_name}", 'checkout_to_migration', checkout_to_migration) redis.multi
redis.hset("vmpooler__vm__#{vm_name}", 'migration_time', finish)
redis.hset("vmpooler__vm__#{vm_name}", 'checkout_to_migration', checkout_to_migration)
redis.exec
end
finish finish
end end

2
spec/fixtures/vmpooler.yaml vendored
View file

@ -36,6 +36,8 @@
- name: 'pool01' - name: 'pool01'
size: 5 size: 5
provider: dummy provider: dummy
ready_ttl: 5
- name: 'pool02' - name: 'pool02'
size: 5 size: 5
provider: dummy provider: dummy
ready_ttl: 5

2
spec/fixtures/vmpooler2.yaml vendored
View file

@ -36,6 +36,8 @@
- name: 'pool03' - name: 'pool03'
size: 5 size: 5
provider: dummy provider: dummy
ready_ttl: 5
- name: 'pool04' - name: 'pool04'
size: 5 size: 5
provider: dummy provider: dummy
ready_ttl: 5

92
spec/helpers.rb
View file

@ -40,97 +40,115 @@ def token_exists?(token)
result && !result.empty? result && !result.empty?
end end
def create_ready_vm(template, name, token = nil) def create_ready_vm(template, name, redis, token = nil)
create_vm(name, token) create_vm(name, redis, token)
redis.sadd("vmpooler__ready__#{template}", name) redis.sadd("vmpooler__ready__#{template}", name)
redis.hset("vmpooler__vm__#{name}", "template", template) redis.hset("vmpooler__vm__#{name}", "template", template)
end end
def create_running_vm(template, name, token = nil, user = nil) def create_running_vm(template, name, redis, token = nil, user = nil)
create_vm(name, token, nil, user) create_vm(name, redis, token, user)
redis.sadd("vmpooler__running__#{template}", name) redis.sadd("vmpooler__running__#{template}", name)
redis.hset("vmpooler__vm__#{name}", 'template', template) redis.hset("vmpooler__vm__#{name}", 'template', template)
redis.hset("vmpooler__vm__#{name}", 'checkout', Time.now) redis.hset("vmpooler__vm__#{name}", 'checkout', Time.now)
redis.hset("vmpooler__vm__#{name}", 'host', 'host1') redis.hset("vmpooler__vm__#{name}", 'host', 'host1')
end end
def create_pending_vm(template, name, token = nil) def create_pending_vm(template, name, redis, token = nil)
create_vm(name, token) create_vm(name, redis, token)
redis.sadd("vmpooler__pending__#{template}", name) redis.sadd("vmpooler__pending__#{template}", name)
redis.hset("vmpooler__vm__#{name}", "template", template) redis.hset("vmpooler__vm__#{name}", "template", template)
end end
def create_vm(name, token = nil, redis_handle = nil, user = nil) def create_vm(name, redis, token = nil, user = nil)
redis_db = redis_handle ? redis_handle : redis redis.hset("vmpooler__vm__#{name}", 'checkout', Time.now)
redis_db.hset("vmpooler__vm__#{name}", 'checkout', Time.now) redis.hset("vmpooler__vm__#{name}", 'clone', Time.now)
redis_db.hset("vmpooler__vm__#{name}", 'token:token', token) if token redis.hset("vmpooler__vm__#{name}", 'token:token', token) if token
redis_db.hset("vmpooler__vm__#{name}", 'token:user', user) if user redis.hset("vmpooler__vm__#{name}", 'token:user', user) if user
end end
def create_completed_vm(name, pool, active = false, redis_handle = nil) def create_completed_vm(name, pool, redis, active = false)
redis_db = redis_handle ? redis_handle : redis redis.sadd("vmpooler__completed__#{pool}", name)
redis_db.sadd("vmpooler__completed__#{pool}", name) redis.hset("vmpooler__vm__#{name}", 'checkout', Time.now)
redis_db.hset("vmpooler__vm__#{name}", 'checkout', Time.now) redis.hset("vmpooler__active__#{pool}", name, Time.now) if active
redis_db.hset("vmpooler__active__#{pool}", name, Time.now) if active
end end
def create_discovered_vm(name, pool, redis_handle = nil) def create_discovered_vm(name, pool, redis)
redis_db = redis_handle ? redis_handle : redis redis.sadd("vmpooler__discovered__#{pool}", name)
redis_db.sadd("vmpooler__discovered__#{pool}", name)
end end
def create_migrating_vm(name, pool, redis_handle = nil) def create_migrating_vm(name, pool, redis)
redis_db = redis_handle ? redis_handle : redis redis.hset("vmpooler__vm__#{name}", 'checkout', Time.now)
redis_db.hset("vmpooler__vm__#{name}", 'checkout', Time.now) redis.sadd("vmpooler__migrating__#{pool}", name)
redis_db.sadd("vmpooler__migrating__#{pool}", name)
end end
def create_tag(vm, tag_name, tag_value, redis_handle = nil) def create_tag(vm, tag_name, tag_value, redis)
redis_db = redis_handle ? redis-handle : redis redis.hset("vmpooler__vm__#{vm}", "tag:#{tag_name}", tag_value)
redis_db.hset("vmpooler__vm__#{vm}", "tag:#{tag_name}", tag_value)
end end
def add_vm_to_migration_set(name, redis_handle = nil) def add_vm_to_migration_set(name, redis)
redis_db = redis_handle ? redis_handle : redis redis.sadd('vmpooler__migration', name)
redis_db.sadd('vmpooler__migration', name)
end end
def fetch_vm(vm) def fetch_vm(vm)
redis.hgetall("vmpooler__vm__#{vm}") redis.hgetall("vmpooler__vm__#{vm}")
end end
def set_vm_data(vm, key, value) def set_vm_data(vm, key, value, redis)
redis.hset("vmpooler__vm__#{vm}", key, value) redis.hset("vmpooler__vm__#{vm}", key, value)
end end
def snapshot_revert_vm(vm, snapshot = '12345678901234567890123456789012') def snapshot_revert_vm(vm, snapshot = '12345678901234567890123456789012', redis)
redis.sadd('vmpooler__tasks__snapshot-revert', "#{vm}:#{snapshot}") redis.sadd('vmpooler__tasks__snapshot-revert', "#{vm}:#{snapshot}")
redis.hset("vmpooler__vm__#{vm}", "snapshot:#{snapshot}", "1") redis.hset("vmpooler__vm__#{vm}", "snapshot:#{snapshot}", "1")
end end
def snapshot_vm(vm, snapshot = '12345678901234567890123456789012') def snapshot_vm(vm, snapshot = '12345678901234567890123456789012', redis)
redis.sadd('vmpooler__tasks__snapshot', "#{vm}:#{snapshot}") redis.sadd('vmpooler__tasks__snapshot', "#{vm}:#{snapshot}")
redis.hset("vmpooler__vm__#{vm}", "snapshot:#{snapshot}", "1") redis.hset("vmpooler__vm__#{vm}", "snapshot:#{snapshot}", "1")
end end
def disk_task_vm(vm, disk_size = '10') def disk_task_vm(vm, disk_size = '10', redis)
redis.sadd('vmpooler__tasks__disk', "#{vm}:#{disk_size}") redis.sadd('vmpooler__tasks__disk', "#{vm}:#{disk_size}")
end end
def has_vm_snapshot?(vm) def has_vm_snapshot?(vm, redis)
redis.smembers('vmpooler__tasks__snapshot').any? do |snapshot| redis.smembers('vmpooler__tasks__snapshot').any? do |snapshot|
instance, sha = snapshot.split(':') instance, _sha = snapshot.split(':')
vm == instance vm == instance
end end
end end
def vm_reverted_to_snapshot?(vm, snapshot = nil) def vm_reverted_to_snapshot?(vm, redis, snapshot = nil)
redis.smembers('vmpooler__tasks__snapshot-revert').any? do |action| redis.smembers('vmpooler__tasks__snapshot-revert').any? do |action|
instance, sha = action.split(':') instance, sha = action.split(':')
instance == vm and (snapshot ? (sha == snapshot) : true) instance == vm and (snapshot ? (sha == snapshot) : true)
end end
end end
def pool_has_ready_vm?(pool, vm) def pool_has_ready_vm?(pool, vm, redis)
!!redis.sismember('vmpooler__ready__' + pool, vm) !!redis.sismember('vmpooler__ready__' + pool, vm)
end end
def create_ondemand_request_for_test(request_id, score, platforms_string, redis, user = nil, token = nil)
redis.zadd('vmpooler__provisioning__request', score, request_id)
redis.hset("vmpooler__odrequest__#{request_id}", 'requested', platforms_string)
redis.hset("vmpooler__odrequest__#{request_id}", 'token:token', token) if token
redis.hset("vmpooler__odrequest__#{request_id}", 'token:user', user) if user
end
def set_ondemand_request_status(request_id, status, redis)
redis.hset("vmpooler__odrequest__#{request_id}", 'status', status)
end
def create_ondemand_vm(vmname, request_id, pool, pool_alias, redis)
redis.sadd("vmpooler__#{request_id}__#{pool_alias}__#{pool}", vmname)
end
def create_ondemand_creationtask(request_string, score, redis)
redis.zadd('vmpooler__odcreate__task', score, request_string)
end
def create_ondemand_processing(request_id, score, redis)
redis.zadd('vmpooler__provisioning__processing', score, request_id)
end

6
spec/integration/api/v1/config_spec.rb
View file

@ -160,7 +160,7 @@ describe Vmpooler::API::V1 do
expect_json(ok = false, http = 400) expect_json(ok = false, http = 400)
expected = { expected = {
ok: false, ok: false,
bad_templates: ['pool10'] not_configured: ['pool10']
} }
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
@ -190,7 +190,7 @@ describe Vmpooler::API::V1 do
expected = { expected = {
ok: false, ok: false,
bad_templates: ['pool1'] not_configured: ['pool1']
} }
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
@ -202,7 +202,7 @@ describe Vmpooler::API::V1 do
expected = { expected = {
ok: false, ok: false,
bad_templates: ['pool1'] not_configured: ['pool1']
} }
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))

362
spec/integration/api/v1/ondemandvm_spec.rb Normal file
View file

@ -0,0 +1,362 @@
require 'spec_helper'
require 'rack/test'
describe Vmpooler::API::V1 do
include Rack::Test::Methods
def app()
Vmpooler::API end
describe '/ondemandvm' do
let(:prefix) { '/api/v1' }
let(:metrics) { Vmpooler::DummyStatsd.new }
let(:config) {
{
config: {
'site_name' => 'test pooler',
'vm_lifetime_auth' => 2,
'max_ondemand_instances_per_request' => 50,
'backend_weight' => {
'compute1' => 5
}
},
pools: [
{'name' => 'pool1', 'size' => 0},
{'name' => 'pool2', 'size' => 0, 'clone_target' => 'compute1'},
{'name' => 'pool3', 'size' => 0, 'clone_target' => 'compute1'}
],
alias: { 'poolone' => ['pool1'] },
pool_names: [ 'pool1', 'pool2', 'pool3', 'poolone' ]
}
}
let(:current_time) { Time.now }
let(:vmname) { 'abcdefghijkl' }
let(:checkoutlock) { Mutex.new }
let(:redis) { MockRedis.new }
let(:uuid) { SecureRandom.uuid }
before(:each) do
app.settings.set :config, config
app.settings.set :redis, redis
app.settings.set :metrics, metrics
app.settings.set :config, auth: false
app.settings.set :checkoutlock, checkoutlock
create_token('abcdefghijklmnopqrstuvwxyz012345', 'jdoe', current_time)
config[:pools].each do |pool|
redis.sadd('vmpooler__pools', pool['name'])
end
end
describe 'POST /ondemandvm' do
context 'with a configured pool' do
context 'with no request_id provided in payload' do
before(:each) do
expect(SecureRandom).to receive(:uuid).and_return(uuid)
end
it 'generates a request_id when none is provided' do
post "#{prefix}/ondemandvm", '{"pool1":"1"}'
expect_json(true, 201)
expected = {
"ok": true,
"request_id": uuid
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'uses a configured platform to fulfill a ondemand request' do
post "#{prefix}/ondemandvm", '{"poolone":"1"}'
expect_json(true, 201)
expected = {
"ok": true,
"request_id": uuid
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'creates a provisioning request in redis' do
expect(redis).to receive(:zadd).with('vmpooler__provisioning__request', Integer, uuid).and_return(1)
post "#{prefix}/ondemandvm", '{"poolone":"1"}'
end
it 'sets a platform string in redis for the request to indicate selected platforms' do
expect(redis).to receive(:hset).with("vmpooler__odrequest__#{uuid}", 'requested', 'poolone:pool1:1')
post "#{prefix}/ondemandvm", '{"poolone":"1"}'
end
context 'with domain set in the config' do
let(:domain) { 'example.com' }
before(:each) do
config[:config]['domain'] = domain
end
it 'should include domain in the return reply' do
post "#{prefix}/ondemandvm", '{"poolone":"1"}'
expect_json(true, 201)
expected = {
"ok": true,
"request_id": uuid,
"domain": domain
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
end
context 'with a resource request that exceeds the specified limit' do
let(:max_instances) { 50 }
before(:each) do
config[:config]['max_ondemand_instances_per_request'] = max_instances
end
it 'should reject the request with a message' do
post "#{prefix}/ondemandvm", '{"pool1":"51"}'
expect_json(false, 403)
expected = {
"ok": false,
"message": "requested amount of instances exceeds the maximum #{max_instances}"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
context 'with request_id provided in the payload' do
it 'uses the given request_id when provided' do
post "#{prefix}/ondemandvm", '{"pool1":"1","request_id":"1234"}'
expect_json(true, 201)
expected = {
"ok": true,
"request_id": "1234"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'returns 409 conflict error when the request_id has been used' do
post "#{prefix}/ondemandvm", '{"pool1":"1","request_id":"1234"}'
post "#{prefix}/ondemandvm", '{"pool1":"1","request_id":"1234"}'
expect_json(false, 409)
expected = {
"ok": false,
"request_id": "1234",
"message": "request_id '1234' has already been created"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
context 'with auth configured' do
it 'sets the token and user' do
app.settings.set :config, auth: true
expect(SecureRandom).to receive(:uuid).and_return(uuid)
allow(redis).to receive(:hset)
expect(redis).to receive(:hset).with("vmpooler__odrequest__#{uuid}", 'token:token', 'abcdefghijklmnopqrstuvwxyz012345')
expect(redis).to receive(:hset).with("vmpooler__odrequest__#{uuid}", 'token:user', 'jdoe')
post "#{prefix}/ondemandvm", '{"pool1":"1"}', {
'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345'
}
end
end
end
context 'with a pool that is not configured' do
let(:badpool) { 'pool4' }
it 'returns the bad template' do
post "#{prefix}/ondemandvm", '{"pool4":"1"}'
expect_json(false, 404)
expected = {
"ok": false,
"bad_templates": [ badpool ]
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
it 'returns 400 and a message when JSON is invalid' do
post "#{prefix}/ondemandvm", '{"pool1":"1}'
expect_json(false, 400)
expected = {
"ok": false,
"message": "JSON payload could not be parsed"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
describe 'GET /ondemandvm' do
it 'returns 404 with message when request is not found' do
get "#{prefix}/ondemandvm/#{uuid}"
expect_json(false, 404)
expected = {
"ok": false,
"message": "no request found for request_id '#{uuid}'"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
context 'when the request is found' do
let(:score) { current_time }
let(:platforms_string) { 'pool1:pool1:1' }
before(:each) do
create_ondemand_request_for_test(uuid, score, platforms_string, redis)
end
it 'returns 202 while the request is waiting' do
get "#{prefix}/ondemandvm/#{uuid}"
expect_json(true, 202)
expected = {
"ok": true,
"request_id": uuid,
"ready": false,
"pool1": {
"ready": "0",
"pending": "1"
}
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
context 'with ready instances' do
before(:each) do
create_ondemand_vm(vmname, uuid, 'pool1', 'pool1', redis)
set_ondemand_request_status(uuid, 'ready', redis)
end
it 'returns 200 with hostnames when the request is ready' do
get "#{prefix}/ondemandvm/#{uuid}"
expect_json(true, 200)
expected = {
"ok": true,
"request_id": uuid,
"ready": true,
"pool1": {
"hostname": [
vmname
]
}
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
context 'with domain set' do
let(:domain) { 'example.com' }
before(:each) do
config[:config]['domain'] = domain
end
it 'should include the domain in the result' do
get "#{prefix}/ondemandvm/#{uuid}"
expected = {
"ok": true,
"request_id": uuid,
"ready": true,
"pool1": {
"hostname": [
vmname
]
},
"domain": domain
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
end
context 'with a deleted request' do
before(:each) do
set_ondemand_request_status(uuid, 'deleted', redis)
end
it 'returns a message that the request has been deleted' do
get "#{prefix}/ondemandvm/#{uuid}"
expect_json(true, 200)
expected = {
"ok": true,
"request_id": uuid,
"ready": false,
"message": "The request has been deleted"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
context 'with a failed request' do
let(:ondemand_request_ttl) { 5 }
before(:each) do
config[:config]['ondemand_request_ttl'] = ondemand_request_ttl
set_ondemand_request_status(uuid, 'failed', redis)
end
it 'returns a message that the request has failed' do
get "#{prefix}/ondemandvm/#{uuid}"
expect_json(true, 200)
expected = {
"ok": true,
"request_id": uuid,
"ready": false,
"message": "The request failed to provision instances within the configured ondemand_request_ttl '#{ondemand_request_ttl}'"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
end
end
end
describe 'DELETE /ondemandvm' do
let(:expiration) { 129_600_0 }
it 'returns 404 with message when request is not found' do
delete "#{prefix}/ondemandvm/#{uuid}"
expect_json(false, 404)
expected = {
"ok": false,
"message": "no request found for request_id '#{uuid}'"
}
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
context 'when the request is found' do
let(:platforms_string) { 'pool1:pool1:1' }
let(:score) { current_time.to_i }
before(:each) do
create_ondemand_request_for_test(uuid, score, platforms_string, redis)
end
it 'returns 200 for a deleted request' do
delete "#{prefix}/ondemandvm/#{uuid}"
expect_json(true, 200)
expected = { 'ok': true }
expect(last_response.body).to eq(JSON.pretty_generate(expected))
end
it 'marks the request hash for expiration in two weeks' do
expect(redis).to receive(:expire).with("vmpooler__odrequest__#{uuid}", expiration)
delete "#{prefix}/ondemandvm/#{uuid}"
end
context 'with running instances' do
let(:pool) { 'pool1' }
let(:pool_alias) { pool }
before(:each) do
create_ondemand_vm(vmname, uuid, pool, pool_alias, redis)
end
it 'moves allocated instances to the completed queue' do
expect(redis).to receive(:smove).with("vmpooler__running__#{pool}", "vmpooler__completed__#{pool}", vmname)
delete "#{prefix}/ondemandvm/#{uuid}"
end
it 'deletes the set tracking instances allocated for the request' do
expect(redis).to receive(:del).with("vmpooler__#{uuid}__#{pool_alias}__#{pool}")
delete "#{prefix}/ondemandvm/#{uuid}"
end
end
end
end
end
end

14
spec/integration/api/v1/status_spec.rb
View file

@ -50,7 +50,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns the number of ready vms for each pool' do it 'returns the number of ready vms for each pool' do
3.times {|i| create_ready_vm("pool1", "vm-#{i}") } 3.times {|i| create_ready_vm("pool1", "vm-#{i}", redis) }
get "#{prefix}/status/" get "#{prefix}/status/"
# of course /status doesn't conform to the weird standard everything else uses... # of course /status doesn't conform to the weird standard everything else uses...
@ -61,8 +61,8 @@ describe Vmpooler::API::V1 do
end end
it 'returns the number of running vms for each pool' do it 'returns the number of running vms for each pool' do
3.times {|i| create_running_vm("pool1", "vm-#{i}") } 3.times {|i| create_running_vm("pool1", "vm-#{i}", redis) }
4.times {|i| create_running_vm("pool2", "vm-#{i}") } 4.times {|i| create_running_vm("pool2", "vm-#{i}", redis) }
get "#{prefix}/status/" get "#{prefix}/status/"
@ -74,8 +74,8 @@ describe Vmpooler::API::V1 do
end end
it 'returns the number of pending vms for each pool' do it 'returns the number of pending vms for each pool' do
3.times {|i| create_pending_vm("pool1", "vm-#{i}") } 3.times {|i| create_pending_vm("pool1", "vm-#{i}", redis) }
4.times {|i| create_pending_vm("pool2", "vm-#{i}") } 4.times {|i| create_pending_vm("pool2", "vm-#{i}", redis) }
get "#{prefix}/status/" get "#{prefix}/status/"
@ -230,8 +230,8 @@ describe Vmpooler::API::V1 do
it 'returns the number of running VMs' do it 'returns the number of running VMs' do
get "#{prefix}/totalrunning" get "#{prefix}/totalrunning"
expect(last_response.header['Content-Type']).to eq('application/json') expect(last_response.header['Content-Type']).to eq('application/json')
5.times {|i| create_running_vm("pool1", "vm-#{i}") } 5.times {|i| create_running_vm("pool1", "vm-#{i}", redis, redis) }
5.times {|i| create_running_vm("pool3", "vm-#{i}") } 5.times {|i| create_running_vm("pool3", "vm-#{i}", redis, redis) }
result = JSON.parse(last_response.body) result = JSON.parse(last_response.body)
expect(result["running"] == 10) expect(result["running"] == 10)
end end

83
spec/integration/api/v1/vm_hostname_spec.rb
View file

@ -33,6 +33,8 @@ describe Vmpooler::API::V1 do
let(:current_time) { Time.now } let(:current_time) { Time.now }
let(:redis) { MockRedis.new }
before(:each) do before(:each) do
app.settings.set :config, config app.settings.set :config, config
app.settings.set :redis, redis app.settings.set :redis, redis
@ -41,7 +43,7 @@ describe Vmpooler::API::V1 do
describe 'PUT /vm/:hostname' do describe 'PUT /vm/:hostname' do
it 'allows tags to be set' do it 'allows tags to be set' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"tags":{"tested_by":"rspec"}}' put "#{prefix}/vm/testhost", '{"tags":{"tested_by":"rspec"}}'
expect_json(ok = true, http = 200) expect_json(ok = true, http = 200)
@ -49,7 +51,7 @@ describe Vmpooler::API::V1 do
end end
it 'skips empty tags' do it 'skips empty tags' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"tags":{"tested_by":""}}' put "#{prefix}/vm/testhost", '{"tags":{"tested_by":""}}'
expect_json(ok = true, http = 200) expect_json(ok = true, http = 200)
@ -57,7 +59,7 @@ describe Vmpooler::API::V1 do
end end
it 'does not set tags if request body format is invalid' do it 'does not set tags if request body format is invalid' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"tags":{"tested"}}' put "#{prefix}/vm/testhost", '{"tags":{"tested"}}'
expect_json(ok = false, http = 400) expect_json(ok = false, http = 400)
@ -69,7 +71,7 @@ describe Vmpooler::API::V1 do
app.settings.set :config, app.settings.set :config,
{ :config => { 'allowed_tags' => ['created_by', 'project', 'url'] } } { :config => { 'allowed_tags' => ['created_by', 'project', 'url'] } }
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"tags":{"created_by":"rspec","tested_by":"rspec"}}' put "#{prefix}/vm/testhost", '{"tags":{"created_by":"rspec","tested_by":"rspec"}}'
expect_json(ok = false, http = 400) expect_json(ok = false, http = 400)
@ -84,7 +86,7 @@ describe Vmpooler::API::V1 do
} } } }
it 'correctly filters tags' do it 'correctly filters tags' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"tags":{"url":"foo.com/something.html"}}' put "#{prefix}/vm/testhost", '{"tags":{"url":"foo.com/something.html"}}'
expect_json(ok = true, http = 200) expect_json(ok = true, http = 200)
@ -93,7 +95,7 @@ describe Vmpooler::API::V1 do
end end
it "doesn't eat tags not matching filter" do it "doesn't eat tags not matching filter" do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"tags":{"url":"foo.com"}}' put "#{prefix}/vm/testhost", '{"tags":{"url":"foo.com"}}'
expect_json(ok = true, http = 200) expect_json(ok = true, http = 200)
@ -105,7 +107,7 @@ describe Vmpooler::API::V1 do
let(:config) { { auth: false } } let(:config) { { auth: false } }
it 'allows VM lifetime to be modified without a token' do it 'allows VM lifetime to be modified without a token' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"lifetime":"1"}' put "#{prefix}/vm/testhost", '{"lifetime":"1"}'
expect_json(ok = true, http = 200) expect_json(ok = true, http = 200)
@ -115,7 +117,7 @@ describe Vmpooler::API::V1 do
end end
it 'does not allow a lifetime to be 0' do it 'does not allow a lifetime to be 0' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"lifetime":"0"}' put "#{prefix}/vm/testhost", '{"lifetime":"0"}'
expect_json(ok = false, http = 400) expect_json(ok = false, http = 400)
@ -125,7 +127,7 @@ describe Vmpooler::API::V1 do
end end
it 'does not enforce a lifetime' do it 'does not enforce a lifetime' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"lifetime":"20000"}' put "#{prefix}/vm/testhost", '{"lifetime":"20000"}'
expect_json(ok = true, http = 200) expect_json(ok = true, http = 200)
@ -137,7 +139,7 @@ describe Vmpooler::API::V1 do
it 'does not allow a lifetime to be initially past config max_lifetime_upper_limit' do it 'does not allow a lifetime to be initially past config max_lifetime_upper_limit' do
app.settings.set :config, app.settings.set :config,
{ :config => { 'max_lifetime_upper_limit' => 168 } } { :config => { 'max_lifetime_upper_limit' => 168 } }
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"lifetime":"200"}' put "#{prefix}/vm/testhost", '{"lifetime":"200"}'
expect_json(ok = false, http = 400) expect_json(ok = false, http = 400)
@ -146,6 +148,19 @@ describe Vmpooler::API::V1 do
expect(vm['lifetime']).to be_nil expect(vm['lifetime']).to be_nil
end end
# it 'does not allow a lifetime to be extended past config 168' do
# app.settings.set :config,
# { :config => { 'max_lifetime_upper_limit' => 168 } }
# create_vm('testhost', redis)
#
# set_vm_data('testhost', "checkout", (Time.now - (69*60*60)), redis)
# puts redis.hget("vmpooler__vm__testhost", 'checkout')
# put "#{prefix}/vm/testhost", '{"lifetime":"100"}'
# expect_json(ok = false, http = 400)
#
# vm = fetch_vm('testhost')
# expect(vm['lifetime']).to be_nil
# end
end end
context '(auth configured)' do context '(auth configured)' do
@ -154,7 +169,7 @@ describe Vmpooler::API::V1 do
end end
it 'allows VM lifetime to be modified with a token' do it 'allows VM lifetime to be modified with a token' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"lifetime":"1"}', { put "#{prefix}/vm/testhost", '{"lifetime":"1"}', {
'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345' 'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345'
@ -166,7 +181,7 @@ describe Vmpooler::API::V1 do
end end
it 'does not allows VM lifetime to be modified without a token' do it 'does not allows VM lifetime to be modified without a token' do
create_vm('testhost') create_vm('testhost', redis)
put "#{prefix}/vm/testhost", '{"lifetime":"1"}' put "#{prefix}/vm/testhost", '{"lifetime":"1"}'
expect_json(ok = false, http = 401) expect_json(ok = false, http = 401)
@ -182,7 +197,7 @@ describe Vmpooler::API::V1 do
end end
it 'deletes an existing VM' do it 'deletes an existing VM' do
create_running_vm('pool1', 'testhost') create_running_vm('pool1', 'testhost', redis)
expect fetch_vm('testhost') expect fetch_vm('testhost')
delete "#{prefix}/vm/testhost" delete "#{prefix}/vm/testhost"
@ -198,7 +213,7 @@ describe Vmpooler::API::V1 do
context '(checked-out without token)' do context '(checked-out without token)' do
it 'deletes a VM without supplying a token' do it 'deletes a VM without supplying a token' do
create_running_vm('pool1', 'testhost') create_running_vm('pool1', 'testhost', redis)
expect fetch_vm('testhost') expect fetch_vm('testhost')
delete "#{prefix}/vm/testhost" delete "#{prefix}/vm/testhost"
@ -209,7 +224,7 @@ describe Vmpooler::API::V1 do
context '(checked-out with token)' do context '(checked-out with token)' do
it 'fails to delete a VM without supplying a token' do it 'fails to delete a VM without supplying a token' do
create_running_vm('pool1', 'testhost', 'abcdefghijklmnopqrstuvwxyz012345') create_running_vm('pool1', 'testhost', redis, 'abcdefghijklmnopqrstuvwxyz012345')
expect fetch_vm('testhost') expect fetch_vm('testhost')
delete "#{prefix}/vm/testhost" delete "#{prefix}/vm/testhost"
@ -218,7 +233,7 @@ describe Vmpooler::API::V1 do
end end
it 'deletes a VM when token is supplied' do it 'deletes a VM when token is supplied' do
create_running_vm('pool1', 'testhost', 'abcdefghijklmnopqrstuvwxyz012345') create_running_vm('pool1', 'testhost', redis, 'abcdefghijklmnopqrstuvwxyz012345')
expect fetch_vm('testhost') expect fetch_vm('testhost')
delete "#{prefix}/vm/testhost", "", { delete "#{prefix}/vm/testhost", "", {
@ -235,7 +250,7 @@ describe Vmpooler::API::V1 do
describe 'POST /vm/:hostname/snapshot' do describe 'POST /vm/:hostname/snapshot' do
context '(auth not configured)' do context '(auth not configured)' do
it 'creates a snapshot' do it 'creates a snapshot' do
create_vm('testhost') create_vm('testhost', redis)
post "#{prefix}/vm/testhost/snapshot" post "#{prefix}/vm/testhost/snapshot"
expect_json(ok = true, http = 202) expect_json(ok = true, http = 202)
expect(JSON.parse(last_response.body)['testhost']['snapshot'].length).to be(32) expect(JSON.parse(last_response.body)['testhost']['snapshot'].length).to be(32)
@ -250,19 +265,19 @@ describe Vmpooler::API::V1 do
it 'returns a 401 if not authed' do it 'returns a 401 if not authed' do
post "#{prefix}/vm/testhost/snapshot" post "#{prefix}/vm/testhost/snapshot"
expect_json(ok = false, http = 401) expect_json(ok = false, http = 401)
expect !has_vm_snapshot?('testhost') expect !has_vm_snapshot?('testhost', redis)
end end
it 'creates a snapshot if authed' do it 'creates a snapshot if authed' do
create_vm('testhost') create_vm('testhost', redis)
snapshot_vm('testhost', 'testsnapshot') snapshot_vm('testhost', 'testsnapshot', redis)
post "#{prefix}/vm/testhost/snapshot", "", { post "#{prefix}/vm/testhost/snapshot", "", {
'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345' 'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345'
} }
expect_json(ok = true, http = 202) expect_json(ok = true, http = 202)
expect(JSON.parse(last_response.body)['testhost']['snapshot'].length).to be(32) expect(JSON.parse(last_response.body)['testhost']['snapshot'].length).to be(32)
expect has_vm_snapshot?('testhost') expect has_vm_snapshot?('testhost', redis)
end end
end end
end end
@ -270,22 +285,22 @@ describe Vmpooler::API::V1 do
describe 'POST /vm/:hostname/snapshot/:snapshot' do describe 'POST /vm/:hostname/snapshot/:snapshot' do
context '(auth not configured)' do context '(auth not configured)' do
it 'reverts to a snapshot' do it 'reverts to a snapshot' do
create_vm('testhost') create_vm('testhost', redis)
snapshot_vm('testhost', 'testsnapshot') snapshot_vm('testhost', 'testsnapshot', redis)
post "#{prefix}/vm/testhost/snapshot/testsnapshot" post "#{prefix}/vm/testhost/snapshot/testsnapshot"
expect_json(ok = true, http = 202) expect_json(ok = true, http = 202)
expect vm_reverted_to_snapshot?('testhost', 'testsnapshot') expect vm_reverted_to_snapshot?('testhost', redis, 'testsnapshot')
end end
it 'fails if the specified snapshot does not exist' do it 'fails if the specified snapshot does not exist' do
create_vm('testhost') create_vm('testhost', redis)
post "#{prefix}/vm/testhost/snapshot/testsnapshot", "", { post "#{prefix}/vm/testhost/snapshot/testsnapshot", "", {
'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345' 'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345'
} }
expect_json(ok = false, http = 404) expect_json(ok = false, http = 404)
expect !vm_reverted_to_snapshot?('testhost', 'testsnapshot') expect !vm_reverted_to_snapshot?('testhost', redis, 'testsnapshot')
end end
end end
@ -295,33 +310,33 @@ describe Vmpooler::API::V1 do
end end
it 'returns a 401 if not authed' do it 'returns a 401 if not authed' do
create_vm('testhost') create_vm('testhost', redis)
snapshot_vm('testhost', 'testsnapshot') snapshot_vm('testhost', 'testsnapshot', redis)
post "#{prefix}/vm/testhost/snapshot/testsnapshot" post "#{prefix}/vm/testhost/snapshot/testsnapshot"
expect_json(ok = false, http = 401) expect_json(ok = false, http = 401)
expect !vm_reverted_to_snapshot?('testhost', 'testsnapshot') expect !vm_reverted_to_snapshot?('testhost', redis, 'testsnapshot')
end end
it 'fails if authed and the specified snapshot does not exist' do it 'fails if authed and the specified snapshot does not exist' do
create_vm('testhost') create_vm('testhost', redis)
post "#{prefix}/vm/testhost/snapshot/testsnapshot", "", { post "#{prefix}/vm/testhost/snapshot/testsnapshot", "", {
'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345' 'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345'
} }
expect_json(ok = false, http = 404) expect_json(ok = false, http = 404)
expect !vm_reverted_to_snapshot?('testhost', 'testsnapshot') expect !vm_reverted_to_snapshot?('testhost', redis, 'testsnapshot')
end end
it 'reverts to a snapshot if authed' do it 'reverts to a snapshot if authed' do
create_vm('testhost') create_vm('testhost', redis)
snapshot_vm('testhost', 'testsnapshot') snapshot_vm('testhost', 'testsnapshot', redis)
post "#{prefix}/vm/testhost/snapshot/testsnapshot", "", { post "#{prefix}/vm/testhost/snapshot/testsnapshot", "", {
'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345' 'HTTP_X_AUTH_TOKEN' => 'abcdefghijklmnopqrstuvwxyz012345'
} }
expect_json(ok = true, http = 202) expect_json(ok = true, http = 202)
expect vm_reverted_to_snapshot?('testhost', 'testsnapshot') expect vm_reverted_to_snapshot?('testhost', redis, 'testsnapshot')
end end
end end
end end

76
spec/integration/api/v1/vm_spec.rb
View file

@ -42,7 +42,7 @@ describe Vmpooler::API::V1 do
describe 'GET /vm/:hostname' do describe 'GET /vm/:hostname' do
it 'returns correct information on a running vm' do it 'returns correct information on a running vm' do
create_running_vm 'pool1', vmname create_running_vm 'pool1', vmname, redis
get "#{prefix}/vm/#{vmname}" get "#{prefix}/vm/#{vmname}"
expect_json(ok = true, http = 200) expect_json(ok = true, http = 200)
response_body = (JSON.parse(last_response.body)[vmname]) response_body = (JSON.parse(last_response.body)[vmname])
@ -63,7 +63,7 @@ describe Vmpooler::API::V1 do
let(:socket) { double('socket') } let(:socket) { double('socket') }
it 'returns a single VM' do it 'returns a single VM' do
create_ready_vm 'pool1', vmname create_ready_vm 'pool1', vmname, redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
post "#{prefix}/vm", '{"pool1":"1"}' post "#{prefix}/vm", '{"pool1":"1"}'
@ -80,7 +80,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns a single VM for an alias' do it 'returns a single VM for an alias' do
create_ready_vm 'pool1', vmname create_ready_vm 'pool1', vmname, redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -106,7 +106,7 @@ describe Vmpooler::API::V1 do
Vmpooler::API.settings.config.delete(:alias) Vmpooler::API.settings.config.delete(:alias)
Vmpooler::API.settings.config[:pool_names] = ['pool1', 'pool2'] Vmpooler::API.settings.config[:pool_names] = ['pool1', 'pool2']
create_ready_vm 'pool1', vmname create_ready_vm 'pool1', vmname, redis
post "#{prefix}/vm/pool1" post "#{prefix}/vm/pool1"
post "#{prefix}/vm/pool1" post "#{prefix}/vm/pool1"
@ -117,7 +117,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns 503 for empty pool referenced by alias' do it 'returns 503 for empty pool referenced by alias' do
create_ready_vm 'pool1', vmname create_ready_vm 'pool1', vmname, redis
post "#{prefix}/vm/poolone" post "#{prefix}/vm/poolone"
post "#{prefix}/vm/poolone" post "#{prefix}/vm/poolone"
@ -128,8 +128,8 @@ describe Vmpooler::API::V1 do
end end
it 'returns multiple VMs' do it 'returns multiple VMs' do
create_ready_vm 'pool1', vmname create_ready_vm 'pool1', vmname, redis
create_ready_vm 'pool2', 'qrstuvwxyz012345' create_ready_vm 'pool2', 'qrstuvwxyz012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -150,9 +150,9 @@ describe Vmpooler::API::V1 do
end end
it 'returns multiple VMs even when multiple instances from the same pool are requested' do it 'returns multiple VMs even when multiple instances from the same pool are requested' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
create_ready_vm 'pool1', '2abcdefghijklmnop' create_ready_vm 'pool1', '2abcdefghijklmnop', redis
create_ready_vm 'pool2', 'qrstuvwxyz012345' create_ready_vm 'pool2', 'qrstuvwxyz012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -177,11 +177,11 @@ describe Vmpooler::API::V1 do
end end
it 'returns multiple VMs even when multiple instances from multiple pools are requested' do it 'returns multiple VMs even when multiple instances from multiple pools are requested' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
create_ready_vm 'pool1', '2abcdefghijklmnop' create_ready_vm 'pool1', '2abcdefghijklmnop', redis
create_ready_vm 'pool2', '1qrstuvwxyz012345' create_ready_vm 'pool2', '1qrstuvwxyz012345', redis
create_ready_vm 'pool2', '2qrstuvwxyz012345' create_ready_vm 'pool2', '2qrstuvwxyz012345', redis
create_ready_vm 'pool2', '3qrstuvwxyz012345' create_ready_vm 'pool2', '3qrstuvwxyz012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -208,9 +208,9 @@ describe Vmpooler::API::V1 do
it 'returns VMs from multiple backend pools requested by an alias' do it 'returns VMs from multiple backend pools requested by an alias' do
Vmpooler::API.settings.config[:alias]['genericpool'] = ['pool1', 'pool2', 'pool3'] Vmpooler::API.settings.config[:alias]['genericpool'] = ['pool1', 'pool2', 'pool3']
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
create_ready_vm 'pool2', '2abcdefghijklmnop' create_ready_vm 'pool2', '2abcdefghijklmnop', redis
create_ready_vm 'pool3', '1qrstuvwxyz012345' create_ready_vm 'pool3', '1qrstuvwxyz012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -231,9 +231,9 @@ describe Vmpooler::API::V1 do
end end
it 'returns the first VM that was moved to the ready state when checking out a VM' do it 'returns the first VM that was moved to the ready state when checking out a VM' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
create_ready_vm 'pool1', '2abcdefghijklmnop' create_ready_vm 'pool1', '2abcdefghijklmnop', redis
create_ready_vm 'pool1', '3abcdefghijklmnop' create_ready_vm 'pool1', '3abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -251,7 +251,7 @@ describe Vmpooler::API::V1 do
end end
it 'fails when not all requested vms can be allocated' do it 'fails when not all requested vms can be allocated' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
post "#{prefix}/vm", '{"pool1":"1","pool2":"1"}' post "#{prefix}/vm", '{"pool1":"1","pool2":"1"}'
@ -262,7 +262,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns any checked out vms to their pools when not all requested vms can be allocated' do it 'returns any checked out vms to their pools when not all requested vms can be allocated' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -273,11 +273,11 @@ describe Vmpooler::API::V1 do
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
expect_json(ok = false, http = 503) expect_json(ok = false, http = 503)
expect(pool_has_ready_vm?('pool1', '1abcdefghijklmnop')).to eq(true) expect(pool_has_ready_vm?('pool1', '1abcdefghijklmnop', redis)).to eq(true)
end end
it 'fails when not all requested vms can be allocated, when requesting multiple instances from a pool' do it 'fails when not all requested vms can be allocated, when requesting multiple instances from a pool' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
post "#{prefix}/vm", '{"pool1":"2","pool2":"1"}' post "#{prefix}/vm", '{"pool1":"2","pool2":"1"}'
@ -288,7 +288,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from a pool' do it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from a pool' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -299,11 +299,11 @@ describe Vmpooler::API::V1 do
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
expect_json(ok = false, http = 503) expect_json(ok = false, http = 503)
expect(pool_has_ready_vm?('pool1', '1abcdefghijklmnop')).to eq(true) expect(pool_has_ready_vm?('pool1', '1abcdefghijklmnop', redis)).to eq(true)
end end
it 'fails when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do it 'fails when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
post "#{prefix}/vm", '{"pool1":"2","pool2":"3"}' post "#{prefix}/vm", '{"pool1":"2","pool2":"3"}'
@ -314,8 +314,8 @@ describe Vmpooler::API::V1 do
end end
it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
create_ready_vm 'pool1', '2abcdefghijklmnop' create_ready_vm 'pool1', '2abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -326,13 +326,13 @@ describe Vmpooler::API::V1 do
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
expect_json(ok = false, http = 503) expect_json(ok = false, http = 503)
expect(pool_has_ready_vm?('pool1', '1abcdefghijklmnop')).to eq(true) expect(pool_has_ready_vm?('pool1', '1abcdefghijklmnop', redis)).to eq(true)
expect(pool_has_ready_vm?('pool1', '2abcdefghijklmnop')).to eq(true) expect(pool_has_ready_vm?('pool1', '2abcdefghijklmnop', redis)).to eq(true)
end end
it 'returns the second VM when the first fails to respond' do it 'returns the second VM when the first fails to respond' do
create_ready_vm 'pool1', vmname create_ready_vm 'pool1', vmname, redis
create_ready_vm 'pool1', "2#{vmname}" create_ready_vm 'pool1', "2#{vmname}", redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).with(vmname, nil).and_raise('mockerror') allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).with(vmname, nil).and_raise('mockerror')
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).with("2#{vmname}", nil).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).with("2#{vmname}", nil).and_return(socket)
@ -349,14 +349,14 @@ describe Vmpooler::API::V1 do
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
expect(pool_has_ready_vm?('pool1', vmname)).to be false expect(pool_has_ready_vm?('pool1', vmname, redis)).to be false
end end
context '(auth not configured)' do context '(auth not configured)' do
it 'does not extend VM lifetime if auth token is provided' do it 'does not extend VM lifetime if auth token is provided' do
app.settings.set :config, auth: false app.settings.set :config, auth: false
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -382,7 +382,7 @@ describe Vmpooler::API::V1 do
it 'extends VM lifetime if auth token is provided' do it 'extends VM lifetime if auth token is provided' do
app.settings.set :config, auth: true app.settings.set :config, auth: true
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -405,7 +405,7 @@ describe Vmpooler::API::V1 do
it 'does not extend VM lifetime if auth token is not provided' do it 'does not extend VM lifetime if auth token is not provided' do
app.settings.set :config, auth: true app.settings.set :config, auth: true
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)

62
spec/integration/api/v1/vm_template_spec.rb
View file

@ -19,7 +19,8 @@ describe Vmpooler::API::V1 do
}, },
pools: [ pools: [
{'name' => 'pool1', 'size' => 5}, {'name' => 'pool1', 'size' => 5},
{'name' => 'pool2', 'size' => 10} {'name' => 'pool2', 'size' => 10},
{'name' => 'poolone', 'size' => 0}
], ],
statsd: { 'prefix' => 'stats_prefix'}, statsd: { 'prefix' => 'stats_prefix'},
alias: { 'poolone' => 'pool1' }, alias: { 'poolone' => 'pool1' },
@ -42,7 +43,7 @@ describe Vmpooler::API::V1 do
describe 'POST /vm/:template' do describe 'POST /vm/:template' do
it 'returns a single VM' do it 'returns a single VM' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -60,7 +61,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns a single VM for an alias' do it 'returns a single VM for an alias' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -86,7 +87,7 @@ describe Vmpooler::API::V1 do
Vmpooler::API.settings.config.delete(:alias) Vmpooler::API.settings.config.delete(:alias)
Vmpooler::API.settings.config[:pool_names] = ['pool1', 'pool2'] Vmpooler::API.settings.config[:pool_names] = ['pool1', 'pool2']
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
post "#{prefix}/vm/pool1" post "#{prefix}/vm/pool1"
post "#{prefix}/vm/pool1" post "#{prefix}/vm/pool1"
@ -97,8 +98,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns 503 for empty pool referenced by alias' do it 'returns 503 for empty pool referenced by alias' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
post "#{prefix}/vm/poolone"
post "#{prefix}/vm/poolone" post "#{prefix}/vm/poolone"
expected = { ok: false } expected = { ok: false }
@ -108,8 +108,8 @@ describe Vmpooler::API::V1 do
end end
it 'returns multiple VMs' do it 'returns multiple VMs' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
create_ready_vm 'pool2', 'qrstuvwxyz012345' create_ready_vm 'pool2', 'qrstuvwxyz012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -130,12 +130,12 @@ describe Vmpooler::API::V1 do
end end
it 'returns multiple VMs even when multiple instances from multiple pools are requested' do it 'returns multiple VMs even when multiple instances from multiple pools are requested' do
create_ready_vm 'pool1', '1abcdefghijklmnop' create_ready_vm 'pool1', '1abcdefghijklmnop', redis
create_ready_vm 'pool1', '2abcdefghijklmnop' create_ready_vm 'pool1', '2abcdefghijklmnop', redis
create_ready_vm 'pool2', '1qrstuvwxyz012345' create_ready_vm 'pool2', '1qrstuvwxyz012345', redis
create_ready_vm 'pool2', '2qrstuvwxyz012345' create_ready_vm 'pool2', '2qrstuvwxyz012345', redis
create_ready_vm 'pool2', '3qrstuvwxyz012345' create_ready_vm 'pool2', '3qrstuvwxyz012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -159,7 +159,7 @@ describe Vmpooler::API::V1 do
end end
it 'fails when not all requested vms can be allocated' do it 'fails when not all requested vms can be allocated' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
post "#{prefix}/vm/pool1+pool2", '' post "#{prefix}/vm/pool1+pool2", ''
@ -170,7 +170,7 @@ describe Vmpooler::API::V1 do
end end
it 'returns any checked out vms to their pools when not all requested vms can be allocated' do it 'returns any checked out vms to their pools when not all requested vms can be allocated' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -181,12 +181,12 @@ describe Vmpooler::API::V1 do
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
expect_json(ok = false, http = 503) expect_json(ok = false, http = 503)
expect(pool_has_ready_vm?('pool1', 'abcdefghijklmnop')).to eq(true) expect(pool_has_ready_vm?('pool1', 'abcdefghijklmnop', redis)).to eq(true)
end end
it 'fails when not all requested vms can be allocated, when requesting multiple instances from a pool' do it 'fails when not all requested vms can be allocated, when requesting multiple instances from a pool' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
create_ready_vm 'pool1', '0123456789012345' create_ready_vm 'pool1', '0123456789012345', redis
post "#{prefix}/vm/pool1+pool1+pool2", '' post "#{prefix}/vm/pool1+pool1+pool2", ''
@ -197,8 +197,8 @@ describe Vmpooler::API::V1 do
end end
it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from a pool' do it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from a pool' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
create_ready_vm 'pool1', '0123456789012345' create_ready_vm 'pool1', '0123456789012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -209,13 +209,13 @@ describe Vmpooler::API::V1 do
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
expect_json(ok = false, http = 503) expect_json(ok = false, http = 503)
expect(pool_has_ready_vm?('pool1', 'abcdefghijklmnop')).to eq(true) expect(pool_has_ready_vm?('pool1', 'abcdefghijklmnop', redis)).to eq(true)
expect(pool_has_ready_vm?('pool1', '0123456789012345')).to eq(true) expect(pool_has_ready_vm?('pool1', '0123456789012345', redis)).to eq(true)
end end
it 'fails when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do it 'fails when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
create_ready_vm 'pool2', '0123456789012345' create_ready_vm 'pool2', '0123456789012345', redis
post "#{prefix}/vm/pool1+pool1+pool2+pool2+pool2", '' post "#{prefix}/vm/pool1+pool1+pool2+pool2+pool2", ''
@ -226,8 +226,8 @@ describe Vmpooler::API::V1 do
end end
it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do it 'returns any checked out vms to their pools when not all requested vms can be allocated, when requesting multiple instances from multiple pools' do
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
create_ready_vm 'pool2', '0123456789012345' create_ready_vm 'pool2', '0123456789012345', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -238,15 +238,15 @@ describe Vmpooler::API::V1 do
expect(last_response.body).to eq(JSON.pretty_generate(expected)) expect(last_response.body).to eq(JSON.pretty_generate(expected))
expect_json(ok = false, http = 503) expect_json(ok = false, http = 503)
expect(pool_has_ready_vm?('pool1', 'abcdefghijklmnop')).to eq(true) expect(pool_has_ready_vm?('pool1', 'abcdefghijklmnop', redis)).to eq(true)
expect(pool_has_ready_vm?('pool2', '0123456789012345')).to eq(true) expect(pool_has_ready_vm?('pool2', '0123456789012345', redis)).to eq(true)
end end
context '(auth not configured)' do context '(auth not configured)' do
it 'does not extend VM lifetime if auth token is provided' do it 'does not extend VM lifetime if auth token is provided' do
app.settings.set :config, auth: false app.settings.set :config, auth: false
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -272,7 +272,7 @@ describe Vmpooler::API::V1 do
it 'extends VM lifetime if auth token is provided' do it 'extends VM lifetime if auth token is provided' do
app.settings.set :config, auth: true app.settings.set :config, auth: true
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)
@ -295,7 +295,7 @@ describe Vmpooler::API::V1 do
it 'does not extend VM lifetime if auth token is not provided' do it 'does not extend VM lifetime if auth token is not provided' do
app.settings.set :config, auth: true app.settings.set :config, auth: true
create_ready_vm 'pool1', 'abcdefghijklmnop' create_ready_vm 'pool1', 'abcdefghijklmnop', redis
allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket) allow_any_instance_of(Vmpooler::API::Helpers).to receive(:open_socket).and_return(socket)

40
spec/integration/dashboard_spec.rb
View file

@ -56,11 +56,11 @@ describe Vmpooler::API do
context 'without history param' do context 'without history param' do
it 'returns basic JSON' do it 'returns basic JSON' do
create_ready_vm('pool1', 'vm1') create_ready_vm('pool1', 'vm1', redis)
create_ready_vm('pool1', 'vm2') create_ready_vm('pool1', 'vm2', redis)
create_ready_vm('pool1', 'vm3') create_ready_vm('pool1', 'vm3', redis)
create_ready_vm('pool2', 'vm4') create_ready_vm('pool2', 'vm4', redis)
create_ready_vm('pool2', 'vm5') create_ready_vm('pool2', 'vm5', redis)
get '/dashboard/stats/vmpooler/pool' get '/dashboard/stats/vmpooler/pool'
@ -90,11 +90,11 @@ describe Vmpooler::API do
end end
it 'returns JSON with history when redis has values' do it 'returns JSON with history when redis has values' do
create_ready_vm('pool1', 'vm1') create_ready_vm('pool1', 'vm1', redis)
create_ready_vm('pool1', 'vm2') create_ready_vm('pool1', 'vm2', redis)
create_ready_vm('pool1', 'vm3') create_ready_vm('pool1', 'vm3', redis)
create_ready_vm('pool2', 'vm4') create_ready_vm('pool2', 'vm4', redis)
create_ready_vm('pool2', 'vm5') create_ready_vm('pool2', 'vm5', redis)
get '/dashboard/stats/vmpooler/pool', :history => true get '/dashboard/stats/vmpooler/pool', :history => true
@ -140,18 +140,18 @@ describe Vmpooler::API do
end end
it 'adds major correctly' do it 'adds major correctly' do
create_running_vm('pool-1', 'vm1') create_running_vm('pool-1', 'vm1', redis)
create_running_vm('pool-1', 'vm2') create_running_vm('pool-1', 'vm2', redis)
create_running_vm('pool-1', 'vm3') create_running_vm('pool-1', 'vm3', redis)
create_running_vm('pool-2', 'vm4') create_running_vm('pool-2', 'vm4', redis)
create_running_vm('pool-2', 'vm5') create_running_vm('pool-2', 'vm5', redis)
create_running_vm('pool-2', 'vm6') create_running_vm('pool-2', 'vm6', redis)
create_running_vm('pool-2', 'vm7') create_running_vm('pool-2', 'vm7', redis)
create_running_vm('pool-2', 'vm8') create_running_vm('pool-2', 'vm8', redis)
create_running_vm('diffpool-1', 'vm9') create_running_vm('diffpool-1', 'vm9', redis)
create_running_vm('diffpool-1', 'vm10') create_running_vm('diffpool-1', 'vm10', redis)
get '/dashboard/stats/vmpooler/running' get '/dashboard/stats/vmpooler/running'

4
spec/unit/api/helpers_spec.rb
View file

@ -128,7 +128,7 @@ describe Vmpooler::API::Helpers do
] ]
allow(redis).to receive(:pipelined).with(no_args).and_return [1,1] allow(redis).to receive(:pipelined).with(no_args).and_return [1,1]
allow(redis).to receive(:get).and_return 1 allow(redis).to receive(:get).and_return(1,0)
expect(subject.get_queue_metrics(pools, redis)).to eq({pending: 2, cloning: 1, booting: 1, ready: 2, running: 2, completed: 2, total: 8}) expect(subject.get_queue_metrics(pools, redis)).to eq({pending: 2, cloning: 1, booting: 1, ready: 2, running: 2, completed: 2, total: 8})
end end
@ -140,7 +140,7 @@ describe Vmpooler::API::Helpers do
] ]
allow(redis).to receive(:pipelined).with(no_args).and_return [1,1] allow(redis).to receive(:pipelined).with(no_args).and_return [1,1]
allow(redis).to receive(:get).and_return 5 allow(redis).to receive(:get).and_return(5,0)
expect(subject.get_queue_metrics(pools, redis)).to eq({pending: 2, cloning: 5, booting: 0, ready: 2, running: 2, completed: 2, total: 8}) expect(subject.get_queue_metrics(pools, redis)).to eq({pending: 2, cloning: 5, booting: 0, ready: 2, running: 2, completed: 2, total: 8})
end end

1413
spec/unit/pool_manager_spec.rb
View file

File diff suppressed because it is too large Load diff

4
spec/unit/providers/base_spec.rb
View file

@ -22,7 +22,9 @@ describe 'Vmpooler::PoolManager::Provider::Base' do
fake_vm fake_vm
} }
subject { Vmpooler::PoolManager::Provider::Base.new(config, logger, metrics, provider_name, provider_options) } let(:redis_connection_pool) { ConnectionPool.new(size: 1) { MockRedis.new } }
subject { Vmpooler::PoolManager::Provider::Base.new(config, logger, metrics, redis_connection_pool, provider_name, provider_options) }
# Helper attr_reader methods # Helper attr_reader methods
describe '#logger' do describe '#logger' do

4
spec/unit/providers/dummy_spec.rb
View file

@ -91,7 +91,9 @@ EOT
) )
} }
subject { Vmpooler::PoolManager::Provider::Dummy.new(config, logger, metrics, 'dummy', provider_options) } let(:redis_connection_pool) { ConnectionPool.new(size: 1) { MockRedis.new } }
subject { Vmpooler::PoolManager::Provider::Dummy.new(config, logger, metrics, redis_connection_pool, 'dummy', provider_options) }
describe '#name' do describe '#name' do
it 'should be dummy' do it 'should be dummy' do

39
spec/unit/providers/vsphere_spec.rb
View file

@ -77,9 +77,15 @@ EOT
let(:connection_options) {{}} let(:connection_options) {{}}
let(:connection) { mock_RbVmomi_VIM_Connection(connection_options) } let(:connection) { mock_RbVmomi_VIM_Connection(connection_options) }
let(:vmname) { 'vm1' } let(:vmname) { 'vm1' }
let(:redis) { MockRedis.new } let(:redis_connection_pool) { Vmpooler::PoolManager::GenericConnectionPool.new(
metrics: metrics,
metric_prefix: 'redis_connection_pool',
size: 1,
timeout: 5
) { MockRedis.new }
}
subject { Vmpooler::PoolManager::Provider::VSphere.new(config, logger, metrics, 'vsphere', provider_options) } subject { Vmpooler::PoolManager::Provider::VSphere.new(config, logger, metrics, redis_connection_pool, 'vsphere', provider_options) }
before(:each) do before(:each) do
allow(subject).to receive(:vsphere_connection_ok?).and_return(true) allow(subject).to receive(:vsphere_connection_ok?).and_return(true)
@ -149,26 +155,19 @@ EOT
allow(destroy_task).to receive(:wait_for_completion) allow(destroy_task).to receive(:wait_for_completion)
allow(vm_object).to receive(:PowerOffVM_Task).and_return(power_off_task) allow(vm_object).to receive(:PowerOffVM_Task).and_return(power_off_task)
allow(vm_object).to receive(:Destroy_Task).and_return(destroy_task) allow(vm_object).to receive(:Destroy_Task).and_return(destroy_task)
$redis = redis
end
it 'should remove redis data and expire the vm key' do
allow(Time).to receive(:now).and_return(now)
expect(redis).to receive(:srem).with("vmpooler__completed__#{pool}", vmname)
expect(redis).to receive(:hdel).with("vmpooler__active__#{pool}", vmname)
expect(redis).to receive(:hset).with("vmpooler__vm__#{vmname}", 'destroy', now)
expect(redis).to receive(:expire).with("vmpooler__vm__#{vmname}", data_ttl * 60 * 60)
subject.destroy_vm_and_log(vmname, vm_object, pool, data_ttl)
end end
it 'should log a message that the vm is destroyed' do it 'should log a message that the vm is destroyed' do
# Ensure Time returns a consistent value so finish is predictable
# Otherwise finish occasionally increases to 0.01 and causes a failure
allow(Time).to receive(:now).and_return(Time.now)
expect(logger).to receive(:log).with('s', "[-] [#{pool}] '#{vmname}' destroyed in #{finish} seconds") expect(logger).to receive(:log).with('s', "[-] [#{pool}] '#{vmname}' destroyed in #{finish} seconds")
subject.destroy_vm_and_log(vmname, vm_object, pool, data_ttl) subject.destroy_vm_and_log(vmname, vm_object, pool, data_ttl)
end end
it 'should record metrics' do it 'should record metrics' do
expect(metrics).to receive(:timing).with('redis_connection_pool.waited', 0)
expect(metrics).to receive(:timing).with("destroy.#{pool}", finish) expect(metrics).to receive(:timing).with("destroy.#{pool}", finish)
subject.destroy_vm_and_log(vmname, vm_object, pool, data_ttl) subject.destroy_vm_and_log(vmname, vm_object, pool, data_ttl)
@ -2573,10 +2572,12 @@ EOT
subject.connection_pool.with_metrics do |pool_object| subject.connection_pool.with_metrics do |pool_object|
expect(subject).to receive(:ensured_vsphere_connection).with(pool_object).and_return(connection) expect(subject).to receive(:ensured_vsphere_connection).with(pool_object).and_return(connection)
expect(subject).to receive(:get_vm_details).and_return(vm_details) expect(subject).to receive(:get_vm_details).and_return(vm_details)
allow($redis).to receive(:hset)
expect(subject).to receive(:run_select_hosts).with(poolname, {}) expect(subject).to receive(:run_select_hosts).with(poolname, {})
expect(subject).to receive(:vm_in_target?).and_return false expect(subject).to receive(:vm_in_target?).and_return false
expect(subject).to receive(:migration_enabled?).and_return true expect(subject).to receive(:migration_enabled?).and_return true
redis_connection_pool.with do |redis|
redis.hset("vmpooler__vm__#{vmname}", 'checkout', Time.now)
end
end end
vm_object.summary.runtime.host = host_object vm_object.summary.runtime.host = host_object
end end
@ -2585,7 +2586,6 @@ EOT
expect(subject).to receive(:select_next_host).and_return(new_host) expect(subject).to receive(:select_next_host).and_return(new_host)
expect(subject).to receive(:find_host_by_dnsname).and_return(new_host_object) expect(subject).to receive(:find_host_by_dnsname).and_return(new_host_object)
expect(subject).to receive(:migrate_vm_host).with(vm_object, new_host_object) expect(subject).to receive(:migrate_vm_host).with(vm_object, new_host_object)
expect($redis).to receive(:hget).with("vmpooler__vm__#{vmname}", 'checkout').and_return((Time.now - 1).to_s)
expect(logger).to receive(:log).with('s', "[>] [#{poolname}] '#{vmname}' migrated from #{parent_host} to #{new_host} in 0.00 seconds") expect(logger).to receive(:log).with('s', "[>] [#{poolname}] '#{vmname}' migrated from #{parent_host} to #{new_host} in 0.00 seconds")
subject.migrate_vm(poolname, vmname) subject.migrate_vm(poolname, vmname)
@ -2603,7 +2603,6 @@ EOT
subject.connection_pool.with_metrics do |pool_object| subject.connection_pool.with_metrics do |pool_object|
expect(subject).to receive(:ensured_vsphere_connection).with(pool_object).and_return(connection) expect(subject).to receive(:ensured_vsphere_connection).with(pool_object).and_return(connection)
expect(subject).to receive(:get_vm_details).and_return(vm_details) expect(subject).to receive(:get_vm_details).and_return(vm_details)
expect($redis).to receive(:hset).with("vmpooler__vm__#{vmname}", 'host', parent_host)
expect(subject).to receive(:run_select_hosts).with(poolname, {}) expect(subject).to receive(:run_select_hosts).with(poolname, {})
expect(subject).to receive(:vm_in_target?).and_return true expect(subject).to receive(:vm_in_target?).and_return true
expect(subject).to receive(:migration_enabled?).and_return true expect(subject).to receive(:migration_enabled?).and_return true
@ -2622,11 +2621,13 @@ EOT
before(:each) do before(:each) do
subject.connection_pool.with_metrics do |pool_object| subject.connection_pool.with_metrics do |pool_object|
expect(subject).to receive(:ensured_vsphere_connection).with(pool_object).and_return(connection) expect(subject).to receive(:ensured_vsphere_connection).with(pool_object).and_return(connection)
expect($redis).to receive(:scard).with('vmpooler__migration').and_return(5)
expect(subject).to receive(:get_vm_details).and_return(vm_details) expect(subject).to receive(:get_vm_details).and_return(vm_details)
expect(subject).to_not receive(:run_select_hosts) expect(subject).to_not receive(:run_select_hosts)
expect(subject).to_not receive(:vm_in_target?) expect(subject).to_not receive(:vm_in_target?)
expect(subject).to receive(:migration_enabled?).and_return true expect(subject).to receive(:migration_enabled?).and_return true
redis_connection_pool.with do |redis|
expect(redis).to receive(:scard).with('vmpooler__migration').and_return(5)
end
end end
vm_object.summary.runtime.host = host_object vm_object.summary.runtime.host = host_object
end end
@ -2691,14 +2692,10 @@ EOT
end end
it' migrates a vm' do it' migrates a vm' do
expect($redis).to receive(:sadd).with('vmpooler__migration', vmname)
expect(subject).to receive(:select_next_host).and_return(new_host) expect(subject).to receive(:select_next_host).and_return(new_host)
expect($redis).to receive(:hset).with("vmpooler__vm__#{vmname}", 'host', new_host)
expect($redis).to receive(:hset).with("vmpooler__vm__#{vmname}", 'migrated', true)
expect(subject).to receive(:find_host_by_dnsname).and_return(new_host_object) expect(subject).to receive(:find_host_by_dnsname).and_return(new_host_object)
expect(subject).to receive(:migrate_vm_and_record_timing).and_return(format('%.2f', (Time.now - (Time.now - 15)))) expect(subject).to receive(:migrate_vm_and_record_timing).and_return(format('%.2f', (Time.now - (Time.now - 15))))
expect(logger).to receive(:log).with('s', "[>] [#{poolname}] '#{vmname}' migrated from host1 to host2 in 15.00 seconds") expect(logger).to receive(:log).with('s', "[>] [#{poolname}] '#{vmname}' migrated from host1 to host2 in 15.00 seconds")
expect($redis).to receive(:srem).with('vmpooler__migration', vmname)
subject.migrate_vm_to_new_host(poolname, vmname, vm_details, connection) subject.migrate_vm_to_new_host(poolname, vmname, vm_details, connection)
end end
end end

1
vmpooler.gemspec
View file

@ -27,6 +27,7 @@ Gem::Specification.new do |s|
s.add_dependency 'net-ldap', '~> 0.16' s.add_dependency 'net-ldap', '~> 0.16'
s.add_dependency 'statsd-ruby', '~> 1.4' s.add_dependency 'statsd-ruby', '~> 1.4'
s.add_dependency 'connection_pool', '~> 2.2' s.add_dependency 'connection_pool', '~> 2.2'
s.add_dependency 'concurrent-ruby', '~> 1.1'
s.add_dependency 'nokogiri', '~> 1.10' s.add_dependency 'nokogiri', '~> 1.10'
s.add_dependency 'spicy-proton', '~> 2.1' s.add_dependency 'spicy-proton', '~> 2.1'
end end

36
vmpooler.yaml.example
View file

@ -195,14 +195,21 @@
# #
# - data_ttl # - data_ttl
# How long (in hours) to retain metadata in Redis after VM destruction. # How long (in hours) to retain metadata in Redis after VM destruction.
# (optional; default: '168') # (default: 168)
#
# - redis_connection_pool_size
# Maximum number of connections to utilize for the redis connection pool.
# (default: 10)
#
# - redis_connection_pool_timeout
# How long a task should wait (in seconds) for a redis connection when all connections are in use.
# (default: 5)
# Example: # Example:
:redis: :redis:
server: 'redis.example.com' server: 'redis.example.com'
# :graphs: # :graphs:
# #
# This section contains the server and prefix information for a graphite- # This section contains the server and prefix information for a graphite-
@ -368,15 +375,19 @@
# #
# - task_limit # - task_limit
# The number of concurrent VM creation tasks to perform. # The number of concurrent VM creation tasks to perform.
# (optional; default: '10') # (default: 10)
#
# - ondemand_clone_limit
# The number of concurrent VM creation tasks to perform for ondemand VM requests.
# (default: 10)
# #
# - timeout # - timeout
# How long (in minutes) before marking a clone in 'pending' queues as 'failed' and retrying. # How long (in minutes) before marking a clone in 'pending' queues as 'failed' and retrying.
# (optional; default: '15') # (default: 15)
# #
# - vm_checktime # - vm_checktime
# How often (in minutes) to check the sanity of VMs in 'ready' queues. # How often (in minutes) to check the sanity of VMs in 'ready' queues.
# (optional; default: '1') # (default: 1)
# #
# - vm_lifetime # - vm_lifetime
# How long (in hours) to keep VMs in 'running' queues before destroying. # How long (in hours) to keep VMs in 'running' queues before destroying.
@ -510,10 +521,21 @@
# Expects a string value # Expects a string value
# (optional) # (optional)
# #
# - max_ondemand_instances_per_request
# The maximum number of instances any individual ondemand request may contain per pool.
# (default: 10)
#
# - ondemand_request_ttl
# The amount of time (in minutes) to give for a ondemand request to be fulfilled before considering it to have failed.
# (default: 5)
#
# - ready_ttl
# How long (in minutes) a ready VM should stay in the ready queue.
# (default: 60)
#
# Example: # Example:
:config: :config: site_name: 'vmpooler'
site_name: 'vmpooler'
logfile: '/var/log/vmpooler.log' logfile: '/var/log/vmpooler.log'
task_limit: 10 task_limit: 10
timeout: 15 timeout: 15