Merge pull request #256 from mattkirby/config_changes_squash

(POOLER-107) Add configuration API endpoint

docs/API.md

@@ -1,8 +1,17 @@
-### API
+# Table of contents
+1. [API](#API)
+2. [Token operations](#token)
+3. [VM operations](#vmops)
+4. [Add disks](#adddisks)
+5. [VM snapshots](#vmsnapshots)
+6. [Status and metrics](#statusmetrics)
+7. [Pool configuration](#poolconfig)
+
+### API <a name="API"></a>
 
 vmpooler provides a REST API for VM management.  The following examples use `curl` for communication.
 
-#### Token operations
+#### Token operations <a name="token"></a>
 
 Token-based authentication can be used when requesting or modifying VMs.  The `/token` route can be used to create, query, or delete tokens.  See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), for information on configuring an authentication store to use when performing token operations.
 
@@ -76,7 +85,7 @@ Enter host password for user 'jdoe':
 }
 ```
 
-#### VM operations
+#### VM operations <a name="vmops"></a>
 
 ##### GET /vm
 
@@ -230,7 +239,7 @@ $ curl -X DELETE --url vmpooler.company.com/api/v1/vm/fq6qlpjlsskycq6
 }
 ```
 
-#### Adding additional disk(s)
+#### Adding additional disk(s) <a name="adddisks"></a>
 
 ##### POST /vm/&lt;hostname&gt;/disk/&lt;size&gt;
 
@@ -270,7 +279,7 @@ $ curl --url vmpooler.company.com/api/v1/vm/fq6qlpjlsskycq6
 
 ````
 
-#### VM snapshots
+#### VM snapshots <a name="vmsnapshots"></a>
 
 ##### POST /vm/&lt;hostname&gt;/snapshot
 
@@ -322,7 +331,7 @@ $ curl X POST -H X-AUTH-TOKEN:a9znth9dn01t416hrguu56ze37t790bl --url vmpooler.co
 }
 ````
 
-#### Status and metrics
+#### Status and metrics <a name="statusmetrics"></a>
 
 ##### GET /status
 
@@ -540,3 +549,97 @@ $ curl -G -d 'from=2015-03-10' -d 'to=2015-03-11' --url vmpooler.company.com/api
   ]
 }
 ```
+
+#### Managing pool configuration via API <a name="poolconfig"></a>
+
+##### GET /config
+
+Returns the running pool configuration
+
+Responses:
+* 200 - OK
+* 404 - No configuration found
+```
+$ curl https://vmpooler.company.com/api/v1/config
+```
+```json
+{
+  "pool_configuration": [
+    {
+      "name": "redhat-7-x86_64",
+      "template": "templates/redhat-7.2-x86_64-0.0.3",
+      "folder": "vmpooler/redhat-7-x86_64",
+      "datastore": "stor1",
+      "size": 1,
+      "datacenter": "dc1",
+      "provider": "vsphere",
+      "capacity": 1,
+      "major": "redhat",
+      "template_ready": true
+    }
+  ],
+  "status": {
+    "ok": true
+  }
+}
+```
+
+Note: to enable poolsize and pooltemplate config endpoints it is necessary to set 'experimental_features: true' in your vmpooler configuration. A 405 is returned when you attempt to interact with these endpoints when this configuration option is not set.
+
+##### POST /config/poolsize
+
+Change pool size without having to restart the service.
+
+All pool template changes requested must be for pools that exist in the vmpooler configuration running, or a 404 code will be returned
+
+When a pool size is changed due to the configuration posted a 201 status will be returned. When the pool configuration is valid, but will not result in any changes, 200 is returned.
+
+Pool size configuration changes persist through application restarts, and take precedence over a pool size value configured in the pool configuration provided when the application starts. This persistence is dependent on redis. So, if the redis data is lost then the configuration updates revert to those provided at startup at the next application start.
+
+An authentication token is required in order to change pool configuration when authentication is configured.
+Responses:
+* 200 - No changes required
+* 201 - Changes made on at least one pool with changes requested
+* 400 - An invalid configuration was provided causing requested changes to fail
+* 404 - An unknown error occurred
+* 405 - The endpoint is disabled because experimental features are disabled
+```
+$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"2","debian-7-x86_64":"1"}' --url https://vmpooler.company.com/api/v1/config/poolsize
+```
+```json
+{
+  "ok": true
+}
+```
+
+##### POST /config/pooltemplate
+
+Change the template configured for a pool, and replenish the pool with instances built from the new template.
+
+All pool template changes requested must be for pools that exist in the vmpooler configuration running, or a 404 code will be returned
+
+When a pool template is changed due to the configuration posted a 201 status will be returned. When the pool configuration is valid, but will not result in any changes, 200 is returned.
+
+A pool template being updated will cause the following actions, which are logged in vmpooler.log:
+* Destroy all instances for the pool template being updated that are in the ready and pending state
+* Halt repopulating the pool while creating template deltas for the newly configured template
+* Unblock pool population and let the pool replenish with instances based on the newly configured template
+
+Pool template changes persist through application restarts, and take precedence over a pool template configured in the pool configuration provided when the application starts. This persistence is dependent on redis. As a result, if the redis data is lost then the configuration values revert to those provided at startup at the next application start.
+
+An authentication token is required in order to change pool configuration when authentication is configured.
+
+Responses:
+* 200 - No changes required
+* 201 - Changes made on at least one pool with changes requested
+* 400 - An invalid configuration was provided causing requested changes to fail
+* 404 - An unknown error occurred
+* 405 - The endpoint is disabled because experimental features are disabled
+```
+$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"templates/debian-7-i386"}' --url https://vmpooler.company.com/api/v1/config/pooltemplate
+```
+```json
+{
+  "ok": true
+}
+```

lib/vmpooler/api/helpers.rb

@@ -378,6 +378,29 @@ module Vmpooler
         result
       end
 
+      def pool_index(pools)
+        pools_hash = {}
+        index = 0
+        for pool in pools
+          pools_hash[pool['name']] = index
+          index += 1
+        end
+        pools_hash
+      end
+
+      def template_ready?(pool, backend)
+        prepared_template = backend.hget('vmpooler__template__prepared', pool['name'])
+        return false if prepared_template.nil?
+        return true if pool['template'] == prepared_template
+        return false
+      end
+
+      def is_integer?(x)
+        Integer(x)
+        true
+      rescue
+        false
+      end
     end
   end
 end

lib/vmpooler/api/v1.rb

@@ -120,6 +120,74 @@ module Vmpooler
       result
     end
 
+    def update_pool_size(payload)
+      result = { 'ok' => false }
+
+      pool_index = pool_index(pools)
+      pools_updated = 0
+      sync_pool_sizes
+
+      payload.each do |poolname, size|
+        unless pools[pool_index[poolname]]['size'] == size.to_i
+          pools[pool_index[poolname]]['size'] = size.to_i
+          backend.hset('vmpooler__config__poolsize', poolname, size)
+          pools_updated += 1
+          status 201
+        end
+      end
+      status 200 unless pools_updated > 0
+      result['ok'] = true
+      result
+    end
+
+    def update_pool_template(payload)
+      result = { 'ok' => false }
+
+      pool_index = pool_index(pools)
+      pools_updated = 0
+      sync_pool_templates
+
+      payload.each do |poolname, template|
+        unless pools[pool_index[poolname]]['template'] == template
+          pools[pool_index[poolname]]['template'] = template
+          backend.hset('vmpooler__config__template', poolname, template)
+          pools_updated += 1
+          status 201
+        end
+      end
+      status 200 unless pools_updated > 0
+      result['ok'] = true
+      result
+    end
+
+    def sync_pool_templates
+      pool_index = pool_index(pools)
+      template_configs = backend.hgetall('vmpooler__config__template')
+      unless template_configs.nil?
+        template_configs.each do |poolname, template|
+          if pool_index.include? poolname
+            unless pools[pool_index[poolname]]['template'] == template
+              pools[pool_index[poolname]]['template'] = template
+            end
+          end
+        end
+      end
+    end
+
+    def sync_pool_sizes
+      pool_index = pool_index(pools)
+      poolsize_configs = backend.hgetall('vmpooler__config__poolsize')
+      unless poolsize_configs.nil?
+        poolsize_configs.each do |poolname, size|
+          if pool_index.include? poolname
+            unless pools[pool_index[poolname]]['size'] == size.to_i
+              pools[pool_index[poolname]]['size'] == size.to_i
+            end
+          end
+        end
+      end
+    end
+
     # Provide run-time statistics
     #
     # Example:
@@ -196,6 +264,8 @@ module Vmpooler
         }
       }
 
+      sync_pool_sizes
+
       result[:capacity] = get_capacity_metrics(pools, backend) unless views and not views.include?("capacity")
       result[:queue] = get_queue_metrics(pools, backend) unless views and not views.include?("queue")
       result[:clone] = get_task_metrics(backend, 'clone', Date.today.to_s) unless views and not views.include?("clone")
@@ -502,6 +572,30 @@ module Vmpooler
       invalid
     end
 
+    def invalid_template_or_size(payload)
+      invalid = []
+      payload.each do |pool, size|
+        invalid << pool unless pool_exists?(pool)
+        unless is_integer?(size)
+          invalid << pool
+          next
+        end
+        invalid << pool unless Integer(size) >= 0
+      end
+      invalid
+    end
+
+    def invalid_template_or_path(payload)
+      invalid = []
+      payload.each do |pool, template|
+        invalid << pool unless pool_exists?(pool)
+        invalid << pool unless template.include? '/'
+        invalid << pool if template[0] == '/'
+        invalid << pool if template[-1] == '/'
+      end
+      invalid
+    end
+
     post "#{api_prefix}/vm/:template/?" do
       content_type :json
       result = { 'ok' => false }
@@ -747,6 +841,95 @@ module Vmpooler
 
       JSON.pretty_generate(result)
     end
+
+    post "#{api_prefix}/config/poolsize/?" do
+      content_type :json
+      result = { 'ok' => false }
+
+      if config['experimental_features']
+        need_token! if Vmpooler::API.settings.config[:auth]
+
+        payload = JSON.parse(request.body.read)
+
+        if payload
+          invalid = invalid_template_or_size(payload)
+          if invalid.empty?
+            result = update_pool_size(payload)
+          else
+            invalid.each do |bad_template|
+              metrics.increment("config.invalid.#{bad_template}")
+            end
+            result[:bad_templates] = invalid
+            status 400
+          end
+        else
+          metrics.increment('config.invalid.unknown')
+          status 404
+        end
+      else
+        status 405
+      end
+
+      JSON.pretty_generate(result)
+    end
+
+    post "#{api_prefix}/config/pooltemplate/?" do
+      content_type :json
+      result = { 'ok' => false }
+
+      if config['experimental_features']
+        need_token! if Vmpooler::API.settings.config[:auth]
+
+        payload = JSON.parse(request.body.read)
+
+        if payload
+          invalid = invalid_template_or_path(payload)
+          if invalid.empty?
+            result = update_pool_template(payload)
+          else
+            invalid.each do |bad_template|
+              metrics.increment("config.invalid.#{bad_template}")
+            end
+            result[:bad_templates] = invalid
+            status 400
+          end
+        else
+          metrics.increment('config.invalid.unknown')
+          status 404
+        end
+      else
+        status 405
+      end
+
+      JSON.pretty_generate(result)
+    end
+
+    get "#{api_prefix}/config/?" do
+      content_type :json
+      result = { 'ok' => false }
+      status 404
+
+      if pools
+        sync_pool_sizes
+        sync_pool_templates
+
+        pool_configuration = []
+        pools.each do |pool|
+          pool['template_ready'] = template_ready?(pool, backend)
+          pool_configuration << pool
+        end
+
+        result = {
+          pool_configuration: pool_configuration,
+          status: {
+            ok: true
+          }
+        }
+
+        status 200
+      end
+      JSON.pretty_generate(result)
+    end
   end
   end
 end

lib/vmpooler/pool_manager.rb

@@ -21,6 +21,9 @@ module Vmpooler
 
       # Our thread-tracker object
       $threads = {}
+
+      # Pool mutex
+      @reconfigure_pool = {}
     end
 
     def config
@@ -187,9 +190,9 @@ module Vmpooler
       end
     end
 
-    def move_vm_queue(pool, vm, queue_from, queue_to, msg)
+    def move_vm_queue(pool, vm, queue_from, queue_to, msg = nil)
       $redis.smove("vmpooler__#{queue_from}__#{pool}", "vmpooler__#{queue_to}__#{pool}", vm)
-      $logger.log('d', "[!] [#{pool}] '#{vm}' #{msg}")
+      $logger.log('d', "[!] [#{pool}] '#{vm}' #{msg}") if msg
     end
 
     # Clone a VM
@@ -482,6 +485,10 @@ module Vmpooler
     #   - Fires when the number of ready VMs changes due to being consumed.
     #   - Additional options
     #       :poolname
+    # :pool_template_change
+    #   - Fires when a template configuration update is requested
+    #   - Additional options
+    #       :poolname
     #
     def sleep_with_wakeup_events(loop_delay, wakeup_period = 5, options = {})
       exit_by = Time.now + loop_delay
@@ -492,6 +499,10 @@ module Vmpooler
         initial_ready_size = $redis.scard("vmpooler__ready__#{options[:poolname]}")
       end
 
+      if options[:pool_template_change]
+        initial_template = $redis.hget('vmpooler__template__prepared', options[:poolname])
+      end
+
       loop do
         sleep(1)
         break if time_passed?(:exit_by, exit_by)
@@ -505,6 +516,14 @@ module Vmpooler
             ready_size = $redis.scard("vmpooler__ready__#{options[:poolname]}")
             break unless ready_size == initial_ready_size
           end
+
+          if options[:pool_template_change]
+            configured_template = $redis.hget('vmpooler__config__template', options[:poolname])
+            if configured_template
+              break unless initial_template == configured_template
+            end
+          end
+
         end
 
         break if time_passed?(:exit_by, exit_by)
@@ -532,6 +551,7 @@ module Vmpooler
           loop_delay = loop_delay_min
           provider = get_provider_for_pool(pool['name'])
           raise("Could not find provider '#{pool['provider']}") if provider.nil?
+          sync_pool_template(pool)
           loop do
             result = _check_pool(pool, provider)
 
@@ -541,7 +561,7 @@ module Vmpooler
               loop_delay = (loop_delay * loop_delay_decay).to_i
               loop_delay = loop_delay_max if loop_delay > loop_delay_max
             end
-            sleep_with_wakeup_events(loop_delay, loop_delay_min, pool_size_change: true, poolname: pool['name'])
+            sleep_with_wakeup_events(loop_delay, loop_delay_min, pool_size_change: true, poolname: pool['name'], pool_template_change: true)
 
             unless maxloop.zero?
               break if loop_count >= maxloop
@@ -555,6 +575,101 @@ module Vmpooler
       end
     end
 
+    def pool_mutex(poolname)
+      @reconfigure_pool[poolname] || @reconfigure_pool[poolname] = Mutex.new
+    end
+
+    def sync_pool_template(pool)
+      pool_template = $redis.hget('vmpooler__config__template', pool['name'])
+      if pool_template
+        unless pool['template'] == pool_template
+          pool['template'] = pool_template
+        end
+      end
+    end
+
+    def prepare_template(pool, provider)
+      provider.create_template_delta_disks(pool) if $config[:config]['create_template_delta_disks']
+      $redis.hset('vmpooler__template__prepared', pool['name'], pool['template'])
+    end
+
+    def evaluate_template(pool, provider)
+      mutex = pool_mutex(pool['name'])
+      prepared_template = $redis.hget('vmpooler__template__prepared', pool['name'])
+      configured_template = $redis.hget('vmpooler__config__template', pool['name'])
+      return if mutex.locked?
+      if prepared_template.nil?
+        mutex.synchronize do
+          prepare_template(pool, provider)
+          prepared_template = $redis.hget('vmpooler__template__prepared', pool['name'])
+        end
+      end
+      return if configured_template.nil?
+      return if configured_template == prepared_template
+      mutex.synchronize do
+        update_pool_template(pool, provider, configured_template, prepared_template)
+      end
+    end
+
+    def drain_pool(poolname)
+      # Clear a pool of ready and pending instances
+      if $redis.scard("vmpooler__ready__#{poolname}") > 0
+        $logger.log('s', "[*] [#{poolname}] removing ready instances")
+        $redis.smembers("vmpooler__ready__#{poolname}").each do |vm|
+          move_vm_queue(poolname, vm, 'ready', 'completed')
+        end
+      end
+      if $redis.scard("vmpooler__pending__#{poolname}") > 0
+        $logger.log('s', "[*] [#{poolname}] removing pending instances")
+        $redis.smembers("vmpooler__pending__#{poolname}").each do |vm|
+          move_vm_queue(poolname, vm, 'pending', 'completed')
+        end
+      end
+    end
+
+    def update_pool_template(pool, provider, configured_template, prepared_template)
+      pool['template'] = configured_template
+      $logger.log('s', "[*] [#{pool['name']}] template updated from #{prepared_template} to #{configured_template}")
+      # Remove all ready and pending VMs so new instances are created from the new template
+      drain_pool(pool['name'])
+      # Prepare template for deployment
+      $logger.log('s', "[*] [#{pool['name']}] preparing pool template for deployment")
+      prepare_template(pool, provider)
+      $logger.log('s', "[*] [#{pool['name']}] is ready for use")
+    end
+
+    def remove_excess_vms(pool, provider, ready, total)
+      return if total.nil?
+      return if total == 0
+      mutex = pool_mutex(pool['name'])
+      return if mutex.locked?
+      return unless ready > pool['size']
+      mutex.synchronize do
+        difference = ready - pool['size']
+        difference.times do
+          next_vm = $redis.spop("vmpooler__ready__#{pool['name']}")
+          move_vm_queue(pool['name'], next_vm, 'ready', 'completed')
+        end
+        if total > ready
+          $redis.smembers("vmpooler__pending__#{pool['name']}").each do |vm|
+            move_vm_queue(pool['name'], vm, 'pending', 'completed')
+          end
+        end
+      end
+    end
+
+    def update_pool_size(pool)
+      mutex = pool_mutex(pool['name'])
+      return if mutex.locked?
+      poolsize = $redis.hget('vmpooler__config__poolsize', pool['name'])
+      return if poolsize.nil?
+      poolsize = Integer(poolsize)
+      return if poolsize == pool['size']
+      mutex.synchronize do
+        pool['size'] = poolsize
+      end
+    end
+
     def _check_pool(pool, provider)
       pool_check_response = {
         discovered_vms: 0,
@@ -683,36 +798,53 @@ module Vmpooler
         end
       end
 
+      # UPDATE TEMPLATE
+      # Evaluates a pool template to ensure templates are prepared adequately for the configured provider
+      # If a pool template configuration change is detected then template preparation is repeated for the new template
+      # Additionally, a pool will drain ready and pending instances
+      evaluate_template(pool, provider)
+
       # REPOPULATE
-      ready = $redis.scard("vmpooler__ready__#{pool['name']}")
+      # Do not attempt to repopulate a pool while a template is updating
-      total = $redis.scard("vmpooler__pending__#{pool['name']}") + ready
+      unless pool_mutex(pool['name']).locked?
+        ready = $redis.scard("vmpooler__ready__#{pool['name']}")
+        total = $redis.scard("vmpooler__pending__#{pool['name']}") + ready
 
-      $metrics.gauge("ready.#{pool['name']}", $redis.scard("vmpooler__ready__#{pool['name']}"))
+        $metrics.gauge("ready.#{pool['name']}", $redis.scard("vmpooler__ready__#{pool['name']}"))
-      $metrics.gauge("running.#{pool['name']}", $redis.scard("vmpooler__running__#{pool['name']}"))
+        $metrics.gauge("running.#{pool['name']}", $redis.scard("vmpooler__running__#{pool['name']}"))
 
-      if $redis.get("vmpooler__empty__#{pool['name']}")
+        if $redis.get("vmpooler__empty__#{pool['name']}")
-        $redis.del("vmpooler__empty__#{pool['name']}") unless ready.zero?
+          $redis.del("vmpooler__empty__#{pool['name']}") unless ready.zero?
-      elsif ready.zero?
+        elsif ready.zero?
-        $redis.set("vmpooler__empty__#{pool['name']}", 'true')
+          $redis.set("vmpooler__empty__#{pool['name']}", 'true')
-        $logger.log('s', "[!] [#{pool['name']}] is empty")
+          $logger.log('s', "[!] [#{pool['name']}] is empty")
-      end
+        end
 
-      if total < pool['size']
+        # Check to see if a pool size change has been made via the configuration API
-        (1..(pool['size'] - total)).each do |_i|
+        # Since check_pool runs in a loop it does not
-          if $redis.get('vmpooler__tasks__clone').to_i < $config[:config]['task_limit'].to_i
+        # otherwise identify this change when running
-            begin
+        update_pool_size(pool)
-              $redis.incr('vmpooler__tasks__clone')
+
-              pool_check_response[:cloned_vms] += 1
+        if total < pool['size']
-              clone_vm(pool, provider)
+          (1..(pool['size'] - total)).each do |_i|
-            rescue => err
+            if $redis.get('vmpooler__tasks__clone').to_i < $config[:config]['task_limit'].to_i
-              $logger.log('s', "[!] [#{pool['name']}] clone failed during check_pool with an error: #{err}")
+              begin
-              $redis.decr('vmpooler__tasks__clone')
+                $redis.incr('vmpooler__tasks__clone')
-              raise
+                pool_check_response[:cloned_vms] += 1
+                clone_vm(pool, provider)
+              rescue => err
+                $logger.log('s', "[!] [#{pool['name']}] clone failed during check_pool with an error: #{err}")
+                $redis.decr('vmpooler__tasks__clone')
+                raise
+              end
             end
           end
         end
       end
 
+      # Remove VMs in excess of the configured pool size
+      remove_excess_vms(pool, provider, ready, total)
+
       pool_check_response
     end
 
@@ -739,6 +871,8 @@ module Vmpooler
       $redis.set('vmpooler__tasks__clone', 0)
       # Clear out vmpooler__migrations since stale entries may be left after a restart
       $redis.del('vmpooler__migration')
+      # Ensure template deltas are created on each startup
+      $redis.del('vmpooler__template__prepared')
 
       # Copy vSphere settings to correct location.  This happens with older configuration files
       if !$config[:vsphere].nil? && ($config[:providers].nil? || $config[:providers][:vsphere].nil?)

lib/vmpooler/providers/base.rb

@@ -217,6 +217,14 @@ module Vmpooler
         def vm_exists?(pool_name, vm_name)
           !get_vm(pool_name, vm_name).nil?
         end
+
+        # inputs
+        #   [Hash] pool : Configuration for the pool
+        # returns
+        #   nil when successful. Raises error when encountered
+        def create_template_delta_disks(pool)
+          raise("#{self.class.name} does not implement create_template_delta_disks")
+        end
       end
     end
   end

lib/vmpooler/providers/vsphere.rb

@@ -197,17 +197,10 @@ module Vmpooler
             target_cluster_name = get_target_cluster_from_config(pool_name)
             target_datacenter_name = get_target_datacenter_from_config(pool_name)
 
-            # Extract the template VM name from the full path
+            # Get the template VM object
-            raise("Pool #{pool_name} did not specify a full path for the template for the provider #{name}") unless template_path =~ /\//
+            raise("Pool #{pool_name} did not specify a full path for the template for the provider #{name}") unless valid_template_path? template_path
-            templatefolders = template_path.split('/')
-            template_name = templatefolders.pop
 
-            # Get the actual objects from vSphere
+            template_vm_object = find_template_vm(pool, connection)
-            template_folder_object = find_folder(templatefolders.join('/'), connection, target_datacenter_name)
-            raise("Pool #{pool_name} specifies a template folder of #{templatefolders.join('/')} which does not exist for the provider #{name}") if template_folder_object.nil?
-
-            template_vm_object = template_folder_object.find(template_name)
-            raise("Pool #{pool_name} specifies a template VM of #{template_name} which does not exist for the provider #{name}") if template_vm_object.nil?
 
             # Annotate with creation time, origin template, etc.
             # Add extraconfig options that can be queried by vmtools
@@ -933,6 +926,37 @@ module Vmpooler
           raise("Cannot create folder #{new_folder}") if folder_object.nil?
           folder_object
         end
+
+        def find_template_vm(pool, connection)
+          datacenter = get_target_datacenter_from_config(pool['name'])
+          raise('cannot find datacenter') if datacenter.nil?
+
+          propSpecs = {
+            :entity => self,
+            :inventoryPath => "#{datacenter}/vm/#{pool['template']}"
+          }
+
+          template_vm_object = connection.searchIndex.FindByInventoryPath(propSpecs)
+          raise("Pool #{pool['name']} specifies a template VM of #{pool['template']} which does not exist for the provider #{name}") if template_vm_object.nil?
+
+          template_vm_object
+        end
+
+        def create_template_delta_disks(pool)
+          @connection_pool.with_metrics do |pool_object|
+            connection = ensured_vsphere_connection(pool_object)
+            template_vm_object = find_template_vm(pool, connection)
+
+            template_vm_object.add_delta_disk_layer_on_all_disks
+          end
+        end
+
+        def valid_template_path?(template)
+          return false unless template.include?('/')
+          return false if template[0] == '/'
+          return false if template[-1] == '/'
+          return true
+        end
       end
     end
   end

spec/integration/api/v1/config_spec.rb

@@ -0,0 +1,250 @@
+require 'spec_helper'
+require 'rack/test'
+
+module Vmpooler
+  class API
+    module Helpers
+      def authenticate(auth, username_str, password_str)
+        username_str == 'admin' and password_str == 's3cr3t'
+      end
+    end
+  end
+end
+
+describe Vmpooler::API::V1 do
+  include Rack::Test::Methods
+
+  def app()
+    Vmpooler::API
+  end
+
+  let(:config) {
+    {
+      config: {
+        'site_name' => 'test pooler',
+        'vm_lifetime_auth' => 2,
+        'experimental_features' => true
+      },
+      pools: [
+        {'name' => 'pool1', 'size' => 5, 'template' => 'templates/pool1'},
+        {'name' => 'pool2', 'size' => 10}
+      ],
+      statsd: { 'prefix' => 'stats_prefix'},
+      alias: { 'poolone' => 'pool1' },
+      pool_names: [ 'pool1', 'pool2', 'poolone' ]
+    }
+  }
+
+  describe '/config/pooltemplate' do
+    let(:prefix) { '/api/v1' }
+    let(:metrics) { Vmpooler::DummyStatsd.new }
+
+    let(:current_time) { Time.now }
+
+    before(:each) do
+      redis.flushdb
+
+      app.settings.set :config, config
+      app.settings.set :redis, redis
+      app.settings.set :metrics, metrics
+      app.settings.set :config, auth: false
+      create_token('abcdefghijklmnopqrstuvwxyz012345', 'jdoe', current_time)
+    end
+
+    describe 'POST /config/pooltemplate' do
+      it 'updates a pool template' do
+        post "#{prefix}/config/pooltemplate", '{"pool1":"templates/new_template"}'
+        expect_json(ok = true, http = 201)
+
+        expected = { ok: true }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails on nonexistent pools' do
+        post "#{prefix}/config/pooltemplate", '{"poolpoolpool":"templates/newtemplate"}'
+        expect_json(ok = false, http = 400)
+      end
+
+      it 'updates multiple pools' do
+        post "#{prefix}/config/pooltemplate", '{"pool1":"templates/new_template","pool2":"templates/new_template2"}'
+        expect_json(ok = true, http = 201)
+
+        expected = { ok: true }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails when not all pools exist' do
+        post "#{prefix}/config/pooltemplate", '{"pool1":"templates/new_template","pool3":"templates/new_template2"}'
+        expect_json(ok = false, http = 400)
+
+        expected = {
+          ok: false,
+          bad_templates: ['pool3']
+        }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'returns no changes when the template does not change' do
+        post "#{prefix}/config/pooltemplate", '{"pool1":"templates/pool1"}'
+        expect_json(ok = true, http = 200)
+
+        expected = { ok: true }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails when a invalid template parameter is provided' do
+        post "#{prefix}/config/pooltemplate", '{"pool1":"template1"}'
+        expect_json(ok = false, http = 400)
+
+        expected = {
+          ok: false,
+          bad_templates: ['pool1']
+        }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails when a template starts with /' do
+        post "#{prefix}/config/pooltemplate", '{"pool1":"/template1"}'
+        expect_json(ok = false, http = 400)
+
+        expected = {
+          ok: false,
+          bad_templates: ['pool1']
+        }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails when a template ends with /' do
+        post "#{prefix}/config/pooltemplate", '{"pool1":"template1/"}'
+        expect_json(ok = false, http = 400)
+
+        expected = {
+          ok: false,
+          bad_templates: ['pool1']
+        }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      context 'with experimental features disabled' do
+        before(:each) do
+          config[:config]['experimental_features'] = false
+        end
+
+        it 'should return 405' do
+          post "#{prefix}/config/pooltemplate", '{"pool1":"template/template1"}'
+          expect_json(ok = false, http = 405)
+
+          expected = { ok: false }
+          expect(last_response.body).to eq(JSON.pretty_generate(expected))
+        end
+      end
+
+    end
+
+    describe 'POST /config/poolsize' do
+      it 'changes a pool size' do
+        post "#{prefix}/config/poolsize", '{"pool1":"2"}'
+        expect_json(ok = true, http = 201)
+
+        expected = { ok: true }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'changes a pool size for multiple pools' do
+        post "#{prefix}/config/poolsize", '{"pool1":"2","pool2":"2"}'
+        expect_json(ok = true, http = 201)
+
+        expected = { ok: true }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails when a specified pool does not exist' do
+        post "#{prefix}/config/poolsize", '{"pool10":"2"}'
+        expect_json(ok = false, http = 400)
+        expected = {
+          ok: false,
+          bad_templates: ['pool10']
+        }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'succeeds with 200 when no change is required' do
+        post "#{prefix}/config/poolsize", '{"pool1":"5"}'
+        expect_json(ok = true, http = 200)
+
+        expected = { ok: true }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'succeeds with 201 when at least one pool changes' do
+        post "#{prefix}/config/poolsize", '{"pool1":"5","pool2":"5"}'
+        expect_json(ok = true, http = 201)
+
+        expected = { ok: true }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails when a non-integer value is provided for size' do
+        post "#{prefix}/config/poolsize", '{"pool1":"four"}'
+        expect_json(ok = false, http = 400)
+
+        expected = {
+          ok: false,
+          bad_templates: ['pool1']
+        }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      it 'fails when a negative value is provided for size' do
+        post "#{prefix}/config/poolsize", '{"pool1":"-1"}'
+        expect_json(ok = false, http = 400)
+
+        expected = {
+          ok: false,
+          bad_templates: ['pool1']
+        }
+
+        expect(last_response.body).to eq(JSON.pretty_generate(expected))
+      end
+
+      context 'with experimental features disabled' do
+        before(:each) do
+          config[:config]['experimental_features'] = false
+        end
+
+        it 'should return 405' do
+          post "#{prefix}/config/poolsize", '{"pool1":"1"}'
+          expect_json(ok = false, http = 405)
+
+          expected = { ok: false }
+          expect(last_response.body).to eq(JSON.pretty_generate(expected))
+        end
+      end
+    end
+
+    describe 'GET /config' do
+      let(:prefix) { '/api/v1' }
+
+      it 'returns pool configuration when set' do
+        get "#{prefix}/config"
+
+        expect(last_response.header['Content-Type']).to eq('application/json')
+        result = JSON.parse(last_response.body)
+        expect(result['pool_configuration']).to eq(config[:pools])
+      end
+    end
+  end
+end

spec/unit/api/helpers_spec.rb

@@ -168,4 +168,74 @@ describe Vmpooler::API::Helpers do
     end
   end
 
+  describe '#pool_index' do
+    let(:pools) {
+      [
+        {
+          'name' => 'pool1'
+        },
+        {
+          'name' => 'pool2'
+        }
+      ]
+    }
+
+    it 'should return a hash' do
+      pools_hash = subject.pool_index(pools)
+
+      expect(pools_hash).to be_a(Hash)
+    end
+
+    it 'should return the correct index for each pool' do
+      pools_hash = subject.pool_index(pools)
+
+      expect(pools[pools_hash['pool1']]['name']).to eq('pool1')
+      expect(pools[pools_hash['pool2']]['name']).to eq('pool2')
+    end
+  end
+
+  describe '#template_ready?' do
+    let(:redis) { double('redis') }
+    let(:template) { 'template/test1' }
+    let(:poolname) { 'pool1' }
+    let(:pool) {
+      {
+        'name' => poolname,
+        'template' => template
+      }
+    }
+
+    it 'returns false when there is no prepared template' do
+      expect(redis).to receive(:hget).with('vmpooler__template__prepared', poolname).and_return(nil)
+
+      expect(subject.template_ready?(pool, redis)).to be false
+    end
+
+    it 'returns true when configured and prepared templates match' do
+      expect(redis).to receive(:hget).with('vmpooler__template__prepared', poolname).and_return(template)
+
+      expect(subject.template_ready?(pool, redis)).to be true
+    end
+
+    it 'returns false when configured and prepared templates do not match' do
+      expect(redis).to receive(:hget).with('vmpooler__template__prepared', poolname).and_return('template3')
+
+      expect(subject.template_ready?(pool, redis)).to be false
+    end
+  end
+
+  describe '#is_integer?' do
+    it 'returns true when input is an integer' do
+      expect(subject.is_integer? 4).to be true
+    end
+
+    it 'returns true when input is a string containing an integer' do
+      expect(subject.is_integer? '4').to be true
+    end
+
+    it 'returns false when input is a string containing word characters' do
+      expect(subject.is_integer? 'four').to be false
+    end
+  end
+
 end

spec/unit/pool_manager_spec.rb

@@ -1503,6 +1503,396 @@ EOT
     end
   end
 
+  describe 'sync_pool_template' do
+    let(:old_template) { 'templates/old-template' }
+    let(:new_template) { 'templates/new-template' }
+    let(:config) { YAML.load(<<-EOT
+---
+:pools:
+  - name: '#{pool}'
+    size: 1
+    template: old_template
+EOT
+      )
+    }
+
+    it 'returns when a template is not set in redis' do
+      expect(subject.sync_pool_template(config[:pools][0])).to be_nil
+    end
+
+    it 'returns when a template is set and matches the configured template' do
+      redis.hset('vmpooler__config__template', pool, old_template)
+
+      subject.sync_pool_template(config[:pools][0])
+
+      expect(config[:pools][0]['template']).to eq(old_template)
+    end
+
+    it 'updates a pool template when the redis provided value is different' do
+      redis.hset('vmpooler__config__template', pool, new_template)
+
+      subject.sync_pool_template(config[:pools][0])
+
+      expect(config[:pools][0]['template']).to eq(new_template)
+    end
+  end
+
+  describe 'pool_mutex' do
+    it 'should return a mutex' do
+      expect(subject.pool_mutex(pool)).to be_a(Mutex)
+    end
+
+    it 'should return the same mutex when called twice' do
+      first = subject.pool_mutex(pool)
+      second = subject.pool_mutex(pool)
+      expect(first).to be(second)
+    end
+  end
+
+  describe 'update_pool_template' do
+    let(:current_template) { 'templates/pool_template' }
+    let(:new_template) { 'templates/new_pool_template' }
+    let(:config) {
+      YAML.load(<<-EOT
+---
+:config: {}
+:pools:
+  - name: #{pool}
+    template: "#{current_template}"
+EOT
+      )
+    }
+    let(:poolconfig) { config[:pools][0] }
+
+    before(:each) do
+      allow(logger).to receive(:log)
+    end
+
+    it 'should set the pool template to match the configured template' do
+      subject.update_pool_template(poolconfig, provider, new_template, current_template)
+
+      expect(poolconfig['template']).to eq(new_template)
+    end
+
+    it 'should log that the template is updated' do
+      expect(logger).to receive(:log).with('s', "[*] [#{pool}] template updated from #{current_template} to #{new_template}")
+
+      subject.update_pool_template(poolconfig, provider, new_template, current_template)
+    end
+
+    it 'should run drain_pool' do
+      expect(subject).to receive(:drain_pool).with(pool)
+
+      subject.update_pool_template(poolconfig, provider, new_template, current_template)
+    end
+
+    it 'should log that a template is being prepared' do
+      expect(logger).to receive(:log).with('s', "[*] [#{pool}] preparing pool template for deployment")
+
+      subject.update_pool_template(poolconfig, provider, new_template, current_template)
+    end
+
+    it 'should run prepare_template' do
+      expect(subject).to receive(:prepare_template).with(poolconfig, provider)
+
+      subject.update_pool_template(poolconfig, provider, new_template, current_template)
+    end
+
+    it 'should log that the pool is ready for use' do
+      expect(logger).to receive(:log).with('s', "[*] [#{pool}] is ready for use")
+
+      subject.update_pool_template(poolconfig, provider, new_template, current_template)
+    end
+  end
+
+  describe 'remove_excess_vms' do
+    let(:config) {
+      YAML.load(<<-EOT
+---
+:pools:
+  - name: #{pool}
+    size: 2
+EOT
+      )
+    }
+
+    before(:each) do
+      expect(subject).not_to be_nil
+    end
+
+    context 'with a 0 total value' do
+      let(:ready) { 0 }
+      let(:total) { 0 }
+      it 'should return nil' do
+        expect(subject.remove_excess_vms(config[:pools][0], provider, ready, total)).to be_nil
+      end
+    end
+
+    context 'when the mutex is locked' do
+      let(:mutex) { Mutex.new }
+      let(:ready) { 2 }
+      let(:total) { 3 }
+      before(:each) do
+        mutex.lock
+        expect(subject).to receive(:pool_mutex).with(pool).and_return(mutex)
+      end
+
+      it 'should return nil' do
+        expect(subject.remove_excess_vms(config[:pools][0], provider, ready, total)).to be_nil
+      end
+    end
+
+    context 'with a total size less than the pool size' do
+      let(:ready) { 1 }
+      let(:total) { 2 }
+      it 'should return nil' do
+        expect(subject.remove_excess_vms(config[:pools][0], provider, ready, total)).to be_nil
+      end
+    end
+
+    context 'with a total size greater than the pool size' do
+      let(:ready) { 4 }
+      let(:total) { 4 }
+      it 'should remove excess ready vms' do
+        expect(subject).to receive(:move_vm_queue).exactly(2).times
+
+        subject.remove_excess_vms(config[:pools][0], provider, ready, total)
+      end
+
+      it 'should remove excess pending vms' do
+        create_pending_vm(pool,'vm1')
+        create_pending_vm(pool,'vm2')
+        create_ready_vm(pool, 'vm3')
+        create_ready_vm(pool, 'vm4')
+        create_ready_vm(pool, 'vm5')
+        expect(subject).to receive(:move_vm_queue).exactly(3).times
+
+        subject.remove_excess_vms(config[:pools][0], provider, 3, 5)
+      end
+    end
+  end
+
+  describe 'prepare_template' do
+    let(:config) { YAML.load(<<-EOT
+---
+:config:
+  create_template_delta_disks: true
+:providers:
+  :mock:
+:pools:
+  - name: '#{pool}'
+    size: 1
+    template: 'templates/pool1'
+EOT
+      )
+    }
+
+    context 'when creating the template delta disks' do
+      before(:each) do
+        allow(redis).to receive(:hset)
+        allow(provider).to receive(:create_template_delta_disks)
+      end
+
+      it 'should run create template delta disks' do
+        expect(provider).to receive(:create_template_delta_disks).with(config[:pools][0])
+
+        subject.prepare_template(config[:pools][0], provider)
+      end
+
+      it 'should mark the template as prepared' do
+        expect(redis).to receive(:hset).with('vmpooler__template__prepared', pool, config[:pools][0]['template'])
+
+        subject.prepare_template(config[:pools][0], provider)
+      end
+    end
+  end
+
+  describe 'evaluate_template' do
+    let(:mutex) { Mutex.new }
+    let(:current_template) { 'templates/template1' }
+    let(:new_template) { 'templates/template2' }
+    let(:config) { YAML.load(<<-EOT
+---
+:config:
+  task_limit: 5
+:providers:
+  :mock:
+:pools:
+  - name: '#{pool}'
+    size: 1
+    template: '#{current_template}'
+EOT
+      )
+    }
+
+    before(:each) do
+      allow(redis).to receive(:hget)
+      expect(subject).to receive(:pool_mutex).with(pool).and_return(mutex)
+    end
+
+    it 'should retreive the prepared template' do
+      expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).and_return(current_template)
+
+      subject.evaluate_template(config[:pools][0], provider)
+    end
+
+    it 'should retrieve the redis configured template' do
+      expect(redis).to receive(:hget).with('vmpooler__config__template', pool).and_return(new_template)
+
+      subject.evaluate_template(config[:pools][0], provider)
+    end
+
+    context 'when the mutex is locked' do
+      before(:each) do
+        mutex.lock
+      end
+
+      it 'should return' do
+        expect(subject.evaluate_template(config[:pools][0], provider)).to be_nil
+      end
+    end
+
+    context 'when prepared template is nil' do
+      before(:each) do
+        expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).and_return(nil)
+      end
+
+      it 'should prepare the template' do
+        expect(subject).to receive(:prepare_template).with(config[:pools][0], provider)
+
+        subject.evaluate_template(config[:pools][0], provider)
+      end
+    end
+
+    context 'when a new template is requested' do
+      before(:each) do
+        expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).and_return(current_template)
+        expect(redis).to receive(:hget).with('vmpooler__config__template', pool).and_return(new_template)
+      end
+
+      it 'should update the template' do
+        expect(subject).to receive(:update_pool_template).with(config[:pools][0], provider, new_template, current_template)
+
+        subject.evaluate_template(config[:pools][0], provider)
+      end
+    end
+  end
+
+  describe 'drain_pool' do
+    before(:each) do
+      allow(logger).to receive(:log)
+    end
+
+    context 'with no vms' do
+      it 'should return nil' do
+        expect(subject.drain_pool(pool)).to be_nil
+      end
+
+      it 'should not log any messages' do
+        expect(logger).to_not receive(:log)
+
+        subject.drain_pool(pool)
+      end
+
+      it 'should not try to move any vms' do
+        expect(subject).to_not receive(:move_vm_queue)
+
+        subject.drain_pool(pool)
+      end
+    end
+
+    context 'with ready vms' do
+      before(:each) do
+        create_ready_vm(pool, 'vm1')
+        create_ready_vm(pool, 'vm2')
+      end
+
+      it 'removes the ready instances' do
+        expect(subject).to receive(:move_vm_queue).twice
+
+        subject.drain_pool(pool)
+      end
+
+      it 'logs that ready instances are being removed' do
+        expect(logger).to receive(:log).with('s', "[*] [#{pool}] removing ready instances")
+
+        subject.drain_pool(pool)
+      end
+    end
+
+    context 'with pending instances' do
+      before(:each) do
+        create_pending_vm(pool, 'vm1')
+        create_pending_vm(pool, 'vm2')
+      end
+
+      it 'removes the pending instances' do
+        expect(subject).to receive(:move_vm_queue).twice
+
+        subject.drain_pool(pool)
+      end
+
+      it 'logs that pending instances are being removed' do
+        expect(logger).to receive(:log).with('s', "[*] [#{pool}] removing pending instances")
+
+        subject.drain_pool(pool)
+      end
+    end
+  end
+
+  describe 'update_pool_size' do
+    let(:newsize) { '3' }
+    let(:config) {
+      YAML.load(<<-EOT
+---
+:pools:
+  - name: #{pool}
+    size: 2
+EOT
+      )
+    }
+    let(:poolconfig) { config[:pools][0] }
+
+    context 'with a locked mutex' do
+
+      let(:mutex) { Mutex.new }
+      before(:each) do
+        mutex.lock
+        expect(subject).to receive(:pool_mutex).with(pool).and_return(mutex)
+      end
+
+      it 'should return nil' do
+        expect(subject.update_pool_size(poolconfig)).to be_nil
+      end
+    end
+
+    it 'should get the pool size configuration from redis' do
+      expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool)
+
+      subject.update_pool_size(poolconfig)
+    end
+
+    it 'should return when poolsize is not set in redis' do
+      expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool).and_return(nil)
+
+      expect(subject.update_pool_size(poolconfig)).to be_nil
+    end
+
+    it 'should return when no change in configuration is required' do
+      expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool).and_return('2')
+
+      expect(subject.update_pool_size(poolconfig)).to be_nil
+    end
+
+    it 'should update the pool size' do
+      expect(redis).to receive(:hget).with('vmpooler__config__poolsize', pool).and_return(newsize)
+
+      subject.update_pool_size(poolconfig)
+
+      expect(poolconfig['size']).to eq(Integer(newsize))
+    end
+  end
+
   describe "#execute!" do
     let(:config) {
       YAML.load(<<-EOT
@@ -1824,6 +2214,7 @@ EOT
       it 'should run startup tasks only once' do
         expect(redis).to receive(:set).with('vmpooler__tasks__clone', 0).once
         expect(redis).to receive(:del).with('vmpooler__migration').once
+        expect(redis).to receive(:del).with('vmpooler__template__prepared').once
 
         subject.execute!(maxloop,0)
       end
@@ -1902,8 +2293,38 @@ EOT
         subject.sleep_with_wakeup_events(loop_delay, wakeup_period, wakeup_option)
       end
     end
+
+    describe 'with the pool_template_change wakeup option' do
+      let(:wakeup_option) {{
+        :pool_template_change => true,
+        :poolname => pool
+      }}
+      let(:new_template) { 'templates/newtemplate' }
+      let(:wakeup_period) { -1 } # A negative number forces the wakeup evaluation to always occur
+
+      context 'with a template configured' do
+        before(:each) do
+          redis.hset('vmpooler__config__template', pool, new_template)
+          allow(redis).to receive(:hget)
+        end
+
+        it 'should check if a template is configured in redis' do
+          expect(subject).to receive(:time_passed?).with(:exit_by, Time).and_return(false, true)
+          expect(redis).to receive(:hget).with('vmpooler__template__prepared', pool).once
+
+          subject.sleep_with_wakeup_events(loop_delay, wakeup_period, wakeup_option)
+        end
+
+        it 'should sleep until a template change is detected' do
+          expect(subject).to receive(:sleep).exactly(3).times
+          expect(redis).to receive(:hget).with('vmpooler__config__template', pool).and_return(nil,nil,new_template)
+
+          subject.sleep_with_wakeup_events(loop_delay, wakeup_period, wakeup_option)
+        end
+      end
+    end
   end
-  
+
   describe "#check_pool" do
     let(:threads) {{}}
     let(:provider_name) { 'mock_provider' }
@@ -2785,6 +3206,52 @@ EOT
         end
       end
 
+      context 'when a pool size configuration change is detected' do
+        let(:poolsize) { 2 }
+        let(:newpoolsize) { 3 }
+        before(:each) do
+          config[:pools][0]['size'] = poolsize
+          redis.hset('vmpooler__config__poolsize', pool, newpoolsize)
+          expect(provider).to receive(:vms_in_pool).with(pool).and_return([])
+        end
+
+        it 'should change the pool size configuration' do
+          subject._check_pool(config[:pools][0],provider)
+
+          expect(config[:pools][0]['size']).to be(newpoolsize)
+        end
+      end
+
+      context 'when a pool template is updating' do
+        let(:poolsize) { 2 }
+        before(:each) do
+          redis.hset('vmpooler__config__updating', pool, 1)
+          expect(provider).to receive(:vms_in_pool).with(pool).and_return([])
+        end
+
+        it 'should not call clone_vm to populate the pool' do
+          expect(subject).to_not receive(:clone_vm)
+
+          subject._check_pool(config[:pools][0],provider)
+        end
+      end
+
+      context 'when an excess number of ready vms exist' do
+
+        before(:each) do
+          allow(redis).to receive(:scard)
+          expect(redis).to receive(:scard).with("vmpooler__ready__#{pool}").and_return(1)
+          expect(redis).to receive(:scard).with("vmpooler__pending__#{pool}").and_return(1)
+          expect(provider).to receive(:vms_in_pool).with(pool).and_return([])
+        end
+
+        it 'should call remove_excess_vms' do
+          expect(subject).to receive(:remove_excess_vms).with(config[:pools][0], provider, 1, 2)
+
+          subject._check_pool(config[:pools][0],provider)
+        end
+      end
+
       context 'export metrics' do
         it 'increments metrics for ready queue' do
           create_ready_vm(pool,'vm1')

spec/unit/providers/vsphere_spec.rb

@@ -283,6 +283,7 @@ EOT
 
     let(:clone_vm_task) { mock_RbVmomi_VIM_Task() }
     let(:new_vm_object)  { mock_RbVmomi_VIM_VirtualMachine({ :name => vmname }) }
+    let(:new_template_object)  { mock_RbVmomi_VIM_VirtualMachine({ :name => vmname }) }
 
     before(:each) do
       allow(subject).to receive(:connect_to_vsphere).and_return(connection)
@@ -305,19 +306,30 @@ EOT
       end
     end
 
-    context 'Given a template path that does not exist' do
+    context 'Given a template that starts with /' do
       before(:each) do
-        config[:pools][0]['template'] = 'missing_Templates/pool1'
+        config[:pools][0]['template'] = '/bad_template'
       end
 
       it 'should raise an error' do
-        expect{ subject.create_vm(poolname, vmname) }.to raise_error(/specifies a template folder of .+ which does not exist/)
+        expect{ subject.create_vm(poolname, vmname) }.to raise_error(/did not specify a full path for the template/)
+      end
+    end
+
+    context 'Given a template that ends with /' do
+      before(:each) do
+        config[:pools][0]['template'] = 'bad_template/'
+      end
+
+      it 'should raise an error' do
+        expect{ subject.create_vm(poolname, vmname) }.to raise_error(/did not specify a full path for the template/)
       end
     end
 
     context 'Given a template VM that does not exist' do
       before(:each) do
         config[:pools][0]['template'] = 'Templates/missing_template'
+        expect(subject).to receive(:find_template_vm).and_raise("specifies a template VM of #{vmname} which does not exist")
       end
 
       it 'should raise an error' do
@@ -327,7 +339,8 @@ EOT
 
     context 'Given a successful creation' do
       before(:each) do
-        template_vm = subject.find_folder('Templates',connection,datacenter_name).find('pool1')
+        template_vm = new_template_object
+        allow(subject).to receive(:find_template_vm).and_return(new_template_object)
         allow(template_vm).to receive(:CloneVM_Task).and_return(clone_vm_task)
         allow(clone_vm_task).to receive(:wait_for_completion).and_return(new_vm_object)
       end
@@ -339,7 +352,7 @@ EOT
       end
 
       it 'should use the appropriate Create_VM spec' do
-        template_vm = subject.find_folder('Templates',connection,datacenter_name).find('pool1')
+        template_vm = new_template_object
         expect(template_vm).to receive(:CloneVM_Task)
           .with(create_vm_spec(vmname,'pool1','datastore0'))
           .and_return(clone_vm_task)
@@ -3461,5 +3474,71 @@ EOT
     end
   end
 
+  describe 'find_template_vm' do
+    let(:vm_object) { mock_RbVmomi_VIM_VirtualMachine() }
 
+    before(:each) do
+      allow(connection.searchIndex).to receive(:FindByInventoryPath)
+    end
+    it 'should raise an error when the datacenter cannot be found' do
+      config[:providers][:vsphere]['datacenter'] = nil
+
+      expect{ subject.find_template_vm(config[:pools][0],connection) }.to raise_error('cannot find datacenter')
+    end
+
+    it 'should raise an error when the template specified cannot be found' do
+      expect(connection.searchIndex).to receive(:FindByInventoryPath).and_return(nil)
+
+      expect{ subject.find_template_vm(config[:pools][0],connection) }.to raise_error("Pool #{poolname} specifies a template VM of #{config[:pools][0]['template']} which does not exist for the provider vsphere")
+    end
+
+    it 'should return the vm object' do
+      expect(connection.searchIndex).to receive(:FindByInventoryPath).and_return(vm_object)
+
+      subject.find_template_vm(config[:pools][0],connection)
+    end
+  end
+
+  describe 'valid_template_path?' do
+
+    it 'should return true with a valid template path' do
+      expect(subject.valid_template_path?('test/template')).to eq(true)
+    end
+
+    it 'should return false when no / is found' do
+      expect(subject.valid_template_path?('testtemplate')).to eq(false)
+    end
+
+    it 'should return false when template path begins with /' do
+      expect(subject.valid_template_path?('/testtemplate')).to eq(false)
+    end
+
+    it 'should return false when template path ends with /' do
+      expect(subject.valid_template_path?('testtemplate/')).to eq(false)
+    end
+  end
+
+  describe 'create_template_delta_disks' do
+    let(:template_object) { mock_RbVmomi_VIM_VirtualMachine({
+        :name => vmname,
+      })
+    }
+
+    before(:each) do
+      allow(subject).to receive(:connect_to_vsphere).and_return(connection)
+    end
+
+    context 'with a template VM found' do
+
+      before(:each) do
+        expect(subject).to receive(:find_template_vm).and_return(template_object)
+      end
+
+      it 'should reconfigure the VM creating delta disks' do
+        expect(template_object).to receive(:add_delta_disk_layer_on_all_disks)
+
+        subject.create_template_delta_disks(config[:pools][0])
+      end
+    end
+  end
 end

vmpooler.yaml.example

@@ -443,6 +443,11 @@
 #     The value represents a percentage and applies to both memory and CPU
 #     (optional; default: 90)
 #
+#   - experimental_features (Only affects API config endpoints)
+#     Enable experimental API capabilities such as changing pool template and size without application restart
+#     Expects a boolean value
+#     (optional; default: false)
+#
 # Example:
 
 :config:
@@ -458,6 +463,7 @@
     - 'project'
   domain: 'company.com'
   prefix: 'poolvm-'
+  experimental_features: true
 
 # :pools:
 #