From d8ba8a3c0b3c90f754bdd0832779b1dadf24a90e Mon Sep 17 00:00:00 2001 From: Scott Schneider Date: Wed, 19 Feb 2014 15:17:52 -0800 Subject: [PATCH 1/3] Initial --- README.md | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..0bd6712 --- /dev/null +++ b/README.md @@ -0,0 +1,132 @@ +![vmpooler](https://raw.github.com/sschneid/vmpooler/master/public/img/logo.jpg) + +# vmpooler + +vmpooler provides configurable 'pools' of available (running) virtual machines. + + +## Usage + +At [Puppet Labs](http://puppetlabs.com) we run acceptance tests on hundreds of disposable VMs every day. Dynamic cloning of VM templates initially worked fine for this, but added several seconds to each test run and was unable to account for failed clone tasks. By pushing these operations to a backend service, we were able to both speed up tests and eliminate test failures due to underlying infrastructure failures. + + +## Installation + +### Prerequisites + +vmpooler requires the following Ruby gems be installed: + +- [json](http://rubygems.org/gems/json) +- [rbvmomi](http://rubygems.org/gems/rbvmomi) +- [redis](http://rubygems.org/gems/redis) +- [sinatra](http://rubygems.org/gems/sinatra) + +It also requires that a [Redis](http://redis.io/) server exists somewhere, as this is the datastore used for vmpooler's inventory and queueing services. + +### Configuration + +The following YAML configuration sets up two pools, `debian-7-i386` and `debian-7-x86_64`, which contain 5 running VMs each: + +``` +--- +:vsphere: + server: 'vsphere.company.com' + username: 'vmpooler' + password: 'swimsw1msw!m' + +:redis: + server: 'redis.company.com' + +:config: + logfile: '/var/log/vmpooler.log' + +:pools: + - name: 'debian-7-i386' + template: 'Templates/debian-7-i386' + folder: 'Pooled VMs/debian-7-i386' + pool: 'Pooled VMs/debian-7-i386' + datastore: 'vmstorage' + size: 5 + - name: 'debian-7-x86_64' + template: 'Templates/debian-7-x86_64' + folder: 'Pooled VMs/debian-7-x86_64' + pool: 'Pooled VMs/debian-7-x86_64' + datastore: 'vmstorage' + size: 5 +``` + +See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for additional configuration options and parameters. + +### Template set-up + +Template set-up is left as an exercise to the reader. Somehow, either via PXE, embedded bootstrap scripts, or some other method -- clones of VM templates need to be able to set their hostname, register themselves in your DNS, and be resolvable by the vmpooler application after completing the clone task and booting up. + + +## API and Dashboard + +vmpooler provides an API and web front-end (dashboard) on port `:4567`. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), to specify an alternative port to listen on. + +### API + +vmpooler provides a REST API for VM management. The following examples use `curl` for communication. + +#### GET /vm + +Retrieve a list of available VM pools. + +``` +$ curl --url vmpooler.company.com/vm +``` +```json +[ + "debian-7-i386", + "debian-7-x86_64" +] +``` + +#### POST /vm/ +Check-out a VM. + +``` +$ curl -d --url vmpooler.company.com/vm/debian-7-i386 +``` +```json +{ + "debian-7-i386": { + "ok": true, + "hostname": "fq6qlpjlsskycq6" + } +} +``` + +#### DELETE /vm/ + +Schedule a checked-out VM for deletion. + +``` +$ curl -X DELETE --url vmpooler.company.com/vm/fq6qlpjlsskycq6 +``` +```json +{ + "ok": true +} +``` + +### Dashboard + +A dashboard is provided to offer real-time statistics and historical graphs. It looks like this in a large installation on a fairly busy day: + +(img) + +[Graphite](http://graphite.wikidot.com/) is required for historical data retrieval. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for details. + + +## Author + +Scott Schneider (sschneid@gmail.com) + + +## License + +vmpooler is distributed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0.html). See the [LICENSE](LICENSE) file for more details. + From cf08289d3e61c2ec5f0c522249c360de1ac1c0fc Mon Sep 17 00:00:00 2001 From: Scott Schneider Date: Wed, 19 Feb 2014 16:05:27 -0800 Subject: [PATCH 2/3] Initial --- vmpooler.yaml.example | 150 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 vmpooler.yaml.example diff --git a/vmpooler.yaml.example b/vmpooler.yaml.example new file mode 100644 index 0000000..4746db2 --- /dev/null +++ b/vmpooler.yaml.example @@ -0,0 +1,150 @@ +--- +# :vsphere: +# +# This section contains the server hostname and authentication credentials +# needed for vmpooler to connect to VMware vSphere. +# +# Available configuration parameters: +# +# - server +# The FQDN hostname of the VMware vSphere server. +# (required) +# +# - username +# The username used to authenticate VMware vSphere. +# (required) +# +# - password +# The password used to authenticate VMware vSphere. +# (required) + +# Example: + +:vsphere: + server: 'vsphere.company.com' + username: 'vmpooler' + password: 'swimsw1msw!m' + +# :redis: +# +# This section contains the server hostname and authentication credentials +# needed for vmpooler to connect to Redis. +# +# Available configuration parameters: +# +# - server +# The FQDN hostname of the Redis server. +# (optional; default: 'localhost') +# +# - username +# The username used to authenticate Redis. +# (optional) +# +# - password +# The password used to authenticate Redis. +# (optional) + +# Example: + +:redis: + server: 'redis.company.com' + +# :config: +# +# This section contains global configuration information. +# +# Available configuration parameters: +# +# - site_name +# The name of your deployment. +# (optional; default: 'vmpooler') +# +# - logfile +# The path to vmpooler's log file. +# (optional; default: '/var/log/vmpooler.log') +# +# - graphite +# The FQDN hostname of the Graphite server. +# (optional) +# +# - task_limit +# The number of concurrent VMware vSphere tasks to perform. +# (optional; default: '10') +# +# - vm_checktime +# How often (in minutes) to check the sanity of VMs in 'ready' queues. +# (optional; default: '15') +# +# - vm_lifetime +# How long (in hours) to keep VMs in 'running' queues before destroying. +# (optional; default: '24') + +# Example: + +:config: + site_name: 'vmpooler' + logfile: '/var/log/vmpooler.log' + graphite: 'graphite.company.com' + task_limit: 10 + vm_checktime: 15 + vm_lifetime: 12 + +# :pools: +# +# This section contains a list of virtual machine 'pools' for vmpooler to +# create and maintain. +# +# Available configuration parameters (per-pool): +# +# - name +# The name of the pool. +# (required) +# +# - template +# The template or virtual machine target to spawn clones from. +# (required) +# +# - folder +# The vSphere 'folder' destination for spawned clones. +# (required) +# +# - pool +# The vSphere 'resource pool' destination for spawned clones. +# (required) +# +# - datastore +# The vSphere 'datastore' destination for spawned clones. +# (required) +# +# - size +# The number of waiting VMs to keep in a pool. +# (required) +# +# - timeout +# How long (in minutes) before marking a clone as 'failed' and retrying. +# (optional; default: '15') +# +# - ready_ttl +# How long (in minutes) to keep VMs in 'ready' queues before destroying. +# (optional) + +# Example: + +:pools: + - name: 'debian-7-i386' + template: 'Templates/debian-7-i386' + folder: 'Pooled VMs/debian-7-i386' + pool: 'Pooled VMs/debian-7-i386' + datastore: 'vmstorage' + size: 5 + timeout: 15 + ready_ttl: 1440 + - name: 'debian-7-x86_64' + template: 'Templates/debian-7-x86_64' + folder: 'Pooled VMs/debian-7-x86_64' + pool: 'Pooled VMs/debian-7-x86_64' + datastore: 'vmstorage' + size: 5 + timeout: 15 + ready_ttl: 1440 + From a83e5caa5dfdc125d987464d3c243b7bd70fc5a1 Mon Sep 17 00:00:00 2001 From: Scott Schneider Date: Fri, 21 Feb 2014 13:53:09 -0800 Subject: [PATCH 3/3] Fix filename typo --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0bd6712..ff52058 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ The following YAML configuration sets up two pools, `debian-7-i386` and `debian- size: 5 ``` -See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for additional configuration options and parameters. +See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), for additional configuration options and parameters. ### Template set-up @@ -64,7 +64,7 @@ Template set-up is left as an exercise to the reader. Somehow, either via PXE, ## API and Dashboard -vmpooler provides an API and web front-end (dashboard) on port `:4567`. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), to specify an alternative port to listen on. +vmpooler provides an API and web front-end (dashboard) on port `:4567`. See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), to specify an alternative port to listen on. ### API @@ -118,7 +118,7 @@ A dashboard is provided to offer real-time statistics and historical graphs. It (img) -[Graphite](http://graphite.wikidot.com/) is required for historical data retrieval. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for details. +[Graphite](http://graphite.wikidot.com/) is required for historical data retrieval. See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), for details. ## Author