From 692f20ff3b4d8eec9baf324ab7cb36ef8080be31 Mon Sep 17 00:00:00 2001 From: Scott Schneider Date: Wed, 19 Feb 2014 15:17:52 -0800 Subject: [PATCH 1/3] Initial --- README.md | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..0bd6712 --- /dev/null +++ b/README.md @@ -0,0 +1,132 @@ +![vmpooler](https://raw.github.com/sschneid/vmpooler/master/public/img/logo.jpg) + +# vmpooler + +vmpooler provides configurable 'pools' of available (running) virtual machines. + + +## Usage + +At [Puppet Labs](http://puppetlabs.com) we run acceptance tests on hundreds of disposable VMs every day. Dynamic cloning of VM templates initially worked fine for this, but added several seconds to each test run and was unable to account for failed clone tasks. By pushing these operations to a backend service, we were able to both speed up tests and eliminate test failures due to underlying infrastructure failures. + + +## Installation + +### Prerequisites + +vmpooler requires the following Ruby gems be installed: + +- [json](http://rubygems.org/gems/json) +- [rbvmomi](http://rubygems.org/gems/rbvmomi) +- [redis](http://rubygems.org/gems/redis) +- [sinatra](http://rubygems.org/gems/sinatra) + +It also requires that a [Redis](http://redis.io/) server exists somewhere, as this is the datastore used for vmpooler's inventory and queueing services. + +### Configuration + +The following YAML configuration sets up two pools, `debian-7-i386` and `debian-7-x86_64`, which contain 5 running VMs each: + +``` +--- +:vsphere: + server: 'vsphere.company.com' + username: 'vmpooler' + password: 'swimsw1msw!m' + +:redis: + server: 'redis.company.com' + +:config: + logfile: '/var/log/vmpooler.log' + +:pools: + - name: 'debian-7-i386' + template: 'Templates/debian-7-i386' + folder: 'Pooled VMs/debian-7-i386' + pool: 'Pooled VMs/debian-7-i386' + datastore: 'vmstorage' + size: 5 + - name: 'debian-7-x86_64' + template: 'Templates/debian-7-x86_64' + folder: 'Pooled VMs/debian-7-x86_64' + pool: 'Pooled VMs/debian-7-x86_64' + datastore: 'vmstorage' + size: 5 +``` + +See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for additional configuration options and parameters. + +### Template set-up + +Template set-up is left as an exercise to the reader. Somehow, either via PXE, embedded bootstrap scripts, or some other method -- clones of VM templates need to be able to set their hostname, register themselves in your DNS, and be resolvable by the vmpooler application after completing the clone task and booting up. + + +## API and Dashboard + +vmpooler provides an API and web front-end (dashboard) on port `:4567`. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), to specify an alternative port to listen on. + +### API + +vmpooler provides a REST API for VM management. The following examples use `curl` for communication. + +#### GET /vm + +Retrieve a list of available VM pools. + +``` +$ curl --url vmpooler.company.com/vm +``` +```json +[ + "debian-7-i386", + "debian-7-x86_64" +] +``` + +#### POST /vm/ +Check-out a VM. + +``` +$ curl -d --url vmpooler.company.com/vm/debian-7-i386 +``` +```json +{ + "debian-7-i386": { + "ok": true, + "hostname": "fq6qlpjlsskycq6" + } +} +``` + +#### DELETE /vm/ + +Schedule a checked-out VM for deletion. + +``` +$ curl -X DELETE --url vmpooler.company.com/vm/fq6qlpjlsskycq6 +``` +```json +{ + "ok": true +} +``` + +### Dashboard + +A dashboard is provided to offer real-time statistics and historical graphs. It looks like this in a large installation on a fairly busy day: + +(img) + +[Graphite](http://graphite.wikidot.com/) is required for historical data retrieval. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for details. + + +## Author + +Scott Schneider (sschneid@gmail.com) + + +## License + +vmpooler is distributed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0.html). See the [LICENSE](LICENSE) file for more details. + From 565e03445e11c8aa4ac47cae3df97108066d65e2 Mon Sep 17 00:00:00 2001 From: Scott Schneider Date: Wed, 19 Feb 2014 16:05:27 -0800 Subject: [PATCH 2/3] Initial --- vmpooler.yaml.example | 150 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 vmpooler.yaml.example diff --git a/vmpooler.yaml.example b/vmpooler.yaml.example new file mode 100644 index 0000000..4746db2 --- /dev/null +++ b/vmpooler.yaml.example @@ -0,0 +1,150 @@ +--- +# :vsphere: +# +# This section contains the server hostname and authentication credentials +# needed for vmpooler to connect to VMware vSphere. +# +# Available configuration parameters: +# +# - server +# The FQDN hostname of the VMware vSphere server. +# (required) +# +# - username +# The username used to authenticate VMware vSphere. +# (required) +# +# - password +# The password used to authenticate VMware vSphere. +# (required) + +# Example: + +:vsphere: + server: 'vsphere.company.com' + username: 'vmpooler' + password: 'swimsw1msw!m' + +# :redis: +# +# This section contains the server hostname and authentication credentials +# needed for vmpooler to connect to Redis. +# +# Available configuration parameters: +# +# - server +# The FQDN hostname of the Redis server. +# (optional; default: 'localhost') +# +# - username +# The username used to authenticate Redis. +# (optional) +# +# - password +# The password used to authenticate Redis. +# (optional) + +# Example: + +:redis: + server: 'redis.company.com' + +# :config: +# +# This section contains global configuration information. +# +# Available configuration parameters: +# +# - site_name +# The name of your deployment. +# (optional; default: 'vmpooler') +# +# - logfile +# The path to vmpooler's log file. +# (optional; default: '/var/log/vmpooler.log') +# +# - graphite +# The FQDN hostname of the Graphite server. +# (optional) +# +# - task_limit +# The number of concurrent VMware vSphere tasks to perform. +# (optional; default: '10') +# +# - vm_checktime +# How often (in minutes) to check the sanity of VMs in 'ready' queues. +# (optional; default: '15') +# +# - vm_lifetime +# How long (in hours) to keep VMs in 'running' queues before destroying. +# (optional; default: '24') + +# Example: + +:config: + site_name: 'vmpooler' + logfile: '/var/log/vmpooler.log' + graphite: 'graphite.company.com' + task_limit: 10 + vm_checktime: 15 + vm_lifetime: 12 + +# :pools: +# +# This section contains a list of virtual machine 'pools' for vmpooler to +# create and maintain. +# +# Available configuration parameters (per-pool): +# +# - name +# The name of the pool. +# (required) +# +# - template +# The template or virtual machine target to spawn clones from. +# (required) +# +# - folder +# The vSphere 'folder' destination for spawned clones. +# (required) +# +# - pool +# The vSphere 'resource pool' destination for spawned clones. +# (required) +# +# - datastore +# The vSphere 'datastore' destination for spawned clones. +# (required) +# +# - size +# The number of waiting VMs to keep in a pool. +# (required) +# +# - timeout +# How long (in minutes) before marking a clone as 'failed' and retrying. +# (optional; default: '15') +# +# - ready_ttl +# How long (in minutes) to keep VMs in 'ready' queues before destroying. +# (optional) + +# Example: + +:pools: + - name: 'debian-7-i386' + template: 'Templates/debian-7-i386' + folder: 'Pooled VMs/debian-7-i386' + pool: 'Pooled VMs/debian-7-i386' + datastore: 'vmstorage' + size: 5 + timeout: 15 + ready_ttl: 1440 + - name: 'debian-7-x86_64' + template: 'Templates/debian-7-x86_64' + folder: 'Pooled VMs/debian-7-x86_64' + pool: 'Pooled VMs/debian-7-x86_64' + datastore: 'vmstorage' + size: 5 + timeout: 15 + ready_ttl: 1440 + From 7efafd25dfdbfb4d92f1ffd1a73e0851599c75ea Mon Sep 17 00:00:00 2001 From: Scott Schneider Date: Fri, 21 Feb 2014 13:53:09 -0800 Subject: [PATCH 3/3] Fix filename typo --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0bd6712..ff52058 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ The following YAML configuration sets up two pools, `debian-7-i386` and `debian- size: 5 ``` -See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for additional configuration options and parameters. +See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), for additional configuration options and parameters. ### Template set-up @@ -64,7 +64,7 @@ Template set-up is left as an exercise to the reader. Somehow, either via PXE, ## API and Dashboard -vmpooler provides an API and web front-end (dashboard) on port `:4567`. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), to specify an alternative port to listen on. +vmpooler provides an API and web front-end (dashboard) on port `:4567`. See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), to specify an alternative port to listen on. ### API @@ -118,7 +118,7 @@ A dashboard is provided to offer real-time statistics and historical graphs. It (img) -[Graphite](http://graphite.wikidot.com/) is required for historical data retrieval. See the provided YAML configuration example, [vmpooler.yaml.example](vmware.yaml.example), for details. +[Graphite](http://graphite.wikidot.com/) is required for historical data retrieval. See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), for details. ## Author