mirror of
https://github.com/puppetlabs/vmpooler.git
synced 2026-01-26 10:08:40 -05:00
(POOLER-158) Add capability to provision VMs on demand
This change adds a capability to vmpooler to provision instances on demand. Without this change vmpooler only supports retrieving machines from pre-provisioned pools. Additionally, this change refactors redis interactions to reduce round trips to redis. Specifically, multi and pipelined redis commands are added where possible to reduce the number of times we are calling redis. To support the redis refactor the redis interaction has changed to leveraging a connection pool. In addition to offering multiple connections for pool manager to use, the redis interactions in pool manager are now thread safe. Ready TTL is now a global parameter that can be set as a default for all pools. A default of 0 has been removed, because this is an unreasonable default behavior, which would leave a provisioned instance in the pool indefinitely. Pool empty messages have been removed when the pool size is set to 0. Without this change, when a pool was set to a size of 0 the API and pool manager would both show that a pool is empty.
This commit is contained in:
kirby@puppetlabs.com
parent
1f6f08d172
commit
811fd8b60f
34 changed files with 3326 additions and 1098 deletions
91
docs/API.md
91
docs/API.md
|
|
@ -6,6 +6,7 @@
|
|||
5. [VM snapshots](#vmsnapshots)
|
||||
6. [Status and metrics](#statusmetrics)
|
||||
7. [Pool configuration](#poolconfig)
|
||||
8. [Ondemand VM provisioning](#ondemandvm)
|
||||
|
||||
### API <a name="API"></a>
|
||||
|
||||
|
|
@ -799,3 +800,93 @@ $ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"1"}' --
|
|||
"ok": true
|
||||
}
|
||||
```
|
||||
|
||||
#### Ondemand VM operations <a name="ondemandvm"></a>
|
||||
|
||||
Ondemand VM operations offer a user an option to directly request instances to be allocated for use. This can be very useful when supporting a wide range of images because idle instances can be eliminated.
|
||||
|
||||
##### POST /ondemandvm
|
||||
|
||||
All instance types requested must match a pool name or alias in the running application configuration, or a 404 code will be returned
|
||||
|
||||
When a provisioning request is accepted the API will return an indication that the request is successful. You may then poll /ondemandvm to monitor request status.
|
||||
|
||||
An authentication token is required in order to request instances on demand when authentication is configured.
|
||||
|
||||
Responses:
|
||||
* 201 - Provisioning request accepted
|
||||
* 400 - Payload contains invalid JSON and cannot be parsed
|
||||
* 401 - No auth token provided, or provided auth token is not valid, and auth is enabled
|
||||
* 403 - Request exceeds the configured per pool maximum
|
||||
* 404 - A pool was requested, which is not available in the running configuration, or an unknown error occurred.
|
||||
* 409 - A request of the matching ID has already been created
|
||||
```
|
||||
$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"4"}' --url https://vmpooler.example.com/api/v1/ondemandvm
|
||||
```
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"request_id": "e3ff6271-d201-4f31-a315-d17f4e15863a"
|
||||
}
|
||||
```
|
||||
|
||||
##### GET /ondemandvm
|
||||
|
||||
Get the status of an ondemandvm request that has already been posted.
|
||||
|
||||
When the request is ready the ready status will change to 'true'.
|
||||
|
||||
The number of instances pending vs ready will be reflected in the API response.
|
||||
|
||||
Responses:
|
||||
* 200 - The API request was successful and the status is ok
|
||||
* 202 - The request is not ready yet
|
||||
* 404 - The request can not be found, or an unknown error occurred
|
||||
```
|
||||
$ curl https://vmpooler.example.com/api/v1/ondemandvm/e3ff6271-d201-4f31-a315-d17f4e15863a
|
||||
```
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"request_id": "e3ff6271-d201-4f31-a315-d17f4e15863a",
|
||||
"ready": false,
|
||||
"debian-7-i386": {
|
||||
"ready": "3",
|
||||
"pending": "1"
|
||||
}
|
||||
}
|
||||
```
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"request_id": "e3ff6271-d201-4f31-a315-d17f4e15863a",
|
||||
"ready": true,
|
||||
"debian-7-i386": {
|
||||
"hostname": [
|
||||
"vm1",
|
||||
"vm2",
|
||||
"vm3",
|
||||
"vm4"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
##### DELETE /ondemandvm
|
||||
|
||||
Delete a ondemand request
|
||||
|
||||
Deleting a ondemand request will delete any instances created for the request and mark the backend data for expiration in two weeks. Any subsequent attempts to retrieve request data will indicate it has been deleted.
|
||||
|
||||
Responses:
|
||||
* 200 - The API request was sucessful. A message will indicate if the request has already been deleted.
|
||||
* 401 - No auth token provided, or provided auth token is not valid, and auth is enabled
|
||||
* 404 - The request can not be found, or an unknown error occurred.
|
||||
```
|
||||
$ curl -X DELETE https://vmpooler.example.com/api/v1/ondemandvm/e3ff6271-d201-4f31-a315-d17f4e15863a
|
||||
```
|
||||
```json
|
||||
{
|
||||
"ok": true
|
||||
}
|
||||
```
|
||||
|
|
|
|||
|
|
@ -74,6 +74,16 @@ The prefix to use while storing Graphite data.
|
|||
The TCP port to communicate with the graphite server.
|
||||
(optional; default: 2003)
|
||||
|
||||
### MAX\_ONDEMAND\_INSTANCES\_PER\_REQUEST
|
||||
|
||||
The maximum number of instances any individual ondemand request may contain per pool.
|
||||
(default: 10)
|
||||
|
||||
### ONDEMAND\_REQUEST\_TTL
|
||||
|
||||
The amount of time (in minutes) to give for a ondemand request to be fulfilled before considering it to have failed.
|
||||
(default: 5)
|
||||
|
||||
## Manager options <a name="manager"></a>
|
||||
|
||||
### TASK\_LIMIT
|
||||
|
|
@ -123,6 +133,11 @@ The target cluster VMs are cloned into (host with least VMs chosen)
|
|||
How long (in minutes) before marking a clone as 'failed' and retrying.
|
||||
(optional; default: 15)
|
||||
|
||||
### READY\_TTL
|
||||
|
||||
How long (in minutes) a ready VM should stay in the ready queue.
|
||||
(default: 60)
|
||||
|
||||
### MAX\_TRIES
|
||||
|
||||
Set the max number of times a connection should retry in VM providers. This optional setting allows a user to dial in retry limits to suit your environment.
|
||||
|
|
@ -130,7 +145,7 @@ Set the max number of times a connection should retry in VM providers. This opti
|
|||
|
||||
### RETRY\_FACTOR
|
||||
|
||||
When retrying, each attempt sleeps for the try count * retry_factor.
|
||||
When retrying, each attempt sleeps for the try count * retry\_factor.
|
||||
Increase this number to lengthen the delay between retry attempts.
|
||||
This is particularly useful for instances with a large number of pools
|
||||
to prevent a thundering herd when retrying connections.
|
||||
|
|
@ -183,6 +198,21 @@ The argument can accept a full path to a file, or multiple files comma separated
|
|||
Expects a string value
|
||||
(optional)
|
||||
|
||||
### ONDEMAND\_CLONE\_LIMIT
|
||||
|
||||
Maximum number of simultaneous clones to perform for ondemand provisioning requests.
|
||||
(default: 10)
|
||||
|
||||
### REDIS\_CONNECTION\_POOL\_SIZE
|
||||
|
||||
Maximum number of connections to utilize for the redis connection pool.
|
||||
(default: 10)
|
||||
|
||||
### REDIS\_CONNECTION\_POOL\_TIMEOUT
|
||||
|
||||
How long a task should wait (in seconds) for a redis connection when all connections are in use.
|
||||
(default: 5)
|
||||
|
||||
## API options <a name="API"></a>
|
||||
|
||||
### AUTH\_PROVIDER
|
||||
|
|
@ -221,3 +251,8 @@ The name of your deployment.
|
|||
Enable experimental API capabilities such as changing pool template and size without application restart
|
||||
Expects a boolean value
|
||||
(optional; default: false)
|
||||
|
||||
### MAX\_LIFETIME\_UPPER\_LIMIT
|
||||
|
||||
Specify a maximum lifetime that a VM may be extended to in hours.
|
||||
(optional)
|
||||
|
|
|
|||
|
|
@ -1,13 +1,15 @@
|
|||
# Setting up a vmpooler development environment
|
||||
|
||||
## Requirements
|
||||
## Docker is the preferred development environment
|
||||
|
||||
The docker compose file is the easiest way to get vmpooler running with any local code changes. The docker compose file expects to find a vmpooler.yaml configuration file in the root vmpooler directory. The file is mapped into the running container for the vmpooler application. This file primarily contains the pools configuration. Nearly all other configuration can be supplied with environment variables.
|
||||
|
||||
## Requirements for local installation directly on your system (not recommended)
|
||||
|
||||
* Supported on OSX, Windows and Linux
|
||||
|
||||
* Ruby or JRuby
|
||||
|
||||
Note - Ruby 1.x support will be removed so it is best to use more modern ruby versions
|
||||
|
||||
Note - It is recommended to user Bundler instead of installing gems into the system repository
|
||||
|
||||
* A local Redis server
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue