there was one API that was falling back on v1 and was returning a domain key.
in order to make it consistent with the changes in v2, the domain is not returned
anymore, and the fqdn is returned if it is available
the vm_ready methods should be implemented per provider and checking in the api
creates issues since provider code is not available. Removing this for the V2 api
The syntax for pipelined has changed and the old syntax will be removed in v5.0.0
Fixing the syntax now since the block syntax has been supported for a while now.
This implements a delete method for pooltemplate and poolsize. The API
removes the override from Redis and then adds an entry in Redis that
causes the pool manager to wake up and process the removal of the
override.
To facilitate this, a new variable has been created in lib/vmpooler.rb
to hold a copy of the original / pre-override config. This supplemental
copy of the pools is then indexed for use as a reference.
When pool manager wakes up to process an override removal, it looks up
the pre-override value from the config via the new variables mentioned
above.
Just as with entering overrides, no restart is needed. Template and pool
size changes are logged so that anyone watching or reviewing the logs
can see what happened when. The new API endpoints also return values for
both the pre-revert and post-revert value.
Prior to this, the encryption settings for LDAP auth were hard coded to
start_tls on port 389 with TLSv1. These are still the defaults, as
insecure as they are, so as to not break existing users. This change
facilitates replacing the defaults so that simple_tls over port 636 via
TLS1.2 can be used.
This adds an "operation" label to the user metrics and moves incrementing from the manager to api, so that the user metrics show when resources are allocated, as well as destroyed. Previously, user metrics were only updated upon destroying a resource.
I think its better suited to increment the metric as part of the api instead of the pool_manger, because it's expected to do so when a user successfully checks out or deletes a VM, but can be problematic when doing so in the provider since it can clone VMs before actually being checked out by a user.
This change removes usage of redis multi from API. Without this change the redis usage can cause issues because no connection pool is used, so another worker may try to use the same backend object causing a failure.
Checkout metric counters were against the template name and not the
actual pool used which prevents us from counting the checkouts in the
pixa4 pools for example.
Note - this is a re-rerun - the last commit on this file over-wrote
the change.
This change fixes an error in validate_token where hset does not have the correct number of parameters. Without this change validate_token causes a error.
This change sets a VM as running in redis as soon as it is checked out. Without this change when allocated several instances it is possible for a machine that has been allocated for a checkout, but not yet marked as active, to be identified as running when it should not be, which was added in POOLER-191. Without this change a machine may be destroyed during checkout by pool_manager if there are several instances being allocated.
Additionally, redis multi is added for vm checkout operations to minimize the round trips to redis during a checkout operation. Without this addition each VM checkout causes several redis interactions.
Checkout metric counters were against the template name and not the
actual pool used which prevents us from counting the checkouts in the
pixa4 pools for example.
* (maint) Speedup the tagging method
While looking at the instrumentation data for the ABS queue processor,
I noticed a lot of time spent in the HTTP PUT method, which in the code
was easy to isolate, as it is only used via the vmpooler tagging functions
ie the API /vm/foobar/ with 'tag' key-value pairs.
While I'm not sure the original hset() make sense to me, there was an easy
way to speed them up by using pipelined. I would expect a very good speed
increase with this turned on.
* tag rubocop to <1.0 because the 1.0 version returns 130 new offenses
This change fixes template alias evaluation to ensure that the correct
data is set when generating on demand requests for pools that have a
backend weight configured for a value of 0. Without this change vmpooler
will return an empty selection in api for template alias evaluation.
To support this change tests are added that first reproduced the
failure, and then verified that it is resolved with the addition of the
patch. Additionally, test coverage is added to ensure that code paths
that include pickup gem usage are covered.
Introducing the Prometheus Stats code into ABS showed that the Clarity
could be improved a bit with better variable naming, some refactoring
to reduce repitition and documenting the Metrics table itself.
Filtering these changes back to the vmpooler code base.
This commit updates the method used for chceking the status of an
ondemand request to ensure that if multiple aliases are used to fulfill
a request that they are correctly presented as a single pool again when
everything is ready. Without this change it is possible for only one
group of an aliased pool to show up in pending or completed requests.
Ensure that the correct stats are registered for the Manager and the api
respectively. E.g. all checkout counters are for the api only, whereas
clone times belong to the manager.
Also new ondemand functionality stats weren't registered, so add these
along with missing delete stats.
Use the example provided in the Ruby Client to provide a customised
collector appropriate to log all calls to the API. The customised
filtering is used to replace individual node names and templates
for the /vm and request ID's for the /ondemand endpoints.
This module was failing our rubocop checks so have updated it since
it now forms part of vmpooler.
Separate trapping for litmus jobs is also included so that they don't
interfere with stats from the jenkins pipelines.
* (POOLER-174) Reduce duplicate of on demand code introduced in POOLER-158
refactored every parsing of request of type 'pool_alias:pool:count' into a
utility class, that is used by pool_manager and the api v1 class
* add some metrics to the od request generation
* fix rubocop offenses, we are now friends
This change adds a capability to vmpooler to provision instances on
demand. Without this change vmpooler only supports retrieving machines
from pre-provisioned pools.
Additionally, this change refactors redis interactions to reduce round
trips to redis. Specifically, multi and pipelined redis commands are
added where possible to reduce the number of times we are calling redis.
To support the redis refactor the redis interaction has changed to
leveraging a connection pool. In addition to offering multiple
connections for pool manager to use, the redis interactions in pool
manager are now thread safe.
Ready TTL is now a global parameter that can be set as a default for all
pools. A default of 0 has been removed, because this is an unreasonable
default behavior, which would leave a provisioned instance in the pool
indefinitely.
Pool empty messages have been removed when the pool size is set to 0.
Without this change, when a pool was set to a size of 0 the API and pool
manager would both show that a pool is empty.
Before this change if the smove returned false, we would continue handing out the VM
which presumably could still be in the 'ready' state. Upon 'delete' that ready VM would not be
picked up and return a 404 which is consistent with the behavior seen. Adding a metric to keep
track of the smove failures since this is not expected. I think some API logging would be good
to add in the future.
Before this PR, the current running time was being inspected to decide if the
vm lifetime could be extended. But since vm lifetime is absolute and not relative
this check is now removed.
This commit fixes a bug in update_clone_target where I believe `=` was
intended, not `==` because `==` just goes to the void context here.
Thanks Rubocop Lint/Void!
This commit updates places where previously we were casting both terms
in a float division into floats in order to ensure that float division
occurs to use the `fdiv` method, which will always do float division and
is available on both `Floats` and `Integers` because they are
both `Numeric`.
This commit updates the dashboard to use `URI.parse` instead of `Kernel#open`
because open can potentially open files on the server and has other
possible security issues.
Also updated the logger to use `File.open` as it is more explicit and
doesn't have the extra potential for abuse like `Kernel#open`
https://rubocop.readthedocs.io/en/latest/cops_security/#securityopen
