The read_timeout on Net::HTTP connections may not reliably interrupt
blocking Java SSL socket reads in JRuby. Add an explicit Timeout.timeout
wrapper around the connection pool block in vms_in_pool, get_vm, and
get_vm_ip_address so that if vSphere hangs mid-operation (after the
health check passes), the Timeout::Error (a StandardError in JRuby 9.x)
propagates to the circuit breaker which counts it as a failure.
After 5 consecutive timeouts the circuit opens and subsequent check_pool
cycles fail immediately rather than blocking all pool threads.
Without this change, the circuit breaker was initialized in base.rb
but never called in the vsphere provider, so vSphere API failures
(TCP timeouts, connection errors) would not trip the circuit open.
Changes:
- Add with_circuit_breaker helper that calls @circuit_breaker.call
if circuit_breaker is configured, otherwise yields directly
- Wrap vms_in_pool, get_vm, create_vm, destroy_vm, get_vm_ip_address
with with_circuit_breaker so vSphere failures trip the circuit
- Replace 'return' with 'next' inside blocks where needed to ensure
circuit breaker on_success is properly called on partial results
This prevents cascading failures: once the circuit opens after
failure_threshold errors, subsequent calls fail fast (CircuitOpenError)
instead of waiting for the full TCP timeout (~18s per pool).
Resolves: P4DEVOPS-9438
VMPooler has the vSphere provider taken out, moving some vSphere related
methods to the provider:
1) pool_folders
2) get_base_folders
And the related spec tests.
At the same time renaming some configuration and code items
to remove harmful terminology.
Note this version of the vsphere provider needs to run on vmpooler
that also contain the renaming changes (version >2.1)
