Merge pull request #92 from puppetlabs/add-google-clouddns

(RE-15124) Bump vmpooler, providers, and add new dns gem
2026-01-26 03:28:41 -05:00 · 2023-04-19 11:06:00 -04:00 · 2023-04-19 11:06:00 -04:00 · 70a41b89d9
commit 70a41b89d9
parent 9b123644b4 db4058f1be
11 changed files with 89 additions and 48 deletions
--- a/.gitignore
+++ b/.gitignore
@ -1,5 +1,6 @@
 /*/**/vendor/bundle/
 /docker/**/vmpooler.yaml
+/docker/data/*.json
 .vagrant/
 .idea/
 helm-charts/vmpooler/charts/
--- a/README.md
+++ b/README.md
@ -1,16 +1,15 @@
 # vmpooler-deployment

- [vmpooler-deployment](#vmpooler-deployment)
-  - [VMPooler Components](#vmpooler-components)
-  - [Docker Registry](#docker-registry)
-  - [Helm Repository](#helm-repository)
-    - [Adding / updating charts](#adding--updating-charts)
-  - [Development](#development)
-    - [Docker Compose URLs](#docker-compose-urls)
-    - [Deploy Chart for Testing](#deploy-chart-for-testing)
-  - [Releasing](#releasing)
-  - [Contributing](#contributing)
-  - [License](#license)
+- [VMPooler Components](#vmpooler-components)
+- [Docker Registry](#docker-registry)
+- [Helm Repository](#helm-repository)
+  - [Adding / updating charts](#adding--updating-charts)
+- [Development](#development)
+  - [Docker Compose URLs](#docker-compose-urls)
+  - [Deploy Chart for Testing](#deploy-chart-for-testing)
+- [Releasing](#releasing)
+- [Contributing](#contributing)
+- [License](#license)

 This repo contains Dockerfiles and a Helm chart that can be used to deploy [VMPooler](https://github.com/puppetlabs/vmpooler). The Release Engineering team at Puppet uses the code here as part of operating our VMPooler instances.

@ -19,6 +18,7 @@ This repo contains Dockerfiles and a Helm chart that can be used to deploy [VMPo
 The docker image gnerated and hosted by this project contain the following VMPooler components:

 - [VMPooler Core](https://github.com/puppetlabs/vmpooler)
+- [VMPooler Google CloudDNS Plugin](https://github.com/puppetlabs/vmpooler-dns-google-clouddns)
 - [VMPooler EC2 Provider](https://github.com/puppetlabs/vmpooler-provider-ec2)
 - [VMPooler GCE Provider](https://github.com/puppetlabs/vmpooler-provider-gce)
 - [VMPooler vSphere Provider](https://github.com/puppetlabs/vmpooler-provider-vsphere)
@ -56,6 +56,8 @@ Prerequisites:
      - [vmpooler-provider-ec2](https://github.com/puppetlabs/vmpooler-provider-ec2)
      - [vmpooler-provider-gce](https://github.com/puppetlabs/vmpooler-provider-gce)
      - [vmpooler-provider-vsphere](https://github.com/puppetlabs/vmpooler-provider-vsphere)
+   - If you are not using Dynamic DNS, then the following DNS plugins can be used to manage records across different compute providers:
+     - [VMPooler Google CloudDNS Plugin](https://github.com/puppetlabs/vmpooler-dns-google-clouddns)
 2. Chose a Development method:
   - Develop via local source:
      1. Clone all of the known vmpooler projects listed at [vmpooler-components](#vmpooler-components) under a common directory, for example:
@ -63,6 +65,7 @@ Prerequisites:
           ```bash
           |-- vmpooler-projects
           |  |--vmpooler-deployment
+           |  |--vmpooler-dns-google-clouddns
           |  |--vmpooler-provider-ec2
           |  |--vmpooler-provider-gce
           |  |--vmpooler-provider-vsphere
@ -72,10 +75,10 @@ Prerequisites:
      3. Run `docker compose -f vmpooler-deployment/docker/docker-compose.local.yml up`

   - Develop via Git source:
-      1. For the component(s) you are developing on, commit and push changes to a branch.
-      2. Change to the `docker` directory and modify the `Gemfile` as needed to pull the gem(s) from your git branch.
-      3. Run `./update-gemfile-lock` to update the `Gemfile.lock`
-      4. Run `docker compose build && docker compose up`.
+      4. For the component(s) you are developing on, commit and push changes to a branch.
+      5. Change to the `docker` directory and modify the `Gemfile` as needed to pull the gem(s) from your git branch.
+      6. Run `./update-gemfile-lock` to update the `Gemfile.lock`
+      7. Run `docker compose build && docker compose up`.

 When a dependency Helm chart is updated, be sure to run `./update-chart-lock` to update the lockfile, otherwise the test and release workflows will fail.

--- a/docker/Dockerfile.local
+++ b/docker/Dockerfile.local
@ -25,6 +25,7 @@ RUN apt-get update -qq && \
    rm -rf /var/lib/apt/lists/*

 COPY vmpooler /opt/vmpooler
+COPY vmpooler-dns-gcp /opt/vmpooler-dns-gcp
 COPY vmpooler-provider-ec2 /opt/vmpooler-provider-ec2
 COPY vmpooler-provider-gce /opt/vmpooler-provider-gce
 COPY vmpooler-provider-vsphere /opt/vmpooler-provider-vsphere
--- a/docker/Gemfile
+++ b/docker/Gemfile
@ -1,13 +1,15 @@
 source ENV['GEM_SOURCE'] || 'https://rubygems.org'

-gem 'vmpooler',                    '~> 2.5'
+gem 'vmpooler',                    '~> 3.0'
+gem 'vmpooler-dns-gcp',            '~> 1.0'
 # Remove temporarily until a stable minor release is published
 # gem 'vmpooler-provider-ec2',       '~> 0.0.1'
-gem 'vmpooler-provider-gce',       '~> 0.5'
-gem 'vmpooler-provider-vsphere',   '~> 2.1'
+gem 'vmpooler-provider-gce',       '~> 1.0'
+gem 'vmpooler-provider-vsphere',   '~> 3.0'

 # For development install via a git branch use something like:
 # gem 'vmpooler', git: 'https://github.com/puppetlabs/vmpooler.git', branch: 'my-feature'
+# gem 'vmpooler-dns-gcp', git: 'https://github.com/puppetlabs/vmpooler-dns-gcp.git', branch: 'my-feature'
 # gem 'vmpooler-provider-ec2', git: 'https://github.com/puppetlabs/vmpooler-provider-ec2.git', branch: 'my-feature'
 # gem 'vmpooler-provider-gce', git: 'https://github.com/puppetlabs/vmpooler-provider-gce.git', branch: 'my-feature'
 # gem 'vmpooler-provider-vsphere', git: 'https://github.com/puppetlabs/vmpooler-provider-vsphere.git', branch: 'my-feature'
--- a/docker/Gemfile.local
+++ b/docker/Gemfile.local
@ -1,6 +1,7 @@
 source ENV['GEM_SOURCE'] || 'https://rubygems.org'

 gem 'vmpooler', path: '/opt/vmpooler'
+gem 'vmpooler-dns-gcp', path: '/opt/vmpooler-dns-gcp'
 gem 'vmpooler-provider-ec2', path: '/opt/vmpooler-provider-ec2'
 gem 'vmpooler-provider-gce', path: '/opt/vmpooler-provider-gce'
 gem 'vmpooler-provider-vsphere', path: '/opt/vmpooler-provider-vsphere'
--- a/docker/Gemfile.lock
+++ b/docker/Gemfile.lock
@ -1,19 +1,19 @@
 GEM
  remote: https://rubygems.org/
  specs:
-    addressable (2.8.1)
+    addressable (2.8.4)
      public_suffix (>= 2.0.2, < 6.0)
    bindata (2.4.15)
    builder (3.2.4)
    concurrent-ruby (1.2.2)
-    connection_pool (2.3.0)
+    connection_pool (2.4.0)
    declarative (0.0.20)
    deep_merge (1.2.2)
    faraday (2.7.4)
      faraday-net_http (>= 2.0, < 3.1)
      ruby2_keywords (>= 0.0.4)
    faraday-net_http (3.0.2)
-    google-apis-compute_v1 (0.63.0)
+    google-apis-compute_v1 (0.66.0)
      google-apis-core (>= 0.11.0, < 2.a)
    google-apis-core (0.11.0)
      addressable (~> 2.5, >= 2.5.1)
@ -24,7 +24,7 @@ GEM
      retriable (>= 2.0, < 4.a)
      rexml
      webrick
-    google-apis-dns_v1 (0.31.0)
+    google-apis-dns_v1 (0.32.0)
      google-apis-core (>= 0.11.0, < 2.a)
    google-cloud-core (1.6.0)
      google-cloud-env (~> 1.0)
@ -52,9 +52,9 @@ GEM
    multi_json (1.15.0)
    mustermann (2.0.2)
      ruby2_keywords (~> 0.0.1)
-    net-ldap (0.17.1)
-    nio4r (2.5.8-java)
-    nokogiri (1.14.2-java)
+    net-ldap (0.18.0)
+    nio4r (2.5.9-java)
+    nokogiri (1.14.3-java)
      racc (~> 1.4)
    opentelemetry-api (1.1.0)
    opentelemetry-common (0.19.6)
@ -101,7 +101,7 @@ GEM
    puma (5.6.5-java)
      nio4r (~> 2.0)
    racc (1.6.2-java)
-    rack (2.2.6.3)
+    rack (2.2.6.4)
    rack-protection (2.2.4)
      rack
    rake (13.0.6)
@ -135,7 +135,7 @@ GEM
    tilt (2.1.0)
    trailblazer-option (0.1.2)
    uber (0.1.0)
-    vmpooler (2.5.0)
+    vmpooler (3.0.0)
      concurrent-ruby (~> 1.1)
      connection_pool (~> 2.2)
      deep_merge (~> 1.2)
@ -156,14 +156,17 @@ GEM
      sinatra (~> 2.0)
      spicy-proton (~> 2.1)
      statsd-ruby (~> 1.4)
-    vmpooler-provider-gce (0.5.0)
-      google-apis-compute_v1 (~> 0.14)
+    vmpooler-dns-gcp (1.0.0)
      google-cloud-dns (~> 0.35.1)
      googleauth (>= 0.16.2, < 1.3.0)
-      vmpooler (~> 2.3, >= 1.3.0)
-    vmpooler-provider-vsphere (2.1.0)
+      vmpooler (~> 3.0)
+    vmpooler-provider-gce (1.0.0)
+      google-apis-compute_v1 (~> 0.14)
+      googleauth (>= 0.16.2, < 1.3.0)
+      vmpooler (~> 3.0)
+    vmpooler-provider-vsphere (3.0.0)
      rbvmomi2 (>= 3.1, < 4.0)
-      vmpooler (~> 2.4)
+      vmpooler (~> 3.0)
    webrick (1.8.1)
    zonefile (1.06)

@ -171,9 +174,10 @@ PLATFORMS
  universal-java-11

 DEPENDENCIES
-  vmpooler (~> 2.5)
-  vmpooler-provider-gce (~> 0.5)
-  vmpooler-provider-vsphere (~> 2.1)
+  vmpooler (~> 3.0)
+  vmpooler-dns-gcp (~> 1.0)
+  vmpooler-provider-gce (~> 1.0)
+  vmpooler-provider-vsphere (~> 3.0)

 BUNDLED WITH
-   2.4.7
+   2.4.8
--- a/docker/data/vmpooler.yaml
+++ b/docker/data/vmpooler.yaml
@ -1,5 +1,18 @@
 ---
+:dns_configs:
+  ddns_domain: 'dynamic.example.com'
+  :gcp-clouddns:
+    project: vmpooler-example
+    domain: vmpooler.example.com
+    dns_zone_resource_name: vmpooler-example-com
+
 :providers:
+  # :gce:
+  #   network_name: projects/vmpooler-example/global/networks/default
+  #   project: vmpooler-example
+  #   title: gce
+  #   zone: us-west1-b
+
  :dummy:
    filename: '/tmp/dummy-backing.yaml'

@ -23,13 +36,19 @@
  allowed_tags:
    - 'created_by'
    - 'project'
-  domain: 'example.com'
-  # domain: 'localhost' # Flip these out for local requests

 :pools:
-  - name: 'debian-10-x86_64'
-    template: 'Templates/debian-7-x86_64'
-    folder: 'Pooled VMs/debian-7-x86_64'
+  # - name: 'fedora-36-x86_64'
+  #   template: 'projects/fedora-cloud/global/images/fedora-cloud-base-gcp-36-20220506-n-0-x86-64'
+  #   size: 0
+  #   provider: 'gce'
+  #   dns_plugin: 'google-clouddns'
+  #   machine_type: 'zones/us-west1-b/machineTypes/n1-standard-2'
+  #   disk_type: 'pd-ssd'
+  #   subnetwork_name: 'projects/vmpooler-example/regions/us-west1/subnetworks/default'
+  - name: 'debian-11-x86_64'
+    template: 'Templates/debian-11-x86_64'
+    folder: 'Pooled VMs/debian-11-x86_64'
    datastore: 'vmstorage'
    size: 5
    timeout: 15
--- a/docker/docker-compose.local.yml
+++ b/docker/docker-compose.local.yml
@ -41,16 +41,21 @@ services:
      - type: bind
        source: ./data/vmpooler.yaml
        target: /etc/vmpooler/vmpooler.yaml
+      # - type: bind
+      #   source: ./data/vmpooler-example-key.json
+      #   target: /etc/vmpooler/vmpooler-example-key.json
    ports:
      - "8082:4567"
    environment:
      - VMPOOLER_DEBUG=true # for use of dummy auth
+      - DEBUG_FLAG=true
      - VMPOOLER_CONFIG_FILE=/etc/vmpooler/vmpooler.yaml
      - REDIS_SERVER=redis-server
      - REDIS_PASSWORD=vmpooler
      - LOGFILE=/dev/null
      - VMPOOLER_TRACING_ENABLED=true
      - VMPOOLER_TRACING_JAEGER_HOST=http://jaeger-aio:14268/api/traces
+      # - GOOGLE_APPLICATION_CREDENTIALS=/etc/vmpooler/vmpooler-example-key.json
    command: api
    links:
      - redis-server
@ -63,16 +68,21 @@ services:
      - type: bind
        source: ./data/vmpooler.yaml
        target: /etc/vmpooler/vmpooler.yaml
+      # - type: bind
+      #   source: ./data/vmpooler-example-key.json
+      #   target: /etc/vmpooler/vmpooler-example-key.json
    ports:
      - "8083:4567"
    environment:
      - VMPOOLER_DEBUG=true # for use of dummy auth
+      - DEBUG_FLAG=true
      - VMPOOLER_CONFIG_FILE=/etc/vmpooler/vmpooler.yaml
      - REDIS_SERVER=redis-server
      - REDIS_PASSWORD=vmpooler
      - LOGFILE=/dev/null
      - VMPOOLER_TRACING_ENABLED=true
      - VMPOOLER_TRACING_JAEGER_HOST=http://jaeger-aio:14268/api/traces
+      # - GOOGLE_APPLICATION_CREDENTIALS=/etc/vmpooler/vmpooler-example-key.json
    command: manager
    links:
      - redis-server
--- a/helm-charts/vmpooler/templates/deployment-api.yaml
+++ b/helm-charts/vmpooler/templates/deployment-api.yaml
@ -46,13 +46,13 @@ spec:
          - 'api'
        livenessProbe:
          httpGet:
-            path: /api/v1/status
+            path: /api/v3/status
            port: {{ .Values.service.port }}
          initialDelaySeconds: 40
          periodSeconds: 10
        readinessProbe:
          httpGet:
-            path: /api/v1/status
+            path: /api/v3/status
            port: {{ .Values.service.port }}
          initialDelaySeconds: 40
          periodSeconds: 10
@ -63,8 +63,6 @@ spec:
          value: {{ .Values.service.port | quote  }}
        - name: SITE_NAME
          value: '<b>{{ .Values.vmpoolerInstance }}</b>.{{ .Values.ingress.domain }}'
-        - name: DOMAIN
-          value: {{ .Values.sut_domain }}
        - name: EXPERIMENTAL_FEATURES
          value: 'true'
        - name: USAGE_STATS
--- a/helm-charts/vmpooler/templates/deployment-manager.yaml
+++ b/helm-charts/vmpooler/templates/deployment-manager.yaml
@ -93,8 +93,6 @@ spec:
          value: '20'
        - name: PURGE_UNCONFIGURED_FOLDERS
          value: 'true'
-        - name: DOMAIN
-          value: {{ .Values.sut_domain }}
        - name: VMPOOLER_CONFIG_FILE
          value: /etc/vmpooler/vmpooler.yaml
        - name: VMPOOLER_TRACING_ENABLED
--- a/helm-charts/vmpooler/values.yaml
+++ b/helm-charts/vmpooler/values.yaml
@ -8,7 +8,6 @@ image:

 jruby_opts: '-J-XX:+UseG1GC'
 configmapName: 'vmpooler-baseconfig'
-sut_domain: 'vmpooler.example.com'
 usage_stats: true
 ondemand_request_ttl: '50'
 ondemand_clone_limit: '50'
@ -17,6 +16,11 @@ vm_lifetime: '2'
 vm_lifetime_auth: '12'
 timeout: '15'

+:dns_configs:
+  :example:
+    dns_class: dynamic-dns
+    domain: 'vmpooler.example.com'
+
 # .json private key, set in ENV var
 gce:
  keyfile: '/vault/secrets/gce-keyfile.json'