Upgrade cloudscale.ch Terraform module from v1 to v2

terraform-openshift4-cloudscale v2.0.0 introduces Puppet-managed load balancer VMs managed by VSHN.

That release includes breaking changes which require manual steps to upgrade existing clusters.

Prerequisites

You need to be able to execute the following CLI tools locally:

ssh-keygen
jq
yq yq YAML processor (Version 4 or newer)
vault Vault CLI
base64
curl
commodore
git

Setup environment variables

Access to API

# For example: https://api.syn.vshn.net
# IMPORTANT: do NOT add a trailing `/`. Commands below will fail.
export COMMODORE_API_URL=<lieutenant-api-endpoint>
export COMMODORE_API_TOKEN=<lieutenant-api-token>

export CLUSTER_ID=<lieutenant-cluster-id> # Looks like: c-<something>
export TENANT_ID=$(curl -sH "Authorization: Bearer ${COMMODORE_API_TOKEN}" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r .tenant)

Update credentials

Add an additional cloudscale.ch token for Floaty and configure GitLab repository as described in Use cloudscale.ch

Connect with Vault

export VAULT_ADDR=https://vault-prod.syn.vshn.net
vault login -method=ldap username=<your.name>

Prepare SSH key

We generate a unique SSH key pair for the cluster for troubleshooting access to the cluster hosts.

SSH_PRIVATE_KEY="$(pwd)/ssh_$CLUSTER_ID"
export SSH_PUBLIC_KEY="${SSH_PRIVATE_KEY}.pub"

ssh-keygen -C "vault@$CLUSTER_ID" -t ed25519 -f $SSH_PRIVATE_KEY -N ''

vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/cloudscale/ssh \
  private_key=$(cat $SSH_PRIVATE_KEY | base64 --wrap 0)

Update component parameters

Compile the cluster catalog to create a local working directory
```
commodore catalog compile "${CLUSTER_ID}"
```

Set the ssh_keys parameter

pubkey=$(cat "${SSH_PUBLIC_KEY}")
yq eval -i '.parameters.openshift4_terraform.terraform_variables.ssh_keys += [strenv(pubkey)]' \
  "inventory/classes/${TENANT_ID}/${CLUSTER_ID}.yml"

Set the Git author (Optional)

GITLAB_CI_USERNAME="Max Mustermann" (1)
GITLAB_CI_EMAIL="mm@example.com" (2)

yq eval -i '.parameters.openshift4_terraform.gitlab_ci.git = {
    "username": strenv(GITLAB_CI_USERNAME),
    "email": strenv(GITLAB_CI_EMAIL),
  }' \
  "inventory/classes/${TENANT_ID}/${CLUSTER_ID}.yml"

1	The Git author name to use when creating hieradata commits from GitLab CI. If not specified, the GitLab CI defaults will be used.
2	The Git author email address to use when creating hieradata commits from GitLab CI. If not specified, the GitLab CI defaults will be used.

Commit and push the changes in the tenant repo

pushd "inventory/classes/${TENANT_ID}"
git commit -av
git push origin master
popd

Compile and push the cluster catalog

commodore catalog compile "${CLUSTER_ID}" --push -i