Restore from Backup

When configuring component-openbao with backup.enabled = true, the component sets up regular backups using k8up. This how-to explains how OpenBao can be restored from such a backup.

Information on the OpenBao Unseal Key and Root Token

component-openbao leverages k8up’s application specific backups to create OpenBao snapshots. Restoring such a snapshot is only possible by providing the OpenBao unseal key and root token. As these are sensitive secrets, they are not backed up as part of the component’s automated backup process.

When setting up OpenBao, you must store the OpenBao unseal key and root token in a safe location in order to be able to restore backups.

In a running OpenBao instance, both of these secrets can be found in the [instance name]-seal secret in the OpenBao instance’s namespace. In order to have a reliable OpenBao backup, this secret must be backed up separately to a safe location.

Prerequisites

  • restic - command line tool to access backups made by k8up

  • bao - command line tool to interact with OpenBao

  • kubectl

  • Write access to the cluster’s tenant repository

  • Read access to the restic repository in which k8up stored the OpenBao backups

  • The OpenBao instance’s unseal key and root token - these must be backed up manually; they are not part of the automated k8up backup.

Procedure

1. Set up new OpenBao instance

  1. Add the openbao application to your cluster configuration.

    1. If your old instance of OpenBao is still running on the cluster, you can use component instantiation to create a second instance by adding openbao as new-openbao to your application list, and configuring it under new_openbao.

  2. Initially disable backups by setting .backups.enabled to false

  3. Compile and push the cluster config and wait for OpenBao to start.

2. Retrieve the OpenBao snapshot

  1. Set up the restic credentials (values correspond to the component parameters backup.bucket and backup.password)

    export AWS_ACCESS_KEY_ID="S3_KEY" # from component configuration: backup.bucket.accesskey
export AWS_SECRET_ACCESS_KEY="S3_SECRET" # from component confiugration: backup.bucket.secretkey
export RESTIC_REPOSITORY="s3:https://path.to.my/bucket"
export RESTIC_PASSWORD="RESTIC_REPO_KEY" # from component configuration: backup.password

  2. Retrieve the latest OpenBao snapshot to your local disk

    mkdir restore
restic restore --target restore/ latest

  3. Verify the snapshot file

    ls restore
# This should show a file named "[instance name]-backup.snapshot"

3. Restore the snapshot

  1. Expose the OpenBao pod

    kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200

  2. In a separate terminal, prepare the environment to access OpenBao

    # Get root token to log in
export VAULT_TOKEN="$(kubectl get secret -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-seal -ojsonpath='{.data.vault-root}' | base64 -d)"
export VAULT_ADDR="http://127.0.0.1:8200"

  3. Restore the backup

    bao operator raft snapshot restore -force restore/${VAULT_INSTANCE_NAME}-backup.snapshot

4. Unseal OpenBao

If you were logged into the OpenBao UI, you should have gotten logged out now. This is expected.

  1. Open your browser at localhost:8200

  2. Use the OpenBao Unseal Key of the OpenBao instance you’ve just restored to unseal OpenBao

  3. Use the OpenBao root token of the OpenBao instance you’ve just restored to log in with the Token method

  4. Verify that the restore worked, and secrets are now restored in OpenBao.

The unseal key and root token of the OpenBao instance you’re restoring need to have been stored separately. Without them, the restore procedure cannot be completed.

5. Update the OpenBao Secret

Without this step, your OpenBao instance will not be able to auto-unseal.

  1. Encode the OpenBao credentials

    export VAULT_UNSEAL_KEY="OLD_UNSEAL_KEY"
export VAULT_ROOT_TOKEN="OLD_ROOT_TOKEN"

echo -n "$VAULT_UNSEAL_KEY" | base64 -w0
echo -n "$VAULT_ROOT_TOKEN" | base64 -w0

  2. Update the OpenBao secret

    kubectl edit secret -n ${VAULT_INSTANCE_NAME} ${VAULT_INSTANCE_NAME}-seal

  3. Update the vault-root and vault-unseal-0 keys to reflect the values you have just encoded

  4. Save the secret

  5. Verify that auto-unseal works:

    1. Restart all OpenBao pods simultaneously:

      kubectl delete pod -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-{0..2}

    2. Expose the OpenBao UI

      kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200

    3. Verify that the OpenBao UI does not prompt you for the unseal key

6. Cleanup

  1. Reenable backups.enabled in the component configuration