Restore from Backup

When configuring component-vault with backup.enabled = true, the component sets up regular backups using k8up. This how-to explains how Vault can be restored from such a backup.

Information on the Vault Unseal Key and Root Token

component-vault leverages k8up’s application specific backups to create Vault snapshots. Restoring such a snapshot is only possible by providing the Vault unseal key and root token. As these are sensitive secrets, they are not backed up as part of the component’s automated backup process.

When setting up Vault, you must store the Vault unseal key and root token in a safe location in order to be able to restore backups.

In a running Vault instance, both of these secrets can be found in the [instance name]-seal secret in the Vault instance’s namespace. In order to have a reliable Vault backup, this secret must be backed up separately to a safe location.

Prerequisites

  • restic - command line tool to access backups made by k8up

  • vault - command line tool to interact with Vault

  • kubectl

  • Write access to the cluster’s tenant repository

  • Read access to the restic repository in which k8up stored the Vault backups

  • The Vault instance’s unseal key and root token - these must be backed up manually; they are not part of the automated k8up backup.

Procedure

1. Set up new Vault instance

  1. Add the vault application to your cluster configuration.

    1. If your old instance of vault is still running on the cluster, you can use component instantiation to create a second instance by adding vault as new-vault to your application list, and configuring it under new_vault.

  2. Initially disable backups by setting .backups.enabled to false

  3. Compile and push the cluster config and wait for Vault to start.

2. Retrieve the Vault snapshot

  1. Set up the restic credentials (values correspond to the component parameters backup.bucket and backup.password)

    export AWS_ACCESS_KEY_ID="S3_KEY" # from component configuration: backup.bucket.accesskey
export AWS_SECRET_ACCESS_KEY="S3_SECRET" # from component confiugration: backup.bucket.secretkey
export RESTIC_REPOSITORY="s3:https://path.to.my/bucket"
export RESTIC_PASSWORD="RESTIC_REPO_KEY" # from component configuration: backup.password

  2. Retrieve the latest Vault snapshot to your local disk

    mkdir restore
restic restore --target restore/ latest

  3. Verify the snapshot file

    ls restore
# This should show a file named "[instance name]-backup.snapshot"

3. Restore the snapshot

  1. Expose the Vault pod

    kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200

  2. In a separate terminal, prepare the environment to access Vault

    # Get root token to log in
export VAULT_TOKEN="$(kubectl get secret -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-seal -ojsonpath='{.data.vault-root}' | base64 -d)"
export VAULT_ADDR="http://127.0.0.1:8200"

  3. Restore the backup

    vault operator raft snapshot restore -force restore/${VAULT_INSTANCE_NAME}-backup.snapshot

4. Unseal Vault

If you were logged into the Vault UI, you should have gotten logged out now. This is expected.

  1. Open your browser at localhost:8200

  2. Use the Vault Unseal Key of the Vault instance you’ve just restored to unseal Vault

  3. Use the Vault root token of the Vault instance you’ve just restored to log in with the Token method

  4. Verify that the restore worked, and secrets are now restored in Vault.

The unseal key and root token of the Vault instance you’re restoring need to have been stored separately. Without them, the restore procedure cannot be completed.

5. Update the Vault Secret

Without this step, your Vault instance will not be able to auto-unseal.

  1. Encode the Vault credentials

    export VAULT_UNSEAL_KEY="OLD_UNSEAL_KEY"
export VAULT_ROOT_TOKEN="OLD_ROOT_TOKEN"

echo -n "$VAULT_UNSEAL_KEY" | base64 -w0
echo -n "$VAULT_ROOT_TOKEN" | base64 -w0

  2. Update the Vault secret

    kubectl edit secret -n ${VAULT_INSTANCE_NAME} ${VAULT_INSTANCE_NAME}-seal

  3. Update the vault-root and vault-unseal-0 keys to reflect the values you have just encoded

  4. Save the secret

  5. Verify that auto-unseal works:

    1. Restart all vault pods simultaneously:

      kubectl delete pod -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-{0..2}

    2. Expose the Vault UI

      kubectl port-forward -n $VAULT_INSTANCE_NAME ${VAULT_INSTANCE_NAME}-0 8200

    3. Verify that the Vault UI does not prompt you for the unseal key

6. Cleanup

  1. Reenable backups.enabled in the component configuration