Lieutenant in vcluster

Vcluster is a virtual Kubernetes Cluster that runs in a single namespace on a host Kubernetes cluster. Installing Lieutenant in a vcluster can be useful as it allows you to run multiple versions of Lieutenant, with different CRD versions on the same host cluster.

The following steps need to be performed to deploy Lieutenant in a vcluster.

Deploy vcluster and enable Syn

First we need a vcluster instance we can deploy Lieutenant to. Follow the setup guide in the component-vcluster documentation and be sure to enable Syn on the newly created virtual cluster.

After that you should be able to deploy to a vcluster and the vcluster component configuration should look similar to the following:

applications:
  - vcluster as lieutenant-prod
parameters:
  lieutenant_prod:
    host_service_cidr: 172.30.0.0/16
    namespace: lieutenant-prod
    ingress:
      host: lieutenant-prod.cluster.example.com
    syn:
      registration_url: https://api.syn.example.com/install/steward.json?token=XRaRSHafWa28afE72F2aCY==

Attach external Vault

Classical deployments of Lieutenant assume that Vault is running on the same cluster. When deploying Lieutenant in a vcluster this isn’t ideal. We will want to configure a connection to an externally running Vault.

For that we need to do two things

Setup Vault to authenticate users against an external Kubernetes cluster. To do that follow the Vault guide on connecting to an external Lieutenant instance.

Configure connection to Vault and enable authentication delegation for service accounts

parameters:
  lieutenant:
    api:
      ingress:
        host: api-prod.syn.example.com
    operator:
      vault:
        addr: https://vault.example.com
        auth_path: lieutenant (1)
        path: clusters/kv
    auth_delegation:
      lieutenant-sa: (2)
        apiGroup: rbac.authorization.k8s.io
        kind: Group
        name: system:serviceaccounts
        namespace: lieutenant

1	The mount path for the configured external Kubernetes Vault authentication method
2	This is needed so that Lieutenant service accounts, such as the ones for the managed clusters, can access Vault

Setup connection to GitLab

The Lieutenant Operator needs to be able to connect to the GitLab API to create repositories for tenants and clusters. Follow the Connecting to GitLab guide. Afterwards the component configuration should look similar to the following:

parameters:
  lieutenant:
    api:
      ingress:
        host: api-prod.syn.example.com
    operator:
      vault:
        addr: https://vault.example.com
        auth_path: lieutenant
        path: clusters/kv
    auth_delegation:
      lieutenant-sa:
        apiGroup: rbac.authorization.k8s.io
        kind: Group
        name: system:serviceaccounts
        namespace: lieutenant
    githosts:
      your-gitlab:
        endpoint: https://git.yourdomain.net/
        token: '?{vaultkv:${cluster:tenant}/${cluster:name}/lieutenant/githosts/your-gitlab/token}'
        host_keys: |
          git.yourdomain.net ssh-rsa AAAAB3Nz...
          git.yourdomain.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO9EkPcVdsz/oVTI2VJkBlq8Mv/dg3rhcbgzAEKyiwUG

(Optional) Enable OIDC

As a last step we can connect the created vcluster to an OIDC provider. This not only allows us to authenticate to the vcluster Kubernetes API using OIDC, but also enables doing so for the Lieutenant API. After enabling this, we can authenticate ourselves to Lieutenant using commodore login (requires Commodore >= 0.15.0).

See the guide on how to configure OIDC for vcluster. After that the vcluster component configuration should look similarly to the following:

applications:
  - vcluster as lieutenant-prod
parameters:
  lieutenant_prod:
    host_service_cidr: 172.30.0.0/16
    namespace: lieutenant-prod
    ingress:
      host: lieutenant-prod.cluster.example.com
    k3s:
      additional_args:
        - --kube-apiserver-arg=oidc-issuer-url=https://id.example.com/auth/realms/main
        - --kube-apiserver-arg=oidc-client-id=lieutenant
        - --kube-apiserver-arg=oidc-username-claim=preferred_username
        - --kube-apiserver-arg=oidc-username-prefix=syn#
        - --kube-apiserver-arg=oidc-groups-claim=groups
    syn:
      registration_url: https://api.syn.example.com/install/steward.json?token=XRaRSHafWa28afE72F2aCY==

Commodore configuration auto discovery

Commodore has the feature to discover the OIDC configuration from the Lieutenant API. That way Commodore user don’t need to know the URL of the IDP or the OIDC client ID.

To use this feature you need to pass this information to the Lieutenant API

parameters:
  lieutenant:
    api:
      ingress:
        host: api-prod.syn.example.com
        env:
          OIDC_CLIENT_ID: lieutenant (1)
          OIDC_DISCOVERY_URL=https://id.example.com/auth/realms/main/.well-known/openid-configuration (2)

1	The same client ID used for vcluster
2	The OIDC discovery endpoint of the IDP