Policies

This documentation explains how the different policies interact to implement the requirements and architectural decisions for APPUiO Cloud.

Namespace ownership

The overall desired architecture for namespace ownership on APPUiO Cloud is documented in the APPUiO Cloud for System Engineers documentation. This component implements the architecture described there with Kyverno policies and the APPUiO Cloud Agent.

This article explains how the different policies managed by the component implement namespace ownership and other adjacent policies.

Looking at the quality requirements for APPUiO cloud, there are three main points which need to be covered by the implementation:

Users should be able to create arbitrary namespaces directly with kubectl create.
Users should be able to create arbitrary namespaces through OpenShift’s Project mechanism, either on the web console or with oc new-project.
Any namespace created by a regular user must be associated with an organization for billing purposes.

Additionally, the following APPUiO Cloud requirements also need to be considered in the implementation:

A single organization can only create a limited number of namespaces on an APPUiO Cloud zone.
Users should be able to use ServiceAccount tokens to create namespaces belonging to the same organization to which the ServiceAccount belongs.
Users should be able to modify selected annotations and labels on namespaces belonging to their organizations. For example, this is required to allow users to reassign namespaces to a different organization.
By default, the namespace’s organization is granted role admin in the namespace.

Request flow and policy execution

The diagram shows how the policies interact with a request to create a Namespace, Project or ProjectRequest.

We only show interactions which are relevant to CREATE requests. Some policies, such as validate-namespace-metadata are also executed on UPDATE requests.

See the detailed policy explanations or the policy reference documentation for details.

1	`Project` resources can’t be created directly by users.
2	`Namespace` resources which are created from a `ProjectRequest` always originate from a privileged system principal which can bypass the namespace restrictions.

Checking whether a principal is allowed to bypass the namespace policies is implemented as exclude rules in the other validating policies. However, to better illustrate the flow of a request in the diagram, we pretend it’s a separate validating policy (labeled PRIV).

Generally, Kyverno policies are evaluated in parallel, but we organize them in a flow-chart style to better illustrate how they interact.

Policies which are executed for all requests by unprivileged principals

The following policies are executed for all namespaces which are created by an unprivileged principal:

disallow-reserved-namespaces (labeled RSVD)

This policy ensures that users can’t create namespaces which match a pattern which is reserved for the system. We need this policy to ensure that users can’t adversely impact the system by using namespace names which might be used by the system in the future.

Effectively, this is a restriction of the requirement that users can choose arbitrary namespace names to ensure overall system availability. The component allows operators to configure the set of disallowed patterns.

check-namespace-quota (labeled NS_QUOT)

This policy denies creation of new namespaces for an organization which has used up their namespace quota on a zone. The component allows operators to adjust the global and per-organization namespace quota.

This policy implements the requirement that a single organization can only create a limited number of namespaces on an APPUiO Cloud zone.

This policy was migrated to the APPUiO Cloud Agent with introduction of the UsageProfile feature. See Control API: UsageProfile.

Overrides of the default quotas are still possible using the same ConfigMap as the Kyverno policy.

validate-namespace-metadata (labeled NS_LBL)

This policy ensures that users can only create or edit selected labels and annotations. In particular, we want to allow users to transfer namespace ownership between organizations of which they’re members.

As shown in the end-user documentation, transferring a namespace is done by changing the namespace’s appuio.io/organization label to the organization which should receive ownership of the namespace.

To protect the cluster, we need to ensure that users can’t modify arbitrary labels or annotations on a namespaces, since OpenShift exposes a number of privileged operations (such as setting a namespace-wide node selector for workloads) as labels and annotations on namespace objects.

This policy is executed when namespaces are created or updated.

default-rolebinding-in-ns (labeled ORG_RBAC)

The Organization RBAC Controller of the APPUiO Cloud Agent makes sure that by default organizations are granted all the permissions necessary to manager their own namespaces.

It does so by creating RoleBindings in every organization namespace that grant configured ClusterRoles. Most importantly, it grants role admin to the organization to which the new namespace belongs.

The controller will only make sure that the RoleBindings exist and won’t modify existing RoleBindings. This allows organizations to further restrict access for their members.

This fulfills the requirement that namespaces are owned by organizations.

quota-and-limit-range-in-ns (labeled RES_QUOT)

This policy is applied to any namespace which is created with an organization label. It generates default ResourceQuota and LimitRange objects in all namespaces belonging to an organization. The policy allows cluster operators to adjust the generated objects by adding appropriate annotations to namespaces.

This policy implements the requirement that the APPUiO Cloud zone is protected from abusive resource usage for resource types which can be managed through Kubernetes ResourceQuota and LimitRange objects. Notably, we deploy a quota limiting the cumulative memory and CPU resource requests and limits of all containers per namespace and a quota limiting the count of other Resources, such as Service and Secret objects, per namespace.

This policy was migrated to the APPUiO Cloud Agent with introduction of the UsageProfile feature. See Control API: UsageProfile.

Overrides of the default quotas are still possible by adding the same annotations as before to the namespace.

Policies which are executed for requests to create a Namespace by specific unprivileged principals

We handle actual validation of namespace creation by users and by ServiceAccounts in different policies:

organization-namespaces (labeled USR_NS_ORG): This policy is executed when a namespace is created by a user. If the user creates a namespace without an explicit appuio.io/organization label, their default organization is set as the value of the label.

The policy denies the request if the user tries to create a namespace for an organization which they’re not a member of. It also denies the request when a user who doesn’t have a default organization tries to create a namespace without an explicit organization label.

This policy implements the requirement that users can create arbitrary namespaces directly with kubectl create Additionally, this policy ensures that user namespaces are associated with an organization for billing purposes for those namespaces.
organization-sa-namespaces (labeled SA_NS_ORG): This policy is executed when a namespace is created by a ServiceAccount. The policy looks up the ServiceAccount’s organization by looking up the organization to which the ServiceAccount’s namespace belongs. If the ServiceAccount creates a namespace without an explicit appuio.io/organization label, it’s organization is set as the value of the label.

The policy denies the request if the ServiceAccount tries to create a namespace for a different organization than the one to which it belongs.

This policy implements the requirement that users should be able to use ServiceAccount tokens to create namespaces. Additionally, this policy ensures that user namespaces are associated with an organization for billing purposes for namespaces created by ServiceAccounts.

Policies which are executed for requests to create an OpenShift project

organization-in-projectrequests (labeled PRJREQ_ORG): This policy is executed when a user creates a ProjectRequest either with oc new-project or through the OpenShift web console. It checks whether the user has a default organization and denies the request if they don’t. This policy is necessary because the request to create a ProjectRequest is the only request in the project creation flow where the principal is the user which wants to create the project.

We need to rely on the user’s default organization for namespaces created through OpenShift projects, because there’s no straightforward way for us to allow users to specify additional metadata in an OpenShift ProjectRequest.

Note that this policy doesn’t itself inject the user’s default organization into the request. Instead, the organization-projects policy ensures that the resulting namespace is created with label appuio.io/organization set to the user’s default organization for any project requests which weren’t denied.

This policy enables the requirement that users can create arbitrary namespaces through OpenShift’s Project mechanism without violating the requirement that user namespaces are associated with an organization for billing purposes.

Injecting the organization label also ensures the policies which generate the default RoleBinding and quota objects are triggered for namespaces created through OpenShift projects.
organization-projects (labeled PRJ_ORG): This policy is executed when the control plane creates a Project resource based on a ProjectRequest created by a user. The policy reads the annotation openshift.io/requester on the Project and uses the value of that annotation to lookup the user which requested the project. The user’s default organization is then injected as label appuio.io/organization on the Project resource^[1].

This policy ensures the requirement that user namespaces are associated with an organization for billing purposes for namespaces created through an OpenShift project.

Policies which don’t interact with organization namespaces

The component also manages Kyverno policies which implement other features than namespace ownership on APPUiO Cloud zones. As shown below, there’s currently only one policy which doesn’t cover some aspect of namespace ownership.

set-runonce-activedeadlineseconds

This policy injects a default value for .spec.activeDeadlineSeconds for run-once pods^[2] which don’t have an explicit value for that field.

This policy implements requirement that the APPUiO Cloud zone is protected from abusive resource usage to protect the cluster from run-once pods with unbounded runtime.

The policy which validates mutations of annotations on namespaces allows users to override the default value which is injected for individual namespaces by annotating the namespace.

References

The policy reference documentation shows what each policy does and how users can configure the policies through the component.
The APPUiO Cloud requirements implemented by the policies can be found in the VSHN Knowledge Base, in the section APPUiO Cloud for Engineers.

1. On OpenShift whenever a Project is created, the control plane automatically creates a Namespace in the background. Labels added on a Project by the policy are applied to the Namespace as well, ensuring that any projects created by a user belong to that user’s default organization.

2. Usually, those are pods created by Kubernetes jobs or cronjobs. However, the policy also affects pods created directly without a controller, for example by applying a Pod manifest to the cluster.