Cloud Agent Policies
This documentation explains how the different admission webhooks and controllers, referenced here as policies, interact to implement the requirements and architectural decisions for APPUiO Cloud.
Namespace ownership
The overall desired architecture for namespace ownership on APPUiO Cloud is documented in the APPUiO Cloud for System Engineers documentation. This component implements the architecture described there with the APPUiO Cloud Agent.
This article explains how the different policies managed by the component implement namespace ownership and other adjacent policies.
Looking at the quality requirements for APPUiO cloud, there are three main points which need to be covered by the implementation:
-
Users should be able to create arbitrary namespaces directly with
kubectl create. -
Users should be able to create arbitrary namespaces through OpenShift’s Project mechanism, either on the web console or with
oc new-project. -
Any namespace created by a regular user must be associated with an organization for billing purposes.
Additionally, the following APPUiO Cloud requirements also need to be considered in the implementation:
-
A single organization can only create a limited number of namespaces on an APPUiO Cloud zone.
-
Users should be able to use ServiceAccount tokens to create namespaces belonging to the same organization to which the ServiceAccount belongs.
-
Users should be able to modify selected annotations and labels on namespaces belonging to their organizations. For example, this is required to allow users to reassign namespaces to a different organization.
-
By default, the namespace’s organization is granted role
adminin the namespace.
Request flow and policy execution
The diagram shows how the policies interact with a request to create a Namespace, Project or ProjectRequest.
|
We only show interactions which are relevant to See the detailed policy explanations or the policy reference documentation for details. |
/---------------\
| cFD0 |
| CREATE |
| Namespace, |
| ProjectRequest|
| |
\-------+-------/
|
|
PRIV v
+---------------------+
| +---------------+ |
| |cBLUVALIDATE | |
| +---------------+ | yes
| Principal is allowed+------------------------------------+
| to bypass policies | |
| {d} | |
+----------+----------+ |
| |
|no |
| |
NMEV v |
+--------------------+ |
| +---------------+ | |
| |cBLUVALIDATE | | |
| +---------------+ | |
yes | Check if requested| |
+---------------------------+ namespace/project | |
| | name matches a | |
| | reserved prefix | |
| | {d} | |
| +----------+---------+ |
| | |
| |no |
| | |
| +-----------------------+-----------------+ |
| | | |
| |CREATE |CREATE |
| |ProjectRequest |Namespace |
| | | |
| | +--------------+--------------+ |
| ORG_NS v | | |
| +-----------------+ |Principal Principal| |
| |+---------------+| |is User is SA| |
| ||cBLUVALIDATE || v v |
| |+---------------+| +----------------------+ +----------------------+ |
| | Check that user| no :ORG_NS | :ORG_NS | |
| | has default +----+ | +-----------------+ | | +----------------+ | |
| | organization | | | | +-------------+ | | | | +-------------+| | |
| | {d} | | | | |cPNKMUTATE | | | | | |cPNKMUTATE || | |
| +-----------------+ | | | +-------------+ | | | | +-------------+| | |
| | | | | Inject user's| | | | Inject SA's | | |
| |yes | | | default | | | | organization| | |
| | | | | organization | | | | if no label | | |
| v | | | if no label | | | | in request | | |
| +--------+ | | | in request | | | | {d} | | |
| | c2A2 | | | | {d} | | | +----------------+ | |
| | ALLOW | | | +-----------------+ | | | | |
| | request| | | | | | v | |
| | | | | v | | +------------------+ | |
| +---+----+ | | +------------------+ | | |+---------------+ | | |
| | | | |+---------------+ | | | ||cBLUVALIDATE | | | |
| | | | ||cBLUVALIDATE | | | | |+---------------+ | | |
| v | | |+---------------+ | | | | Check that NS | | |
| /------------------\ | | | Check that NS | | | | belongs to | | |
| | cFD0 | | | | belongs to | | | | organization | | |
| | CREATE Project[1]| | | | organization | | | | and that it | | |
| | | | | | and user is | | | | matches SA's | | |
| \--------+---------/ | | | member of it | | | | organization | | |
| | | | | {d} | | | | {d} | | |
| | | | +------------------+ | | +------------------+ | |
| PRJ_ORG v | | | | | |
| +---------------+ | +-------+------+-------+ +-------+------+-------+ |
| |+-------------+| | | | | | |
| ||cPNKMUTATE || | | | | | |
| |+-------------+| | | | | | |
| | Inject | | | | | | |
| | organization | | | | | | |
| | from project | | no | | | | |
| | annotation | +----------|------+---------------------|------+ |
| | {d} | | | | |
| +-------+-------+ | | | |
| | | | yes | |
| | | +------------+---------------+ |
| v | | |
| /--------------------\ | | |
| | cFD0 | | | |
| | triggers | | | |
| | CREATE Namespace[2]| | | |
| | | | NS_QUOT v |
| \---------+----------/ | +-----------------+ |
| | | |+---------------+| |
| | | ||cBLUVALIDATE || |
| | | |+---------------+| |
| | | no | Check if | |
| | +--------------+ organization | |
| | | | can create | |
| | | | more namespaces| |
| | | | {d} | |
| | | +--------+--------+ |
| | | | |
| | | |yes |
| | | | |
| | | NMEV v |
| | | +-----------------+ |
| | | |+---------------+| |
| | | ||cBLUVALIDATE || |
| | | |+---------------+| |
| | | no | Check if | |
| | +--------------+ the request only| |
| | | | contains allowed| |
| | | | labels and | |
| | | | annotations | |
| | | |{d} | |
| | | +--------+--------+ |
| | | | |
| | | |yes |
| | | | |
| | v v |
| | +--------+ +--------+ |
| | | cRED | | c2A2 | |
+------------|--------->| DENY | | ALLOW |<----------------------------+
| | request| | request|
| | | | |
| +--------+ +---+----+
| |
| |
| v
| /----------------\
| | cFD0 |
| | CREATE |
+----------------------------->| Namespace with |
| organization |
| label |
| |
\-------+--------/
|
|
+--------------+--------------+
| |
ORG_RBAC v RES_QUOT v
+-----------------+ +---------------------+
|+---------------+| | +---------------+ |
||cYELRECONCILE || | |cYELRECONCILE | |
|+---------------+| | +---------------+ |
| Grant the | | Create |
| organization | | ResourceQuota and |
| Role "admin" | | LimitRange objects|
| in the NS | | in the namespace |
|{d} | |{d} |
+-----------------+ +---------------------+
| 1 | Project resources can’t be created directly by users. |
| 2 | Namespace resources which are created from a ProjectRequest always originate from a privileged system principal which can bypass the namespace restrictions. |
|
Checking whether a principal is allowed to bypass the namespace policies is implemented as exclude rules in the other policies.
However, to better illustrate the flow of a request in the diagram, we pretend it’s a separate policy (labeled |
|
Mutating policies are executed before validating policies, both are executed in parallel with other policies of the same type. We organize them in a flow-chart style to better illustrate how they interact. |
Policies which are executed for all requests by unprivileged principals
The following policies are executed for all namespaces which are created by an unprivileged principal:
namespace_metadata_validator(labeledNMEV)-
This policy ensures that users can’t create namespaces which match a pattern which is reserved for the system. We need this policy to ensure that users can’t adversely impact the system by using namespace names which might be used by the system in the future.
Effectively, this is a restriction of the requirement that users can choose arbitrary namespace names to ensure overall system availability. The component allows operators to configure the set of disallowed patterns.
This policy also ensures that users can only create or edit selected labels and annotations. In particular, we want to allow users to transfer namespace ownership between organizations of which they’re members.
+
As shown in the end-user documentation, transferring a namespace is done by changing the namespace’s appuio.io/organization label to the organization which should receive ownership of the namespace.
+ To protect the cluster, we need to ensure that users can’t modify arbitrary labels or annotations on a namespaces, since OpenShift exposes a number of privileged operations (such as setting a namespace-wide node selector for workloads) as labels and annotations on namespace objects.
+ TIP: This policy is executed when namespaces are created or updated.
namespace_quota_validator.go(labeledNS_QUOT)-
This policy denies creation of new namespaces for an organization which has used up their namespace quota on a zone. The component allows operators to adjust the global and per-organization namespace quota.
This policy implements the requirement that a single organization can only create a limited number of namespaces on an APPUiO Cloud zone.
This policy has two modes: Legacy mode: The policy reads the namespace quota from a ConfigMap in the same namespace as the policy. UsageProfile mode: See Control API: UsageProfile. Overrides of the default quotas are still possible using the same ConfigMap as the legacy policy.
org_rbac_controller.go(labeledORG_RBAC)-
The Organization RBAC Controller of the APPUiO Cloud Agent makes sure that by default organizations are granted all the permissions necessary to manager their own namespaces.
It does so by creating
RoleBindingsin every organization namespace that grant configuredClusterRoles. Most importantly, it grants roleadminto the organization to which the new namespace belongs.The controller will only make sure that the
RoleBindingsexist and won’t modify existingRoleBindings. This allows organizations to further restrict access for their members.This fulfills the requirement that namespaces are owned by organizations.
legacy_resource_quota_controller.go(labeledRES_QUOT)-
This policy is applied to any namespace which is created with an organization label. It generates default
ResourceQuotaandLimitRangeobjects in all namespaces belonging to an organization. The policy allows cluster operators to adjust the generated objects by adding appropriate annotations to namespaces.This policy implements the requirement that the APPUiO Cloud zone is protected from abusive resource usage for resource types which can be managed through Kubernetes
ResourceQuotaandLimitRangeobjects. Notably, we deploy a quota limiting the cumulative memory and CPU resource requests and limits of all containers per namespace and a quota limiting the count of other Resources, such asServiceandSecretobjects, per namespace.This policy is superseeded by the UsageProfile feature. See Control API: UsageProfile.
Overrides of the default quotas are still possible by adding the same annotations as before to the namespace.
A validating policy (
reserved_resourcequota_limitrange_validator.go) forbids modification of the reservedResourceQuotaandLimitRangeobjects in the namespace.
Policies which are executed for requests to create a Namespace by specific unprivileged principals
We handle actual validation of namespace creation by users and by ServiceAccounts in different policies:
namespace_project_organization_mutator.go(labeledORG_NS)-
This policy is executed when a namespace or OpenShift project is created by a user or a ServiceAccount.
User flow
If the user creates a namespace without an explicit appuio.io/organization label, their default organization is set as the value of the label.
+ The policy denies the request if the user tries to create a namespace for an organization which they’re not a member of. It also denies the request when a user who doesn’t have a default organization tries to create a namespace without an explicit organization label.
+
This policy implements the requirement that users can create arbitrary namespaces directly with kubectl create
Additionally, this policy ensures that user namespaces are associated with an organization for billing purposes for those namespaces.
ServiceAccount flow
The policy looks up the ServiceAccount’s organization by looking up the organization to which the ServiceAccount’s namespace belongs.
If the ServiceAccount creates a namespace without an explicit appuio.io/organization label, it’s organization is set as the value of the label.
+ The policy denies the request if the ServiceAccount tries to create a namespace for a different organization than the one to which it belongs.
+ This policy implements the requirement that users should be able to use ServiceAccount tokens to create namespaces. Additionally, this policy ensures that user namespaces are associated with an organization for billing purposes for namespaces created by ServiceAccounts.
OpenShift Project flow
This policy is executed when the control plane creates a Project resource based on a ProjectRequest created by a user.
The policy reads the annotation openshift.io/requester on the Project and uses the value of that annotation to lookup the user which requested the project.
The user’s default organization is then injected as label appuio.io/organization on the Project resource[1].
+ This policy ensures the requirement that user namespaces are associated with an organization for billing purposes for namespaces created through an OpenShift project.
Policies which don’t interact with organization namespaces
The component also manages policies which implement other features than namespace ownership on APPUiO Cloud zones.
pod_runonce_active_deadline_seconds_mutator.go-
This policy injects a default value for
.spec.activeDeadlineSecondsfor run-once pods[2] which don’t have an explicit value for that field.This policy implements requirement that the APPUiO Cloud zone is protected from abusive resource usage to protect the cluster from run-once pods with unbounded runtime.
The policy which validates mutations of annotations on namespaces allows users to override the default value which is injected for individual namespaces by annotating the namespace. pod_node_selector_mutator.go-
This policy injects a default value for
.spec.nodeSelectorfor pods which don’t have an explicit value for that field.This allows users to choose a default node class value for their workloads if they don’t specify a node selector in their pod or pod controller manifests.
The policy which validates mutations of annotations on namespaces allows users to override the default value which is injected for individual namespaces by annotating the namespace. ratio_validator.go-
This policy validates that the ratio of CPU and memory requests to limits in a pod is within a configured range.
On violation of the ratio, the policy warns the user that they might be billed more than expected for their workloads. It does not however deny the request.
Fair-usage ratios differ between cloud providers and node node classes.
References
-
The APPUiO Cloud requirements implemented by the policies can be found in the VSHN Knowledge Base, in the section APPUiO Cloud for Engineers.
Project is created, the control plane automatically creates a Namespace in the background. Labels added on a Project by the policy are applied to the Namespace as well, ensuring that any projects created by a user belong to that user’s default organization.