Red Hat Best Practices for Kubernetes

A set of three base images (Minimal, Standard, and Multi-service) are provided to provide optimum starting points for a variety of use cases.

See test case platform-alteration-isredhat-release

Red Hat created the Node Tuning Operator for low latency nodes.

The emergence of Edge computing in the area of Telco plays a key role in reducing latency, congestion, and improving application performance. Many of the deployed applications in the Telco space require low latency and zero packet loss. OpenShift Container Platform provides a Node Tuning Operator to implement automatic tuning to achieve low latency performance for applications. The Node Tuning Operator is a meta-operator that leverages MachineConfig, Tuned and KubeletConfig resources, Topology Manager, and CPU Manager, to optimize the nodes.

In OpenShift Container Platform, nodes/hosts must pre-allocate huge pages.

For more information, see Configuring huge pages.

To request hugepages, pods must supply the following within the pod.spec for each container:

See test cases platform-alteration-hugepages-2m-only, platform-alteration-hugepages-config

The Node Tuning Operator manages host CPUs by dividing them into reserved CPUs for cluster and operating system housekeeping duties, and isolated CPUs for workloads. CPUs that are used for low latency workloads are set as isolated.

Device interrupts are load balanced between all isolated and reserved CPUs to avoid CPUs being overloaded, with the exception of CPUs where there is a guaranteed pod running. Guaranteed pod CPUs are prevented from processing device interrupts when the relevant annotations are set for the pod.

Topology Manager collects hints from the CPU Manager, Device Manager, and other Hint Providers to align pod resources, such as CPU, SR-IOV VFs, and other device resources, for all Quality of Service (QoS) classes on the same non-uniform memory access (NUMA) node. This topology information and the configured Topology manager policy determine whether a workload is accepted or rejected on a node.

The following Topology manager policies are available and dependent on the requirements of the workload can be enabled. For high performance workloads making use of SR-IOV VFs, NUMA awareness follows the NUMA node to which the SR-IOV capable network adapter is connected.

Using CPU Manager and Topology Manager.

Applications should discover services via DNS by doing an AAAA and A query. If an application gets a AAAA response the application should prefer using the IPv6 address in the AAAA response for application sockets.

In OpenShift Container Platform 4.7+, you can declare ipFamilyPolicy: PreferDualStack which will present an IPv4 and IPv6 address in the service.

For more information, see IPv4/IPv6 dual-stack.

To configure IPv4/IPv6 dual-stack, set dual-stack cluster network assignments:

Services advertised to other CNFs should utilize VIPs on the SPK that are part of the platform. This creation of external services is done via pushing a configmap with F5 AS3 formatted configuration tocreation of CRDs for SPK into the Kubernetes API.

Creation of IPv6 external services and mapping them to IPv4 internal services within the clusters is possible and happens automatically. The SPK will terminate the IPv6 traffic destined to the pods in the cluster and translate the traffic to an IPv4 Pod destination IP address and source NAT the traffic to an IPv4 address on the load balancer.

In order to reach an IPv6 external service from an IPv4 pod, the service must be configured in Verizon’s external DNS systems to the cluster. Once this is done reachability is achieved by the pod generating a DNS A and/or AAAA query for the IPv6 external service. This query is forwarded by the cluster through the SPK which consumes the request and re-originates a request on behalf of the cluster. When a DNS response comes back with only a AAAA record response, the F5 will reserve an IPv4 address local to it that is reachable from the cluster and return to the pod via the cluster DNS both an A and the external AAAA record. This IPv4 address that is reserved is a 1 to 1 mapping to the IPv6 destination. Because the pods only have an IPv4 address and socket they attempt to reach the IPv4 address that is resident on the SPK, when the load balancer receives traffic destined to that IPv4 address it translates the traffic to the IPv6 destination address that was in the AAAA response and source NATs the pod’s IPv4 address to an IPv6 source NAT address that will allow the traffic to reach the IPv6 destination.

Virtual routing and forwarding (VRF) provides a way to have separate routing tables on the device enabling multiple L3 routing domains concurrently. This allows for traffic in different VRF to be treated independently of each other.

Generally a Load Balancer is used within the platform for L4 services and sometimes L7 load balancing services. In a multi-tenant environment, Network Functions (NFs) can be deployed within a single namespace. Supporting applications like an OAM platform for multiple NFs from the same vendor should be run in an additional separate namespace. The CNI interface should be used as the default mechanism for accessing VRFs. For traffic inbound to an application this is done through allocation of a VIP on the load balancer via the Kubernetes API on the appropriate VRF. For traffic outbound from an application selection of the VRF is done on the application’s behalf via the Load Balancer and destination routing. Multus will be supported within the platform for additional NICs within containers. However Multus should be used only for those cases that cannot be supported by the load balancer.

The POD and Services networks are unrouted address space, they are only reachable via service VIPs on the load balancers. The POD network will be NATed as traffic egresses the load balancer. Traffic inbound will be destination NATed to Service/Pod IP addresses.

Applications should use Network Policies for firewalling the application. Network Policies should be written with a default deny and only allow ports and protocols on an as needed basis for any pods and services.

See test case networking-network-policy-deny-all

The following ports are reserved by OpenShift and should not be used by any application. These ports are blocked by iptables on the nodes and traffic will not pass. Port list:

A workload which handles user plane traffic or latency-sensitive payloads at line rate falls into this category, such as load balancing, routing, deep packet inspection, and so on. Some of these workloads may also need to process the packets at a lower level.

This kind of workload may need to:

As CAP_IPC_LOCK is mandatory for allocating hugepage memory, this capability is granted to DPDK-based applications. Additionally, if the workload is latency-sensitive and needs the determinacy provided by the real-time kernel, the CAP_SYS_NICE is also required.

Here is an example pod manifest of a DPDK application:

It is not recommended for an application to access following resources on the host.

It is not necessary to mount host /sys/ or host /dev/ directories as a volume in a pod in order to use a network device such as SR-IOV VF. The moving of a network interface into the pod network namespace is done automatically by CNI. Mounting the whole /sys/ or /dev/ directory in the container will overwrite the network device descriptor inside the container which causes device not found or no such file or directory error.

Network interface statistics can be queried inside the container using the same /sys/ path as was done when running directly on the host. When running network interfaces in containers, relevant /sys/ statistics interfaces are available inside the container, such as /sys/class/net/net1/statistics/, /proc/net/tcp and /proc/net/tcp6.

For running DPDK applications with SR-IOV VF, device specs (in case of vfio-pci) are automatically attached to the container via the Device Plugin. There is no need to mount the /dev/ directory as a volume in the container as the application can find device specs under /dev/vfio/ in the container.

Application pods must avoid using hostNetwork. Applications may not use the host network, including nodePort for network communication. Any networking needs beyond the functions provided by the pod network and ingress/egress proxy must be serviced via a MULTUS connected interface.

The default capabilities that are allowed via the restricted SCC are as follows (see default cri-o Linux capabilities)

IPC_LOCK capability is required if any of these functions are used in an application:

Even though mlock() is not necessary on systems where page swap is disabled (for example on OpenShift), it may still be required as it is a function that is built into DPDK libraries, and DPDK based applications may indirectly call it by calling other functions.

See test case access-control-ipc-lock-capability-check

NET_ADMIN capability is required to perform various network related administrative operations inside container such as:

This doesn’t include:

All the administrative operations (except ethtool) mentioned above that require the NET_ADMIN capability should already be supported on the host by various CNIs in Openshift.

This capability is very powerful and overloaded. It allows the application to perform a range of system administration operations to the host. So you should avoid requiring this capability in your application.

In the case that a workload is running on a node and is using DPDK, SYS_NICE will be used to allow DPDK application to switch to SCHED_FIFO.

See test case access-control-sys-nice-realtime-capability

This capability is required when using Process Namespace Sharing. This is used when processes from one Container need to be exposed to another Container. For example, to send signals like SIGHUP from a process in a Container to another process in another Container. See Share Process Namespace between Containers in a Pod for more details. For more information on these capabilities refer to Linux Capabilities in OpenShift.

See test case access-control-sys-ptrace-capability

The SR-IOV Network Operator also supports configuring the following parameters for SR-IOV VFs. Refer to Configuring an SR-IOV Ethernet network attachment for more information.

In OpenShift, multicast is supported for both the default interface (OVN or OpenShift-SDN) and the additional interfaces such as macvlan, SR-IOV, etc. Multicast is disabled by default. To enable it, refer to the following procedures:

NOTE

OpenShift SDN CNI is deprecated as of OpenShift Container Platform 4.14. As of OpenShift Container Platform 4.15, the network plugin is not an option for new installations. In a subsequent future release, the OpenShift SDN network plugin is planned to be removed and no longer supported. Red Hat will provide bug fixes and support for this feature until it is removed, but this feature will no longer receive enhancements. As an alternative to OpenShift SDN CNI, you can use OVN Kubernetes CNI instead. For more information, see OpenShift SDN CNI removal in OCP 4.17.

Secrets objects in OpenShift provide a way to hold sensitive information such as passwords, config files and credentials. There are 4 types of secrets; service account, basic auth, ssh auth and TLS. Secrets can be added via deployment configurations or consumed by pods directly. For more information on secrets and examples, see the following documentation.

Providing sensitive data to pods

Permissions to use an SCC is done by adding a cluster role that has uses permissions for the SCC and then rolebindings for the users within a namespace to that role for users that need that SCC. Application admins can create their own role/rolebindings to assign permissions to a Service Account.

The following best practices highlight some key principles of cloud-native application design.

See test cases lifecycle-container-shutdown, lifecycle-container-startup

It is recommended that container images be built utilizing Red Hat’s Universal Base Image as they will have a solid security baseline as well as support from Red Hat.

Vendors must satisfy 3 requirements related to maintaining proper workload isolation in a containerized environment:

The Red Hat UBI is able to meet these requirements and enables images built with it to meet these requirements. UBI is supported by a dedicated, full-time team providing releases of base image. UBI has the following features:

UBI is designed to be a foundation for cloud-native and web applications use cases developed in containers. You can build a containerized application using UBI, push it to your choice of registry server, easily share it with others - and because it’s freely redistributable — even deploy it on non-Red Hat platforms. And since it’s built on Red Hat Enterprise Linux, UBI is a platform that is reliable, secure, and performant.

Workloads should use the service name only as a configuration parameter for attaching to a service within your namespace, the cluster will append namespace name and kubernetes service nomenclature on behalf of the application via search string in DNS. This allows a generic name for a service that works in all clusters no matter what the namespace name is and what the cluster base FQDN is.

For more information, see Kubernetes upstream reference for pod/service names and DNS.

It is recommended that container images be built utilizing Red Hat’s Universal Base Image as they will have a solid security baseline as well as support from Red Hat.

Must satisfy 3 requirements related to maintaining proper workload isolation in a containerized environment:

The Red Hat UBI is able to meet these requirements and enables images built with it to meet these requirements. UBI is supported by a dedicated, full-time team providing releases of base image. UBI has the following features:

UBI is designed to be a foundation for cloud-native and web applications use cases developed in containers. You can build a containerized application using UBI, push it to your choice of registry server, easily share it with others - and because it’s freely redistributable — even deploy it on non-Red Hat platforms. And since it’s built on Red Hat Enterprise Linux, UBI is a platform that is reliable, secure, and performant.

Workloads should use the service name only as a configuration parameter for attaching to a service within your namespace, the cluster will append namespace name and kubernetes service nomenclature on behalf of the application via search string in DNS. This allows a generic name for a service that works in all clusters no matter what the namespace name is and what the cluster base FQDN is.

If an application deploys a service in the namespace clspcoykvzwcscp-y-xx-w1-001 and is attempting to access a service named worker, the application should just configure the client of the service with an FQDN of worker.

The DNS search suffix will append clspcoykvzwcscp-y-xx-w1-001.svc.cluster.local to the end of the name and result in a successful query for worker.clspcoykvzwcscp-y-xx-w1-001.svc.cluster.local. This allows an application to be less aware of the application’s namespace name and genericize the configuration of the application.

For more information, see Kubernetes upstream reference for pod/service names and DNS.