Kubernetes Best Practices for Far Edge Applications

The most basic requirement for the lifecycle management of pods in OpenShift is the ability to start and stop correctly. When starting up, health probes like liveness and readiness checks can be put into place to ensure the application is functioning properly.

There are different ways a pod can be stopped in Kubernetes. One way is that the pod can remain alive but non-functional. Another way is that the pod can crash and become non-functional. In the first case, if the administrator has implemented liveness and readiness checks, OpenShift can stop the pod and either restart it on the same node or a different node in the cluster. For the second case, when the application in the pod stops, it should exit with a code and write suitable log entries to help the administrator diagnose what the issue was that caused the problem.

Pods should use terminationMessagePolicy: FallbackToLogsOnError to summarize why they crashed and use stderr to report errors on crash

See test case observability-termination-policy

OpenShift Container Platform and Kubernetes give application instances time to shut down before removing them from load balancing rotations. However, applications must ensure they cleanly terminate user connections as well before they exit.

On shutdown, OpenShift Container Platform sends a TERM signal to the processes in the container. Application code, on receiving SIGTERM, stop accepting new connections. This ensures that load balancers route traffic to other active instances. The application code then waits until all open connections are closed, or gracefully terminate individual connections at the next opportunity, before exiting.

After the graceful termination period expires, a process that has not exited is sent the KILL signal, which immediately ends the process. The terminationGracePeriodSeconds attribute of a pod or pod template controls the graceful termination period (default 30 seconds) and can be customized per application as necessary.

There are different reasons that a pod may need to shutdown on an OpenShift cluster. It might be that the node the pod is running on needs to be shut down for maintenance, or the administrator is doing a rolling update of an application to a new version which requires that the old versions are shutdown properly.

When pods are shut down by the platform they are sent a SIGTERM signal which means that the process in the container should start shutting down, closing connections and stopping all activity. If the pod doesn’t shut down within the default 30 seconds then the platform may send a SIGKILL signal which will stop the pod immediately. This method isn’t as clean and the default time between the SIGTERM and SIGKILL messages can be modified based on the requirements of the application.

Pods should exit with zero exit codes when they are gracefully terminated.

CaaS Platform has a default scheduler that is responsible for being aware of the current available resources on the platform, and placing containers / applications on the platform appropriately.

All pods should have a resource request that is the minimum amount of resources the pod is expected to use at steady state for both memory and CPU.

There are several options for volumes and reading and writing files in OpenShift. When the requirement is temporary storage and given the option to write files into directories in containers versus an external filesystems, choose the emptyDir option. This will provide the administrator with the same temporary filesystem - when the pod is stopped the dir is deleted forever. Also, the emptyDir can be backed by whatever medium is backing the node, or it can be set to memory for faster reads and writes.

Using emptyDir with requested local storage limits instead of writing to the container directories also allows enabling readonlyRootFilesystem on the container or pod.

As part of the pod lifecycle, the OpenShift platform needs to know what state the pod is in at all times. This can be accomplished with different health checks. There are at least three states that are important to the platform: startup, running, shutdown. Applications can also be running, but not healthy, meaning, the pod is up and the application shows no errors, but it cannot serve any requests.

When an application starts up on OpenShift it may take a while for the application to become ready to accept connections from clients, or perform whatever duty it is intended for.

Two health checks that are required to monitor the status of the applications are liveness and readiness. As mentioned above, the application can be running but not actually able to serve requests. This can be detected with liveness checks. The liveness check will send specific requests to the application that, if satisfied, indicate that the pod is in a healthy state and operating within the required parameters that the administrator has set. A failed liveness check will result in the container being restarted.

There is also a consideration of pod startup. Here the pod may start and take a while for different reasons. Pods can be marked as ready if they pass the readiness check. The readiness check determines that the pod has started properly and is able to answer requests. There are circumstances where both checks are used to monitor the applications in the pods. A failed readiness check results in the container being taken out of the available service endpoints. An example of this being relevant is when the pod was under heavy load, failed the readiness check, gets taken out of the endpoint pool, processes requests, passes the readiness check and is added back to the endpoint pool.

For more information, see Configure Liveness, Readiness and Startup Probes.

See test cases lifecycle-liveness-probe, lifecycle-readiness-probe, lifecycle-startup-probe

If there is a situation where the container dies and needs to be restarted, the image pull policy becomes important. There are three image pull policies available: Always, Never and IfNotPresent. It is generally recommended to have a pull policy of IfNotPresent. This means that the if pod needs to restart for any reason, the kubelet will check on the node where the pod is starting and reuse the already downloaded container image if it’s available. OpenShift intentionally does not set AlwaysPullImages as turning on this admission plugin can introduce new kinds of cluster failure modes. Self-hosted infrastructure components are still pods: enabling this feature can result in cases where a loss of contact to an image registry can cause redeployment of an infrastructure or application pod to fail. We use PullIfNotPresent so that a loss of image registry access does not prevent the pod from restarting.

See test case lifecycle-image-pull-policy

Do not use naked Pods (that is, Pods not bound to a ReplicaSet, or StatefulSet deployment). Naked pods will not be rescheduled in the event of a node failure.

See test case lifecycle-pod-owner-type

init containers can be used for running tools or commands or any other action that needs to be done before the actual pod is started. For example, loading a database schema, or constructing a config file from a definition passed in via ConfigMap or Secret.

See Using init containers to perform tasks before a pod is deployed for more information.

It is not recommended for an application to access following resources on the host.

It is not necessary to mount host /sys/ or host /dev/ directories as a volume in a pod in order to use a network device such as SR-IOV VF. The moving of a network interface into the pod network namespace is done automatically by CNI. Mounting the whole /sys/ or /dev/ directory in the container will overwrite the network device descriptor inside the container which causes device not found or no such file or directory error.

Network interface statistics can be queried inside the container using the same /sys/ path as was done when running directly on the host. When running network interfaces in containers, relevant /sys/ statistics interfaces are available inside the container, such as /sys/class/net/net1/statistics/, /proc/net/tcp and /proc/net/tcp6.

For running DPDK applications with SR-IOV VF, device specs (in case of vfio-pci) are automatically attached to the container via the Device Plugin. There is no need to mount the /dev/ directory as a volume in the container as the application can find device specs under /dev/vfio/ in the container.

Application pods must avoid using hostNetwork. Applications may not use the host network, including nodePort, for network communication. Any networking needs beyond the functions provided by the pod network and ingress/egress proxy must be serviced via a MULTUS connected interface.

Linux Capabilities allow you to break apart the power of root into smaller groups of privileges. Platform administrators can use Pod Security Policy (PSP) to control permissions for pods. Users can also specify the necessary Security Context in the pod annotations.

Even though ‘mlock’ is not necessary on systems where page swap is disabled, it may still be required as it is a function that is built into DPDK libraries, and DPDK based applications may indirectly call it by calling other functions.

This doesn’t include:

All the administrative operations (except ethtool) mentioned above that require the NET_ADMIN capability should already be supported on the host by various CNIs on CaaS Platform.

The burstable workload can float around on the CPUs depending on the availability of resources. More information can be found in the Low latency tuning documentation.

The Workload that is not sensitive to CPU migration and context switching, e.g. OAM, is suitable for using the shared CPU pool. For more information see the Kubernetes Pod QoS class docs.

CPU resource request example:

CPUsets constrain the CPU and memory placement of task to the resources within a task’s current cpuset. Within cgroup, the workload can set CPU affinity to reduce the context switching, however CPU scheduler still performs load balances for non-cpu-pinned workload across all the CPUs in the cpuset. In addition, the following annotation is required to accompany the workload. See Disabling CPU CFS quota:

A workload should use Application exclusive CPU pool if it has CPU low-latency requirement. However, the workload should allow context switching with other application threads and kernel threads in the cpuset. An example workload that uses application exclusive pool is a vRAN RT task.

CPU resource request example:

The above applies for guaranteed and non-guaranteed QoS within a single Pod.

It is possible to create a Pod with multiple containers and only pin some of the CPUs.

Follow these rules:

Before enabling PTP, ensure that NTP is disabled for the required nodes. This can be done by disabling the chrony time service (chronyd) using a MachineConfig custom resource. For more information, see Disabling chrony time service.

After disabling the NTP service, the next thing is to properly assign labels to the cluster nodes. Labels ensure that the PTP operator applies PTP profiles to the correct cluster node. The use of labels facilitates in-cluster object searches, which are leveraged by the PTP operator (using a match field) to select the right cluster nodes and configure the ptp4l and phc2sys programs accordingly. See the following examples:

These labels are used by the match field in PtpConfig CR to match a profile to one or more nodes.

Configuring PTP is a matter of installing the PTP Operator, selecting a capable NIC using the NodePtpDevice CR and setting up the PtpConfig CR with the appropriate values to work as:

Notice, however, that to fulfill Far Edge requirements regarding low latency the PTP Operator linuxptp services must be set to allow threads to run with a SCHED_FIFO policy.

The follow topology illustrates the PTP operator running on cluster nodes where the O-RAN workloads are hosted:

The GM device sends its timestamps downstream and these are received in the boundary clock slave ports of a vDU (denoted in the image with a capital S). These timestamps are then used by the ptp4l program to adjust the local PHC in the corresponding cluster node. This also happens from the vDU master ports toward RU ordinary clocks slave ports.

At the same time, the phc2sys process running in both the O-DU and O-RU nodes read the PHC timestamps from the ptp4l program through a shared Unix Domain Socket (UDS) and finally adjusting the SYSTEM_CLOCK offset, as shown in the figure below.

This process is continuously repeated keeping all the PHC instances in the O-RAN deployment effectively synchronized to the GNSS source clock, which acts as the GM in this scenario.

The recommended NICs with hardware PTP support are:

This framework mitigates workload errors by allowing cluster nodes to directly communicate PTP clock sync status to the vRAN application running as part of the DU. These event notifications are available to RAN applications running on the same DU node using a publish/subscribe REST API approach that provides event notifications via a fast messaging bus.

Fast event notifications are generated by the PTP Operator in the OpenShift Container Platform for every PTP-capable network interface. Specifically, it uses an AMQP event notification bus provided by the AMQ Interconnect Operator that delivers flexible routing of messages between any AMQP-enabled endpoints. A high-level overview of the PTP fast events framework is below:

From the Openshift PTP perspective, the idea is to [install the AMQ Interconnect Operator and configure the PTP Operator.

Then, update the PtpConfig CR to include the relevant values in each case (sample values for ptpClockThreshold are shown below):

The ptpClockThreshold configures how long the PTP operator should wait after losing the PTP master clock signal, triggering that error as a PTP event:

From the DU application perspective, a new cloud-event-proxy sidecar container is appended to the DU application Pod that is loosely coupled to the main DU application container on the DU node. It provides an event publishing framework that allows you to subscribe to DU applications to any published PTP event. A detailed explanation of how it works can be found here.

The steps to get a DU application subscribed are described here, along with several examples of use.

Monitoring PTP fast event metrics PTP fast event metrics can be monitored in the OpenShift Container Platform web console by using the pre-configured and self-updating Prometheus monitoring stack as explained here. From the DU application perspective, the Prometheus metrics support helps to set up custom alerting manager systems for early detection issues.

It is not recommended for an application to access the following resources on the host.

It is not necessary to mount host /sys/ or host /dev/ directory as a volume in a pod in order to use a network device such as SR-IOV VF. The moving of a network interface into the pod network namespace is done automatically by CNI. Mounting the whole /sys/ or /dev/ directory in the container will overwrite the network device descriptor inside the container which causes 'device not found' or 'no such file or directory' error.

Network interface statistics can be queried inside the container using the same /sys/ path as was done when running directly on the host. When running network interfaces in containers, relevant /sys/ statistics interfaces are available inside the container, such as /sys/class/net/net1/statistics/, /proc/net/tcp and /proc/net/tcp6.

For running DPDK applications with SR-IOV VF, device specs (in case of vfio-pci) are automatically attached to the container via the Device Plugin. There is no need to mount the /dev/ directory as a volume in the container as the application can find device specs under /dev/vfio/ in the container.

Application pods must avoid using hostNetwork. Applications may not use the host network, including nodePort for network communication. Any networking needs beyond the functions provided by the pod network ingress/egress proxy must be serviced via a multus connected interface.

The default capabilities that are allowed via the restricted SCC are as follows.

IPC_LOCK capability is required if any of these functions are used in an application:

Even though mlock() is not necessary on systems where page swap is disabled (for example on OpenShift), it may still be required as it is a function that is built into DPDK libraries, and DPDK based applications may indirectly call it by calling other functions.

See test case access-control-ipc-lock-capability-check

NET_ADMIN capability is required to perform various network related administrative operations inside container such as:

This doesn’t include:

All the administrative operations (except ethtool) mentioned above that require the NET_ADMIN capability should already be supported on the host by various CNIs in Openshift.

CNI TAP can be used to create tap devices and avoid NET_ADMIN.

This capability is very powerful and overloaded. It allows the application to perform a range of system administration operations to the host. So you should avoid requiring this capability in your application.

In the case that a workload is running on a node using the real-time kernel, SYS_NICE will be used to allow DPDK application to switch to SCHED_FIFO.

See test case access-control-sys-nice-realtime-capability

This capability is required when using Process Namespace Sharing. This is used when processes from one Container need to be exposed to another Container. For example, to send signals like SIGHUP from a process in a Container to another process in another Container. See Share Process Namespace between Containers in a Pod for more details. For more information on these capabilities refer to Linux Capabilities in OpenShift.

See test case access-control-sys-ptrace-capability

The SR-IOV Network Operator also supports configuring the following parameters for SR-IOV VFs. Refer to Configuring an SR-IOV Ethernet network attachment for more information.

In OpenShift, multicast is supported for both the default interface (OVN or OpenShift-SDN) and the additional interfaces such as macvlan, SR-IOV, etc. Multicast is disabled by default. To enable it, refer to the following procedures:

If your application works as a multicast source and you want to utilize the additional interfaces to carry the multicast traffic, then you don’t need the NET_ADMIN capability. Follow the instructions in Using high performance multicast to set the correct multicast route in the pod’s routing table.

Secrets objects in OpenShift provide a way to hold sensitive information such as passwords, config files and credentials. There are 4 types of secrets; service account, basic auth, ssh auth and TLS. Secrets can be added via deployment configurations or consumed by pods directly. For more information on secrets and examples, see the following documentation.

Providing sensitive data to pods