Application Best Practices

On the Contain Platform, we provide a secure and resilient foundation, but the security and reliability of the overall system is a shared responsibility. The way you build, configure, and deploy your applications plays a critical role.

This guide provides a comprehensive set of best practices for developing applications that are secure, resilient, and well-behaved on a Kubernetes platform. Following these guidelines will not only enhance your application's security posture but will also improve its stability, maintainability, and performance.

The practices are organized into four key areas:

1. Supply Chain Security: Securing your application before it's deployed.
2. Runtime Security: Hardening your application's configuration.
3. Reliability and Resilience: Ensuring your application is stable and highly available.
4. Resource and Lifecycle Management: Making your application a "good citizen" in the cluster.

---

1. Supply Chain Security

A secure application starts with a secure supply chain. These practices focus on the integrity and security of your container images.

Use Minimal, Trusted Base Images

The smaller your container image, the smaller your attack surface.

• Use distroless or minimal alpine images: Avoid using full OS base images like ubuntu or centos. Images from projects like Google's Distroless contain only your application and its runtime dependencies, nothing else.
• Use multi-stage builds: A multi-stage Dockerfile allows you to build your application in a "builder" image that contains all the build tools, and then copy only the final compiled artifact into a minimal production image.

Use Specific, Immutable Image Tags

Never use the :latest tag for your container images in production. This tag is mutable and makes your deployments unpredictable and difficult to roll back reliably.

Always use specific and immutable tags that clearly map to a version of your code.

• Good: my-app:v1.2.3 (Semantic Version)
• Better: my-app:v1.2.3-a1b2c3d (Semantic Version + Git SHA)

Use Private, Scanned, and Signed Images

• Use a Private Registry: Always pull images from a trusted, private container registry, such as our managed Container Registry Service. This prevents the use of unvetted public images and protects you from public registry rate-limiting or outages.
• Scan for Vulnerabilities: Integrate vulnerability scanning into your CI/CD pipeline. Our managed registry provides this automatically, allowing you to block deployments that contain high-severity vulnerabilities.
• Use Signed Images: For the highest level of trust, cryptographically sign your container images using a tool like Cosign. Our platform can be configured with policies to verify these signatures, ensuring that only images built by your trusted CI/CD pipelines can run in your cluster.

---

2. Runtime Security

Once you have a secure image, the next step is to ensure it runs securely in the cluster.

Harden Your Pods with securityContext

The securityContext is the most critical tool for locking down your running containers. Apply these settings to all your production workloads.

# In your Deployment, under spec.template.spec
spec:
  template:
    spec:
      containers:
      - name: my-app
        image: ...
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 1000
          runAsGroup: 3000
          readOnlyRootFilesystem: true
          capabilities:
            drop: [ "ALL" ]

• runAsNonRoot: true: The single most important setting. It prevents your container from running as the powerful root user.
• allowPrivilegeEscalation: false: Prevents a process from gaining more privileges than its parent, closing a common attack vector.
• readOnlyRootFilesystem: true: Makes the container's filesystem immutable. If your application needs to write temporary files, mount a dedicated tmpfs volume for that purpose.
• capabilities: { drop: ["ALL"] }: Adheres to the principle of least privilege by removing all Linux capabilities. Most web applications do not require any.

Mandatory Configuration

The Contain Platform enforces these security settings via the Restricted Pod Security Standard. Pods that do not meet these requirements will be rejected. For detailed, step-by-step instructions on how to configure your workloads, see the guide:

• Guide: Configuring Pod Security Contexts

Isolate Network Traffic with Network Policies

The platform enforces a default-deny network model by default. All traffic is denied unless explicitly allowed by a NetworkPolicy. This prevents lateral movement by attackers and is a critical security control. Be specific in your policies; for example, if your API only needs to talk to your database on port 5432, create a policy that only allows that specific connection.

Manage Secrets Securely

Never store secrets in your Git repository or in plain ConfigMap resources. Use a dedicated secret store like HashiCorp Vault or OpenBao and sync them into the cluster using our External Secrets Operator. This ensures sensitive data is only ever stored securely in the vault and in the cluster's memory at runtime.

---

3. Reliability and Resilience

These practices ensure your application is stable, highly available, and handles updates gracefully.

Implement Health Probes

Health probes are critical. Without them, Kubernetes has no way of knowing if your application is running correctly, which can lead to downtime during deployments and failures.

• livenessProbe: Tells Kubernetes if your application is deadlocked or has crashed. If this probe fails, Kubernetes will restart the container. Use this to recover from unrecoverable errors.
• readinessProbe: Tells Kubernetes if your application is ready to serve traffic. If this probe fails, Kubernetes will not send traffic to the pod. This is crucial for zero-downtime rolling updates.
• startupProbe: Useful for slow-starting applications. It disables the other probes until the application has finished starting up, preventing Kubernetes from killing it prematurely.

# In your Deployment, under spec.template.spec.containers[]
spec:
  containers:
  - name: my-app
    # ...
    ports:
    - containerPort: 8080
      name: http
    readinessProbe:
      httpGet:
        path: /healthz
        port: http
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      httpGet:
        path: /healthz
        port: http
      initialDelaySeconds: 15
      periodSeconds: 20

Ensure High Availability

• Run Multiple Replicas: For any production workload, run a minimum of replicas: 2. This prevents downtime during routine node maintenance or unexpected node failures.
• Use Pod Disruption Budgets (PDBs): A PodDisruptionBudget protects your application from voluntary disruptions (like a cluster upgrade). It guarantees that a minimum number of replicas will always be available. For an application with 3 replicas, a PDB with minAvailable: 2 is a strong choice.

Handle Graceful Shutdown

When Kubernetes needs to terminate a pod (e.g., during a deployment), it sends a SIGTERM signal to your application's process. Your application should catch this signal and begin shutting down gracefully by:

1. Stopping to accept new connections.
2. Finishing any in-flight requests.
3. Closing connections to databases or other downstream services.

If your application does not handle SIGTERM and instead shuts down immediately, you risk dropping user requests and leaving orphaned connections.

---

4. Resource and Lifecycle Management

These practices ensure your application is a "good citizen" that coexists well with other workloads in the cluster.

Manage Resources Wisely (Requests and Limits)

Setting resource requests and limits is one of the most critical aspects of running applications on Kubernetes. It ensures stability for both your application and the cluster as a whole. However, the best practices for CPU and Memory are different and should be understood separately.

Our platform may enforce policies that reject pods that do not have requests set for both CPU and memory.

Memory: Always Set Requests and Limits

Memory is an incompressible resource. If an application tries to use more memory than is available, the node's kernel will terminate the process with an Out Of Memory (OOM) error. To prevent this and ensure stability, you must always set both memory requests and limits.

• memory requests: Guarantees your pod a minimum amount of memory. This is the primary value Kubernetes uses to decide where to schedule your pod.
• memory limits: Sets a hard cap on the memory your pod can use. If it exceeds this limit, Kubernetes will terminate it.

Best Practice: Set requests and limits to the same value (requests: 1Gi, limits: 1Gi). This gives your pod a "Guaranteed" Quality of Service (QoS) class, making it less likely to be killed during node pressure events.

CPU: Set Requests, Avoid Limits

CPU is a compressible resource. If a pod needs more CPU, it can be throttled, but it won't be terminated. While setting a CPU limit seems like a good way to prevent a "noisy neighbor," it often causes more problems than it solves.

• The Problem with CPU Limits: When a container hits its CPU limit, the Linux kernel forcefully throttles it. This can cause significant, unexpected latency spikes in your application, even if the node has plenty of idle CPU cycles available. It prevents your application from handling legitimate bursts in traffic or processing.

• cpu requests: This is the most important setting. It guarantees your pod a minimum amount of CPU time and is used by Kubernetes to make scheduling decisions. It also determines the pod's share of CPU when there is contention on the node.

• cpu limits: In most cases, you should not set CPU limits. Omitting the limit allows your application to "burst" and use any available, unused CPU on the node, leading to better performance without harming other applications.

Best Practice:

1. Always set CPU requests: Analyze your application's typical usage (e.g., its 95th percentile CPU usage) and set a request based on that.
2. Do not set CPU limits unless necessary: Only enforce limits for specific, non-critical workloads that you know can be disruptive (e.g., batch jobs that are not latency-sensitive). For most applications, especially web services, omitting the limit is the better choice.

# In your Deployment, under spec.template.spec.containers[]
spec:
  containers:
  - name: my-app
    image: ...
    resources:
      requests:
        cpu: "250m"      # Set based on typical usage
        memory: "512Mi"  # Set based on application needs
      limits:
        # Do NOT set a CPU limit in most cases
        memory: "512Mi"  # Set memory limit equal to request

Use Recommended Labels

Labels are critical for organizing and selecting Kubernetes resources. While you can use any labels you want, adopting a standardized set makes your applications easier to manage, observe, and automate. Kubernetes recommends a set of common labels that can be applied to all your resources.

Key	Description	Example
app.kubernetes.io/name	The name of the application	hello-world
app.kubernetes.io/instance	A unique name identifying the instance	hello-world-abcxyz
app.kubernetes.io/version	The current version of the application	0.1.0
app.kubernetes.io/component	The component within the architecture	frontend
app.kubernetes.io/part-of	The name of a higher-level application this one is part of	tutorials

Using these labels consistently allows tools and team members to quickly understand the structure of your application. For example, you can easily find all the database components for your wordpress application.

# In your Deployment, under metadata
metadata:
  name: my-app-db
  labels:
    app.kubernetes.io/name: mysql
    app.kubernetes.io/instance: my-app-db
    app.kubernetes.io/version: "8.0"
    app.kubernetes.io/component: database
    app.kubernetes.io/part-of: my-app

Clean Up Completed Jobs

If you run batch tasks using Kubernetes Jobs, use the ttlSecondsAfterFinished field to automatically garbage collect the Job and its Pods after they complete. This prevents cluttering the API server with thousands of completed jobs over time.