Manage Kompass agent resources

Compute Optimization automatically manages the CPU and memory resources used by the Kompass agent and its components, with the following features:

• Self-rightsizing: Adjusts resource settings in place as component needs change.

• Guardian: Raises CPU and memory resources when a component needs more.

Both features are enabled by default.

This topic explains how they work and how to manage them.

How self-rightsizing works

Self-rightsizing automatically applies Pod Rightsizing to all Kompass agent components running as these workload types:

• Deployments

• StatefulSets

• DaemonSets

It does not apply to CronJobs, Jobs, or installation and upgrade hooks. Compute Optimization manages Kompass agent components through a dedicated policy. You do not need to configure the policy, and custom policies cannot manage these components.

Self-rightsizing changes the resources of running Pods in place. It does not restart Pods or change workload manifests.

For more information, see Pod Rightsizing.

How Guardian works

Guardian monitors Kompass agent components for OOM kills and sustained CPU throttling. When it detects either condition, Guardian raises the affected component’s CPU and memory requests and limits, adds a Guardian annotation, and rolls out the component.

Guardian only increases CPU and memory requests and limits. It never lowers them. It saves the raised value as a safe minimum for the container and restores it if a later Helm upgrade or Git sync sets a lower value. Compute Optimization determines how much Guardian raises resources and may tune this amount between releases.

Scope and expected behavior

Self-rightsizing and Guardian manage Kompass agent components and bundled components such as VictoriaMetrics, Grafana, kube-state-metrics, cert-manager, and metrics-server. They do not manage customer workloads. Although other workloads in the Kompass agent’s namespace follow their regular policies and namespace exclusions, install the Kompass agent in a dedicated namespace.

When Guardian intervenes, it replaces the affected Pods one by one. During recovery, normal rightsizing for that component pauses until the new Pods are healthy and the default 10-minute stabilization period has passed.

If the increased resource request does not fit on an available node, the replacement Pod remains Pending until capacity is available. Guardian does not add nodes, and the previous Pods continue running according to the workload’s rollout strategy.

A Helm upgrade can trigger an extra rollout if it sets values below the safe minimum, because Guardian restores that minimum. Argo CD may report drift because Guardian changes the workload.

Requirements and disabling self-rightsizing and Guardian

Self-rightsizing and Guardian require Kubernetes in-place Pod resize support. If your cluster does not support in-place Pod resize, disable both features.

Disable options

Both features are enabled by default.

You can disable both features for ongoing use or temporarily disable either feature while troubleshooting.

Disable both features for ongoing use

To disable both features for ongoing use, use the resources-without-guardian.yaml file that ships with the chart. This file keeps the Kompass agent’s default resource settings. Apply your own values file after it so your settings take precedence:

helm upgrade <release> <kompass-chart> -n <kompass-namespace> \

  -f resources-without-guardian.yaml \

  -f <your-values>.yaml

Temporarily disable either feature while troubleshooting

To disable either feature temporarily while troubleshooting, set the corresponding Helm value and upgrade the chart:

kompassWorkloadsRightsizingEnabled: false

guardian:

  enabled: false

Set each value you changed back to true when the issue is resolved.

Troubleshooting

Start by checking the Guardian logs. The logs include reason codes that explain Guardian’s decisions. Replace <kompass-namespace> with the namespace where you installed the Kompass agent:

kubectl logs deploy/kompass-guardian -n <kompass-namespace>

To check whether a component is recovering, look for the guardian.kompass.zesty.co/recovery-state annotation. While this annotation is present, Compute Optimization pauses normal rightsizing for that component.

Guardian stores safe minimums in the kompass-guardian-store-config ConfigMap in the Kompass agent namespace.

A component restarted and now requests more resources

This is expected when Guardian responds to an OOM kill or sustained CPU throttling. The raised values are kept as safe minimums. Check the Guardian logs to see what changed and why.

Replacement Pods remain Pending

If no node can fit the larger resource request, the Guardian logs show the reason code RecoveryNotApplied. Add capacity or allow the cluster autoscaler to scale up. Recovery continues when capacity is available.

OOM kills continue and resources are not raised further

Guardian may have reached the per-container ceiling. The log reason code is CeilingReached. Contact Zesty Support to review the ceiling.

Rightsizing actions fail with InPlaceResizeUnsupported

The cluster does not support in-place Pod resize. Upgrade the cluster or use resources-without-guardian.yaml to disable self-rightsizing and Guardian.

Values change back after a Git sync

Argo CD may be reverting Guardian’s changes. The log reason code is StoredFloorViolation. If Argo CD manages the cluster externally, configure it to ignore fields that Guardian and other Kompass agent components manage, as described below.

Values are raised again after a Helm upgrade

This is expected when the upgrade sets values below a safe minimum. The log reason code is StoredFloorViolation. To intentionally use lower values, reset the safe minimum.

Guardian is not ready after a restart

If the saved Guardian data is invalid, the log reason code is FloorStoreInvalid or RecoveryStateInvalid. Contact Zesty Support. Do not delete or edit the ConfigMap yourself.

Allow Guardian changes in Argo CD

If the Argo CD Application runs in the same cluster as the Kompass agent, a Compute Optimization webhook adds the ignore rules automatically. If Argo CD manages the cluster from a separate management cluster, add these rules to the Application so Argo CD ignores changes made by Guardian and other Kompass agent components:

spec:
  ignoreDifferences:
    - group: "*"
      kind: "*"
      managedFieldsManagers:
        - kompass-guardian
    - group: "*"
      kind: "*"
      managedFieldsManagers:
        - action-taker
    - group: "*"
      kind: "*"
      managedFieldsManagers:
        - insights-agent
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

The webhook also configures Argo CD to respect these ignore rules and ignores its own changes to the Application.

Reset a safe minimum

Use this procedure only when you deliberately want a component to run below the value Guardian set.

1. Scale the kompass-guardian Deployment to 0 replicas.

2. Remove the component’s entry from the kompass-guardian-store-config ConfigMap. If the component has a guardian.kompass.zesty.co/recovery-state annotation, remove it too.

3. Apply the resource values you want.

4. Scale the kompass-guardian Deployment back up.

Deleting the entire kompass-guardian-store-config ConfigMap clears every saved safe minimum.