Customizing Transparent Proxy Configuration in Kubernetes Mode

The default transparent proxy configuration works well for most scenarios, but there are cases where adjustments are needed. This guide explains the various methods available for modifying the configuration, along with their limitations and recommendations on when to use each one. While these methods can be used individually, they can also be combined when necessary.

Terminology overview

• Workload: In this guide, workload refers to an application running in a Kubernetes cluster, usually represented by a Pod. Kubernetes resources like Deployment, ReplicaSet, and StatefulSet are used to define and manage these workloads, resulting in one or more Pods where the application actually runs.

• ConfigMap: In this guide, ConfigMap refers to the Kubernetes ConfigMap resource. It’s used to store configuration data as key-value pairs that can be easily accessed by other resources in the cluster, such as Pods, Deployments, and StatefulSets.

Method 1: Control plane runtime configuration

The control plane runtime configuration is a section of Kuma’s configuration that contains important settings for how the control plane operates, including options for the transparent proxy and other key components.

Some transparent proxy settings can be adjusted here, and for certain settings, this is the only place they can be modified.

Currently, it’s best to use the control plane runtime configuration as the main place to store transparent proxy settings that apply to all workloads. In the future, once the Configuration in ConfigMap ^{(experimental)} feature is fully supported, we’ll recommend using ConfigMaps for these settings. However, a few restricted settings that are rarely customized will still need to be in the control plane’s runtime configuration.

Below is a subset of the configuration focused on transparent proxy settings, with default values and corresponding environment variables for each field. For further details, see the Modifying control plane runtime configuration section.

runtime:
  kubernetes:
    injector:
      cniEnabled: false # KUMA_RUNTIME_KUBERNETES_INJECTOR_CNI_ENABLED
      applicationProbeProxyPort: 9001 # KUMA_RUNTIME_KUBERNETES_APPLICATION_PROBE_PROXY_PORT
      sidecarContainer:
        uid: 5678 # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_UID
        ipFamilyMode: dualstack # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_IP_FAMILY_MODE
        redirectPortInbound: 15006 # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_REDIRECT_PORT_INBOUND
        redirectPortOutbound: 15001 # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_REDIRECT_PORT_OUTBOUND
      sidecarTraffic:
        excludeInboundPorts: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_INBOUND_PORTS
        excludeOutboundPorts: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_PORTS
        excludeInboundIPs: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_INBOUND_IPS
        excludeOutboundIPs: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS
      builtinDNS:
        enabled: true # KUMA_RUNTIME_KUBERNETES_INJECTOR_BUILTIN_DNS_ENABLED
        port: 15053 # KUMA_RUNTIME_KUBERNETES_INJECTOR_BUILTIN_DNS_PORT
      ebpf:
        enabled: false # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_ENABLED
        instanceIPEnvVarName: INSTANCE_IP # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_INSTANCE_IP_ENV_VAR_NAME
        bpffsPath: /sys/fs/bpf # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_BPFFS_PATH
        cgroupPath: /sys/fs/cgroup # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_CGROUP_PATH
        tcAttachIface: "" # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_TC_ATTACH_IFACE
        programsSourcePath: /tmp/kuma-ebpf # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_PROGRAMS_SOURCE_PATH

Settings restricted to control plane runtime configuration

Some transparent proxy settings can only be changed through the control plane’s runtime configuration because they are shared with other Kuma components. These settings handle essential tasks like creating Dataplane resources for workloads and setting up the kuma-dp sidecar container alongside them. For example, the DNS port used to redirect traffic is shared between the transparent proxy and the kuma-dp run command in the sidecar container. Keeping these settings consistent across workloads prevents misconfigurations. These settings are:

• runtime.kubernetes.injector.sidecarContainer.redirectPortInbound
• runtime.kubernetes.injector.sidecarContainer.redirectPortOutbound
• runtime.kubernetes.injector.sidecarContainer.uid
• runtime.kubernetes.injector.builtinDNS.enabled
• runtime.kubernetes.injector.builtinDNS.port

If you try to change these settings with methods described later in this guide, but with values different from the runtime configuration, the effects will vary:

• runtime.kubernetes.injector.sidecarContainer.redirectPortInbound and runtime.kubernetes.injector.sidecarContainer.redirectPortOutbound

  Attempts to change these settings using kuma.io/transparent-proxying-inbound-port and kuma.io/transparent-proxying-outbound-port annotations, or by updating redirect.inbound.port and redirect.outbound.port in the ConfigMap, will trigger warnings in the control plane logs, but the values will be ignored.

• runtime.kubernetes.injector.sidecarContainer.uid

  Attempts to modify this setting using the kuma.io/sidecar-uid annotation or kumaDPUser in the ConfigMap will result in:

  • A warning in control plane logs if the Configuration in ConfigMap ^{(experimental)} feature is enabled, and the values will be ignored.
  • Silent ignoring of the change without warnings if the feature is disabled.

• runtime.kubernetes.injector.builtinDNS.enabled and runtime.kubernetes.injector.builtinDNS.port

  Attempts to modify these settings using deprecated annotations kuma.io/builtin-dns and kuma.io/builtin-dns-port, or redirect.dns.enabled and redirect.dns.port in the ConfigMap, will have the following effects:

  • If the Configuration in ConfigMap ^{(experimental)} feature is enabled, warnings will appear in the control plane logs, and values will be ignored.

  • If the feature is disabled, changes through annotations will still apply, possibly leading to broken DNS redirection. This can result in kuma-dp not starting the DNS server or listening on the wrong port, which may create issues in the environment.

    ⚠️ This behavior differs from that of other settings. Ensure you’re aware of this distinction and never attempt to modify these settings using the mentioned annotations.

Modifying control plane runtime configuration

To modify the configuration, you have two main options:

1. Store the desired settings in the kuma-control-plane-config ConfigMap:

   This method is recommended because it simplifies future modifications. Rather than reinstalling Kuma or manually updating its Deployment to change environment variables, you can just update the ConfigMap and restart the kuma-control-plane Pod to apply the changes.

   You can provide the full runtime configuration in the kuma-control-plane-config ConfigMap within the kuma-system namespace.

   Here’s an example that excludes specific outbound IPs from transparent proxy redirection:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kuma-control-plane-config
     namespace: kuma-system
   data:
     config.yaml: |
       runtime:
         kubernetes:
           injector:
             sidecarTraffic:
               excludeOutboundIPs: [10.1.0.254, 172.10.1.254]
   

   Instead of manually updating the ConfigMap in the cluster, you can specify the configuration it will store during the Kuma installation. Follow these steps:

   1. Create a values.yaml file with the following content:

      controlPlane:
        config: |
          runtime:
            kubernetes:
              injector:
                sidecarTraffic:
                  excludeOutboundIPs: [10.1.0.254, 172.10.1.254]
      

      Make sure the content under controlPlane.config is a string representation of the control plane configuration.

   2. Install Kuma using kumactl or Helm:

      • kumactl
      • Helm

      kumactl install control-plane --values values.yaml | kubectl apply -f -

      Before using Kuma with Helm, ensure that you’ve followed these steps to configure your local Helm repository.

      helm install \
        --create-namespace \
        --namespace kuma-system \
        --values values.yaml \
        kuma kuma/kuma

   If you’re deploying Kuma using Helm, you can place the configuration in a separate file and pass it during installation with the --set-file flag.

   For example, create a cp-conf.yaml file with the following content:

   runtime:
     kubernetes:
       injector:
         sidecarTraffic:
           excludeOutboundIPs: [10.1.0.254, 172.10.1.254]

   Then, during Helm installation, run:

   helm install \
     --create-namespace \
     --namespace kuma-system \
     --set-file "controlPlane.config=cp-conf.yaml" \
     kuma kuma/kuma

   This will have the same effect as using --values values.yaml, as described earlier.

2. Configure specific settings through environment variables:

   You can also set individual configuration values using environment variables. For example, to exclude specific outbound IPs from the transparent proxy, update the kuma-control-plane Deployment by setting the environment variable KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS with the desired IP addresses, separated by commas. For instance: 10.1.0.254,172.10.1.254.

   Manually editing Kubernetes manifests for Kuma is strongly discouraged and should only be done for testing in non-production environments. Making manual changes can increase the risk of misconfiguration and may cause unexpected issues. For dynamic configuration updates without reinstalling Kuma, use the kuma-control-plane-config ConfigMap as described above.

   Setting environment variables during the initial installation, as shown below, is safe for any environment. While the ConfigMap is the preferred method, environment variables are also a reliable option for setup.

   To apply environment variables to Kuma’s control plane automatically, configure them during installation using the controlPlane.envVars setting. For example:

   • kumactl
   • helm

   kumactl install control-plane \
     --set "controlPlane.envVars.KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS=10\.1\.0\.254\,172\.10\.1.254" \
     | kubectl apply -f -

   Before using Kuma with helm, please follow these steps to configure your local helm repo.

   helm install --create-namespace --namespace kuma-system \
     --set "controlPlane.envVars.KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS=10\.1\.0\.254\,172\.10\.1.254" \
     kuma kuma/kuma

Overriding list-based configuration settings

For list-based settings like runtime.kubernetes.injector.sidecarTraffic.excludeOutboundPorts, only the value from the highest-precedence method will be applied, and values from other methods will not be combined.

For example, if runtime.kubernetes.injector.sidecarTraffic.excludeOutboundPorts is set in the control plane runtime configuration but an annotation like traffic.kuma.io/exclude-outbound-ports is applied to a specific workload, the annotation will override the control plane setting. This means the transparent proxy will exclude only the ports specified in the annotation.

See the Order of Precedence section to understand how different methods affect the final configuration applied.

Method 2: Configuration in ConfigMap (experimental)

Until Kuma 2.9, transparent proxy settings could only be modified through the Control Plane Runtime Configuration and Kubernetes Annotations, which had several limitations:

• Not all settings were available through both methods; some could only be adjusted with annotations, while others were limited to the runtime configuration.
• Control plane runtime settings were applied globally to all workloads with injected data planes, making it hard to customize settings for specific groups of workloads.
• Annotation-based settings had to be added to each workload individually, making it difficult to manage the same configuration across multiple workloads.

To address these issues, Kuma now supports storing transparent proxy settings in dedicated ConfigMaps. These ConfigMaps can be applied at the namespace or individual workload level, offering greater flexibility and easier configuration.

To enable this feature, set transparentProxy.configMap.enabled during installation:

kumactl install control-plane \
  --set "transparentProxy.configMap.enabled=true" \
  | kubectl apply -f -

helm install --create-namespace --namespace kuma-system \
  --set "transparentProxy.configMap.enabled=true" \
  kuma kuma/kuma

ConfigMap auto-creation and configuration

During installation, Kuma will automatically create a ConfigMap in the kuma-system namespace. The ConfigMap will be named based on the transparentProxy.configMap.name setting, and its content will come from the YAML configuration defined in transparentProxy.configMap.config, which holds the transparent proxy settings.

Here is an example of how to modify parts of this configuration during installation:

kumactl install control-plane \
  --set "transparentProxy.configMap.config.enabled=true" \
  --set "transparentProxy.configMap.config.redirect.outbound.excludePortsForIPs=10\.1\.0\.254\,172\.10\.1.254" \
  --set "transparentProxy.configMap.config.redirect.verbose=false" \
  | kubectl apply -f -

helm install --create-namespace --namespace kuma-system \
  --set "transparentProxy.configMap.config.enabled=true" \
  --set "transparentProxy.configMap.config.redirect.outbound.excludePortsForIPs=10\.1\.0\.254\,172\.10\.1.254" \
  --set "transparentProxy.configMap.config.redirect.verbose=false" \
  kuma kuma/kuma

You can apply the same method of providing these values as explained in the Store the desired settings in the kuma-control-plane-config ConfigMap option in the Modifying Control Plane Runtime Configuration section.

ConfigMap lookup strategy

Kuma follows a specific order when searching for the ConfigMap:

• If the workload is annotated with traffic.kuma.io/transparent-proxy-configmap-name, Kuma first looks for the resource with the specified name in the workload’s namespace. If not found, it will then check the kuma-system namespace for the same ConfigMap.

• If the workload is not annotated, or the ConfigMap specified in the annotation is not found, Kuma will search for the resource named in the transparentProxy.configMap.name setting. It first looks in the workload’s namespace, and if not found, it checks the kuma-system namespace.

Custom ConfigMap name

The name of the ConfigMap is defined by the transparentProxy.configMap.name setting. By default, this name is set to kuma-transparent-proxy-config.

To apply a custom name for all workloads, you can modify the transparentProxy.configMap.name setting during installation:

kumactl install control-plane \
  --set "transparentProxy.configMap.name=custom-name" \
  | kubectl apply -f -

helm install --create-namespace --namespace kuma-system \
  --set "transparentProxy.configMap.name=custom-name" \
  kuma kuma/kuma

To use a different resource for specific workloads, apply the traffic.kuma.io/transparent-proxy-configmap-name annotation to those workloads. For example:

kubectl annotate pods <pod_name> "traffic.kuma.io/transparent-proxy-configmap-name=custom-name"

This allows flexibility, enabling different workloads to use distinct ConfigMap settings while maintaining a global default.

Method 3: Kubernetes annotations

Kubernetes annotations can be applied to individual workloads to modify the transparent proxy configuration. These annotations allow fine-tuning of specific behaviors for a single workload without affecting others.

Below you can find a list of annotations that can be used to configure specific transparent proxy settings.

• traffic.kuma.io/exclude-inbound-ips

• traffic.kuma.io/exclude-inbound-ports

• traffic.kuma.io/exclude-outbound-ips

• traffic.kuma.io/exclude-outbound-ports

• traffic.kuma.io/exclude-outbound-ports-for-uids

• traffic.kuma.io/drop-invalid-packets

• traffic.kuma.io/iptables-logs

• kuma.io/transparent-proxying-ebpf

• kuma.io/transparent-proxying-ebpf-bpf-fs-path

• kuma.io/transparent-proxying-ebpf-cgroup-path

• kuma.io/transparent-proxying-ebpf-tc-attach-iface

• kuma.io/transparent-proxying-ebpf-instance-ip-env-var-name

• kuma.io/transparent-proxying-ebpf-programs-source-path

• kuma.io/transparent-proxying-ip-family-mode

The following annotations also affect the configuration, but their values are automatically managed by Kuma and cannot be manually adjusted. Values of these annotations will always match those specified in the Control Plane Runtime Configuration. For more details, refer to Settings restricted to control plane runtime configuration.

• kuma.io/sidecar-uid

• kuma.io/transparent-proxying-inbound-port

• kuma.io/transparent-proxying-outbound-port

The following annotation indirectly adjusts the transparent proxy configuration by configuring other components, resulting in changes to the transparent proxy behavior. While its value is automatically set by Kuma, you can override it by manually providing a value.

• kuma.io/application-probe-proxy-port

  This annotation defines the port (default: 9001) to be excluded from incoming traffic redirection, similar to using the traffic.kuma.io/exclude-inbound-ports annotation or other configuration methods.

  For more information, refer to:

  • The Kubernetes section in the Service Health Probes documentation.
  • The Introduction to Application Probe Proxy and deprecation of Virtual Probes section in the Upgrade Instructions.

The following annotations differ from others mentioned earlier as they are related to the transparent proxy, but do not directly or indirectly configure specific individual settings.

• kuma.io/transparent-proxying

  This annotation is automatically applied to workloads with injected sidecar containers. In Kubernetes environments, the transparent proxy is mandatory meaning you cannot disable it. Any attempt to explicitly disable it using this annotation will trigger a warning and have no effect. Therefore, the value of this annotation will always be set to true.

• traffic.kuma.io/transparent-proxy-config

  This annotation is automatically applied in environments using Kuma CNI instead of init containers. It contains the final configuration for installing the transparent proxy. Manually modifying or setting this annotation has no effect, regardless of whether Kuma CNI is used. Its primary purpose is to efficiently pass the transparent proxy configuration between Kubernetes workloads and the Kuma CNI, which handles the actual installation of the transparent proxy for those workloads.

• traffic.kuma.io/transparent-proxy-configmap-name

  This annotation lets you specify a custom name for the ConfigMap that holds the transparent proxy configuration when the Configuration in ConfigMap ^{(experimental)} feature is enabled. For more details, refer to the Custom ConfigMap name section.

Automatically applied annotations

Certain annotations are automatically added to workloads with injected sidecar containers, regardless of whether they are explicitly defined. These annotations reflect the final values used to configure the transparent proxy. For settings that can be manually specified, these annotations will still be applied, even if not explicitly provided, using values from the Control Plane Runtime Configuration.

The automatically applied annotations include:

• kuma.io/transparent-proxying
• kuma.io/sidecar-uid
• kuma.io/transparent-proxying-inbound-port
• kuma.io/transparent-proxying-outbound-port
• kuma.io/builtin-dns
• kuma.io/builtin-dns-port
• kuma.io/application-probe-proxy-port
• kuma.io/transparent-proxying-ebpf
• kuma.io/transparent-proxying-ebpf-bpf-fs-path
• kuma.io/transparent-proxying-ebpf-cgroup-path
• kuma.io/transparent-proxying-ebpf-tc-attach-iface
• kuma.io/transparent-proxying-ebpf-instance-ip-env-var-name
• kuma.io/transparent-proxying-ebpf-programs-source-path
• kuma.io/transparent-proxying-ip-family-mode
• traffic.kuma.io/transparent-proxy-config (only in environments using Kuma CNI)

These annotations ensure that the proper configuration is automatically applied to each workload, aligning with the global and per-workload settings.

Order of precedence

When using multiple configuration methods, it’s important to understand the order in which they are applied to avoid conflicts and ensure the correct settings are used.

1. Defaults (refer to Transparent Proxy Configuration Reference for exact values)

2. Control Plane Runtime Configuration

3. Configuration in ConfigMap ^{(experimental)}

4. Kubernetes Annotations