Careful!

You are browsing documentation for a version of Kuma that is not the latest release.

Looking for even older versions? Learn more.

Customizing Transparent Proxy Configuration in Kubernetes Mode

The default transparent proxy configuration works well for most scenarios, but there are cases where adjustments are needed. This guide explains the various methods available for modifying the configuration, along with their limitations and recommendations on when to use each one. While these methods can be used individually, they can also be combined when necessary.

It’s best to stick to one method whenever possible. Using more than one can make things more complicated and harder to troubleshoot, as it may not be clear where each setting comes from. If you need to combine methods, check the Order of Precedence to see what the final configuration will look like based on the priority of each setting.

Terminology overview

Workload: In this guide, workload refers to an application running in a Kubernetes cluster, usually represented by a Pod. Kubernetes resources like Deployment, ReplicaSet, and StatefulSet are used to define and manage these workloads, resulting in one or more Pods where the application actually runs.
ConfigMap: In this guide, ConfigMap refers to the Kubernetes ConfigMap resource. It’s used to store configuration data as key-value pairs that can be easily accessed by other resources in the cluster, such as Pods, Deployments, and StatefulSets.

Method 1: Control plane runtime configuration

The control plane runtime configuration is a section of Kuma’s configuration that contains important settings for how the control plane operates, including options for the transparent proxy and other key components.

For more details, see the Control Plane Configuration Reference under runtime.kubernetes.

Some transparent proxy settings can be adjusted here, and for certain settings, this is the only place they can be modified.

Below is a subset of the configuration focused on transparent proxy settings, with default values and corresponding environment variables for each field. For further details, see the Modifying control plane runtime configuration section.

runtime:
  kubernetes:
    injector:
      cniEnabled: false # KUMA_RUNTIME_KUBERNETES_INJECTOR_CNI_ENABLED
      applicationProbeProxyPort: 9001 # KUMA_RUNTIME_KUBERNETES_APPLICATION_PROBE_PROXY_PORT
      sidecarContainer:
        uid: 5678 # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_UID
        ipFamilyMode: dualstack # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_IP_FAMILY_MODE
        redirectPortInbound: 15006 # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_REDIRECT_PORT_INBOUND
        redirectPortOutbound: 15001 # KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_REDIRECT_PORT_OUTBOUND
      sidecarTraffic:
        excludeInboundPorts: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_INBOUND_PORTS
        excludeOutboundPorts: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_PORTS
        excludeInboundIPs: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_INBOUND_IPS
        excludeOutboundIPs: [] # KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS
      builtinDNS:
        enabled: true # KUMA_RUNTIME_KUBERNETES_INJECTOR_BUILTIN_DNS_ENABLED
        port: 15053 # KUMA_RUNTIME_KUBERNETES_INJECTOR_BUILTIN_DNS_PORT
      ebpf:
        enabled: false # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_ENABLED
        instanceIPEnvVarName: INSTANCE_IP # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_INSTANCE_IP_ENV_VAR_NAME
        bpffsPath: /sys/fs/bpf # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_BPFFS_PATH
        cgroupPath: /sys/fs/cgroup # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_CGROUP_PATH
        tcAttachIface: "" # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_TC_ATTACH_IFACE
        programsSourcePath: /tmp/kuma-ebpf # KUMA_RUNTIME_KUBERNETES_INJECTOR_EBPF_PROGRAMS_SOURCE_PATH

Settings restricted to control plane runtime configuration

Some transparent proxy settings can only be changed through the control plane’s runtime configuration because they are shared with other Kuma components. These settings handle essential tasks like creating Dataplane resources for workloads and setting up the kuma-dp sidecar container alongside them. For example, the DNS port used to redirect traffic is shared between the transparent proxy and the kuma-dp run command in the sidecar container. Keeping these settings consistent across workloads prevents misconfigurations. These settings are:

runtime.kubernetes.injector.sidecarContainer.redirectPortInbound
runtime.kubernetes.injector.sidecarContainer.redirectPortOutbound
runtime.kubernetes.injector.sidecarContainer.uid
runtime.kubernetes.injector.builtinDNS.enabled
runtime.kubernetes.injector.builtinDNS.port

Modifying control plane runtime configuration

To modify the configuration, you have two main options:

Store the desired settings in the kuma-control-plane-config ConfigMap:

This method is recommended because it simplifies future modifications. Rather than reinstalling Kuma or manually updating its Deployment to change environment variables, you can just update the ConfigMap and restart the kuma-control-plane Pod to apply the changes.

You can provide the full runtime configuration in the kuma-control-plane-config ConfigMap within the kuma-system namespace.

Here’s an example that excludes specific outbound IPs from transparent proxy redirection:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: kuma-control-plane-config
  namespace: kuma-system
data:
  config.yaml: |
    runtime:
      kubernetes:
        injector:
          sidecarTraffic:
            excludeOutboundIPs: [10.1.0.254, 172.10.1.254]
```
Instead of manually updating the ConfigMap in the cluster, you can specify the configuration it will store during the Kuma installation. Follow these steps:
1. Create a values.yaml file with the following content:
```
controlPlane:
  config: |
    runtime:
      kubernetes:
        injector:
          sidecarTraffic:
            excludeOutboundIPs: [10.1.0.254, 172.10.1.254]
```
  Make sure the content under controlPlane.config is a string representation of the control plane configuration.
2. Install Kuma using kumactl or Helm:
  - kumactl
  - Helm
  kumactl install control-plane --values values.yaml | kubectl apply -f -
  Before using Kuma with Helm, ensure that you’ve followed these steps to configure your local Helm repository.
  
  helm install \ --create-namespace \ --namespace kuma-system \ --values values.yaml \ kuma kuma/kuma
If you’re deploying Kuma using Helm, you can place the configuration in a separate file and pass it during installation with the --set-file flag.

For example, create a cp-conf.yaml file with the following content:
```
runtime:
  kubernetes:
    injector:
      sidecarTraffic:
        excludeOutboundIPs: [10.1.0.254, 172.10.1.254]
```
Then, during Helm installation, run:
```
helm install \
  --create-namespace \
  --namespace kuma-system \
  --set-file "controlPlane.config=cp-conf.yaml" \
  kuma kuma/kuma
```
This will have the same effect as using --values values.yaml, as described earlier.
Configure specific settings through environment variables:

You can also set individual configuration values using environment variables. For example, to exclude specific outbound IPs from the transparent proxy, update the kuma-control-plane Deployment by setting the environment variable KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS with the desired IP addresses, separated by commas. For instance: 10.1.0.254,172.10.1.254.

Manually editing Kubernetes manifests for Kuma is strongly discouraged and should only be done for testing in non-production environments. Making manual changes can increase the risk of misconfiguration and may cause unexpected issues. For dynamic configuration updates without reinstalling Kuma, use the kuma-control-plane-config ConfigMap as described above.

Setting environment variables during the initial installation, as shown below, is safe for any environment. While the ConfigMap is the preferred method, environment variables are also a reliable option for setup.

To apply environment variables to Kuma’s control plane automatically, configure them during installation using the controlPlane.envVars setting. For example:
- kumactl
- helm
kumactl install control-plane \ --set "controlPlane.envVars.KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS=10\.1\.0\.254\,172\.10\.1.254" \ | kubectl apply -f -
Before using Kuma with helm, please follow these steps to configure your local helm repo.
helm install --create-namespace --namespace kuma-system \ --set "controlPlane.envVars.KUMA_RUNTIME_KUBERNETES_SIDECAR_TRAFFIC_EXCLUDE_OUTBOUND_IPS=10\.1\.0\.254\,172\.10\.1.254" \ kuma kuma/kuma

Overriding list-based configuration settings

For list-based settings like runtime.kubernetes.injector.sidecarTraffic.excludeOutboundPorts, only the value from the highest-precedence method will be applied, and values from other methods will not be combined.

For example, if runtime.kubernetes.injector.sidecarTraffic.excludeOutboundPorts is set in the control plane runtime configuration but an annotation like traffic.kuma.io/exclude-outbound-ports is applied to a specific workload, the annotation will override the control plane setting. This means the transparent proxy will exclude only the ports specified in the annotation.

See the Order of Precedence section to understand how different methods affect the final configuration applied.

Method 2: Kubernetes annotations

Kubernetes annotations can be applied to individual workloads to modify the transparent proxy configuration. These annotations allow fine-tuning of specific behaviors for a single workload without affecting others.

Below you can find a list of annotations that can be used to configure specific transparent proxy settings.

traffic.kuma.io/exclude-inbound-ips
traffic.kuma.io/exclude-inbound-ports
traffic.kuma.io/exclude-outbound-ips
traffic.kuma.io/exclude-outbound-ports
traffic.kuma.io/exclude-outbound-ports-for-uids
traffic.kuma.io/drop-invalid-packets
traffic.kuma.io/iptables-logs
kuma.io/transparent-proxying-ebpf
kuma.io/transparent-proxying-ebpf-bpf-fs-path
kuma.io/transparent-proxying-ebpf-cgroup-path
kuma.io/transparent-proxying-ebpf-tc-attach-iface
kuma.io/transparent-proxying-ebpf-instance-ip-env-var-name
kuma.io/transparent-proxying-ebpf-programs-source-path
kuma.io/transparent-proxying-ip-family-mode

The following annotations also affect the configuration, but their values are automatically managed by Kuma and cannot be manually adjusted. Values of these annotations will always match those specified in the Control Plane Runtime Configuration. For more details, refer to Settings restricted to control plane runtime configuration.

kuma.io/sidecar-uid
kuma.io/transparent-proxying-inbound-port
kuma.io/transparent-proxying-outbound-port

The following two annotations are deprecated and will be removed in future releases. They are provided here for reference only, and we strongly advise against using them.

kuma.io/builtin-dns
kuma.io/builtin-dns-port

The following annotation differs from others mentioned earlier as they are related to the transparent proxy, but do not directly or indirectly configure specific individual settings.

kuma.io/transparent-proxying

This annotation is automatically applied to workloads with injected sidecar containers. In Kubernetes environments, the transparent proxy is mandatory meaning you cannot disable it. Any attempt to explicitly disable it using this annotation will trigger a warning and have no effect. Therefore, the value of this annotation will always be set to true.

Automatically applied annotations

Certain annotations are automatically added to workloads with injected sidecar containers, regardless of whether they are explicitly defined. These annotations reflect the final values used to configure the transparent proxy. For settings that can be manually specified, these annotations will still be applied, even if not explicitly provided, using values from the Control Plane Runtime Configuration.

The automatically applied annotations include:

kuma.io/transparent-proxying
kuma.io/sidecar-uid
kuma.io/transparent-proxying-inbound-port
kuma.io/transparent-proxying-outbound-port
kuma.io/builtin-dns
kuma.io/builtin-dns-port
kuma.io/application-probe-proxy-port
kuma.io/transparent-proxying-ebpf
kuma.io/transparent-proxying-ebpf-bpf-fs-path
kuma.io/transparent-proxying-ebpf-cgroup-path
kuma.io/transparent-proxying-ebpf-tc-attach-iface
kuma.io/transparent-proxying-ebpf-instance-ip-env-var-name
kuma.io/transparent-proxying-ebpf-programs-source-path
kuma.io/transparent-proxying-ip-family-mode

These annotations ensure that the proper configuration is automatically applied to each workload, aligning with the global and per-workload settings.

Order of precedence

When using multiple configuration methods, it’s important to understand the order in which they are applied to avoid conflicts and ensure the correct settings are used.

Defaults
Control Plane Runtime Configuration
Kubernetes Annotations