On May 25 2019, a security relevant bug has been reported to us via the documented security disclosure channel. Thanks to l14n for the excellent bug report! It was soon identified that multiple vendors are affected by this vulnerability. This lead to an embargo period which is being lifted today.

The bug allows, under certain circumstances, to bypass network security policies. See below for details on the vulnerability and the mitigation.

Who is affected?: Users operating Cilium in encapsulation mode (VXLAN or Geneve) while hosting untrusted workloads with an egress policy that allows pods to emit UDP encapsulation traffic to other worker nodes.

The vulnerability is being tracked by CVE-2019-13119

We are releasing Cilium 1.5.4, 1.4.5, and 1.3.7 to fix the security vulnerability.

Vulnerability details

An endpoint can emit encapsulation traffic to another node running Cilium and given certain conditions are met, that node will accept the encapsulated traffic and route it. This can bypass egress security policies. If the endpoint is able to guess security identities correctly, the endpoint can also impersonate other security identities.

Conditions that must be met:

• Cilium must be running in encapsulation mode. This is the default configuration mode.

• The egress policy of the endpoint must allow UDP on the configured encapsulation port. This port is different depending on whether VXLAN or Geneve is being used as the encapsulation protocol. Alternatively, if the endpoint has access to an external LB such as a NodePort which redirects to the configured UDP encapsulation port on any node, the policy must allow for this. Redirection to a UDP encapsulation port using a CluserIP or headless service is not affected as the load-balancing decision happens before egress policy.

• The endpoint must be aware of at least one node IP or be aware of an external LB that redirects to a node IP.

• Masquerading of egress traffic to worker node IP must be enabled, if masquerading is disabled, the remote tunnel will reject the traffic. This means that known or guessed worker node IP must be a node IP which is considered outside of the cluster so masquerading is performed.

• The endpoint must guess a Cilium security identity or be aware of well-known security identities which is allowed by the ingress policy of the destination service. Even if no ingress policy is used, the guessed security identity must be valid, i.e. it must be a well-known security identity or an security identity that is actively in-use by a pod.

  In the reported attack scenario, the reporter has used tcpdump on one of the nodes to derive the security identity by inspecting the wire and replaying encapsulation traffic. Cilium provides support for node level encryption. The attack assumes that no encryption is performed. However, the security identity is a construct intended for scale. The 32 bit numeric space used to group workloads with identical security identity together can be subject to brute-force attacks.

Mitigation

In order to mitigate this, the following fix is applied:

Any UDP traffic emitted by an endpoint to one of the supported encapsulation ports is now dropped with a distinct drop reason:

xx drop (Encapsulation traffic is prohibited) flow 0xcab0685a to endpoint 0, identity 3087->2: 10.16.209.20:39258 -> 192.168.122.124:8472 udp

As Cilium performs load-balancing (enabled by default) before the egress policy enforcement is applied, any traffic subject to a Kubernetes ClusterIP service that load-balances to an encapsulation port is also rejected. This is not the case if you are using kube-proxy as the load-balancing happens after the standard egress policy enforcement point.

Questions

Thanks again to l14n for disclosing the vulnerability to us via the private security mailing list so we could provide a bugfix relese before disclusing the details publicly. If you have any questions, feel free to reach out to us on Slack.