# Cilium

> Cilium is an open source, CNCF graduated project that uses eBPF to deliver networking, security, and observability for Kubernetes and cloud native platforms. It provides a high-performance Container Network Interface (CNI), an eBPF-based kube-proxy replacement, identity-based network policy, transparent encryption, multi-cluster connectivity, a sidecar-free service mesh, and deep network observability through Hubble. Cilium was created by Isovalent part of Cisco and runs in production at Google, Amazon, Microsoft Azure, Adobe, Bell Canada, The New York Times, Datadog, Bloomberg, TikTok, Alibaba Cloud, and thousands of other companies. Official site: https://cilium.io. Documentation: https://docs.cilium.io

**What is Cilium**

Cilium is a cloud native networking and security platform for Kubernetes built on eBPF, a Linux kernel technology that runs sandboxed programs inside the kernel without changing kernel source or loading modules. By programming the dataplane in eBPF, Cilium replaces iptables-based packet processing with a more scalable, lower-latency, identity-aware dataplane.

Cilium operates at the network (L3/L4) and application (L7) layers. It assigns a security identity to each Kubernetes workload, enforces policy based on identity rather than IP address, and provides visibility into every network flow. The Hubble sub project adds flow logs, metrics, and a service dependency map; the Tetragon sub project adds eBPF-based runtime security and enforcement.

Cilium graduated in the Cloud Native Computing Foundation (CNCF) in 2023 and is the default or recommended CNI for many managed Kubernetes platforms and distributions, including Google GKE Dataplane V2, Microsoft AKS, Digital Ocean Kubernetes, Red Hat OpenShift, kind, and more.

Cilium is often compared to other CNIs such as Calico and Flannel. It differs primarily in its eBPF dataplane, which enables identity-aware policy, scalability, and observability that IP-table-based CNIs can't provide natively. For service mesh, Cilium's sidecar-free approach and eBPF dataplane separate it from others like Istio or Linkerd.

**Core capabilities**

- **Container networking (CNI):** Kubernetes pod networking with an eBPF dataplane, IPAM, overlay (VXLAN/Geneve) and native/direct routing modes, ENI/Azure delegated IPAM, and BIG TCP and netkit for high throughput.
- **kube-proxy replacement:** eBPF-based service load balancing for ClusterIP, NodePort, and LoadBalancer services, with Maglev consistent hashing, Direct Server Return (DSR), and XDP acceleration, eliminating iptables scaling bottlenecks.
- **Network policy and security:** identity-based microsegmentation with Kubernetes NetworkPolicy and CiliumNetworkPolicy (CNP) at L3/L4/L7, DNS/FQDN-aware policy, and host firewall for zero-trust networking.
- **Observability with Hubble:** identity-aware L3/L4/L7 network flow logs, DNS visibility, Prometheus and OpenTelemetry metrics, Grafana dashboards, and a real-time service dependency map.
- **Transparent encryption:** WireGuard and IPsec pod-to-pod in-transit encryption with no application changes and ztunnel for mTLS, supporting compliance and FIPS requirements.
- **Multi-cluster (Cluster Mesh):** cross-cluster service discovery, global services, and pod-to-pod connectivity, observability, and security across clusters, regions, and clouds.
- **Sidecar-free service mesh:** eBPF and Envoy-based L7 traffic management and mutual TLS (mTLS) without per-pod sidecar proxies, reducing latency and resource overhead.
- **Runtime security (Tetragon):** eBPF-based process execution, file access, and network monitoring with kernel-level enforcement for threat detection and forensics.

---

This is the **full** version (`llms-full.txt`). [llms.txt](https://cilium.io/llms.txt) is the map and this file is the territory: it contains one section per page linked from llms.txt, in the same order. Content pages on cilium.io are inlined in full, the Blog entry expands to the full text of every post published on the site plus an index of external posts, and map entries without substantive on-site text keep their one-line description. For deep technical specifications, see the official documentation at https://docs.cilium.io.

## Use cases

### High Performance Cloud Native Networking (CNI)

URL: https://cilium.io/use-cases/cni
Category: Networking

_Enhance the speed and efficiency of your Kubernetes and cloud native networks_

**How can I have scalable and consistent networking across clouds?**

There are dozens of CNIs available for Kubernetes but, their features, scale, and performance vary greatly. Many of them rely on a legacy technology (iptables) that cannot handle the scale and churn of Kubernetes environments leading to increased latency and reduced throughput. Most CNIs also only offer support for L3/L4 Kubernetes network policy but little beyond. Many Cloud Provider have their own custom CNIs which results in operational complexity for customers operating in multi-cloud environments.

#### High Performance Cloud Native Networking CNI

Cilium’s control and data plane has been built from the ground up for large-scale and highly dynamic cloud native environments where 100s and even 1000s of containers are created and destroyed within seconds. Cilium’s control plane is highly optimized, running in Kubernetes clusters with 1,000s of nodes and 100K pods. Cilium’s data plane uses eBPF for efficient load-balancing and incremental updates, avoiding the pitfalls of large iptables rulesets.

#### Scalability

Cilium is built to scale. Whether you're running a few nodes or managing a cluster with thousands, Cilium can handle it. Cilium’s eBPF-powered networking is optimized for large scale operations. This means you can grow your operations without worrying about the network becoming a bottleneck.

#### Network Security

Cilium doesn't just excel in performance, it also brings robust security features. With identity-based security that goes beyond traditional IP address-based ACLs, Cilium provides fine-grained policy enforcement. This allows you to control who can talk to whom, reducing the attack surface and helping secure your cloud native applications.

#### Integration

Cilium integrates seamlessly with Kubernetes, providing networking and security through a CNI plugin. Cilium has been tested, validated, and optimized across multiple clouds and Kubernetes distributions. All major cloud providers have already standardized on Cilium for cloud native networking and security needs with Kubernetes. Managed Kubernetes offerings from Google Cloud, AWS, Azure, Alibaba, DigitalOcean, and several smaller platforms like Civo, Linode, etc, leverage Cilium for their CNI.

_Adopters & resources referenced on this page:_ AWS picks Cilium for Networking & Security on EKS Anywhere; Cilium in Anthos and Google Kubernetes Engine (GKE) as Dataplane V2; Azure CNI Powered by Cilium in Azure Kubernetes Service; Alibaba Cloud turns to Cilium for improved Network Performance and Scalability; SysEleven Picks Cilium as the Preferred Networking Solution for their Platform; CNI Benchmark: Understanding Cilium Network Performance.

### Kube-proxy Replacement

URL: https://cilium.io/use-cases/kube-proxy
Category: Networking

_Enhanced networking speed and efficiency for your Kubernetes clusters_

**Liberating Kubernetes From kube-proxy and IPtables**

IPtables and Netfilter are the two foundational technologies of kube-proxy for implementing the Service abstraction. They carry legacy accumulated over 20 years of development grounded in more traditional networking environments that are typically far more static than your average Kubernetes cluster. In the age of cloud native, they are no longer the best tool for the job, especially in terms of performance, reliability, scalability, and operations.

#### Kubernetes Without Kube-Proxy

Cilium’s control and data plane has been built from the ground up for large-scale and highly dynamic cloud native environments where 100s and even 1000s of containers are created and destroyed within seconds. Cilium’s control plane is highly optimized, running in Kubernetes clusters of up to 5K nodes and 100K pods. Cilium’s data plane uses eBPF for efficient load-balancing and incremental updates, avoiding the pitfalls of large IPtables rulesets.

#### Seamless Integration and Upgrade

If you already have kube-proxy running as a DaemonSet, transitioning to Cilium is a breeze. Replacing kube-proxy with Cilium is a straightforward process, as Cilium provides a Kubernetes-native implementation that is fully compatible with the Kubernetes API. Existing Kubernetes applications and configurations can continue to work seamlessly with Cilium.

#### Unlock Advanced Configuration Modes

Cilium's kube-proxy replacement offers advanced configuration modes to cater to your specific needs. Features like client source IP preservation ensure that your service connections remain intact, while Maglev Consistent Hashing enhances load balancing and resiliency. With support for Direct Server Return (DSR) and Hybrid DSR/SNAT modes, you can optimize traffic routing and improve performance.

_Adopters & resources referenced on this page:_ Post Finance picks Cilium for Cloud Native Networking; SmartNews leverages Cilium to improve performance and scaling; Service Load-Balancing Without Kube-Proxy on Kubermatic; Kakao improves network performance and lowers costs with Cilium; Why is the kernel community replacing iptables with BPF?.

### Layer 4 Load Balancer

URL: https://cilium.io/use-cases/load-balancer
Category: Networking

_High performance load balancing with low overhead_

**How can I implement efficient L4 load balancing with low overhead and cost?**

Configuring and managing load balancing into your cluster can be challenging due to the complexity involved in setting up connectivity and synchronization between the clusters and the outside world. Traditional hardware load balancers can be very costly while software load balancers may not provide the performance you need. External-to-Pod (North-South) LB also typically requires additional tooling, adding more complexity, cost, and overhead.

#### XDP and eBPF powered scalable Load Balancing and Ingress

Cilium can attract traffic with BGP and accelerate it leveraging XDP and eBPF. Together these technologies provide a very robust and secure implementation of Load Balancing. Cilium and eBPF operate at the kernel layer. With this level of context intelligent decisions can be made about how to connect different workloads whether on the same node or between clusters. With eBPF and XDP Cilium enables significant improvements in latency and performance. Cilium's standalone load balancer offers a high-performance LB, providing huge throughput gains at a reduced CPU overhead.

#### Cilium Standalone Layer 4 Load Balancer XDP

Cilium's high performance, robust load balancing implementation is tuned for the scale and churn of cloud native environments. You can replace expensive legacy boxes in your network with Cilium as a standalone load balancer. This unlocks the potential of DSR and Maglev for handling north/south traffic in on-premises environments without requiring Kubernetes to manage the network border.

_Adopters & resources referenced on this page:_ Efficiently handling production traffic with Cilium Standalone Layer 4 Load Balancer XDP; Software L4 Load Balancing for Kubernetes Services at Yahoo.

### Advanced Network Policy

URL: https://cilium.io/use-cases/network-policy
Category: Security

_Maintain identity based policies effectively at scale_

**How can I implement granular security policies when IPs change quickly?**

Kubernetes network policies provide an application-centric construct for defining security policies at L3/L4 level. One of the primary challenges is how to effectively enforce security policies when traditional IP rules don't apply. Modern systems often churn IPs dynamically making it difficult to rely entirely on TCP/UDP ports and IP addresses for scaling security policies.

#### Application and DNS Aware Policies with Cilium

Cilium implements Kubernetes Network Policies for L3/L4 level and extends with L7 policies for fine grained API-level security for common protocols such as HTTP, Kafka, gRPC, etc. For example, the endpoint with label role=frontend can only perform the REST API call GET /userdata/[0-9]+, all other API interactions with role=backend are restricted.

#### Scaling policies with Identities not IPs

Cilium decouples security from network addressing using workload identity derived from labels and metadata, allowing for more flexible and efficient scaling without constant security rule updates.

#### Policy visualization and editing

Cilium provides a simple and intuitive network policy editor UI easing the cognitive overhead of writing network policies. It can often be painful to get the YAML syntax and formatting right when implementing network policies. There are many subtleties in the behavior of the network policy specification (e.g. default allow/deny, namespacing, wildcarding, rules combination, etc) that can result in misconfiguration.

#### Multi-cluster Policies

Cluster Mesh, Cilium's multi-cluster implementation features Network policy enforcement spanning multiple clusters. The same policy enforcement you are familiar with from a single cluster simply expands and works across multiple clusters.

#### Cluster-wide Policies

Cilium also features cluster wide policies which are non-namespaced and cluster scoped via the extended CiliumClusterwideNetworkPolicy CRD. Using cluster-wide policies, administrators can enforce consistent policies across all namespaces, simplifying network management.

_Adopters & resources referenced on this page:_ Scaling for the future with Cilium; Observability for a highly available multi cluster environment with Hubble; How ClickHouse is Using Cilium to Implement Efficient Network Policies; Self-service, Zero Trust Network Security; Enforcing Network Policies for Host Processes via eBPF; Migrating to Cilium for Better Networking, Visibility and Security; Implementing Zero Trust Security with Cilium.

### Transparent Encryption

URL: https://cilium.io/use-cases/transparent-encryption
Category: Security

_Encryption without operational headache_

**How can I encrypt traffic on my clusters while minimizing operational overhead?**

Many compliance frameworks require encryption, but Kubernetes lacks native pod-to-pod encryption. Two common solutions to this problem are embedding encryption within the application or using a service mesh. Embedding encryption within the app is complex and requires application and security expertise. On the other hand, most service mesh implementations are very complex and challenging to manage and operate.

#### What does Cilium provide?

Cilium provides a straightforward solution for enabling the encryption of all node-to-node traffic with just one switch, no application changes or additional proxies. Cilium features automatic key rotation with overlapping keys, efficient datapath encryption through in-kernel IPsec or WireGuard, and can encrypt all traffic, including non-standard traffic like UDP. Simply configuring all nodes across all clusters with a common key and all communication between nodes is automatically encrypted.

_Adopters & resources referenced on this page:_ Achieving HIPAA compliance with Cilium’s transparent encryption; Seamless Network Security and Privacy with Cilium.

### Cluster Mesh

URL: https://cilium.io/use-cases/cluster-mesh
Category: Networking

_Unleashing the power of multi-cluster networking with Cilium Cluster Mesh_

**Seamless connectivity for multiple Kubernetes clusters**

Multi-cluster Kubernetes setups are often adopted for reasons like fault isolation, scalability, and geographical distribution. This approach can lead to networking complexities. With such multi-cluster setups, traditional networking models struggle with service discovery, network segmentation, policy enforcement, and load balancing across clusters. Additionally, managing security protocols and policies across multiple environments can be a challenging endeavour due to the distributed nature of services.

#### Cluster Mesh, Cilium's Multi-Cluster Networking Implementation

Cilium Cluster Mesh allows you to connect the networks of multiple clusters in such as way that pods in each cluster can discover and access services in all other clusters of the mesh, provided all the clusters run Cilium as their CNI. This allows effectively joining multiple clusters into a large unified network, regardless of the Kubernetes distribution or location each of them is running.

#### High Availability and Fault Tolerance

Cluster Mesh enhances your services high availability and fault tolerance. It supports the operation of Kubernetes clusters in multiple regions or availability zones. If resources become temporarily unavailable, are misconfigured in one cluster, or offline for upgrades, it enables failover to other clusters, ensuring your services remain accessible at all times.

#### Transparent Service Discovery

Cluster Mesh automates service discovery across your Kubernetes clusters. Using standard Kubernetes services, it automatically merges services with identical names and namespaces across clusters into a global service. This means your applications can discover and interact with services, irrespective of the cluster they reside in, greatly simplifying cross-cluster communication.

#### Effortless Pod IP Routing

Cluster Mesh is able to handle Pod IP routing across multiple Kubernetes clusters at native performance. By using either tunneling or direct-routing, it circumvents the need for any gateways or proxies. This allows your pods to communicate across clusters seamlessly, enhancing the overall efficiency of your microservice architecture.

#### Shared Services Across Clusters

Cluster Mesh enables sharing of services such as secrets management, logging, monitoring, or DNS between all clusters. This reduces operational overhead, simplifies management, and maintains isolation between tenant clusters.

#### Uniform Network Policy Enforcement

Cluster Mesh extends Cilium's Layer 3-7 network policy enforcement to all clusters in the mesh. It standardizes the application of network policies, ensuring a consistent security approach across your entire Kubernetes deployment, irrespective of the number of clusters involved.

_Adopters & resources referenced on this page:_ Most mature multi cluster application management technology; Transforming Networking with Cilium at Ecco; Multi-cluster networking with Cilium at Form3; Building ClickHouse Cloud From Scratch in a Year; How Wildlife Studios built a Global Multi Cluster Gaming Infrastructure with Cilium; Strengthening Security Across Distributed Kubernetes Clusters; High Performance Data Backend for Your AI/ML with MinIO and Cilium Cluster Mesh.

### Cilium Service Mesh

URL: https://cilium.io/use-cases/service-mesh
Category: Networking

_Revolutionizing networking and simplify operations_

**Performance Meets Simplicity**

Traditional service meshes, despite their benefits, can present significant challenges. These include the complexity and error-prone nature of IP and port-based network policies, performance overhead due to their proxy-based architecture, limited granularity in visibility of service-to-service communication, interoperability issues with existing infrastructure, scalability challenges as the number of services and traffic volumes increase, and operational and resource overhead.

#### Cilium's Next-Generation Service Mesh

Cilium Service Mesh redefines traditional service mesh frameworks by integrating the mesh layer directly into the kernel using eBPF, thus eliminating the need for sidecar proxies. It manages connectivity at both the networking and application protocol layers, handling protocols like IP, TCP, UDP, HTTP, Kafka, gRPC, and DNS with greater efficiency.

#### Superior Networking with eBPF

At the heart of Cilium is eBPF, a revolutionary technology built into the Linux kernel. With eBPF, Cilium delivers lightning-fast, efficient, and scalable networking. This bypasses the performance drawbacks of traditional proxies, enabling direct and efficient communication between your services.

#### Control Plane Options

Cilium Service Mesh gives users the choice of control plane options for the ideal balance of complexity and richness, from simpler options such as Ingress and Gateway API to richer options with Istio, to the full power of Envoy via the Envoy CRD.

#### Sidecar-free Option

With Cilium Service Mesh, users now have the choice to run a service mesh with sidecars or without them, based on their specific requirements and constraints. This flexibility reduces the complexity and overhead impact of sidecars.

#### Hands-On: eCHO Playlist

Whether you’re bootstrapping a service mesh from the ground up or adopting Cilium’s eBPF-powered service mesh later in your service mesh journey, the eCHO Recaps: Cilium Service Mesh playlist shows the entire timeline starting from the first demo and hands-on commands. Follow each episode as to see how service mesh includes ingress and Gateway API traffic management, all on real clusters with CLI walk throughs you can replay on-demand.

_Adopters & resources referenced on this page:_ How Roche Manages Network Connectivity for 1000+ Edge Clusters; Strengthening Security Across Distributed Kubernetes Clusters; Revolutionizing Mesh Layers: Transitioning from Istio to Cilium at the New York Times; Simplifying Service Mesh with Encryption from Cilium.

### Gateway API

URL: https://cilium.io/use-cases/gateway-api
Category: Networking

_Robust Kubernetes Ingress for traffic management_

**Evolving Past Ingress**

Gateway API addresses the limitations of the Kubernetes Ingress API, which has traditionally been used to route traffic into Kubernetes clusters. While the Ingress API supports basic routing based on path and host rules, it lacks support for advanced routing features, only supports HTTP and HTTPS traffic, doesn’t separate out user/operator concerns, and can lead to inconsistencies due to vendor-specific annotations. Gateway API overcomes these constraints, providing a more robust, extensible, and role-oriented approach to traffic engineering.​

#### Advanced Routing and Protocol Support

Cilium’s Gateway API implementation provides a fully conformant implementation of Kubernetes Ingress and offers advanced routing capabilities that go beyond the limitations of the Ingress API. It supports features such as traffic splitting, header modification, and URL rewriting. Additionally, it extends beyond HTTP and HTTPS traffic, with support of other protocols such as TCP, UDP, and gRPC. This allows for a more flexible and sophisticated routing strategies.

#### Role-Oriented and Extensible

Cilium’s Gateway API implementation was designed with different operational roles in mind, such as Infrastructure Providers, Cluster Operators, and Application Developers. By decomposing the Ingress API into several Gateway API objects, different personas can be assigned the appropriate access and privileges according to their responsibilities. This means that, for example, application developers would be allowed to create Route objects in a specified namespace, but would not be able to modify the Gateway configuration or edit Route objects in other namespaces.​

#### Reducing Tool Sprawl

Cilium aims to simplify operations by reducing the number of cloud native networking, observability, and security tools required. It natively supports not just the Gateway API but also features like a sidecar-less Service Mesh, Transparent Encryption, Network Policies, and built-in observability with Hubble. This approach makes life easier for cluster operators, who only need a single tool to manage and upgrade for many, if not all, of their Kubernetes networking needs​​.

#### Gateway API for Service Mesh (GAMMA)

By leveraging the standard Kubernetes Gateway API for Mesh Management and Administration(GAMMA), Cilium can be used for sophisticated East-West traffic management within the cluster. Cilium support for GAMMA translates into simplicity for operators while providing advanced traffic management.

### Ingress

URL: https://cilium.io/use-cases/ingress
Category: Networking

_Enhancing Kubernetes Ingress With Cilium_

**How can I expose my services with a protocol-aware mechanism?**

The Kubernetes ecosystem is rich with ingress controllers, yet they often introduce complexity and limitations. Kubernetes ingress solutions often fall short when it comes to scalability, visibility, and security in dynamic cloud native environments. Many rely on iptables or other legacy networking constructs, which can lead to inefficient resource usage, performance bottlenecks, and limited observability. Additionally, ensuring source IP visibility, seamless TLS handling, and integration with advanced network policies often requires complex workarounds or additional tools, complicating operations and increasing overhead.

#### Flexible Load Balancer for Simplified Management

Cilium Ingress introduces flexible load balancer modes, enabling you to choose between dedicated and shared configurations tailored to your needs. In shared mode, resources are conserved by utilizing a single load balancer across all ingress resources. In dedicated mode, each ingress receives an independent load balancer, preventing conflicts like overlapping path prefixes. This adaptability ensures that scaling your application architecture never compromises performance or resource efficiency.

#### Seamless Source IP Visibility

Cilium Ingress ensures that backend applications retain access to the original source IP, a common pain point with other solutions. By leveraging TPROXY and the intelligent configuration of Envoy, Cilium maintains transparency in HTTP headers such as X-Forwarded-For and X-Envoy-External-Address. This seamless visibility is vital for debugging, logging, and implementing IP-based access controls without additional overhead.

#### Advanced TLS Management

With support for TLS termination and passthrough, Cilium Ingress simplifies secure traffic handling. It enables multi-TLS backend sharing while dynamically routing based on SNI. This eliminates configuration complexity and ensures robust security, even in highly dynamic environments. Applications benefit from simplified setup and seamless performance.

#### eBPF Powered Network Security

Cilium embeds security directly into the network layer using eBPF. This approach allows fine-grained enforcement of CiliumNetworkPolicies for traffic entering and exiting the cluster. Traffic passes through an Envoy proxy integrated with Cilium's policy engine, enabling precise control over network behavior. This dual-layer security ensures compliance with organizational policies while protecting against advanced threats.

### BGP

URL: https://cilium.io/use-cases/bgp
Category: Networking

_Harness the infinite potential of Border Gateway Protocol (BGP) with Cilium_

**Supercharge Your Routing Protocol**

Traditional IP routing can be static and inflexible, especially in cloud native environments where workloads are constantly being created, moved, and deleted. BGP's inherent ability to manage complex network topologies and routing data makes it an excellent fit for the dynamic and distributed nature of modern cloud native environments.

#### Elevate your Networking Infrastructure with Cilium’s BGP

Cilium amplifies the power of BGP, the backbone of the internet, to bring you high-speed, scalable, and secure routing for your cloud environments. Seamlessly integrating with existing infrastructure, Cilium's BGP is perfect for a range of deployments, from hybrid to multi cloud to edge. With advanced traffic engineering features, you'll have granular control over your network traffic, optimizing your network's performance and security.

#### Effortless Integration of Existing Network Infrastructure

Cilium's BGP support is designed to be simple and easy to integrate with your existing networking infrastructure. Wherever your application is running, Cilium can ensure efficient BGP routing for all your workloads. Many network infrastructures already use BGP for routing. By leveraging BGP, Cilium can seamlessly integrate with existing infrastructure, allowing Kubernetes pods to communicate with other parts of the network.

_Adopters & resources referenced on this page:_ Trip.com's Hybrid Cloud Approach with Cilium for Workload Routing; Sidecar BGP Peering Pattern with Cilium at Preferred Networks; Leveraging Cilium’s BGP in Private Data Centers.

### Egress Gateway

URL: https://cilium.io/use-cases/egress-gateway
Category: Networking

_Enhancing network security and traffic control exiting Kubernetes environments_

**Integrating Unpredictable IPs**

Pods typically have ever-changing IP addresses in Kubernetes environments. Even if masquerading is used to mitigate this, the IP addresses of nodes can also change frequently. Egress gateways provide a way to route all outbound traffic from certain pods through a specific node with a predictable IP address. This predictable IP can be useful for scenarios where the traffic destination requires a known source IP, for instance, when working with legacy systems or firewall rules.

#### Harness the Power of Egress Gateway with Cilium

Egress Gateway with Cilium fundamentally transforms Kubernetes networking by addressing dynamic IP challenges, ensuring seamless integration with legacy systems and enhancing network security. It provides precise control over traffic routing, enabling selective direction of pod traffic through stable, predictable IP addresses. This feature enables granular traffic management, effective monitoring and filtering, and workload-specific routing, all while facilitating interoperability with systems requiring known source IPs.

#### Selective Traffic Control

The egress gateway allows fine-grained control over which pods' traffic should be routed through the gateway node. This is done by applying egress gateway policies that use label selectors to target specific pods. This selective routing can help in implementing security policies, achieving network isolation, and managing network costs.

#### Workload-Specific Routing

In multi-tenant Kubernetes clusters, different workloads might need to interact with different external systems that have specific network requirements. Egress gateways can help meet these requirements by allowing the configuration of workload-specific routing rules.

_Adopters & resources referenced on this page:_ Zero Trust Networking at Scale (20k+ VCPUs, 100+ Dev Teams); Cilium Egress Gateway at Corner Banca SA.

### Host Firewall

URL: https://cilium.io/use-cases/host-firewall
Category: Networking

_Host Security That Scales With Kubernetes_

**How can I secure the host namespace with same consistent security model?**

Kubernetes nodes are the backbone of any cluster, but securing them presents unique challenges beyond the capabilities of traditional firewalls. Without granular host-level controls, nodes are vulnerable to unauthorized access, exposing critical cluster components, like kube-apiserver or etcd, to potential breaches. Static firewall rules struggle to adapt to the dynamic nature of Kubernetes environments, leaving gaps in protection during workload changes or updates. Additionally, misconfigurations in restrictive policies can disrupt essential communications, while limited visibility into host-level traffic makes monitoring and troubleshooting difficult.

#### Consistent Security for Nodes and Pods

Traditionally, securing Kubernetes environments required managing separate security models for nodes and pods, resulting in operational complexity and potential blind spots. Cilium Host Firewall eliminates this inconsistency by applying the same network policy model to the host. This approach extends Kubernetes declarative, policy-driven security model to the nodes hosting your workloads, delivering seamless, consistent protection across your entire environment. By leveraging YAML manifests, administrators can define, apply, and manage host-level policies with the same ease and precision as Kubernetes Network Policies, creating a unified approach to securing both pods and their underlying hosts.

#### Host Security That Scales With Kubernetes

Cilium Host Firewall empowers administrators to enforce fine-grained policies for node-level traffic. By matching on node labels, you can create targeted rules that allow or deny traffic based on specific needs, such as permitting only SSH or ICMP traffic to specific nodes. This level of control ensures that your nodes are protected from unauthorized access while maintaining the flexibility required for smooth operations. Cilium Host Firewall ensures consistent, granular, and adaptable security for every node in your cluster, giving you the confidence to scale securely in even the most demanding environments.

### Bandwidth and Latency Optimization

URL: https://cilium.io/use-cases/bandwidth-optimization
Category: Networking

_Simple and intuitive network performance optimization_

**Offering Latency and throughput improvements while controlling pod network contention**

Kubernetes lacks native traffic control capabilities, making Traffic Rate-Limiting essential for optimal resource consumption and to prevent bandwidth exhaustion. While Kubernetes does offer Bandwidth Rate-Limiting, it is still experimental and can have detrimental effects on latency. Furthermore, connecting to external-facing Kubernetes clusters may result in a poor user experience due to most TCP congestion protocols not being designed for today's diverse networks.

#### Optimal Bandwidth Management with Cilium’s Bandwidth Manager

Cilium’s Bandwidth Manager allows rate-limiting per Pod with just one line of YAML. Compared to other alternatives, the Bandwidth Manager provides a 4x reduction in latency, ensuring a smooth network experience and isn’t detrimental to performances and is designed for multi-queue and multi-core NICs.

#### Accelerate network performance with Cilium BBR

Cilium supports BBR, a congestion control algorithm developed by Google, making it the first platform to do so. Google observed up to 2,700x improvement in throughput when testing BBR, making it a valuable addition for optimizing network performance. Cilium’s BBR provides exceptional improvements in throughput and latency for external-facing applications, offering consumers a delightful user experience

#### 100Gbit/S Clusters With Cilium: Building Tomorrows Networking

Cilium BIG TCP allows for larger packets than the traditional 64KB limit by leveraging IPv6's Hop-by-Hop header, which can specify payload lengths up to 512KB. This is particularly useful for organizations building networks capable of 100Gbps and beyond. BIG TCP does not require modifying the MTU on network devices, making it easier to implement than Jumbo Frames. With BIG TCP, Cilium offers enhanced network performance for nodes, enabling users to extract as much performance as possible from the network.

_Adopters & resources referenced on this page:_ How Flipkart Implements Bandwidth QoS with Cilium; BIG Performances with BIG TCP on Cilium.

### IPv6-Native Kubernetes Networking with Cilium

URL: https://cilium.io/use-cases/ipv6
Category: Networking

_Cloud native networking at IPv6 scale, powered by eBPF_

**IPv6-Native Kubernetes Networking with Cilium**

IPv6 adoption is accelerating across modern infrastructure. Address exhaustion, large-scale cluster growth, and cloud native connectivity are pushing platform teams toward IPv6-first designs. Cilium brings IPv6 deep into the Kubernetes datapath, enabling clusters to operate at scale without sacrificing performance, visibility, or control. Rather than treating IPv6 as a compatibility layer, Cilium integrates it directly into how traffic flows through the cluster. IPv6 unlocks massive address space, simplified routing, and long-term scalability. Cilium turns those advantages into practical outcomes for Kubernetes by combining IPv6-native networking with high-performance eBPF data paths. The result is a Kubernetes networking layer ready for the next decade of growth.

#### IPv6 Underlay with Overlay Simplicity

Operating Kubernetes across diverse infrastructure often requires abstraction from the physical network. Encapsulation modes such as VXLAN and Geneve provide that abstraction, allowing clusters to behave consistently regardless of where they run. Cilium enables these overlays to operate on top of an IPv6-only physical network. The underlay can be entirely IPv6 while pod-to-pod traffic continues to flow through a stable encapsulated datapath. This allows platform teams to modernize their networks to IPv6 without rethinking how Kubernetes connectivity is designed or operated. The result is an IPv6-native foundation with the same predictable networking model teams already trust.

#### IPv6-Aware Egress Control

Service networking is one of the most performance-sensitive paths in Kubernetes. Legacy approaches based on iptables introduce latency, complexity, and scaling challenges, problems that become more pronounced in large IPv6 environments. Cilium delivers IPv6 service handling directly in eBPF. Load balancing, service translation, and traffic steering all happen at the kernel level without relying on iptables or kube-proxy. IPv6 traffic benefits from the same high-performance, low-latency datapath that has made Cilium a standard for large-scale clusters. This creates a clean, efficient service model for IPv6 workloads that scales naturally as clusters grow.

#### A Unified IPv6 Datapath Built on eBPF

IPv6 in Cilium is implemented with the same eBPF-based datapath that powers IPv4 networking, security, and observability. This creates a unified model in which IPv4, IPv6, and dual-stack clusters share the same architecture and operational semantics. For platform teams, this means fewer special cases, simpler troubleshooting, and confidence that IPv6 workloads behave exactly as expected at scale.

_Adopters & resources referenced on this page:_ TikTok's IPv6 Journey To Cilium: Pitfalls and Lessons Learned; IPv6 First, Not Just Ready: Kubernetes Without IPv4 Using Cilium at ESnet.

### Multicast

URL: https://cilium.io/use-cases/multicast
Category: Networking

_eBPF-Powered Efficiency for Multicast in Kubernetes_

**Smarter Network Utilization**

IP multicast is a bandwidth-conserving technology that reduces traffic by simultaneously delivering a single stream of information to potentially thousands of recipients. Multicast addresses the technical challenge of efficiently delivering data from a single source to multiple recipients simultaneously over a network. Multicast has found its use case in traditional networking, but Kubernetes networking often relies on unicast and broadcast methods.<br /><br />In data transmission to containers within a Kubernetes cluster, packets are replicated and sent to every container regardless of their need. This approach might not be ideal for some applications with specific requirements and can lead to significant network congestion, high bandwidth, and overhead.

#### Muticast the Cloud Native Way

Cilium's multicast feature enables efficient network communication. By leveraging multicast, Cilium can efficiently transmit and replicate a single data stream to reach multiple consumer containers in a Kubernetes cluster, reducing the overall network bandwidth and overhead. This approach is particularly beneficial for industries like financial services, media, and entertainment, which typically have multiple services and endpoints that need to communicate simultaneously.

#### Bringing the eBPF Revolution to Multicast

Cilium's multicast implementation is not just efficient, it's also technically advanced. It's based on eBPF, a technology known for its exceptional efficiency and performance. By leveraging eBPF, Cilium ensures that the same level of optimization is brought to multicast as it does to a variety of other networking use cases.

### Identity-aware L3/L4/DNS Network Flow Logs

URL: https://cilium.io/use-cases/network-flow-logs
Category: Observability

_Holistic view of your network traffic_

**How can you trace traffic to a specific workload in a complex environment?**

Traditional network flow logs often lack the necessary level of detail and context needed to monitor and troubleshoot network issues in complex environments making it difficult to trace traffic back to specific workloads. This makes debugging difficult and time-consuming and makes it challenging to identify the source and scope of security incidents.

#### Real time visibility for Network Flows

By providing real-time visibility into network flows with enhanced metadata, including identity-based information about Kubernetes workloads, Cilium simplifies the process of monitoring and troubleshooting network traffic in Kubernetes clusters. This makes it easier to trace traffic back to specific workloads, simplifying the process of monitoring, troubleshooting, and addressing network issues, and enabling operators to quickly identify and respond to security incidents.

#### Kubernetes-aware Network Flow Logs

Cilium enhances network flow logs with additional information about the identity of the Kubernetes workloads generating or receiving the traffic. This identity information is based on the Kubernetes labels and annotations assigned to the workloads, as well as any other metadata that can be retrieved from the Kubernetes API server.

_Adopters & resources referenced on this page:_ Observability for a highly available multi cluster environment with Hubble; Building Data Sandboxes at Bloomberg.

### Metrics & Tracing Export

URL: https://cilium.io/use-cases/metrics-export
Category: Observability

_Valuable insights for troubleshooting, optimization, and performance monitoring_

**Monitor and analyze the performance and behavior of your Cilium-managed Kubernetes environment**

Metrics alone may lack context for effective issue diagnosis. Configuring metrics exporters and integrating them with monitoring systems can be error-prone and manual. Inadequate, inconsistent, or incorrect metrics can be misleading, resulting in incorrect conclusions about the application state. Troubleshooting often requires correlating metrics with traces for a comprehensive system understanding.

#### Metrics and Tracing With Cilium

Cilium's Metrics and Tracing export feature provides a seamless and integrated solution empowering users to monitor, analyze, and optimize their Kubernetes environments with ease. By leveraging the power of Prometheus metrics, combined with Hubble's network behavior insights, Cilium enables users to gain deep visibility into their applications and network while simplifying the setup and configuration process. Cilium also integrates with various tracing systems, such as Jaeger, Zipkin, and OpenTelemetry, to provide distributed tracing capabilities. Cilium is optimized to handle high data volumes without compromising on performance.

#### Comprehensive and Rich Set of Metrics

Cilium captures a plethora of metrics, including latency, request rates, and error rates for your applications. These metrics are exported in a standardized Prometheus format, which can be easily integrated with your existing monitoring and visualization tools, enabling you to track your network performance in real-time.

#### Distributed Tracing

Cilium supports popular distributed tracing frameworks like Jaeger and Zipkin. With this, you can visualize request flow through your services, identify bottlenecks and optimize for efficiency. Cilium’s tracing provides a granular view of service interactions, bringing clarity to complex distributed systems.

_Adopters & resources referenced on this page:_ eBPF-powered observability for Kubernetes and cloud native infrastructure; Cilium-powered Enterprise Kubernetes Management for Platform Teams.

### Advanced Network Protocol Visibility

URL: https://cilium.io/use-cases/protocol-visibility
Category: Observability

_Insights into workload communication at the protocol level_

**How can I understand my workload’s communication at the protocol level?**

Traditional network observability tools provide packet-level visibility, which can be insufficient in cloud native and other environments with complex communication patterns and diverse application protocols. In such scenarios, troubleshooting network issues can be challenging, and gaining protocol-level visibility becomes critical to ensure observability and security.

#### Gain deep insights into your workload communication at the protocol level with Cilium

Cilium's protocol-aware visibility provides application owners with deep insights into their workload’s communications at the protocol level. This means that Cilium is capable of native understanding of various application protocols, such as TLS, gRPC, Kafka, DNS, and HTTP, and other ones like SCTP enabling fine-grained observability of API-specific endpoints and DNS-identities for external endpoints.

#### Hubble, eBPF-based Observability for Kubernetes

Hubble is an eBPF-based observability platform for Kubernetes. Hubble leverages the Cilium CNI and eBPF technology to provide fine-grained visibility into network traffic and application behaviour, without the need to modify applications. With Hubble, administrators to interactively troubleshoot complex network issues and write custom metrics to benefit from eBPF's superpowers without the need to write or understand any kernel code.

_Adopters & resources referenced on this page:_ Enhanced observability with Hubble; A more consumable network infrastructure with Cilium at Ikea.

### Service Map

URL: https://cilium.io/use-cases/service-map
Category: Observability

_Detailed visibility into the communication and behavior of your workloads_

**How can I gain detailed visibility into the communication and behavior of my workloads?**

When troubleshooting cloud native environments, the issue could be lurking between any layer of the network, environment, or its dependencies. For instance, it may be difficult to determine if DNS is working properly, or if an application is failing due to policy-related issues. Identifying the latest policy-related drops or determining if essential traffic between service components is being blocked can also prove to be challenging. On the other hand, examining logs for answers can be a difficult and time-consuming process.

#### Hubble, eBPF powered Network, Service, and Security Observability for Kubernetes

Hubble provides a range of monitoring capabilities, including service dependencies and communication maps, network monitoring, application monitoring, and security observability. By relying on eBPF, all visibility is programmable and allows for a dynamic approach that minimizes overhead while providing deep and detailed visibility.

#### Service Dependency Graph

Simply looking at `kubectl get pods` does not indicate dependencies between each service or external APIs or databases. Hubble offers effortless automatic discovery of service dependencies within Kubernetes Clusters at L3/L4 and L7 levels. This allows for user-friendly visualization and filtering of dataflows as a Service Map, enabling easy management of service dependencies.

_Adopters & resources referenced on this page:_ Building a Secure and Maintainable PaaS; Saving Cost and Accelerating Insights with Hubble; Providing Observability to Customers With Hubble; Transforming Scientific Computing Infrastructure for IPv6 with Cilium at ESnet.

### Runtime Security

URL: https://cilium.io/use-cases/runtime-security
Category: Security

_Prevent threats and enforce policies in real-time_

**Achieve threat prevention in cloud native environments while maintaining operational agility**

Cloud native environments are often dynamic and distributed, requiring a security approach that encompasses detection and prevention. Observing and filtering events in user space can be resource-intensive and lead to blind spots in security monitoring, leaving systems vulnerable to attacks.

#### Security Observability and Runtime Enforcement with Cilium’s Tetragon

<a href='https://tetragon.io' style='text-decoration: underline;'>Tetragon</a> enables transparent security observability and real-time runtime enforcement through its eBPF-based technology. It provides deep visibility without requiring changes to the application and operates with low overhead through in-kernel filtering and aggregation logic built into the eBPF-based kernel-level collector. Tetragon's embedded runtime enforcement layer offers access control capabilities at various enforcement levels, including system call control.

#### Kubernetes-aware Real Time Enforcement

<a href='https://tetragon.io' style='text-decoration: underline;'>Tetragon</a> is Kubernetes-aware, meaning it recognizes Kubernetes identities like namespaces and pods. This enables security event detection that can be tailored to individual workloads. Using eBPF, Tetragon can access the Linux kernel state and combine it with Kubernetes awareness and user policy to generate rules that are enforced by the kernel in real-time. This allows for capabilities like process namespace and capabilities annotation and enforcement, process file descriptor to filename association, and socket to process control.

_Adopters & resources referenced on this page:_ Don't Get Blown up! Avoiding Configuration Gotchas for Tetragon Newbies; Integrating Tetragon for Secured Build Pipelines; Pollenating Build Attestations on Kubernetes with Tetragon and eBPF; Hubble + Tetragon at Palantir; Who Owns Your Pod? Observing and Blocking Unwanted Behavior at eBay With eBPF.

## Business outcomes

### Zero Trust Networking

URL: https://cilium.io/outcomes/zero-trust

_Enforce what matters, where it matters._

Traditional network security measures can be ineffective in cloud native systems due to the transient nature of IP addresses, the dynamic nature of workloads, and the rapid growth of east-west traffic. These internal communication routes are undefended by standard firewall-based security, and it becomes next to impossible to maintain static IP rules or network segments across numerous environments. Zero Trust Networking directly addresses these challenges by shifting the focus from trusting the network to explicitly verifying each connection and workload. Instead of assuming anything inside the network is safe, Zero Trust Networking requires all communication to be authenticated and authorized based on what a workload is, not where it runs. Cilium makes this practical in cloud native environments by enforcing identity-aware policies tied to Kubernetes service accounts and labels, applying them efficiently in the kernel with eBPF, supporting multi-cluster and multi-cloud deployments, and providing deep visibility into and security for service-to-service communication.

#### Keep internal traffic secure at any scale

In modern infrastructure, internal communication is often the largest and most vulnerable attack surface. In the absence of Zero Trust controls, attackers who breach one service have the ability to extend their attacks laterally, gain access to sensitive data, and escalate their attacks. For regulatory frameworks like PCI DSS, GDPR, and HIPAA, protection of internal communication is becoming increasingly necessary in order to comply, and firewall-based controls aren’t enough. Cilium helps you meet these security and compliance demands by enforcing Zero Trust Networking policies that protect every connection, no matter how large or complex your environment becomes.

#### Enforcing identity driven policies instead of IP-based rules

As workloads scale, restart, or move across nodes and clusters, IP addresses change constantly, creating gaps or needing constant rule updates. Cilium solves this by applying security policies tied to Kubernetes-native identities like service accounts, pod labels, and namespaces. These identities stay consistent even as workloads shift or restart, so policies automatically follow the correct services wherever they run, removing the need to manage static network rules and ensuring reliable Zero Trust Networking enforcement at scale.

#### Implement policies in the kernel

Typically, enforcing Zero Trust Networking relies on proxies, sidecars, or userspace components to inspect and control traffic. These methods can make things more complicated to run, especially as systems scale. Cilium takes a different approach by using eBPF to apply security policies directly in the Linux kernel. This allows high-performance enforcement at the source of the traffic without extra components in the data path. The result is fast, low-latency security that scales cleanly with your infrastructure.

#### Secure at the API and application layer, not just at the network layer

In cloud native environments, many microservices communicate through APIs. Traditional security solutions stop at IP addresses and ports, leaving critical API traffic and application-level interactions vulnerable to misuse. This creates blind spots that attackers can exploit once inside the network. Cilium extends Zero Trust Networking to Layer 7, enabling network policies that inspect and control traffic based on API paths, HTTP methods, gRPC calls, DNS queries, and more. This ensures services communicate only as intended and minimizes unnecessary exposure within your environment.

#### Built-in distributed firewalling for Zero Trust Networking at scale

Rather than relying on centralized firewalls or sidecar proxies, Cilium enforces security policies directly at the source of traffic using eBPF inside the Linux kernel. This distributed model applies identity-aware policies consistently across all nodes, clusters, and environments, without introducing bottlenecks. With visibility and control from Layer 3 to Layer 7, Cilium prevents unauthorized lateral movement and delivers scalable, low-overhead Zero Trust enforcement across cloud native infrastructure.

#### Keep Zero Trust enforcement consistent across clusters and clouds

Applications rarely run in just one cluster, region, or cloud. As environments grow more distributed, keeping security policies consistent across all of them becomes complex, and any gaps in enforcement between environments creates risk. Cilium makes it simple to define and apply Zero Trust Networking policies uniformly, no matter where your workloads run. Whether you operate in a single Kubernetes cluster, across multiple clouds, or in hybrid setups, Cilium helps ensure security stays consistent without duplication or increased complexity.

_Adopters & resources referenced on this page:_ Implementing Zero Trust in highly regulated financial systems; Enforcing Zero Trust isolation in a multi-tenant platform; Building Zero Trust into the Choreo platform; Strengthening security and compliance across multi-cloud HR systems; Enforcing Zero Trust data protection and access controls; Network Policy; Advanced Network Protocol Visibility; Identity-aware L3/L4/DNS Network Flow Logs; Host Firewall.

### Tool Consolidation

URL: https://cilium.io/outcomes/tool-consolidation

_Simplify your networking stack without sacrificing visibility or control_

As cloud native environments grow in complexity, teams often rely on a patchwork of tools for networking, observability, and security, each with its own learning curve, performance trade-offs, and integration overhead. Cilium replaces this fragmented tooling with a unified platform built on eBPF that provides connectivity, security, and deep visibility all in one.

#### Replacing a patchwork of tools with Cilium

Managing and integrating multiple tools for service mesh, firewalling, load balancing, ingress, observability, and policy enforcement slows teams down. Cilium helps teams consolidate without compromise with one tool for managing network traffic, service discovery, API-aware security, and real-time network visibility. This reduces operational costs, accelerates onboarding, and improves performance, while maintaining full control.

#### Consolidating your networking stack

With Cilium, you get high-performance networking, API-aware security, and deep observability in a single tool powered by eBPF. It removes the need for standalone network plugins, sidecar-based service meshes, and third-party observability layers, while integrating natively with Kubernetes.

#### Observe traffic in real time without extra tools

Cilium’s Hubble brings powerful network visibility directly into your Kubernetes environment, surfacing flow logs, a service map, and protocol-level insights (HTTP, gRPC, Kafka, etc.) without installing separate monitoring or tracing tools.

#### Enforce policies at Layer 3 to Layer 7

Unlike traditional firewalls, Cilium enforces identity-aware policies tied to Kubernetes primitives and inspects traffic all the way up to Layer 7. That means you can secure microservices based on actual behavior, from DNS queries to API calls, without additional tooling.

_Adopters & resources referenced on this page:_ One CNI to Power Customer and Internal Infrastructure; Unified networking and observability for MetaKube; One Platform to Manage Global Scale; Hubble; Kube-proxy Replacement; Gateway API; Border Gateway Protocol (BGP); Layer 4 Load Balancer; Service Mesh.

### Multi-Cloud Connectivity

URL: https://cilium.io/outcomes/multi-cloud-connectivity

_Seamless service-to-service communication across clouds, clusters, and regions._

Connecting workloads across multiple clouds is often fragile and complex, introducing issues like overlapping IPs and inconsistent rules. Cilium solves this with a unified, eBPF-powered network that offers native service discovery, identity-aware security, and scalable performance that works wherever your infrastructure lives.

#### Unified connectivity and service discovery across clusters

Cilium’s Cluster Mesh enables services across multiple clusters in any cloud or on-prem environment to connect and discover each other natively, as if running in the same cluster. Connections are fast, encrypted, and identity-aware by default, with no need for complex networking configurations.

#### Trusted across cloud providers

Cilium powers the default networking for major Kubernetes platforms like GKE, EKS-A, and AKS, and is trusted by leading organizations operating at scale. Its performance and security model have been validated across a wide range of public clouds, private data centers, and edge environments, making it a reliable foundation for multi-cloud and hybrid infrastructure. Whether you're running in one cloud or many, Cilium makes sure your clusters have consistent connectivity, performance, and security.

_Adopters & resources referenced on this page:_ From Network Outages to Seamless Connectivity; Scaling secure payments across regions; Simplifying global service communication at scale; Reliable, flexible connectivity for data automation; Bridging Clouds for smarter operations; High Performance Container Network Interface (CNI); Cluster Mesh.

### Network Automation

URL: https://cilium.io/outcomes/network-automation

_Automate and simplify Kubernetes networking at scale_

Manual network configuration can’t keep up with the speed of cloud native environments. In Kubernetes, where workloads are dynamic and infrastructure is distributed, traditional approaches to networking create bottlenecks, inconsistencies, and operational risk. Cilium brings automation to the core of Kubernetes networking. Cilium makes it possible to configure, secure, and observe network behavior automatically through identity-aware policies that adapt to workload changes, dynamic service discovery, an eBPF-powered datapath, and seamless integration with CI/CD and GitOps workflows.

#### Make your networking configuration as dynamic as your workloads

As workloads in cloud native environments scale, shift, and redeploy across clusters and clouds, manual network configuration quickly becomes unsustainable. Engineers are left to manage rigid IP rules, duplicate policies across environments, and react to changes rather than design systems that adapt automatically. Cilium addresses this by making the network programmable and declarative. It allows teams to specify intent once, using workload identity and Kubernetes-native components, and then trust that connectivity, security, and observability will respond in real time as the environment changes. This means less time spent developing and debugging YAML, fewer outages due to misconfigured rules, and increased confidence when deploying infrastructure or adding new services. By removing manual steps from the networking stack, Cilium helps platform and security teams move faster, reduce risk, and operate more reliably in even the most dynamic environments.

#### Enforcing identity-driven policies instead of IP-based rules

As workloads scale, restart, or move across nodes and clusters, IP addresses change constantly, creating configuration gaps, or a need for continual manual updates. Cilium solves this by applying security policies tied to Kubernetes-native identities like service accounts, pod labels, and namespaces. These identities remain consistent even as workloads shift, so policies follow services automatically without requiring engineers to rewrite or reapply static rules. This keeps network behavior predictable, secure, and in sync with your infrastructure design, with little to no manual intervention.

#### Dynamic connectivity without manual updates

Services in Kubernetes are always changing, either scaling up, down, or shifting across nodes. Manually updating routing or access rules to stay up-to-date is inefficient and error-prone. Cilium automates this by integrating deeply with Kubernetes. As services come and go, Cilium updates routing and policies in real time, ensuring that communication paths stay correct and secure without any effort.

#### Consistent policy across clusters and clouds

Cilium’s Cluster Mesh and cluster-wide policies make it possible to apply security rules consistently across namespaces, regions, or even different cloud providers. This unified approach removes the need to duplicate network configurations for each environment, reduces operational overhead, and minimizes the risk of inconsistencies, ensuring reliable protection no matter where workloads run.

#### eBPF programs and secures the network in the kernel

Rather than relying on centralized firewalls or sidecar proxies, Cilium enforces security policies directly at the source of traffic using eBPF inside the Linux kernel. This distributed model applies identity-aware policies consistently across all nodes, clusters, and environments, without introducing bottlenecks. With visibility and control from Layer 3 to Layer 7, Cilium prevents unauthorized lateral movement and delivers scalable, low-overhead [Zero Trust Networking](https://cilium.io/outcomes/zero-trust/) enforcement across cloud native infrastructure.

#### Automated networking built for GitOps and CI/CD

Modern infrastructure is managed as code, and your networking should be too. With Cilium, network policies are declarative, version-controlled, and Kubernetes-native, making them easy to integrate into the same pipelines and workflows you use for deploying applications. You can define policies in Git, review them through pull requests, and apply them automatically through CI/CD or GitOps tools. This makes your network configuration more consistent, easier to audit, and easier to automate, while ensuring that your infrastructure stays secure and in sync across all environments.

_Adopters & resources referenced on this page:_ Solving latency and scaling network performance; Delivering high-performance cloud networking; Automating scalable networking across KubeSphere; Building a scalable, future-ready networking platform; Automating Kubernetes networking and security at scale; Kube-proxy Replacement; Network Policy.

### Cost and Carbon Savings

URL: https://cilium.io/outcomes/cost-and-carbon-savings

_Do more with less_

Manual and legacy network systems often come with hidden overhead like inefficient IP usage, many network redirects, and cloud charges that add up fast. Cilium removes these inefficiencies with a streamlined, eBPF-powered network engine, lowering costs while reducing compute waste and carbon dioxide emissions.

#### Why Sustainability Starts at the Network Layer

​​In Kubernetes environments, inefficiencies in networking can grow quickly. Traditional solutions are often heavy on resource consumption, requiring extra nodes, IP addresses, and compute power. This not only inflates costs but also increases energy use and carbon emissions. Cilium changes that by optimizing network paths, removing unnecessary components, and providing granular observability, meaning teams can reclaim resources, cut bills, and improve sustainability.

#### eBPF-powered dataplanes cut CPU costs

Cilium replaces traditional kube-proxy and iptables-based networking with an eBPF-based dataplane. This removes the need for userspace proxies and reduces the overhead of managing long rule chains. By processing traffic closer to the source, Cilium shortens the network path, lowers CPU consumption, and frees up compute resources for actual workloads, saving both money and energy.

#### High pod density with lean IP allocation

Particularly when used at scale, other CNIs may require large blocks of IP addresses for every node, limiting pod density and resulting in wasted resources. Because Cilium manages IP allocation more efficiently, you can run more pods per node without running out of IPs. This improves resource utilisation and decreases infrastructure and electricity costs by reducing the number of nodes needed.

#### Eliminate cloud load balancer charges

With Cilium's Cluster Mesh, native pod-to-pod communication between clusters is possible without the need for costly cloud load balancers. This simplifies your architecture and prevents excessive data transfer fees. By removing external dependencies, teams gain more control over traffic, reduce latency, and cut down significantly on cloud networking costs.

_Adopters & resources referenced on this page:_ Cutting networking overhead at scale; Lowering network costs while debugging faster; Lower costs with high-performance; Cutting storage costs and cloud networking complexity; Reducing cloud bills and IP footprint; Bandwidth and Latency Optimization; High Performance Container Network Interface (CNI); Kube-proxy Replacement.

## Ecosystem and subprojects

### Hubble

URL: https://docs.cilium.io/en/stable/observability/hubble/

Networking and security observability layer built on Cilium and eBPF: flow logs, metrics, and service map.

### Tetragon

URL: https://tetragon.io

EBPF-based runtime security and observability for process, file, and network events.

### eBPF

URL: https://ebpf.io

The Linux kernel technology Cilium is built on.

## Key pages

### llms-full.txt

URL: https://cilium.io/llms-full.txt

Full-text version with the complete prose of every use-case, outcome, and industry page and every blog post inlined. (this file).

### Home

URL: https://cilium.io/

The Cilium project home page.

### Get started

URL: https://cilium.io/get-started

What is Cilium and how to get started.

### Adopters

URL: https://cilium.io/adopters

Adopters and production users.

### Enterprise

URL: https://cilium.io/enterprise

Enterprise distributions and training.

### Blog

URL: https://cilium.io/blog

Full text of all 147 posts published on the cilium.io blog, newest first, followed by an index of posts that link to articles hosted elsewhere.

#### Zero-Trust Developer Platforms with Cilium Network Policies

URL: https://cilium.io/blog/2026/08/28/zero-trust-developer-platforms-with-cilium-network-policies
Date: 2026-08-28
Categories: Technology, Security

Every platform team eventually hits the same wall. Security wants zero trust where no workload talks to another unless the connection is explicitly sanctioned. Engineering wants to ship on Friday afternoon without filing a ticket with the platform engineering team to get an IP whitelisted. Both are right, and the network is where the argument plays out.

This doesn't have to be a trade-off for a platform team. It's a design problem at the platform layer itself, and it has a known solution: make network policies a _byproduct_ of deployment rather than a prerequisite to it.

##### The limits of IP-based security in Kubernetes

Traditional network security assumes an address means something. A firewall rule permitting `10.4.2.17` to reach `10.4.8.9:5432` encodes a real intent, _the orders service may query the orders database_, but it encodes that intent in a fragile form.

Kubernetes breaks that form completely. Pods are scheduled, rescheduled, scaled, evicted, and rolled. The IP address that identified your payments service this morning may belong to a batch job by lunchtime. Any security model that depends on addresses staying still is accurate only until the next reschedule.

Standard Kubernetes NetworkPolicies improve on this by letting you declare intent with label selectors instead of addresses. But two problems remain.

First, even though the intent declaration is by label, enforcement still happens by address. The CNI resolves selectors into concrete pod IPs and programs iptables chains accordingly, reprogramming on every reschedule. On a busy cluster, these rule sets can grow into thousands of sequentially-evaluated entries: a well-documented source of latency and control-plane load.

Second, and more limiting, the NetworkPolicy API stops at Layer 4. A policy can allow port 8080 on a workload. However, it cannot say “allow `GET /orders` but not `DELETE /orders`”, or “this workload may reach _`api.stripe.com`_ and nothing else on the internet”. For most real threat models that's too broad; the interesting attacks happen inside connections you already allowed.

##### The shift to identity-based network policies with Cilium

Cilium changes the primitive. Instead of translating labels into IP-keyed firewall rules, it assigns every workload a security identity derived from its labels and namespace, and enforces policy on that identity via eBPF programs attached directly to the kernel's networking hooks.

This maps naturally onto ephemeral workloads in Kubernetes. When a pod is rescheduled onto a different node with a different IP, nothing about the policy needs to change. The new pod carries the same labels, resolves to the same identity, and inherits the same permissions the moment it starts. Policy stops depending on infrastructure details.

Cilium can also reach higher up the OSI stack. For L7 rules, eBPF transparently redirects matched traffic to a per-node Envoy proxy with no sidecar and no application change. CiliumNetworkPolicy expresses application-aware rules: HTTP methods and paths, gRPC services, and DNS/FQDN-based egress. That’s what makes the least-privilege model actually achievable: "this service may call that service" becomes "this service may call exactly this operation on that service."

##### Zero trust in developer platforms

Installing Cilium does not give you zero trust. It gives you the mechanism for it; a fast, identity-based enforcement engine that can express the exact rules you want. Orchestrating the policies to enforce zero trust while giving platform teams a way to govern them should come as a property of the developer platform built on top of Cilium.

There are several ways to achieve zero trust in a developer platform.

- **Developers author their own policies.** Nobody understands a service's dependencies better than the team that built it, so this is the most flexible option, and the policies it produces can be exactly right, at least at first. The difficulty isn't competence. CiliumNetworkPolicy becomes one more thing to learn and maintain, alongside the language and frameworks the service is actually written in. Services gain endpoints, drop dependencies, and get refactored, while the policy sits in a separate file that someone has to remember to update. Some drift open, because nothing breaks when they do. Some are never written at all. Some are written carefully and then quietly rot.

- **Platform teams pre-provision isolated environments with policy already applied.** This is a good, common pattern: developers receive an environment that is default-deny from the moment they get it, and they never touch a CiliumNetworkPolicy. The limits are granularity and change. A policy scoped to the whole environment can't know that one component listens on 8080 and another on 9090, so it either permits more than it should inside the boundary, or it needs a request to the platform team for every new endpoint, the exact ticket queue an Internal Developer Platform (IDP) exists to remove.

- **The platform derives policy, the platform team governs it.** If a developer states an endpoint's intended reach as part of deploying it, the platform has everything it needs to generate the policy itself. Policy stops being an artifact anyone maintains and becomes an output of deployment, while the platform team keeps control over any exceptions.

That third option is what makes a developer platform zero-trust by default. It rests on the following properties, which are not specific to any vendor:

- **Policy is derived from declared intent** rather than authored separately, so intent and enforcement cannot disagree.
- **Policy is regenerated as the workload changes**, so it cannot drift.
- **Default-deny is a byproduct**, not a checklist item. A workload is restricted because it was deployed, not because someone remembered.
- **The boundary is owned by the platform**, so it is held by construction rather than by convention.

None of that is achievable by just installing a CNI, and none of it requires developers to learn one. It requires abstractions that carry enough intent for the platform to act on.

##### What this looks like in practice: OpenChoreo and Cilium

Those four properties are easy to state and harder to build. They require a platform that already knows what a workload is, not just where it's running. OpenChoreo is one example of a developer platform built this way, and it's a useful one to walk through concretely, since it wires the same identity-based model straight into Cilium.

[OpenChoreo](https://openchoreo.dev/) is an open-source internal developer platform for Kubernetes, and a CNCF sandbox project. Developers work with a simpler set of Kubernetes-native abstractions (Project, Component, Environment,...) wired to a Backstage-based portal, CI/CD, GitOps, and observability, while platform teams keep full visibility of the underlying cluster. Individual capabilities ship as swappable modules from a growing ecosystem, including a networking module that builds on Cilium as the cluster CNI.

Its runtime model and abstractions provide exactly what is needed to have zero trust by default.

Each OpenChoreo **Project** becomes a [**Cell**](https://openchoreo.dev/docs/concepts/runtime-model/) at runtime: a secure, isolated runtime boundary that encapsulates all components of an application domain. Concretely, every (project, environment) pair maps to its own dedicated data-plane namespace, and every pod deployed into it is labelled with its full platform identity: which namespace, project, component, and environment. Components inside a cell communicate freely. Anything crossing a cell boundary goes through a defined gateway.

The other half of the picture is a single field that developers already fill in. When a developer declares an endpoint on their workload, they state its visibility:

| Visibility | Who can reach it |
|:---------- |:------------------------------------------------------------------------------------------------------- |
| `project` | Only components in the same project, within the same environment _(implicit — every endpoint gets this)_ |
| `namespace` | Any project in the same namespace within the same environment |
| `internal` | Across all namespaces in the platform |
| `external` | Exposed publicly, via the external gateway |

That's the entire developer-facing surface. No selectors, no port lists, no CRDs.

From there, OpenChoreo's built-in controllers do the translation. When a component is deployed, the platform renders the workload's Kubernetes resources and injects a network policy alongside them, built from the component's pod identity labels and its declared endpoint visibilities. The policy is regenerated on every reconcile, so it tracks the component as it changes rather than drifting from it.

For a single HTTP endpoint with `project` visibility, the result is a policy that selects the component by identity and admits traffic only from inside its own cell:

```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: openchoreo-web-app
spec:
  endpointSelector:
    matchLabels:
      openchoreo.dev/component: web-app
      openchoreo.dev/project: my-project
  ingress:
    - fromEndpoints:
        - {} # any endpoint in this cell's namespace
      toPorts:
        - ports:
            - port: '8080'
              protocol: TCP
          rules:
            http: [{}] # routed via Envoy for L7 visibility
```

This YAML was not written by a developer to allow ingress into their component. It's rendered at deployment time. The important part is what it omits: because a CiliumNetworkPolicy selects this component, everything not listed is denied. Default-deny isn't a separate policy the platform team must remember to apply, it's the consequence of the component having a policy at all. Raise the endpoint to `namespace` visibility and an additional rule appears, scoped to the same namespace and environment. Mark it `external` and a rule admitting ingress from the platform's gateway appears. Neither requires the developer to know any of this happened.

> _Fig 1: OpenChoreo’s Backstage portal Kubernetes artifacts view for a deployed component, showing the generated `CiliumNetworkPolicy` listed alongside the Deployment, Service, and HTTPRoute._

For the platform engineer, adopting this is deliberately simple. Install Cilium as the CNI on the data-plane cluster, then annotate the `DataPlane`/`ClusterDataPlane` OpenChoreo custom resource to explicitly mark it as a Cilium-powered data plane.

```shell
kubectl annotate clusterdataplanes.openchoreo.dev default \
  openchoreo.dev/networkpolicyprovider=cilium --overwrite
```

Every subsequent deployment renders CiliumNetworkPolicies instead of standard Kubernetes NetworkPolicies. Without the annotation, OpenChoreo emits plain NetworkPolicies, so clusters without Cilium still get project isolation, just at L3/L4. Same developer abstraction, different enforcement engine underneath. That's what makes the abstraction worth having: the platform team can upgrade the enforcement model without touching a single application.

Currently, the generated policies use Cilium's L7 hook for network observability: an empty HTTP rule allows all requests on the port while routing them through Envoy so Hubble can observe them. They also cover ingress only: OpenChoreo does not yet derive egress rules from a developer-facing abstraction.

That doesn't leave egress ungoverned, it just moves the control point from the endpoint to the cell boundary. A `ProjectType` carries a list of resource templates that OpenChoreo renders into the cell namespace of every project of that type, and a template can be any namespace-scoped manifest, including a CiliumNetworkPolicy. The platform engineer writes the policy once and every cell created from that project type inherits it:

```yaml
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterProjectType
metadata:
  name: standard-service
spec:
  resources:
    - id: cell-namespace # Isolated runtime namespace for workloads
      targetPlane: dataplane
      template:
        apiVersion: v1
        kind: Namespace
        metadata:
          name: ${metadata.namespace}
    - id: cell-egress # Templated network policy to secure egress from the cell boundary
      targetPlane: dataplane
      template:
        apiVersion: cilium.io/v2
        kind: CiliumNetworkPolicy
        metadata:
          name: cell-egress
          namespace: ${metadata.namespace}
        spec:
          endpointSelector: {} # every pod in the cell
          egress:
            - toFQDNs: # egress denied except for api.stripe.com
                - matchName: api.stripe.com
            - toEndpoints: # allow dns resolution
                - matchLabels:
                    k8s-app: kube-dns
                    k8s:io.kubernetes.pod.namespace: kube-system
              toPorts:
                - ports:
                    - port: '53'
                      protocol: UDP
                    - port: '53'
                      protocol: TCP
                  rules:
                    dns:
                      - matchPattern: '*'
```

Because those templates are evaluated with CEL against the project's parameters and per-environment configs, a platform engineer can expose the allowlist as a project parameter, rather than hard-coding it into the template. Developers then declare which hosts they need; the platform engineer controls the shape of the policy those declarations produce, and production can end up stricter than development. It’s the same derive-and-govern split described earlier for ingress, applied to egress using a general-purpose mechanism instead of a purpose-built one.

That distinction matters: this is a pattern, not a native abstraction. The platform engineer writes and maintains CiliumNetworkPolicy directly, and unlike endpoint visibility it does not fall back to plain NetworkPolicies on a non-Cilium data plane, since FQDN-based egress is a Cilium-specific capability. A first-class egress abstraction is on the roadmap, and the identity foundation is what makes that addition incremental rather than architectural.

##### Identity that enforces zero trust also makes the network observable

Identifying workloads by platform identity instead of IP has a useful side effect: the network observability data arrives labelled with concepts developers actually use.

Because Hubble sees every flow and knows each endpoint's project, component, and environment, OpenChoreo can render network behaviour in the developer's own context rather than as a table of IP addresses. The [Cilium networking module](https://openchoreo.dev/ecosystem/item/networking-cilium/) for OpenChoreo surfaces the following network observability features:

- A runtime cell diagram showing which components are genuinely talking to each other, derived from live Hubble flows.
- Real-time wirelogs: a live stream of Hubble flow events, scoped per component, with source and destination resolved to platform identities rather than pod IPs.
- HTTP throughput and latency metrics per component, derived from Hubble's HTTP metrics.

> _Fig 2: Live component-to-component traffic within a cell_

> _Fig 3: Streaming network wirelogs for a component_

> _Fig 4: HTTP throughput and latency per component_

Teams often hesitate to lock things down because they can't tell what depends on what. Seeing the actual traffic removes the guesswork, and that's what makes a policy safe to tighten.

##### Conclusion: Secure defaults, full velocity

Zero trust doesn't have to mean zero velocity. The two only conflict when security is a task appended to the end of the delivery path instead of a property of the path itself.

Identity-based policy makes the enforcement model durable enough to automate; a platform with real abstractions makes the automation invisible. Put them together and a developer declaring an endpoint's visibility produces an eBPF-enforced, default-deny, identity-based network boundary that survives every reschedule their workload will ever experience. All without the developer knowing the complexity underneath.

##### References

- [Cilium documentation](https://docs.cilium.io/) — eBPF datapath, identities, and network policy reference
- [OpenChoreo Cilium networking module](https://openchoreo.dev/ecosystem/item/networking-cilium/) — installation, configuration, and compatibility matrix
- [OpenChoreo runtime model](https://openchoreo.dev/docs/concepts/runtime-model/) — cells, gateway topology, and isolation boundaries
- [OpenChoreo repository](https://github.com/openchoreo/openchoreo)

#### Troubleshooting Kubernetes Networking

URL: https://cilium.io/blog/2026/08/16/troubleshooting-kubernetes-networking
Date: 2026-08-16
Categories: Networking, Technology

##### Troubleshooting Kubernetes Networking

##### 1.0 Introduction

Networking is notoriously one of the most common failure points encountered in workloads deployed within a Kubernetes environment. Because a cluster consists of hundreds or thousands of ephemeral compute resources (pods, nodes, and services) constantly spinning up, shutting down, and scaling, the underlying network layer bears the immense responsibility of keeping everything connected. When an application breaks in Kubernetes, **_it is probably a network issue_**.

Before container orchestration, it was somewhat easy to troubleshoot outages led by networking challenges. Servers had static IP addresses, physical switches had predictable ports, and firewalls relied on static rules. Kubernetes completely shatters this model.
The challenges of debugging Kubernetes networks stem directly from its highly dynamic architecture:

- **Ephemeral IP Addresses**: Pod IPs change every time a deployment is scaled, restarted, or rescheduled to a new node. Tracking down a connection failure using a raw IP address from an hour-old log file is often useless, as that IP may now belong to a completely different application.
- **Complex Abstractions**: Traffic routing relies on overlapping, virtualized abstractions like ClusterIPs, NodePorts, Ingress controllers, and CoreDNS.
- **The iptables Black Box**: Historically, Kubernetes relied heavily on kube-proxy and massive, sequential lists of iptables rules to route traffic. When a packet is unexpectedly dropped, tracing it through thousands of generic rules to find the exact misconfiguration is incredibly difficult and completely lacks appliscreen readercation-level context.

Misconfiguration is consistently reported as the leading source of Kubernetes incidents [\[Red Hat 2024\]](https://www.redhat.com/en/resources/kubernetes-adoption-security-market-trends-overview), while networking and DNS faults, though less frequent, are disproportionately difficult to localise.

##### 2.0 OSI Stack

Kubernetes abstracts away physical infrastructure, but the OSI model does not disappear; it shifts. Every packet sent between pods, services, nodes, or external clients still traverses all seven layers. What changes is who owns each layer: the Linux kernel, the CNI plugin, kube-proxy (or its replacement), CoreDNS, Ingress controllers, and application code. Each operates at different OSI levels, and when something breaks, the layer boundary is usually where you start your investigation.

Cilium's datapath is unique because it operates simultaneously at Layers 2-7 through kernel hooks, including TC (traffic control), XDP (Layer 2/3), kprobes, and socket-level filters (Layer 4), as well as Envoy-integrated Layer 7 parsing. This means a single Cilium-equipped cluster can surface and debug issues at every OSI layer through one unified toolchain: Hubble, cilium-dbg, and the BPF map inspection commands.

_Figure 1: OSI stack. Image credit: Cloudflare, "What is the OSI Model?" Cloudflare Learning Center. [https://www.cloudflare.com/learning/ddos/glossary/open-systems-interconnection-model-osi/](https://www.cloudflare.com/learning/ddos/glossary/open-systems-interconnection-model-osi/)_

###### Physical Layer: Node and NIC Failures

Although Kubernetes doesn't manage physical hardware directly, L1 failures surface immediately in cluster networking.

###### Symptoms

- Node becomes NotReady in `kubectl get nodes`
- All pods on the affected node become unreachable
- cilium status reports Cluster health: 0/N reachable for the affected node

Common causes in cloud and on-prem

- MTU mismatch between the host NIC and the overlay tunnel. Cilium's VXLAN encapsulation adds 50 bytes of overhead. If the underlay MTU is 1500 and Cilium uses 1500 for pod traffic, frames are silently dropped during fragmentation.
- NIC queue saturation on high-throughput nodes.

[Read More about MTU Overhead](https://docs.cilium.io/en/stable/network/concepts/routing/)

Cilium Commands

`# Inspect the route MTU on the cilium_host interface`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- ip link show cilium_host
4: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 5e:9d:ba:cf:a2:84 brd ff:ff:ff:ff:ff:ff

```

`# Check MTU configuration used by Cilium`

```shell
root@server:~# kubectl -n kube-system exec -it ds/cilium -- cilium status --verbose | grep -i mtu
                                           │   ├── mtu
                                           │   │   ├── job-endpoint-mtu-updater                        [OK] Endpoint MTU updated (14m, x1)
                                           │   │   └── job-mtu-updater                                 [OK] MTU updated (1500) (14m, x1)

```

`# Verify MTU on the overlay interface (if using tunnel mode)`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- ip link show cilium_vxlan
5: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/ether 3a:0d:29:2d:1f:14 brd ff:ff:ff:ff:ff:ff

```

###### Data Link: ARP, MAC Resolution, Bridge Issues

With symptoms like pods on the same node failing to communicate, and intermittent packet loss between pods on adjacent nodes.

Common causes

- ARP flux: nodes with multiple interfaces can reply to ARP requests from the wrong interface, causing asymmetric routes.
- Bridge netfilter disabled: if net.bridge.bridge-nf-call-iptables is off, kube-proxy-based clusters break at L2/L3 boundary (less relevant with Cilium's full eBPF replacement, but still matters during migration)
- Cilium is using veth device mode, where the veth peer goes missing after pod restart, due to race conditions.

`# List all endpoints and their MAC / IP state`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6   IPv4           STATUS
           ENFORCEMENT        ENFORCEMENT
2179       Disabled           Disabled          1          reserved:host                                       ready
2216       Disabled           Disabled          8          reserved:ingress                     10.244.2.108   ready
2450       Disabled           Disabled          4          reserved:health                      10.244.2.19    ready
root@server:~#

```

`# Check if the veth pair for a specific endpoint is intact`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- ip link show | grep lxc
7: lxc_health@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
root@server:~#

```

###### Network Layer: IP Routing, IPAM, Overlay Tunnels

This is the most frequently hit layer in Kubernetes networking incidents. Most "pod can't reach pod on another node" issues live here.

Most of the time, issues in this layer are characterised by a timeout in ping between two pods, an internode traffic drop while intra-node traffic is ok, and CoreDNS unreachable with a timeout.

Common causes:

- IPAM exhaustion: Cilium's default cluster-pool IPAM allocates pod CIDRs per node. If the CIDR is exhausted, new pods fail to launch with IP assignment errors.
- Missing or stale BPF route entries after node restarts.
- VXLAN tunnel mismatch: Two nodes on different overlay VTEP configurations can't establish a tunnel.
- IP masquerade misconfigured: pods can reach the cluster network but not external services because SNAT rules aren't applied correctly.

Cilium Commands
`# Check IPAM allocation state`
`kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ipmasq list`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ipmasq list
No entries found.
root@server:~#

```

`# Inspect the IP address manager`
`kubectl -n kube-system exec ds/cilium -- cilium-dbg ip list`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg ip list
IP                IDENTITY                                                                            SOURCE
0.0.0.0/0         reserved:world
10.244.0.0/24     reserved:world
10.244.1.0/24     reserved:world
10.244.0.76/32    reserved:ingress
10.244.0.127/32   reserved:remote-node
10.244.0.139/32   reserved:health
10.244.1.112/32   reserved:remote-node
10.244.1.230/32   reserved:health
10.244.1.251/32   reserved:ingress
10.244.2.17/32    k8s:app=local-path-provisioner                                                      custom-resource
                  k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=local-path-storage
                  k8s:io.cilium.k8s.policy.cluster=kind-kind
                  k8s:io.cilium.k8s.policy.serviceaccount=local-path-provisioner-service-account
                  k8s:io.kubernetes.pod.namespace=local-path-storage
10.244.2.72/32    reserved:host
10.244.2.73/32    reserved:ingress
10.244.2.76/32    k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system          custom-resource
                  k8s:io.cilium.k8s.policy.cluster=kind-kind
                  k8s:io.cilium.k8s.policy.serviceaccount=coredns
                  k8s:io.kubernetes.pod.namespace=kube-system
                  k8s:k8s-app=kube-dns
10.244.2.127/32   reserved:health
10.244.2.149/32   k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system          custom-resource
                  k8s:io.cilium.k8s.policy.cluster=kind-kind
                  k8s:io.cilium.k8s.policy.serviceaccount=coredns
                  k8s:io.kubernetes.pod.namespace=kube-system
                  k8s:k8s-app=kube-dns
172.18.0.2/32     reserved:remote-node
172.18.0.3/32     reserved:kube-apiserver
                  reserved:remote-node
172.18.0.4/32     reserved:host
root@server:~#

```

`# List all BPF routing entries`
`kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ipcache list`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ipcache list
IP PREFIX/ADDRESS   IDENTITY
10.244.0.139/32     identity=4 encryptkey=0 tunnelendpoint=172.18.0.3 flags=hastunnel
10.244.1.0/24       identity=2 encryptkey=0 tunnelendpoint=172.18.0.2 flags=hastunnel
10.244.2.72/32      identity=1 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
10.244.0.0/24       identity=2 encryptkey=0 tunnelendpoint=172.18.0.3 flags=hastunnel
10.244.1.112/32     identity=6 encryptkey=0 tunnelendpoint=172.18.0.2 flags=hastunnel
10.244.2.17/32      identity=3388 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
172.18.0.2/32       identity=6 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
172.18.0.3/32       identity=7 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
172.18.0.4/32       identity=1 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
10.244.1.230/32     identity=4 encryptkey=0 tunnelendpoint=172.18.0.2 flags=hastunnel
10.244.2.73/32      identity=8 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
10.244.2.149/32     identity=48339 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
10.244.0.76/32      identity=8 encryptkey=0 tunnelendpoint=172.18.0.3 flags=hastunnel
10.244.0.127/32     identity=6 encryptkey=0 tunnelendpoint=172.18.0.3 flags=hastunnel
10.244.1.251/32     identity=8 encryptkey=0 tunnelendpoint=172.18.0.2 flags=hastunnel
10.244.2.76/32      identity=48339 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
10.244.2.127/32     identity=4 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
0.0.0.0/0           identity=2 encryptkey=0 tunnelendpoint=0.0.0.0 flags=<none>
root@server:~#

```

`# Check the ipcache — maps pod IPs to identities and tunnel endpoints`
`kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ipcache get <pod-ip>`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ipcache get 10.244.1.45
10.244.1.45 maps to identity identity=6026 encryptkey=0 tunnelendpoint=172.18.0.2 flags=hastunnel
root@server:~#

```

`# Verify tunnel map (VXLAN/Geneve mode)`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg node list
Name                           IPv4 Address   Endpoint CIDR   IPv6 Address   Endpoint CIDR   Source
kind-kind/kind-control-plane   172.18.0.3     10.244.0.0/24                                  custom-resource
kind-kind/kind-worker          172.18.0.4     10.244.2.0/24                                  local
kind-kind/kind-worker2         172.18.0.2     10.244.1.0/24                                  custom-resource
root@server:~#

```

`# Run Cilium's built-in connectivity test`

```shell
root@server:~# cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [kind-kind] Creating namespace cilium-test-1 for connectivity check...
✨ [kind-kind] Deploying echo-same-node service...
✨ [kind-kind] Deploying DNS test server configmap...
✨ [kind-kind] Deploying same-node deployment...
✨ [kind-kind] Deploying client deployment...
✨ [kind-kind] Deploying client2 deployment...
✨ [kind-kind] Deploying client3 deployment...
✨ [kind-kind] Deploying echo-other-node service...
✨ [kind-kind] Deploying other-node deployment...
✨ [host-netns] Deploying kind-kind daemonset...
✨ [host-netns-non-cilium] Deploying kind-kind daemonset...
ℹ️  Skipping tests that require a node Without Cilium
⌛ [kind-kind] Waiting for deployment cilium-test-1/client to become ready...

```

Hubble (L3 drops)

`# Watch all dropped flows at L3 with reason`

```shell
root@server:~# hubble observe --type drop --last 200 -o jsonpb | jq '.flow | {src: .source.pod_name, dst: .destination.pod_name, reason: .drop_reason_desc}'
{
  "src": null,
  "dst": null,
  "reason": "UNSUPPORTED_L3_PROTOCOL"
}
{
  "src": null,
  "dst": null,
  "reason": "UNSUPPORTED_L3_PROTOCOL"
}

```

`# Watch traffic between two specific pods`

```shell
root@server:~# hubble observe --from-pod default/frontend --to-pod default/backend --follow

```

######

###### Transport Layer: TCP/UDP Services

Network issues in this layer are characterised by failure in TCP connections despite being able to ping it, NodePort services being reachable from only a few nodes but not others, uneven distribution of traffic in Load Balancer services, and exhausted connections in the nf_conntrack table.
Common Causes:

- Cilium service map out of sync with Kubernetes Endpoints: endpoints change faster than Cilium's reconciliation loop, especially during rolling deploys.
- kube-proxy running alongside Cilium in partial replacement mode, creating conflicting NAT rules.
- BPF NAT map full (default 524288 entries) on very high-connection-rate nodes
- Session affinity misconfiguration is causing stickiness to terminate pods.

Cilium Commands

`# List all Cilium-managed services and their backends`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg service list
ID   Frontend                 Service Type   Backend
1    10.96.0.1:443/TCP        ClusterIP      1 => 172.18.0.4:6443/TCP (active)
2    0.0.0.0:30464/TCP        NodePort
4    0.0.0.0:30977/TCP        NodePort
6    10.96.183.162:80/TCP     ClusterIP
7    10.96.183.162:443/TCP    ClusterIP
8    10.96.248.144:443/TCP    ClusterIP      1 => 172.18.0.3:4244/TCP (active)
9    0.0.0.0:31234/TCP        NodePort       1 => 10.244.1.248:4245/TCP (active)
11   10.96.218.16:80/TCP      ClusterIP      1 => 10.244.1.248:4245/TCP (active)
12   0.0.0.0:31235/TCP        NodePort       1 => 10.244.1.174:8081/TCP (active)
14   10.96.108.211:80/TCP     ClusterIP      1 => 10.244.1.174:8081/TCP (active)
15   10.96.0.10:53/TCP        ClusterIP      1 => 10.244.1.82:53/TCP (active)
                                             2 => 10.244.1.102:53/TCP (active)
16   10.96.0.10:53/UDP        ClusterIP      1 => 10.244.1.82:53/UDP (active)
                                             2 => 10.244.1.102:53/UDP (active)
17   10.96.0.10:9153/TCP      ClusterIP      1 => 10.244.1.82:9153/TCP (active)
                                             2 => 10.244.1.102:9153/TCP (active)
18   172.18.255.200:80/TCP    LoadBalancer
19   172.18.255.200:443/TCP   LoadBalancer
20   10.96.141.89:80/TCP      ClusterIP      1 => 10.244.2.163:80/TCP (active)
root@server:~#

```

`# Check a specific service's backend health`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg service list
ID   Frontend                 Service Type   Backend
1    10.96.0.1:443/TCP        ClusterIP      1 => 172.18.0.4:6443/TCP (active)
2    0.0.0.0:30464/TCP        NodePort
4    0.0.0.0:30977/TCP        NodePort
6    10.96.183.162:80/TCP     ClusterIP
7    10.96.183.162:443/TCP    ClusterIP
8    10.96.248.144:443/TCP    ClusterIP      1 => 172.18.0.3:4244/TCP (active)
9    0.0.0.0:31234/TCP        NodePort       1 => 10.244.1.248:4245/TCP (active)
11   10.96.218.16:80/TCP      ClusterIP      1 => 10.244.1.248:4245/TCP (active)
12   0.0.0.0:31235/TCP        NodePort       1 => 10.244.1.174:8081/TCP (active)
14   10.96.108.211:80/TCP     ClusterIP      1 => 10.244.1.174:8081/TCP (active)
15   10.96.0.10:53/TCP        ClusterIP      1 => 10.244.1.82:53/TCP (active)
                                             2 => 10.244.1.102:53/TCP (active)
16   10.96.0.10:53/UDP        ClusterIP      1 => 10.244.1.82:53/UDP (active)
                                             2 => 10.244.1.102:53/UDP (active)
17   10.96.0.10:9153/TCP      ClusterIP      1 => 10.244.1.82:9153/TCP (active)
                                             2 => 10.244.1.102:9153/TCP (active)
18   172.18.255.200:80/TCP    LoadBalancer
19   172.18.255.200:443/TCP   LoadBalancer
20   10.96.141.89:80/TCP      ClusterIP      1 => 10.244.2.163:80/TCP (active)
root@server:~#

```

`# Inspect the BPF load balancer map (frontend VIP → backends)`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list
SERVICE ADDRESS              BACKEND ADDRESS (REVNAT_ID) (SLOT)
10.96.218.16:80/TCP (1)      10.244.1.248:4245/TCP (11) (1)
10.96.248.144:443/TCP (1)    172.18.0.3:4244/TCP (8) (1)
10.96.141.89:80/TCP (1)      10.244.2.163:80/TCP (20) (1)
10.96.0.10:53/TCP (0)        0.0.0.0:0 (15) (0) [ClusterIP, non-routable]
10.96.108.211:0/ANY (0)      0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
0.0.0.0:31235/TCP (1)        10.244.1.174:8081/TCP (12) (1)
10.96.141.89:0/ANY (0)       0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
172.18.0.3:31234/TCP (0)     0.0.0.0:0 (10) (0) [NodePort]
172.18.0.3:31235/TCP (1)     10.244.1.174:8081/TCP (13) (1)
....
```

`# Check BPF NAT table (active SNAT mappings)`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf nat list
TCP OUT 172.18.0.3:34178 -> 54.234.221.194:443 XLATE_SRC 172.18.0.3:34178 Created=408sec ago NeedsCT=1
TCP IN 54.234.221.194:443 -> 172.18.0.3:34178 XLATE_DST 172.18.0.3:34178 Created=408sec ago NeedsCT=1
TCP IN 172.18.0.4:6443 -> 172.18.0.3:58624 XLATE_DST 172.18.0.3:58624 Created=667sec ago NeedsCT=1
TCP OUT 172.18.0.3:59112 -> 172.18.0.4:4240 XLATE_SRC 172.18.0.3:59112 Created=297sec ago NeedsCT=1
TCP OUT 172.18.0.3:54010 -> 13.249.228.49:443 XLATE_SRC 172.18.0.3:54010 Created=406sec ago NeedsCT=1

```

`# Check connection tracking table`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ct list global
TCP OUT 172.18.0.3:58624 -> 172.18.0.4:6443 expires=19466 Packets=0 Bytes=0 RxFlagsSeen=0x18 LastRxReport=11461 TxFlagsSeen=0x1a LastTxReport=11461 Flags=0x0010 [ SeenNonSyn ] RevNAT=0 SourceSecurityID=0 BackendID=0 NatPort=0
TCP OUT 10.244.2.148:38566 -> 10.244.0.34:4240 expires=19462 Packets=0 Bytes=0 RxFlagsSeen=0x1a LastRxReport=11462 TxFlagsSeen=0x1a LastTxReport=11462 Flags=0x0010 [ SeenNonSyn ] RevNAT=0 SourceSecurityID=0 BackendID=0 NatPort=0

```

`# Compare Kubernetes services with what Cilium has programmed`
`# (discrepancies = sync lag or bug)`

```shell
root@server:~# kubectl get svc --all-namespaces
NAMESPACE     NAME              TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default       backend-service   ClusterIP      10.96.141.89    <none>           80/TCP                       10m
default       kubernetes        ClusterIP      10.96.0.1       <none>           443/TCP                      3h11m
kube-system   cilium-envoy      ClusterIP      None            <none>           9964/TCP                     15m
kube-system   cilium-ingress    LoadBalancer   10.96.183.162   172.18.255.200   80:30977/TCP,443:30464/TCP   15m
kube-system   hubble-metrics    ClusterIP      None            <none>           9965/TCP                     15m
....
root@server:~#

```

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg service list
ID   Frontend                 Service Type   Backend
1    10.96.0.1:443/TCP        ClusterIP      1 => 172.18.0.4:6443/TCP (active)
2    0.0.0.0:30464/TCP        NodePort
4    0.0.0.0:30977/TCP        NodePort
6    10.96.183.162:80/TCP     ClusterIP
7    10.96.183.162:443/TCP    ClusterIP
8    10.96.248.144:443/TCP    ClusterIP      1 => 172.18.0.3:4244/TCP (active)
9    0.0.0.0:31234/TCP        NodePort       1 => 10.244.1.248:4245/TCP (active)
11   10.96.218.16:80/TCP      ClusterIP      1 => 10.244.1.248:4245/TCP (active)
12   0.0.0.0:31235/TCP        NodePort       1 => 10.244.1.174:8081/TCP (active)
14   10.96.108.211:80/TCP     ClusterIP      1 => 10.244.1.174:8081/TCP (active)
15   10.96.0.10:53/TCP        ClusterIP      1 => 10.244.1.82:53/TCP (active)
                                             2 => 10.244.1.102:53/TCP (active)
...
```

Hubble for L4 Drops

`# Watch TCP-specific drops`

```shell
root@server:~# hubble observe --type drop --protocol TCP --follow

```

`# Filter by specific port`

```shell
root@server:~# hubble observe --to-port 80 --follow

```

`# Check policy verdicts at L4`

```shell
root@server:~# hubble observe --type policy-verdict --follow

```

###### Session Layer: Connection Tracking and Keep-Alive Issues

Kubernetes doesn't surface a dedicated session layer, but TCP session management is a real operational concern in long-lived microservice connections.

Connections appear active in the application but not in BPF ConTrack Maps.

**Common Causes:**

- BPF connection tracking idle timeout (default 300s for TCP established) terminates long-lived connections silently.
- NAT timeout is too aggressive for idle database connections.
- TCP keep-alive is not configured at the application level, and connections silently close mid-stream.

Cilium Commands

`# View CT timeouts and current CT table`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf ct list global | head -50
TCP IN 10.244.1.46:56068 -> 10.244.2.219:4240 expires=11750 Packets=0 Bytes=0 RxFlagsSeen=0x1b LastRxReport=11740 TxFlagsSeen=0x1b LastTxReport=11740 Flags=0x0413 [ RxClosing TxClosing SeenNonSyn FromTunnel ] RevNAT=0 SourceSecurityID=6 BackendID=0 NatPort=0
TCP OUT 10.244.2.148:44772 -> 10.244.1.106:4240 expires=11711 Packets=0 Bytes=0 RxFlagsSeen=0x1b LastRxReport=11701 TxFlagsSeen=0x1b LastTxReport=11701 Flags=0x0013 [ RxClosing TxClosing SeenNonSyn ] RevNAT=0 SourceSecurityID=0 BackendID=0 NatPort=0

```

`# Check Cilium config for current CT settings`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg config | grep -i conntrack
ConntrackAccounting               : Disabled
root@server:~#

```

###### Presentation Layer: TLS, mTLS, Certificate Issues

This is normally characterised by TLS handshake timeout or certificate verification failed; mTLS between pods fails after certificate rotation.

Common Causes

- Cilium WireGuard encryption is enabled on some nodes but not all (mixed encryption state).
- certificate expiry in the SPIFFE/SPIRE integration causes identity verification failures.
- Ingress TLS secret missing or expired, causing 503 from the Ingress controller.

Commands

`# Check WireGuard encryption status on all nodes`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg encrypt status
Encryption: Disabled
root@server:~#

```

`# Check node-to-node encryption state`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg node list
Name                           IPv4 Address   Endpoint CIDR   IPv6 Address   Endpoint CIDR   Source
kind-kind/kind-control-plane   172.18.0.4     10.244.0.0/24                                  custom-resource
kind-kind/kind-worker          172.18.0.3     10.244.2.0/24                                  local
kind-kind/kind-worker2         172.18.0.2     10.244.1.0/24                                  custom-resource
root@server:~#

```

Hubble for TLS Drops

```shell
# Watch flows with encryption status
hubble observe --follow -o jsonpb | jq 'select(.flow.is_reply == false) | {src: .flow.source.pod_name, dst: .flow.destination.pod_name, encrypted: .flow.traffic_direction}'

{
  "src": null,
  "dst": "hubble-relay-5bbf66658c-k6958",
  "encrypted": "INGRESS"
}
{
  "src": null,
  "dst": "hubble-relay-5bbf66658c-k6958",
  "encrypted": "INGRESS"
}

```

###### Application Layer: HTTP, gRPC, DNS, FQDN Policies

This is where Cilium visibly exceeds. Cilium extends network policy enforcement to Layers 3, 4, and 7, enabling more granular control over ingress and egress traffic based on application behavior, including DNS support and mTLS.

The following are the common challenges in the application layer.

- Misconfigured Network Policies are a common issue
- DNS resolution failure

Symptoms

- HTTP 403/404 responses that only affect specific paths, not the whole service
- DNS resolution fails for external FQDNs, but pod IPs are reachable
- gRPC calls time out on specific methods but not others
- CiliumNetworkPolicy with HTTP rules is blocking traffic that should be allowed

`# List all endpoints with L7 policy`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                              IPv6   IPv4           STATUS
           ENFORCEMENT        ENFORCEMENT
180        Disabled           Disabled          10064      k8s:app=frontend                                                                10.244.2.199   ready
                                                           k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
                                                           k8s:io.cilium.k8s.policy.cluster=kind-kind
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default

```

`# Inspect FQDN cache (is the DNS entry resolved and cached?)`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs
root@server:~#

```

`# Watch all HTTP flows cluster-wide (requires L7 visibility enabled)`

```shell
root@server:~# hubble observe --protocol http --follow

```

`# Watch DNS queries and responses`

```shell
root@server:~# hubble observe --protocol dns --follow

```

`# Trace a specific pod's HTTP traffic`

```shell
root@server:~# hubble observe --from-pod default/frontend --protocol http --follow

```

`# Watch TCP specifically`

```shell
root@server:~# hubble observe --protocol tcp --follow
Jun 24 23:10:00.032: 172.18.0.3:58624 (host) -> 172.18.0.4:6443 (kube-apiserver) to-network FORWARDED (TCP Flags: ACK, PSH)
Jun 24 23:10:00.286: 172.18.0.3:58636 (host) -> 172.18.0.4:6443 (kube-apiserver) to-network FORWARDED (TCP Flags: ACK, PSH)
Jun 24 23:10:00.669: 10.244.0.99:37982 (world) -> kube-system/hubble-relay-5bbf66658c-k6958:4245 (ID:14331) to-overlay FORWARDED (TCP Flags: ACK, PSH)

```

`# Show L7 policy verdicts (ALLOW vs DENY with path detail)`

```shell
root@server:~# hubble observe --type policy-verdict --protocol http --follow

```

###### Cross-layer Challenges Common in Kubernetes

These challenges often happen when failures occur across different layers of the stack.

- MTU Cascade: Incorrect MTU settings can result in packet drops and performance degradation. This is particularly difficult to diagnose, as it might only happen when transferring large payloads. To detect this in your cluster, prioritise using Hubble to monitor and analyse your network traffic. It is important to verify Cilium MTU configuration.
- Policy Drift: This occurs when the current state of your cluster configuration deviates from the defined state as specified in code templates. To detect this in your cluster, prioritise using Hubble to monitor.

##### 2.0 The Cilium Diagnostic Toolkit

Cilium provides a comprehensive set of built-in troubleshooting tools that leverage its eBPF data plane to expose information that is impossible to obtain with traditional networking tools. When a pod can't connect to a service, traceroute and tcpdump provide incomplete answers because they operate outside the kernel eBPF context where Cilium performs its work.

`# Overall Healthcheck`

```shell
root@server:~# cilium status --verbose
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet              cilium-envoy             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-relay             Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-ui                Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 3
                       cilium-envoy             Running: 3
                       cilium-operator          Running: 1
                       clustermesh-apiserver
                       hubble-relay             Running: 1
                       hubble-ui                Running: 1

```

`# Check all Cilium pods`

```shell
root@server:~# kubectl get pods -n kube-system -l k8s-app=cilium
NAME           READY   STATUS    RESTARTS   AGE
cilium-5pks9   1/1     Running   0          8m50s
cilium-rl6n5   1/1     Running   0          8m50s
cilium-wdvhp   1/1     Running   0          8m50s
root@server:~#

```

`# Perform endpoint inspection`

```shell
root@server:~# kubectl exec -n kube-system  cilium-6vxzs -- cilium-dbg endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                   IPv6   IPv4           STATUS
           ENFORCEMENT        ENFORCEMENT
126        Disabled           Disabled          8          reserved:ingress                                                     10.244.0.14    ready
839        Disabled           Disabled          4          reserved:health                                                      10.244.0.196   ready
2424       Disabled           Disabled          1          k8s:node-role.kubernetes.io/control-plane                                           ready
                                                           k8s:node.kubernetes.io/exclude-from-external-load-balancers
                                                           reserved:host
root@server:~#

root@server:~# kubectl exec -n kube-system  cilium-6vxzs -- cilium-dbg endpoint get 839
[
  {
    "id": 839,
    "spec": {
      "label-configuration": {},
      "options": {
        "ConntrackAccounting": "Disabled",
        "Debug": "Disabled",
        "DebugLB": "Disabled",
        "DebugPolicy": "Disabled",
        "DropNotification": "Enabled",
        "MonitorAggregationLevel": "Medium",
        "PolicyAccounting": "Enabled",
        "PolicyAuditMode": "Disabled",
        "PolicyVerdictNotification": "Enabled",
        "SourceIPVerification": "Enabled",
        "TraceNotification": "Enabled"
      }
    },
...

```

`# Running Connectivity Test`

```shell
root@server:~# cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [kind-kind] Creating namespace cilium-test-1 for connectivity check...
✨ [kind-kind] Deploying echo-same-node service...
✨ [kind-kind] Deploying DNS test server configmap...
✨ [kind-kind] Deploying same-node deployment...
✨ [kind-kind] Deploying client deployment...
✨ [kind-kind] Deploying client2 deployment...
✨ [kind-kind] Deploying client3 deployment...
✨ [kind-kind] Deploying echo-other-node service...
✨ [kind-kind] Deploying other-node deployment...
✨ [host-netns] Deploying kind-kind daemonset...
✨ [host-netns-non-cilium] Deploying kind-kind daemonset...
ℹ️  Skipping tests that require a node Without Cilium
✨ [kind-kind] Deploying Ingress resource...

```

_Figure 2: Cilium connectivity troubleshooting decision tree_

Policy Troubleshooting

Cilium offers command-line utilities to interact with and troubleshoot network policies. The infrastructure team can simulate the policy decision for a specific source or destination port combination and show exactly which rule is permitting or denying the flow, without sending real traffic.

`# Get an effective policy for an endpoint`

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg policy get
[]
Revision: 1
root@server:~#
```

Log Inspection

Logs are the audit trail for everything the agent tried and failed to do. Cilium agent logs are verbose by default; the key is filtering for actionable signals rather than reading every line.

`# Get Cilium logs`

```shell
root@server:~# kubectl -n kube-system logs -f ds/cilium -c cilium-agent
Found 3 pods, using pod/cilium-6th6g
time=2026-06-27T14:00:11.756428436Z level=info msg="Memory available for map entries (0.250% of 8322646016B): 20806615B"
time=2026-06-27T14:00:11.756524087Z level=info msg="option bpf-ct-global-tcp-max set by dynamic sizing to 131072"
time=2026-06-27T14:00:11.756538881Z level=info msg="option bpf-ct-global-any-max set by dynamic sizing to 65536"
...
```

Endpoint regeneration log lines are particularly useful. Every time a pod's policy changes (new NetworkPolicy, label update, pod restart), Cilium regenerates the endpoint's eBPF programs. A regeneration failure means the pod's policy is stale.

##### 3.0 Cluster Network State

One of the most dangerous failure modes in Kubernetes networking is the one that looks like nothing. Pods schedule, nodes appear Ready, **_kubectl get pods_** shows green, but traffic silently fails, DNS resolutions time out, or cross-namespace calls drop without a clear error.

Silence in Cilium can mean:

- A controller is reconciling but silently erroring. Controllers retry on failure and don't surface errors unless you explicitly check controller status. Run **_cilium-dbg status \--verbose_** and look for controllers with a non-zero failure count.
- Hubble flows are missing. If Hubble shows no flows for a pod that should be receiving traffic, the eBPF program may not be loaded on that node, or the endpoint hasn't been picked up by the agent. Check the cilium-dbg endpoint list and verify the pod's endpoint exists.
- eBPF programs aren't attached. The agent may be running but failing to attach programs to a new interface. This produces no obvious error in kubectl get pods, but traffic will bypass Cilium entirely. Verify with cilium-dbg bpf metrics list.
- KVStore lag. If you're using etcd as the KVStore, a lagging or partitioned etcd means node identities and policies are stale. Traffic may still flow based on cached state until the cache expires.

###### Cilium Agent Health Check

The Cilium agent runs as a DaemonSet pod. A pod in Running state doesn't guarantee the agent is fully functional; it means the container started. True health requires checking what the agent reports about itself.

**Check agent status across all nodes.**

```shell
root@server:~# kubectl -n kube-system get pods -l k8s-app=cilium -o wide
NAME           READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
cilium-9p44h   1/1     Running   0          86s   172.18.0.4   kind-worker2         <none>           <none>
cilium-hlntk   1/1     Running   0          86s   172.18.0.2   kind-worker          <none>           <none>
cilium-jzkj4   1/1     Running   0          86s   172.18.0.3   kind-control-plane   <none>           <none>
root@server:~#

```

Look for OOMKill events, failed readiness probes, and init container failures. Cilium's init container (cilium-init) is responsible for setting up the host network before the main agent starts. Failures here mean the agent never reached a healthy state.

**Check the Cilium operator.**

```shell
root@server:~# kubectl -n kube-system get pods -l name=cilium-operator
NAME                               READY   STATUS    RESTARTS   AGE
cilium-operator-57f4d55695-gvhln   1/1     Running   0          5m19s
root@server:~#

root@server:~# kubectl -n kube-system logs deployment/cilium-operator --tail=100
time=2026-06-27T15:15:04.847030224Z level=info msg="Start hook executed" module=operator function=*job.groupHooks.Start duration=645ns
time=2026-06-27T15:15:04.847041107Z level=info msg="Start hook executed" module=operator function=*job.groupHooks.Start duration=572ns
```

The operator manages cluster-scoped resources, CiliumNode objects, IPAM pools, and CRD reconciliation. An unhealthy operator can leave nodes unable to acquire IP addresses, even when agent pods themselves look fine.

###### Node Readiness

A node being Ready from Kubernetes' perspective and a node being fully operational from Cilium's perspective are different things. Kubernetes marks a node ready when its kubelet is healthy; Cilium readiness depends on the eBPF datapath being set up correctly.

```shell
root@server:~# kubectl get ciliumnode kind-worker2 -o yaml
apiVersion: cilium.io/v2
kind: CiliumNode
metadata:
  creationTimestamp: "2026-06-27T15:15:03Z"
  generation: 1
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: kind-worker2
    kubernetes.io/os: linux
...

```

The CiliumNode resource represents the node from Cilium's perspective. Its status section contains the IPAM state, including which IP addresses are allocated, which are available, and whether the node has successfully registered with the Cilium operator. A CiliumNode that hasn't been created, or whose spec diverges from the actual node's configuration, is a strong signal of an operator or RBAC problem.

**Verifying eBPF programs are loaded.**

```shell
root@server:~# kubectl -n kube-system exec cilium-jzkj4 -c cilium-agent -- bpftool prog list | head -n 5
61: cgroup_device  tag 3918c82a5f4c0360
        loaded_at 2026-06-27T13:19:14+0000  uid 0
        xlated 64B  jited 45B  memlock 4096B
64: cgroup_device  tag 3918c82a5f4c0360
        loaded_at 2026-06-27T13:19:14+0000  uid 0
root@server:~#
```

##### 4.0 Pod Connectivity Problems

###### When the Pod Is Not Running

A pod that hasn't reached the running state has a networking problem that happens before networking can be verified. Don't reach for Hubble yet; the issue is earlier in the lifecycle.
Start with kubectl describe pod.

```shell
root@server:~# kubectl describe pod nginx | head -n 5
Name:             nginx
Namespace:        default
Priority:         0
Service Account:  default
Node:             kind-worker2/172.18.0.4
root@server:~#
```

In the Events section, confirm the following details:

**FailedScheduling**: the pod was never placed on a node. This is a scheduler problem, not a network problem. But if it's failing due to IPAM exhaustion, it will appear here.

**NetworkPluginNotReady**: the CNI plugin (Cilium) hasn't signaled readiness to the kubelet yet. The Cilium DaemonSet pod on that node may be crashing or not yet fully initialized.

**Failed to create pod sandbox**: the container runtime couldn't set up the network namespace. This usually means the CNI binary isn't present at /opt/cni/bin/ or the CNI config at /etc/cni/net.d/ is malformed.

###### When the Pod Is Running but Unreachable

A running pod that doesn't respond to traffic is one of the most common Cilium troubleshooting scenarios. The most likely causes are: NetworkPolicy blocking the traffic, the pod's eBPF endpoint not being regenerated after a policy change, or the pod's IP not being correctly advertised.

_First: Confirm the pod has an endpoint._

```shell
root@server:~# kubectl -n kube-system exec cilium-vnwrl  -- cilium-dbg endpoint list | grep 10.244.2.57
2043       Disabled           Disabled          64004      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default          10.244.2.57   ready
root@server:~#

```

The endpoint should be in a ready state. If it's not-ready, regenerating, or disconnected, the datapath isn't set up for that pod yet. Give it 10 \- 15 seconds; if it doesn't transition, check agent logs for regeneration errors.

###### Second: Check Hubble for dropped flows

```shell
root@server:~# hubble observe --to-pod default/nginx --verdict DROPPED --follow

```

Then attempt to connect to the pod from a source. If drops appear in Hubble, the output will include the drop reason, typically **_Policy denied_** or **_CT: missing entry_**. A Policy denied drop means a NetworkPolicy is blocking the traffic. An **_CT: missing entry_** drop means the connection tracking table doesn't have a return-path entry, which suggests asymmetric routing.

_Third: Trace the policy._

```shell
root@server:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                              IPv6   IPv4          STATUS
           ENFORCEMENT        ENFORCEMENT
381        Disabled           Disabled          54455      k8s:app=client                                                                  10.244.1.83   ready
                                                           k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
                                                           k8s:io.cilium.k8s.policy.cluster=kind-kind
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default
....

root@server:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium policy get 1994
[]
Revision: 1
root@server:~#

```

The trace output shows you the exact policy decision. If no ingress rule matches, Cilium will tell you. This is far faster than reading NetworkPolicy YAML and mentally evaluating label selectors.

_Fourth: Verify Service routing_
If you're hitting the pod through a Service, it’s important to check the BPF service map.

```shell
root@server:~# CILIUM_AGENT_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}')

kubectl -n kube-system exec $CILIUM_AGENT_POD -- cilium-dbg bpf lb list | grep 10.96.194.145
10.96.194.145:80/TCP (0)     0.0.0.0:0 (20) (0) [ClusterIP, non-routable]
10.96.194.145:0/ANY (0)      0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
root@server:~#
```

A Service that has no backend entries in the BPF load balancer table will silently drop traffic even if the pod itself is healthy.

###### Cross-Node Pod Connectivity

Cross-node failures are network-topology failures. When a pod on **Node A** can't reach a pod on **Node B**, the problem is usually in one of three layers: the underlay (physical or cloud routing), the Cilium tunnel or native routing configuration, or the eBPF programs on the receiving node.

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose | grep -E "Routing|Tunnel"
KubeProxyReplacement:   True   [eth0    172.18.0.2 fc00:f853:ccd:e793::2 fe80::b4cd:bdff:fe8f:95db (Direct Routing)]
Routing:                Network: Tunnel [vxlan]   Host: Legacy
  Devices:              eth0    172.18.0.2 fc00:f853:ccd:e793::2 fe80::b4cd:bdff:fe8f:95db (Direct Routing)
root@server:~#
```

Cilium supports two modes:

- **VXLAN / Geneve (tunnel mode);** pod traffic is encapsulated between nodes. Node-to-Node UDP connectivity on port 8472 (VXLAN) or 6081 (Geneve) must be allowed.
- **Native routing:** pod CIDR routes must be present in the underlay (via BGP, cloud routing tables, or static routes). No encapsulation means any routing gap in the underlay creates a black hole.

A missing route means the pod CIDR for the destination node hasn't been advertised. In BGP mode, check that the Cilium BGP control plane has established a session and is advertising prefixes:

```shell
kubectl -n kube-system exec <cilium-pod> -- cilium-dbg bgp peers

```

###### Running Connectivity Tests with Cilium

Rather than constructing test traffic manually, Cilium includes a built-in connectivity test suite that validates the entire network stack end-to-end. It's the fastest way to confirm that a cluster's networking is fully operational after installation, an upgrade, or a configuration change.

_Run the full connectivity test._

```shell
cilium connectivity test

```

This deploys a set of test pods across your cluster and runs a structured battery of tests, including:

- Pod-to-pod connectivity within the same node
- Pod-to-pod connectivity across nodes
- Pod-to-Service connectivity
- Pod-to-external connectivity (egress)
- NetworkPolicy enforcement (allow and deny cases)
- DNS resolution
- L7 policy enforcement (if Hubble and Envoy are enabled)

_Run a subset of tests._

```shell
# Only pod-to-pod tests
cilium connectivity test --test pod-to-pod

# Exclude tests requiring external access
cilium connectivity test --test '!external-1.1.1.1'
```

The test name tells you the scenario. The flow line tells you the exact src/dst. From here, use **_hubble observe \--verdict DROPPED_** filtered to those pods to identify the drop cause.

_Use connectivity tests as a regression gate._

After any change to NetworkPolicy, Cilium configuration, or cluster topology, run the connectivity tests before marking the change complete. A clean pass means the full network stack datapath, policy engine, identity system, DNS, and Service load balancing are working as expected across all nodes.

##### 5.0 Service and Load Balancing Connectivity Problems

Service failures in Kubernetes are deceptive. The pod may be healthy, the deployment may show all replicas ready, and yet traffic never arrives. With Cilium’s kube-proxy replacement, service load balancing runs entirely in the eBPF datapath, no iptables rules, no kube-proxy chains. That means the debugging path is different and faster than with traditional CNIs.

[Read More About Service Load Balancing](https://cilium.io/blog/2026/04/25/understanding-kubernetes-load-balancing/)

###### Is the Service Selecting Any Pod?

The first question when a Service doesn't work is whether it has any backends at all. A Service with no matching pods will accept connections and immediately drop them silently, from the caller's perspective.

```shell
root@server:~# kubectl get endpoints nginx-service
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME            ENDPOINTS         AGE
nginx-service   10.244.1.112:80   17s
root@server:~#

```

If the **_ENDPOINTS_** column is empty or shows **_\<none\>_**, the selector isn't matching any pods. Cross-check the Service's **_spec. selector_** against actual pod labels.

```shell
root@server:~# kubectl get svc nginx-service  -o jsonpath='{.spec.selector}'
{"run":"nginx"}
root@server:~#

root@server:~# kubectl get pods  --show-labels
NAME    READY   STATUS    RESTARTS   AGE     LABELS
nginx   1/1     Running   0          2m19s   run=nginx
root@server:~#

```

A single mismatched label, a typo, a version suffix, or a missing key is enough to produce zero backends. The Service exists, the pods exist, but they never find each other.

Also, verify pods are actually running and passing readiness probes. A pod in Running but failing its readiness check will be excluded from EndpointSlices automatically.

```shell
root@server:~# kubectl get pods  --show-labels
NAME    READY   STATUS    RESTARTS   AGE     LABELS
nginx   1/1     Running   0          2m19s   run=nginx

```

###### ClusterIP Not Responding

You can validate the correct datapath state by running cilium-dbg bpf lb list to inspect the contents of the eBPF load balancing maps.

[Read More About Cilium Troubleshooting](https://docs.cilium.io/en/stable/operations/troubleshooting/)

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep 10.96.69.57
10.96.69.57:80/TCP (1)       10.244.1.112:80/TCP (20) (1)
10.96.69.57:80/TCP (0)       0.0.0.0:0 (20) (0) [ClusterIP, non-routable]
10.96.69.57:0/ANY (0)        0.0.0.0:0 (0) (0) [ClusterIP, non-routable]
root@server:~#
```

If the ClusterIP appears in the table with no backends listed, the endpoint hasn't been programmed into the BPF map. Check the Cilium operator logs and verify EndpointSlices are in sync. If the ClusterIP doesn't appear at all, the Service hasn't been picked up by Cilium's service reconciler.

Run **_cilium-dbg debuginfo_** and look for the section **_k8s-service-cache_**. In that section, you will find the contents of the service correlation cache. This cache is what Cilium uses to reconcile Kubernetes Services into BPF map entries. A stale or missing entry here explains why the datapath isn't programmed correctly.

###### NodePort Not Reachable Externally

With the help of the cilium-dbg service list command, you can validate that Cilium's eBPF kube-proxy replacement created the NodePort service. Services with the NodePort will have entries for each device on every node.

[Read More About Kube Proxy Replacement](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/)

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg service list | grep NodePort

1    0.0.0.0:31234/TCP        NodePort       1 => 10.244.2.211:4245/TCP (active)
4    0.0.0.0:31235/TCP        NodePort       1 => 10.244.2.123:8081/TCP (active)
12   0.0.0.0:30452/TCP        NodePort
14   0.0.0.0:32142/TCP        NodePort
root@server:~#
```

If NodePort entries are missing for a node interface, the \--devices configuration may not include that interface. Check the Cilium ConfigMap:

```shell
root@server:~# kubectl -n kube-system get cm cilium-config -o jsonpath='{.data.devices}'
eth0
root@server:~#
```

The device list must include the interface on which external traffic arrives. If your nodes have multiple network interfaces (common in cloud environments), an interface not listed here will not have NodePort BPF programs attached, and external traffic arriving there will be ignored.
For external traffic specifically, it’s important to verify that your cloud provider’s security groups or firewall rules allow the NodePort default range (30000-32767 TCP/UDP) from external sources.

If the container runtime in your cluster is running in cgroup namespace mode, Cilium's kube-proxy replacement-based load balancing may not be effective, leading to connectivity issues. Verify cgroup attachment.

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- bpftool cgroup tree /run/cilium/cgroupv2/
CgroupPath
ID       AttachType      AttachFlags     Name
/run/cilium/cgroupv2
734      cgroup_inet_sock_release multi           cil_sock_release
731      cgroup_inet4_post_bind multi           cil_sock4_post_bind
735      cgroup_inet6_post_bind multi           cil_sock6_post_bind
745      cgroup_inet4_connect multi           cil_sock4_connect
...

```

###### Debugging EndpointSlices

Cilium consumes EndpointSlices directly; it doesn't rely on the older Endpoints API. If EndpointSlices are missing, stale, or out of sync, the BPF load balancing table will not reflect the current backend state.

```shell
# List EndpointSlices for a service
root@server:~# kubectl get endpointslices -l kubernetes.io/service-name=nginx-service
NAME                  ADDRESSTYPE   PORTS   ENDPOINTS      AGE
nginx-service-krzb2   IPv4          80      10.244.1.112   15m
root@server:~#

# Inspect a specific slice
root@server:~# kubectl describe endpointslice nginx-service-krzb2
Name:         nginx-service-krzb2
Namespace:    default
Labels:       endpointslice.kubernetes.io/managed-by=endpointslice-controller.k8s.io
              kubernetes.io/service-name=nginx-service
              run=nginx
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2026-07-19T12:25:21Z
AddressType:  IPv4
Ports:
  Name     Port  Protocol
  ----     ----  --------
  <unset>  80    TCP
Endpoints:
  - Addresses:  10.244.1.112

```

The following details must be present in the output

- addressType: should be IPv4 or IPv6 for dual stack.
- conditions. ready: only ready endpoints will be picked up.
- targetRef: should point to an existing pod in the expected namespace.

```shell
root@server:~# kubectl get endpointslices -l kubernetes.io/service-name=nginx-service -o jsonpath='{range .items[*]}{"addressType: "}{.addressType}{"\n"}{range .endpoints[*]}{"  - targetRef: "}{.targetRef.kind}{"/"}{.targetRef.name}{"\n    ready:     "}{.conditions.ready}{"\n"}{end}{end}'
addressType: IPv4
  - targetRef: Pod/nginx
    ready:     true
root@server:~#
```

###### The Load Balancer Service Is Stuck in Pending

A LoadBalancer Service showing _\<pending\>_ in the EXTERNAL-IP column means no external IP has been assigned. The cause depends on your environment.

On bare metal or on-premise clusters using Cilium LB IPAM:
The ExternalIP field having a value of _\<pending\>_ means no LB IPs have been assigned. When LB IPAM is unable to allocate or assign IPs for the service, it will update the service conditions in the status.

To check for service status conditions:

```shell
root@server:~# kubectl describe svc nginx-service
Name:                     nginx-service
Namespace:                default
Labels:                   run=nginx
Annotations:              <none>
Selector:                 run=nginx
Type:                     ClusterIP
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.69.57
IPs:                      10.96.69.57
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
Endpoints:                10.244.1.112:80
Session Affinity:         None
Internal Traffic Policy:  Cluster
Events:                   <none>
root@server:~#
```

A reason for no_pool means no **_CiliumLoadBalancerIPPool_** has been created, or the pool's **_serviceSelector_** doesn't match this service.
LB IPAM is always enabled but dormant. The controller is only activated when the first IP Pool is added to the cluster. Create a pool if none exists

[Read More About LB IPAM](https://docs.cilium.io/en/stable/network/lb-ipam/)

###### Session Affinity

Session affinity ensures that repeated connections from the same client are routed to the same backend pod. Without it, stateful applications or anything maintaining in-memory session state will behave erratically when connections are distributed across replicas.

Cilium implements session affinity natively in eBPF. The source for the affinity depends on the origin of a request. If a request is sent from outside the cluster to the service, the request's source IP address is used for determining the endpoint affinity. If a request is sent from inside the cluster, the client's network namespace cookie is used as the source when socket-LB is in use. This allows affinity to be implemented at the socket layer, where the endpoint selection happens before a network packet has been built.

[Read More About Cilium Session Affinity](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#session-affinity)

##### 6.0 DNS Resolution Failures

DNS is a common cause for outages and incidents in Kubernetes clusters. When errors occur during DNS resolution, they are typically referred to as DNS resolution errors or DNS issues. Errors during connection to the returned IP are generally referred to as network connectivity issues. Keeping that distinction sharp saves significant debugging time; the symptoms can look identical at the application layer.

###### Running DNS Resolution Tests inside Pods

When DNS issues occur, they often go unnoticed until significant damage has been done.

The first ideal step is to check whether nslookup is working properly.

```shell
root@server:~# kubectl run dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 --command -- sleep 3600
pod/dnsutils created
root@server:~# kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

root@server:~#
```

The following are common issues associated with DNS in Kubernetes

1. Delayed response: the DNS server is slow to respond due to congestion or server overload.
2. Incorrect DNS resolutions: usually because of out-of-date or incomplete records.
3. DNS records conflict: multiple pods or services use the same name or IP.
4. DNS lookup failures: When a lookup fails, it is most often due to the server being unreachable.
5. NXDOMAIN errors: the server can't find a request domain within its records.
6. External domain failures: Kubernetes DNS can’t resolve the external domain name, likely due to failure to connect to the external DNS server.

_Figure3:DNS resolution queries_

###### Checking DNS Logs

Logs can be checked using the following command:

```shell
kubectl logs --namespace=kube-system -l k8s-app=kube-dns
```

Warnings or error messages in the logs may help you hone in on the source of your DNS problem, especially if the root of the issue lies with kube-dns itself as opposed to the network.

###### Check kube-dns Resource Utilization

Checking the CPU utilization of the kube-DNS service can help you determine whether the server is simply overloaded. To view real-time CPU metrics, run:

```shell
root@server:~# kubectl logs --namespace=kube-system -l k8s-app=kube-dns
maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
.:53
[INFO] plugin/reload: Running configuration SHA512 = 61c4e97fd46a540ac8dda919cc898bd4e91207731b7b08b141e56fbd76fd591d82a4524df658474b8e3d867076fc7ba04e95992926333c1bcfaa75c46969808c
CoreDNS-1.12.0
linux/amd64, go1.23.3, 51e11f1
maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
.:53
[INFO] plugin/reload: Running configuration SHA512 = 61c4e97fd46a540ac8dda919cc898bd4e91207731b7b08b141e56fbd76fd591d82a4524df658474b8e3d867076fc7ba04e95992926333c1bcfaa75c46969808c
CoreDNS-1.12.0
linux/amd64, go1.23.3, 51e11f1
root@server:~#

```

##### 7.0 Network Policy Drops

Network policy changes are among the most frequent operations in a Kubernetes cluster. They are also among the most delicate, as even a small mistake can lead to widespread traffic disruption. This section walks through several methods to make policy management safer, especially in Day 2 operations or brownfield deployments where clusters already run critical workloads. It shows how to test and validate changes before enforcing them, helping teams adopt a more reliable approach to policy rollout.

An endpoint can be put into default deny mode at ingress or egress if a rule selects the endpoint and contains the respective rule section. Any rule selecting the endpoint will have this effect. Understanding that default-deny is triggered implicitly, not by a dedicated deny rule, is the foundation of everything that follows.

Cilium network policies build upon Kubernetes NetworkPolicy, extending it with deeper visibility and more flexible rule types. Policies are used to allow or deny traffic based on defined rules, which can apply to ingress, egress, or both. These rules are evaluated at the datapath level, meaning enforcement takes place directly in the kernel through eBPF.

[Read More About Safely Managing Cilium Network Policies](https://www.cncf.io/blog/2025/11/06/safely-managing-cilium-network-policies-in-kubernetes-testing-and-simulation-techniques/)

###### Identifying Policy-Denied Drops

The fastest way to identify a policy drop is through Hubble. Unlike iptables-based CNIs, where you parse iptables \-L output and mentally trace rule ordering, Cilium surfaces every policy decision as an observable flow event, including the direction and reason for each drop.

_Watch for drops in real time:_

```shell
# All policy-denied drops in the cluster
root@server:~# hubble observe --verdict DROPPED --follow

# Filter to a specific destination
hubble observe --verdict DROPPED \
  --to-pod default/backend --follow

# Filter by source pod
root@server:~# hubble observe --verdict DROPPED \
  --to-pod default/nginx --follow

# Show only policy-verdict events (broader than just drops)
root@server:~# hubble observe -t policy-verdict --follow

```

Using cilium-dbg monitor for low-level drop events:
When Hubble is not available, or you need lower-level detail:

```shell
root@server:~# kubectl -n kube-system exec ds/cilium --  cilium-dbg monitor --type drop
Listening for events on 8 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit

```

###### Policy Tracing

Policy tracing simulates the policy decision for a specific traffic flow without sending any real traffic. It's the most direct way to answer the question: "Should this traffic be allowed right now, given the policies in effect?"

```shell
# Trace by pod name
kubectl -n kube-system exec ds/cilium -- \
  cilium-dbg policy trace \
  --src-k8s-pod default/frontend \
  --dst-k8s-pod default/backend \
  --dport 8080/TCP

# Trace by label selectors
kubectl -n kube-system exec ds/cilium -- \
  cilium-dbg policy trace \
  --src-identity k8s:app=frontend,k8s:io.kubernetes.pod.namespace=default \
  --dst-identity k8s:app=backend,k8s:io.kubernetes.pod.namespace=default \
  --dport 8080/TCP

```

###### Rendering the Effective Policy

The policy in effect for an endpoint is not simply the sum of kubectl get networkpolicy output. Cilium computes a realized policy per endpoint by evaluating all matching rules from Kubernetes NetworkPolicies, CiliumNetworkPolicies, and CiliumClusterwideNetworkPolicies and merging them into a single enforcement state. The realized policy is what actually runs in the datapath.

To determine which policy rules are currently in effect for an endpoint, the data from cilium-dbg endpoint list and cilium-dbg endpoint get can be paired with the data from cilium-dbg policy get. cilium-dbg endpoint get will list the labels of each rule that applies to an endpoint.

Step 1: Find the endpoint ID

```shell
root@server:~# # 1. Find the Cilium pod handling your nginx pod
TARGET_NODE=$(kubectl get pod nginx -o jsonpath='{.spec.nodeName}')
CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium --field-selector spec.nodeName=$TARGET_NODE -o jsonpath='{.items[0].metadata.name}')

# 2. Extract the Endpoint ID
EP_ID=$(kubectl -n kube-system exec $CILIUM_POD -- cilium-dbg endpoint list | grep nginx | awk '{print $1}')

# 3. Verify it found the ID
echo "Your Nginx Endpoint ID is: $EP_ID"
Your Nginx Endpoint ID is: k8s:run=nginx
root@server:~#
```

Note the endpoint ID and the security identity from the output.

Step 2: Check policy enforcement state

```shell
root@server:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                   IPv6   IPv4           STATUS
           ENFORCEMENT        ENFORCEMENT
98         Disabled           Disabled          8          reserved:ingress                                                     10.244.0.12    ready
1050       Disabled           Disabled          1          k8s:node-role.kubernetes.io/control-plane                                           ready
                                                           k8s:node.kubernetes.io/exclude-from-external-load-balancers
                                                           reserved:host
2314       Disabled           Disabled          4          reserved:health                                                      10.244.0.227   ready
root@server:~#
```

The POLICY (ingress) and POLICY (egress) columns show Enabled or Disabled for each endpoint. An endpoint with 'Disabled' on both means no policy is selecting it; it's in default-allow mode. The moment a policy is selected, the selected direction becomes the enforced default-deny. If you see Disabled when you expect Enabled, check whether the policy's endpointSelector actually matches the pod's labels.

###### Policymap Pressure

Each Cilium-managed endpoint has its own BPF policy map, a fixed-size hash table in kernel memory that holds all allowed traffic rules for that endpoint. The default maximum number of entries in the endpoint policy map is 16,384 per endpoint, configured via \--bpf-policy-map-max.
In large clusters with complex policies, especially those using many CIDR rules, fine-grained label selectors, or toFQDNs, this map can fill up. When it does, policy entries can't be added, and traffic that should be allowed gets dropped.

Common Pitfalls

1\. Implicit default-deny on first policy application
The most common surprise in Cilium policy management: the moment you apply the first NetworkPolicy or CiliumNetworkPolicy that selects a pod, all traffic not explicitly permitted by that policy is denied. Applying the first network policy triggers the default-deny behavior, which may unintentionally block traffic not yet accounted for in your policy definitions.
The fix is to apply audit mode first, observe all traffic, then write policies that cover every observed flow before enforcing.

2\. Forgetting to allow egress DNS
Any policy restricting egress must explicitly allow DNS on port 53\. Without it, the pod loses service discovery. This is the single most common policy-related incident in production Cilium clusters. Always include this egress rule in any policy applied to a namespace:

3\. Policy is per-direction, not per-endpoint
A policy that allows egress from Pod A to Pod B does not automatically allow ingress at Pod B. Note that while an egress rule may allow traffic from an endpoint, the receivers of that egress traffic may have ingress rules that deny it. Policy must be configured on both sides, the sender and the receiver. If both pods have policies applied, you need matching egress rules on the sender and ingress rules on the receiver.

4\. Label selector scope and namespace boundaries
Network policies imported directly with the Cilium API apply to all namespaces unless a namespace selector is specified. A CiliumClusterwideNetworkPolicy without a namespace selector applies globally. A regular CiliumNetworkPolicy scopes to its own namespace. Getting this wrong produces either overly permissive or unexpectedly restrictive behavior. Always check the namespace context of your policy resource type.

5\. toServices and toPorts cannot be combined
You cannot create an egress rule that simultaneously specifies a service and restricts ports. You must choose between defining traffic by service identity or by port. Attempting to combine toServices with toPorts in the same egress rule produces a policy that silently ignores one of the two constraints, leading to unexpected behavior.

6\. The cluster entity does not span ClusterMesh
A common misunderstanding when configuring entity-based L3 policies is that the cluster entity includes only endpoints in a local cluster and does not include other clusters in a ClusterMesh. If you're running ClusterMesh and your toEntities: cluster rule isn't allowing cross-cluster traffic, you need explicit selectors for the remote cluster's identities.

7\. toFQDNs rules without a DNS inspection rule
As covered in section 6.0, a toFQDNs rule does nothing without a companion L7 DNS rule that tells Cilium to intercept DNS responses. The traffic will be denied because Cilium never learned the IP mapping for the FQDN.

8\. Policy revision mismatch after rollout
After applying a new policy, verify that the agent has picked it up on every node.

[Read More About Cilium Network Policy Troubleshooting](https://docs.cilium.io/en/stable/security/policy/troubleshooting/)

##### 8.0 Summary

Reactive troubleshooting, diagnosing a problem after it's already impacting production, is expensive. The gap between a network misconfiguration and its detection is often measured in minutes or hours, not seconds. The goal of this section is to close that gap by building a detection layer that surfaces issues before users notice them.

###### Troubleshooting Hierarchy

Every Cilium networking problem falls into one of four layers. Working top-down through this order eliminates entire categories of causes quickly

| Layer | Check first |
|:----------------- |:--------------------------------------------- |
| Agent health | cilium status all subsystems OK? |
| Endpoint state | cilium-dbg endpoint list all endpoints ready? |
| Traffic visibility | hubble observe \--verdict DROPPED any drops? |
| Datapath state | cilium-dbg bpf \* maps reflect expected state? |

It is also important to utilise Prometheus metrics for alerting as an early warning on every major failure.

#### How k0rdent and Cilium Turn Kubernetes Sprawl into a Platform

URL: https://cilium.io/blog/2026/07/08/how-k0rdent-and-cilium-turn-kubernetes-sprawl-into-a-platform
Date: 2026-07-08
Categories: Technology

If you have spent time running Kubernetes in the real world, you know the catch: standing up one cluster is the easy part. The hard part starts when one cluster becomes ten, then a hundred, spread across cloud providers, bare metal, and on-prem environments.

That is where k0rdent and Cilium fit together very well.

**k0rdent** gives platform teams a single control plane to provision, template, and manage fleets of clusters declaratively and consistently. **Cilium**, built on eBPF, provides the networking, identity-based security, and observability those clusters need for production.

Together, they help turn cluster sprawl into a repeatable platform pattern. Instead of bolting on networking, policy, and visibility one cluster at a time, teams can make them part of the cluster definition from the start.

In this post, we will look at why platform teams pair k0rdent and Cilium, then walk through a hands-on example using a k0rdent `ClusterDeployment` with Cilium as the CNI.

Let's dig in.

##### Built Once, But is it Consistent Everywhere?

If you run a platform team, you're already juggling a growing fleet, workloads that need management, and a list of things that must stay consistent across every cluster. k0rdent and Cilium work well together because they address both sides of that problem: how clusters get built and how they behave once they are running.

The drift usually shows up in small but painful ways. One cluster has Cilium configured differently, another is missing a policy that exists everywhere else, and another has incomplete visibility when a service-to-service call starts failing. Those differences are easy to create by hand and hard to debug at 2 a.m.

k0rdent's answer is to treat the cluster as a template, and Cilium fits into that model directly. Cilium is available in the k0rdent catalog as a validated service template, meaning it can be attached to a cluster as a reusable add-on. Define it once, and every cluster k0rdent provisions can come up with the same Cilium foundation: CNI, policies, and observability included from the start.

Visibility is also a factor. "Service A can't reach service B" is easy or hard depending entirely on what you can see. Cilium ships with Hubble, which gives you L3 through L7 flow visibility without sidecars by leveraging eBPF, and k0rdent helps make that visibility part of the standard cluster template rather than something enabled one cluster at a time. When something breaks, you can see where the traffic was dropped and why, and on which cluster it happened.

Put it together and the practical result is that the platform team can run a much larger fleet, because the clusters are built the same way and behave predictably.

##### From Template to Running Cluster

With the why out of the way, let's see what this actually looks like in practice. In this section we'll walk through deploying a k0rdent `ClusterDeployment` that uses Cilium as its CNI, so that the cluster comes up already networked the way we want rather than waiting for someone to wire it in afterward.

A quick note on assumptions before we start. This walkthrough assumes you already have a k0rdent Management cluster up and running. If you don't, the [official quickstart](https://docs.k0rdent.io/latest/quickstarts/quickstart-1-mgmt-node-and-cluster/#install-a-kubernetes-cluster-as-the-management-cluster) walks you through setting one up. We'll also be deploying onto bare metal hosts, which means we'll be using the `remote-cluster` `ClusterTemplate` to bring those existing machines under management.

A `ClusterTemplate` is k0rdent's reusable definition for how a cluster should be created. The `remote-cluster` template is useful when the infrastructure already exists, as it lets k0rdent connect to those machines over SSH and bootstrap them into a managed child cluster..

###### Deploying the Remote Cluster Template

For this example we're working with three Linux machines on the same network. One is the k0rdent node running our Management cluster. The other two, `worker1` and `worker2`, are the bare metal hosts that will become our child cluster.

Because k0rdent provisions bare metal remote machines over SSH, the first thing we need is root SSH access to both workers. Let's generate a dedicated key pair for this so we're not reusing anything else, then copy it out to each host.

```shell
ssh-keygen -f ~/.ssh/idk0r
ssh-copy-id -i ~/.ssh/idk0r.pub root@worker1
ssh-copy-id -i ~/.ssh/idk0r.pub root@worker2
```

Next, k0rdent needs a credential object so it knows how to authenticate against those workers. We install the `remote-credential` chart, then base64-encode the private key we just created and patch it into the secret k0rdent expects.

```shell
helm install remote-credential oci://ghcr.io/k0rdent/catalog/charts/remote-credential -n kcm-system
REMOTE_SSH_KEY_B64=$(cat ~/.ssh/idk0r | openssl base64 -A)
kubectl patch secret remote-ssh-key -n kcm-system -p='{"data":{"value":"'$REMOTE_SSH_KEY_B64'"}}'
```

Here's where the pairing actually comes together. We install Cilium into k0rdent as a service template, which is what makes it something we can reference from a `ClusterDeployment` later.

```shell
helm upgrade --install cilium oci://ghcr.io/k0rdent/catalog/charts/kgst \
--set "chart=cilium:1.19.0" -n kcm-system
```

Once it's installed, a quick check confirms the template is registered and valid.

```shell
kubectl get servicetemplates -A | grep cilium
```

Now, we set up the cluster we want. We can do this with a `ClusterDeployment` that references the `remote-cluster` template for the bare metal hosts and the `cilium` service template for the CNI.

Create a file called `remote-cld-cilium.yaml`:

```
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: remote
  namespace: kcm-system
  labels:
    type: remote
spec:
  template: remote-cluster-1-0-22
  credential: remote-credential
  propagateCredentials: false
  config:
    controlPlaneNumber: 1
    k0smotron:
      service:
        type: NodePort
    machines:
    - name: worker1
      address: 192.168.1.5
      user: root
      port: 22
    - name: worker2
      address: 192.168.1.6
      user: root
      port: 22
    k0s:
      version: v1.35.3+k0s.0
    network: # prepare for cilium
      calico: null
      provider: custom
    kubeProxy:
      disabled: true
  serviceSpec:
    services:
    - template: cilium-1-19-0
      name: cilium
      namespace: kube-system
      values: |
        cilium:
          cluster:
            name: cilium
          hubble:
            tls:
              enabled: false
            auto:
              method: helm
              certManagerIssuerRef: {}
            ui:
              enabled: false
              ingress:
                enabled: false
            relay:
              enabled: false
          ipv4:
            enabled: true
          ipv6:
            enabled: false
          envoy:
            enabled: false
          egressGateway:
            enabled: false
          kubeProxyReplacement: "true"
          serviceAccounts:
            cilium:
              name: cilium
            operator:
              name: cilium-operator
          localRedirectPolicy: true
          ipam:
            mode: cluster-pool
            operator:
              clusterPoolIPv4PodCIDRList:
              - "192.168.224.0/20"
              - "192.168.210.0/20"
              clusterPoolIPv6PodCIDRList:
              - "fd00::/104"
          tunnelProtocol: geneve
          k8sServiceHost: "{{ .Cluster.spec.controlPlaneEndpoint.host }}"
          k8sServicePort: "{{ .Cluster.spec.controlPlaneEndpoint.port }}"
```

A couple of things worth calling out. We're disabling `kubeProxy` and setting `calico: null` because Cilium is going to take over that job with its eBPF datapath, and `kubeProxyReplacement` is set to `"true"` to make that explicit.

Now, we can apply the `remote-cld-cilium.yaml` file, creating the cluster.

```shell
kubectl apply -f remote-cld-cilium.yaml
```

From there, k0rdent takes over, reaching out to the workers over SSH, bootstraps k0s, and rolls out Cilium as the CNI as part of the same process.

We can get the status of the `ClusterDeployment` with:

```shell
kubectl get cld -A
```

Once you see the status of "READY" being "True", the cluster is ready to go\!

```shell
NAMESPACE    NAME     READY   SERVICES   TEMPLATE                MESSAGES          AGE
kcm-system   remote   True    1/1        remote-cluster-1-0-22   Object is ready   15m
```

###### Running a Cilium Test

Once the `ClusterDeployment` reports ready, we can pull the child cluster's kubeconfig straight from the secret k0rdent created and have a look around.

Extract the secret, decode it, and save it to a file:

```shell
kubectl get secret remote-kubeconfig -n kcm-system -o=jsonpath={.data.value} | base64 -d > kcfg_remote.yaml
```

Verify Cilium is up and running:

```shell
kubectl --kubeconfig kcfg_remote.yaml get pods -A -l app.kubernetes.io/part-of=cilium
```

The output should show the Cilium pods running in the `kube-system` namespace:

```shell
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   cilium-mgw7b                       1/1     Running   0          4h42m
kube-system   cilium-operator-758fc644d7-jjq8h   1/1     Running   0          4h43m
kube-system   cilium-operator-758fc644d7-nhdr4   1/1     Running   0          4h43m
kube-system   cilium-rxcfw                       1/1     Running   0          4h38m
```

Now that we verified that Cilium is running, we can do a quick `CiliumNetworkPolicy` test.

First, we'll deploy a small sample workload: a `webapp` that serves both a `/public` and a `/private` endpoint, along with a `client` pod we can use to reach it.

```shell
kubectl --kubeconfig kcfg_remote.yaml \
apply -f \
https://gist.githubusercontent.com/christianh814/0d4c0d9529fe630775fa3f54f1725f80/raw/8e89d8d5d89f014e3d7899e32b4d71fe6498e93d/sample-workload.yaml
```

With no policy in place, the `client` pod can reach both endpoints. Both of these calls return data:

```shell
$ kubectl --kubeconfig kcfg_remote.yaml exec -it client-pod -- curl http://webapp/public
[
  {
    "id": 1,
    "body": "public information"
  }
]

$ kubectl --kubeconfig kcfg_remote.yaml exec -it client-pod -- curl http://webapp/private
[
  {
    "id": 1,
    "body": "secret information"
  }
]
```

That `/private` endpoint handing back "secret information" to anyone who asks is exactly the kind of thing we'd want to lock down. So let's write a `CiliumNetworkPolicy` that only allows the `client` to issue an HTTP `GET` against `/public`, and nothing else.

```shell
kubectl --kubeconfig kcfg_remote.yaml apply -f -<<EOF
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-public-get"
spec:
  description: "Allow HTTP GET /public from app=client to app=webapp"
  endpointSelector:
    matchLabels:
      app: webapp
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: client
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/public"
EOF
```

Note that this isn't a simple "can these two pods talk" rule. We're filtering at Layer 7, allowing a specific method on a specific path. That's something a traditional L3/L4 CNI can't express on its own. With the policy applied, let's run the exact same two requests again.

```shell
$ kubectl --kubeconfig kcfg_remote.yaml exec -it client-pod -- curl http://webapp/public
[
  {
    "id": 1,
    "body": "public information"
  }
]

$ kubectl --kubeconfig kcfg_remote.yaml exec -it client-pod -- curl http://webapp/private
Access denied
```

The `/public` request still works exactly as before, but the call to `/private` now comes back with `Access denied`. Cilium is inspecting the HTTP request and enforcing our intent at the application layer, and we got there without deploying a service mesh, a sidecar, or anything beyond the cluster k0rdent handed us. The CNI we declared in the template is already doing real work.

##### Summary

In this walkthrough, k0rdent gave us a declarative way to stand up a cluster on bare metal, and because Cilium was defined in the template as a service, that cluster came up already networked, already secured, and ready to enforce a Layer 7 policy the moment it was online. Nothing was bolted on after the fact. The networking was part of the cluster definition from the start.

The value becomes clearer when you move from a single cluster to a fleet. The same template that produced a working CNI and policy engine here gives every cluster the same foundation, whether it lands on a Cloud Service Provider or a rack in your own data center. Networking, security, and observability stop being things each team wires up on their own and become part of what a cluster means in your environment. That consistency is what lets a platform team support a growing fleet without the drift, snowflakes, and the guesswork during late-night troubleshooting.

k0rdent and Cilium are each strong on their own, but together they close the gap between a cluster that's running and a cluster that's ready for production. That's why platform teams are choosing to pair the two in their environments.

#### Multi-Cluster Kubernetes Explained

URL: https://cilium.io/blog/2026/06/13/multi-cluster-kubernetes-explained
Date: 2026-06-13
Categories: Technology

##### **Multi-Cluster Kubernetes Explained**

##### **Introduction**

As enterprises scale their cloud native infrastructure, a single Kubernetes cluster often becomes not enough for meeting requirements related to isolation, availability, scale, and/or geographic distribution. Multi-cluster Kubernetes is an architectural approach that involves deploying and managing multiple independent clusters to support a unified application environment.

By dividing resources into targeted groups, companies can localize for efficiency, isolate potential failures, and tailor specific technology or cluster needs without overlapping operational concerns.

There are various ways to segment workloads from one another in Kubernetes. You can run them on different pods or set up different namespaces. But if you want the maximum level of granularity and the high availability and performance benefits that it can bring, a multi-cluster Kubernetes deployment is the way to go.

Edge computing can also require connecting hundreds of clusters across disparate locations and infrastructures. Without a networking solution to manage this scale and complexity, you will just have a bunch of computers talking to themselves rather than each other and your customers.

##### **I. Why Multi-Cluster?**

Transitioning from a single large cluster to multiple smaller clusters can be driven by several critical operational concerns:

- **Low Latency**: In a single-cluster architecture, workloads are deployed to one region or data center. This means users interacting with your application from across the globe will inevitably experience high network delays due to the physical distance data must travel. By adopting a multi-cluster approach, you can distribute workloads geographically, placing compute resources in regions closer to your end-users or edge devices. This drastically reduces round-trip times and ensures a highly responsive application experience.

- **High Availability and Disaster Recovery**: Distributing workloads across clusters in different availability zones or regions ensures that the failure of a single cluster or provider region does not lead to total application downtime.
- **Workload scalability:** Running multiple clusters may enhance your ability to scale workloads up when necessary. If everything runs in a single cluster, it’s harder to determine which specific workloads require more resources or additional replicas, especially if you lack performance data for individual workloads (which may be the case if you are only tracking cluster-level health). You are also more likely to run into “noisy neighbor” issues when running everything in a single cluster. And for very large clusters, you may hit the ceiling on the cluster size that Kubernetes supports; currently, you can’t have more than 5,000 nodes per cluster.
- **Compliance and Data Sovereignty:** Specific clusters can be localized to certain jurisdictions to meet regulatory requirements regarding data residency.
- **Hard Multitenancy**: While namespaces provide soft isolation, separate clusters offer a stronger security boundary for different teams, environments (Dev/UAT/Prod), or sensitive workloads.
- **Organizational Growth and Acquisitions**: As companies scale organically or engage in Mergers and Acquisitions (M\&A), they frequently inherit disparate infrastructure, different cloud providers, or existing Kubernetes environments. Instead of undertaking a massive, high-risk migration to consolidate these discrete workloads into a single monolith, connecting them via a multi-cluster architecture provides a seamless, non-disruptive path to infrastructure integration while allowing acquired teams to maintain their operational autonomy.

##### **II. Multi-Cluster Architecture Patterns**

Organizations deploy Kubernetes across multiple clusters for various reasons such as resilience, compliance, regulatory isolation, team autonomy, or geographic distribution. Before choosing a topology, it's important to understand the trade-offs each pattern introduces in terms of operational complexity, blast radius, latency, and cost.

###### **Active-Active**

In an active-active topology, two or more clusters simultaneously serve live production traffic. Workloads are replicated across all clusters, and load is balanced between them. This pattern reduces blast radius and enables failover, supporting resilience through active-active approaches. With Cilium ClusterMesh, global services can span clusters so that pods in Cluster A and Cluster B both act as backends for the same service; traffic flows to whichever is healthy.

**Best for:** High-availability applications, geo-distributed workloads, zero-downtime deployments.
**Trade-off:** Requires state synchronization between clusters; data consistency logic adds complexity.

###### **Active-Standby**

Here, one cluster handles all live traffic (active), while a second cluster sits warm and ready to take over (standby). The standby cluster runs replicated workloads but does not serve traffic under normal conditions. Failover is triggered manually or automatically when the active cluster degrades.

With Cilium's service.cilium.io/affinity annotation, you can express topology-aware preference, for instance, pinning traffic to the local cluster under normal conditions and allowing spillover to the remote cluster only during failure. Setting the affinity to local marks local endpoints as preferred, and changing the value to remote shifts traffic to the remote cluster's endpoints.

**Best for:** Disaster recovery, regulated workloads requiring a clear primary/secondary designation.
**Trade-off:** Standby capacity is idle under normal operations, increasing cost.

[Read More About Service Affinity](https://docs.cilium.io/en/latest/network/clustermesh/affinity/)

###### **Cluster per Environment**

Each Software Development Lifecycle environment (development, staging, production) gets its own cluster. This is one of the most common patterns in enterprise Kubernetes adoption and provides hard boundaries between lifecycle stages.

Separating environments (dev/staging/prod) or tenants into dedicated clusters avoids noisy-neighbor effects and misconfiguration leakage. ClusterMesh is not always required here; typically, inter-cluster communication between environments is intentionally restricted, but it can be used to mirror services or support integration testing across environment boundaries with a fine-grained network policy.

###### **Cluster per Region**

As enterprises diversify product offerings across the globe, it’s important to make sure that users connect to their services from the nearest access point. Hence, instead of attempting to stretch a single Kubernetes cluster across geographic regions, which often results in high latency, organizations deploy distinct clusters in different geographic regions.

Another reason for an enterprise to run a cluster per region is data residency compliance requirements. GDPR mandates strict data residency requirements, ensuring personal data of EU residents is stored and processed within specific geographic locations or under adequate safeguards.

###### **Cluster per Tenant**

Also known as hard multitenancy, this pattern provisions a dedicated Kubernetes cluster for each customer, business unit, or internal department. While Kubernetes namespaces, Network Policies, and RBAC offer soft isolation, they still share the same control plane and underlying host kernels, which might not meet strict compliance requirements.

By assigning a cluster per tenant, you eliminate the noisy neighbor problem and ensure a strict security boundary. A common challenge here is IP exhaustion or overlapping IPs across dozens of tenant clusters.

###### **Hub and Spoke Topology**

In a hub-and-spoke topology, a central hub cluster acts as the core shared-services center, connecting out to multiple spoke clusters where the actual workloads run. Instead of deploying operational tooling in every single cluster, the hub hosts centralized services like observability stacks (Hubble/Prometheus), secrets management (HashiCorp Vault), CI/CD runners, or centralized DNS.

With Cilium ClusterMesh, you can expose these specific shared services from the Hub to the Spokes without requiring the Spokes to be meshed with each other.

##### **III. Implementing Inter-Cluster Connectivity via Cilium ClusterMesh**

Cilium enables seamless multi-cluster connectivity through a feature called **ClusterMesh**.

_Fig 1: Cilium ClusterMesh connecting heterogeneous Kubernetes environments across Azure, AWS, Red Hat OpenShift, and VMware through unified Ingress, Egress, and Transit Gateways._

Cluster Mesh connectivity is high performance, as data flows directly from one worker node to another without intermediate proxies. It preserves the workload identity for cross-cluster traffic, meaning that network observability tooling, such as Cilium’s Hubble, as well as network policies, continue to operate just as they do within a single cluster.

###### **Key Capabilities of ClusterMesh**

- **Multi-Cluster Connectivity:** Cilium provides the ability to connect multiple clusters across different cloud providers or on-premises environments.
- **Identity-Based Security:** Security identities assigned to pods are maintained across cluster boundaries, allowing for consistent L3/L4 and L7 network policies regardless of where a pod is running.
- **Global Services:** Services can be defined as "global," enabling traffic to be load-balanced across pods in multiple clusters. This allows for automatic failover; if a service in Cluster A fails, traffic is redirected to the healthy backends in Cluster B.
- **Transparent Encryption:** Cilium can enable node-to-node encryption across clusters without requiring application changes or sidecar proxies.

_Fig 2: Cilium ClusterMesh global service load balancing, frontend and backend services spanning two clusters with automatic failover to healthy backends in Cluster 2._

###### **Architectural Requirements Checklist**

To successfully implement Cilium ClusterMesh and achieve inter-cluster connectivity, your infrastructure must meet a specific set of foundational network requirements:

1. **Non-overlapping Pod CIDRs**: In a standard ClusterMesh deployment, every cluster must be assigned a unique, non-overlapping Pod IP subnet (CIDR). This ensures that Pod IPs are globally unique across the entire mesh, allowing Cilium to route traffic directly to a specific pod without relying on complex Network Address Translation (NAT) rules.
2. **Node-to-node IP Connectivity**: The worker nodes in all participating clusters must be able to reach each other over the network. Clusters are connected and routed across cloud providers using native VPC peering, dedicated interconnects (like AWS Direct Connect or Azure ExpressRoute), or via secure IPSec/WireGuard tunnels.
3. **Native Routing CIDR**: When using direct routing instead of encapsulation (tunneling), the network infrastructure must know how to route the Pod CIDRs. This requires configuring the underlying VPC network to recognize the pod IP ranges so that traffic can flow natively between nodes across different clusters.
4. **Consistent Datapath Mode:** While it is possible to mix some configurations, it is highly recommended that meshed clusters use a consistent datapath mode (e.g., all clusters using VXLAN/Geneve tunneling, or all clusters using direct native routing). This reduces troubleshooting complexity and ensures consistent performance.
5. **DNS Across Clusters:** For seamless service discovery, the Kubernetes DNS service (like CoreDNS) needs to be aware of the ClusterMesh. Cilium integrates with the service routing plane underneath Kubernetes DNS. When CoreDNS resolves a global service hostname to a local virtual ClusterIP, Cilium's eBPF layer translates that IP target into a list of globally available backend pods, regardless of which cluster they reside in.

_Fig 3: Cilium ClusterMesh internal architecture, two clusters connected via VPC Peering, showing etcd state synchronization, Cilium agents, Load Balancers, and the VXLAN encapsulated network packet structure for cross-cluster pod traffic._

[Read More About Cilium Cluster Mesh](https://isovalent.com/blog/post/introducing-cilium-mesh/)

##### **IV. Deep Dive: How ClusterMesh Operates Under the Hood**

Unlike traditional multi-cluster setups that rely on heavy routing layers or centralized API gateways, Cilium leverages the Linux kernel to create a seamless, flat network topology across your entire multi-cluster environment.

In many traditional architectures, cross-cluster communication requires traffic to traverse a complex maze of ingress controllers, egress gateways, and sidecar proxies. Cilium fundamentally changes this paradigm. Cilium Cluster Mesh implements IP forwarding without proxies or gateways. Because the routing logic is embedded directly in the kernel, data flows directly from one worker node to another without intermediate proxies.

###### **The clustermesh-apiserver**

At the heart of Cilium's multi-cluster architecture is the clustermesh-apiserver. Rather than relying on a single, global database that could act as a single point of failure, Cilium takes a decentralized approach.

Cilium's Cluster Mesh uses etcd as an external key-value store to exchange information across multiple Cilium instances. The clustermesh-apiserver manages this dedicated etcd instance for the cluster, tracking the state of endpoints, services, and network identities. To allow other clusters to connect and read this state, the Cilium control plane is exposed to the VPCs using the service annotation 'Internal' for load balancer type.

###### **KVStoreMesh**

Once the _cluster mesh-apiserver_ exposes the cluster's state, the clusters need a safe way to synchronize this data. This is where the KVStoreMesh concept comes in.

Instead of pushing data directly into a remote cluster's primary database (which is risky), each cluster in the network operates its own etcd cluster with replication occurring on a read-only basis to ensure that failures in a cluster do not bring down other clusters. Cilium agents watch the remote KVStores in a pull-based, read-only fashion. If the network link between Cluster A and Cluster B goes down, both clusters simply continue operating securely using their locally cached state.

###### **Identity Propagation**

One of the most complex challenges in multi-cluster networking is maintaining the security context when a packet leaves one cluster and enters another. Traditional networks rely on IP addresses, which are ephemeral and lose meaning across cluster boundaries.

Cilium solves this through identity propagation. The key to seamless pod-to-pod networking is Cilium's implementation of identity-based networking that identifies which pods and services can communicate using labels instead of IP addresses to enforce access controls. Because this identity is embedded in the eBPF datapath, Cilium preserves workload identity for cross-cluster traffic, meaning that network observability tooling, such as Cilium's Hubble, as well as network policies, continue to operate just as they do within a single cluster.

###### **Certificate Management**

Because ClusterMesh involves control plane data (etcd synchronization) and data plane traffic (pod-to-pod communication) traversing underlying networks often crossing different VPCs or public internet links, security is paramount.

- **Control Plane Security**: TLS authenticates the client and server with the certificates and keys that are managed as Kubernetes secrets. This ensures that only authorized clusters can connect to the remote clustermesh-apiserver to read the KVStore state.
- **Data Plane Encryption**: For the actual application traffic flowing between nodes across clusters, encryption between all endpoints, clusters, pods, and services can be automatically and transparently applied at the platform level without requiring application modifications. This transparent encryption (using IPsec or WireGuard) ensures that data in transit remains fully secure regardless of the underlying cloud provider or network infrastructure.

##### **V. Pod-to-Pod Connectivity Across Clusters**

The minimum requirement for a successful multi-cluster network is for Kubernetes pods to easily communicate with one another across clusters. Historically, connecting isolated Kubernetes clusters required exposing workloads via Ingress controllers, API gateways, or complex NAT rules.

Cilium ClusterMesh eliminates this friction. Cluster Mesh connectivity is high performance, as data flows directly from one worker node to another without intermediate proxies. This flattens the network, allowing a pod in one cluster to directly address the IP of a pod in another cluster just as if they were on the same physical node.

_Fig 4: Pod-to-Pod connectivity across Cilium ClusterMesh, frontend pod in Cluster A (10.10.0.0/16) communicating directly with inventory pod in Cluster B (10.20.0.0/16) via eBPF datapath and VXLAN/native routing, with bidirectional control-plane synchronization of EndpointSlices, services, and identities._

###### **How Pod IPs become routable Across Clusters**

To achieve this flat network topology, several networking prerequisites must align:

- Non-Overlapping Pod CIDRs: The most critical requirement is that each cluster must be assigned a unique, non-overlapping Pod IP subnet (CIDR). If Cluster A uses 10.10.0.0/16, Cluster B must use something different, like 10.20.0.0/16. This ensures that every pod across your entire infrastructure has a globally unique IP address.

- Node-to-Node Reachability: ClusterMesh does not magically bypass firewalls. The worker nodes across different clusters must be able to route traffic to one another. Clusters are connected and routed through VPCs using either the standard API from any cloud provider, an on-premise infrastructure via regular IPSec-based VPN gateways and tunnels, or directly through network headers.

###### **Encapsulation Mode vs Native Routing Across Cluster Boundaries**

When a packet leaves a pod in Cluster A destined for a pod in Cluster B, Cilium can route the pod IPs in a few different ways. You must choose the right datapath mode based on your underlying infrastructure:

**Encapsulation Mode (Tunneling)**: By default, Cilium wraps the pod traffic inside a VXLAN or Geneve tunnel. The underlying cloud provider or physical network only sees traffic moving from Node A's IP to Node B's IP; the internal pod IPs are hidden inside the tunnel.

Pros: Incredibly easy to set up. It works on almost any infrastructure because the underlying network doesn't need to know how to route pod IPs.
Cons: The encapsulation process adds a small amount of overhead (header size and CPU cycles) to every packet.

**Native Routing (Direct Routing)**: In this mode, Cilium relies on the underlying network (like AWS VPC CNI, Azure Delegated IPAM, or on-premises BGP routers) to natively understand and route the pod IPs. No tunneling is used.

Pros: Maximum performance. By avoiding the encapsulation overhead of tunnel mode, Native Routing provides the lowest possible latency and highest throughput, crucial for data-heavy workloads like AI/ML training or massive databases.

Cons: Requires advanced configuration of the underlying network infrastructure to ensure it can natively route the distinct Pod CIDRs across cluster boundaries.

###### **EndpointSlice Synchronization**

Routing packets is only half the battle; the clusters also need to know where to send them. This is where service discovery comes into play.

In a single Kubernetes cluster, when you create a Service, the Kubernetes control plane generates an EndpointSlice, a scalable list of the healthy pod IPs backing that service. In a multi-cluster setup, Cilium agents securely synchronize these endpoint, identity, and service states across cluster boundaries.

Recent advancements in Cilium have taken this a step further by natively supporting the Multi-Cluster Services API (MCS-API), a Kubernetes SIG standard. By using EndpointSlice synchronization, Cilium can dynamically reconcile EndpointSlices from remote clusters. If a backend pod in Cluster B scales up or crashes, the Cilium clustermesh-apiserver instantly synchronizes that state change over the etcd control plane, updating the available endpoints for Cluster A in real-time. This allows load balancing to shift dynamically without relying on external DNS polling.

###### **Verifying Cross-Cluster Pod Connectivity**

Once your ClusterMesh is established, verifying and troubleshooting the connectivity requires the right tools. Because Cilium preserves workload identity for cross-cluster traffic, you can use native Cilium and Hubble tooling to debug cross-cluster flows just as easily as local ones.

- Verify the Mesh Status: Use the Cilium CLI to ensure the clusters are actively peering and the etcd control planes are synchronizing.
- Run the Multi-Cluster Connectivity Test: The Cilium CLI includes a robust, automated testing suite that deploys test pods in both clusters and validates IP routing, global service load balancing, and network policy enforcement across the boundary.
- Observe with Hubble: If a connection fails, you don't need to guess if a network policy or a routing rule dropped it. Because the network is fully observable without additional code changes, you can use the Hubble CLI to instantly trace the flow.

[Read More About Routing & Service Mesh](https://isovalent.com/blog/post/topology-aware-routing-and-service-mesh-across-clusters-with-cluster-mesh/)

##### **VI. Global Services and Cross-Cluster Load Balancing**

In a multi-cluster environment, developers need a way to expose an application running in multiple clusters under a single, unified service name. Cilium handles this natively without requiring complex external load balancers or API gateways.

###### **Global Services**

Cilium routes connections between pods in multiple clusters if a 'global services' annotation is added to a Kubernetes service deployment. To enable this, you deploy a standard Kubernetes Service in each cluster and attach the following metadata annotation: io.cilium/global-service: "true".

Implementing this annotation immediately solves the pod-to-pod communication problem and eliminates the need for internal load balancers and DNS records with cross-cluster identity-based networking. Cilium connects clusters without proxies or gateways by representing each global workload as if the workload runs as a pod.

_Fig 5: Cilium Global Service load balancing across two clusters, Pod A in Cluster 1 (us-east) resolves web-service via DNS, with the Global Service LB distributing 50% of traffic to local endpoints and 50% to remote endpoints in Cluster 2 (us-west) through bidirectional ClusterMesh API state synchronization._

###### **How Global Load Balancing Works**

When a pod in Cluster A attempts to reach a Global Service, the hostname resolves to a standard local ClusterIP via internal CoreDNS. The eBPF datapath then hooks the connection at the socket layer, dynamically translating that single virtual IP into the direct backend Pod IPs of healthy instances across the entire synchronized multi-cluster mesh.

###### **Service Affinity**

While load balancing across clusters is powerful, cross-cluster traffic often incurs higher latency and egress costs. Cilium solves this using the service.cilium.io/affinity annotation.

- By setting affinity to local, Cilium will prioritize routing traffic to backend pods within the same cluster.
- It only spills traffic over to the remote cluster if all local endpoints fail or degrade, ensuring optimal performance under normal conditions while maintaining disaster recovery capabilities.

_Fig 6: Cilium service affinity traffic is prioritized to local backend pods within each cluster, with cross-cluster spillover (dashed) triggered only when all local endpoints in Cluster 1 fail or degrade._

###### **Shared Services**

Using a Hub and Spoke topology, global services allow organizations to centralize common operational tools. Cilium Cluster Mesh provides cloud architects and platform operators with the flexibility to centralize commonly used services like shared secrets, DNS, and other global services from a hub. This prevents you from having to deploy heavy monitoring or vault instances into every cluster.

###### **Handling Unreachable Clusters**

Because Cilium leverages the underlying capabilities of Kubernetes, there is no need to modify any of Kubernetes' fundamental architectures. Services that are available to other clusters are kept healthy using Kubernetes' liveness and readiness probes, where services are added and removed as necessary when pods scale up and down, or they become unhealthy. If a cluster goes offline, the cluster mesh-apiserver instantly removes those endpoints from the mesh, ensuring traffic is only routed to healthy clusters.

[Read About Setting Up a ClusterMesh](https://docs.cilium.io/en/stable/network/clustermesh/clustermesh/)

##### **VII. Network Policy Across Cluster Boundaries**

Extending connectivity across clusters introduces new security challenges. If Cluster A is compromised, you must ensure the attacker cannot freely pivot into Cluster B.

###### **How Identity-Based Policy Works Across Clusters**

Because network policies are implemented at the kernel level, all traffic can be observed and monitored with Prometheus and visualized with Hubble without requiring developers to instrument their code.

The key to seamless pod-to-pod networking is Cilium's implementation of identity-based networking that identifies which pods and services can communicate using labels instead of IP addresses to enforce access controls. When traffic leaves one cluster, the security identity (represented as a numeric ID) is embedded in the network packet. The receiving cluster evaluates this identity against its own local network policies before allowing the packet to reach the destination pod.

###### **Writing Cross-Cluster Policies**

Writing a cross-cluster policy is identical to writing a local CiliumNetworkPolicy. Because identities are synchronized, you can restrict traffic based on cluster names. For example, using the **_io.cilium.k8s.policy.cluster_** label, you can write a policy stating that the frontend pod in cluster-1 is only allowed to talk to the backend pod in cluster-2, effectively micro-segmenting your mesh.

###### **Default-Deny Across Cluster Boundaries**

The best practice in Kubernetes security is the Default-Deny posture. By creating a baseline policy that denies all ingress and egress traffic, you force developers to explicitly define what cross-cluster communication is permitted. Cilium honors this posture seamlessly; if a global service does not have an explicit network policy allowing the cross-cluster identity, the eBPF datapath will drop the packets at the source node.

###### **FQDN Policy in Multi-Cluster**

Cilium also supports DNS-based (FQDN) policies across clusters. If a pod in a remote cluster needs to reach an external API (like api.stripe.com), you can enforce egress security at the DNS level, ensuring that even within a vast multi-cluster mesh, workloads can only communicate with approved external domains.

###### **Limitations**

It is important to note that standard ClusterMesh deployments require non-overlapping Pod CIDRs across all clusters. While Isovalent Enterprise for Cilium offers advanced NAT capabilities for overlapping IPs, using standard open source Cilium with overlapping subnets can severely complicate identity propagation and direct pod routing.

##### **VIII. Multi-Cluster Observability with Hubble**

Maintaining visibility in a multi-cluster environment is crucial for troubleshooting and security purposes. When traffic traverses multiple infrastructure boundaries, blind spots are unacceptable.

###### **Hubble**

Hubble is built on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure. Because the identity-aware datapath spans the mesh, Hubble provides platform teams with a single management plane for load-balancing, observability, and security enforcement between nodes from multiple Kubernetes clusters.

_Fig 7: Cilium Hubble UI showing cluster mesh visualization._

###### **Filtering Flows by Cluster**

Using the Hubble CLI or UI, operators do not need to guess where a packet was dropped. Because the cluster name is attached to the flow identity, an engineer can instantly filter network telemetry to show only cross-cluster traffic (e.g., traffic originating in cluster-aws and destined for cluster-onprem).

###### **Cross-Cluster Dependency Mapping**

Through the Hubble UI, security and platform teams can view a real-time, graphical dependency map. If a global service spans three clusters, Hubble visually maps the traffic flows between them, indicating protocol types, bandwidth usage, and highlighting dropped packets in red. This allows SREs to instantly pinpoint if a multi-cluster outage is due to a crashed pod, a misconfigured network policy, or a cloud provider network partition.

##### **IX. Summary**

Multi-cluster Kubernetes is no longer an edge case; it is a necessity for enterprise-grade resilience and scale. Organizations now require the ability to control and observe multi-cluster architectures that can seamlessly discover services, load balance across zones, perform cross-cloud disaster recovery, and deliver low-latency services globally.

Although Kubernetes makes it simple to scale distributed applications on a single cluster, its native networking model introduces significant operational complexity and performance overhead when attempting to connect workloads across multiple clusters. Historically, bridging these environments meant relying on heavy proxies, API gateways, and external load balancers.

By leveraging eBPF-native solutions like Cilium, organizations are overcoming these networking bottlenecks. Through features like ClusterMesh, Cilium provides the high-performance, identity-aware infrastructure required to treat multiple clusters as a single, cohesive environment. It preserves workload identity for all cross-cluster traffic, meaning that security policies and network observability tooling such as Cilium's Hubble continue to operate exactly as they do within a local cluster.

#### Kubernetes Runtime Security: The Guide

URL: https://cilium.io/blog/2026/06/07/kubernetes-runtime-security-the-guide
Date: 2026-06-07
Categories: Security, Technology

##### **Kubernetes Runtime Security: The Guide**

##### **I. Introduction**

Runtime security is the continuous process of protecting applications, workloads, and infrastructure while they are actively running. While pre-deployment security focuses on securing the blueprint, runtime security is about defending the live environment. It is the active surveillance and defense of your cluster against zero-day exploits, insider threats, and post-exploitation behavior.

For years, the gold standard of container security has focused heavily on pre-deployment gates, such as vulnerability image scanning and static configuration checks. While crucial, this approach is entirely predictive; it only catches known risks.
True security requires moving from static gates to active protection. Security cannot stop when a container passes a scan and enters production. Runtime security picks up exactly where the continuous integration/continuous deployment (CI/CD) pipeline ends, monitoring real-time system calls, network behavior, and file integrity as code executes.

This highlights the need for deep visibility into all system calls and processes executing within the operating system, as well as the workloads that run on top of it.

_Figure 1: Key aspects of Runtime Security that the Cilium ecosystem offers._

###### **Why Scanned Images Still Need Active Runtime Monitoring**

A clean container scan is a green light, not a lifetime guarantee. Relying solely on pre-deployment image scanning leaves a massive blind spot for several critical reasons:

- Zero-Day Vulnerabilities: An image scan only flags vulnerabilities that are already cataloged in public CVE databases. It cannot protect against zero-day exploits or vulnerabilities that exist in your running code but have not yet been discovered by the security community.
- Configuration Drift and Runtime Injections: Even if an image is perfectly secure, the surrounding configuration can change. Attackers can exploit exposed Kubernetes APIs or environment variables to inject malicious payloads, modify binaries, or execute unauthorized code directly into memory at runtime.
- Insider Threats and Credential Theft: If an attacker steals valid credentials or compromises a trusted third-party dependency (a supply chain attack), they can bypass perimeter checks entirely. In this case, without runtime monitoring, these events can’t be spotted and mitigated.

###### **Cilium Tetragon**

**Tetragon** is a specialized cloud native runtime security and observability solution built from the ground up to leverage eBPF. By compiling and executing security logic directly within the Linux kernel, Tetragon transforms runtime security from a slow, reactive log-parsing exercise into an intelligent, low-overhead, and identity-aware shield.

Instead of sitting outside the application looking in, Tetragon operates at the lowest level of the operating system to deliver deep behavioral monitoring and enforcement.

###### **Why Tetragon**

Tetragon brings the following capabilities:
**Kernel-Level Precision:** Tetragon intercepts system execution at the kernel boundary. It doesn't just watch network packets; it tracks system calls (syscalls), file access, namespace changes, and process execution with cryptographic certainty.

**Kubernetes-Aware Identity:** Tetragon automatically maps raw kernel-level events (like a process execution or a file write) to high-level Kubernetes metadata (Pods, Namespaces, Deployments, and Labels). Instead of telling you that "PID 4052 executed a binary," Tetragon tells you that "the frontend pod in the production namespace executed a shell."

**Real-Time Mitigation and Inline Enforcement:** Unlike other tools that alert you _after_ a breach has occurred, Tetragon can be configured to take action. Because it operates in the kernel, it can instantly block a malicious action, such as killing a compromised process or shutting down a socket connection just after a security policy is violated, preventing lateral movement before an attacker can take a single step. All these happen via user-defined policies of what should be allowed and denied.

_Figure 2. Tetragon Architecture for Real-Time Enforcement. This diagram illustrates how Tetragon leverages eBPF within the Linux kernel to monitor deep system activities like process execution, file access, and network traffic._

[Read More about Runtime Security](https://isovalent.com/blog/post/what-is-runtime-security/)

##### **II. The Kubernetes Runtime Threat Model**

To defend a Kubernetes cluster, you must understand how attackers operate once they bypass your initial defenses. In a cloud native world, the perimeter is porous. Attackers do not necessarily need to breach a firewall; they might exploit a zero-day vulnerability in a public-facing web app or leverage a compromised third-party library.

Kubernetes runtime security threat model unfolds via the following vectors.

**i) Container Escape & Privilege Escalation**
Containers are not virtual machines; they are just isolated processes. If an attacker compromises a pod, their first goal is often to escape it. By exploiting a vulnerability in the shared Linux kernel or abusing misconfigured privileges (like mounting the host filesystem or running as root), an attacker can break out of their containerized namespace to gain root access to the underlying Kubernetes worker node.

**ii) Lateral Movement**
Once a foothold is established, attackers attempt to map the internal network and move "east-west" across the cluster. In a default, flat Kubernetes network, a compromised frontend pod can often communicate directly with a backend database or internal API. Attackers exploit this lack of identity-aware segmentation to pivot to higher-value targets.

**iii) Cryptomining and Resource Abuse**
Not all attacks are designed to steal data. Often, attackers simply want to hijack your computing resources. Cryptomining is one of the most common runtime threats, where attackers silently deploy malicious payloads that consume your cluster's CPU and memory limits, driving up cloud bills and starving legitimate applications of resources.

**iv) Data Exfiltration**
The ultimate goal of many breaches is stealing sensitive information, customer databases, proprietary code, or API keys. At runtime, data exfiltration often looks like a legitimate pod suddenly initiating outbound connections to an unknown external IP address or using DNS tunneling to quietly push data out of the cluster.

**v) Supply Chain Attacks**
Sometimes, the threat is already inside the house. Supply chain attacks occur when malicious code is injected into a trusted open source dependency or base image. These attacks lie dormant during CI/CD checks and only wake up at runtime to execute unauthorized processes or phone home to command-and-control (C2) servers.

###### **Why Container Isolation is not Enough**

Containers don’t provide a hard security boundary in the system they run. Relying on basic container isolation and Kubernetes-native controls is not enough for defending against advanced runtime threats.

**i) The Linux Kernel is Shared**
Unlike virtual machines, which each have their own operating system, every container on a Kubernetes node shares the same underlying Linux kernel. Containers rely on Linux features like namespaces (for isolation) and cgroups (for resource limits). If an attacker finds an exploit in the shared kernel, the namespace illusion vanishes. Compromising the kernel means compromising every single pod running on that node.

**ii) Runtime dynamics**
Image scanners are essentially spell-checkers for vulnerabilities. They compare your container's filesystem against a known list of bad signatures (CVEs). However, they cannot prevent an attack where an attacker uses legitimate, scanned tools (like curl, wget, or bash) to download malware and execute it in memory.

**iii) Kubernetes RBAC Does Not Control Syscalls**
Kubernetes Role-Based Access Control (RBAC) is excellent for the control plane. It determines who can delete a pod, read a secret, or scale a deployment. However, RBAC has zero visibility into the data plane or the host OS. RBAC cannot control which system calls applications can make.

[Read Why Runtime Security Matters in Modern Cloud Environments](https://isovalent.com/blog/post/what-is-runtime-security/#why-runtime-security-matters-in-modern-cloud-environments)

##### **III. How Tetragon Works: eBPF-Based Kernel Visibility**

To understand why Cilium Tetragon is so highly regarded for runtime security, you have to look at where it operates: directly inside the Linux kernel via eBPF. Tetragon is not like most security tools that sit in user-space, guessing what applications are doing. Instead, it turns the kernel itself into a security sensor and enforcement engine.

_Figure 3. Tetragon's eBPF-Based Kernel Visibility. This diagram demonstrates how Tetragon bypasses user-space limitations by operating directly inside the Linux kernel._

###### **Kernel is a Zero Ground**

In any Linux-based system (which includes virtually all Kubernetes worker nodes), the kernel is the zero ground. Applications and containers cannot access the network, read a file, or start a new process on their own. They must politely ask the kernel to do it for them via a System Call (syscall).

Because every single action must pass through the kernel, it is the ideal place for security enforcement. If a security tool lives in user-space, a compromised container with root privileges can blind, bypass, or kill the tool. But when security is embedded in the kernel via eBPF, it is effectively invisible and untouchable to the workloads running above it. Tetragon deploys eBPF programs as sensors, observing the entire system from a privileged vantage point.

###### **Probes**

In the eBPF ecosystem, **probes** are the mechanisms used to attach an eBPF program to a specific execution point in the system. They act as sensors that wake up and collect data the exact moment a specific function is called. Probes are generally used for observation and tracing.

###### **Types of Probes**

There are three primary types of probes used by Tetragon:

**i) kprobes (Kernel Probes)**
These are dynamic. They allow Tetragon to attach an eBPF program to almost any function inside the Linux kernel. If you want to know exactly when the kernel function responsible for opening a file (do_sys_open) is triggered, a kprobe intercepts it, records the data, and returns control to the kernel.

**ii) tracepoints**
These are static, hardcoded markers intentionally placed by Linux kernel developers in the source code specifically for debugging and tracing. They are highly stable across different kernel versions, making them incredibly reliable for observing standard kernel behavior (like network packet transmission or system calls).

**iii) uprobes (User Probes)**
While kprobes watch the kernel, uprobes watch user-space applications. Tetragon can use uprobes to hook directly into the functions of a running application or a shared library. A common use case is attaching a uprobe to an SSL/TLS library (like OpenSSL) to inspect data before it gets encrypted and sent over the network.

###### **Hooks**

Hooks are designed for enforcement, while probes are meant for watching. Hooks are designed for active enforcement, which is facilitated by the Linux Security Module (LSM).
LSM hooks are used strategically to remediate access to kernel internal objects.

###### **Process Ancestry**

Ancestry is a genealogical process of how a process is born. In Linux, every process (except the very first init process) is spawned by another process, creating a parent-child relationship. When monitoring for kernel security events, a single isolated process can’t tell the whole story about itself.
Knowing who started a process is what determines the intent, whether it’s a runtime breach or just a normal action.
Tetragon tracks the process execution chain, which gives a context of what started with who. A curl being executed by a node worker agent sounds like a routine, while a curl executed by a system user tells a different intent.

###### **O(1) Efficiency and In-kernel Filtering**

Traditional security and networking tools often struggle with performance at scale. For example, legacy network filtering (like iptables) processes rules sequentially (an O(n) operation). If you have 10,000 rules, the system checks them one by one, causing massive CPU overhead and latency.
eBPF solves this by using eBPF Maps, highly efficient, constant-time (O(1)) hash tables that live in the kernel. When Tetragon checks if an IP is malicious or if a process is allowed, it performs a near-instantaneous lookup, regardless of how many rules are applied.

###### **Context Enrichment**

A raw kernel event is extremely low-level. The kernel only knows that Process ID (PID) 4096 opened a network connection. To a security analyst, PID 4096 is meaningless. Tetragon solves this via Kubernetes Context Enrichment.
It continuously maps raw PIDs, network namespaces, and cgroups to high-level Kubernetes constructs. When Tetragon outputs a security event, it tells you exactly which Kubernetes Namespace, Pod Name, Container ID, and label were responsible for the kernel action.

_Figure 4. Kubernetes Context Enrichment. This enriched process tree illustrates how Tetragon translates meaningless, low-level kernel events (like raw PIDs) into actionable security insights mapped directly to specific Kubernetes resources._

[Read More About Tetragon Events](https://tetragon.io/docs/concepts/events/)

##### **IV. Process Execution Monitoring**

The most fundamental signal in runtime security is the execution of processes. Every attack, whether it is a sophisticated data exfiltration campaign, a cryptominer, or a simple lateral movement script, must eventually execute code. If you can definitively answer what is running, spawned by what, and as which user, you have the foundation to stop almost any runtime threat.

Because Cilium Tetragon operates directly within the kernel via eBPF, it observes process execution with perfect fidelity. Clever user-space tricks or fast-executing scripts cannot bypass it.

###### **Process Exec and Exit Events**

In a Linux system, **process_exec** is used to trace when a process starts, and when it exits, either by completing a termination signal, **process_exit** is invoked.
Tetragon captures an active user ID, command line arguments passed, and a binary name being executed; likewise, on exit, a process exit code is captured.

###### **Why Process Ancestry Matters**

A single process execution rarely tells the whole story. Being capable of knowing the process call tree on the other end gives a clearer intent on who is doing what using what.

To accurately detect threats without generating a massive amount of false positives, security tools must understand the process ancestry.

###### **Detecting Reverse Shells**

A reverse shell is a remote access technique where a compromised target machine initiates an outbound connection to an attacker-controlled system. This is sometimes called the phone home technique. Since most of the default configurations leave the egress traffic unprotected, outbound connections are likely allowed.

Tetragon’s ability to correlate process execution and network activity can help to easily detect what is actually initiating those outbound connections.

**Detecting Fileless Execution**
Most anti-virus scanners and sanitizing programs rely on examining files on a disk. Attackers know this, which changes their strategies to fileless malware.
Using utilities like memfd_create, attackers can create anonymous files and return the file descriptor that references them. This makes it possible to download payloads/scripts directly into RAM and execute them without writing the file to disk.

Up to that point, the attackers can clearly bypass scanners and anti-virus; the only point to catch them is in the kernel.

**Binary Allow-Listing**
The runtime security model has shifted from reactive to proactive enforcement. Instead of a reactive block listing, you proactively allow which binaries to execute. Since container runtimes are built with a purpose, all the utility programs they rely on are known, and only those are allowed to be executed at runtime.

[Read More About eBPF-based Security Observability & Runtime Enforcement](https://isovalent.com/blog/post/2022-05-16-tetragon/)

##### **V. Network Runtime Security**

Network runtime security protects actively running applications and workloads by monitoring live traffic and system behavior in real-time. It operates on the assumption that some threats will bypass perimeter defenses, actively blocking malicious actions like data exfiltration, unauthorized connections, and lateral movement as they happen.

Network runtime security requires connecting the runtime process identity to network behavior. You must answer not just "which IP is talking to which IP," but "which specific process, spawned by which user, is opening this socket."

###### **Process to Network Correlation**

Cilium Tetragon can link every single network connection to the exact process that opened it, complete with its full ancestry tree.
Imagine a security alert flags an outbound connection from your web server pod to an external IP. A traditional network monitor only tells you the pod's IP and the destination IP.

With eBPF and Tetragon, you get the full story:

- Scenario A: The connection was opened by the nginx application binary itself, routing legitimate user traffic.
- Scenario B: The connection was opened by curl, which was spawned by bash, which was spawned by nginx.

Scenario B tells a drastically different story: the web server was exploited to open a reverse shell, which is now attempting to download a malicious payload. Tetragon’s process-to-network correlation makes this distinction instantly obvious.

_Figure 5. Process to Network Correlation. Tetragon visualizes the exact process ancestry for every network connection, instantly distinguishing legitimate application traffic from suspicious spawned child processes._

###### **Detecting Unexpected Outbound Connections**

In the event of a security breach on a host system, once an attacker gains a foothold in a container, their next move is almost always to establish a connection to an external Command and Control (C2) server or to download additional malware.

Because Tetragon monitors the kernel's network socket requests in real time, it can detect and block unexpected outbound connections the millisecond a compromised process attempts them. By defining baseline network behaviors for your workloads, any deviation, like a Redis database suddenly initiating an outbound HTTPS request, is immediately flagged and blocked at the kernel level.

###### **DNS Exfiltration Detection**

Data exfiltration doesn't always happen over obvious HTTP/TCP connections. Attackers frequently use DNS tunneling to sneak sensitive data out of a cluster. They encode stolen data into DNS queries (e.g., _user-password-hash.malicious-domain.com_), knowing that most firewalls allow outbound DNS traffic by default.

Tetragon monitors DNS requests directly at the socket and process level. It can enforce policies that restrict which processes are allowed to make DNS queries or flag abnormally large/frequent DNS requests originating from non-system processes, neutralizing a common exfiltration vector.

###### **Monitoring tcp_connect and tcp_close**

Most network monitoring tools rely on packet sniffing (like tcpdump or deep packet inspection) to analyze traffic. This is incredibly CPU-intensive and struggles at scale.

Instead of looking at the packets on the wire, Tetragon hooks directly into the Linux kernel functions that manage the lifecycle of a connection, specifically tcp_connect (when a connection is initiated) and tcp_close (when it ends). This provides granular, O(1) efficiency visibility into network state, duration, and bandwidth usage without the overhead of capturing and parsing raw packets.

###### **Runtime Network Enforcement vs. Network Policy**

It is essential to distinguish between standard Kubernetes Network Policies or Cilium Network Policies and Tetragon's runtime network enforcement. Cilium Network Policy (CNP) operates at the network layer.

Tetragon Runtime Enforcement operates at the process layer. It dictates that only the node process inside Pod A is allowed to initiate the connection to Pod B. If an attacker breaches Pod A and attempts to use netcat to connect to Pod B, Tetragon blocks the connection, even though the CNP technically allows pod-to-pod traffic.

Using both together creates a zero-trust architecture that secures both the network perimeter and the internal execution environment.

_Figure 6. Runtime Network Enforcement. Tetragon operates at the granular process layer via eBPF to filter system calls and file access, blocking malicious application behaviors even if network-layer policies technically permit the connection._

1. [Read More About Hyperscaling Security With Isovalent Enterprise](https://isovalent.com/blog/post/hyperscaler-security/)
2. [Read About Top Tetragon Use Cases (1)](https://isovalent.com/blog/post/top-tetragon-use-cases/)
3. [Read About Top Tetragon Use Cases (2)](https://isovalent.com/blog/post/top-tetragon-use-cases-part-2/)

##### **VI. Privilege Escalation Detection**

Privilege escalation is a form of cyber attack where an attacker exploits a bug, design flow, or misconfiguration in the operating system or application to gain elevated unauthorized access to resources.

This escalation can be vertical, where a user elevates privileges of the same user account, or horizontal, which involves an attacker switching from one user with low privileges to another one with high privileges.

In a Kubernetes environment, privilege escalation rarely looks like a traditional password cracking attempt. Instead, it involves abusing the shared Linux kernel and the porous boundaries of containerization. Attackers look for misconfigurations such as pods running with privileged: true, mounted host filesystems, or overly permissive Linux capabilities.

When a pod is compromised, the attacker's immediate goal is to mutate their current restricted identity into a highly privileged one (usually root on the host operating system). Tetragon monitors the exact system calls that facilitate this mutation.

_Figure 7. Kubernetes Privilege Escalation. This diagram illustrates how an attacker exploits a compromised, non-secured pod to execute malicious commands and break out of container isolation to access underlying host resources._

[Read More About Container Escape Attacks](https://isovalent.com/blog/post/2021-11-container-escape/)

###### **SUID Binary Execution Monitoring**

One of the oldest privilege escalation techniques in Linux involves SUID (Set Owner User ID) binaries. If a binary has the SUID bit set, it executes with the privileges of the file's owner (often root), regardless of who actually ran the command (like sudo or passwd). Attackers hunt for custom or vulnerable SUID binaries within a container to execute arbitrary code as root.

Tetragon hooks into process execution (process_exec). It evaluates the file metadata at the kernel level, instantly detecting and logging when an unauthorized user attempts to execute an SUID binary to elevate their permissions.

[Read More About Process Execution](https://tetragon.io/docs/use-cases/process-lifecycle/process-execution/)

##### **VII. Tracing Policy: Writing and Managing Runtime Policies**

Visibility and enforcement are only good as the rules that govern them. In the Cilium and Tetragon ecosystem, these rules are defined declaratively using a Kubernetes Custom Resource definition (CRD) known as TracingPolicy, for clusterwide rules, TracingPolicyNamespaced for namespace-specific rules.

###### **Tracing Policy Anatomy**

A tracing policy has three components:
i) Hook Point: The exact location in the Linux kernel where Tetragon will attach its eBPF sensor can be kprobes for internal kernel functions or tracepoints for static kernel markers, and uprobes for user-space application functions.

ii) Selectors: The in-kernel BPF filtering logic that determines if the hooked event should be processed or ignored.

iii) Actions: The specific response Tetragon executes when an event matches the hook and selectors criteria.

###### **Selector Types**

Passing every single kernel event to the user space for analysis would immediately overload a system. Tetragon addresses this by using selectors to evaluate context and drop irrelevant events directly inside the kernel.

Selectors allow security teams to create highly granular rules.
i) matchArgs & matchReturnArgs: filters based on the exact arguments passed to a system call.
ii) matchBinaries & matchParentBinaries: restricts policy to specific execution chains.
iii) matchNamespaces & matchCapabilities: detects when a process attempts to change its isolation boundaries or elevate its Linux privileges.
iv) Kubernetes Identity Filters: ensure policies only apply to specific workloads based on Kubernetes labels, rather than blanket application of rules.

[Read More About Policy Selectors](https://tetragon.io/docs/concepts/tracing-policy/selectors/)

###### **Action Types**

Once an event matches the hook and passes the selector filters, Tetragon executes an Action. Because the eBPF program runs synchronously within the kernel, these actions occur before the system call completes.

1. Post (Default): Generates a rich, JSON-formatted observability event and sends it to the user-space agent or a SIEM integration.
2. NoPost: Silently drops the event. This is incredibly useful for filtering out known, safe noise.
3. Sigkill: The cornerstone of runtime enforcement. It instantly sends a fatal kill signal to the process attempting the malicious action, terminating the threat before it executes.
4. Override: Instead of killing the process, this action seamlessly alters the return value of the system call. For example, it can return an EPERM (Permission Denied) error to the application, thwarting the attack without crashing the entire container.

_Figure 8: Tetragon hook points across the Linux kernel file I/O and network execution paths._

[Read About How to Use Tetragon Without Cilium](https://isovalent.com/blog/post/can-i-use-tetragon-without-cilium-yes/)

###### **Policy Rollout Strategy**

When applying for a new policy, it is advised not to start with enforcement directly until the impact has been analyzed.

New policies should be applied with detection mode first, using the POST action first. Monitor the output in SIEM tools.
Once a legitimate application is confirmed, the policy can then be put in enforcement mode.

[Read More About Tetragon Tracing Policy](https://tetragon.io/docs/concepts/tracing-policy/)

##### **VIII. Compliance Use Cases**

Security is about stopping the attack; compliance is about proving to an auditor that you can and did stop the attack.
In Kubernetes, where pods are ephemeral, workloads share the underlying compute resources. Three different application pods can be scheduled on the same host. This increases the risk of security attacks, as a single compromised host grants an attacker access to most workloads, provided network policies permit connectivity.
To remediate the risk, Tetragon can be used to implement security compliance standards.

###### **Egress Security**

Many compliance mandates require strict control over how data is transmitted outside the environment. However, businesses often need their modern Kubernetes microservices to communicate with legacy, on-premises databases, or third-party APIs (like payment gateways) that are protected by firewalls.

Because Kubernetes pod IPs constantly change, configuring a traditional firewall to allow traffic from a cluster usually means opening wide IP ranges, a massive compliance violation.

Cilium solves this through two primary egress controls:

i) DNS-Based Egress (toFQDNs): Instead of allowing an IP range, Cilium allows egress policies based on domain names. You can write a policy stating that a pod can only communicate with api.stripe.com. Cilium dynamically resolves the IP and enforces the policy at the kernel level, regardless of how often Stripe's IPs change.

ii) Static Egress Gateway: Cilium can route traffic from specific, dynamic Kubernetes namespaces through a highly available gateway node, assigning the traffic a stable, static IP address. This allows you to integrate dynamic Kubernetes workloads seamlessly with firewalls, maintaining strict, IP-based compliance without sacrificing cloud-native scalability.

**\*Figure 9\. Cilium Static Egress Gateway.** This illustrates how Cilium routes dynamic cluster traffic through a central gateway node, providing a stable, predictable IP address to satisfy strict external firewall requirements.\*

###### **Host-Level Protection**

A core requirement of almost all security frameworks is protecting the underlying operating system. If an attacker bypasses a container and compromises the worker node, the entire cluster is at risk.

Tetragon’s eBPF visibility does not stop at the container boundary; it extends to the host itself. It provides native **File Integrity Monitoring (FIM)** to detect unauthorized changes to critical host files (like /etc/shadow or /etc/kubernetes/manifests), logs all host-level syscall activity, and alerts on privilege escalation attempts. This ensures that the infrastructure housing your workloads is just as auditable as the workloads themselves.

_Figure 10. File Integrity Monitoring. File Monitoring with eBPF and Tetragon._

[Read About Top Tetragon Use Cases (2)](https://isovalent.com/blog/post/top-tetragon-use-cases-part-2/)

###### **Compliance Frameworks**

i) **PCI-DSS (Payment Card Industry Data Security Standard)**: PCI-DSS requires strict microsegmentation of the Cardholder Data Environment (CDE). Using Cilium’s identity-aware Layer 3, Layer 4, and Layer 7 Network Policies, you can enforce a Zero Trust architecture, cryptographically proving that out-of-scope pods cannot communicate with the CDE. Furthermore, Cilium's transparent encryption (IPsec/WireGuard) seamlessly encrypts cardholder data in transit without requiring application code changes.

ii) **SOC 2 Type II**: SOC 2 requires continuous monitoring, strict access controls, and robust incident response capabilities over an extended period. Tetragon’s ability to track process ancestry and correlate it with network activity provides a tamper-proof audit trail. If an auditor asks, "Who accessed this database last month?", Tetragon provides the exact pod identity, binary, and user that opened the connection.

iii) **NIST SP 800-190 (Application Container Security Guide):** This NIST framework specifically dictates the need to monitor container runtime behaviors, network traffic, and filesystem access. Tetragon is essentially purpose-built for NIST 800-190, directly addressing its requirements by enforcing binary allow-lists, restricting sensitive system calls, and blocking anomalous network connections synchronously in the kernel.

**iv) CIS Kubernetes Benchmark:** While the CIS Benchmark is primarily a prescriptive guide for static configuration (like disabling root access or securing the Kubelet API), Tetragon acts as a powerful compensating control. If a legacy application must run with elevated privileges (violating a CIS benchmark), Tetragon can heavily restrict and monitor exactly what that privileged container is allowed to do at runtime, satisfying auditors who require mitigated risk.

[Read More About Cilium for Compliance](https://isovalent.com/blog/post/implementing-cilium-for-compliance-use-cases-controlplane-isovalent-whitepaper/)

##### **IX. Summary**

The cloud-native ecosystem has reached a tipping point. However, this massive adoption has created an equally massive, dynamic attack surface that legacy tools simply cannot protect.

The industry is collectively realizing that preventing attacks at the perimeter is no longer sufficient; security must exist where the applications actually live and execute.
Moving forward, organizations must adopt a true Zero-Trust architecture that spans from the network layer (Cilium CNI) all the way up to the application and process layer (Tetragon). Threats like fileless malware, supply chain compromises, and zero-day exploits become more sophisticated.

#### Understanding Kubernetes Network Security

URL: https://cilium.io/blog/2026/06/07/understanding-kubernetes-network-security
Date: 2026-06-07
Categories: Security, Technology

##### Understanding Kubernetes Network Security

##### 1\. Introduction

The ephemeral nature of cloud-native workloads has fundamentally changed the Network Security models. In a microservices architecture where resources are constantly being created and destroyed, firewalls based on IP addresses are not enough. The current state of the art is shifting towards Kubernetes-native, Identity-Based Security, a necessary evolution to keep pace with the container orchestrator.

Kubernetes network security is transitioning from IP/port-based rules to intent-based policies defined by Kubernetes-native identity and metadata, such as Pod names and labels. Cilium, leveraging eBPF technology, facilitates this shift by enabling security policy enforcement directly inside the Linux kernel’s data plane. This approach provides a robust, high-performance solution that understands the application context of communication, a key requirement for cloud-native platforms.

_Figure 1: Cilium Agent node-level architecture translating Kubernetes network policies and identities into eBPF enforcement rules for pod traffic._

###### Network Attack Surfaces

A significant security hole is the Layer 4 (L4) limitation of legacy tools and native Kubernetes Network Policies. While a policy might correctly restrict traffic to a specific port, such as Port 8080/TCP, it offers no visibility into the actual application-layer intent. Once the port is open, an attacker who compromises a Pod can execute unauthorized actions like DELETE /orders or POST /admin/shutdown just as easily as a legitimate GET /items. This Layer 7 (L7) blindness exposes the application to API-based attacks, highlighting the risk of relying solely on basic Layer 3/4 filtering.

##### 2\. Limitations of the Legacy Network Security Model

The foundational principles of traditional network security were designed for a static world, making them insufficient to handle the dynamic nature of cloud-native environments and Kubernetes. The shortcomings of legacy approaches are clear in four key areas:

###### Inability to Handle Ephemeral IP Addresses

In Kubernetes, Pods are ephemeral, constantly spinning up, moving, and being destroyed, which leads to unpredictable IP address changes (IP Churn). Traditional security models rely on fixed IP addresses to define firewall rules, an approach that breaks down when IP addresses are volatile. The modern approach, championed by Cilium, shifts this focus from volatile IP addresses to Identity-based security, where unique security identities are assigned to Pods based on Kubernetes metadata and labels. This means that the security policy is based on intent (e.g., "Allow role: frontend to talk to role: backend") rather than specific, rapidly changing IP addresses.

###### Performance Bottlenecks

Most networking technologies often introduce a performance bottleneck, frequently relying on complex rule processing. For instance, the use of technologies like iptables or IPVS limits both performance and scalability. With these technologies, adding more Pods means the firewall has to check more rules, resulting in an O(n) complexity. Cilium, by leveraging eBPF (Extended Berkeley Packet Filter), moves networking and security logic directly into the Linux kernel. This allows security enforcement to happen with constant time complexity, or O(1), by using kernel-level eBPF maps (high-performance hash tables) for identity and policy verification, enabling faster policy evaluation with low CPU overhead.

###### Lack of Identity-Awareness

Modern distributed applications rely on technologies such as containers to facilitate agility in the deployment of new versions of their application and to scale out on demand. This results in a large number of containers starting in a short period of time. Typical container firewalls secure workloads by filtering source IP addresses, destination IP addresses, and ports. IP addresses in Kubernetes are ephemeral. Traditional firewalls are also not cloud-native-aware and are mostly not capable of being programmed dynamically when applications scale out or new versions are deployed. Updating the Firewall constantly to adapt to the constant changes becomes impossible at scale.

###### Default-Allow

The default state of Kubernetes networks is a flat network structure where, by default, all pods can communicate with each other. This "default-allow" posture creates a significant security hole, allowing an attacker who compromises a single container to move laterally throughout the cluster (lateral movement). To mitigate this risk, a Default Deny strategy must be implemented as part of a Zero Trust Architecture. This approach aggressively blocks all traffic or actions by default, allowing only explicitly authorized activities. Tools like Hubble, Cilium's observability layer, help in this transition by visualizing existing traffic before enforcement, allowing teams to prototype and test policies in Audit Mode before moving to full enforcement.

##### 3\. Kubernetes Network Security with eBPF

The shift from IP-based security to Kubernetes-native identity requires a foundation that can operate at the data plane with speed and context. This foundation is the Extended Berkeley Packet Filter (eBPF). Cilium uses eBPF (extended Berkeley Packet Filter) to overcome these limitations by moving security to the kernel level.

By leveraging eBPF, Cilium gets the ability to insert security rules based on service/pods/container identity rather than an IP address for identification, as in the traditional system. As a result, eBPF makes applying security policies in a dynamic container environment scalable by decoupling security from IP addressing, providing stronger security isolation

###### The Kernel as the Ideal Enforcement Point

The Linux kernel is the primary handler for all networking traffic. eBPF allows network and security logic to be safely embedded directly into the kernel's data path, executing security policy decisions at the earliest possible moment without relying on slow, outdated methods like iptables or IPVS.

_Figure 2: Runtime observability interface correlating container process trees with specific network connection events._

###### Cilium Network Policies and Layer 7

Cilium leverages eBPF to extend basic Layer 3/4 Kubernetes Network Policies to Layer 7 (Application Layer) controls. This enables fine-grained, API-aware segmentation, meaning policies can allow specific HTTP methods (e.g., GET /items) while blocking others (e.g., POST /admin/shutdown), strictly enforcing the Principle of Least Privilege.

_Figure 3: Comparison of L3, L4, and L7 network policies, illustrating Cilium's ability to enforce fine-grained, API-aware security controls._

##### 4\. Pillars of Kubernetes Network Security

Kubernetes network security is strongest when it is treated as a layered model, not a single control. In dynamic clusters, pod IPs change, services scale up and down, and traffic patterns shift constantly. Cilium’s approach, powered by eBPF, focuses on enforcing security close to the workload while using Kubernetes identity, metadata, and application-aware context instead of relying only on IP addresses.

_Figure 4: Core networking, security, and observability layers unifying multiple Kubernetes clusters into a single, seamless environment._

###### Identity-Based Access Control

The first pillar is identity. Traditional firewalls often depend on IP addresses and ports, but Kubernetes workloads are short-lived and constantly rescheduled. Cilium decouples security from network addressing by assigning identities based on Kubernetes metadata such as labels, namespaces, pods, and service accounts.

This enables policies such as “frontend can talk to backend on port 443” or “only this service account can call this API,” even as pods move across nodes or receive new IPs. Isovalent describes this as a foundation for microsegmentation and least-privilege access across services.

###### Network Policy and Microsegmentation

Network policies define which workloads can communicate, in which direction, and on which ports or protocols. Kubernetes NetworkPolicy provides basic ingress and egress controls at L3/L4, while Cilium extends this with CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy for richer enforcement across L3-L7.
With Cilium, teams can implement:

- Namespace isolation
- Default-deny policies
- Pod-to-pod and service-to-service allow rules
- Egress controls for external services
- Cluster-wide policies for shared security baselines
- Layer 7 controls for HTTP, DNS, and other application protocols

This turns the cluster network into a set of explicit trust boundaries instead of a flat network where every workload can reach every other workload by default.

###### Application-Aware L7 Security

Kubernetes security needs more than “allow TCP/80.” Many services share the same ports, especially HTTP and HTTPS APIs, so port-based controls alone cannot express real application intent.

Cilium supports Layer 7 policy enforcement, allowing teams to define controls based on application-layer behavior. For example, a policy can allow one service to make only GET /public requests while denying other HTTP methods or paths. This helps reduce blast radius when a service is compromised because the attacker inherits only the precise network permissions the workload actually needs.

###### Encryption and Mutual Authentication

Encryption protects data in transit between workloads and nodes. Cilium supports transparent encryption using IPsec or WireGuard, allowing traffic between Cilium-managed endpoints, and in some configurations, host traffic to be encrypted without requiring application changes.
For stronger workload identity verification, Cilium also supports mutual authentication through its service mesh capabilities. This helps ensure that communication is not only encrypted but also tied to verified identities, which is especially important for zero-trust Kubernetes environments.

[Read More About Encryption](https://docs.cilium.io/en/latest/security/network/encryption/)

##### 5\. The Anatomy of a Rule

A NetworkPolicy is a Kubernetes object, and its rule structure (the "anatomy") consists of four primary parts: Selector (who is this for), direction (which way is traffic moving), the peer, and the action.

###### Subject Selectors and Identity

Cilium Network Policies shift from volatile IP addresses to Identity-Based Security. Policies use Kubernetes metadata and labels known as Subject Selectors to define the enforcement point for both source and destination workloads (e.g., _role: frontend_ talking to _role: backend_). This logical abstraction is pre-computed by the control plane into a single, cluster-wide Numeric Security Identity ($O(1)$ lookup complexity), allowing the eBPF kernel maps to execute high-speed packet filtering instantly.

###### Default Deny Posture

A core element of a Zero Trust architecture, the Default Deny posture aggressively blocks all communication unless it is explicitly permitted by a policy. Teams can safely transition to this model by using Hubble (Cilium's observability layer) to visualize existing traffic and test policies in Audit Mode before moving to full enforcement.

###### Ingress & Egress Semantics

Rules control the flow of traffic _into_ the workload (Ingress) and _out of_ the workload (Egress). For external communication, Cilium Egress Gateway can be used to secure egress access by routing traffic to specific external services through designated nodes, providing a stable, predictable source IP that satisfies traditional firewalls outside the cluster.

###### Layer 7 Inspection

This capability extends security beyond Layer 3/4 filtering to address the Layer 7 blindness in network policies. It allows for fine-grained, API-aware security (e.g., DNS-aware rules) by enabling policies to inspect the application content. For instance, a policy can allow a _GET /public_ request while explicitly blocking a sensitive _POST /admin/shutdown_ command, strictly enforcing the Principle of Least Privilege.

[Read More About Cilium Network Policies](https://isovalent.com/blog/post/intro-to-cilium-network-policies/)

##### 6\. Advanced Security Patterns

Cilium supports stronger patterns such as transparent encryption, mutual authentication, and host-layer segmentation to protect traffic and infrastructure without adding unnecessary operational complexity.

**Transparent Encryption (mTLS)**
Kubernetes lacks native pod-to-pod encryption, often requiring complex service mesh sidecars. Cilium solves this by providing **Transparent Encryption** for node-to-node traffic using technologies like IPsec or WireGuard, without requiring application changes. Recently, Cilium introduced **Native mTLS with ztunnel**, which unifies authentication and encryption directly within the Kubernetes data path, offering high-performance, identity-aware security for East-West traffic.

_Figure 5: Transparent mTLS encryption and identity management for pod-to-pod communication powered by Cilium and eBPF._

**Host-Layer Segmentation**
Security strategies must also extend to the Node itself to prevent attackers who compromise the host from bypassing pod-level controls. **Cilium Host Policies** apply the same identity-aware security logic used for applications to secure the host network namespace, segmenting control plane traffic and SSH access.

##### 7\. Day 2 Operations: Observability, Auditing, and Compliance

Day 2 operations represent the ongoing, post-deployment phase of the IT lifecycle, focused on managing, monitoring, and optimizing systems already in production. Day 2 Operations include the longest operational phase after software has been deployed and operationalized.

Day-2 operations can be succinctly characterized as 'business as usual' tasks, constituting the phase that sustains product functionality, enabling customers to enjoy it seamlessly whenever needed.

_Figure 6: The IT operational lifecycle illustrating the progression from initial Day 0 planning to continuous Day 2 operations and maintenance._

###### Hubble for Network Observability

As the observability layer built on eBPF, Hubble provides deep, real-time insights into networking, service, and application behavior, helping teams turn reactive firefighting into proactive optimization. It provides a Service Map and flow logs enriched with Kubernetes identity metadata (Pod names, Labels, Security ID), allowing operators to troubleshoot issues and understand the conversation in terms of services, not volatile IP addresses.

_Figure 7: Hubble Service Map providing real-time, identity-aware visualization of Kubernetes network flows._

###### Auditing and Compliance

For regulated industries (FinTech, Healthcare), meeting compliance frameworks like SOC2, PCI-DSS, and GDPR requires to be detailed auditing. Cilium facilitates this by enforcing strict segmentation and generating detailed, immutable network flow logs. Tetragon further extends this capability by providing runtime security and visibility into system calls, supporting real-time threat protection and streamlined compliance reporting.

###### Importance of Day 2 Operations

It's important to have Day 2 operations in mind when laying the groundwork for your software in Day 0 and Day 1, especially for cloud-native technologies. Having the right maintenance tools in place earlier on will help your organization avoid issues in the future.
Some common challenges during Day 2 operations include having trouble visualizing the performance of your software and difficulty integrating updates. Managing all the moving parts of your software is especially challenging with cloud-native systems such as Kubernetes, as they become increasingly more complex with scale.

##### 8\. Summary

Kubernetes network security works best as a layered model, rather than relying on a single feature or firewall rule. Because workloads are dynamic, short-lived, and constantly moving across nodes, teams need security controls that follow workload identity rather than static IP addresses. A strong approach combines identity-based network policies, microsegmentation, transparent encryption, application-aware controls, observability, and host-level protection. Together, these layers reduce unnecessary access, limit lateral movement, and facilitate the enforcement of the least privilege across the cluster.

Cilium strengthens this model by using eBPF to enforce security close to workloads in the Linux kernel, while integrating with Kubernetes identities such as labels, namespaces, and service accounts. It gives teams more precise control over which services can communicate, how traffic is secured, and how policy decisions are enforced across pods, nodes, and clusters. With Hubble, teams also gain the visibility needed to understand service dependencies, troubleshoot denied traffic, audit policy behavior, and validate compliance requirements. The result is a Kubernetes network that is not only more secure but also easier to observe, operate, and improve over time.

#### Understanding Kubernetes Microsegmentation

URL: https://cilium.io/blog/2026/05/24/understanding-kubernetes-microsegmentation
Date: 2026-05-24
Categories: Security, Technology

##### Understanding Kubernetes Microsegmentation

###### Introduction

Kubernetes Microsegmentation is the security practice of dividing a cluster into small, isolated segments to limit the blast radius of a potential breach. Kubernetes microsegmentation leverages workload identity, which encompasses metadata such as labels and namespaces, making it possible to define granular security policies that follow the application, regardless of where it is scheduled or what IP it is assigned.

_Figure 1. The Segmentation Spectrum. This diagram outlines three levels of security isolation, moving from broad to granular: Macro (compute regions), Micro (workloads and containers), and Nano (kernel-level processes via eBPF)._

###### 1\. Why traditional firewalls don’t work in cloud native environments

When the first network firewalls appeared in the late 1980s and early 1990s, they served as barriers at the edge, acting as controls between inside and outside networks. That perimeter model worked when most traffic was north–south (entering or leaving the data centre).

###### Why IP Addresses are the Wrong Primitive for Cloud Native Security

In traditional networking, an IP address was a reliable proxy for identity. If you knew the IP, you knew the server. In Kubernetes and cloud native architectures, this logic breaks down for three primary reasons:

###### i) Ephemeral Nature of Resources

In Kubernetes, Pods are designed to be cattle, not pets. They are created and destroyed constantly due to scaling, updates, or failures.

IP Reuse: A Pod might live for only five minutes. Once it dies, its IP address is released back into the pool and may be assigned to a completely different service moments later.

###### ii) Lack of Contextual Identity

An IP address is a network coordinate, not an identity. It tells you where a packet is coming from, but not who is sending it.

The "_What_" Matters; Security teams need to define rules like "_The Payment Service can talk to the Credit Card Database_." To enforce this via IPs, you would need to constantly update firewall rules every time a new Pod is scheduled. This creates a massive "policy lag" where security cannot keep up with the speed of development.

###### iii) Scalability and Performance

Traditional Linux networking relies on iptables to manage IP-based rule chains.
iptables checks rules sequentially. In a cluster with thousands of Pods, an IP-based rule list can grow so long that network latency increases significantly because the system has to "read" through thousands of lines for every single packet. Cilium uses eBPF to replace sequential iptables rule evaluations with high-performance BPF hash maps. When a packet arrives, eBPF instantly maps the source and destination IP addresses to their corresponding Numeric Security Identities using a cluster-wide cache, then executes a constant-time $O(1)$ policy check.

###### 2\. What is Kubernetes Microsegmentation?

Kubernetes Microsegmentation is the security practice of isolating individual workloads (Pods) and strictly controlling the traffic between them. Unlike traditional segmentation, which creates broad perimeters around entire data centers or VLANs, microsegmentation applies security policies at the individual workload level.

###### i) The Zero-Trust Principle: "Never Trust, Always Verify."

In a microsegmented environment, the network operates under the assumption that the inside is just as dangerous as the outside.

- Default Deny: By default, all communication between Pods is blocked.
- Explicit Allowing: A connection is only permitted if there is a specific security policy that allows it.
- Identity-Based: Instead of trusting an IP address, the system verifies the identity of the workload (using Kubernetes labels, such as app: settings or role: database) before allowing a packet to pass.

###### ii). Eliminating the "Flat Network."

By default, Kubernetes has a flat network structure, meaning any Pod can talk to any other Pod across the entire cluster. While this makes deployment easy, it creates a massive security risk. Microsegmentation breaks this flat structure into thousands of tiny, isolated segments, effectively putting a miniature firewall in front of every single container.

###### 3\. Identity-Based Security (The Cilium Way)

In a traditional network, a firewall rule looks like this: "_Allow traffic from 10.0.1.5 to 10.0.2.10_." As we’ve established, this fails in Kubernetes because those IPs are constantly changing. Cilium solves this by replacing volatile IPs with Security Identities.
Security Identity is a logical abstraction that represents a group of workloads (Pods) sharing the same security properties.

###### i) From IPs to Logical Labels

Instead of managing a list of thousands of IP addresses, Cilium uses the metadata already present in Kubernetes.
Your security policy becomes human-readable and intent-based: "_Allow role: frontend to talk to role: backend._"

Cilium doesn't care what the IP address is. If a Pod has the label role: frontend, it is automatically granted the permissions assigned to that identity.

###### ii) How Numeric Identities Work

To keep the network fast, Cilium’s control plane translates complex strings of labels into a simple Numeric Identity (e.g., role: frontend might become ID 501).

**Assignment of Numeric Identity:**
The local Cilium agent collects the pod's labels upon creation. These security-relevant labels are sorted and hashed, and the Cilium control plane assigns a globally synchronized Numeric Identity to that unique label footprint.

**Grouping:**
If you have 100 replicas of a frontend pod, they all share the same numeric identity; that is to say, all pods sharing the same security-relevant labels receive the same numeric identity.

**The Identity Table:**
Cilium maintains a map across the cluster that links every IP address to its current Numeric Identity. When a packet is sent, Cilium stamps the packet with this identity.

_Figure 2. The Cilium Identity Table. This diagram shows how Cilium uses a central Key-Value store to map workload labels to unique numeric identities, sharing this cluster-wide mapping across individual nodes for identity-aware networking._

###### iii) O(1) Efficiency in the eBPF Datapath

The eBPF datapath achieves high-speed enforcement by performing lookups in kernel-level eBPF maps, which function like high-performance hash tables. In legacy systems, as you add more pods, the firewall has to check more rules (this is known as O(n) complexity). With Cilium and eBPF, security enforcement happens in constant time, or O(1).

**Identity Lookup:**
When a packet enters or leaves a pod, the eBPF program retrieves the source identity from a local eBPF Endpoint Map. It simply looks at the numeric ID stamped on the packet and checks it against a HashMap.

**Policy Verification:**
Instead of scanning complex lists of firewall rules, the datapath consults a Policy Map. A combination of source identity and destination identity is used as a key.

**Packet Marking:**
For cross-node traffic, the source identity is often embedded directly into the packet header (using VXLAN or Geneve metadata) so that the receiving node doesn’t have to resolve it.

_Figure 3. Identity-Based Policy Interface in Hubble UI. This dashboard visualizes how network rules are constructed and monitored using logical workload identities._

###### 4\. Kubernetes vs Cilium Network Policies

Kubernetes includes a NetworkPolicy resource, which is often insufficient for modern security requirements. The primary difference lies in visibility and depth.

Kubernetes Network Policies are application-centric constructs for defining network segmentation rules at various levels in the cluster. Kubernetes Network Policies serve as firewalls, enabling security, namespace isolation, and traffic restriction (ingress/egress) through the use of labels and selectors.

We can define network security policies at layers 3 and 4 of the OSI model, i.e., IP, TCP, UDP, and SCTP. Although the network policy specification is part of Kubernetes, the implementation is handled by a Container Network Interface (CNI); therefore, using a CNI that supports network policies is imperative.

[Read More about Cilium Network Policies](https://isovalent.com/blog/post/intro-to-cilium-network-policies/)

###### The Layer 3/4 Limitation

Native Kubernetes Network Policies operate at Layer 3 (IP) and Layer 4 (Port/Protocol). This means you can write a rule that says:
"_Allow the Frontend Pod to talk to the Carts Service on Port 8080/TCP_."

The problem is that, once that port is open, the Frontend can do anything on that port. If an attacker compromises the Frontend, they can send DELETE /orders or POST /admin/shutdown just as easily as a legitimate GET /items.
At Layer 4, the network is blind to the actual intent of the traffic.

###### Layer 7 API-Aware Segmentation

Cilium extends the Kubernetes network policy model by adding Layer 7 (Application Layer) visibility. This allows you to define policies based on the specific API calls being made.
Instead of just opening a port, you can enforce the Principle of Least Privilege at the API level.

Example of policy granularity
HTTP: Allow GET /public/ but block POST /admin/.
gRPC: Allow the GetProduct method but block UpdateProduct.

###### Supported Protocols

Kubernetes Native Network policy only supports IP, TCP, UDP, and SCTP protocols. Cilium Network policies extend support to protocols like gRPC and HTTP.

| Feature | Kubernetes Network Policy | Cilium Network Policy |
| ----------------- |:-------------------------------: |:--------------------------------------------------------: |
| Enforcement Layer | L3 / L4 (IP & Port) | L3, L4, and L7 (API) |
| Protocol Support | TCP, UDP, SCTP | HTTP, gRPC, DNS |
| Granularity | All traffic on a port is allowed. | Only specific API methods/paths are allowed. |
| Observability | Only "Allow/Deny" logs. | Deep visibility into API paths, headers, and status codes. |
| Scope | Namespace-only. | Namespace and Cluster-wide. |

_Table 1: Kubernetes Native Network Policies vs Cilium Network Policies._

_Figure 4. Kubernetes vs. Cilium Network Policies. This Venn diagram compares standard Kubernetes network policies with Cilium's extended capabilities._

###### 5\. Implementing a "Default Deny" Strategy

The default deny strategy is an aggressive network policy strategy that involves configuring security controls (firewalls, applications) to block all traffic or actions by default, allowing only specifically authorized activities. Default Deny is a zero-trust approach that requires explicit permission for communication.

###### Phase 1: See Everything with Hubble

Before you can restrict traffic, you must first identify the existing traffic. Hubble is Cilium's observability layer, providing a real-time service map of your cluster.

**Service Map:**
Visualizes exactly which services are talking to each other.

**Flow Logs:**
Audit every connection attempt, including metadata like which labels were used and which protocol was detected.

**Identify Shadow Traffic:**
Discover forgotten connections like a legacy monitoring agent or an external API call that would be broken by a new security policy.

_Figure 5. Network Policy Visualization and Flow Auditing in Hubble UI. This dashboard provides a unified interface for authoring and monitoring security rules._

###### Phase 2: Refine Your Labels

Identity-based security is only as good as your metadata. Since Cilium creates Security Identities based on labels, you must ensure your labeling scheme is consistent and granular.
Instead of broad labels like env: prod, use specific pairs:

- app: billing-api
- tier: backend
- version: v2.1

Good labeling doesn't just help with security; it makes your entire observability stack and CI/CD pipeline easily observable and organized, but also makes it possible to communicate well about policy enforcement decisions and effects.

###### Phase 3: Audit Mode, The Test Drive

Cilium allows you to test policies without actually dropping packets. This is known as Audit Mode. This is useful for prototyping and testing the side effects of policies in the same environment we are about to enforce.

When you deploy a policy in Audit Mode, Cilium will log what would have been dropped if the policy were active. This allows you to:

1. Apply your "Default Deny" policy in audit mode.
2. Check Hubble logs for "Policy Deny" events.
3. Add "Allow" rules for legitimate traffic that you missed.
4. Once the "Deny" logs hit zero, switch the policy to Enforcement Mode.

###### Phase 4: Full Enforcement

Once you are confident in your rules, you move to true Microsegmentation. At this stage, any packet that doesn't match an "Allow" rule is silently dropped by the eBPF datapath. This ensures that even if a new, malicious process starts in a Pod, it has nowhere to go.

1. [Read Kubernetes Network Policies Comprehensive Guide](https://isovalent.com/blog/post/kubernetes-network-policies-ebook/)
2. [Read about Cilium Network Policies & Enforcement Modes](https://docs.cilium.io/en/stable/security/policy/intro/)

###### 6\. Advanced Microsegmentation Architectures

Depending on the size of the cluster and the number of workloads running, it will grow to a point where Pod-to-Pod rules aren’t enough. You will need to manage traffic that crosses boundaries, from internal namespaces, external firewalls, or the underlying host operating system.

###### Cross-Namespace Communication

By default, Kubernetes NetworkPolicy is namespace-scoped. Cilium uses the **_CiliumClusterwideNetworkPolicy_** resource, which allows you to define a single security rule that applies across the entire cluster, regardless of namespace. It’s perfect for enforcing global rules, like ensuring no pod in any namespace can reach a specific sensitive database unless explicitly permitted.

###### Egress Gateways: Bridging the New and the Old

A common pain point in Kubernetes clusters is connecting to external services (like a legacy Oracle DB) that sit behind a traditional hardware firewall. These firewalls expect a static IP, but Kubernetes pods have dynamic IPs.

**_Cilium Egress Gateway_** allows you to designate specific nodes as gateways. Traffic from your pods to a specific external IP is routed through these gateways, which mask the pod IP with a stable, predictable node IP. This lets you maintain microsegmentation inside the cluster while satisfying the static requirements of the legacy world outside.

###### Host-Layer Segmentation

Most security strategies focus on reducing the attack surface on the Pods, but what about the Node itself? If an attacker gets onto the host (the virtual machine), they can often bypass Pod-level controls.
**Cilium Host Policies** provides a mechanism to secure the host network namespace of Kubernetes nodes. It allows you to segment the control plane traffic, SSH access, and node-to-node communication using the same identity-aware logic you use for your applications.

###### 7\. Compliance and Auditing

In regulated industries (FinTech, Healthcare), being secure isn't enough; you have to prove it. Microsegmentation is the foundation for meeting frameworks like PCI-DSS, SOC2, and GDPR.
Cilium helps satisfy audit requirements by enforcing strict segmentation and providing detailed, immutable logs of all network communications. Cilium is a framework-agnostic compliance solution for Kubernetes.

Tetragon further extends these capabilities by offering visibility into runtime behaviors and system calls, facilitating the enforcement of runtime security policies, and supporting potential work around anomaly detection or streamlined compliance reporting.

[Read the complete Whitepaper on Implementing Cilium for Compliance Use Cases](https://isovalent.com/blog/post/implementing-cilium-for-compliance-use-cases-controlplane-isovalent-whitepaper/)

###### 8\. Summary

As architectures continue to evolve toward AI-driven workloads and multi-cloud meshes, microsegmentation isn't just a nice-to-have; it is the fundamental building block of a resilient cloud native strategy.

By implementing microsegmentation, you achieve three critical outcomes:

1. Zero Trust Architecture: You treat the internal network with the same scrutiny as the public internet, ensuring every connection is authorized.
2. Attack Surface Reduction: You prevent lateral movement, ensuring a single compromised container does not lead to a cluster-wide breach.
3. Future-Proof Compliance: As highlighted in the Blueprint for Kubernetes Compliance, microsegmentation isn't just a technical configuration; it is a strategic foundation. It allows teams to map technical eBPF controls directly to global frameworks like NIST SP 800–53 and 800–190.

For leadership teams, microsegmentation provides the observability required to pass audits without the manual overhead of legacy systems.

Microsegmentation ensures that your environment is not only secure but also compliant by design, allowing your organization to navigate the ever-changing landscape of cloud native regulations with confidence.

#### Securing CI/CD for an open source project: lessons from Cilium

URL: https://cilium.io/blog/2026/05/06/securing-cicd-open-source-lessons-from-cilium
Date: 2026-05-06
Categories: Technology

<style>{`
.supply-chain-post pre { font-size: 0.82em; line-height: 1.45; }
`}</style>

The last twelve months have been rough on the open source supply chain. [Axios was compromised on npm](https://www.stepsecurity.io/blog/axios-compromised-on-npm-malicious-versions-drop-remote-access-trojan) and shipped a remote access trojan inside otherwise normal-looking releases. [LiteLLM's PyPI package was hijacked](https://futuresearch.ai/blog/litellm-pypi-supply-chain-attack/) to exfiltrate environment variables. [Typosquatted forks of Trivy](https://rosesecurity.dev/2026/03/20/typosquatting-trivy.html) were published to catch people who fat-finger `go install`. And the canonical example, the [2020 SolarWinds breach](https://en.wikipedia.org/wiki/2020_United_States_federal_government_data_breach), is still the cautionary tale we keep coming back to: attackers got into the build system and pushed malware through normal Orion updates to roughly 18,000 organizations, including U.S. federal agencies, NATO, and Microsoft. The malware sat dormant for months. The breach went undetected for the better part of a year.

Cilium runs in the kernel-level networking path of millions of Kubernetes pods. If our supply chain were compromised, the blast radius would not be small. Hardening the project against that scenario is something we work on continuously, and we wanted to write down what we actually do, in detail. Most of what follows isn't Cilium-specific: any open source project running CI/CD on GitHub Actions can apply these patterns. We've also called out where we still fall short, in case any of it makes a useful starting point for someone else.

##### TL;DR

If you don't have time to read the whole thing, here's what Cilium does to harden its supply chain today, organized by which layer of the pipeline each control lives at:

| Layer | Control | What it does |
| ------------------------------------------------ | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Who triggers builds** | [Trigger control via Ariane](#workflow-trigger-restrictions-with-ariane) | Only verified org members can fire CI workflows from PR comments, against an explicit allow-list of workflows. |
| **What code CI executes** | [Two-phase checkouts for `pull_request_target`](#separating-trusted-and-untrusted-code-in-ci) | Trusted code (composite actions, scripts, signing logic) is loaded from the base branch; the PR head is only used as Docker build context, never executed as a script. |
| **Who reviews CI changes** | [CODEOWNERS gates](#codeowners-as-a-review-gate) | Anything under `.github/` requires review from the security-focused CI team, and `auto-approve.yaml` requires a maintainer. |
| **What dependencies CI pulls in** | [SHA-pinned actions and images](#pinning-github-actions-by-sha-digest) | Every `uses:` references a 40-character commit SHA; container images are pinned by `@sha256:` digest. [Renovate](#automated-updates-with-a-trust-boundary) keeps the pins fresh and waits 5 days before picking up new releases. |
| **What Go modules ship in the binary** | [Vendored Go dependencies](#go-module-vendoring) | Everything is checked into `vendor/` and reviewed by the `@cilium/vendor` team, so a typosquatted or hijacked module shows up as a diff at review time. |
| **What workflows are even allowed to look like** | [Static analysis on workflows](#catching-mistakes-with-static-analysis) | CodeQL enforces explicit `permissions:` on every workflow, actionlint catches unsafe patterns, and both flag GitHub Actions expression injection in `run:` blocks. |
| **What credentials are reachable** | [CI vs. production credential isolation](#ci-vs-production-credential-isolation) | CI credentials can only push to `*-ci` development tags; production registry credentials sit behind a protected `release` environment that requires maintainer approval. |
| **What consumers can verify** | [Signed releases](#signing-and-attesting-what-we-ship) | Every release image and Helm chart is signed with [Sigstore Cosign](https://github.com/sigstore/cosign) using keyless OIDC, with SBOM attestations attached. |
| **Where we still fall short** | [Gaps we're still closing](#what-were-still-working-on) | No SLSA provenance yet, no PR-time dependency review, no `govulncheck` in CI, and a handful of internal `@main` references that need to move to a dedicated composite-actions repo. |

The rest of the post walks through each row in more depth, including the design decisions behind them and the things we deliberately chose _not_ to do (like forking every third-party action into our own org).

##### Controlling who runs what

The first question in any CI supply chain story is: who can trigger a build, and what code does it execute? Plenty of CI compromises start right here, by tricking the system into running attacker-controlled code with elevated privileges.

###### Workflow trigger restrictions with Ariane

[Ariane](https://github.com/cilium/ariane) is a GitHub bot we wrote in-house to dispatch CI workflows from PR comments. When a maintainer types `/test` or `/ci-eks` on a pull request, Ariane checks that the commenter belongs to the `organization-members` team, figures out which workflows to fire (including dependencies, like tests that need a fresh image build first), and dispatches them via `workflow_dispatch`.

The interesting bit is the allow-list. Only verified org members can trigger workflows, and the set of workflows that can be triggered is enumerated by hand in the config:

[`.github/ariane-config.yaml`](https://github.com/cilium/cilium/blob/main/.github/ariane-config.yaml)

```yaml
allowed-teams:
  - organization-members

triggers:
  /test\s*:
    workflows:
      - conformance-aws-cni.yaml
      - conformance-clustermesh.yaml
      - conformance-eks.yaml
    # ...and so on
    depends-on:
      - /build-images-dependency
  /ci-aks:
    workflows:
      - conformance-aks.yaml
    depends-on:
      - /build-images-dependency
```

A random external commenter typing `/test` in a PR is ignored. They can't kick off our expensive cloud-provider conformance suites or burn through our CI minutes.

###### Separating trusted and untrusted code in CI

When somebody opens a PR we need to build their code, but we obviously can't trust it. This is the classic [`pull_request_target` problem](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/). We avoid `pull_request_target` where we can, but a handful of workflows still need it, and we wrap those in mitigating controls.

The image build workflow is the canonical example. It splits the checkout in two:

[`.github/workflows/build-images-ci.yaml`](https://github.com/cilium/cilium/blob/main/.github/workflows/build-images-ci.yaml)

```yaml
- name: Checkout base or default branch (trusted)
  uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
  with:
    ref: ${{ github.base_ref || github.event.repository.default_branch }}
    persist-credentials: false

# ...trusted setup steps run here, including loading composite actions...

# Warning: since this is a privileged workflow, subsequent workflow job
# steps must take care not to execute untrusted code.
- name: Checkout pull request branch (NOT TRUSTED)
  uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
  with:
    persist-credentials: false
    ref: ${{ steps.tag.outputs.sha }}
```

The first checkout grabs the _base branch_ (code that's already been reviewed and merged) so we can load our composite actions, scripts, and the Cosign signing logic from a known-good source. Only after that does the workflow check out the PR head, and that checkout is used purely as build context for `docker build`. Nothing from the PR branch is ever executed as a script.

We get security reports about this pattern fairly regularly. Automated scanners and well-meaning researchers see "`pull_request_target` plus a second checkout" and flag it as a vulnerability. In the general case they're right too. In ours, the workflow is intentionally designed so the pattern is safe:

- **No `run:` steps execute scripts from the untrusted checkout.** Every shell block after the second checkout is written inline in the workflow YAML (disk usage checks, file copies, digest output). Nothing is sourced from the PR branch.
- **No composite actions are loaded from the untrusted checkout either.** All composite actions (`set-runtime-image`, `cosign`, `set-env-variables`) come from the trusted base-branch checkout or from the saved `../cilium-base-branch/` directory. We're also working on moving these composite actions into a dedicated repository so we don't have to check out source to run them at all.
- **Docker BuildKit does execute the untrusted Dockerfile**, and that's the whole point of building a CI image from a PR. BuildKit runs in isolation: no GitHub Actions environment variables, no repo secrets, no access to the runner's Docker credential store. The build args we pass contain no secrets, just the runtime image reference and the operator variant name.
- **Untrusted data flows into exactly one trusted action.** The `runtime-image*.txt` file from the PR is fed into the trusted `set-runtime-image` action, which checks the image reference starts with `quay.io/cilium/` and strips newlines so an attacker can't smuggle in a `GITHUB_ENV` injection. There's no way to repoint the build to anything outside the Cilium namespace.
- **Only CI credentials are in scope.** The Docker login uses `QUAY_USERNAME_CI` / `QUAY_PASSWORD_CI`, which can only push to the `-ci` development registry. Production credentials aren't on the runner at all.

The worst-case outcome of a compromised PR build is a malicious CI image landing in the development registry, which is the same blast radius any CI system that builds contributor code carries. We do appreciate every report and read each one carefully, but this pattern is intentional.

###### CODEOWNERS as a review gate

We lean on [CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) pretty heavily so that changes always land in front of the people with the most context. For CI configuration that means anything under `.github/` is owned by `@cilium/github-sec` (our security-focused CI team) plus `@cilium/ci-structure`, and the `auto-approve.yaml` workflow is owned by `@cilium/cilium-maintainers`:

[`CODEOWNERS`](https://github.com/cilium/cilium/blob/main/CODEOWNERS)

```
/.github/                          @cilium/github-sec @cilium/ci-structure
/.github/ariane-config.yaml        @cilium/github-sec @cilium/ci-structure
/.github/renovate.json5            @cilium/github-sec @cilium/ci-structure
/.github/workflows/                @cilium/github-sec @cilium/ci-structure
/.github/workflows/auto-approve.yaml  @cilium/cilium-maintainers
```

Nobody can change the CI pipeline without an explicit review from the team responsible for keeping it safe.

##### Locking down dependencies

Once you control who triggers builds, the next question is what code those builds pull in. A pinned workflow that fetches a compromised dependency is still a compromised workflow.

###### Pinning GitHub Actions by SHA digest

The single highest-leverage thing any project can do here is stop trusting mutable tags.

Every `uses:` directive in our workflow files references actions by full 40-character commit SHA, with the human-readable version stuck on the end as a comment:

```yaml
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
```

If somebody compromises the `v6` tag on `actions/checkout` and force-pushes malicious code, our workflows won't pull it. They're pinned to a specific commit. Same story for every third-party action we use: `docker/build-push-action`, `sigstore/cosign-installer`, `golangci/golangci-lint-action`, dozens more. We pin container images used directly in workflow steps the same way, by `@sha256:` digest, so even the tools we run inside CI are content-addressed.

Pinning has one annoying blind spot, which is transitive dependencies. When we pin `actions/checkout@de0fac2e...` we know exactly which code runs for that action. But if `actions/checkout` itself references another action by tag (`uses: some-org/some-helper@v1`), that resolution happens at runtime and is invisible to us. An attacker who pops the nested dependency can still reach our pipeline.

A fix is on the way: workflow-level dependency locking was announced in GitHub's [2026 Actions security roadmap](https://github.blog/news-insights/product-news/whats-coming-to-our-github-actions-2026-security-roadmap/). It would add a `dependencies:` section to workflow YAML that locks all direct _and transitive_ action dependencies by commit SHA, similar to what `go.mod` + `go.sum` do for Go. We'll adopt it as soon as it ships.

###### Automated updates with a trust boundary

Maintaining SHA pins by hand would be miserable, so we don't. Our [Renovate configuration](https://github.com/cilium/cilium/blob/main/.github/renovate.json5) extends the `helpers:pinGitHubActionDigests` preset and sets `pinDigests: true` globally. When a new action version drops, Renovate opens a PR bumping the SHA. We stay current without ever falling back to a mutable ref.

Renovate runs as a [self-hosted bot](https://github.com/cilium/cilium/blob/main/.github/workflows/renovate.yaml) on an hourly schedule, using a dedicated GitHub App with fine-grained permissions instead of a personal access token. `vulnerabilityAlerts` is on, so known CVEs in the dependency tree turn into PRs straight away.

We [recently added](https://github.com/cilium/cilium/pull/45491) a Renovate cooldown so we don't pick up brand-new releases the moment they appear. Given the current pace of supply chain attacks, those few days are usually the window in which a compromised package gets noticed and yanked:

[`.github/renovate.json5`](https://github.com/cilium/cilium/blob/main/.github/renovate.json5)

```json5
{
  // Dependency cooldown: skip versions published less than 5 days ago
  "matchUpdateTypes": ["major", "minor", "patch"],
  "minimumReleaseAge": "5 days"
},
{
  "matchPackageNames": [
    "actions/{/,}**",                  // GitHub's official actions
    "docker/{/,}**",                   // Official Docker actions
    "cilium/{/,}**",                   // Our own ecosystem
    "k8s.io/{/,}**",                   // Kubernetes official
    "sigs.k8s.io/{/,}**",              // Kubernetes SIGs
    "golang.org/x/{/,}**",             // Go experimental
    "github.com/golang/{/,}**",        // Go official org
    "github.com/prometheus/{/,}**",
    "github.com/hashicorp/{/,}**",
    "go.etcd.io/etcd/{/,}**",
    // ...trimmed
  ],
  "automerge": true,
  "automergeType": "pr",
  "groupName": "auto-merge-trusted-deps",
  "reviewers": ["ciliumbot"]
}
```

Updates from this allow-list auto-merge after CI passes. Everything else needs a human review.

The [auto-approve workflow](https://github.com/cilium/cilium/blob/main/.github/workflows/auto-approve.yaml) adds another belt-and-suspenders check: it verifies that the PR was created by `cilium-renovate[bot]` _and_ that the review request was actually triggered by the bot itself, not by a human pretending to be it:

```yaml
if: ${{
  github.event.pull_request.user.login == 'cilium-renovate[bot]' &&
  (github.triggering_actor == 'cilium-renovate[bot]' ||
  github.triggering_actor == 'auto-committer[bot]')
  }}
```

If those conditions don't hold, no auto-approval happens.

###### Go module vendoring

All Go dependencies are vendored and committed to the repo. CI [verifies there's no drift](https://github.com/cilium/cilium/blob/main/.github/workflows/lint-go.yaml) between `go.mod`, `go.sum`, and `vendor/`. Builds are reproducible and don't talk to external module proxies at build time, so a tampered module on a proxy never reaches us. We also run license checks (`go run./tools/licensecheck`) to keep dependencies with unwanted licenses out of the tree.

###### Would forking actions into our own org be even safer?

In theory, yes. If we forked every third-party action into `cilium/` and pinned to our own fork's SHA, an upstream compromise wouldn't reach us at all. Some high-security projects do exactly this.

We've decided against it, mostly because the operational cost is real and the security win is smaller than it first looks:

- **Maintenance burden.** We use dozens of third-party actions. Keeping forks in sync with upstream security patches becomes a part-time job, and a stale fork with unpatched vulnerabilities is itself a security problem.
- **Missed improvements.** Upstream actions regularly fix bugs and ship security features. Forks add friction to picking those up.
- **Renovate complexity.** Our update pipeline would have to track upstream releases, open PRs against each fork, and then update the consuming workflows. The chain doubles in length.

SHA pinning gives us the immutability guarantee that actually matters: a specific commit is a specific commit, regardless of which org hosts it. Combined with Renovate proposing updates as new versions come out, we get the security benefit without the operational tax. If a major action provider got repeatedly compromised, forking the high-risk ones is a reasonable escalation, but we haven't been pushed to that point.

###### The same tradeoff applies to Go dependencies

The "should we fork it?" question applies just as much to our Go dependency tree. Cilium pulls in hundreds of Go modules: Kubernetes client libraries, gRPC, etcd, Prometheus, the works. Forking and maintaining all of them isn't realistic.

Go is in a slightly better starting position than npm or PyPI because import paths explicitly include the source (`github.com/stretchr/testify`), which kills off the [Dependency Confusion](https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610) attack class entirely. Typosquatting is still a real threat, though. [Michael Henriksen's research](https://michenriksen.com/archive/blog/finding-evil-go-packages/) found typosquatted Go packages in the wild, including a fork of `urfave/cli` registered as `utfave` (one transposed letter) that phoned home with hostname, OS, and architecture. Swapping that callback for a reverse shell would have been a one-line change.

And typosquatting isn't the worst case. SolarWinds showed that a legitimate, widely-trusted vendor can have its build pipeline compromised and then push malware through normal updates. Same can happen to any Go module: an attacker who gets into a maintainer's account publishes a malicious release, the proxy caches it, and anyone running `go get` pulls it in. That's why we vendor: it moves the trust decision from build time, where it's invisible, to review time, where a human can see the diff.

Vendoring is the main defense here. A typosquatted import path shows up as a diff in `vendor/` during code review instead of silently resolving from a module proxy. It doesn't catch the typo at the moment it's introduced (it relies on a reviewer noticing the unfamiliar path in the PR), but combined with CODEOWNERS gating it has held up well so far.

We're also deliberate about which dependencies we take on. The Renovate config has an explicit list of disabled dependencies that we manage by hand, either because they need coordinated updates (like `sigs.k8s.io/gateway-api` alongside conformance tests), because we maintain a fork with project-specific patches (like `github.com/cilium/dns`), or because the dependency is one we develop ourselves and want to bump deliberately (like `github.com/cilium/ebpf`, which isn't a fork but a standalone Go library maintained under the Cilium org). Changes to `vendor/` are reviewed by the dedicated [`@cilium/vendor`](https://github.com/cilium/cilium/blob/main/CODEOWNERS) team via the same CODEOWNERS mechanism above.

There's a Go proverb worth quoting here: ["A little copying is better than a little dependency."](https://go-proverbs.github.io/) We take that one seriously beyond style. We [periodically audit our third-party libraries](https://github.com/cilium/cilium/pull/45078) and actively shrink the tree. If a dependency exists only to provide a small utility function, we replace it with a few lines copied inline. Every dependency you remove is one that can never be compromised, the vendor tree gets smaller, and reviewing future dependency changes gets easier. The benefits compound.

##### Catching mistakes with static analysis

Even with the right policies in place, mistakes happen. A well-meaning contributor can add a workflow without `permissions:`, or use `ubuntu-latest` instead of a pinned runner. We use static analysis to catch this stuff before review.

Where workflows need write access (release signing, OIDC for Cosign), they declare only the specific scope they need, like `id-token: write` or `contents: write`. Where they don't, they declare `permissions: read-all` or `permissions: {}` to opt out of the broader defaults. We don't rely on memory for this, though. [CodeQL runs on every push and PR](https://github.com/cilium/cilium/blob/main/.github/workflows/codeql.yaml) with the `actions/missing-workflow-permissions` rule turned on, and the workflow fails any modified workflow file that doesn't set permissions explicitly.

On top of that, [actionlint](https://github.com/cilium/cilium/blob/main/.github/workflows/lint-workflows.yaml) statically checks every workflow file for syntax errors, unsafe patterns, and misconfigurations. The same lint pipeline also enforces project conventions: every job and step has a `name`, no job uses the floating `ubuntu-latest` runner tag (we pin to `ubuntu-24.04`), and there's no trailing whitespace in workflow files.

One vulnerability class is worth singling out: **GitHub Actions expression injection**. The `${{ }}` syntax in workflow YAML is a text substitution that happens before bash sees the line at all. If an attacker controls the value being substituted (a PR title, a branch name), they can inject arbitrary shell commands via `;`, `$(...)`, or backticks. Bash has no idea where the value came from. The fix is to assign the value to an environment variable first and reference it as `"$MY_VAR"` in the `run:` block, so bash treats it as a single variable regardless of contents. The GitHub security team reported this to us a while back, and we fixed every instance. It's a subtle bug that's easy to introduce and hard to spot in review, which is exactly why static analysis matters: both [actionlint](https://github.com/cilium/cilium/blob/main/.github/workflows/lint-workflows.yaml) and [CodeQL](https://github.com/cilium/cilium/blob/main/.github/workflows/codeql.yaml) flag `${{ }}` usage in `run:` blocks where untrusted input flows in.

##### Protecting credentials

We assume any individual layer can fail. If a CI workflow ever does get compromised, the question becomes: what can the attacker actually reach? The answer should be: nothing that matters.

###### Strong defaults

By default our `GITHUB_TOKEN`s are scoped to [minimal read permissions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#setting-the-permissions-of-the-github_token-for-your-repository) on `contents` and `packages`. Workflows that need anything more have to opt in explicitly, so a workflow that forgets to declare permissions doesn't end up with broad org-wide write access.

###### CI vs. production credential isolation

We keep two distinct sets of registry credentials behind separate GitHub [protected environments](https://docs.github.com/en/actions/deployment/targeting-different-environments/managing-environments-for-deployment):

- **CI credentials** can push to our development image registry (`quay.io/cilium/*-ci`) and are available to CI builds. Even if a CI workflow is compromised somehow, these credentials cannot push to production image tags.
- **Production credentials** sit behind the [`release` environment](https://docs.github.com/en/actions/deployment/targeting-different-environments/managing-environments-for-deployment), which requires an explicit maintainer approval before a workflow run can touch them. No fork, no feature branch, and no CI build can reach those secrets. Only tag-triggered release builds that a maintainer has approved can.

Worst-case, in a CI compromise, the attacker can publish a malicious `-ci` image. They cannot publish to `quay.io/cilium/cilium:v1.x.x` or `docker.io/cilium/cilium:v1.x.x`. The credentials simply aren't on the runner.

Every `actions/checkout` call also sets `persist-credentials: false`, so the `GITHUB_TOKEN` never ends up in the runner's git config where a later step could grab it.

##### Signing and attesting what we ship

The previous sections are about preventing bad things from getting into the pipeline. This one is about letting consumers verify what comes out of it.

Every container image we release (`cilium`, `operator-*`, `hubble-relay`, `clustermesh-apiserver`) is signed with [Sigstore Cosign](https://github.com/sigstore/cosign) using keyless OIDC. There are no long-lived signing keys for anyone to steal.

A reusable composite action handles the signing pipeline:

[`.github/actions/cosign/action.yaml`](https://github.com/cilium/cilium/blob/main/.github/actions/cosign/action.yaml)

```yaml
- name: Install Cosign
  uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003 # v4.1.1

- name: Generate SBOM
  uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
  with:
    artifact-name: sbom_${{ inputs.sbom_name }}.spdx.json
    output-file: ./sbom_${{ inputs.sbom_name }}.spdx.json
    image: ${{ inputs.image_tag }}

- name: Sign Container Image
  shell: bash
  run: cosign sign -y "${{ inputs.image }}"

- name: Attach SBOM Attestation
  shell: bash
  run: |
    cosign attest -y \
      --predicate "./sbom_${{ inputs.sbom_name }}.spdx.json" \
      --type spdxjson \
      "${{ inputs.image }}"
```

This runs for every release image build and for our Helm chart OCI artifacts. Verification instructions are in the [Cilium docs](https://docs.cilium.io/en/stable/configuration/verify-image-signatures/#verify-signed-container-images).

Release builds also run inside [protected environments](https://docs.github.com/en/actions/deployment/targeting-different-environments/managing-environments-for-deployment) (`release`, `release-tool`, `release-helm`) so production registry credentials are gated behind environment protection rules. You can't trigger a release build from a fork or a feature branch.

##### The Cilium security team

If you've ever reported a security issue to the project (via [GitHub security advisories](https://github.com/cilium/cilium/security/advisories) or [security@cilium.org](mailto:security@cilium.org)), you've already interacted with [Cilium's Security Team](https://github.com/cilium/community/blob/main/roles/Security-Team.md). Beyond triaging vulnerability reports, the team also runs the operational side of supply chain security:

- Auditing and rotating credentials and permissions across the GitHub organization.
- When necessary, carrying out incident investigation and audits.
- Monitoring for patterns in our security issues and industry developments in order to propose mitigations and controls in areas where our security posture is weak.

##### Additional layers

A few smaller things worth mentioning:

- **Tag immutability.** Once a GitHub release is published, the tags and assets attached to it can't be modified. The setting lives in the repository's _Settings → Releases_ page.
- **DCO sign-off enforcement.** Every commit must carry a `Signed-off-by` line. Our [maintainers-little-helper](https://github.com/cilium/cilium/blob/main/.github/maintainers-little-helper.yaml) config blocks merges with a `dont-merge/needs-sign-off` label until a sign-off is present.
- **Third-party security audits.** We've been audited by [ADA Logics](https://adalogics.com), and we maintain a published [threat model](https://docs.cilium.io/en/latest/security/threat-model/).

##### What we're still working on

We audited our `.github/` directory against current best practices (OpenSSF Scorecard, SLSA, StepSecurity recommendations) and turned up a number of real gaps. The bigger ones:

- **No SLSA provenance.** Every `docker/build-push-action` call sets `provenance: false`. We sign images with Cosign, but we don't generate SLSA build provenance attestations. Consumers can verify _who_ signed an image, but not _how_ it was built. Adopting `slsa-framework/slsa-github-generator` (or at minimum enabling BuildKit-native provenance) is on the list.
- **No dependency review at PR time.** We rely on Renovate's `vulnerabilityAlerts` to flag known-vulnerable dependencies, but that's reactive. Wiring in [actions/dependency-review-action](https://github.com/actions/dependency-review-action) would catch malicious or vulnerable new dependencies _before_ they merge.
- **No `govulncheck` in CI.** We fuzz and we lint, but we don't yet run Go's official vulnerability scanner, which checks whether our code actually calls vulnerable functions rather than just whether a vulnerable package shows up in `go.sum`.
- **68 internal `@main` references.** A bunch of conformance and scale-test workflows reference `cilium/cilium/.github/actions/set-commit-status@main`, which is a mutable branch ref. It's lower risk than a third-party tag, but inconsistent with our SHA-pinning policy. The plan is to move all of our composite actions out of cilium/cilium into a dedicated repository, which removes the need for `@main` here.

A few smaller items in the same audit:

- No [OpenSSF Scorecard](https://securityscorecards.dev/) workflow for continuous supply chain health monitoring.
- Our `SECURITY-INSIGHTS.yml` expired in January 2025 and hasn't been updated. (We actually noticed this while writing this post.)
- No `go mod verify` step to validate vendor directory integrity against `go.sum` checksums.

If any of these look like a good first issue and you want to send a PR, we'd take it.

---

##### GitHub's 2026 Actions security roadmap and how it maps to what we do

In April 2026, GitHub published their [Actions security roadmap](https://github.blog/news-insights/product-news/whats-coming-to-our-github-actions-2026-security-roadmap/) describing platform-level changes across three layers: ecosystem, attack surface, and infrastructure. Reading it felt like validation of problems we've been working around for years, and a real signal that the platform is finally catching up to what large open source projects need. Here's how it maps to what we do today.

###### Dependency locking: making SHA pinning first-class

We pin every action by SHA and lean on Renovate to keep those pins current, but we still have a blind spot for transitive references. GitHub's planned `dependencies:` section in workflow YAML would lock all direct _and transitive_ dependencies by commit SHA, with hash verification before execution starts. That closes the gap.

###### Policy-driven execution: centralizing what we enforce per-file today

We restrict who can trigger workflows (Ariane's allow-list), which events are allowed (per-workflow configuration), and who can approve releases (protected environments). All of that is currently encoded across dozens of YAML files plus a custom bot, and auditing the full picture means reading every file.

GitHub's planned workflow execution protections, built on rulesets, would let us define those controls centrally at the org level: which actors can trigger workflows, which events are permitted, which repositories the rules apply to. We could prohibit `pull_request_target` org-wide except for the workflows where we've intentionally designed a safe two-phase checkout, instead of relying on code review and CODEOWNERS to enforce it.

###### Scoped secrets: closing the implicit inheritance gap

CI vs. production credential isolation is one of our strongest controls, but within a given environment, secrets are still scoped pretty broadly: any workflow running in that environment can access them.

Scoped secrets would let us bind credentials to specific workflow paths, branches, or even individual reusable workflows. A release credential could be restricted not just to the `release` environment but to the specific `release.yaml` workflow file, so a new workflow added to that environment (by accident or by an attacker) wouldn't inherit the credentials. That's a meaningful step beyond what protected environments alone provide.

The roadmap also separates secret management from repository write access. Today anyone with write access to a repo can manage its secrets. GitHub plans to move secret management into a dedicated custom role, which lines up with the least-privilege principle we already apply to workflow permissions but can't currently apply to secret administration.

###### Native egress firewall

GitHub's planned native egress firewall would restrict outbound network access from GitHub-hosted runners. It runs outside the runner VM at L7, so it's immutable even if an attacker gets root inside the runner. Organizations would define allowed domains, IP ranges, and HTTP methods, and anything else gets blocked.

For Cilium it's less critical than the rest. Our most security-sensitive workflows (release builds, image signing) already run with credential isolation and least-privilege permissions, which limits what a compromised step could do even with unrestricted network access. Building an accurate egress allow-list for a project that talks to container registries, Go module proxies, cloud APIs, and Sigstore would be a significant chunk of work. Public preview is expected in 6 to 9 months, so we'll evaluate then.

###### Actions Data Stream: making CI observable

Our workflows produce logs, but we don't have centralized telemetry for them. If a workflow starts behaving oddly (resolving unexpected dependencies, running longer than usual, making strange network calls), we'd have to notice it manually.

Actions Data Stream would deliver near real-time execution telemetry to external systems (S3, Azure Event Hub), covering workflow execution details, dependency resolution patterns, and eventually network activity. For an open source project with hundreds of workflow runs per day, that's a blind spot worth closing.

##### The point

Supply chain security is mostly the practice of repeatedly asking "what if this thing I trust gets compromised?" and adding a layer that limits the blast radius when it does.

We've tried to build defense in depth: access controls so only trusted people can trigger builds, pinned digests so a compromised tag can't reach us, least-privilege permissions so a rogue action can't exfiltrate secrets, credential isolation so CI can never touch production, and signatures so users can verify what they're running.

None of this makes us invulnerable. But security by obscurity isn't really a thing, and the inverse is also true: the more open source projects share their defenses openly, the higher the collective bar for attackers. We've shown you ours, including the parts that aren't great yet. If you're running CI/CD for an open source project and you've solved something we haven't, open an issue, write your own post, or come tell us on Slack. The open source supply chain is only as strong as its weakest project, and the only way to strengthen it is together.

---

_Relevant resources: [OpenSSF Scorecard](https://securityscorecards.dev/) · [SLSA Framework](https://slsa.dev/) · [Sigstore](https://www.sigstore.dev/) · [StepSecurity Harden Runner](https://github.com/step-security/harden-runner) · [GitHub Actions Security Hardening](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-for-github-actions) · [GitHub Actions 2026 Security Roadmap](https://github.blog/news-insights/product-news/whats-coming-to-our-github-actions-2026-security-roadmap/)_

#### Understanding Kubernetes Load Balancing

URL: https://cilium.io/blog/2026/04/25/understanding-kubernetes-load-balancing
Date: 2026-04-25
Categories: Technology

##### Understanding Kubernetes Load Balancing

##### I.Introduction

Kubernetes Load Balancing is a means to distribute network traffic to application instances. This can be within the same cluster or different compute regions or clusters.

To ensure proper utilization and desired performance, each healthy running instance of your application has to get a portion of the traffic. Traffic has to be distributed by load balancing between these instances; the distribution weight will depend on the algorithm.

###### Services in Kubernetes

Services are used to expose applications running inside the cluster behind a single outward-facing endpoint, even when the workload is split across multiple backends. This is what makes load balancing possible.

###### Why Kubernetes Load Balancing?

The main goals that Load Balancing is trying to achieve are Service Reliability, Availability, and Performance (Horizontal Scalability).
Load balancing ensures reliability by using health checks and readiness probes to steer traffic away from failing pods. By distributing incoming requests across multiple replicas, it guarantees high availability, ensuring that the failure of a single pod or even an entire physical node does not result in application downtime.

##### II.How does Kubernetes Load Balancing Work?

Load Balancing in Kubernetes clusters can be split into two perspectives.

###### Types of Load Balancing

Not all load balancing works the same way. Different layers of the network stack offer different trade-offs between performance, intelligence, and flexibility. Understanding these distinctions helps in choosing the right approach for a given workload, whether you need raw throughput at the network edge or fine-grained routing logic at the application level.

###### Internal Load Balancing

Internal load balancing refers to the distribution of traffic within a Kubernetes cluster itself among pods of the same application or service.
Kubernetes uses services to implement internal load balancing. For internal load balancing services with designated cluster IPs (reachable within the cluster), fit the purpose.

When pod A needs to talk to pod B inside the same cluster, it doesn’t use pod B’s direct IP; instead, it uses a Kubernetes service (stable virtual IP).
By default, Virtual IPs are managed by kube-proxy, a network agent running on each node. Kube-proxy watches the Kubernetes API for changes to Services and Endpoints and translates them into local networking rules.

Originally, this was done using iptables, which relies on sequential list processing, or IPVS, which improved performance through hash tables and advanced algorithms like Round-robin and Least Connection.

However, as clusters scale, the overhead of managing thousands of iptables rules can degrade performance. To improve performance, Cilium can replace kube-proxy and iptables. By using eBPF, Cilium processes packets at the lowest level of the network stack without the context switching required by iptables. This provides significantly higher throughput, lower latency, and more granular security.

While kube-proxy is limited to Layer 4 (IP/Port), Cilium can perform Layer 7 (HTTP/gRPC) load balancing and observability, providing a more identity-aware networking layer that is both faster and more resilient at scale.

###### External Load Balancing

External load balancing is the gateway that connects outside users to services running inside the cluster. While internal load balancing manages East-West traffic (pod-to-pod), external load balancing handles North-South traffic (internet-to-pod).

This is achieved through three primary methods:

- NodePort, which exposes a specific port on every node's IP.
- LoadBalancer, which integrates with infrastructure providers.
- Ingress, which acts as a Layer 7 smart router (handling hostnames and SSL), sits behind one of the aforementioned service types.
- Gateway API is a family of API kinds that provide dynamic infrastructure provisioning and advanced traffic routing. Gateway API is the successor to Ingress. Cilium has native Gateway API support built directly into its Envoy and eBPF data path; no separate ingress controller is needed. A Gateway resource backed by Cilium gets L7 routing, native load balancing, and full Hubble observability on every request without an additional proxy in the path.

For users not running in a cloud environment, such as those on bare metal or on-premises data centers, Kubernetes does not have a native "out-of-the-box" load balancer implementation.

In these cases, tools like Cilium Load Balancer and MetalLB are used, acting as a software-defined load balancer that monitors the cluster and assigns a virtual IP (VIP) from a pre-configured range to the service. It then uses standard network protocols like ARP (Layer 2\) or BGP (Layer 3\) to announce to the local network that the service is reachable at that specific IP.
This allows local environments to have the same "Type: LoadBalancer" experience as cloud users without relying on a cloud provider's hardware.

###### Layers of Kubernetes Load Balancing

In Kubernetes, load balancing happens across distinct layers of the OSI model, depending on where traffic originates and what information is needed to route it.

###### Data Link Layer: L2-Aware Load Balancers

The Data Link layer is the second layer of the OSI model, responsible for node-to-node data transfer between directly connected devices.

L2-Awareness allows Kubernetes services to be reachable via Address Resolution Protocol (ARP) for IPv4 or Neighbor Discovery Protocol (NDP) for IPv6. Using a feature like Cilium L2 Announcements, a leader node responds to network queries by broadcasting its own MAC address as the destination for a Service's Virtual IP. This bridges the gap between the physical switch and the virtual cluster, making services visible on the local area network (LAN) without requiring complex routing protocols like BGP.

Before Kubernetes can distribute traffic to pods, the external network must first know which physical node in the cluster owns the Service IP. In cloud environments, this is handled by the provider’s Software Defined Network. However, in on-premises or bare-metal environments, the cluster must manage its own physical identity.

###### L2-Aware Load Balancing and Service Announcement in Cilium

L2-Aware Load Balancer introduces the ability for Kubernetes services to be reachable via Address Resolution Protocol (ARP) announcements. L2 Announcements is a feature that makes services visible and reachable on the local area network. This feature is primarily intended for on-premises deployments within networks without BGP-based routing, such as office or campus networks or home labs.

When used, this feature will respond to ARP/NDP queries for ExternalIPs and/or LoadBalancer IPs. These IPs are Virtual IPs (not installed on network devices) on multiple nodes, so for each service, one node at a time will respond to ARP/NDP queries and respond with its MAC address. This node will perform load balancing with the service load balancing feature, thus acting as a north/south load balancer. To use this mode, Kube-proxy Replacement must be enabled.

**Failover and Availability**
To ensure that external traffic always has a reliable path into the cluster, Cilium implements an automated Failover mechanism. Since local networks typically associate a single Service IP with a single physical node at any given time, Cilium must ensure there is always one "active" responder while preventing conflicts where multiple nodes try to claim the same traffic.

**L2 Pods Announcements**
Layer 2-pod announcement is a means that allows individual pods to be directly visible on the local network by assigning each pod an IP address from the local network range and broadcasting its presence to the physical switch.

Read More: [https://docs.cilium.io/en/stable/network/l2-announcements/](https://docs.cilium.io/en/stable/network/l2-announcements/)

###### Transport Layer: L4 Load Balancers

The transport layer is the fourth layer of the OSI model, responsible for managing end-to-end communication, flow control, segmentation, and error correction between host systems.

Once a packet has physically reached a node, Layer 4 load balancing determines which specific pod should receive it based on IP addresses and TCP/UDP ports. L4 load balancers are highly efficient because they do not inspect the data inside the packet; they simply forward traffic based on the connection header.

In Kubernetes, standard ClusterIP and LoadBalancer services operate at this layer. Traditionally managed by kube-proxy using iptables or IPVS, modern CNIs like Cilium now handle this via eBPF. This allows for high-performance North-South (external to internal) and East-West (pod to pod) distribution with minimal latency.

###### Application Layer: L7 Load Balancers

The application layer is the top-most layer of the OSI model, which is a direct interface between end-user applications and the network.

Layer 7 Load Balancers operate using application contexts, including HTTP headers, URL paths, or gRPC methods. This allows advanced routing such as sending /api traffic to one set of pods and /web traffic to another. Ingress controllers and Application Load Balancers operate here.

##### III. Cilium Standalone XDP L4 Load Balancer

XDP (eXpress Data Path) is a high-performance packet processing framework built on eBPF. Traditional load balancers process packets after the kernel has already done significant work; interrupts, buffer allocation, and traversing firewall chains. XDP skips all of that by intercepting packets as early as possible, before any of that overhead kicks in.

The Cilium XDP L4LB comes with full IPv4/v6 dual-stack support that can be deployed and programmed independently of Kubernetes.

Read More: [https://cilium.io/blog/2022/04/12/cilium-standalone-L4LB-XDP/](https://cilium.io/blog/2022/04/12/cilium-standalone-L4LB-XDP/)

###### How Cilium Implements XDP L4LB

In a Cilium-managed cluster, the XDP load balancer operates as a "stand-alone" or "integrated" gateway. When an external packet destined for a Service IP hits the node:

1. XDP Hook: The eBPF program attached to the NIC intercepts the packet.
2. Maglev Lookup: Cilium uses a consistent hashing algorithm (Maglev) to select a backend Pod, ensuring that traffic from the same flow always hits the same destination.
 Kubernetes service load balancing implemented by Cilium or Kube-proxy selects backends randomly and ensures that the traffic remains sticky to that backend. In the case of node failure, the upstream load balancer selects a different load balancing node that has no context of the failed node. This can lead to an unexpected disruption on connection oriented protocols.
 Maglev consistent hashing minimizes such disruption by ensuring each load-balancing node has a consistent view and ordering of the backend lookup table.
3. Encapsulation/DSR: The packet is either encapsulated (VXLAN/Geneve) or sent via Direct Server Return (DSR) to the target node.
4. XDP_TX: The modified packet is sent back out the same interface immediately, completely bypassing the host's networking stack. XDP_TX is an XDP action that involves TX bouncing the received packet page back out the same NIC it arrived on.

Read More: [https://cilium.io/blog/2022/04/12/cilium-standalone-L4LB-XDP/](https://cilium.io/blog/2022/04/12/cilium-standalone-L4LB-XDP/)

###### Operating Modes

L4 Load Balancers work in two modes:

###### Passthrough Mode

Packets are forwarded without inspecting application data, and the same TCP connection is used from the client to the backend. There is no connection termination in L4 LB. Connections are terminated by backend servers, and responses from backend servers are sent directly to clients with Direct Server Return (DSR). Pass-through LoadBalancers can not terminate SSL certificates.

###### Proxy Mode

In this operating mode, the load balancer terminates a connection from the client and initiates a different connection to the downstream service. In proxy mode, the Load Balancer fully handles the TCP handshake, resulting in more flexibility for routing decisions to destinations.

##### IV. Summary

Kubernetes load balancing is a critical architecture for managing traffic across ephemeral, scaled application instances. By decoupling fixed service endpoints from transient pod IP addresses, it ensures Service Reliability, High Availability, and Performance.

To achieve production-grade resilience, these layers use health checks and automated failover mechanisms. Whether through traditional proxy modes or high-performance passthrough with Direct Server Return (DSR) and Maglev consistent hashing, Kubernetes load balancing ensures that network traffic is distributed predictably and efficiently across the entire cluster lifecycle.

Cilium redefines this stack by replacing legacy components like kube-proxy and iptables with a unified, eBPF-native data path. This shift provides three distinct advantages across the networking layers **Foundation (L2/L3)** Cilium bridges the gap for on-premises and bare-metal environments, **Performance (L4/XDP)** By leveraging XDP (eXpress Data Path), Cilium processes packets at the earliest possible point in the network driver and **Intelligence (L7/Ingress)**, At the application level, Cilium provides identity-aware routing.

#### Understanding Kubernetes Networking

URL: https://cilium.io/blog/2026/04/25/understanding-kubernetes-networking
Date: 2026-04-25
Categories: Technology

##### Understanding Kubernetes Networking

##### I.Introduction: Rules of the Game

Kubernetes is all about orchestrating applications across machines. Kubernetes networking enables communication across nodes, pods, and external services using a flat network[^1] structure and allows pods, services, nodes, and external resources to communicate inside a Kubernetes cluster. Container runtimes use an implementation of the Container Network Interface (CNI) specification to manage the network.

A CNI plugin is not built into Kubernetes itself. Instead, Kubernetes calls the network plugin when pods are created. A CNI plugin ensures all pods are assigned IPs, configures routes, and ensures that traffic reaches the destination.

###### Fundamental Rules of Kubernetes Networking

- Flat Network Structure: all pods can reach each other without special gateways, whether on the same or different nodes.
- Every pod gets its own IP address. **The "IP-per-Pod" concept**

The goal is for each pod to have an IP in a flat shared networking namespace that has full communication with other pods across the network. IP-per-pod creates a clean, backward-compatible model where pods can be treated much like VMs or physical hosts from the perspectives of port allocation, networking, naming, service discovery, load balancing, application configuration, and migration.

- No Internal NAT: Internal pods communication doesn’t use Network Address translation.
- The same network namespace for containers inside the same pod: containers in the same pod share an IP address and share port space.
- Services as stable access points: services provide virtual IPs that map to backend pods.
- Overlay or underlay networks: Some CNIs use overlay networks to encapsulate traffic. Others rely on Layer 3 routing to encapsulate traffic.

Read More: [https://isovalent.com/blog/post/what-is-kubernetes-networking/](https://isovalent.com/blog/post/what-is-kubernetes-networking/)

###### Core Networking Requirements

- Highly coupled Container to Container communications: containers within the same pod share the same network namespace. Communication is done via localhost.
- Pod-to-Pod communications:
 Every pod can communicate directly with any other pod across nodes without NAT since each pod is assigned a real IP. This enables all naming or discovery mechanisms to work out of the box.
- Pod to Service communications: A service is an abstraction of a group of pods. Services are assigned Virtual IPs proxied to pods in a service. Since pods are ephemeral workloads, they don’t need to care about pod IP changes; Kubernetes automatically tracks these via Service EndpointSlice. Kube-proxy keeps track of the iptables to track VIP services; Cilium does this smartly by replacing iptables with eBPF. With eBPF, packet decisions happen at the earliest hook, and faster network policy evaluation with a low CPU overhead.
- External to Internal communications: Exposing and accessing services, such as external services or databases, from outside the Kubernetes cluster.

Read More: [https://github.com/kubernetes/design-proposals-archive/blob/main/network/networking.md](https://github.com/kubernetes/design-proposals-archive/blob/main/network/networking.md).

###### Why does traditional networking fail in a dynamic, containerized world?

Containerised environments often run resources with an ephemeral nature, and traditional networking technologies struggle to keep up with quickly changing environments.

##### II. Why Cilium as your CNI?

Container Network Interface is a CNCF project that specifies the relationship between a Container Runtime interface (CRI), such as containerd, responsible for container creation, and a CNI plugin tasked with configuring network interfaces within the container upon execution.

Read More: [https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)
[https://isovalent.com/blog/post/demystifying-cni/](https://isovalent.com/blog/post/demystifying-cni/)

Many CNI plugins satisfy basic Kubernetes networking requirements, but Cilium addressed the networking requirements with a different approach, building directly in the Linux kernel with eBPF.
eBPF allows Cilium to embed logic or programs into the Linux kernel dynamically.

The following are key capabilities that Cilium offers to support the Kubernetes networking model.

1. Performance and Scalability: Cilium avoids old routing technologies like iptables or IPVS that limit performance and scalability.
2. Identity-based security: Instead of managing security via ephemeral IP addresses, Cilium assigns a unique security identity to pods based on metadata. Most CNIs can enforce network policies at L3 and L4, but Cilium extends this capability to provide L7 control capabilities too.
3. Multi-cluster connectivity: Cilium provides the ability to connect multiple clusters across different clouds.
4. Transparent Encryption: Kubernetes lacks native pod-to-pod encryption. Two common solutions to this problem are embedding encryption within the application or using a service mesh. Embedding encryption within the app is complex and requires application and security expertise. Cilium provides a straightforward solution for enabling the encryption of all node-to-node traffic with just one switch, no application changes, or additional proxies.
5. Observability: Cilium's Hubble empowers users to monitor, analyze, and optimize their Kubernetes networking environments with ease. With both a CLI and service map, Hubble helps quickly identify problems in the network.

##### III.How do packets move?

After we have all pods allocated with IP addresses, here comes the Cilium data path. It is a set of eBPF programs Cilium uses to process, route, and forward packets within the Kubernetes cluster. The CNI determines the actual path that data takes through the Linux kernel and Kubernetes cluster.

###### Building Blocks

1. Network Namespaces

Every pod is given an illusion to assume it’s running its own network namespace, providing a dedicated network stack (interfaces, routing tables, and firewall rules).

2. Veth Pairs

Virtual Ethernet (veth) pairs are connected virtual Ethernet interfaces that act as a tunnel between network namespaces. Data sent into one interface appears on the other, facilitating communication between isolated networking stacks.

###### Layer 1: Intra-Node Networking

When packets are sent from pod A to pod B on the same host, packets exit the pod A namespace through the peer interface.

With a traditional CNI, this packet might be passed through slow iptables chains. Cilium instead intercepts this packet immediately using the eBPF program running on the veth pair interfaces. It then consults internal maps for security policies and identity. Hence, decisions are made directly in the data path at the earliest possible moment. This gives Cilium the capability to scale without CPU overhead. Finally, the packet is forwarded to pod B’s interface.

###### Layer 2: Inter-Node Networking

When a packet is destined for a Pod on a different node, Cilium must decide how to move it across the physical network fabric. There are two primary routing models.

1. ### Native Routing Mode

In native routing, the underlying network is aware of your Pod IP address. In this mode, Cilium is more than just a CNI as it’s capable of BGP peering with your router or TOR devices. Cilium advertises all IP addresses it assigns to the router. In this setup, Cilium communicates directly with an external BGP-capable router, FRR. This allows external traffic to be routed directly to nodes. With this model, you need to think carefully about IPAM and address ownership, BGP peering topology and failure domains, route aggregation and scale, interaction between L2/L3 modes, and the physical network.

In native routing mode, when a packet from pod A in node A is dispatched, it’s routed through the host network stack and placed directly onto the physical network without any encapsulation. The underlying fabric uses the destination pod IP to route the traffic.

Read More: [https://docs.cilium.io/en/latest/network/concepts/routing/\#native-routing](https://docs.cilium.io/en/latest/network/concepts/routing/#native-routing)

2. ### Encapsulation/Overlay Mode

In overlay mode, Cilium creates a virtual “mesh” of tunnels over your existing physical network. By using UDP-based encapsulation protocols like VXLAN or Geneve, Cilium effectively abstracts the Pod network from the underlying infrastructure. Overlay mode hides the complexity of Pod IP addresses, meaning the physical switch or router only needs to know how to reach the Nodes, not the individual Pods.

###### How does the Data Travel

When a pod sends a packet, it transforms, leaving the host:

- The Inner Packet: Contains the original Pod-to-Pod data and the source/destination Pod IPs.
- The Outer Header: The original packet is wrapped (encapsulated) in a UDP header that uses the Node’s IP address as the source and destination.

Because the physical network only sees Node-to-Node traffic, connectivity is highly resilient.

###### The Trade-offs

While Overlay mode introduces a small amount of encapsulation overhead (extra bytes in the packet header), it provides operational simplicity. In this mode, Cilium creates a virtual network that sits on top of the existing physical infrastructure.

Read More: [https://docs.cilium.io/en/latest/network/concepts/routing/\#encapsulation](https://docs.cilium.io/en/latest/network/concepts/routing/#encapsulation)

| Feature | Native Routing | Encapsulated (Overlay) |
|:---------- |:------------------------------------------------ |:-------------------------------- |
| Complexity | High (requires configuration on physical network) | Lower (Works out of the box) |
| Performance | High (no overhead) | Moderate (encapsulation overhead) |
| Visibility | Pod IPs visible to the network | Pod IPs hidden from the network |

##### IV. Service, Service Discovery & Cluster DNS

Since pods are ephemeral and each time a new pod spins up, it is assigned a different IP address by the CNI, other applications and clients need to use pod identities that do not change. A group of pods running an application is exposed as a network application using services. Services in Kubernetes are just an abstraction to help expose groups of pods over the network.

A Service defines a logical set of endpoints (pods) aligned with a policy on how to make those pods accessible. The set of pods targeted by a service is determined by a label selector, which identifies pods based on assigned key-value pairs.

Every service is assigned a DNS name that maps to the cluster IP of the service.

###### Service Types

The following are the kinds of services supported in Kubernetes:

1. ### Cluster IP

Exposes a service using an IP address that is accessible only within the cluster. This is a default type when a service type is not specified. You can ingest traffic from the public internet to a cluster IP service using ingress or the Gateway API.

2. ### NodePort

Exposes a service on each node's IP at a static port. Kubernetes creates a Cluster IP to which the node port traffic will be forwarded.

3. ### LoadBalancer

Exposes the service externally using an external LoadBalancer. Traffic from the external LoadBalancer is directed to the backend pods. While Cloud providers (like AWS or GCP) automatically provision a managed load balancer for this purpose, Bare Metal environments require a specialised controller to handle this traffic.
In bare metal environments, Cilium facilitates this in two primary ways: Border Gateway Protocol (BGP) and L2 Announcements.

4. ### ExternalName

Maps the service to a DNS name rather than a set of Pods. Instead of using selectors to route traffic to internal Pods, it acts as a Canonical Name (CNAME) record within the cluster’s DNS.
When a Pod tries to reach a service of kind externalName, the cluster’s DNS service returns the value defined in the externalName field instead of a cluster IP.

5. ### Headless

A Headless service is a service that allows a client to connect to whatever Pod it prefers, directly.
Headless services don't configure routes and packet forwarding using virtual IP addresses and proxies; instead, headless Services report the endpoint IP addresses of the individual pods via internal DNS records, served through the cluster's DNS service. Headless services are useful for stateful applications.

Read More: [https://kubernetes.io/docs/concepts/services-networking/service/](https://kubernetes.io/docs/concepts/services-networking/service/)

###### Cluster DNS

Cluster DNS can be set using add-ons. DNS stands for Domain Name System. It provides a naming system for computers on the Internet. It translates domain names to IP addresses. In a cluster, services are accessed with service names. The Kubernetes DNS server is the only way to access externalName services.

###### Service Discovery

Service discovery in Kubernetes may be provided in different ways; DNS-Based Service Discovery is the most common means of discovery and the recommended method.

When a Service is created, the Cluster DNS automatically generates records mapping the Service name to its corresponding Cluster IP, allowing applications to communicate via stable hostnames rather than volatile IP addresses. To facilitate this, the Kubelet configures each Pod’s /etc/resolv.conf file to point toward the internal DNS service, enabling containers to perform lookups natively. This resolution is context-aware: while a standard query is scoped to the Pod's local namespace by default, services in other namespaces can be reached by using their Fully Qualified Domain Name (FQDN), ensuring precise service discovery across the entire cluster.

###### Virtual IPs and Service Proxies

Every node runs a kube proxy (if not replaced completely). Kube proxy implements a virtual IP mechanism for services of type other than ExternalName. Kube proxy watches the control plane for the addition or removal of service and endpointSlice objects. In most deployments of Cilium, kube-proxy is replaced completely by eBPF programs.

###### How Cilium Optimizes Service Discovery?

While Kubernetes manages DNS records, Cilium handles high-performance routing that makes discovery work at scale.

- eBPF vs. iptables: Traditionally, kube-proxy uses iptables rules to route traffic. Cilium replaces this with eBPF hash lookup tables, which are very optimized.
- DNS queries visibility: With Hubble, you can see which pods are querying DNS records and whether those queries are succeeding or not.
- DNS proxy: Cilium uses a DNS proxy to listen for DNS traffic. This adds a layer of granularity for security enforcement. This allows you to write security policies based on FQDN and not only IPs.

##### V. Summary

The Kubernetes networking model requires an understanding of the shift from static, physical infrastructure to dynamic, software-defined environments. While the fundamental rules, such as the IP-per-Pod model and Flat Network Structure, provide the necessary consistency for distributed applications, traditional networking tools often struggle to keep pace with the ephemeral nature of containers.

Cilium doesn't just fulfill the CNI specification; it has become the industry-standard foundation for cloud native infrastructure by moving networking logic directly into the Linux kernel via eBPF. This shift has led to massive adoption across the ecosystem, with major cloud providers (including Google, AWS, and Azure) choosing Cilium as their default networking layer. As a CNCF-graduated project with a vibrant, global community, Cilium has effectively set the new benchmark for networking, security, observability, and scale in the modern enterprise.

[^1]: A flat network is a network architecture where all devices can communicate to each other without going through any NAT.

#### Native mTLS for Cilium: Transparent Encryption Meets Cloud Native Identity with ztunnel

URL: https://cilium.io/blog/2026/03/23/native-mtls-cilium
Date: 2026-03-23
Categories: Technology

##### Native mTLS for Cilium: Transparent Encryption Meets Cloud Native Identity

*Announcing native mutual TLS in the Cilium datapath — zero-drop, inline encryption with Kubernetes-native identity, no sidecars required.*

Today at [KubeCon + CloudNativeCon Europe 2026](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/) in Amsterdam, we are excited to announce **native mTLS support in Cilium** — a major step forward for transparent, zero-trust encryption in Kubernetes. This capability brings inline mutual TLS directly into the Cilium datapath, unifying authentication and encryption into a single operation. No sidecars, no dropped first packets, and no application changes required.

Mutual TLS has long been the gold standard for workload-to-workload security in cloud native environments, but achieving it has historically meant accepting significant operational complexity. With this release, Cilium eliminates that tradeoff.
##### How Open Source Drives Innovation: Lessons from the Evolution of the Service Mesh

In the Cilium project, we’ve always believed that open source isn’t just a way to share code, it’s also a way to drive innovation. By putting ideas into the public domain, engineers learn from them, challenge them, and build on them to drive progress forward. The evolution of the service mesh towards ztunnel is a perfect case study of this process in action. Projects like Cilium, Istio, and Linkerd have each contributed unique approaches, learning from one another to enhance performance, scalability, and user experience.

##### Different Paths, Same Destination

When the service mesh pattern first emerged, sidecars were the dominant model. Projects like Istio and Linkerd standardized on injecting a proxy alongside each application instance, offering consistent L7 observability, security, and traffic management. But as these meshes grew in adoption, so did the operational complexity and performance overhead of sidecar-heavy architectures.

Interestingly, Linkerd actually started with a per-node proxy model in its earliest versions. However, they encountered real limitations which led them to pivot toward a sidecar-based approach. It was a pragmatic move: sidecars gave them per-pod context and control, which was hard to achieve with the technologies available at the time.

##### Cilium: Rethinking Service Mesh with eBPF

Fast forward a few years, and the landscape had changed. Cilium introduced a sidecarless service mesh built on eBPF, which enabled per-pod visibility, identity, and policy enforcement, all without the need for injected sidecar proxies. With eBPF running directly in the Linux kernel, Cilium was able to revisit the original promise of node-local simplicity while delivering per-pod context and control.

This was a major inflection point. By removing the operational burden of sidecars while maintaining strong security and observability guarantees, Cilium opened the door to a new generation of service mesh architectures that removed the need for a sidecar injected into each and every pod.

##### Enter ztunnel

Cilium’s introduction of the sidecarless approach led to months of debate and discussion about the pros and cons. In our view, healthy discussion and probing different technical positions can lead to real progress. The following year, Istio Ambient was announced (listen to an interview with the authors on why they created it [here](https://kubernetespodcast.com/episode/189-ambient-mesh/)), proposing a new data plane model for Istio that eliminates sidecars in favor of a per-node ztunnel component for Layer 4 security, and an optional Layer 7 proxy layer for application traffic control.

[Performance results](https://arxiv.org/pdf/2411.02267) for Ambient are promising—lower CPU usage, faster pod starts, and improved network efficiency compared to Istio’s traditional sidecar deployments. But what really piqued our interest is that for encrypted TCP connections, the ztunnel approach even out-performs Cilium.

One of the main reasons for ztunnel’s better performance is that by encrypting a TCP payload stream, it can avoid the per-packet encryption overhead inherent in Cilium’s use of WireGuard/IPsec.

But this performance boost does come with a few trade-offs.

One area to consider is upgrades. Cilium’s eBPF programs can be left in the kernel during a control plane upgrade, allowing data to continue flowing while user space components are replaced or restarted. With ztunnel, any TCP connections will need to be torn down during upgrade.

Another factor, which may be important to some users, is that WireGuard and IPsec can be used to encrypt any IP traffic, whereas ztunnel is limited to TCP traffic for now.

##### Bringing ztunnel to Cilium

When we started looking at [bringing ztunnel support into Cilium](https://github.com/cilium/cilium/issues/38548), it seemed like we could combine the best of both worlds. The top-tier performance of ztunnel, combined with Cilium’s ease of policy control, and the ability to support all types of IP traffic.

Bringing ztunnel support into Cilium brings native mTLS to the data path and enables enhanced performance for ztunnel encrypted connections.

###### The Challenge: Network-Level Authentication, Not Session-Based

Cilium's [existing mutual authentication](https://isovalent.com/blog/post/2022-05-03-servicemesh-security/) capability (introduced as beta in Cilium 1.14) was a significant innovation, using eBPF to verify workload identity at the datapath level. However, it had some important limitations:

- **Network-level, not session-based authentication.** The mutual authentication handshake had nodes authenticate each other and then encapsulate all network traffic to build an authenticated and encrypted virtual network. This meant authentication happened at the node level rather than on a per-session basis — so, for example, pod-to-pod encryption on the same node was not possible.
- **First packets were dropped.** When a new identity pair communicated for the first time, the initial packet was dropped while an out-of-band TLS 1.3 handshake completed between Cilium agents. Applications had to tolerate this retry penalty.
- **External dependencies were required.** A full SPIRE deployment was mandatory — adding infrastructure complexity for teams that simply wanted encrypted pod-to-pod communication.

The following diagram illustrates this existing flow:

```
                     FIRST PACKET (DROPPED)

  Pod A          BPF Datapath         Cilium Agent (A)       Cilium Agent (B)
    |                 |                     |                       |
    |-- 1. Packet --> |                     |                       |
    |                 |-- 2. Policy check:  |                       |
    |                 |   auth_type = SPIRE |                       |
    |                 |-- 3. auth_map       |                       |
    |                 |   lookup -> MISS    |                       |
    |                 |                     |                       |
    |   X DROPPED     |-- 4. Signal ----->> |                       |
    |                 |                     |-- 5. Get SPIFFE cert  |
    |                 |                     |   from SPIRE          |
    |                 |                     |== 6. TLS 1.3 ======>> |
    |                 |                     |   (out-of-band)       |
    |                 |                     |   7. Validate cert  <<|
    |                 |<<-- 8. Write -------|                       |
    |                 |   auth_map entry    |                       |

                     SUBSEQUENT PACKETS (ALLOWED, BUT PLAINTEXT)

  Pod A          BPF Datapath (A)                      BPF Datapath (B)    Pod B
    |                 |                                       |               |
    |-- 9. Packet --> |                                       |               |
    |                 |-- auth_map lookup -> HIT              |               |
    |                 |========= PLAINTEXT ================>> |-- forward --> |
```

While this approach is elegant for pure identity verification, it leaves a critical gap in environments that require confidentiality guarantees — which is, increasingly, every production Kubernetes cluster.

###### Introducing Native mTLS: Ztunnel Integration

Cilium's native mTLS is powered by [ztunnel](https://github.com/cilium/ztunnel) — a lightweight, per-node Rust proxy originally from Istio's ambient mesh — reengineered to integrate directly with the Cilium control plane and **SPIRE** as the certificate authority. We introduced native SPIRE support into ztunnel ([cilium/ztunnel#4](https://github.com/cilium/ztunnel/pull/4)), replacing the upstream CA dependency with SPIRE's Delegated Identity API. SPIRE is the **only** CA mode supported in Cilium's mTLS solution, providing a production-grade, [CNCF-graduated](https://www.cncf.io/projects/spiffe/) identity foundation.

Critically, because ztunnel operates at the pod level rather than the node level, **mTLS encryption applies to all pod-to-pod communication — whether the pods are on the same node or on different nodes**. Even two pods scheduled on the same host have their traffic encrypted through ztunnel, ensuring consistent security guarantees regardless of pod placement.

While we are currently maintaining a fork to accelerate these capabilities, our goal is to align with and eventually upstream these improvements to the broader ecosystem.

Here is what this looks like in practice for cross-node communication:

```
  Pod A             Ztunnel              Ztunnel              Pod B
  (enrolled)        (Node A)             (Node B)             (enrolled)
    |                  |                    |                    |
    |-- App traffic -->|                    |                    |
    |   (plaintext)    |                    |                    |
    |                  |                    |                    |
    |   PLAINTEXT      |  ENCRYPTED mTLS    |    PLAINTEXT       |
    |  --------------> |  ==============>   |  ----------------> |
    |                  |  HBONE tunnel      |                    |
    |                  |  HTTP/2 CONNECT    |  [ztunnel decrypts |
    |                  |  over mTLS 1.3     |   and delivers     |
    |                  |  to :15008         |   into pod]        |
    |                  |                    |                    |
```

For same-node communication, the flow is similar — traffic from Pod A is still intercepted by ztunnel, encrypted via mTLS, and delivered to Pod B through ztunnel, even though both pods reside on the same host:

```
                        Same Node
  Pod A             Ztunnel                   Pod B
  (enrolled)        (local)                   (enrolled)
    |                  |                        |
    |-- App traffic -->|                        |
    |   (plaintext)    |                        |
    |                  |-- ENCRYPTED mTLS ----->|
    |                  |   HBONE tunnel         |
    |                  |   (loopback)           |
    |                  |   [ztunnel encrypts,   |
    |                  |    then decrypts and   |
    |                  |    delivers into pod]  |
    |                  |                        |
```

**Every packet from an enrolled pod is encrypted — regardless of where the destination pod is scheduled.** There is no plaintext window, no dropped first packets, and no separate WireGuard or IPsec layer to configure. The connection is held inline by ztunnel until the mTLS tunnel is established, then traffic flows bidirectionally through an [HBONE](https://istio.io/latest/docs/ambient/architecture/hbone/) (HTTP/2 CONNECT) tunnel.

###### How It Works

Traffic interception is handled through lightweight in-pod iptables rules:

1. **Pod enrollment**: When a namespace is labeled with `io.cilium/mtls-enabled=true`, the Cilium agent enrolls all pods in that namespace. It enters each pod's network namespace and installs iptables rules that redirect outbound traffic to ztunnel on port `15001`.

2. **mTLS tunnel establishment**: Ztunnel on the source node looks up the destination workload via XDS (streamed from the Cilium agent) and initiates an mTLS 1.3 connection to the destination node's ztunnel on port `15008`. The SPIFFE identity — `spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>` — is embedded in the certificate and verified at both ends.

3. **Traffic delivery**: The destination ztunnel decrypts the traffic and delivers it into the target pod, bypassing the interception rules via an in-pod mark. The application sees a normal plaintext connection and is completely unaware encryption happened.

4. **Certificate management via SPIRE**: Workload certificates are issued by **SPIRE**, the CNCF-graduated implementation of the SPIFFE standard. The Cilium operator automatically registers SPIRE entries for enrolled service accounts, and ztunnel obtains short-lived X.509-SVIDs directly from the local SPIRE agent via workload attestation. This provides a robust, standards-based identity foundation with automatic certificate rotation.

###### Architecture Deep Dive

The control plane is cleanly split across the Cilium agent (per-node) and the Cilium operator (cluster-wide), following Cilium's Hive cell architecture.

###### Agent-Side Control Plane

Each Cilium agent runs two servers that communicate with the local ztunnel instance, while SPIRE handles certificate issuance:

```
┌──────────────────────────────────────────────────────────────────┐
│  cilium-agent (per node)                                          │
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐     │
│  │ ZDS Server (Unix socket: /var/run/cilium/ztunnel.sock)   │     │
│  │ Pod enrollment/disenrollment via netns FD passing        │     │
│  └──────────────────────────────────────────────────────────┘     │
│  ┌──────────────────────────────────────────────────────────┐     │
│  │ XDS Server (Unix socket: /var/run/cilium/xds.sock)       │     │
│  │ Streams workload addresses + authorization policies      │     │
│  └──────────────────────────────────────────────────────────┘     │
│                              │                                    │
│                              ▼                                    │
│                         ztunnel proxy                             │
│                              │                                    │
│                              ▼                                    │
│                    SPIRE Agent (per node)                         │
│                    Workload attestation + X.509-SVID issuance     │
└──────────────────────────────────────────────────────────────────┘
```

- **ZDS (Ztunnel Discovery Service)**: Handles pod lifecycle. On enrollment, it enters the pod's network namespace, creates iptables redirect rules, and passes the netns file descriptor to ztunnel via protobuf over a Unix domain socket. On disenrollment, it cleans up.

- **XDS (xDS Discovery Service)**: Watches `CiliumEndpoint` and `CiliumEndpointSlice` resources and streams them as `istio.workload.Address` protos to ztunnel using gRPC Delta Aggregated Discovery Service (dADS). This tells ztunnel which workloads exist, their IPs, and whether they speak HBONE.

- **SPIRE Agent**: Runs as a DaemonSet alongside ztunnel on each node. Ztunnel obtains short-lived X.509-SVIDs (SPIFFE Verifiable Identity Documents) directly from the SPIRE agent through workload attestation. The SPIRE agent validates the workload's identity using Kubernetes selectors (namespace + service account) and issues certificates signed by the SPIRE server's trust chain.

###### Operator-Side Control Plane

The Cilium operator handles cluster-wide SPIRE identity registration:

```
┌──────────────────────────────────────────────────────────────────┐
│  cilium-operator (cluster-wide)                                   │
│                                                                   │
│  Namespace Reflector ──► EnrolledNamespace Table (StateDB)        │
│  ServiceAccount Reflector ──► ServiceAccount Table (StateDB)      │
│                      │                                            │
│                      ▼                                            │
│           Enrollment Reconciler ──► SPIRE Server (entry CRUD)     │
└──────────────────────────────────────────────────────────────────┘
```

When a namespace is enrolled, the operator queries all service accounts in that namespace and creates corresponding SPIRE registration entries with SPIFFE IDs in the format `spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>`, parented under `/ztunnel`. When a namespace is un-enrolled, those entries are cleaned up.

###### Enrollment Model: Gradual Rollout by Namespace

A key design decision is that mTLS enrollment is **per-namespace**, controlled by a single label:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    io.cilium/mtls-enabled: "true"
```

This gives platform teams a natural rollout mechanism: enable mTLS for individual namespaces as they are ready, without affecting the rest of the cluster. Mixed traffic scenarios work predictably:

| Source | Destination | Result |
|--------|-------------|--------|
| Enrolled | Enrolled | **Encrypted** (mTLS over HBONE) |
| Enrolled | Non-enrolled | Plaintext passthrough |
| Non-enrolled | Enrolled | Plaintext (captured by ztunnel, but not encrypted) |
| Non-enrolled | Non-enrolled | Normal Cilium datapath (no ztunnel involvement) |

Encryption only occurs when **both** pods are enrolled. Mixed traffic degrades gracefully to plaintext without breakage or hard failures.

###### Why This Matters

###### 1. mTLS by Default, Without Operational Overhead

Enrolled namespaces get mTLS with zero application changes, code modifications, sidecar injection, or certificate management burden on developers. Platform teams label a namespace and encryption is on.

###### 2. No Dropped Packets

Unlike the existing mutual auth approach, ztunnel holds connections inline while the mTLS tunnel is established. There is no "first packet drop" penalty — connections succeed on the first attempt.

###### 3. Kubernetes-Native Identity

Workload identity is expressed as `spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>`, aligning directly with Kubernetes RBAC primitives.

###### 4. Production-Grade Identity with SPIRE

Workload certificates are managed by [SPIRE](https://spiffe.io/), the CNCF-graduated reference implementation of the SPIFFE standard. SPIRE provides automatic certificate rotation, workload attestation via Kubernetes node and pod selectors, and a trust chain rooted in a dedicated SPIRE server. The Cilium operator automates SPIRE registration entry management — when a namespace is enrolled, all service account identities are automatically registered with SPIRE.

###### 5. Best of Both Worlds: Performance & Flexibility

By encrypting a TCP payload stream, ztunnel can avoid the per-packet encryption overhead inherent in IPsec. This provides a significant performance boost for TCP-heavy workloads. Meanwhile, users requiring encryption for non-TCP traffic or hitless upgrades can still opt for Cilium’s WireGuard or IPsec modes.

###### 6. Ecosystem Convergence

By integrating ztunnel's L4 proxy capabilities with Cilium's eBPF-powered datapath and SPIRE's identity framework, this solution brings together three CNCF projects into a cohesive, standards-based mTLS stack. It establishes a clean path toward L7 waypoint proxies in the future, and it means Cilium users benefit from the combined innovation across the cloud native ecosystem.

###### Comparison: Before and After

| | Existing Mutual Auth | Native mTLS (Ztunnel) |
|----------------------|-----------------------------------|----------------------------------|
| **Encryption** | Requires enabling IPsec/WireGuard | Built-in mTLS on every packet |
| **First packet** | Dropped until handshake completes | Held inline — no drops |
| **Identity** | `spiffe://.../identity/<num-id>` | `spiffe://.../ns/<ns>/sa/<sa>` |
| **Enrollment** | Per CiliumNetworkPolicy rule | Per namespace label |
| **Cert source** | SPIRE only | SPIRE (automated registration) |
| **Granularity** | Per identity pair (BPF cache) | Per workload pair (H2 mux) |
| **App changes** | None | None |

###### Getting Started

Enabling native mTLS requires three things:

1. **Enable the ztunnel feature** in your Cilium Helm values:

```yaml
ztunnel:
  enabled: true
```

2. **Deploy ztunnel** as a DaemonSet (managed via Helm).

3. **Enroll namespaces** by adding the label:

```bash
kubectl label namespace <name> io.cilium/mtls-enabled=true
```

That's it. All pods in enrolled namespaces will automatically have their traffic encrypted with mTLS. You can verify the status with:

```bash
cilium status
```

The ztunnel component will appear in the output, showing enrollment state and certificate health.

###### A Joint Effort: Microsoft and Isovalent

Native mTLS in Cilium is the product of a deep, joint engineering collaboration between **Microsoft** and **Isovalent** (the creators of Cilium, now part of Cisco). Microsoft's team has been **the driving force of the ztunnel feature** in the upstream Cilium codebase, a reflection of the shared commitment to building this capability in the open, for the entire community.

The work spans multiple areas:

**Core control plane** — merged upstream into Cilium:
- Remote workload awareness ([#41945](https://github.com/cilium/cilium/pull/41945))
- ZDS server for pod enrollment ([#42364](https://github.com/cilium/cilium/pull/42364))
- Namespace-based mTLS enrollment ([#41944](https://github.com/cilium/cilium/pull/41944))
- XDS namespace filtering ([#43128](https://github.com/cilium/cilium/pull/43128))
- `cilium status` integration ([#43227](https://github.com/cilium/cilium/pull/43227))

**Datapath and deployment** — enabling production readiness:
- Idempotent in-pod iptables rules ([#42122](https://github.com/cilium/cilium/pull/42122))
- go-iptables library migration ([#42160](https://github.com/cilium/cilium/pull/42160))
- Helm-managed ztunnel DaemonSet ([#43763](https://github.com/cilium/cilium/pull/43763))
- Configurable SPIRE client ([#44136](https://github.com/cilium/cilium/pull/44136))
- Namespace enrollment reconciler for SPIRE ([#44275](https://github.com/cilium/cilium/pull/44275))

**Ztunnel fork** ([cilium/ztunnel](https://github.com/cilium/ztunnel)) — native SPIRE support:

A key piece of this collaboration was introducing native SPIRE support directly into ztunnel. Upstream ztunnel was designed exclusively for Istio's control plane and its built-in CA (istiod) — it had no concept of SPIRE as a certificate authority. We introduced SPIRE's Delegated Identity API into ztunnel ([#4](https://github.com/cilium/ztunnel/pull/4)), enabling ztunnel to obtain X.509-SVIDs directly from the local SPIRE agent via workload attestation. This includes PID-based attestation where each workload is individually verified by its container process ID, providing stronger security guarantees than metadata-only approaches. SPIRE is the **only** CA mode supported in Cilium's mTLS solution — there is no fallback to Istio's CA. We look forward to contributing these changes back to the upstream ztunnel project to ensure long-term ecosystem alignment.

- Unix socket support for XDS ([#1](https://github.com/cilium/ztunnel/pull/1))
- Native SPIRE workload attestation via Delegated Identity API ([#4](https://github.com/cilium/ztunnel/pull/4))
- CI: container image build pipeline ([#7](https://github.com/cilium/ztunnel/pull/7))

**Testing** — comprehensive validation:
- Encryption connectivity tests ([#43229](https://github.com/cilium/cilium/pull/43229))
- Feature checks and e2e scenarios ([#43255](https://github.com/cilium/cilium/pull/43255), [#43803](https://github.com/cilium/cilium/pull/43803))

##### Conclusion: Open Source Drives Us All Forward

The journey to native mTLS in Cilium is a perfect case study in how open source drives innovation. By putting ideas into the public domain, engineers learn from, challenge, and build upon them. The evolution of the service mesh has seen many paths, from the early sidecar models of Istio and Linkerd to Cilium’s eBPF-powered sidecarless approach. Each iteration has been a response to the community's need for better performance and lower operational complexity.

When Istio Ambient introduced ztunnel, it offered a new data plane model that offered new efficiency gains in encrypting TCP payload streams. By integrating ztunnel support into Cilium, we are embracing the "best of both worlds." We combine the top-tier performance of ztunnel for TCP and native mTLS with Cilium’s robust policy control and the flexibility of WireGuard/IPsec for non-TCP traffic and hitless upgrades.

This integration is the result of a healthy feedback loop where great ideas are shared, refined, and reimagined. At Isovalent and Microsoft, we are proud to be part of a community where learning goes both ways. By converging the innovations of Cilium, Istio, and SPIRE, we ensure that the ecosystem remains modular, resilient, and focused on the user.

Because when open source wins, _users win_.
###### Join Us at KubeCon Europe 2026

Come see native mTLS in action at [CiliumCon](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/co-located-events/ciliumcon/) and visit the Cilium project booth in the Project Pavilion throughout KubeCon + CloudNativeCon Europe 2026 in Amsterdam. Our team will be demonstrating the feature live and answering questions.

###### Getting Involved

- [Cilium Documentation](https://docs.cilium.io/en/stable/)
- [Cilium GitHub](https://github.com/cilium/cilium)
- [Join the Cilium Slack](https://slack.cilium.io/)
- [Cilium Community](https://cilium.io/get-involved/)
- [CiliumCon Europe 2026](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/co-located-events/ciliumcon/)

---

*Native mTLS for Cilium — transparent, inline encryption with SPIRE-backed SPIFFE identity. No sidecars. No dropped packets. No compromises.*

#### Celebrating 10 Years of Cilium: What’s New and What’s Next

URL: https://cilium.io/blog/2026/03/23/2026-03-23-ciliumcon-momentum
Date: 2026-03-23
Categories: Community

**_Author: Katie Meinders, Isovalent_**

Cilium is officially 10 years old and has firmly established itself as the default CNI for production cloud native deployments. It has been adopted by all of the major cloud providers, and, driven by customer demand, many other clouds and Kubernetes distributions are investing heavily in Cilium as their CNI of choice. According to the recent State of Kubernetes Networking report, Cilium represents over 60% of CNI deployments, more than double the next alternative.

This stability is also a reflection of the community behind the technology. Cilium is now made up of a growing community of over 1,010 developers, and the project now has over 24,000 GitHub stars. Annual development activity has grown 55x since year one, with nearly 10,000 PRs contributed in 2025 alone.

Looking ahead to the next decade, Cilium is already established as the networking data plane for AI. Microsoft and Google use Cilium to run some of the largest AI training clusters in the world. Organizations like ESnet and TikTok are running Cilium in IPv6-only data centers with massive scale. Simultaneously, Tetragon is positioned to redefine the runtime security landscape while tools like Cluster Mesh and KubeVirt together will enable organizations to run both VMs and containers on a consistent networking plane.

The Cilium community is coming together this week during CiliumCon and KubeCon + CloudNativeCon in Amsterdam to celebrate 10 years of the project and to align the roadmap with the workload demands of the next decade. Some of the recent project developments include the release of Cilium 1.19, a complete guide to the project with Cilium Up and Running, a new Children’s book, and new production case studies from OVHcloud and Zynga.

##### Cilium 1.19

Released in February, Cilium 1.19 continues the project's momentum with improvements across networking, policy, and observability.

- **ztunnel integration for mTLS:** Enroll namespaces into ztunnel for transparent Layer 4 mutual TLS (mTLS) pod to pod traffic, without requiring application changes. [Read more about the significance of this new feature and the work that has gone into it.](https://cilium.io/blog/2026/03/23/native-mtls-cilium/)
- **Expanded IPv6 support.** IPv6 support was added to both Cilium's L2 service advertising and tunnel networking, removing limitations for teams running IPv6-first or dual stack environments.
- **Multi-Pool IPAM reaches stable.** Multi-Pool IPAM graduated from beta to stable and got a meaningful upgrade. Teams can now assign pod IPs based on workload identity, enforce strict pool matching, and preserve real source IPs for routed traffic, giving operators both flexibility and security without tradeoffs.
- **Smarter DNS network policies.** A new wildcard prefix makes it easier to write DNS-based egress policies that cover full subdomain hierarchies in a single rule, reducing complexity and the risk of overly broad access permissions.
- **Hubble flow log aggregation.** Hubble can now aggregate flow logs before export, grouping traffic by namespace, service, or verdict over a set interval. For busy clusters, this dramatically reduces log volume while keeping the context needed for monitoring and analytics.

##### Cilium Up and Running

[Cilium: Up and Running](https://isovalent.com/blog/post/cilium-up-and-running/), the latest book in O’Reilly’s “ Up and Running” series is now available.

The book is a complete guide to Cilium and covers all angles: use cases, configuration, and gotchas plus practical tutorials and an accompanying lab. It is for anyone with an interest in Kubernetes networking, whether just starting out or looking to add features in production workloads.

Nico Vibert, Filip Nicolic, and James Laverack spent more than a year writing, testing, refining the content based on hands-on experience and real world Cilium user feedback.

The book is available for purchase, a digital copy can be downloaded for free from Isovalent’s website, or signed copies will be available this week from Isovalent’s booth #730.

##### Cilium Children’s Book

Following the release of Buzzing Across Space: The Illustrated Children’s Guide to eBPF comes episode two, Buzzing Beyond Clouds: The Illustrated Children’s Guide to Cilium. The story follows Obee and L4LB as they seek to connect and protect the cloud native galaxy.

To avoid the lure of the Dark Side, education for all, and particularly for children, is essential. This new story will take readers on a trip to a galaxy far, far away, and if you’re not already familiar with Cilium, you may learn a thing or two about the project, as well.

The book will be available as a digital copy on the Cilium website, and signed copies will be available this week from Isovalent’s booth #730.

##### Cilium in Production: new case studies: Zynga and OVHcloud

Cilium now has nearly 100 case studies and production user stories, and more than 170 public adopters, with use cases growing increasingly sophisticated. This month, Cilium has published case studies with [OVHcloud](https://www.cncf.io/case-studies/ovhcloud/) and [Zynga](https://www.cncf.io/case-studies/zynga/).

OVHcloud adopted Cilium as the default CNI in its Managed Kubernetes Service, driven by customer demand and the need for a future-ready networking layer. Cilium's eBPF-based architecture reduces load on management clusters serving thousands of tenants, while advanced network policies enforce strict tenant separation at scale. "Many users want the ability to choose their CNI during cluster creation, and Cilium is the clear leader in this space,” said Joël Le Corre, Cloud Architect, Containers & Orchestration at OVHcloud.

At Zynga, the team migrated to Cilium to solve scalability challenges and consolidate their networking stack, replacing AWS VPC CNI, kube-proxy, and Istio with a single platform. By replacing kube-proxy with eBPF-based load balancing, conntrack entries dropped from around 300,000 per node, or 5x the default Linux limit, to effectively zero. "Once we switched over to Cilium, connection limits were no longer a concern. That issue is gone completely."

Both organizations already have new Cilium features on their roadmap. OVHcloud is exploring Cluster Mesh for multi-region connectivity, while Zynga is rolling out Gateway API and evaluating L7 network policies for more granular application-layer traffic control.

##### Join Us in Amsterdam

Ten years in, Cilium is not just keeping pace with the demands of cloud native infrastructure — it is helping define what comes next. From AI training clusters at hyperscale to IPv6-only data centers, from a growing library of production case studies to a new O'Reilly book and a children's book, the project is as active and community-driven as ever.

If you're at [KubeCon + CloudNativeCon](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/) this week, come celebrate with us. Find the Cilium booth in the Project Pavillion, join the conversations at CiliumCon, and pick up a signed copy of Cilium: Up and Running and Buzzing Beyond Clouds at Isovalent’s booth #730. Here's to the next decade.

#### Cilium at Cloud Native Days France 2026

URL: https://cilium.io/blog/2026/02/25/cloud-native-days-france-2026-cilium
Date: 2026-02-25
Categories: Community

Two weeks ago, I attended **[Cloud Native Days France 2026](https://www.cloudnativedays.fr/)** for the very first time and it felt like stepping right into the heart of the French (and European) cloud native ecosystem.

More than **1,700 people**, **40 partners**, and **50+ talks** gathered at the CENTQUATRE-PARIS for a full day dedicated to Cloud Native, Kubernetes, DevOps, Open Source, and Platform Engineering.

Exactly the kind of conference where you want to be when you work on open source infrastructure.

##### An opening keynote that set the bar very high

The day kicked off with an amazing keynote hosted by Jérôme Petazzoni, co-creator of Docker and one of the early voices who helped shape the modern DevOps movement. For many of us, his work influenced how we think about containers and infrastructure today. He's also an excellent speaker, sharp, funny, and incredibly clear.

Then came **Ricardo Rocha**, Chair of the CNCF Technical Advisory Board and Head of Infrastructure at CERN, with a mind-blowing talk:

**10 PB/s without exploding the budget, how CERN scales its infrastructure with cloud native and AI.**

With the next upgrade of the LHC (Large Hadron Collider), the massive particle accelerator at CERN, the amount of data produced by experiments will increase tenfold. We're talking about detectors generating petabytes of data every second, but the computing budget isn't increasing. They need to process 10× more data, with the same money.

To make that possible, CERN had to rethink its entire infrastructure, intelligently sharing GPUs and accelerators, supporting both large-scale batch processing and interactive workflows (yes, even SSH), running real-time AI models directly on detectors 100 meters underground, and embedding sustainability and carbon awareness directly into the architecture.

YouTube link: [Keynote d'ouverture Cloud Native Days France 2026](https://www.youtube.com/watch?v=lJXUhqHWCDo&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=1&t=5s&pp=iAQB)

##### Cloud Native in the public sector

Another highlight was seeing open source used at the state level, with talks from DINUM (France's central digital government authority) and DGFIP (the national public finance administration) explaining how their inter-ministerial collaboration resulted in a shared, open source Kubernetes foundation powering government infrastructure.

Even more exciting was the announcement of closer collaboration with the CNCF, along with the intention to open-source parts of these projects. As a French engineer working in open source, that felt particularly meaningful, s because public investment, community, and long-term impact really align.

##### Cilium… everywhere

And yes **Cilium was everywhere**.

###### Multi-cluster networking, made real

The talk _"Le réseau multi-cluster, enfin simplifié"_ by **Arthur Outhenin-Chalandre** and **Quentin Swiech** was a perfect example of production-grade cloud native.

They gave a clear introduction to Cilium ClusterMesh, explained the architecture and configuration without hand-waving, and then… ran a live demo connecting two Kubernetes clusters, one in Paris, one in New York.

Seeing Hubble flows live, with packets moving across clusters and continents in real time demands huge respect for flawlessly pulling off live demos. Huge respect for pulling off live demos flawlessly.

YouTube Link: [REX Ledger - Le réseau multi-cluster, enfin simplifié: Guide pratique de Cilium ClusterMesh](https://www.youtube.com/watch?v=Zmakri4Kqyo&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=10&pp=iAQB0gcJCYcKAYcqIYzv)

###### Inside Cilium: Direct Server Return (DSR)

Another standout was _"Inside Cilium: Deep Dive sur DSR"_ by **Alexis La Goutte**.

In just 15 minutes, Alexis managed to demystify Direct Server Return, explain how eBPF and Cilium implement it efficiently, and visualize everything using Hubble and Wireshark.

Reducing latency, improving performance, and still keeping observability all explained clearly, keeping me awake even though it was directly after lunch.

YouTube Link: [Inside Cilium: Deep Dive sur DSR (Direct Server Return)](https://www.youtube.com/watch?v=1jAhnw18dn0&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=17&pp=iAQB)

###### Training at scale: 100,000 cloud native networking & security lab sessions

There was also a great session from my colleague Raphaël Pinson, "Training at scale: 100,000 cloud native networking & security lab sessions," where he shared how they built an interactive lab platform around Cilium, Tetragon, and Kubernetes more broadly. Using Kind, Containerlab, GitOps workflows, and distribution via Instruqt, these labs have already powered more than 100,000 hands-on sessions.

Do you want to test the labs? [https://isovalent.com/labs/](https://isovalent.com/labs/)

I've personally gone through several of these labs, and I genuinely love the approach: a ready-to-use, reproducible, ephemeral environment where you learn by doing. For me, that's the best way to learn, experiment, break things, observe what happens, and try again.

These labs help onboard engineers, support demos, validate new features, and are directly useful for customers who want a safe, hands-on environment to explore Cilium and Tetragon on their own. They also create feedback loops that continuously improve both the content and the product itself.

YouTube Link: [Formation à l'échelle: 100 000 sessions de labs réseau & sécurité cloud-native](https://www.youtube.com/watch?v=n_g60hLXZOk&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=32)

##### Hallway track > everything else

Beyond the talks, the real magic happened in the corridors. I randomly met several people running Cilium in production (always a good sign), including [**Noe Charmet**](https://www.linkedin.com/in/charmet/), co-founder and CTO at [Shipfox](https://www.shipfox.io/), who shared how they operate highly optimized GitHub runners at scale. We discussed some of their current challenges and explored whether Tetragon could help, in particular how eBPF-based probes might bring deeper runtime visibility and stronger security signals with minimal overhead (and yes, their booth also had some very cool goodies, which never hurts to break the ice).

I also crossed paths with [**Edgar Oblette**](https://www.linkedin.com/in/eoblette/), who benchmarked Cilium against Istio in terms of CPU usage and power consumption during his internship at Orange.

##### More great sessions (and too many to see all)

Like every good conference, Cloud Native Days FR forced some tough choices, with several strong talks running in parallel.

- **La famille k0s — simplifier le multi-cluster Kubernetes**, by **Luc Juggery (Exoscale)**, presented the k0s ecosystem and how its lightweight, pragmatic tooling helps reduce operational complexity when managing multiple Kubernetes clusters.

YouTube Link: [La famille k0s: une boîte à outils pour simplifier la gestion multi-cluster de Kubernetes](https://www.youtube.com/watch?v=Edo3e7KFqN8&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=14)

- **Du chaos à la clarté — VictoriaLogs & LogsQL**, by **Diana Todea (VictoriaMetrics)**, offered a clear and concrete look at building efficient, cost-effective logging pipelines using LogsQL, with strong integration into OpenTelemetry and Grafana.

YouTube Link: [Du chaos à la clarté: booster votre pipeline de logs avec VictoriaLogs et LogsQL](https://www.youtube.com/watch?v=la8coUmqriQ&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=6)

- **Questions pour un conteneur – édition Supply Chain**, by **Aurélie Vache** and **Shérine Khoury**, used an interactive format to make software supply chain security approachable while staying grounded in real-world concerns.

YouTube Link: [Questions pour un conteneur - édition Supply Chain](https://www.youtube.com/watch?v=FShI5Ot0Uxk&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=9)

- **Passons au cloud-native durable**, by **Henrik Rexed**, connected Kubernetes efficiency, AI workloads, and environmental impact, highlighting CNCF projects like Kepler, KubeGreen, and vCluster as concrete paths toward greener platforms.

YouTube Link: [Passons au cloud-native durable](https://www.youtube.com/watch?v=ZNo90IMUQyw&list=PLmZ3gFl2Aqt_4_F40zEoUEweEctmtx4p2&index=37&pp=iAQB)

Together, these sessions reinforced the key theme of the day. Modern cloud native systems must balance performance, cost, security, and sustainability, all at once.

##### Communities at the core!

Cloud Native Days FR is a community-driven, francophone event with a strong focus on open source and concrete feedback from the field. Developers, SREs, platform engineers, architects, but also CTOs and decision-makers all shared the same space and same conversations.
That diversity really showed throughout the day: some talks were deeply technical, others more human or organizational. But all of them shared one thing, a desire to build sustainable, understandable, and production-grade cloud native systems.

The density of communities represented and the ease with which conversations started. Throughout the day, I crossed paths with members of [**Merge Forward**](https://community.cncf.io/merge-forward/), [**TechNovations**](https://technovationchallenge.org/fr/) (formerly Women Techmakers), [MongoDB user groups](https://www.meetup.com/pro/mongodb-user-groups/), including [**Stéphane**](https://www.linkedin.com/in/sestegra/), [**Ahlem**](https://www.linkedin.com/in/ahlem-jaouadi/), and [**Abdul**](https://www.linkedin.com/in/abed-attal/), with whom I had great conversations, as well as CNCF community contributors, all sharing the same curiosity and willingness to learn from each other.

These informal discussions, sometimes between two talks, sometimes at dedicated booths or while waiting for swag, were a strong reminder that beyond technology, the cloud native ecosystem is above all a network of people building, sharing, and growing together.

And the momentum doesn't stop there: the next major cloud native event in France will be [**Kubernetes Community Days Provence**](https://cloudnative-provence.fr/), taking place on **December 10th**. I am very excited to join the organizing team, so stay tuned, more to come very soon

And a huge shout-out to the volunteers who made the event run so smoothly. Conferences like this don't happen by magic.

#### Migrating from Falco to Tetragon: A Guide for Transitioning Your Runtime Security Stack

URL: https://cilium.io/blog/2026/01/19/tetragon-falco-migrate
Date: 2026-01-19
Categories: Community

This guide helps current Falco users transition to Tetragon. It outlines key differences, explains how Falco constructs map to Tetragon, and addresses features without direct equivalents by suggesting alternative approaches. You’ll also learn how to integrate Tetragon into your runtime security stack. Whether you are migrating from Falco to Tetragon or considering it, this guide provides the background and technical details needed to make the transition confidently.

##### How Tetragon Compares to Falco

Falco is primarily a visibility engine. It uses various event sources, including kernel modules, eBPF probes, and external plugins, to evaluate event streams against security rules and alert on suspicious activity. However, remediation actions such as killing a process or blocking a network connection must be handled externally. Teams often need to intervene manually, build automation pipelines, or integrate Falco with other tools for threat remediation. Tetragon takes a quite different approach. It offers deep kernel-level visibility and enforcement across the full spectrum of eBPF capabilities for observability, tracing, and security. These enforcement capabilities include killing offending processes, sending signals, overriding kernel function return values, and more. Tetragon security policies allow for a declarative description of what should be observed, what the system is allowed to do, and the corresponding enforcement actions when a security policy is violated.

Another core difference is Falco’s exclusive reliance on syscall monitoring for visibility. Falco automatically selects a set of syscalls to trace using the union between a base set of syscalls needed for its internal state management (eg: open, close, clone…) and the minimum set of syscalls required by the ruleset (adaptive syscall selection). While effective for many use cases, syscall-based monitoring can be susceptible to [TOCTOU (time-of-check/time-of-use) vulnerabilities](https://www.youtube.com/watch?v=yFl_ScKA300) where certain operations can occur between the syscall check and the actual kernel action, creating opportunities to evade monitoring.

Tetragon avoids this class of attacks by exposing the full spectrum of kernel primitives, including Linux Security Module (LSM) hooks, tracepoints, kprobes, and even uprobes. This lets users select observability and enforcement points that reflect the kernel's actual behavior. For example, file access policies can rely on LSM hooks rather than syscalls, ensuring decisions happen at the most authoritative point in the system.

This degree of flexibility that Tetragon offers also introduces a learning curve, and the level of depth requires users to consider which kernel hook points best match their security intent. Choosing the optimal hook point may require experimentation and familiarity with the kernel behavior for that specific security intent. But for users who care about deep kernel-level visibility and precise control, this is precisely the type of low-level control they seek.

Tetragon filters and aggregates events directly in the kernel using eBPF. In contrast, Falco sends raw events to userspace for rule evaluation. Tetragon only forwards “matched” events to userspace when they satisfy a policy, which significantly reduces the volume of data crossing the userspace/kernel boundary. As a result of this design, Tetragon is much less likely to drop events under high load.
Furthermore, Tetragon's performance advantage stems from its use of targeted kernel instrumentation points rather than broad system call monitoring. While Falco must hook into the entry/exit of every syscall to maintain visibility, Tetragon hooks directly into the specific kernel instrumentation point relevant to your policy. In real-world scenarios, this precision results in a much lower performance overhead.

There are additional advantages that Tetragon offers over Falco that this migration-focused guide cannot cover in detail. But in a nutshell, Tetragon is designed with a very lean performance footprint at scale. It can also observe operating system integrity, including kernel module activities and interactions with the eBPF subsystem. Because this visibility and enforcement operate at the kernel layer, Tetragon can even monitor the activities of other security tools, including Falco itself. Finally, Tetragon is cloud native, in a sense that it is able to react to Kubernetes state changes and relies upon native Kubernetes resources; Falco needs the recent falco-operator to have Kubernetes awareness.

##### Migrating Falco Rules to Tetragon Tracing Policies

The rough equivalent of Falco rules in the Tetragon context is Tetragon tracing policies. Falco rules express conditions on syscall events or plugin-generated data; on the other hand, Tetragon tracing policies define which kernel hook points (kprobes, tracepoints, uprobes, LSM BPF) to attach to, which fields to extract from these hook points, and which corresponding enforcement or visibility actions to take.

However, not all Falco rules have a direct mapping to tracing policies. Some Falco rules depend on event sources that do not originate from the kernel and therefore cannot be expressed as Tetragon tracing policies. Although Tetragon ingests Kubernetes workload metadata and container runtime information to provide context for kernel events, it does not ingest Kubernetes audit logs, user-driven control-plane activity, cloud provider logs, etc. This means rules that rely on non-kernel event sources cannot be migrated into tracing policies. Some examples of Falco event sources that cannot be translated into Tetragon tracing policies include: Kubernetes Audit Events (API server audit logs, user actions, RBAC changes), AWS CloudTrail logs, Okta identity and authentication events, and most Falco external plugin events.

Tetragon is focused on kernel-level visibility and enforcement. As a result, rules based on these external sources should be fed into your SIEM or log analytics pipeline, where they can be correlated with Tetragon’s kernel and workload events to provide end-to-end context.

For Falco rules that rely on behaviors that are visible to the kernel, such as process execution, file access, privilege changes, networking activity, or syscall anomalies, you can translate them into their equivalent Tetragon tracing policies by selecting the appropriate kernel hook (LSM hooks, kprobes, tracepoints, etc.) and expressing conditions and enforcement logic there.

###### Rewriting Falco Rules as Tetragon Tracing Policies

Falco Tetragon by default without any tracing policy loaded tracks all executions across Kubernetes clusters, virtual machines, and bare metal systems. This creates the foundation that allows Tetragon to attribute all system behavior back to a specific binary and its associated metadata (container, pod, node, and cluster).

NB: New users are often unclear about the scope of Tetragon’s default execution monitoring. Although Tetragon offers extensive visibility by default, it does not cover all scenarios. Many security intents require a specific Tracing Policy to be deployed. This confusion occurs because the default process execution monitoring already covers several important use cases, such as sudo invocations, SUID binary execution, and fileless execution, without extra configuration. For so many other security use cases, users must deploy a tracing policy. To support this, the Tetragon community offers a [vetted library of tracing policies](https://tetragon.io/docs/policy-library/observability/), which is an excellent starting point for users to explore writing tracing policies.
Consider this Falco rule below and see how we translate this into the equivalent Tetragon tracing policy. This Falco rule monitors attempts to access the /etc/shadow directory.

```yaml
- rule: Unauthorized Access to Sensitive Files
  desc: Detects attempts to access /etc/shadow by unauthorized users
  condition: >
    evt.type in (open, openat) and
    fd.name = "/etc/shadow" and
    not user.name = "root"
  output: >
    Unauthorized file access detected
    (user=%user.name file=%fd.name command=%proc.cmdline container=%container.id)
  priority: WARNING
  tags: [filesystem, security]
```

There are multiple ways to go about writing the equivalent Tetragon tracing policy. For file access related tracing policies, LSM BPF is the recommended hook point because they always represent real enforcement points.

```yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: 'block-sensitive-file-access'
spec:
  kprobes:
    - call: 'security_file_permission'
      syscall: false
      args:
        - index: 0
          type: 'file' # The file being accessed
      selectors:
        - matchArgs:
            - index: 0
              operator: 'Equal'
              values:
                - '/etc/shadow'
          matchActions:
            - action: Sigkill # Kill the process immediately
```

Tetragon tracing policies focus on behaviors rather than just events. The first step is to decide which behaviors to monitor, such as process execution or file access, and to define the security intent. Policies are then built around the basic operations that best show these behaviors.

###### Migrating User-Defined Syscalls

Falco users often customize the syscalls monitored by Falco to optimize performance or focus on specific behaviors. This is typically achieved through Falco's adaptive syscall selection or user-defined syscall sets. During a migration, any syscall-based monitoring objectives must be mapped into their corresponding Tetragon tracing policies.

However, you will quickly discover that syscalls are not always the ideal hook point when crafting tracing policies. Syscall monitoring can be vulnerable to TOCTOU (time-of-check/time-of-use) issues and may miss crucial kernel-level state transitions. For example, when monitoring file access, relying solely on syscalls can create blind spots. In Tetragon, LSM (Linux Security Module) BPF hooks generally provide a more authoritative, real-time view of the action and are typically the better choice for both monitoring and enforcement.

Thus, migration is not always a 1:1 syscall translation; sometimes the optimal Tetragon hook point that best addresses your security intent is not a syscall at all, but could be a tracepoint or a kprobes for a specific kernel function. This blog post, titled [“Linux Kernel Fundamentals for Effectively Writing Tetragon Tracing Policies,”](https://cilium.io/blog/2025/09/16/kernel-basics-for-tetragon/) covers some prerequisite Linux fundamentals that can enable effectively writing good tracing policies.

###### Rebuilding Your Alert Pipeline When Migrating From Falco to Tetragon

Falco users rely on Falcosidekick as a central component of their alert forwarding pipeline. Falcosidekick acts as an HTTP proxy for Falco events and provides a unified way to fan out alerts to chat platforms, storage services, SIEMs, incident-response tools, log stores, alerting systems, streaming systems, and more. It supports dozens of integrations out of the box.

When migrating to Tetragon, it is crucial to understand that there is no direct equivalent of Falcosidekick in the open source Tetragon ecosystem. Similar to Falco outputs, Tetragon emits events using a gRPC endpoint or JSON logs. It does not provide a built-in multi-destination alert routing layer like Falcosidekick. This layer can be achieved with minimal effort using a variety of tools in the CNCF open source ecosystem. With minimal effort, Tetragon can be integrated with SIEMs, incident response tools, log stores, and more.

###### Conclusion

Beyond syscall monitoring, Tetragon extends runtime security to include deep kernel-level visibility and real-time enforcement, leveraging the full capabilities of eBPF. Successfully migrating from Falco to Tetragon requires a rethink of how security intent is expressed, choosing the appropriate kernel hook points for each security intent rather than relying exclusively on syscalls. Some Falco use cases, especially those derived from non-kernel event sources, will remain outside the scope of Tetragon and should instead be handled by SIEMs or log analytics platforms, where they can be correlated for full context.

Ultimately, what Tetragon offers is a powerful and authoritative foundation for runtime security, especially for teams that value enforcement, flexibility, performance, and kernel-level visibility. By understanding these differences and approaching migration as an opportunity to modernize your security architecture, you can transition from Falco to Tetragon with confidence and build a resilient runtime security stack that caters to the ever evolving security threat landscape.

##### Additional Resources

- [Linux Kernel Fundamentals for Effectively Writing Tetragon Tracing Policies](https://cilium.io/blog/2025/09/16/kernel-basics-for-tetragon/)
- [Securing the Modern Process with Tetragon: Runtime Security for the Cloud-Native Kernel](https://cilium.io/blog/2025/11/4/process-tetragon/)
 [Tetragon Enterprise Distributions & Training](https://tetragon.io/enterprise/)

#### Tetragon: An End-of-Year 2025 Reflection

URL: https://cilium.io/blog/2025/12/31/tetragon-2025-review
Date: 2025-12-31
Categories: Community

As we close out another incredible year, we want to pause and say thank you to the entire Tetragon community. From contributors and maintainers to users, speakers, and the entire community, this year was shaped by your energy, curiosity, and commitment to advancing the standard of runtime security.

Tetragon has continued to grow because of the amazing people behind it: those writing code, testing edge cases, sharing feedback, publishing blogs, and teaching others what’s possible with eBPF-powered security observability and runtime enforcement. Together, we made meaningful progress both technically and as a community, laying a strong foundation for the future.

Below are some of the highlights that defined this year.

##### Major Feature Highlights

###### Tetragon for Windows (Preview)

This year marked an important milestone with the ongoing work to bring Tetragon’s security observability and runtime enforcement to the Windows ecosystem. The current preview already supports `process_exec_` and `process_exit_` events on Windows, opening the door to consistent, cross-platform visibility. While there’s more work ahead, this is a significant step toward expanding Tetragon’s reach beyond Linux.

###### Persistent Enforcement

First introduced in the Tetragon 1.12 release in late 2024, persistent enforcement remains one of the most impactful features. With persistent enforcement, policies continue running even if the Tetragon agent goes down, closing a critical security gap during agent restarts or failures. To enable persistent enforcement, configure Tetragon with `--keep-sensors-on-exit`. This ensures that when the Tetragon process exits, enforcements policies stays active because they’re pinned in sysfs bpf tree under the `/sys/fs/bpf/tetragon` directory.

###### Flexible Enforcement Modes for Tracing Policies

This year also brought greater flexibility with the addition of enforcement modes. Tracing policies can now run in: Monitoring mode, where enforcement actions are ignored, or Enforcement mode, where policies are actively enforced. This makes it easier to test policies safely, validate behavior, and avoid unintended disruptions. Enforcement modes can be configured directly in the policy, at load time, or dynamically at runtime via gRPC.

###### Attribute Resolution

Attribute resolution significantly improved the experience of writing tracing policies. It allows policy authors to dynamically extract specific attributes from kernel structures passed into kprobes and LSM hooks making policies more expressive and easier to reason about. For example, the policy below demonstrates extracting a parent process’s comm during binary execution:

```yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: 'lsm'
spec:
  lsmhooks:
    - hook: 'bprm_check_security'
      args:
        - index: 0 # struct linux_binprm *bprm
          type: 'string'
          resolve: 'mm.owner.real_parent.comm'
      selectors:
        - matchActions:
            - action: Post
```

Using attribute resolution still requires you to read the kernel structure definitions to find what you’re looking for in the hook parameter attributes.

###### Userspace Hooks (Uprobes & USDTs)

A major portion of userspace tracing support landed this year with uprobes and USDTs. These hooks allow Tetragon to attach directly to userspace functions and statically defined tracepoints, extending the same observability and enforcement model beyond the kernel. With userspace hooks, Tetragon now delivers deeper visibility into application behavior—bringing kernel-grade security insights to user-space programs.

###### Community Voices

One quote from a security engineer at a large social networking company captured this sentiment perfectly:

_“The best situation is when other services and teams don't even realize that a security tool is running. That’s when you know you’ve done your job well, and that’s what Tetragon gives us. Tetragon had the best performance among all the tools we tested.”_

###### Community Talks & Learning

The Tetragon community showed up in a big way across conferences, webinars, podcasts, and tutorials. In no particular order, highlights included:

- [Tetragon and eBPF for Network Engineers](https://www.youtube.com/watch?v=NZ2il-rZTUc)
- [eCHO Episode 196: Exploring Tetragon on Linux Servers](https://www.youtube.com/watch?v=Sjew34d7i1w)
- [Security Superpowers With eBPF and Tetragon - Liz Rice, Isovalent at Cisco](https://www.youtube.com/watch?v=u4lSpHSVZOw)
- [Agentic Bee: How To Get AI Agents To Talk To Tetragon](https://www.youtube.com/watch?v=YXl9xuIxylY)
- [Master Linux Security with Tetragon and Wazuh: Real-Time Monitoring and Seamless SIEM Integration](https://www.youtube.com/watch?v=Mfnrp3X1E-k)
- [Tetragon detects io_uring file access](https://www.youtube.com/watch?v=ujZnwkC08Hk&t=8s)

###### User Stories & Blog Posts

This year also brought a steady stream of thoughtful user stories and technical deep dives, including:

- [Tetragon User Story: Why This Social Networking Company Made Tetragon a Default for their Kubernetes Clusters](https://cilium.io/blog/2025/04/15/tetragon-social-networking-user-story/)
- [Sinad User Story: Delivering Security and Observability for Workloads with Confidence](https://cilium.io/blog/2025/04/15/tetragon-user-sinad/)
- [Securing the Modern Process with Tetragon: Runtime Security for the Cloud-Native Kernel](https://cilium.io/blog/2025/11/4/process-tetragon/)
- [Linux Kernel Fundamentals for Effectively Writing Tetragon Tracing Policies](https://cilium.io/blog/2025/09/16/kernel-basics-for-tetragon/)
- [What is Runtime Security?](https://isovalent.com/blog/post/what-is-runtime-security/)
- [Why Tetragon Should Be Standard in Every Kubernetes Cluster: The Missing Runtime Security Layer](https://isovalent.com/blog/post/tetragon-security-standard/)

Each story reinforced a common theme: Tetragon is the standard for runtime security

###### Looking Ahead

As we look toward the next year, we’re excited to keep building with you. More platforms, deeper visibility, stronger enforcement, and continued focus on performance and usability still remain the priority. Most importantly, we’re excited to continue growing a community that believes runtime security should be powerful, practical, and open.
Thank you for an amazing year of collaboration, learning, and progress. We can’t wait to see what 2026 holds for the Tetragon community.

Here’s to another year doing great things with eBPF and Tetragon.

See you in 2026!

For the Tetragon Community:

#### Contributing to Cilium Through the LFX Mentorship Program

URL: https://cilium.io/blog/2025/11/29/peacesandy-cilium-internship-experience
Date: 2025-11-29
Categories: Community

The past three months have been both educational and transformative, as I have had the opportunity to contribute to Cilium through the LFX Mentorship Program. This journey brought growth, hands-on learning, and meaningful collaboration with my mentor and the broader open-source community. Working closely with the my mentor helped me expand my technical skills while experiencing the true spirit of open-source contribution. In this article, I share my whole experience in the LFX Mentorship Program and my work on evaluating and improving Cilium’s SEO, AEO, and AIO.

##### What is the LFX Mentorship Program?

The LFX Mentorship Program is an initiative by The Linux Foundation designed to encourage global participation in open-source development. Similar to programs like Google Summer of Code, it offers contributors the opportunity to work on real open-source projects under the guidance of experienced project maintainers for 3 months. It provides hands-on experience, mentorship, and a pathway to becoming an active member of the open-source community.

##### About Cilium

Cilium is an open-source, cloud-native networking solution that provides secure, observable, and high-performance connectivity between workloads, powered by eBPF, a revolutionary Linux kernel technology. It enables advanced networking, load balancing, and security capabilities without requiring any changes to applications.

Trusted by organizations running large-scale, modern infrastructure, Cilium is a core building block of cloud-native platforms. Contributing to Cilium gave me valuable exposure to real-world challenges in networking, cloud infrastructure, and open-source development. I first discovered Cilium as an exciting open-source project, applied for the mentorship, and was thrilled and immensely grateful to be selected.

##### My Internship Experience

At the start of my internship, I spent time gaining a deep understanding of the project by reading the codebase, learning its architecture, and identifying key pain points. My mentor, Bill Mulligan, played a considerable role in shaping my experience. He set up weekly meetings and stayed consistently available through one-on-one Slack conversations to support my progress.

From our first introductory call, I knew this internship would offer immense learning opportunities, and it exceeded my expectations. My mentor is incredibly supportive, which makes the experience rewarding. Along the way, I learned a lot about eBPF, cloud-native, and Kubernetes, which have become very valuable to me.

Over the course of the internship, I sharpened my technical skills, improved my understanding of cloud-native networking, and gained confidence working on a widely adopted open-source project. I feel truly fulfilled contributing to Cilium, and I’m proud of the impact I made.

Cilium is an exciting and welcoming project for anyone interested in open-source, and I’m grateful to have been part of it.

##### My Contributions

Once my internship officially began, we aligned on a clear, ambitious goal: improve Cilium's SEO, AEO, and AIO so that both search engines and modern AI systems could better understand, surface, and recommend Cilium’s content. To stay aligned throughout the project, my mentor set up consistent weekly check-ins and a Slack communication channel. These regular touchpoints helped us maintain momentum, review progress, and rapidly iterate on improvements as insights emerged.

We approached the project systematically. I started by addressing foundational issues on the site, fixing broken links, cleaning up metadata, and enhancing the SEO component to support richer structured data across the Cilium website. From there, I moved on to updating metadata for key pages, refining blog titles and descriptions, and ensuring every page had a canonical URL to prevent duplicate content issues. I also reorganized the heading hierarchy across multiple sections to make them more transparent for both humans and search engines. I refactored the blog tagging system to highlight only the most frequently used tags.

After improving the site's overall health, I shifted my focus to more advanced optimizations. I updated article-level metadata (author, publication date, and descriptions), added alt text for accessibility, and introduced standardized frontmatter for automation, especially on blog pages. I also improved user experience and search performance by adding TL;DR sections to key documentation pages and implementing FAQ schemas where appropriate. I wrote the FAQ and added a dedicated page for it on the website, improving its chances of appearing in featured snippets and AI-generated summaries.

On the AI optimization side, I reworked meta descriptions to be more keyword-specific and user-focused, reinforced a clean heading hierarchy, and added unique IDs to major sections for better deep linking. I added metadata blocks at the top of pages for better machine readability and implemented answer-first formatting, ensuring the core answer to a query appears early in the content. These changes were significant for AI search engines like Perplexity, Bing Copilot, and Google SGE, which rely heavily on structured clarity.

Although some pages didn’t require major restructuring, I still added custom anchor IDs and deep links so AI assistants could accurately reference key sections. Throughout this process, every update was tested, reviewed, and refined through our weekly check-ins, ensuring that improvements aligned with Cilium’s broader documentation and website goals.

To close out the project, I established baseline performance metrics in Google Search Console and keyword ranking tools. After the updates went live, I tracked whether Cilium began appearing in rich results such as FAQs, snippets, and AI-generated answer panels. The early results were positive, showing increased visibility, improved metadata usage, and better alignment with modern search and AI engines.

This internship was a uniquely rewarding experience. It allowed me to work at the intersection of technical writing, SEO strategy, AI search-readiness, and documentation architecture, while contributing meaningfully to a major open-source project. I’m proud of the improvements I made, and even more excited to see how Cilium continues to grow in visibility and accessibility across both traditional search engines and the emerging landscape of AI-driven discovery.

##### Impact

The improvements made during this internship significantly strengthened the discoverability, accessibility, and overall clarity of Cilium’s website. By enhancing SEO, implementing structured metadata, refining heading hierarchies, and optimizing content for AI-driven search, cilium.io is now better positioned to appear in traditional search results and emerging AI-generated answer platforms.

These updates help users, whether new contributors, engineers, or organizations evaluating Cilium, find accurate, well-structured information faster. The implementation of FAQ schema, improved meta descriptions, and TL;DR sections also ensures that key concepts are more easily understood and more likely to surface in featured snippets, rich results, and AI chat interfaces.

Beyond immediate gains, the changes introduced more consistency across the website and established a scalable foundation that future contributors and maintainers can build on. By making Cilium’s content more machine-readable and search-friendly, this project supports long-term visibility, better onboarding for newcomers, and stronger engagement across the broader open-source ecosystem.

##### Summary

I am deeply grateful to everyone who made this internship experience meaningful. To my mentor, Bill Mulligan, thank you for believing in me and guiding me with patience, clarity, and kindness. Your feedback, your time, and your willingness to review every PR and answer every question made this journey so much smoother. And to Paul Arah, thank you for your thoughtful reviews and comments; they strengthened my work in ways I truly appreciate.

A special thank you to my friends, Oluchi Nwenyi and Victoria Nduka, LFX Intern alumni, for always being there to answer my endless questions and for offering invaluable support throughout this internship.

I have genuinely enjoyed this experience and learned so much more than I expected. To anyone considering participating in the LFX mentorship: please apply. There is an incredible range of skills needed in open-source projects, and you don’t need to “fit perfectly” or feel like you already have the right experience. Don’t count yourself out. There is room for you to grow, contribute, and thrive.

The Cilium ecosystem will always hold a special place in my heart.

And finally, thank you to the Linux Foundation for creating the LFX Mentorship Program. It is truly a life-changing opportunity.

#### Securing the Modern Process with Tetragon: Runtime Security for the Cloud-Native Kernel

URL: https://cilium.io/blog/2025/11/4/process-tetragon
Date: 2025-11-04
Categories: Community

**_Author: Paul Arah, Isovalent@Cisco_**

The evolution into cloud native architectures fundamentally changed how we think about processes from a security perspective. Before cloud native environments, processes were relatively stable and long-lived. You could audit them, understand their behavior patterns, and apply security controls with reasonable confidence. Containers changed this model. Processes now spawn and die in milliseconds, run in isolated namespaces, share the same kernel with dozens of other workloads, and operate at a scale that makes manual security review next to impossible.

The need for runtime security has never been more important. Tetragon's eBPF-based security observability and runtime enforcement provides the visibility and control needed to secure processes in these dynamic environments. But to use it effectively, you need to understand both the Linux process model and how attackers exploit it in containerized systems.

##### The Linux Process Model in the Container Context

When you execute a container, you're creating processes that run on the same kernel as every other process on that node, just with different namespace and cgroup configurations. Every container process is still just a Linux process with a process ID, parent-child relationships, memory space, and file descriptors. The container runtime manages the creation of these processes with specific isolation primitives. When Kubernetes schedules a pod, it is instructing the container runtime to fork processes with carefully configured namespaces for PID, network, mount, UTS, and IPC isolation.

This model creates some interesting challenges from a security point of view. For example, a container escape involves manipulating these namespace boundaries or exploiting kernel vulnerabilities that affect all processes regardless of their namespace configuration. The shared kernel between processes becomes both a performance benefit and a security concern.

##### Process Namespaces and Security Implications

The process namespace is particularly relevant for security. Inside a container, processes see themselves starting from PID 1, but on the host, these processes have different PIDs in the root namespace. Monitoring tools that run inside containers only see the container's namespace view, missing the broader context of what's happening on the node; on the other hand, monitoring tools that lack namespace awareness miss the container context.

Because Tetragon operates at the kernel level, it has visibility across all namespaces, seeing both the container's and the host's views. This kernel-level visibility allows Tetragon to track process relationships even when they cross namespace boundaries. Being able to monitor processes across namespace boundaries is essential for detecting container escapes, where a process might spawn in one namespace and then attempt actions in another namespace.

##### Why the Shared Kernel Model Matters

Because all containers on a node share the kernel, a vulnerability in kernel code affects every workload. This is fundamentally different from virtual machines, where each VM runs its own kernel. The [Dirty Pipe vulnerability](https://nvd.nist.gov/vuln/detail/cve-2022-0847) exemplified this perfectly: an unprivileged process in any container could exploit the vulnerability to overwrite files across namespace boundaries, potentially gaining control over the entire node.

##### Understanding Linux Capabilities in Containers

Before diving into specific attacks, we need to understand Linux capabilities because they're central to how container security works. In the old world, processes were either root (UID 0) or standard users (!=UID 0). This binary system was too rigid. Sometimes unprivileged processes need specific privileges. The kernel introduced capabilities to provide more granular security controls. Capabilities can be implemented on files or processes.

Container runtimes use capabilities to implement security boundaries. A typical container runs with a restricted capability set compared to root. The default Docker capability set includes things like `CAP_NET_RAW` (raw sockets) and `CAP_CHOWN` (change file ownership), but drops dangerous capabilities like `CAP_SYS_ADMIN`.

However, privileged containers get all capabilities, including `CAP_SYS_ADMIN`, which is essentially the "new root" in Linux. This is why privileged containers are so dangerous. With `CAP_SYS_ADMIN`, you can load kernel modules, mount arbitrary filesystems, change namespaces, and perform numerous other operations that break container isolation.

###### The runc Vulnerability (CVE-2019-5736)

The Runc container escape [(CVE-2019-5736)](https://kubernetes.io/blog/2019/02/11/runc-and-cve-2019-5736/) remains one of the most instructive examples. The vulnerability allowed a malicious container to overwrite the runc binary on the host. When an administrator executed commands like docker exec, the compromised runc would execute with host privileges.

From a process perspective, this attack exploited how runc handles file descriptors. The malicious container would write to `/proc/self/exe`, a symbolic link pointing to the running process's binary. Because runc was the process executing into the container, this link pointed to the host's runc binary. The write operation could overwrite the actual runc binary on the host filesystem, bypassing container isolation entirely.

What makes this particularly deadly is that the attack leverages legitimate behaviour. Writing to `/proc/self/exe` isn't inherently suspicious in all contexts, but from a container it absolutely is. This is where context-aware monitoring becomes crucial. We can write tracing policies that can detect this pattern by monitoring file write operations to sensitive paths with awareness of whether the acting process is containerized:

```yaml
spec:
  kprobes:
    # https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#c.security_file_permission
    - call: 'security_file_permission'
      syscall: false
      args:
        - index: 0
          type: 'file'
        - index: 1
          type: 'int'
      selectors:
        - matchArgs:
            - index: 0
              operator: 'Equal'
              values:
                - '/usr/bin/runc' #adjust to your actual runc path
            # MAY_WRITE
            - index: 1
              operator: 'Mask'
              values:
                - '2'
          #Only suspicious when NOT in the host PID namespace
          matchNamespaces:
            - namespace: Pid
              operator: NotIn
              values:
                - 'host_ns'
          #kill the offending process
          matchActions:
            - action: Sigkill
```

This policy hooks into the kernel function that installs file descriptors, watching for attempts to open `/proc/self/exe` for writing. The enforcement action kills the process attempting this operation.

###### The Dirty Pipe vulnerability (CVE-2022-0847)

The Dirty Pipe vulnerability [(CVE-2022-0847)](https://nvd.nist.gov/vuln/detail/cve-2022-0847) demonstrates how kernel-level bugs enable privilege escalation regardless of container configuration. The vulnerability existed in the Linux kernel's pipe handling code, specifically in how the kernel handled copy-on-write semantics for pipe buffers.

The attack worked by creating a pipe, writing data to it, then splicing that data to overwrite arbitrary file contents, even read-only files. An attacker could overwrite setuid binaries or critical system files like `/etc/passwd` to gain root access.

What's particularly interesting from a monitoring perspective is the attack pattern. The exploit required specific sequences of system calls: creating pipes with `pipe()`, writing data with `write()`, and then using `splice()` to copy data into file page caches. While each individual system call is legitimate, the combination in this specific pattern is highly suspicious.

##### When Legitimate Tools Become Weapons

Modern attackers increasingly use legitimate system tools for malicious purposes, a technique called ["living off the land."](https://www.crowdstrike.com/en-us/cybersecurity-101/cyberattacks/living-off-the-land-attack/) Instead of dropping custom malware that could be easily detected, they chain together native utilities to achieve their objectives. A typical example: an attacker gains initial access through a web application vulnerability, uses curl to download a script, executes it with bash, uses find to locate sensitive files, compresses them with tar, and exfiltrates with nc or curl. Every binary involved is legitimate. Relying on signature-based detection would not suffice. This is where behavioral monitoring shines. The sequence of operations is suspicious, even though individual commands aren't.

A web server process spawning bash is unusual. That bash process immediately running curl to download from the internet is even more suspicious. The downloaded script executing a find for SSH keys and database credentials seals the deal.
Tetragon's child process visibility is crucial here. You can create policies that understand parent-child relationships. For example, you monitor unexpected shells launched from web-facing applications. To detect these behaviors, you can match on the parent binary and ask Tetragon to automatically follow all child processes it spawns.

```yaml
spec:
  # https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#c.security_bprm_check
  lsmhooks:
    - hook: 'bprm_check_security'
      args:
        - index: 0
          type: 'string'
          resolve: 'filename'
      selectors:
        - matchBinaries:
            - operator: 'In'
              values:
                - '/usr/sbin/nginx'
                - '/usr/bin/node'
                - '/usr/local/bin/python'
              followChildren: true
          matchArgs:
            - index: 0
              operator: 'In'
              values:
                - '/bin/bash'
                - '/bin/sh'
          matchActions:
            - action: Post
```

This sample tracing policy monitors execution starting from nginx, node, or python and follows the children they spawn. If any of those processes creates a shell, Tetragon detects this with full lineage visibility. An activity like this is a strong signal of remote code execution or “living off the land” activity.

##### Building Observability-Driven Prevention Policies

The most powerful of Tetragon is translating detection events into prevention policies. This is different from traditional security tools that rely on signature-based detection or predefined rule sets. Instead, you observe actual behavior of your workloads in your environment, understand what the expected behavior is, and finally create policies that block deviations from this expected behavior. There are generally two ways to go about this:

###### The Allowlist Approach: Least Privilege

Allowlist policies specify what actions applications are allowed to perform and block everything else. This is the ideal security posture: only grant the minimal capabilities and privileges an application needs. The challenge has always been determining what an application actually needs. Trial and error (remove capabilities until something breaks) is frustrating and risky. Static analysis of code doesn't account for runtime behavior. Observability solves this. Deploy your application with full monitoring, exercise all its functionality (including edge cases and error paths), and observe what capabilities it uses, what files it accesses, and what network connections it makes. This baseline behaviour becomes your allowlist policy.

For example, you might observe that your frontend application:

- Executes only node and npm binaries
- Accesses only files in /app and /tmp
- Makes network connections only to backend services on ports 8080 and 6379
- Runs with only CAP_NET_BIND_SERVICE capability

Your allowlist policy enforces exactly this behavior and blocks everything else. If an attacker compromises the application and tries to run bash, download additional tools, access /etc/shadow, or connect to external IPs, the policy blocks it.

###### The Denylist Approach: Known Bad Behavior

Denylists specify behaviors that should be blocked while allowing everything else. They're less secure than allowlists (there's more opportunity for attackers to maneuver), but easier to implement and less likely to break legitimate functionality. The key to effective denylists is learning from real attacks. Don't guess what malicious behavior looks like; observe it during security exercises, CTF challenges, or incident response. An observability-driven approach means your denylist is based on actual attacker behavior in your environment.

##### Practical Deployment Strategies

When deploying Tetragon in production, follow these practices to maximize effectiveness while minimizing operational risk.

- **Start Passive:** Deploy Tetragon in monitoring enforcement mode before enabling any enforcement. This establishes baselines and identifies legitimate behaviors that might otherwise be blocked.
- **Focus on Security Significant Events:** You don't need to monitor everything equally. For example, production namespaces running internet-facing services warrant more scrutiny than development environments. Workloads handling sensitive data need stricter policies than internal tools.
- **Use Kubernetes Identity Aware Policies:** Leverage namespaces and labels to create targeted policies. Your database pods have different legitimate behaviors than your frontend pods. Policies should reflect this.
- **Test Thoroughly**: Before enforcing policies in production, test in development and staging. Reproduce workload patterns and verify policies don't break legitimate operations. Treat security policies as code: version control, review, test, then deploy.
- **Iterate Continuously**: Applications evolve, new vulnerabilities emerge, and attack techniques change. Regularly review Tetragon events, identify new patterns, and update policies accordingly.
 Learn by Attacking: Participate in CTF challenges or run red team exercises with Tetragon monitoring active. This teaches both attack techniques and improves your detection capabilities.

##### Conclusion

Process security in Kubernetes requires understanding that containers are just Linux processes with namespace isolation, not virtualized systems. The shared kernel model that makes containers efficient also makes them vulnerable. A kernel exploit in any container affects all workloads on that node.

Traditional security tools struggle with this reality. They lack namespace awareness, miss ephemeral workloads, and can't correlate process behavior with Kubernetes identity. Static analysis catches known vulnerabilities but misses zero-days and misconfigurations. Network monitoring sees packets but loses context when pod IPs are reused.

Tetragon transforms process security by operating at the kernel level with full namespace awareness. Tetragon observes every process execution, file access, and network connection across all workloads while correlating the Kubernetes context. It correlates events through the process lifecycle, revealing attack patterns that individual events wouldn't show.
More importantly, Tetragon enables prevention through observability-driven policy. By monitoring normal application behavior, you can build least-privilege policies that block deviations from expected behavior without breaking legitimate functionality. By observing attacks during security exercises, you can create targeted defenses against real-world threats.

##### Additional Resources

- [Container Security, 2nd Edition](https://isovalent.com/books/container-security/)
- [eBPF for Creating Least Privileged Policies: What Do I Need to Know to Prepare for the Next CVEs?](https://www.youtube.com/watch?v=-sfOB1s6mvs&list=PLDg_GiBbAx-kvhwkGkCDdnQzTIeEdrJ_F)
- [Past, Present, Future of Tetragon- First Production Use Cases, Lessons Learnt, Where Are We Heading?](https://www.youtube.com/watch?v=2BIe4VmSYyQ&list=PLDg_GiBbAx-kvhwkGkCDdnQzTIeEdrJ_F&index=16)
- [Don't Get Blown up! Avoiding Configuration Gotchas for Tetragon Newbies](https://www.youtube.com/watch?v=YNDp7Id7Bbs&t=452s)
- [What Is a Standard Container: Diving Into the OCI Runtime. Spec Containers Aren't Linux Processes](https://iximiuz.com/en/posts/oci-containers/)

#### Linux Kernel Fundamentals for Effectively Writing Tetragon Tracing Policies 

URL: https://cilium.io/blog/2025/09/16/kernel-basics-for-tetragon
Date: 2025-09-16
Categories: Community

**_Author: Paul Arah, Isovalent@Cisco_**

When you write Tetragon tracing policies, you’re not writing arbitrary sets of rules; you're programming against the kernel execution path itself. Every policy you create hooks directly into kernel functions, intercepts system calls, and examines kernel data structures. This power comes with responsibility. Without an understanding of how the Linux kernel works, you'll find yourself writing policies that are ineffective, overly broad, or worse, missing the exact events you are trying to find.

This blog post is meant to be a pointer guide; it won’t make you a kernel hacker overnight, but it aims to cover some core Linux knowledge essential for crafting effective Tetragon tracing policies. We’ll connect kernel fundamentals such as user vs. kernel space, system calls, process structures, namespaces, and more to practical tracing policy examples. A basic familiarity with Linux and a base-level understanding of what Tetragon is will be enough to follow along.

##### Kernel Space vs User Space

One of the most fundamental concepts in Linux system programming is the distinction between kernel space and user space.

**User space** is where apps like Bash, Nginx, VS Code run. Userspace programs run with restricted privileges and cannot do things like access arbitrary memory locations, execute privileged instructions, directly control hardware, or access kernel data structures.

**Kernel space** is where kernel code runs with unrestricted access to all system resources. Here, code can access any memory location, execute privileged CPU instructions, directly control hardware devices, modify system-wide data structures, and more.

##### The System Call Interface

User programs can’t talk to the kernel directly. They use system calls(syscalls) like `openat`, `write`, or `execve`. System calls are the controlled entry points that allow user space programs to request kernel services. When a user space program wants to open a file, allocate memory, or create a network connection, it must go through the system call interface. For example, when a program calls `open("name.text")`, it isn’t the library function itself that touches the file. Behind the scenes, this becomes a `sys_openat` system call. The kernel then processes the request, checks permissions, and returns a reference that the program can use.

Most programming languages offer some sort of standard library that provides a high-level abstraction over the system call interface; this way, application developers typically never have to access the system call interface directly. When you write tracing policies, we’re working with the kernel’s perspective, where syscalls are the actual events being invoked.

##### Hook Points

When you write a tracing policy, you have to tell Tetragon where to look in the system. These attachment points are called hook points. There are two complementary perspectives we can view this from: how the Linux kernel itself defines them, and how Tetragon exposes them for us. Understanding both perspectives helps us choose the right hook point for the security observability scenario.

###### The Linux Kernel’s Point of View

In the kernel, a hook point is just a place in the execution flow where code can be instrumented. Different subsystems provide different mechanisms for instrumentation:

- **Kprobes** provide dynamic probes on almost any kernel function. They let you intercept functions like `fd_install()` whenever they’re called. Kprobes are powerful, but are tightly coupled with your kernel version since kernel functions can change across versions.
- **Tracepoints** are essentially built-in static markers inside the kernel. For example, `sched_process_exec` fires every time a process runs a new program. Tracepoints are more stable than kprobes and work across kernel versions.
- **Uprobes** are like kprobes, but for user-space programs. For example, you can hook into the readline() function in Bash to see when someone types a command.
- **BPF LSM** essentially allows instrumenting Linux Security Module (LSM) hooks at runtime. A good way to think of LSM hooks is as some kind of built-in checkpoint that asks, “Is this action allowed?” before letting a process do something sensitive. Security systems like SELinux or AppArmor use LSM hooks. Tetragon can also use LSM hooks for access control and observability. LSM hooks are reliable, less prone to race conditions like TOCTOU, and always represent real enforcement points.

This blog post titled: [Linux tracing systems & how they fit together](https://jvns.ca/blog/2017/07/05/linux-tracing-systems/) by Julia Evans, is a good resource for learning about the tracing systems in the Linux kernel.

###### Tetragon Points of View

Tetragon abstracts these raw attachment points into policy targets. **A hook point in Tetragon is simply a declaration of where you want to monitor and what arguments you want extracted.**
With a kprobe spec for example, you declare, and Tetragon takes care of attaching to the kernel function and extracting the arguments.

```yaml
spec:
  kprobes:
    - call: 'fd_install'
      syscall: false
```

With a tracepoint spec, Tetragon subscribes to the stable kernel tracepoints, decoding the arguments.

```yaml
spec:
  tracepoints:
    - subsystem: 'sched'
      event: 'sched_process_exec'
```

This same principle applies to every hook point in Tetragon. With LSM hooks, for example, you point at security checks(`file_open`, `bprm_check_security`) and Tetragon handles the attachment.

From the Tetragon point of view, **a hook point is a declarative contract that specifies what to watch(functions, tracepoints, LSM hooks), which arguments to pull out(pid, file, etc), and how to filter or act on the event.**

###### Writing Kprobe-based Tracing Policies for Monitoring System Calls

Before rounding off this section, it is important to highlight one interesting abstraction Tetragon provides for kprobe-based policies that monitor system calls. Different CPU architectures implement system calls differently, and this can create portability challenges. On x86_64, system call handlers have names like `__x64_sys_write`, while on ARM64, they're named `__arm64_sys_write`.
Tetragon provides elegant abstraction here. Instead of writing architecture-specific policies like this:

```yaml
# Architecture-specific (don't do this)
spec:
  kprobes:
    - call: '__x64_sys_write' # Only works on x86_64
      syscall: true
```

You can write portable policies that work across different architectures.

```yaml
# Portable across architectures
spec:
  kprobes:
    - call: 'sys_write' # Works on any architecture
      syscall: true
```

Tetragon automatically translates `sys_write` to the correct architecture-specific function name. This abstraction is crucial for policies that need to work across diverse environments.

###### Choosing the Right Hook Point

The best hook point for your policy depends on your specific security observability objective and how much stability you need across environments. If your goal prioritizes portability, tracepoints are often the safest choice. They are built into the kernel source and tend to remain stable across kernel versions.
If you’re writing security-sensitive policies, LSM hooks are a better option. Because they operate on kernel-owned memory after user input has already been validated, they naturally avoid time-of-check to time-of-use (TOCTOU) pitfalls.
When you need fine-grained insight into kernel internals, kprobes give you the flexibility to attach almost anywhere. The tradeoff is that they are more tightly coupled to your kernel version, since function names and prototypes can change between releases. To use them effectively, you need to be comfortable browsing kernel symbols and understanding the calling conventions of the functions you hook.
Finally, if your focus is on application-level behavior, uprobes let you trace functions inside user-space binaries and libraries. The prerequisite here is being able to explore how a particular program is laid out at the binary level.
Whichever option you choose, the common theme is that you’re navigating the Linux kernel (or user-space program) at the level of functions, structures, and symbols. An understanding of these internals is what allows you to pick the right hook.

##### Process Management

In the kernel's view, every running program is represented by a `task_struct` data structure. This massive structure (over 1,000 lines in recent kernels) contains everything the kernel needs to know about a process, including the process and thread group IDs (PID/TGID), memory management information, file descriptor table, security credentials, scheduling information, and signal handling state.
When your Tetragon examines process-related information, it is often looking at fields within the current process's task_struct. Process lifecycle monitoring is a core use case for Tetragon, and by default, without deploying any additional tracing policy, Tetragon observes the process lifecycle. The process section of the [Linux kernel teaching lab](https://linux-kernel-labs.github.io/refs/heads/master/lectures/processes.html) covers in detail how processes work in Linux.

###### Process Monitoring with Tetragon

Consider a policy that tracks process creation using the sched_process_exec tracepoint:

```yaml
spec:
  tracepoints:
    - subsystem: 'sched'
      event: 'sched_process_exec'
      args:
        - index: 0
          type: 'int'
          resolve: 'pid'
        - index: 2
          type: 'linux_binprm'
```

This policy hooks into the scheduler subsystem when a new program is executed. The `resolve: "pid"` directive tells Tetragon to extract the PID from the first argument (a `task_struct` pointer), while the `linux_binprm` type captures information about the binary being executed.

Understanding that process creation involves multiple kernel subsystems (the scheduler, memory manager, and file system) helps you choose the right hook points for your monitoring and enforcement objectives.

###### Process Hierarchies and Namespaces

Container environments add complexity to process management through namespaces. Containers use chroot, namespaces, and cgroups to isolate processes. A good developer-centric resource that covers the Linux internals of how containers work is [Containers From Scratch](https://www.youtube.com/watch?v=8fi7uSYlOdc) and [Crafting Containers By Hand – What Are Containers?](https://btholt.github.io/complete-intro-to-containers/what-are-containers)

A process might have different PIDs in different PID namespaces; PID 1 inside a container might be PID 12345 from the host perspective. Tetragon policies need to account for this. When you're filtering by PID, consider whether you want the namespace PID or the host PID:

```yaml
selectors:
  - matchPIDs:
      - operator: In
        followForks: true
        isNamespacePID: true # Use container-internal PID
        values:
          - 1
```

##### File System and File Descriptors

Whenever a program works with files, for example, opening `/etc/passwd`, writing logs, or reading configs; it goes through the Linux file system layer. The kernel doesn’t let programs touch files directly. Instead, it hands them a file descriptor: a small number like 3, 4, or 5 that represents an open file. File descriptors 0, 1, and 2 are always reserved for standard input, output, and error(`stdin`, `stdout`, `stderr`).

Tetragon can hook into the kernel functions that create or use these file descriptors. For example, the `fd_install()` function runs whenever the kernel adds a new open file to a process. By attaching a kprobe here, you can see which files are being opened and by which process:

```yaml
spec:
  kprobes:
    - call: 'fd_install'
      syscall: false
      args:
        - index: 0
          type: int # the file descriptor number
        - index: 1
          type: file # the file being opened
      selectors:
        - matchArgs:
            - index: 1
              operator: 'Prefix'
              values:
                - '/etc'
```

The big idea here is that every file action in Linux is funneled through file descriptors, and by watching them, you can write focused policies that only report on the paths you care about.

##### Networking and Sockets

In Linux, all network traffic flows through sockets. A socket works like a file descriptor, but instead of pointing to a file, it represents a network connection (IP + port). When an app calls connect() or send(), the kernel manages the socket. This means every HTTP request, DNS lookup, or database call is visible at the socket layer.

With Tetragon, you can hook into functions like `__sys_connect` to catch new connections and filter them by port, address, or namespace.

```yaml
spec:
  kprobes:
    - call: '__sys_connect'
      syscall: true
      args:
        - index: 0
          type: 'sockaddr'
      selectors:
        - matchArgs:
            - index: 0
              operator: 'DPortPriv'
```

Sockets are the kernel’s gateway for networking, and Tetragon gives you the hooks to watch or restrict how they’re used.

##### Conclusion: Building Your Kernel Knowledge

Understanding Linux kernel fundamentals for effectively writing Tetragon tracing policies is an ongoing journey. The kernel is a complex, evolving system, and effective policy development requires understanding how its various subsystems interact.

Start by focusing on the areas most relevant to your monitoring objectives:

- For file system security: study VFS, file descriptors, and path resolution
- For process monitoring: understand task structures, process lifecycle, and namespaces
- For network security: learn socket structures, network stack flow, and connection tracking
- For container security: understand namespaces, cgroups, and container runtime interactions

The investment in kernel knowledge pays dividends in the form of more effective, efficient, and reliable tracing policies. Remember that kernel internals can change between versions, so staying current with kernel development and testing your policies across different kernel versions is essential for production deployments

##### Resources and References:

- [Linux Kernel Teaching](https://linux-kernel-labs.github.io/refs/heads/master/index.html)
- [Linux tracing systems & how they fit together](https://jvns.ca/blog/2017/07/05/linux-tracing-systems/)
- [Learning eBPF by Liz Rice](https://isovalent.com/books/learning-ebpf/)
- [Containers From Scratch](https://www.youtube.com/watch?v=8fi7uSYlOdc)
- [Crafting Containers By Hand – What Are Containers?](https://btholt.github.io/complete-intro-to-containers/what-are-containers)
- [File Monitoring with eBPF and Tetragon](https://isovalent.com/blog/post/file-monitoring-with-ebpf-and-tetragon-part-1/)
- [Tetragon Documentation](https://tetragon.io/docs/)
- [The Linux Programming Interface](https://www.amazon.com/Linux-Programming-Interface-System-Handbook/dp/1593272200)

#### Cilium at KubeCon + CloudNativeCon and CiliumCon North America 2025

URL: https://cilium.io/blog/2025/08/20/cilium-at-kubecon-na-2025
Date: 2025-08-20
Categories: Community

**_Author: Katie Meinders, Isovalent_**

The Cilium community is buzzing to reunite again this November in Atlanta for [KubeCon + CloudNativeCon and CiliumCon North America 2025](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/register/?utm_source=cilium&utm_medium=homepage&utm_campaign=10608228-KubeCon-NA-2025&utm_content=hero). Learn where to find Cilium during the show, which talks to attend, and how to engage with end users, core contributors, and industry leaders.

##### A look back at KubeCon + CloudNativeCon and CiliumCon Europe in London

Earlier this year, the community gathered in London during KubeCon + CloudNativeCon and CiliumCon Europe. It was a packed house for CiliumCon where attendees heard talks from Isovalent’s Liz Rice and Microsoft’s Neha Aggarwal about configuring Cilium for performance and scale, G-Research’s Luigi Zhou on how the organization uses Cilium as the core network for on-premise, bare-metal clusters and ML workloads, simplifying multi-cluster networking with the multi cluster services API, and much more.

KubeCon keynote attendees also heard the latest Cilium updates from Liz Rice from reporting live at London landmarks.

Contributors also gathered during the [Cilium Developer Summit](https://github.com/cilium/dev-summits/tree/main/2025-EU) to discuss the future of the Cilium project. Representatives from Google, Isovalent, Microsoft, Seznam, and more discussed the status of existing and new features like Cluster Mesh and StateDB-based load balancing and the project roadmap.

During the Maintainers track, the community came together to celebrate nearly a decade of Cilium, to hear the latest updates to CIlium, Hubble, and Tetragon, and hear how DBSchenker is using Cilium for networking and observability and why Cilium provides the core networking features in the GKE dataplane.

From co-los to demos and lightning and maintainer talks, we will continue the celebration of 10 years of Cilium in Atlanta!

##### CiliumCon

Join us on Monday, November 10, for a morning of talks, community updates, and end user stories during CiliumCon. The schedule is full of insightful talks around Cilium, Hubble, Tetragon and eBPF. Here’s a look at what you can expect:

[CiliumCon | Welcome + Opening Remarks - Hemanth Malla, Program & Bill Mulligan, Co-Chairs](https://colocatedeventsna2025.sched.com/event/28F61/ciliumcon-welcome-+-opening-remarks-bill-mulligan-hemanth-malla-program-committee-co-chairs?iframe=no)

09:00 - 09:05 AM

Event co-chairs, Bill & Hemanth, will share the latest updates from the project and community.

[Rewiring Cilium: Operator-Driven Scale and Security With CiliumEndpointSlice - Shreya Jayaraman & Tamilmani Manoharan, Microsoft](https://colocatedeventsna2025.sched.com/event/28Czl/rewiring-cilium-operator-driven-scale-and-security-with-ciliumendpointslice-shreya-jayaraman-tamilmani-manoharan-microsoft?iframe=no)

9:10 - 9:35 AM

As clusters scale, the control plane often bears the brunt of the growing pains. But what if scale didn’t have to mean stress? In this session, we’ll explore the CiliumEndpointSlice feature that has enabled more scalable container networking and laid the foundation for future optimizations in the Cilium ecosystem.

We start by exploring the default Cilium configuration and the strain it places on the kube-apiserver at scale. Then, we explore how enabling CiliumEndpointSlice helped us unlock higher scale at AKS, with details on its design, how it works & the performance gains we observed. We also share our experiences with fine-tuning clusters at scale & the tradeoffs involved. Finally, we take a glimpse at what’s next in the quest to centralize the Operator and make policy enforcement in Cilium more scalable.

Whether you're running Cilium at scale or just getting started, this talk will equip you with practical insights and architectural strategies to run high-scale Cilium clusters.

[Operating Cilium Mesh With Confidence: From Zero To Production in 90 Days - Andrew Timmes, SeatGeek](https://colocatedeventsna2025.sched.com/event/28D0I/operating-cilium-mesh-with-confidence-from-zero-to-production-in-90-days-andrew-timmes-seatgeek?iframe=no)

9:45 - 10:10 AM

In late 2024, the Cloud Platform team at SeatGeek was tasked with migrating hundreds of microservices from their existing service mesh to Cilium Mesh in just three months.

In this session, we'll talk about how our engineers got up to speed on the inner workings of Cilium in short order, how all the pieces of Cilium's CNI and mesh systems fit together, and walk through production incidents and how we reasoned about a brand-new eBPF networking stack, with the goal of providing you with the context and methodologies to do the same either tomorrow or 5 years from now.

[CiliumCon Sponsored Keynote: Liz Rice, Isovalent at Cisco](https://colocatedeventsna2025.sched.com/event/28F6G/ciliumcon-sponsored-keynote-liz-rice-isovalent-at-cisco?iframe=no)

10:15 - 10:20 AM

[IPv6 First, Not Just Ready: Kubernetes Without IPv4 Using Cilium at ESnet - Kapil Agrawal, ESnet](https://colocatedeventsna2025.sched.com/event/28D0g/ipv6-first-not-just-ready-kubernetes-without-ipv4-using-cilium-at-esnet-kapil-agrawal-esnet?iframe=no)

10:40 - 11:05 AM

ESnet, the U.S. Department of Energy’s high-performance research network, runs Kubernetes on-prem in IPv6 only network using Cilium as our CNI. While Kubernetes introduced IPv6 support in version 1.22, deploying production grade clusters without IPv4 presents unique challenges that require careful architectural design such as:

- Support for DNS64, NAT64 in the underlying network
- Obtaining a stable egress IPv6 address
- Traffic patterns (ingress vs. egress)
- To masquerade or not to
- Allocating PodCIDR, ServiceCIDR sizes correctly
- Choosing between tunneling vs. native routing mode and it's implications

Attendees will learn how we solved these problems using Cilium features such as:

- eBPF datapath using kube proxy replacement
- BGP, load balancer IPAM, GatewayAPI
- NAT46x64Gateway in absence of NAT64 gateway on the network

We will also discuss steps to deploy an IPv6 only KIND cluster using Cilium and how that can be useful for testing applications during development.

[Containing Wildfires in a Sprawling Multi-Cluster Network: The Network's Immune System - Anmol Krishan Sachdeva & Paras Mamgain, Google](https://colocatedeventsna2025.sched.com/event/28D1Y/containing-wildfires-in-a-sprawling-multi-cluster-network-the-networks-immune-system-anmol-krishan-sachdeva-paras-mamgain-google?iframe=no)

11:15 - 11:40 AM

In a sprawling multi-cluster network, a security breach in one pod is never just a local problem. It’s the potential start of a wildfire. The real danger isn’t the initial compromise, it's the attacker's lateral movement across clusters, turning a small incident into a catastrophic failure.

Traditional incident response is too slow; by the time a human can analyze logs and apply new firewall rules, the damage is done. How can we build a system that detects a threat and automatically "shrink-wraps" the blast radius in real-time?

This session presents a cutting-edge, automated threat containment system built on the combined power of Cilium, Tetragon, and Hubble. We demonstrate a closed-loop workflow that transforms your network from a passive conduit into an active immune system.

Join us for a live-fire demo where we’ll compromise a service and watch the mesh quarantine the breach, proving you can contain wildfires before they rage out of control.

[Can We Really Parse DNS in eBPF? Improving Cilium ToFQDN With In-kernel Policy Updates - Hemanth Malla, Microsoft](https://colocatedeventsna2025.sched.com/event/28D2Q/can-we-really-parse-dns-in-ebpf-improving-cilium-tofqdn-with-in-kernel-policy-updates-hemanth-malla-microsoft?iframe=no)

11:50 AM - 12:15 PM

Parsing DNS in eBPF has been “tricky” for a long time. One needs to handle DNS compression, follow an unknown number of pointers and parse an unknown number of answers. Things get complicated with DNS over TCP. What if the DNS response is split across multiple packets requiring us to parse TCP streams in eBPF? Luckily eBPF is now sufficiently advanced to handle such complex logic.

So, what does this mean for toFQDN policies in CIlium? Cilium’s toFQDN policies currently rely on a userspace DNS proxy to intercept responses and populate the endpoint's policy map. A userspace high availability mode for the proxy can avoid hard dependency on the agent at runtime. However this results in increased complexity and need for co-ordination between agent and out of process proxy.

So can we do better natively with eBPF? Yes. This talk will explore how we can implement such a system and what it takes to integrate it into Cilium’s toFQDN policy sub-system.

[Lightning Talk: Taming the BPF LRU: Eliminating TCP Resets in Cilium - Gyu Tae Bae, Kakao Corp.](https://colocatedeventsna2025.sched.com/event/28D2o/cllightning-talk-taming-the-bpf-lru-eliminating-tcp-resets-in-cilium-gyu-tae-bae-kakao-corp?iframe=no)

12:20 - 12:30 PM

At scale, Cilium users often face mysterious TCP connection failures from unexpected RST packets. This session explores a critical bug where Cilium's BPF-based SNAT and its LRU eviction policy prematurely terminate active sessions. We will dissect the root cause in the eBPF datapath and reveal the elegant fix, now merged upstream in Pull Request #37747: proactively restoring the original NAT entry on the reverse traffic path. This solution, born from a real-world production issue, reduced connection failures from up to 10% to nearly zero.

This talk is a must for operators debugging network instability and developers tackling real-world eBPF challenges. You will leave with a clear diagnosis for this "silent killer" and key insights into building robust, high-performance cloud networking.

##### Cilium at other Co-located Events

There are a couple of great talks on Cilium during other KubeCon co-lo events, including Cloud Native University and Observability Day, also happening November 10.

[From Squire To Network Knight: Defending the Kubernetes Castle the Cloud Native Way - Donia Chaiehloudj, Isovalent at Cisco](https://colocatedeventsna2025.sched.com/event/28D0X/from-squire-to-network-knight-defending-the-kubernetes-castle-the-cloud-native-way-donia-chaiehloudj-isovalent-at-cisco?iframe=no&w=100%25&sidebar=yes&bg=no)

Cloud Native University - Monday, November 10, 10:40 - 11:05 AM

In medieval times, a squire was the first step toward becoming a knight. In Kubernetes, many of us start the same way eager to protect our clusters, but armed with the wrong tools and assumptions. In this talk, we’ll start with a wide-open Kubernetes cluster — no policies, no protections — and walk through what’s missing, why it matters, and how to build security the cloud native way.

We’ll contrast static, perimeter-based firewalls with Kubernetes native policy and show how to evolve from static IPs to identity-based enforcement. Step by step, we’ll build protections using Network Policies and Cilium’s identity-aware, DNS-enforcing, and observability-rich capabilities.

Along the way, you’ll learn to debug common issues, avoid the traps of “firewall thinking,” and think in terms of intent, not infrastructure. By the end, you’ll have the tools and mindset to go from squire to network knight.

[How UW Madison Monitors 25+ Clusters With Cilium and Prometheus Agent on a Student-Sized Budget - Cory Sherman, University of Wisconsin & Ryan Etten, Red Hat](https://colocatedeventsna2025.sched.com/event/28D14/how-uw-madison-monitors-25+-clusters-with-cilium-and-prometheus-agent-on-a-student-sized-budget-cory-sherman-university-of-wisconsin-ryan-etten-red-hat)

Observability Day - Monday, November 10, 10:40 - 11:05 AM

Budgets keep shrinking while observability demands explode—so how do you capture L7 insight across 25+ Kubernetes clusters without paying a sidecar tax? At UW Madison, we paired Cilium’s eBPF networking with the Prometheus Agent’s remote write, streaming rich flow telemetry to a central Grafana stack at a fraction of the usual footprint. Live on stage, we’ll fire test traffic with curl from a single app pod and watch Cilium Flow Metrics spike and Alertmanager light up within seconds—no sidecars, no context switching. You’ll leave with a field-tested playbook: scrape-target pruning, discovery patterns that survive air-gapped research zones, and the real story of what broke first—and how we fixed it. This case study demonstrates light weight mesh and light weight observability for a university budget.

##### KubeCon + CloudNativeCon North America 2025 Talks

Don't miss Cilium during the main event. There are several exciting talks featuring Cilium from community members and end users.

[10 Years of Cilium: Connecting, Securing, and Simplifying the Cloud Native Stack - Bill Mulligan & Paul Arah, Isovalent @ Cisco; Neha Aggarwal, Microsoft; Satish Krishnan, UBS](https://kccncna2025.sched.com/event/27Nnr/10-years-of-cilium-connecting-securing-and-simplifying-the-cloud-native-stack-bill-mulligan-paul-arah-isovalent-cisco-neha-aggarwal-microsoft-satish-krishnan-ubs?iframe=yes&w=100%&sidebar=yes&bg=no)

Tuesday, November 11, 11:15 - 11:45 AM

Join us as we celebrate a decade of Cilium, now the de-facto standard CNI for Kubernetes and a cornerstone of cloud native networking and security. This session provides updates on the latest Cilium release and showcases how its unified eBPF-powered stack is transforming Kubernetes environments and beyond by replacing fragmented toolchains with seamless, secure, scalable, and simplified solutions.

We’ll showcase advancements in multi-cluster connectivity and support for massively scalable clusters. You’ll also hear updates from sub-project Tetragon for runtime enforcement and security observability.

Contributors and adopters from Isovalent, Microsoft, and UBS will share how they’re using Cilium to streamline operations and reshape the cloud native stack cementing Cilium’s role as the networking and security data plane for modern infrastructure for the next decade to come.

[QEMU in the Fast Lane: Accelerating KubeVirt Networking With eBPF - Daniel Borkmann & Anton Protopopov, Isovalent at Cisco](https://kccncna2025.sched.com/event/27FXg/qemu-in-the-fast-lane-accelerating-kubevirt-networking-with-ebpf-daniel-borkmann-anton-protopopov-isovalent-at-cisco?iframe=yes&w=100%&sidebar=yes&bg=no)

Tuesday, November 11, 5:00 - 5:30 PM

As organizations converge infrastructure by running VMs alongside containers in Kubernetes for cost savings and simplified operations, networking performance remains a key challenge especially for latency and throughput sensitive workloads. In this talk, we'll show how eBPF with AF_XDP can improve KubeVirt networking by putting QEMU in the fast lane (all without losing the ability to observe and enforce policy on traffic in the host unlike SR-IOV).

We'll unpack an architecture built on Cilium, netkit, and upstream Linux kernel enhancements that enable KubeVirt Pods to launch QEMU/KVM instances backed by high-performance AF_XDP interfaces. You'll learn how our contributions to QEMU and the kernel enable this accelerated path, and how it stacks up against KubeVirt defaults using real world benchmarks.

Expect kernel spelunking, performance graphs, and a vision for adding a fast lane to VM networking.

[Community Capital: Making OSS and Businesses Successful Together - Liz Rice, Isovalent at Cisco](https://kccncna2025.sched.com/event/27FaR/community-capital-making-oss-and-businesses-successful-together-liz-rice-isovalent-at-cisco?iframe=yes&w=100%&sidebar=yes&bg=no)

Wednesday, November 12, 3:00 - 3:30 PM

Just as open source success is about more than great code, building a successful business on OSS relies on more than pricing. This talk explores how ecosystems thrive when maintainers, vendors, and users build on shared values and trust. We'll unpack why timing matters when open sourcing a project or contributing it to a foundation, how vendors can grow real businesses by adding value around open source rather than trying to control it, and why vendor success matters to the projects themselves.

Drawing on Liz’s experience with the Cilium project and as former Chair of the TOC, she’ll look at examples from the CNCF and beyond, to show how shared values can lead to collective success, and draw out the relationships between project health and vendor viability. Expect thoughtful metaphors, practical takeaways, and a reminder that open source isn’t a zero-sum game, and commercial success can amplify community impact.

[Portable MTLS for Kubernetes: A QUIC-Based Plugin Compatible With Any CNI - Apurup Chevuru & Michael Zappa, Microsoft](https://kccncna2025.sched.com/event/27FaL/portable-mtls-for-kubernetes-a-quic-based-plugin-compatible-with-any-cni-apurup-chevuru-michael-zappa-microsoft?iframe=yes&w=100%&sidebar=yes&bg=no)

Wednesday, November 12, 3:00 - 3:30 PM

Most mTLS solutions in Kubernetes are tightly coupled to service meshes or L7 proxies, introducing operational complexity, performance overhead, and limited flexibility across CNIs. In this session, we present a portable mTLS plugin built on QUIC, a modern transport protocol designed for performance and security. Our plugin offers native, transparent L4 encryption with automatic certificate management via SPIRE, and is designed to work seamlessly with any third-party CNI including Cilium, Calico, and others without sidecars or mesh dependencies. We'll deep-dive into the architecture, QUIC+eBPF redirection, SPIRE integration, and benchmarks that highlight performance gains over Envoy-based solutions.

[TikTok's IPv6 Journey To Cilium: Pitfalls and Lessons Learned - Giri Kuncoro & Joseph Pallamidessi, ByteDance](https://kccncna2025.sched.com/event/27FbS/tiktoks-ipv6-journey-to-cilium-pitfalls-and-lessons-learned-giri-kuncoro-joseph-pallamidessi-bytedance?iframe=yes&w=100%&sidebar=yes&bg=no)

Wednesday, November 12, 4:45 - 5:15 PM

Cilium has been the standard for Kubernetes networking and security. TikTok migrated clusters to use Cilium for its advanced security features like mutual authentication, along with high performance networking and enhanced observability. The main challenge was executing this on TikTok IPv6 only datacenters, as Cilium has been battle tested with IPv4 and dual-stack, but not with IPv6 only environments.

This talk shares the journey of making Cilium work for IPv6 only Kubernetes, highlighting the limitations and techniques to overcome them. First, Cilium doesn't support tunneling over IPv6, native routing mode must be configured. Second, we encountered several bugs related to IPv6 only: NDP traffic getting dropped by Cilium Network Policy due to incorrect identification; DNS policy not allowing traffic for IPv6 DNS servers; broken cilium debug tools when IPv4 related BPF maps not found. Finally, the NodePort timeout issue was blocking us from enabling Cilium to fully replace kube-proxy.

**Cilium Project Booth:** Be sure to visit Cilium’s booth in the Project Pavilion to see live demos, explore new features, and connect with the community.

Whether you are attending hands-on workshops, lightning talks, or networking with industry peers, you will find countless opportunities to interact with the Cilium community. Be sure to stop by with any questions or just say hello!

#### Securing the Node: A Primer on Cilium’s Host Firewall

URL: https://cilium.io/blog/2025/08/15/host-firewall-primer
Date: 2025-08-15
Categories: Community, Technology

###### July 6th, 2025

###### Author: Paul Arah, Isovalent@Cisco

#####

When discussing Kubernetes network security, much of the attention focuses on pod-to-pod traffic, ingress controllers, and service meshes. But what about the underlying nodes themselves, the very foundation on which our workloads run? The attack surface area that Kubernetes nodes expose is vast, and if left unprotected, can become a golden ticket for malicious actors.

Cilium host firewall is built to lock down the host network namespace with precision, visibility, and control, extending the same familiar declarative Kubernetes network policy model to the underlying host. In this blog post, we’ll explore what Cilium Host Firewall is, how it works, and why it should be a core part of your Kubernetes security.

##### The Node as a Blind Spot

Kubernetes native network policies don’t apply to host-level traffic. This means any communication that enters or leaves the host directly (for example, SSH, kubelet, or external monitoring agents) is largely invisible to traditional Kubernetes policy enforcement. While some firewalling is possible via firewalld or external systems, managing those rules is brittle and lacks integration with Kubernetes. At the core of it, this is the problem Cilium host firewall solves. Leveraging eBPF, Cilium introduces host firewalling directly into the fabric of the cluster.

##### How Cilium’s Host Firewall Works

Cilium treats the node as a special type of endpoint with the label reserved:host. This lets us apply policies just like we would for pods, except these apply to traffic to and from the node(s) themselves. Cilium host firewall operates at the interface level. You can explicitly set which devices it attaches to (eth0, eth1, etc.), or let Cilium auto-detect them.

##### Enabling Host Firewall

The most convenient way to try out Cilium’s host firewall while keeping the user experience close to that of a production cluster is to set up your Kubernetes environment using a tool that spins up virtual machines locally. This makes it easier to test SSH access, simulate real network interfaces, and apply node-level policies. My personal favourites are [Minikube](https://minikube.sigs.k8s.io/docs/) with the [virtualbox driver](https://minikube.sigs.k8s.io/docs/drivers/virtualbox/), or [k3s](https://k3s.io/) combined with [Multipass](https://canonical.com/multipass) for lightweight VM-based nodes that you can SSH into. We won’t dive into setting up a cluster for the sake of brevity. That said, you’re free to set up your cluster however you like, as long as it allows you to test Cilium in a way that includes host-level visibility and networking.

If you are installing Cilium via the Cilium CLI, you can enable host firewall with the flag below.

```shell
cilium install \
  --version 1.17.4 \
  --set hostFirewall.enabled=true \
  --set kubeProxyReplacement=true
```

Via Helm, you can also enable the host firewall like below.

```shell
helm install cilium ./cilium \
  --namespace kube-system \
  --set hostFirewall.enabled=true \
  --set devices='{eth0}'
```

Once Cilium has been deployed, it starts managing the selected interfaces and the host endpoint. You can verify the status with:

```shell
kubectl exec -n kube-system ds/cilium -- cilium-dbg status | grep 'Host firewall'
```

We can also verify that the Host Firewall feature is activated using the Cilium CLI:

```shell
cilium config view | grep host-firewall
```

To apply policies, first label the target node:

```shell
kubectl label node k8s1 node-access=ssh
```

This allows you to scope policies to specific nodes based on purpose or function (e.g., node-access=ssh, type=ingress-worker).

##### Audit Mode

Before enforcing a new host policy, you can enable Policy Audit Mode. This mode logs what would have been dropped without actually enforcing the rules. It's an essential step to avoid self-inflicted outages such as accidentally cutting off access to the kube-apiserver or SSH.

```shell
    kexec cilium-dbg endpoint config $HOST_EP_ID PolicyAuditMode=Enabled
```

Monitor traffic verdicts with:

```shell
    kexec cilium-dbg monitor -t policy-verdict --related-to $HOST_EP_ID
```

This provides real-time insights into which flows would have been denied, letting you fine-tune policies before flipping the enforcement switch.

##### Observe Network Traffic with Hubble

Cilium assigns a special identity of “1” to the nodes in a cluster. We can use this identity to filter and observe the node’s network traffic.

```shell
    hubble observe --to-identity 1 --port 22 -f
```

##### Writing Host Network Policies

Host network policies are defined using the CiliumClusterwideNetworkPolicy (CCNP) custom resource. Anatomically, the major difference between host policies and network policies is that traffic is matched based on the nodeSelector field rather than podSelector or endpointSelector like in Cilium network policies. This essentially means policies are scoped to the node itself, rather than to specific pods or workloads.

This table summarizes the major differences between Cilium host policies and Cilium network policies.

| Feature | Cilium Network Policy | Cilium Host Policy |
| --------------- | ----------------------------------- | ------------------------------------ |
| Selector Type | endpointSelector (pod-level) | nodeSelector (node-level) |
| Namespace Scope | Namespaced or cluster-wide | Always cluster-wide |
| Applies To | Pods (in or out of host networking) | Host namespace + host-networked pods |
| L3/L4 Rules | Yes | Yes |
| L7 Rules | Yes | No |

Here’s an example host network policy that allows only SSH (TCP/22) and ICMP (ping) traffic from outside the cluster, while permitting all internal cluster communications to the host:

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: 'demo-host-policy'
spec:
  nodeSelector:
    matchLabels:
      node-access: ssh
  ingress:
    - fromEntities:
        - cluster
    - toPorts:
        - ports:
            - port: '22'
              protocol: TCP
    - icmps:
        - fields:
            - type: EchoRequest
              family: IPv4
```

##### Enforcing the Policy

Once you're confident your policy won't break anything, you can disable audit mode:

```shell
    kexec cilium-dbg endpoint config $HOST_EP_ID PolicyAuditMode=Disabled
```

Immediately, Cilium will begin enforcing the host policy. Any disallowed communication, like a rogue external SSH attempt, will be blocked and logged with a clear verdict:

```shell
    Policy verdict log: action deny, match none, 10.0.2.2:49038 -> 10.0.2.15:21 tcp SYN
```

Here’s an example of a real-world production policy that permits just the minimum required ingress traffic to a worker node:

```yaml
piVersion: 'cilium.io/v2'
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: 'lock-down-ingress-worker-node'
spec:
  nodeSelector:
    matchLabels:
      type: ingress-worker
  ingress:
    - fromEntities:
        - remote-node
        - health
    - toPorts:
        - ports:
            - port: '22'
              protocol: TCP
            - port: '6443'
              protocol: TCP # kube-apiserver
            - port: '2379'
              protocol: TCP # etcd
            - port: '4240'
              protocol: TCP # health checks
            - port: '8472'
              protocol: UDP # VXLAN
```

##### Best Practices and Troubleshooting Tips

- **Label Nodes Clearly**: Make sure `nodeSelector` labels in your policy match what’s actually set on the nodes.
- **Use Audit Mode First**: Always test in audit mode to avoid breaking the control plane or SSH access.
- **Watch Monitor Logs**: Use `cilium-dbg monitor` to get clear visibility into dropped or allowed packets.
- **Match Devices**: If host firewall seems inactive, verify that Cilium is managing the correct interfaces with `cilium-dbg status`.
- **Hubble is Your Friend**: Flow logs from Hubble come in very handy when writing and debugging policies.

##### Conclusion

Cilium Host Firewall brings Kubernetes-native, eBPF-powered network security to the host layer, giving operators fine-grained control over traffic to and from the Kubernetes nodes themselves. Using the same familiar Kubernetes network policy declarative model, it's never been easier to treat your nodes as first-class citizens.

##### Additional Resources:

- [Cilium Host Firewall Lab](https://isovalent.com/labs/cilium-host-firewall/)
- [eCHO Episode 40: Cilium Host Firewall](https://www.youtube.com/watch?v=GLLLcz398K0&t=288s)
- [eCHO Episode 184: Securing Kubernetes Nodes with Cilium Host Firewall](https://www.youtube.com/watch?v=fLYHG07VdNc)
- [Kubernetes Node Firewalling from the Inside Out - Jef Spaleta & Justin Garrison](https://www.youtube.com/watch?v=-3Fbb3BNjjE)

#### Installing Cilium on EKS in Overlay(BYOCNI) and CNI Chaining Mode

URL: https://cilium.io/blog/2025/07/08/byonci-overlay-install
Date: 2025-07-08
Categories: Community

###### July 6th, 2025

###### Author: Paul Arah, Isovalent@Cisco

##### Installing Cilium on EKS in Overlay(BYOCNI) and CNI Chaining Mode

In the first part of this EKS series, we covered setting up an EKS cluster and installing Cilium in ENI mode.
In this second and last of the EKS series, we’ll cover installing Cilium in overlay mode and CNI chaining mode.

##### Installing Cilium on EKS in CNI chaining mode

Installing Cilium in different modes follows a similar process. The main distinction lies in whether the AWS VPC CNI is disabled and Helm flags used during installation. For brevity sake, I'll skip certain repetitive steps in the installation process with the assumption that you've read the [previous blog post](https://cilium.io/blog/2025/06/19/eks-eni-install/) in this series, or can refer back to it as needed.

##### Prerequisites

The following prerequisites need to be taken into account:

- An active AWS Account
- Install [kubectl](https://kubernetes.io/releases/download/#kubectl), [Helm](https://helm.sh/docs/intro/install/), [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/setting-up.html), [awscli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), and [Cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli)

##### Creating our EKS cluster

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: cluster1
  region: us-east-1
  version: '1.30'
iam:
  withOIDC: true
addonsConfig:
  disableDefaultAddons: true
addons:
  - name: coredns
  - name: vpc-cni
```

Notice that in the cluster config file, the AWS VPC CNI has been added to the list of add-ons with which we create the cluster. This is because in CNI chaining mode, we want both the AWS VPC CNI and Cilium present in the cluster as opposed to ENI mode, where we entirely remove the AWS VPC CNI.

- Using eksctl, create the cluster.

```
eksctl create cluster -f cilium-eks-config.yaml
```

- Since we have no node group yet, there will be no nodes in our cluster, and the pods will be stuck in a pending state.

````shell
kubectl get pod -n kube-system

NAME                      READY   STATUS    RESTARTS   AGE
coredns-c7bbdfbb8-j2wdg   0/1     Pending   0          2m24s
coredns-c7bbdfbb8-mcgt5   0/1     Pending   0          2m24s

- Next, we need to install Cilium in our cluster. We’ll do this using Helm. But first, we need to grab the value of our Kubernetes service host by running the command below:

```shell
kubectl cluster-info

Kubernetes control plane is running at https://xxxxxxxxxxxxxxxxxxxx.yyy.eu-west-2.eks.amazonaws.com
CoreDNS is running at https://xxxxxxxxxxxxxxx.gr7.eu-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
````

- Up next, we install Cilium using Helm with the command below

```shell
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --version 1.17.5 \
 --namespace kube-system \
 --set cni.chainingMode=aws-cni \
 --set cni.exclusive=false \
 --set enableIPv4Masquerade=false \
 --set routingMode=native\
 --set k8sServiceHost=xxxxxxxxxxxxxx.gr7.eu-west-2.eks.amazonaws.com\
 --set k8sServicePort=443

```

Notice we're installing Cilium with a different set of flags? So what these flags do:

| Helm Flag                           | Description                                                                                                                                     |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `--namespace kube-system`           | Specifies the Kubernetes namespace where Cilium will be installed.                                                                              |
| `--set cni.chainingMode=aws-cni`    | Sets the CNI chaining mode to `aws-cni`. This allows Cilium to integrate with the AWS CNI plugin.                                               |
| `--set cni.exclusive=false`         | Sets the CNI exclusive mode to false, meaning Cilium will not be the only CNI plugin and can coexist with other CNI plugins.                    |
| `--set enableIPv4Masquerade=false`  | Disables IPv4 masquerading. This means packets exiting the cluster will not have their source IP addresses translated to the node's IP address. |
| `--set routingMode=native`          | Enables native routing mode. This mode does not rely on encapsulation (e.g., VXLAN or Geneve) for routing packets between nodes.                |
| `--set endpointRoutes.enabled=true` | Enables endpoint routes. This allows Cilium to manage routing at the endpoint level, creating more specific routes for the network traffic.     |

- Next, we create a corresponding node group for our cluster.

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
 name: cluster1
 region: eu-west-2

managedNodeGroups:
 - name: ng-1
 desiredCapacity: 2
 privateNetworking: true
```

To create the node group, run the command below

```
eksctl create nodegroup -f nodegroup.yaml
```

- Afterwards, we can check the status of the nodes to ensure that they are in a ready state.

````shell
kubectl get nodes

NAME STATUS ROLES AGE VERSION
ip-192-168-125-58.eu-west-2.compute.internal Ready <none> 3h25m v1.30.11-eks-473151a
ip-192-168-128-147.eu-west-2.compute.internal Ready <none> 3h25m v1.30.11-eks-473151a

- Validate health check: cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking and connectivity. You can check node-to-node health with cilium-health status:

```shell

kubectl -n kube-system exec ds/cilium -- cilium status

````

- Additionally, you can run the connectivity tests to ensure your installation works properly.

##### Installing Cilium on EKS in BYOCNI or Overlay Mode

BYOCNI mode ("Bring your own CNI"), or overlay, creates an overlay network for pods using UDP-based encapsulation protocols such as VXLAN or Geneve. This installation mode has the advantage of being independent of pod network sizing/IP addressing limitations of AWS ENI.

The steps for Cilium in overlay mode is quite similar. We create a cluster with the AWS VPC CNI disabled and then install Cilium with right set of helm flags.

- To get started, we’ll create a ClusterConfig file. In the file below, we’ve disabled kube-proxy and AWS VPC CNI.

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster1
  region: eu-west-2
  version: '1.30'
addonsConfig:
  disableDefaultAddons: true
addons:
  - name: coredns
```

Notice this time our cluster config file doesn’t include the AWS VPC CNI or kube-proxy add-on.

- Using eksctl, create the cluster.

```shell
eksctl create cluster -f cilium-eks-config.yaml
```

We need to grab the value of our Kubernetes service host by running the command below:

```shell
kubectl cluster-info

Kubernetes control plane is running at https://xxxxxxxxxxxxxxxxxxxx.yyy.eu-west-2.eks.amazonaws.com
CoreDNS is running at https://xxxxxxxxxxxxxxx.gr7.eu-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
```

Up next, we install Cilium using Helm with the command below

```shell
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --version 1.17.5 \
  --namespace kube-system \
  --set egressMasqueradeInterfaces=eth0\
  --set k8sServiceHost=xxxxxxxxxxxxxxxxxxxxxx.gr7.eu-west-2.eks.amazonaws.com\
  --set k8sServicePort=443
```

Next, we create a corresponding node group for our cluster.

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: cluster1
  region: eu-west-2
managedNodeGroups:
  - name: ng-1
    desiredCapacity: 2
    privateNetworking: true
```

To create the node group, run the command below

```yaml
eksctl create nodegroup -f nodegroup.yaml
```

- Afterwards, we can check the status of the nodes to ensure that they are in a ready state, use the Cilium health tool and connectivity tests to validate our installation.

##### A Primer on Encapsulation vs Native/Direct Routing in Cilium

Throughout this series, we’ve explored different ways to install Cilium in an EKS cluster. To better understand the trade-offs between these methods and help determine which one is right for you, it's important to examine a core concept in Cilium: routing.

Cilium supports two primary networking modes for routing traffic between Kubernetes nodes: encapsulation and native routing each with its own strengths and drawbacks.

Encapsulation uses overlay networks like VXLAN or Geneve to tunnel traffic between nodes, forming a mesh of virtual paths. This simplifies setup, avoids IP conflicts, and works well in complex or multi-cloud environments. However, it comes with performance overhead and potential [maximum transfer unit(MTU)](https://docs.cilium.io/en/stable/network/concepts/routing/) issues due to the added packet headers.

Native routing, on the other hand, skips tunneling entirely and relies on the underlying network to route PodCIDRs directly. This results in better performance and eliminates MTU concerns, but it requires a network that is aware of all pod IPs typically achievable in on-prem setups or cloud environments.

##### Is AWS ENI is considered a form of native routing in Cilium?

Yes! AWS ENI is implemented via a specialized Cilium datapath optimized for AWS VPC’s native capabilities. Cilium delegates packet forwarding to the underlying network (or routing layer) rather than using encapsulation like VXLAN or Geneve. AWS ENI fits this model because Pod IPs are directly routable within the AWS VPC, they are allocated from the ENI IP ranges that are natively integrated into the AWS network. The AWS VPC acts as the native network that routes packets between ENIs across nodes. This model simplifies communication of pod traffic within VPCs and avoids the need for SNAT.

##### Conclusion

In this two-part series, we explored how to deploy Cilium on EKS using different networking modes—starting with ENI mode, and now wrapping up with overlay (BYOCNI) and CNI chaining modes. Each mode offers distinct advantages depending on your infrastructure needs. As you evaluate which deployment mode best fits your environment, understanding the trade-offs between encapsulation and native routing helps you make informed decisions that align with your goals on EKS.

##### Additional Resources

- [eCHO Episode 106: Live Migration to Cilium in AWS](https://www.youtube.com/watch?v=kurMo3r4Ol4)
- [Life of a packet with Cilium in EKS in CNI chain mode](https://www.youtube.com/watch?v=A91iQS0F9Ug)
- [AWS VPC CNI plugin - Cilium Docs](https://docs.cilium.io/en/latest/installation/cni-chaining-aws-cni/)
- [AWS ENI - Cilium Docs](https://docs.cilium.io/en/latest/network/concepts/routing/#aws-eni-datapath)
- [Routing - Cilium Docs](https://docs.cilium.io/en/latest/network/concepts/routing/#aws-eni-datapath)
- [Helm Reference](https://docs.cilium.io/en/stable/helm-reference/)

#### Explore Our New eCHO Playlist Library

URL: https://cilium.io/blog/2025/16/01/themed-echo-playlists
Date: 2025-07-01
Categories: Community, Technology

###### July 01st, 2025

###### Author: Donia Chaiehloudj, Isovalent@Cisco

eCHO (“eBPF & Cilium Office Hours”) is our weekly live-stream where maintainers, users, and special guests pull back the
curtain on all things Cilium, eBPF, Hubble, and Tetragon. Since episode #1 we’ve:

- Shipped 180 + sessions ranging from five-minute demos to full architecture deep-dives
- Answered live YouTube questions in real time
- Captured every recording so you can binge-watch at your own pace.

To make the library easier to navigate we’ve created theme-based playlist - think of them as fast-track learning paths.
Pick a topic, hit play, and work through the sessions in any order that fits your day. In total you’ll find 16 playlists,
grouped into six themes that mirror the Cilium journey: Cilium in Production (installs and upgrades), Networking for Kubernetes
(policies and datapath), Runtime Security (Tetragon), Mesh Networking (service and cluster mesh), Gateway-API & Traffic Management,
All Things eBPF (development and observability), and Ecosystem & Community (releases, conferences, culture).

###### Cilium in Production

Running Cilium in a home lab is fun, but running it in anger across EKS, bare-metal, and on-prem clusters requires repeatable workflows and a few battle-tested tricks. The Cilium in Production playlist condenses those lessons into one place.

###### What you’ll learn

- Choosing (and changing) data-plane modes on managed Kubernetes (EKS, AKS, GKE).
- Safe blue/green upgrades with Helm and Pulumi.
- Debugging the odd corner cases on OpenShift, vSphere, or air-gapped bare metal.
- Real-world migration stories from Calico, Flannel, and others.

###### More resources

- Hands-on lab: [Install Cilium on EKS](https://isovalent.com/resource-library/labs/) – spin up a sandbox in minutes.
- Book: [Cilium Up and Running](https://isovalent.com/books/cilium-up-and-running/)

###### Networking for Kubernetes

Kubernetes ships with an “allow-all” networking and leaves the policy enforcement and performance tuning to your CNI.
The Networking for Kubernetes playlist tackles everything from writing first NetworkPolicy to packet-level performance tuning.

###### What you’ll learn

- Writing a default-deny policy and graduating to L7 HTTP and DNS rules.
- Following a single packet through XDP, tc, and the eBPF conntracker (“Life of a Packet”).
- Benchmarking NodePort vs Maglev vs BGP mode and knowing when to switch.
- Pros and cons of popular CNIs and why teams migrate to Cilium.

###### More resources

- Hands-on lab: [Isovalent Enterprise for Cilium: Network Policies](https://isovalent.com/labs/cilium-network-policies/)
- eBook: [Kubernetes Networking and Cilium for the Network Engineer](https://isovalent.com/blog/post/introducing-the-new-kubernetes-networking-and-cilium-for-the-network-engineer-ebook/)

###### Mesh Networking

From in-cluster L7 routing to cross-cluster fail-over, Cilium delivers mesh traffic without sidecars or iptables overhead.
The two playlists below cover both layers: one for the Service Mesh inside a cluster,
and one for Cluster Mesh that stretches services across clusters.

###### What you’ll learn

- Stretching services across clusters with ClusterMesh fail-over.
- Integrating Envoy, and Gateway API without iptables overhead.

###### More resources

- Hands-on lab: [Cilium Cluster Mesh Lab](https://isovalent.com/labs/cilium-cluster-mesh/), [Cilium Gateway API](https://isovalent.com/labs/cilium-gateway-api/)
- Blog post: [Cilium Mesh - One Mesh to Connect Them All](https://isovalent.com/blog/post/introducing-cilium-mesh/)

##### Runtime Security with Tetragon

Need process-level enforcement and instant CVE detection? The Tetragon playlist shows how eBPF can hook every `execve`,
file write, and socket call before turning them into actionable policies.

###### What you’ll learn

- Building allow/deny rules at PID and binary granularity.
- Detecting supply-chain attacks (e.g. XZ Utils CVE) in real time.
- Capturing least-privilege baselines for production workloads.

###### More resources

- Hands-on lab: [Tetragon Getting Started](https://isovalent.com/labs/tetragon-getting-started/)
- Blog post: [Detecting the XZ Utils CVE with Tetragon](https://isovalent.com/blog/post/ebpf-tetragon-xz-utils-cve-policy/)
- Doc: Quick-start guide — [docs.tetragon.io](http://docs.tetragon.io)

##### All Things eBPF

Whether you’re writing custom programs or just want better observability, these playlists cover the kernel side of life.

###### What you’ll learn

- eBPF architecture, verifier workflow, and key helper functions.
- Building custom programs and loaders with libbpf.
- End-to-end observability with BPFTrace, Hubble, and continuous profiling.
- Real-world use cases beyond containers like live VM migration by Loophole labs.

###### More resources

- Hands-on lab: [Getting started with eBPF](https://isovalent.com/labs/ebpf-getting-started/)
- Book: [Learning eBPF](https://isovalent.com/books/learning-ebpf/), O'Reilly book by Liz Rice

##### Ecosystem & Community

Stay current with release overviews, conference recaps, and behind-the-scenes chats with Cilium maintainers.

###### What you’ll learn

- Highlights and release notes for every Cilium, Hubble, and Tetragon release.
- Previews and recaps from KubeCon, eBPF Summit and more.
- Contributor AMAs, lightning talks, and behind the scenes culture stories.

###### More resources

- Blog posts:
 - [Cilium Release 1.17](https://isovalent.com/blog/post/isovalent-networking-kubernetes-1-17/)
 - [KubeCon Europe 2025 Wrap-Up](https://isovalent.com/blog/post/kubecon-europe-2025-wrap-up/)

##### How to use this library

- Choose your theme above.
- Start the playlist episode.
- Dive deeper with the linked blog post, hands-on lab, or downloadable eBook.

###### Recap

| Theme | Playlist | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| **Cilium in Production** | [**eCHO Recaps: Cilium in the Clouds**](https://www.youtube.com/playlist?PLDg_GiBbAx-mmWNec1zMhEal5f6WurrJL) | Installing and operating Cilium on EKS, AKS and GKE. |
| | [**eCHO Recaps: Deploying Cilium**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-m6W4UfeVxFNT5vtPDvel4a) | Helm values, upgrades and CI/CD pipelines for day-2 ops. |
| | [**eCHO Recaps: Cilium with on-premise clusters**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-maWggVCCczWOWq4BxMZD5d) | Blue/green roll-outs and war stories from other CNIs (Calico, Flannel). |
| | [**eCHO Recaps: Migrating to Cilium**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-kTu6e-og1plHJAR1VNo6NN) | Blue/green roll-outs and war stories from other CNIs (Calico, Flannel). |
| **Networking for Kubernetes** | [**eCHO Recaps: Cilium Network Policy**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-nD4ps3RpgRxc3oum6t51yB) | From default-deny to L7 HTTP/DNS rules and policy tracing. |
| | [**eCHO Recaps: Life of a Packet**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-nBjiLpW5QDGEZzQL1Y80Tz) | Step-by-step tour of a packet through the Linux datapath. |
| | [**eCHO Recaps: CNIs**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-m2yEWqmYtD_yz1s7VLsHrl) | Comparative dives into Cilium vs Calico, Flannel and more. |
| **Runtime Security** | [**eCHO Recaps: Tetragon**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-n-Jo3KifgM57-9FS83pOd8) | Runtime process security and real-time CVE detection. |
| **Mesh Networking** | [**eCHO Recaps: Cilium Cluster Mesh**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-lTuFY7ho-N7NktqIgjhMDU) | Multi-cluster service discovery, fail-over and global policies. |
| | [**eCHO Recaps: Cilium Service Mesh**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-l5U7CdEHtg1DJL2kkOlPTE) | Sidecar-less service-mesh, Envoy filters and transparent mTLS. |
| **All things eBPF** | [**eCHO Recaps: Developing eBPF Applications**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-nD4ps3RpgRxc3oum6t51yB) | Verifier tricks, helper functions and user-space loaders. |
| | [**eCHO Recaps: eBPF-Powered Runtime Observability**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-mxtjKVjFTooh6TpOlMlNsM) | Tracing, profiling and flow visibility with Hubble & BPFTrace. |
| | [**eCHO Recaps: eBPF performance**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-k0eD5DDtci5MsHDVVOKotp) | Measuring, benchmarking, and tuning eBPF programs. |
| **Ecosystem & Community** | [**eCHO Recaps: Releases**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-nWmNVyJXSBc3ll-EDN70xx) | What’s new in every Cilium, Hubble and Tetragon release. |
| | [**eCHO Recaps: Conferences**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-mPS0Y5X86Gj_M1D_gY-Zf4) | Key take-aways and demos from KubeCon, eBPF Summit and beyond. |
| | [**eCHO Recaps: Community & Culture**](https://www.youtube.com/playlist?list=PLDg_GiBbAx-mAs5fGNhVcr6QnTTp9Uvyt) | AMAs, contributor spotlights and behind-the-scenes stories. |

###### See you live!

- Subscribe on [YouTube](https://www.youtube.com/@eBPFCilium) and tap the bell for stream reminders.
- Find all the notes for each episode on [HackMD](https://hackmd.io/@eCHO-live).
- Join our [Slack Workspace Cilium](https://slack.cilium.io/) for between episode chat.
- Submit your episode idea or guest request on [GitHub](https://github.com/isovalent/eCHO/issues/new/choose).

---

Happy watching and happy building with Cilium & eBPF!

#### Installing Cilium on EKS in ENI Mode

URL: https://cilium.io/blog/2025/06/19/eks-eni-install
Date: 2025-06-19
Categories: Community

_June 19th, 2025_

_Author: Paul Arah, Isovalent@Cisco_

##### Installing Cilium on EKS in ENI Mode

Amazon EKS is one of the most widely used managed Kubernetes services, chosen by many teams to offload the complexity of cluster management. Deploying Cilium on EKS unlocks powerful networking, security, and observability capabilities that go far beyond the default setup.
In this blog post series, we’ll explore the different ways to install Cilium on Amazon EKS. This first post focuses on deploying and configuring Cilium in ENI mode.

##### Why Cilium on EKS?

EKS provides a managed Kubernetes control plane backed by AWS scalable infrastructure. Since Kubernetes doesn't include a native network interface, networking plugins like Cilium and the default AWS VPC CNI provide this functionality. Cilium elevates EKS networking by offering a consistent, high-performance cloud native experience. Cilium seamlessly integrates with AWS's software-defined networking (SDN) while adding eBPF-based policy enforcement, advanced observability, multi-cluster capabilities, service mesh functionalities, all with superior performance. It is also noteworthy to mention that Cilium is the default built-in networking addon for EKS-Anywhere.

##### What is ENI Mode?

EKS clusters by default come with a set of networking add-ons, including Amazon VPC CNI, CoreDNS, and Kube-Proxy. These add-ons provide critical functionality that enables pod and service operations. Installing Cilium in ENI mode is the recommended installation method for Amazon EKS clusters. When running in ENI mode, Cilium manages ENIs and IP address allocation in place of the Amazon VPC CNI plugin. This method is highly performant, as it eliminates the need for overlay encapsulation or NAT.

##### Prerequisites

The following prerequisites need to be taken into account:

- An active AWS Account
- Install [kubectl](https://kubernetes.io/releases/download/#kubectl), [Helm](https://helm.sh/docs/intro/install/), [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/setting-up.html), [awscli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), and [Cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli)

##### Creating our EKS cluster

When using Cilium in EKS, to avoid conflicting behavior with the default networking add-ons, the default networking add-ons have to be disabled. There are generally two ways to achieve this: The first way is to create the cluster with the default networking add-ons and then delete or prevent these add-ons from running. In July 2024, [AWS announced](https://aws.amazon.com/about-aws/whats-new/2024/06/amazon-eks-cluster-creation-flexibility-networking-add-ons/) the ability to create EKS clusters without the default networking add-ons, and this is the approach we’ll be using in the tutorial.

- To get started, we’ll create a ClusterConfig file. In the file below, we’ve disabled kube-proxy and AWS VPC CNI.

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster1
  region: eu-west-2
  version: '1.30'

addonsConfig:
  disableDefaultAddons: true
addons:
  - name: coredns
```

- Using eksctl, create the cluster.

```shell
eksctl create cluster -f cilium-eks-config.yaml
```

- Check the status of the pods; the core-dns pods should be running in a pending state. This is because we currently do not have a CNI in the cluster.

##### Installing and Configuring Cilium

Next, we need to install Cilium in our cluster. We’ll do this using Helm.

- But first, we need to grab the value of our Kubernetes service host by running the command below:

```shell
kubectl cluster-info

Kubernetes control plane is running at https://xxxxxxxxxxxxxxxxxxxx.yyy.eu-west-2.eks.amazonaws.com
CoreDNS is running at https://xxxxxxxxxxxxxxx.gr7.eu-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
```

- Up next, we install Cilium using Helm with the command below

```shell
helm install cilium cilium/cilium --version 1.17.4 \
  --namespace kube-system \
  --set eni.enabled=true \
  --set ipam.mode=eni \
  --set egressMasqueradeInterfaces=eth+ \
  --set routingMode=native\
  --set kubeProxyReplacement=true\
  --set k8sServiceHost=xxxxxxxxxxxxxxxxx.yyy.eu-west-2.eks.amazonaws.com\
  --set k8sServicePort=443
```

**N/B**: _You should remove the “https://” from the Kubernetes services host value before including it in the value of the k8sServiceHost flag. Failure to do so would result in the Cilium agent crashing._

What these flags do:

| Option | Description |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `--set eni.enabled=true` | Enables the Elastic Network Interface (ENI) mode, which is specific to AWS environments. |
| `--set ipam.mode=eni` | Sets the IP Address Management (IPAM) mode to ENI, which allows Cilium to manage IP addresses. |
| `--set egressMasqueradeInterfaces=eth0` | Specifies the interface (`eth0`) on which egress masquerading (NAT) should be performed. |
| `--set routingMode=native` | Configures Cilium to use native routing instead of encapsulation (e.g., VXLAN or Geneve). |
| `--set kubeProxyReplacement=true` | Enables kube-proxy replacement, allowing Cilium to take over service load balancing and networking functions. |
| `--set k8sServiceHost=${API_SERVER_IP}` | Sets the IP address of the Kubernetes API server. |
| `--set k8sServicePort=${API_SERVER_PORT}` | Sets the port of the Kubernetes API server. |

- Next, we create a corresponding node group for our cluster.

```yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster1
  region: eu-west-2

managedNodeGroups:
  - name: ng-1
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    desiredCapacity: 2
    privateNetworking: true
```

**N/B**: _You might be curious why we did not create the EKS cluster with a managed node group in one go. Creating EKS clusters and corresponding node groups with add-ons disabled is currently not supported. This is why we created the cluster and subsequently added the node group._

To create the node group, run the command below

```yaml
eksctl create nodegroup -f nodegroup.yaml
```

- Afterwards, we can check the status of the nodes to ensure that they are in a ready state.

```
kubectl get nodes

NAME                                            STATUS   ROLES    AGE     VERSION
ip-192-168-125-58.eu-west-2.compute.internal    Ready    <none>   3h25m   v1.30.11-eks-473151a
ip-192-168-128-147.eu-west-2.compute.internal   Ready    <none>   3h25m   v1.30.11-eks-473151a

```

- Check if any AWS daemonsets are present in the cluster. You should see only the cilium and cilium-envoy daemonset

```
kubectl get ds -A

NAMESPACE     NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   cilium         2         2         2       2            2           kubernetes.io/os=linux   5m11s
kube-system   cilium-envoy   2         2         2       2            2           kubernetes.io/os=linux   5m11s

```

- Validate health check
 cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking and connectivity. You can check node-to-node health with cilium-health status:

```
kubectl -n kube-system exec ds/cilium -- cilium status

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
E0618 14:07:29.226958   97532 websocket.go:296] Unknown stream id 1, discarding message
KVStore:                 Disabled
Kubernetes:              Ok         1.30+ (v1.30.13-eks-5d4a308) [linux/amd64]
Kubernetes APIs:         ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    True   [ens5   192.168.190.239 fe80::de:f7ff:fe5a:d4ad (Direct Routing), ens6   192.168.186.8 fe80::62:48ff:fe8a:1133]
Host firewall:           Disabled
SRv6:                    Disabled
CNI Chaining:            none
CNI Config file:         successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                  Ok   1.17.4 (v1.17.4-55aecc0f)
NodeMonitor:             Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok
IPAM:                    IPv4: 4/12 allocated,
IPv4 BIG TCP:            Disabled
IPv6 BIG TCP:            Disabled
BandwidthManager:        Disabled
Routing:                 Network: Native   Host: Legacy
Attach Mode:             Legacy TC
Device Mode:             veth
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]
Controller Status:       32/32 healthy
Proxy Status:            OK, ip 192.168.167.197, 0 redirects active on ports 10000-20000, Envoy: external
Global Identity Range:   min 256, max 65535
Hubble:                  Ok              Current/Max Flows: 1184/4095 (28.91%), Flows/s: 8.84   Metrics: Disabled
Encryption:              Disabled
Cluster health:          2/2 reachable   (2025-06-18T13:06:28Z)
Name                     IP              Node   Endpoints
Modules Health:          Stopped(0) Degraded(0) OK(59)

```

- Cilium Connectivity Test (Optional)

The Cilium connectivity test deploys a series of services and deployments, and CiliumNetworkPolicy will use various connectivity paths to connect. Connectivity paths include with and without service load-balancing and various network policy combinations.

```
cilium connectivity test

ℹ️  Monitor aggregation detected, will skip some flow validation steps
ℹ️  Skipping tests that require a node Without Cilium
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for deployment cilium-test-1/client to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for deployment cilium-test-1/client2 to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for deployment cilium-test-1/echo-same-node to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for deployment cilium-test-1/client3 to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for deployment cilium-test-1/echo-other-node to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client2-57cf4468f-ncgvj to reach DNS server on cilium-test-1/echo-same-node-ff6f76ccc-4km59 pod...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client3-67f959dd9b-45k7d to reach DNS server on cilium-test-1/echo-same-node-ff6f76ccc-4km59 pod...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client-7b7776c86b-l4wsc to reach DNS server on cilium-test-1/echo-same-node-ff6f76ccc-4km59 pod...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client-7b7776c86b-l4wsc to reach DNS server on cilium-test-1/echo-other-node-6c497457bc-9vlq7 pod...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client2-57cf4468f-ncgvj to reach DNS server on cilium-test-1/echo-other-node-6c497457bc-9vlq7 pod...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client3-67f959dd9b-45k7d to reach DNS server on cilium-test-1/echo-other-node-6c497457bc-9vlq7 pod...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client-7b7776c86b-l4wsc to reach default/kubernetes service...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client2-57cf4468f-ncgvj to reach default/kubernetes service...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for pod cilium-test-1/client3-67f959dd9b-45k7d to reach default/kubernetes service...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for Service cilium-test-1/echo-other-node to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for Service cilium-test-1/echo-other-node to be synchronized by Cilium pod kube-system/cilium-fxr77
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for Service cilium-test-1/echo-other-node to be synchronized by Cilium pod kube-system/cilium-mxfz7
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for Service cilium-test-1/echo-same-node to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for Service cilium-test-1/echo-same-node to be synchronized by Cilium pod kube-system/cilium-fxr77
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for Service cilium-test-1/echo-same-node to be synchronized by Cilium pod kube-system/cilium-mxfz7
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for NodePort 192.168.130.253:30556 (cilium-test-1/echo-other-node) to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for NodePort 192.168.130.253:32358 (cilium-test-1/echo-same-node) to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for NodePort 192.168.190.239:30556 (cilium-test-1/echo-other-node) to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for NodePort 192.168.190.239:32358 (cilium-test-1/echo-same-node) to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for DaemonSet cilium-test-1/host-netns-non-cilium to become ready...
⌛ [cluster1.eu-west-2.eksctl.io] Waiting for DaemonSet cilium-test-1/host-netns to become ready...
ℹ️  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.17.4
🏃[cilium-test-1] Running 115 tests ...
[=] [cilium-test-1] Test [no-unexpected-packet-drops] [1/115]
..
[=] [cilium-test-1] Test [no-policies] [2/115]
.................................................
[=] [cilium-test-1] Skipping test [no-policies-from-outside] [3/115] (skipped by condition)
[=] [cilium-test-1] Test [no-policies-extra] [4/115]
............
[=] [cilium-test-1] Test [allow-all-except-world] [5/115]
........................
[=] [cilium-test-1] Test [client-ingress] [6/115]
......
[=] [cilium-test-1] Test [client-ingress-knp] [7/115]
......
[=] [cilium-test-1] Test [allow-all-with-metrics-check] [8/115]
......
[=] [cilium-test-1] Test [all-ingress-deny] [9/115]
............
[=] [cilium-test-1] Skipping test [all-ingress-deny-from-outside] [10/115] (skipped by condition)
[=] [cilium-test-1] Test [all-ingress-deny-knp] [11/115]
............
[=] [cilium-test-1] Test [all-egress-deny] [12/115]
........................
[=] [cilium-test-1] Test [all-egress-deny-knp] [13/115]
........................
[=] [cilium-test-1] Test [all-entities-deny] [14/115]
............
[=] [cilium-test-1] Test [cluster-entity] [15/115]
...
[=] [cilium-test-1] Skipping test [cluster-entity-multi-cluster] [16/115] (skipped by condition)
[=] [cilium-test-1] Test [host-entity-egress] [17/115]
......
[=] [cilium-test-1] Test [host-entity-ingress] [18/115]
....
[=] [cilium-test-1] Test [echo-ingress] [19/115]
......
[=] [cilium-test-1] Skipping test [echo-ingress-from-outside] [20/115] (skipped by condition)
[=] [cilium-test-1] Test [echo-ingress-knp] [21/115]
......
[=] [cilium-test-1] Test [client-ingress-icmp] [22/115]
......
[=] [cilium-test-1] Test [client-egress] [23/115]
......
[=] [cilium-test-1] Test [client-egress-knp] [24/115]
......
[=] [cilium-test-1] Test [client-egress-expression] [25/115]
......
[=] [cilium-test-1] Test [client-egress-expression-port-range] [26/115]
......
[=] [cilium-test-1] Test [client-egress-expression-knp] [27/115]
...E0618 14:50:03.068727   10146 websocket.go:296] Unknown stream id 1, discarding message
...
[=] [cilium-test-1] Test [client-egress-expression-knp-port-range] [28/115]
......
[=] [cilium-test-1] Test [client-with-service-account-egress-to-echo] [29/115]
......
[=] [cilium-test-1] Test [client-with-service-account-egress-to-echo-port-range] [30/115]
......
[=] [cilium-test-1] Test [client-egress-to-echo-service-account] [31/115]
......
[=] [cilium-test-1] Test [client-egress-to-echo-service-account-port-range] [32/115]
......
[=] [cilium-test-1] Test [to-entities-world] [33/115]
.........
[=] [cilium-test-1] Test [to-entities-world-port-range] [34/115]
.........
[=] [cilium-test-1] Test [to-cidr-external] [35/115]
......
[=] [cilium-test-1] Test [to-cidr-external-knp] [36/115]
......
[=] [cilium-test-1] Skipping test [seq-from-cidr-host-netns] [37/115] (skipped by condition)
[=] [cilium-test-1] Test [echo-ingress-from-other-client-deny] [38/115]
..........
[=] [cilium-test-1] Test [client-ingress-from-other-client-icmp-deny] [39/115]
............
[=] [cilium-test-1] Test [client-egress-to-echo-deny] [40/115]
............
[=] [cilium-test-1] Test [client-egress-to-echo-deny-port-range] [41/115]
............
[=] [cilium-test-1] Test [client-ingress-to-echo-named-port-deny] [42/115]
E0618 14:53:36.939961   10146 websocket.go:296] Unknown stream id 1, discarding message
....
[=] [cilium-test-1] Test [client-egress-to-echo-expression-deny] [43/115]
....
[=] [cilium-test-1] Test [client-egress-to-echo-expression-deny-port-range] [44/115]
....
[=] [cilium-test-1] Test [client-with-service-account-egress-to-echo-deny] [45/115]
....
[=] [cilium-test-1] Test [client-with-service-account-egress-to-echo-deny-port-range] [46/115]
....
[=] [cilium-test-1] Test [client-egress-to-echo-service-account-deny] [47/115]
..
[=] [cilium-test-1] Test [client-egress-to-echo-service-account-deny-port-range] [48/115]
..
[=] [cilium-test-1] Test [client-egress-to-cidr-deny] [49/115]
......
[=] [cilium-test-1] Test [client-egress-to-cidrgroup-deny] [50/115]
......
[=] [cilium-test-1] Test [client-egress-to-cidrgroup-deny-by-label] [51/115]
......
[=] [cilium-test-1] Test [client-egress-to-cidr-deny-default] [52/115]
......
[=] [cilium-test-1] Skipping test [clustermesh-endpointslice-sync] [53/115] (skipped by condition)
[=] [cilium-test-1] Test [health] [54/115]
..
[=] [cilium-test-1] Skipping test [north-south-loadbalancing] [55/115] (Feature node-without-cilium is disabled)
[=] [cilium-test-1] Test [pod-to-pod-encryption] [56/115]
.
[=] [cilium-test-1] Skipping test [pod-to-pod-with-l7-policy-encryption] [57/115] (Feature encryption-pod is disabled)
[=] [cilium-test-1] Skipping test [pod-to-pod-encryption-v2] [58/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [pod-to-pod-with-l7-policy-encryption-v2] [59/115] (skipped by condition)
[=] [cilium-test-1] Test [node-to-node-encryption] [60/115]
...
[=] [cilium-test-1] Skipping test [seq-egress-gateway] [61/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [egress-gateway-excluded-cidrs] [62/115] (Feature enable-ipv4-egress-gateway is disabled)
[=] [cilium-test-1] Skipping test [seq-egress-gateway-with-l7-policy] [63/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [pod-to-node-cidrpolicy] [64/115] (Feature cidr-match-nodes is disabled)
[=] [cilium-test-1] Skipping test [north-south-loadbalancing-with-l7-policy] [65/115] (Feature node-without-cilium is disabled)
[=] [cilium-test-1] Skipping test [north-south-loadbalancing-with-l7-policy-port-range] [66/115] (Feature node-without-cilium is disabled)
[=] [cilium-test-1] Test [echo-ingress-l7] [67/115]
..................
[=] [cilium-test-1] Test [echo-ingress-l7-via-hostport] [68/115]
......
[=] [cilium-test-1] Test [echo-ingress-l7-named-port] [69/115]
..................
[=] [cilium-test-1] Test [client-egress-l7-method] [70/115]
..................
[=] [cilium-test-1] Test [client-egress-l7-method-port-range] [71/115]
.............E0618 14:59:09.879325   10146 websocket.go:296] Unknown stream id 1, discarding message
.....
[=] [cilium-test-1] Test [client-egress-l7] [72/115]
...............
[=] [cilium-test-1] Test [client-egress-l7-port-range] [73/115]
...............
[=] [cilium-test-1] Test [client-egress-l7-named-port] [74/115]
...............
[=] [cilium-test-1] Test [client-egress-tls-sni] [75/115]
.........
[=] [cilium-test-1] Test [client-egress-tls-sni-denied] [76/115]
.........
[=] [cilium-test-1] Test [client-egress-l7-tls-headers-sni] [77/115]
...
[=] [cilium-test-1] Test [client-egress-l7-tls-headers-other-sni] [78/115]
...
[=] [cilium-test-1] Test [client-egress-l7-set-header] [79/115]
......
[=] [cilium-test-1] Test [client-egress-l7-set-header-port-range] [80/115]
......
[=] [cilium-test-1] Skipping test [echo-ingress-auth-always-fail] [81/115] (Feature mutual-auth-spiffe is disabled)
[=] [cilium-test-1] Skipping test [echo-ingress-auth-always-fail-port-range] [82/115] (Feature mutual-auth-spiffe is disabled)
[=] [cilium-test-1] Skipping test [echo-ingress-mutual-auth-spiffe] [83/115] (Feature mutual-auth-spiffe is disabled)
[=] [cilium-test-1] Skipping test [echo-ingress-mutual-auth-spiffe-port-range] [84/115] (Feature mutual-auth-spiffe is disabled)
[=] [cilium-test-1] Skipping test [pod-to-ingress-service] [85/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [pod-to-ingress-service-allow-ingress-identity] [86/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [pod-to-ingress-service-deny-all] [87/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [pod-to-ingress-service-deny-backend-service] [88/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [pod-to-ingress-service-deny-ingress-identity] [89/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [pod-to-ingress-service-deny-source-egress-other-node] [90/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [outside-to-ingress-service] [91/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [outside-to-ingress-service-deny-all-ingress] [92/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [outside-to-ingress-service-deny-cidr] [93/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Skipping test [outside-to-ingress-service-deny-world-identity] [94/115] (Feature ingress-controller is disabled)
[=] [cilium-test-1] Test [dns-only] [95/115]
...............
[=] [cilium-test-1] Test [to-fqdns] [96/115]
............
[=] [cilium-test-1] Skipping test [pod-to-controlplane-host] [97/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [pod-to-k8s-on-controlplane] [98/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [pod-to-controlplane-host-cidr] [99/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [pod-to-k8s-on-controlplane-cidr] [100/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [local-redirect-policy] [101/115] (Feature enable-local-redirect-policy is disabled)
[=] [cilium-test-1] Skipping test [local-redirect-policy-with-node-dns] [102/115] (skipped by condition)
[=] [cilium-test-1] Test [pod-to-pod-no-frag] [103/115]
.
[=] [cilium-test-1] Skipping test [seq-bgp-control-plane-v1] [104/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [seq-bgp-control-plane-v2] [105/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [multicast] [106/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [strict-mode-encryption] [107/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [strict-mode-encryption-v2] [108/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [host-firewall-ingress] [109/115] (skipped by condition)
[=] [cilium-test-1] Skipping test [host-firewall-egress] [110/115] (skipped by condition)
[=] [cilium-test-1] Test [seq-client-egress-l7-tls-deny-without-headers] [111/115]
...
[=] [cilium-test-1] Test [seq-client-egress-l7-tls-headers] [112/115]
...
[=] [cilium-test-1] Test [seq-client-egress-l7-extra-tls-headers] [113/115]
......
[=] [cilium-test-1] Test [seq-client-egress-l7-tls-headers-port-range] [114/115]
...
[=] [cilium-test-1] Test [check-log-errors] [115/115]
..................

✅ [cilium-test-1] All 72 tests (630 actions) successful, 43 tests skipped, 0 scenarios skipped.

```

##### Conclusion

In the first part of our series, we walked through the deployment of Cilium on Amazon EKS in ENI mode. This configuration offers native AWS networking performance while leveraging Cilium's powerful eBPF-based capabilities. From creating an EKS cluster with the default networking add-ons disabled to validating Cilium's health, you've now seen what it takes to get a clean, kube-proxy-free EKS cluster up and running with Cilium managing ENIs and IP allocation.

In the next part of this series, we'll cover how to install Cilium in overlay mode and explore CNI chaining, giving you more flexibility to integrate Cilium with existing CNIs or tailor your cluster's networking model to suit hybrid environments. Stay tuned!

#### Application-Aware Security Policies with Cilium Layer 7 Network Policies

URL: https://cilium.io/blog/2025/05/20/cilium-l7-policies
Date: 2025-05-20
Categories: Community

_May 20th, 2025_

_Author: Paul Arah, Isovalent@Cisco_

In Kubernetes, network policies define how pods can communicate with each other and other network endpoints. Network policies are a key building block for securing Kubernetes workloads. The standard Kubernetes network policy is limited to Layer 3 and Layer 4 of the OSI model. That means traffic can be filtered based on IP address, ports, and protocols, but not based on the actual application-layer content, such as HTTP methods or URLs. Cilium extends this model to support Layer 7 policies that understand and control traffic based on application-level protocol semantics. These include HTTP, Kafka, and DNS protocols. With L7 policies, we can enforce rules like “only allow GET /public HTTP requests” or “only permit Kafka produce access to a specific topic.” This level of specificity enables you to lock down workloads with precision without disrupting valid business logic.

In this blog post, we’ll explore how Cilium Layer 7 network policies work across HTTP, gRPC, Kafka, and DNS protocols. If you’re new to Cilium network policies, check out [Introduction to Cilium Network Policies](https://isovalent.com/blog/post/intro-to-cilium-network-policies/) and [Tutorial: Cilium Network Policy in Practice](https://isovalent.com/blog/post/tutorial-cilium-network-policy/) for a primer on Cilium network policies.

##### Why Layer 7 Policies Matter

In modern microservice architectures, services rarely communicate over a low-level protocol like TCP or UDP alone. Instead, they expose APIs over a high-level protocol such as HTTP or gRPC. These high-level protocols carry rich protocol-specific semantics such as HTTP methods, Kafka topics, or gRPC service names representing the true intent behind the application communication.

Standard Layer 3/4 network policies, which only understand IP addresses and ports, are oblivious to this communication intent. L7 network policies in Cilium address this gap by enabling policy verdicts based on the content and context of the traffic, not just its source or destination. This allows us to go beyond saying, “Service A can only talk to Service B on port 8080,” to expressing rules like, “Service A can only perform a POST to /login path” or “Service B can only consume from the order-events Kafka topic”. By aligning the policy enforcement with the API behaviour, Cilium Layer 7 policies ensure that services can only perform the specific operations they are designed for, eliminating implicit access to sensitive resources. This approach is consistent with zero-trust principles, where services are granted the least privileged access to resources. By providing fine-grained control over the exact HTTP methods, gRPC calls, and Kafka topics a service can make, L7 policies reduce the attack surface. Even if a pod is compromised, the attacker’s ability to abuse service-to-service communication is constrained at the protocol level.

##### HTTP Policies

For HTTP, Cilium Layer 7 policies allow you to define specific methods, paths, host headers, and even required HTTP headers. These rules are enforced by a node-local Envoy proxy that inspects and authorizes traffic before it reaches the application.
A common use case is restricting access to specific REST endpoints. For example, you can allow only _GET /public_ requests to your app while blocking all other paths or methods. Here's an example:

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'public-api-only'
spec:
  endpointSelector:
    matchLabels:
      app: service
  ingress:
    - fromEndpoints:
        - matchLabels:
            env: prod
      toPorts:
        - ports:
            - port: '80'
              protocol: TCP
          rules:
            http:
              - method: 'GET'
                path: '/public'
```

This policy ensures only HTTP GET requests to _/public_ are allowed on port 80, effectively restricting access to private API paths or other HTTP methods like _POST_ or _PUT_. Unlike Layer 3/4 policies, L7 violations return protocol-specific errors (e.g., HTTP 403), providing a better user experience.

##### gRPC Policies

Now lets see how Cilium extends this model to gRPC where HTTP/2 semantics allow similar fine grained control. Because gRPC is built on top of HTTP/2, Cilium supports gRPC Layer 7 policy enforcement via its HTTP parser. This means you can apply HTTP-based rules to gRPC traffic, allowing you to control access to specific services and methods.
For example, you can restrict access to specific gRPC service methods _(like helloworld.Greeter/SayHello)_ using the HTTP path filter, since gRPC requests are encoded as HTTP/2 POST requests with paths like _/package.service/method_. Here's a sample policy:

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'grpc-restrict-methods'
spec:
  endpointSelector:
    matchLabels:
      app: grpc-server
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: grpc-client
      toPorts:
        - ports:
            - port: '50051'
              protocol: TCP
          rules:
            http:
              - method: 'POST'
                path: '/helloworld.Greeter/SayHello'
```

This ensures only authorized gRPC clients can call the _SayHello_ method on the _helloworld.Greeter_ service. All other gRPC methods, including potentially sensitive or admin-only endpoints, will be denied with a [gRPC status code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html) 7 (PERMISSION_DENIED).

##### Kafka Policies

Cilium also supports Kafka-specific Layer 7 rules, allowing us to define what Kafka operations are permitted at the Kafka protocol level. You can restrict access based on Kafka topics and roles, like _produce_ or _consume_, or match specific apiKeys for a request as per the Kafka protocol reference.
Here's how you can allow only the _produce_ role for specific Kafka topics:

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'kafka-produce'
spec:
  endpointSelector:
    matchLabels:
      app: kafka
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: empire-hq
      toPorts:
        - ports:
            - port: '9092'
              protocol: TCP
          rules:
            kafka:
              - role: 'produce'
                topic: 'empire-announce'
```

Kafka support is currently in beta, but is already powerful enough to enforce complex Kafka-aware security policies.

##### DNS Policies

Cilium’s DNS-aware L7 policies allow you to filter DNS queries directly _(e.g., only allow queries to \*.cilium.io)_ and dynamically enforce L3 policies based on the resolved IPs.
Here’s an example policy that allows only DNS queries for specific domains and restricts egress traffic based on the returned IPs:

```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: 'dns-filter'
spec:
  endpointSelector:
    matchLabels:
      any:org: alliance
  egress:
    - toEndpoints:
        - matchLabels:
            'k8s:io.kubernetes.pod.namespace': kube-system
            'k8s:k8s-app': kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: ANY
          rules:
            dns:
              - matchPattern: '*.cilium.io'
    - toFQDNs:
        - matchPattern: '*.cilium.io'
      toPorts:
        - ports:
            - port: '443'
              protocol: TCP
```

This combination of DNS query control and dynamic IP resolution ensures your applications only communicate with known and verified domains, reducing the risk of data exfiltration to malicious endpoints.

##### Conclusion

Cilium’s Layer 7 network policies give Kubernetes operators powerful, application-aware security controls that go far beyond the capabilities of IP and port-based network policies. L7 policies enable precise, protocol security control that is aligned with how applications actually communicate. In this post, we explored Cilium Layer 7 network policies across HTTP, gRPC, Kafka, and DNS; with some base examples of policy enforcement. As Kubernetes environments mature, adopting L7-aware security polices becomes less of a “nice to have” and more of a foundational requirement for zero-trust architectures.

If you have questions or feedback, reach out via the network policy channel in the [Cilium Slack](https://slack.cilium.io).

##### Additional Resources

- [Securing a Kafka Cluster - Cilium docs](https://docs.cilium.io/en/stable/security/kafka/)
- [Securing gRPC - Cilium docs](https://docs.cilium.io/en/stable/security/grpc/)
- [eCHO Episode 180: Enforcing Kafka-aware Security Policies with Cilium](https://www.youtube.com/watch?v=IIeUIcwwhmE)
- [Cloud Native Live: Using Cilium to enforce gRPC-aware security policies](https://www.youtube.com/watch?v=00b1KN3141k&t=1s)

#### Agentic Bee: How to get AI Agents to talk to Tetragon?

URL: https://cilium.io/blog/2025/05/15/tetragon-ai-agents-canopus
Date: 2025-05-15
Categories: Community

_May 15th, 2025_

_Authors: Himal Kumar, Bhaskar Dutta, Arman Pashamokhtari @CanopusAI.com_

##### Overview

Canopus, a cloud-native cybersecurity company, faced the challenge of managing container vulnerabilities across a complex hybrid cloud environment. By leveraging eBPF-powered observability from Tetragon and autonomous workflows using AI agents, Canopus significantly reduced security noise and enabled highly accurate runtime vulnerability prioritization. This transformation enhanced operational efficiency for small DevSecOps teams while strengthening security posture across hundreds of containers and multiple Kubernetes clusters.

##### Challenge

Operating in a hybrid cloud infrastructure with development and production workloads, Canopus confronted several key challenges:

- **Alert Fatigue**: Traditional CI/CD vulnerability scanners flooded teams with alerts – often dozens per container – making it difficult for smaller security teams to identify and act on the most critical issues.
- **Lack of Runtime Context**: Static analysis tools failed to provide insight into whether vulnerabilities were actually exploitable in real-world scenarios.
- **Manual Processes**: Security insights required manual dashboard interaction, resulting in inefficient workflows and slower response times.

##### Solution

Canopus developed a system called FlowPulse AI, which integrates eBPF telemetry from Tetragon with autonomous AI agent workflows. This system provides high-fidelity runtime data, contextual intelligence, and automated analysis to accurately prioritize actionable vulnerabilities.

**Key Components:**

- **Tetragon (eBPF-powered)**: Captures four “golden signals” of security observability – Network, Process, FileSystem, and Identity – at the kernel level.
- **Contextualization Engine**: Uses LLMs to classify container roles, and behavioural baselining algorithms to assess privileges, trace data flows, and correlate vulnerabilities to real-time usage.
- **Grype + Syft Integration**: Scans for static vulnerabilities and builds SBOMs to monitor actual use of vulnerable packages.
- **AI Agentic Workflow (CrewAI + Claude)**:
 - Funnels vulnerabilities from general to runtime-exploitable.
 - Assesses exposure using asset classification and network flow data.
 - Tracks remediation and change history.
 - Generates concise risk reports automatically.

##### Results

- **100x Reduction in Immediate Action Items**: By filtering out non-exploitable vulnerabilities, Canopus cut down the number of alerts requiring immediate action by a factor of 100.
- **Faster Decision-Making**: AI-generated reports are delivered in one-to-two minutes, consuming about $0.13 to $0.23 per report.
- **Operational Efficiency**: Enabled a small security team to proactively manage a large, dynamic container environment.
- **Comprehensive Visibility**: Continuous monitoring delivers an always-current view of runtime behavior across infrastructure.

##### Implementation

**High-fidelity data collection with Tetragon:**
FlowPulse uses high-fidelity eBPF telemetry from Tetragon to collect runtime events from the cloud-native infrastructure. Tetragon is an open-source CNCF project that enables extraction of four golden signals of security observability on Network, Processes, FileSystems and Identity, by configuring dynamic Tracing Policies.

**Data Contextualisation in real-time:**
FlowPulse takes a stream of events from Tetragon through its gRPC client, and processes them to generate various baseline and runtime profiles, such as process and network baselines, data flow patterns, run-time malicious evidences, vulnerability scans, and asset classifications. Key contextual factors that are used specifically for vulnerability prioritisation are:

- **Asset Classification**: FlowPulse leverages a local LLM (Llama) to classify each container based on its activity into its role, such as database, monitoring, or message queue.
- **Data Flows**: To assess risk, FlowPulse constructs network flows by processing network events and enriching them with contextual details such as src/dst containers and domain name. This is done using kprobes tcp_recvmsg & tcp_sendmsg. We additionally implement stateful tracking & aggregations to reduce the sheer volume of network events.
- **Privileges**: Some containers have elevated privileges that allow them to access and modify host resources. For example, the Linux NET_ADMIN capability enables containers to configure network interfaces and traffic control settings, while sharing host namespaces like PID grants direct access to host processes. FlowPulse detects privileged containers by analysing Tetragon’s runtime events and incorporates this information into its contextual data to further assess risk.
- **Static Vulnerabilities**: FlowPulse utilises Grype to scan container images running in monitored environments. Grype identifies vulnerabilities in installed software packages and provides metadata such as CVE ID, description, severity, package name, package type, installed version, and fixed version (if available). Since Grype’s vulnerability descriptions lack a standard format, we use the local Llama model to generate concise yet accurate summaries, ensuring that vulnerability data remains both informative and AI-friendly.
- **Runtime Vulnerabilities**: Static vulnerabilities do not indicate whether those vulnerabilities pose an active risk. A package becomes a real threat only if it is actively used by running processes. To determine this, FlowPulse uses Syft to generate a Software Bill of Materials (SBOM) for containers with vulnerable packages. The SBOM provides details on the location of installed software and the files associated with them. FlowPulse then dynamically updates Tetragon Tracing Policies to monitor processes that access vulnerable files. When a process (e.g., a Java application) loads a vulnerable package (e.g., Log4J), Tetragon generates an event containing both process details and the accessed file. FlowPulse then correlates this information with known vulnerabilities, adding critical runtime context that is essential for prioritising vulnerabilities.

**AI Agents Workflows**

Traditional cybersecurity tools rely heavily on dashboards and visual interfaces, requiring users to manually search through data, drill down, and write queries (e.g., SQL, KQL, SPL) to extract meaningful insights. With recent developments in AI Agents frameworks and LLM Models, we leverage an autonomous Agentic workflow to produce the final actionable report, while providing interactivity to the user.
FlowPulse utilizes the ReAct (Reason & Act) agentic framework from CrewAI to build out the workflow, and uses the Anthropic Claude LLM model for reasoning. The Agentic workflow consists of four main steps and is triggered whenever there is a change/update to the application environment:

- AI Agents first generate the severity funnel by filtering down vulnerabilities from Critical to Fixable to Privileged to RuntimeActive, significantly reducing the scope of investigation.
- Second, Agentic Crew assess risk by utilising asset classifications and network data flows to determine the exposure.
- Third, the crew correlates findings from previous runs to determine what has caused a new vulnerability to appear, or whether a previous alert has been actioned and fixed. This is done by correlating vector and summary indexes on the LLM generated reports.
- Finally, the Agents generate a concise context report outlining the scope of risk and a set of actions that require immediate attention.

The system also consists of a no-dashboard interactive UI using which the security teams can perform further investigative queries to understand the context better, or prompt the agents to generate reports for the next set of vulnerabilities having fixed all that were flagged.
The exact number of LLM calls per report is unpredictable, however, we observed that the crew invokes the underlying LLM approximately 10 times per report, with each report taking between one to two minutes to generate. LLM tokens consumption varies significantly, ranging from 30,000 to 60,000 tokens, which results in a cost of $0.13 to $0.23 USD per report.

##### Why eBPF?

eBPF was chosen for its unparalleled ability to observe and act on runtime events directly in the kernel, capturing critical signals such as process, network, and filesystem activity with minimal overhead. This high-fidelity data is foundational to Canopus’s proactive security architecture and the effectiveness of its AI-driven workflows.

##### Next Steps

Canopus is expanding its use of eBPF with Tetragon to develop a runtime threat detection system. This new system will: (a) Leverage behavioral evidence to detect anomalies and malicious activities; (b) Use AI agents to classify threats against the Kubernetes Threat Matrix; and (c) Enforce real-time policies (e.g., kill processes, block commands) directly in the kernel using Tetragon's enforcement features.

#### Sinad User Story: Delivering Security and Observability for Workloads with Confidence

URL: https://cilium.io/blog/2025/04/15/tetragon-user-sinad
Date: 2025-04-25
Categories: Community

_April 25th, 2025_

_Author: Paul Arah, Isovalent@Cisco_

**Challenge**

SINAD, a company specializing in Cloud, DevOps, and Kubernetes, with a focus on making Kubernetes management seamless, provides consulting and in-house solutions to help clients optimize their cloud-native infrastructure. Their flagship product, EzyKube, simplifies Kubernetes operations. However, they also knew that customers were looking for granular control over security and observability while maintaining ease of use.

SINAD identified a key pain point among its target clients: The need for granular control over their data and operations, with customizable options for observability and security. SINAD sought a solution to this challenge while providing robust observability and security capabilities. They needed a tool that could integrate seamlessly with their Kubernetes-focused product, EzyKube, and offer real-time insights and policy enforcement.

**Solution**

The SINAD team discovered Tetragon through Cilium, which they already used for Kubernetes networking. They chose Tetragon for security observability and runtime enforcement capabilities. KubeKnight, one key component of the EzyKube platform responsible for security and observability, integrates Tetragon to deploy customizable security policies, enabling real-time monitoring and enforcement. With Tetragon, SINAD team can deploy policies covering a wide range of security and observability use cases for its organization and target clients.

**Impact**

Using Tetragon’s monitoring and enforcement capabilities, SINAD is able to effectively observe and secure its client’s environments and workloads. Tetragon has enabled SINAD to offer its clients a wide array of benefits, such as deep visibility into workloads, network traffic, and file integrity while maintaining fine-tuned control over security policies, leading to increased trust in SINAD’s solutions. Tetragon’s unique efficiency and ease of use empowers SINAD to equip organizations to maximize their return on investment in Kubernetes and cloud native technologies. Integrating Tetragon into the EzyKube platform also helps teams streamline compliance, making it easier to meet regulatory requirements. In summary, Tetragon enables SINAD’s customers to focus on security outcomes rather than infrastructure challenges.

##### Simplifying Security and Observability for Kubernetes

SINAD was founded with a mission to make Kubernetes easier to manage, providing consulting services and developing in-house solutions like EzyKube, their flagship Kubernetes management platform. From the outset, SINAD wanted to provide deeper control over observability and security capabilities.

eBPF offered the visibility and enforcement capabilities SINAD needed, but using it directly was too complex and resource-intensive. _“We knew eBPF was the ideal solution for our security and observability needs,”_ explained Arezki OUHENIA, co-founder of SINAD. _“But managing maps, using helpers, and writing safe kernel code is extremely complex. It’s the kernel after all.”_

The challenge of using eBPF directly extended beyond the scope and specialty of the SINAD team. A tool that required manual eBPF programming or kernel modifications would be impractical. “

##### Discovering Tetragon Through Cilium

Initially, SINAD wasn’t aware of Tetragon. When they discovered Tetragon's runtime security observability capabilities, the team had already been using Cilium for Kubernetes networking. Arezki OUHENIA recalled, _“At first, we didn’t know about Tetragon. We found it through using Cilium, and it turned out to be exactly what we needed.”_

The powerful and flexible abstraction Tetragon provides on top of eBPF eliminates the need for the SINAD team to write eBPF programs directly for the EzyKube platform while still providing all the immense benefits and flexibility eBPF brings to cloud native security. Tetragon made it easy for SINAD to leverage eBPF for observability and security without requiring deep eBPF or kernel expertise from the team. Arezki, highlighted that Tetragon’s ability to simplify eBPF was a game changer. _“Rather than dealing with the complexities of writing eBPF programs, we wanted to focus on delivering value for our clients, and we needed a tool that made eBPF easy to use. To SINAD, Tetragon is eBPF made simple. It provides precise observability and efficient security, exactly what we needed.”_

The simplicity of collecting various security significant events with Tetragon meant that the SINAD team could be as broad or granular as possible with the data they collected depending on the use case. In Arezki’s words: _“Tetragon gave us the flexibility to decide what data we want to collect without enforcing any restrictions on us.” This flexibility empowered SINAD Team to customize their solution to the varying needs and requirements of their target clients._

##### Real-Time Observability Without Overhead

Tetragon’s efficiency and lightweight design were also major selling points. Unlike many, it didn’t introduce unnecessary performance overhead. _“Tetragon provides real-time visibility into workloads and allows us to enforce policies at the kernel level,”_ Arezki OUHENIA explained. _“That kind of deep insight was something we needed but couldn’t easily achieve before.”_

Another key advantage was network observability. _“Being able to track and enforce network policies in real time without introducing latency is crucial,”_ Arezki OUHENIA noted. Tetragon’s fine-grained controls offer several capabilities, for example, they enable SINAD to monitor live production sessions, promptly detect unauthorized network connections, and log critical file modifications in real time.

##### Building a Tetragon Enabled Platform

Tetragon is the underlying driver of KubeKnight, the component of the EzyKube platform responsible for observability and security. KubeKnight integrates with Tetragon to provide fine-grained monitoring, runtime enforcement, and security observability. This allowed SINAD to address use cases such as file integrity monitoring, network observability, and privilege escalation detection.

_“Tetragon enables us to provide deep insights, visibility, and control,”_ Arezki OUHENIA explained. _“With Tetragon, we’ve been able to deliver a solution that our clients feel confident and at ease with. Providing this high level of assurance to our clients is a key measure of success, and Tetragon is behind this.”_

One of KubeKnight’s key features is enforcement. Tetragon enables SINAD to block security violations in real time. “Tetragon allows us to block activities at runtime, preventing misconfigurations or malicious actions before they cause damage.”

By integrating Tetragon into EzyKube, SINAD streamlined compliance efforts, which is core to their offering.

##### Future Plans

SINAD plans to expand its use of Tetragon within EzyKube by introducing more user-friendly, easily accessible Kubernetes and cloud computing capabilities.

_“Tetragon stands as a powerhouse within EzyKube, delivering real-time enforcement, deep observability, and streamlined security; Maximizing return on investment (ROI) while reducing operational complexity,”_ said the co-founder of SINAD, Arezki OUHENIA

#### Tetragon User Story: Why This Social Networking Company Made Tetragon a Default for their Kubernetes Clusters

URL: https://cilium.io/blog/2025/04/15/tetragon-social-networking-user-story
Date: 2025-04-15
Categories: Community

_April 15th, 2025_

_Author: Paul Arah, Isovalent@Cisco_

_This user story comes from a company that provides communication and social networking services._

**Problem**

As this social networking company’s Kubernetes footprint expanded from 12 to 35 clusters in just two years, its security and observability tools struggled to keep up. Like most companies, the ratio of security engineers to developers made it nearly impossible to manually ensure that every service adhered to the company's security best practices. The open source Osquery, initially deployed for visibility, frequently caused production outages due to high resource consumption and contention and noisy neighbor issues. They needed a cloud native, low-overhead solution capable of delivering detailed security observability while seamlessly integrating with its Kubernetes-based infrastructure.

**Solution**

After evaluating various tools and doing in-depth performance overhead testing based on metrics such as process overhead, network monitoring overhead, and file access overhead, this social networking company adopted Tetragon, an eBPF-powered runtime security observability tool, for its Kubernetes environments. Tetragon’s daemonset-based deployment model ensured consistent coverage across all clusters, while its granular configuration allowed them to focus on high-value process events without overwhelming their logging pipeline. Critical features like ARM architecture support, enforcement capabilities, and integration with their data lake via a gRPC logging pipeline made Tetragon the ideal fit for their evolving needs.

With 100 percent coverage of Kubernetes clusters at this company, Tetragon is now part of the bare minimum requirements for every new Kubernetes cluster created there, regardless of whether it is a development, staging, or production cluster.

**Outcome**

Tetragon transformed the company’s approach to security observability by providing full visibility into process events with minimal impact on resources. Prior to migrating to Tetragon, production outages due to performance impacts from security tools were a common challenge. Tetragon’s low overhead has translated into adjacent teams not even realizing that a security tool is running in the background. Production workloads are shielded from disruptions, enabling their security team to focus on proactive threat detection. This has resulted in deeper trust across the organization, knowing their infrastructure is continuously monitored and safeguarded against evolving threats. With comprehensive coverage across all clusters, the team now identifies anomalies faster and enforces better security practices without compromising performance or developer productivity.

##### The Journey to Reliable Security Observability with Tetragon

The company is a network of thousands of vibrant communities that connect millions of users daily to discuss topics they’re passionate about. Behind the scenes, its infrastructure has grown significantly to meet its users' demands. Over the past few years, the company’s Kubernetes adoption surged from 12 clusters to 35, with over 80% of workloads now running on Kubernetes. Supporting this growth is a small but dedicated infrastructure security team of five engineers tasked with securing the environment for over 1,200 developers. The rapid scale of Kubernetes adoption revealed critical limitations in the company’s existing security observability stack. Osquery, an open source tool previously used to monitor processes and network activity, became a source of frustration. As a security engineer at this company explained, _“Osquery caused numerous outages in production. It was consuming a lot of resources, creating contention issues for our workloads. The tool ended up being a noisy neighbor that disrupted the very systems it was supposed to protect.”_ Even switching from a self hosted to a managed solution didn’t help mitigate the problem.

Performance wasn’t the only issue. Osquery’s high data volume overwhelmed their logging pipeline. Generating excessive logs from test clusters alone, Osquery’s default configuration also provided little actionable insight. _“With Osquery, we often found ourselves sifting through stacks of irrelevant data to find something meaningful,”_ The security engineer added. The lack of Kubernetes-native integration and real-time enforcement capabilities further compounded the challenges.

##### Testing and Performance Benchmarking Security Tools

Seeking a modern, Kubernetes-aware solution, the company’s security team evaluated multiple options. Performance testing and benchmarking of tools primarily focused on the overhead introduced for process monitoring, file access, and network monitoring. Tetragon came to the top of the list with the least overhead.

_“Tetragon’s overhead compared to any other tool has been very minimal. We did in-depth performance testing in terms of process monitoring, file access monitoring, and network monitoring. We tested the baseline behavior without any security tool. Then we compared it with all the security tools we were testing against, and it turned out that Tetragon had the best performance among all the tools we tested out and for all the criteria we were testing for.”_ - Security engineer on the team.

Tetragon’s gRPC logging pipeline also reduced resource consumption compared to JSON-based logging. By directing logs into their Kafka-backed observability pipeline and storing them in BigQuery, the team ensured scalability and fast querying of security events. _“Tetragon offers granular configuration for monitoring processes, file access, and network traffic, and there’s also no requirement to maintain a logging backend. We can customize how we want the logs sent to our data lake and how they get delivered to various destinations,”_ The security engineer explained.

##### Kubernetes and ARM Support Additional Benefits

Another important factor the team considered when choosing their next security observability tool was first-class support for Kubernetes. Osquery was primarily designed with bare metal and VM workloads in mind. This meant the team had to customize Osquery to work for their Kubernetes workloads, which introduced an extra layer of complexity and maintenance burden on the team. A security engineer on the team highlighted Tetragon’s key differentiator “Tetragon’s Kubernetes compatibility stood out immediately. It supports Kubernetes as a first-class citizen and offers granular monitoring for processes, file access, and network activity.”

The company also has some of its workloads running ARM-based architectures, and when choosing a tool, they looked for in-built support ARM so they wouldn’t have to maintain any custom ARM images. _“A major factor in choosing Tetragon was its support for the arm64 architecture. With increasing workloads running on ARM-based processors, we are shifting further in this direction to leverage the efficiency and cost benefits. Tetragon's native support for arm64 eliminates the need for us to maintain custom images for our ARM-based deployments.”_

Finally, the team needed a tool that provided flexibility on the kind of security observability data collected. They were keen about this tool not requiring any custom vendor solution to funnel this data into their security observability pipeline. With the gRPC pipeline, the team could streamline exporting and analyzing security events without relying on proprietary or custom vendor solutions. _“We were looking for a security observability tool that allowed us to customize what kind of data we wanted to gather. As a company, we rely a lot on open source tools; this means we didn't want a situation where we had to wait for a custom vendor solution or request for a specific kind of data.”_

Deployment was straightforward. _“Tetragon’s daemonset-based model made implementation seamless,”_ the security engineer said. The company’s automated infrastructure ensured Tetragon was installed with baseline configurations across all new clusters. This provided them with instant visibility into process-level events without requiring extensive manual intervention.

##### Granular Control of High Value Events

Tetragon’s granular policy controls allowed them to focus on high-value process events, such as privilege escalations, execution of binaries from temp directories, and unauthorized container access.

_“Tetragon offers granular configuration for monitoring processes, file access, and network traffic, and there's also no requirement to maintain a logging backend. We can customize how we want the logs sent to our data lake and how they get delivered to various destinations.”_ - A Security Engineer on The Team

Tetragon’s ability to track vulnerabilities at runtime also proved invaluable. “If a new vulnerability affects specific binaries or libraries, we could write a Tetragon tracing policy to identify the workload affected. Tracking what service uses what library can be challenging. A service can have thousands of dependencies. Tetragon makes detecting any new vulnerability type that could impact us easier,” The security engineer said.

##### Confidence in Baseline Security Posture

With Tetragon deployed across all Kubernetes clusters, the security team gained comprehensive visibility into its environment. _“We can now detect anomalies faster, establish baseline behaviors, and confidently identify deviations,”_ The security engineer said. This enhanced confidence extended beyond the security team to the company’s leadership, who appreciated the improved reliability and reduced risk of production outages.

They have also been able to make security disappear into the background. _“The best situation is when other services and teams don't even realize that a security tool is running. That’s when you know you’ve done your job well, and that’s what Tetragon gives us.”_
The company’s infrastructure automation now includes Tetragon as a baseline component for every new Kubernetes cluster, whether it is for development, staging, or production. Tetragon is deployed with predefined configurations and baseline tracing policies. Events generated by Tetragon are logged locally and sent to their security observability pipeline. Currently, the company focuses on process visibility and file access monitoring with Tetragon.

##### Enforcement For the Future

The team plans to expand their usage of Tetragon to include various use cases such as monitoring all sudo invocations, monitoring the execution of binaries /tmp directory, monitoring privilege escalations, monitoring kubectl exec, monitoring execution of deleted binaries, and monitoring of eBPF subsystem interactions to provide visibility into the activity of other programs utilizing eBPF within their environment. They also want to implement network monitoring and improve metric-based anomaly detection. _“Metrics are incredibly powerful. They help us identify patterns and potential issues before they escalate,”_ the security engineer explained. Training team members to craft policies and integrating advanced enforcement mechanisms are on the roadmap.

Enforcement capabilities, though not yet fully implemented, are also a promising feature for the company. _“Tetragon allows us to block activities at runtime, which becomes crucial when managing a large number of violations. The SOC team cannot manually intervene with each developer, saying, ‘Hey, this is something you should not be doing.’ Tetragon allows us to automate the shift left to the first deployment, which is very helpful because once things are deployed in production, reversing them is challenging,”_ the security engineer noted

Tetragon has revolutionized the company’s approach to security observability, transforming a once-frustrating process into an efficient, proactive system. By providing detailed visibility without compromising performance, Tetragon empowers the company to secure its rapidly growing Kubernetes infrastructure. The security engineer summarized, _“Tetragon has become the baseline of our security observability strategy.”_

#### Cilium at KubeCon + CloudNativeCon Europe 2025 and CiliumCon

URL: https://cilium.io/blog/2025/02/21/cilium-at-kubecon-eu-2025
Date: 2025-02-21
Categories: Community

**_Author: Paul Arah, Isovalent_**

The Cilium community is buzzing to reunite again this April in London for KubeCon + CloudNativeCon EU and CiliumCon. As one of the most widely adopted open source projects in the cloud native ecosystem, Cilium continues to revolutionize networking, security, and observability. This year’s agenda for CiliumCon is designed to take you on a deep dive into the world of Cilium, Hubble, and Tetragon, featuring sessions led by end users, core contributors, and industry leaders. These talks cover everything from optimizing Cilium for performance and scale to exploring its role in bare metal machine learning workloads.

##### Reminiscing KubeCon NA 2024 and Cilium + eBPF Day in Salt Lake City, Utah

The last time the Cilium community was at KubeCon + CloudNativeCon NA in Salt Lake City, Utah. KubeCon + CloudNativeCon NA was an immersive exploration into the world of Cilium, Tetragon, Hubble, and eBPF. Some of the biggest highlights include Cilium being voted the #1 most useful and mature Multi-Cluster Application Management tool in the CNCF end-user survey, Cilium Maintainer Joe Stringer being named CNCF Top Committer of 2024, and a lot of insightful talks from around the ecosystem.

Data from the KubeCon event’s team indicated that Cilium + eBPF Day at KubeCon was the most attended co-located event last year. This shows how much impact Cilium continues to have in the cloud native community and how the growth of the Cilium community continues to accelerate.

KubeCon NA 2024 featured [fantastic talks](https://www.google.com/url?q=https://cilium.io/blog/2024/10/04/cilium-kubecon-na-2024/) from well-known companies like Microsoft, eBay, Datagod, Samsung, Sony, Red Hat, and Reddit. If you want the one minute update from the project, check out the Keynote Cilium Project Update.

##### Anticipating CiliumCon and KubeCon + CloudNativeCon Europe 2025 Talks

This year’s CiliumCon promises excellent talks diving into the world of Cilium, Hubble, and Tetragon. Here’s a glimpse of what’s in store:

[CiliumCon | Welcome + Opening Remarks - Hemanth Malla, Program & Bill Mulligan, Co-Chairs](https://colocatedeventseu2025.sched.com/event/1u5f2/ciliumcon-welcome-opening-remarks-hemanth-malla-program-bill-mulligan-co-chairs)

Tuesday, April 1, 2025, 09:00 - 09:05 BST

The opening session for CiliumCon Europe 2025 by the co-chairs.

[That’s Just My Cup of Tea: Configuring Cilium for Performance and Scale - Liz Rice, Isovalent at Cisco & Neha Aggarwal, Microsoft](https://colocatedeventseu2025.sched.com/event/1u5f5/thats-just-my-cup-of-tea-configuring-cilium-for-performance-and-scale-liz-rice-isovalent-at-cisco-neha-aggarwal-microsoft)

Cilium’s out-of-the-box default settings prioritize compatibility over performance, making it easy to deploy and get started. However, for production-grade environments, it’s essential to tune the settings to unlock Cilium’s full potential for performance and scalability.

This talk will explore settings and options that make a real difference, from eBPF-based host routing and kube-proxy replacement to eBPF map sizing and multi-cluster setups. We’ll also look at cutting-edge features like BIG TCP and Netkit and future performance features the project has on the roadmap. Whether you’re looking to optimize your performance or scale to the next level, we’ll provide the tools to get your Cilium environment running at top speed—without “waiting for the kettle to boil”!

[High-Scale Networking for ML Workloads With Cilium - Luigi Zhou, G-Research](https://colocatedeventseu2025.sched.com/event/1u5f8/high-scale-networking-for-ml-workloads-with-cilium-luigi-zhou-g-research)

Tuesday, April 1, 2025, 09:45 - 10:10 BST

In G-Research’s ML environment of over 10,000 nodes, Cilium is used as the core network for on-premise, bare-metal clusters that scale to 1,000 nodes each. In this talk, we’ll discuss several Cilium features used in detail:

- Network policy to enforce strict security controls for segmenting and protecting market-sensitive information
- Host firewall to remove the need for external firewall appliances
- High-performance eBPF dataplane that directly improves ML job performance

We’ll also cover the implications of limiting Cilium’s identity labels to reduce policy map pressure, tuning conntrack garbage collection, and the performance implications of different policies at scale. Attendees will learn how to use Cilium’s built-in tools to observe and measure large deployments and what to look out for in large Kubernetes clusters.

[Sponsored Keynote: Cilium’s Honeycomb: Cultivating Technical Excellence in a Diverse Ecosystem - Donia Chaiehloudj, Isovalent at Cisco](https://colocatedeventseu2025.sched.com/event/1u5fB/sponsored-keynote-ciliums-honeycomb-cultivating-technical-excellence-in-a-diverse-ecosystem-donia-chaiehloudj-isovalent-at-cisco)

Tuesday April 1, 2025 10:15 - 10:20 BST

Join us as we celebrate the collective intelligence that forms Cilium's success. By highlighting diverse use cases, supported environments, users, and dedicated contributors, we will illustrate how this convergence of minds, ideas, and implementations cultivates technical excellence within the Cilium project.

[Improving Network Efficiency: The Power of Integrating Delegated IPAM With Cilium - Tamilmani Manoharan & Will Daly, Microsoft](https://colocatedeventseu2025.sched.com/event/1u5fE/improving-network-efficiency-the-power-of-integrating-delegated-ipam-with-cilium-tamilmani-manoharan-will-daly-microsoft)

Tuesday, April 1, 2025, 10:40 - 11:05 BST

Are you curious about how Azure integrated their native IPAM implementation with Cilium through the power of open standards? The secret is "Delegated IPAM," a widely used part of the CNI specification. Unlike other IPAM implementations that embed vendor-specific code in Cilium, delegated IPAM allows seamless integration with any platform using out-of-tree plugins. Azure users benefit from fast, scalable IPAM and native routing that avoids the encapsulation overhead of Cilium tunnel mode. Delegated IPAM acts as a bridge combining the benefits of both worlds without any compromises. Session participants will learn how to leverage delegated IPAM to provide similar integrations for cloud or on-prem environments with zero changes in the Cilium code.

[Simplifying Multi-Cluster Networking With Cilium and MCS-API: A Technical Deep Dive - Arthur Outhenin-Chalandre, Ledger & Marco Iorio, Isovalent at Cisco](https://colocatedeventseu2025.sched.com/event/1u5fH/simplifying-multi-cluster-networking-with-cilium-and-mcs-api-a-technical-deep-dive-arthur-outhenin-chalandre-ledger-marco-iorio-isovalent-at-cisco)

Tuesday April 1, 2025, 11:15 - 11:40 BST

Multi-Cluster Services API (MCS-API), a standard driven by SIG Multicluster, extends Services across multiple clusters and is now supported in Cilium. While Cilium already enables multi-cluster services via annotations, MCS-API support brings it to the next level: create a ServiceExport resource referencing an existing Service to make it available to all clusters, enjoy DNS integration through the clusterset.local domain, and support for advanced features like Gateway API. All complemented by EndpointSlice synchronization to enable external ingress controllers and more.

In this deep dive talk, expect to learn about the Cilium Cluster Mesh architecture, how we implemented MCS-API support, and all the lessons we learned. We’ll also demo scenarios unlocked by MCS-API support and hidden secrets like how we (ab)used the Kubernetes EndpointSlice controller to reconcile EndpointSlices from remote clusters with minimal code changes.

[Lightning Talk: Agentic Bee: How To Get AI Agents To Talk To Tetragon - Himal Kumar, FlowPulse.AI](https://colocatedeventseu2025.sched.com/event/1u5fK/cl-lightning-talk-agentic-bee-how-to-get-ai-agents-to-talk-to-tetragon-himal-kumar-flowpulseai)

Tuesday April 1, 2025 11:45 - 11:55 BST

Tetragon delivers powerful runtime security logs, offering unmatched granularity. However, these detailed insights can overwhelm users with an endless stream of alerts and events. Usual approaches include tracing policy tuning, aggregating, and then visualising data on dashboards, however, not everything is meant for human consumption.

This talk introduces a fresh approach: integrating Tetragon’s runtime telemetry with LLM-powered AI agents. By leveraging a simple Retrieval-Augmented Generation (RAG) architecture, we enable AI agents to automate workflows directly on top of Tetragon events. These agents can perform tasks such as correlating vulnerabilities with runtime context and extracting actionable insights for human consumption—without relying on dashboards.

[Lightning Talk: Mastering Zero Downtime Migrations Between Cilium Identity Allocation Modes - Anton Ippolitov, Datadog](https://colocatedeventseu2025.sched.com/event/1u5fN/cl-lightning-talk-mastering-zero-downtime-migrations-between-cilium-identity-allocation-modes-anton-ippolitov-datadog)

Tuesday April 1, 2025 12:00 - 12:10 BST

Cilium offers two options for storing security identities: KVStore mode and CRD mode. Both have their pros and cons when it comes to scalability, operational complexity, and cost.

Previously, there wasn't a straightforward method to migrate live Kubernetes clusters between these two modes without incurring downtime. In this talk, we'll discuss the challenges Datadog faced and our upstream contributions to seamlessly migrate hundreds of clusters from KVStore to CRD mode without causing network disruptions for users.

[Lightning Talk: High Availability for Cilium Egress Gateway - Angelo Conforti, Corner Banca SA](https://colocatedeventseu2025.sched.com/event/1u5fQ/cl-lightning-talk-high-availability-for-cilium-egress-gateway-angelo-conforti-corner-banca-sa)

Tuesday April 1, 2025 12:15 - 12:25 BST

Cilium is a leading CNCF project that has become the de-facto standard for Kubernetes networking. Among its many features is the Egress Gateway, which allows routing outgoing traffic from one or more workloads to a specific egress IP. However, this feature lacks built-in high availability, and the egress IP must be managed externally. The Cilium HAEgress Operator addresses this limitation by providing a high-availability solution for egress traffic, ensuring continuity during node failures through dynamic virtual IP migration between nodes.
This lightning talk introduces the project, its current state, and future developments to extend this standard Cilium feature.

[CiliumCon | Closing Remarks - Hemanth Malla, Program & Bill Mulligan, Co-Chairs](https://colocatedeventseu2025.sched.com/event/1u5fT/ciliumcon-closing-remarks-hemanth-malla-program-bill-mulligan-co-chairs)

Tuesday April 1, 2025 12:25 - 12:30 BST

The closing session for CiliumCon Europe 2025 by the co-chairs

[Simplifying the Networking and Security Stack With Cilium, Hubble, and Tetragon - Bill Mulligan & Anna Kapuścińska, Isovalent at Cisco; Dorde Lapcevic, Google; Amir Kheirkhahan, DBSchenker](https://kccnceu2025.sched.com/event/1tcxw/simplifying-the-networking-and-security-stack-with-cilium-hubble-and-tetragon-bill-mulligan-anna-kapuscinska-isovalent-at-cisco-dorde-lapcevic-google-amir-kheirkhahan-dbschenker?iframe=no&w=100%&sidebar=yes&bg=no)

Wednesday April 2, 2025 15:15 - 15:45 BST

oin us as we celebrate nearly a decade of Cilium, now the de-facto standard CNI for Kubernetes and a cornerstone of cloud native networking, observability, and security. This session provides updates on the latest Cilium release and showcases how its unified eBPF-powered stack is transforming Kubernetes environments by replacing fragmented toolchains with seamless, secure, scalable, and simplified solutions.

We’ll explore features like multi-cluster networking, scaling to 65,000 nodes, and service mesh use cases and dive into sub-projects Hubble for network observability and Tetragon’s security observability and runtime enforcement. Hear from contributors and adopters DB Schenker, Google, and Isovalent about how Cilium is simplifying the cloud native stack and solidifying its role as the comprehensive networking and security solution for modern cloud native architectures.

**Cilium Project Booth:** Finally, visit Cilium’s kiosk at 15B in the Project Pavilion to see live demos, explore new functionalities, and connect with the community.

Whether you are attending hands-on workshops, lightning talks, or networking with industry peers, you will find countless opportunities to interact with the Cilium community. Be sure to say Hi!

See you in London!

#### Cilium at KubeCon + CloudNativeCon and CiliumCon Europe 2026

URL: https://cilium.io/blog/2026/01/23/cilium-at-kubecon-eu-2026
Date: 2025-01-23
Categories: Community

**_Author: Katie Meinders, Isovalent_**

As tulips begin to bloom across Amsterdam this March, the Cilium community is buzzing to reunite for [KubeCon + CloudNativeCon](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/) and [CiliumCon](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/co-located-events/ciliumcon/) Europe 2026. From ClusterMesh deployments spanning continents to eBPF-powered security at scale, this year's lineup showcases how Cilium continues to reshape networking, security, and observability across cloud native infrastructure. Here's your guide to the must-attend sessions and where to find the community during the show.

##### CiliumCon

Join us on **Monday, 23 March**, for an afternoon of talks, community updates, and production use cases during CiliumCon. This year's agenda features end user stories from organizations like Roche and Etraveli Group, deep dives into new features like flow aggregation, and cutting-edge innovations including best practices for running Tetragon and multi-cluster networking at scale. Here’s a look at what you can expect:

[**Welcome + Opening Remarks**](https://colocatedeventseu2026.sched.com/event/2DZMa/ciliumcon-welcome-+-opening-remarks-hemanth-malla-mahe-tardy-program-committee-co-chairs?iframe=no)
*Hemanth Malla & Mahé Tardy, Program Committee Co-Chairs | 13:25 - 13:30 CET*

Event co-chairs Mahé and Hemanth, will share the latest updates from the project and community.

[**Network Flow Aggregation: Pay for the Logs You Care About!**](https://colocatedeventseu2026.sched.com/event/2DY58/network-flow-aggregation-pay-for-the-logs-you-care-about-mereta-degutyte-anubhab-majumdar-microsoft?iframe=yes&w=100%&sidebar=yes&bg=no)
*Mereta Degutyte & Anubhab Majumdar, Microsoft | 13:35 - 14:00*

Learn how a new flow aggregation feature reduces Hubble storage costs while maintaining observability quality through in-agent aggregation based on user-defined fields.

[**It Is Always DNS… Even at the Edge: Taming Proxy-Only Lookups With Cilium and eBPF**](https://colocatedeventseu2026.sched.com/event/2DY5u/it-is-always-dns-even-at-the-edge-taming-proxy-only-lookups-with-cilium-and-ebpf-hector-monsalve-roche-thomas-gosteli-tactummotum-gmbh?iframe=yes&w=100%&sidebar=yes&bg=no)
*Hector Monsalve, Roche & Thomas Gosteli, TactumMotum GmbH | 14:10 - 14:35*

Discover how Roche is solving DNS challenges in protected environments where external lookups must occur through HTTP proxies, using Cilium's network policy traffic redirect and CoreDNS filtering, including a live demo.

[**Lightning Talk: Accelerating Cilium: Our Journey on Offloading Cilium To DPU**](https://colocatedeventseu2026.sched.com/event/2DY66/cllightning-talk-accelerating-cilium-our-journey-on-offloading-cilium-to-dpu-vijay-inavolu-marvell-technologies?iframe=yes&w=100%&sidebar=yes&bg=no)
*Vijay Inavolu, Marvell Technologies | 14:40 - 14:50*

See how Marvell is transparently offloading Cilium to DPUs with minimal code changes, including metrics on resource savings and a demo of hardware-accelerated security policies.

[**Sponsored Keynote - Built in the Open: Why Cisco Is Betting on eBPF, Cilium, and Tetragon**](https://colocatedeventseu2026.sched.com/event/2DZMf/ciliumcon-sponsored-keynote-tba-isovalent-at-cisco?iframe=yes&w=100%&sidebar=yes&bg=no)
*Bill Mulligan, Isovalent at Cisco | 14:55 - 15:00*

Learn how, by investing upstream in Cilium, eBPF, and Tetragon, Cisco is aligning product innovation with community driven standards ensuring that what runs in Kubernetes today can extend consistently across clouds, data centers, and the network itself.

[**Connecting the World: Your Hands-on Guide To Cilium ClusterMesh and MCS-API**](https://colocatedeventseu2026.sched.com/event/2DY6j/connecting-the-world-your-hands-on-guide-to-cilium-clustermesh-and-mcs-api-arthur-outhenin-chalandre-ledger-quentin-swiech-s3ns?iframe=yes&w=100%&sidebar=yes&bg=no)
*Arthur Outhenin-Chalandre, Ledger & Quentin Swiech, S3NS | 15:20 - 15:45*

S3NS shares their GitOps-ready blueprint for building secure multi-cluster platforms using ClusterMesh and Multi-Cluster Service API, scaling to hundreds of clusters worldwide.

[**Cloud Native Promises, On-Premises: Bump Your Load Balancing To V2 With Cilium**](https://colocatedeventseu2026.sched.com/event/2DY7S/cloud-native-promises-on-premises-bump-your-load-balancing-to-v2-with-cilium-mikael-johansson-lansberg-etraveli-group-ab?iframe=yes&w=100%&sidebar=yes&bg=no)
*Mikael Johansson Länsberg, Etraveli Group AB | 15:55 - 16:20*

Learn how Etraveli Group replaced proprietary hardware appliances with Cilium's eBPF-powered load balancing to handle millions of daily flight searches with unified control and cloud native agility.

[**One Policy To Rule Them All? Scaling Tetragon Without Flooding Your Cluster**](https://colocatedeventseu2026.sched.com/event/2DY8B/one-policy-to-rule-them-all-scaling-tetragon-without-flooding-your-cluster-alessio-biancalana-suse?iframe=yes&w=100%&sidebar=yes&bg=no)
*Alessio Biancalana, SUSE | 16:30 - 16:55*

SUSE explores per-workload security models with Tetragon, sharing techniques to achieve strong policy enforcement while minimizing performance and memory overhead at scale.

[**Lightning Talk: What's Happening in Cilium? Current Projects You Need To See**](https://colocatedeventseu2026.sched.com/event/2DY8x/cllightning-talk-whats-happening-in-cilium-current-projects-you-need-to-see-martynas-pumputis-isovalent-at-cisco?iframe=yes&w=100%&sidebar=yes&bg=no)
*Martynas Pumputis, Isovalent at Cisco | 17:00 - 17:10*

A fast-paced visual tour of major developments in the Cilium ecosystem, featuring video introductions from project authors themselves.

[**Lightning Talk: Closing the Gap: Fair North–South Bandwidth Management for Tenants**](https://colocatedeventseu2026.sched.com/event/2DY8x/cllightning-talk-whats-happening-in-cilium-current-projects-you-need-to-see-martynas-pumputis-isovalent-at-cisco?iframe=yes&w=100%&sidebar=yes&bg=no)
*Weizhou Lan, Daocloud | 17:15 - 17:25*

Daocloud presents a lightweight approach combining Cilium and Spiderpool to enable tenant-level ingress bandwidth management for load balancer services within Kubernetes.

##### Featured KubeCon + CloudNativeCon Europe 2026 Talks

Beyond CiliumCon, you'll find Cilium featured throughout the main conference, from Bill Mulligan's celebration of Cilium's 10-year anniversary to Asana using invisible guardrails to create a secure platform with Cilium. Here are some of the highlights:

[**Project Lightning Talk: Translating eBPF Into Business Value: Lessons From The Cilium Website**](https://kccnceu2026.sched.com/event/2EFwm/project-lightning-talk-translating-ebpf-into-business-value-lessons-from-the-cilium-website-bill-mulligan-maintainer?iframe=yes&w=100%&sidebar=yes&bg=no)
*Bill Mulligan, Cilium Maintainer | Monday, 23 March | 09:52 - 09:57*

Practical patterns for translating deep technical capabilities into clear business outcomes through case studies and real-world results, without losing technical credibility.

[**10 Years of Cilium: Connecting, Securing, and Simplifying the Cloud Native Stack**](https://kccnceu2026.sched.com/event/2EF3l/10-years-of-cilium-connecting-securing-and-simplifying-the-cloud-native-stack-bill-mulligan-isovalent-at-cisco?iframe=yes&w=100%&sidebar=yes&bg=no)
*Bill Mulligan, Isovalent at Cisco | Tuesday, 24 March | 14:30 - 15:00*

Celebrate Cilium's decade as the de-facto standard CNI with updates on the latest release, service mesh evolution, Tetragon's security observability, and how users are reshaping the cloud native stack.

[**Demystifying the Kubernetes Network Stack (From Pod to Pod)**](https://kccnceu2026.sched.com/event/2CW0A/demystifying-the-kubernetes-network-stack-from-pod-to-pod-simone-rodigari-microsoft?iframe=yes&w=100%&sidebar=yes&bg=no)
*Simone Rodigari, Microsoft | Tuesday, 24 March | 17:00 - 17:30*

Trace a packet's journey through Kubernetes networking with diagrams and live demos, learning how CNIs like Cilium, kube-proxy, and DNS work together, plus practical debugging techniques.

[**From Classroom To Container: Teaching Distributed Systems With CNCF & Cloud Native Technologies**](https://kccnceu2026.sched.com/event/2CW2s/from-classroom-to-container-teaching-distributed-systems-with-cncf-cloud-native-technologies-matthias-haeussler-cgi?iframe=yes&w=100%&sidebar=yes&bg=no)
*Matthias Haeussler, CGI | Wednesday, 25 March | 15:00 - 15:30*

A roadmap for integrating CNCF technologies like Cilium, Envoy, and OpenTelemetry into education, translating distributed systems theory into hands-on cloud native development.

[**Invisible Guardrails: Enabling Developer Velocity With a Secure Platform**](https://kccnceu2026.sched.com/event/2CW34/invisible-guardrails-enabling-developer-velocity-with-a-secure-platform-james-elias-sigurdarson-vignir-hafsteinsson-asana?iframe=yes&w=100%&sidebar=yes&bg=no)
*James Elías Sigurðarson & Vignir Hafsteinsson, Asana | Wednesday, 25 March | 15:00 - 15:30*

Asana's approach to making the fastest developer path also the most secure, using Crossplane and Cilium to build invisible security guardrails directly into their platform.

[**Observing Chaos: Real-Time Monitoring of AI-Driven Kubernetes Destruction**](https://kccnceu2026.sched.com/event/2CW1T/observing-chaos-real-time-monitoring-of-ai-driven-kubernetes-destruction-josh-halley-cisco-ricardo-aravena-cncf?iframe=yes&w=100%&sidebar=yes&bg=no)
*Josh Halley, Cisco & Ricardo Aravena, CNCF | Wednesday, 25 March | 17:30 - 18:00*

AI agents play DOOM against real Kubernetes workloads while OpenTelemetry and Cilium provide real-time observability, turning static chaos tests into adaptive challenges.

##### A look back at KubeCon + CloudNativeCon and CiliumCon North America in Atlanta

This past November, the community gathered in Atlanta during KubeCon + CloudNativeCon and [CiliumCon North America](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/co-located-events/ciliumcon/). CiliumCon attendees heard from Microsoft’s Shreya Jayaraman and Tamilmani Manoharan about scalable container networking with CiliumEndpointSlice, ESnet’s Kapil Agrawal about the organization’s use of Cilium for IPv6-only networking, and co-chair Hemanth Malla about parsing DNS in eBPF, plus much more.

Cilium was also featured in talks at other co-located events, including Cloud Native University, Observability Day, and VM on Kubernetes Day.

During the maintainers track, Bill Mulligan, Thomas Graf, and Neha Aggarwal [celebrated a decade of Cilium](https://www.youtube.com/watch?v=mCKwX74xIs4) as the de-facto standard CNI for Kubernetes, and looked ahead to the next ten years. Giri Kuncoro and Joseph Pallamidessi [shared insights](https://www.youtube.com/watch?v=y0qlhiKtDGo) into TikTok’s journey of making Cilium work for IPv6 only Kubernetes.

Contributors gathered during the [Cilium Developer Summit](https://github.com/cilium/dev-summits/blob/main/2025-NA/README.md) to discuss the project’s future, with representatives from Google, Isovalent, Microsoft, and more tackling topics like datapath plugins, scaling with network policies, and multi-network support.

##### Connect with the Cilium Community in Amsterdam

**Cilium Project Booth**: Be sure to visit Cilium’s booth in the Project Pavilion throughout KubeCon + CloudNativeCon for live demos, hands-on exploration of new features, and the chance to connect directly with maintainers, contributors, and community members. Whether you have technical questions, want to learn about contributing, or just want to share your Cilium story, stop by and say hello. We look forward to seeing you in Amsterdam!

Beyond the sessions, join us for hallway conversations, impromptu demos, and the chance to meet the contributors behind the features you use every day. Whether you're troubleshooting a deployment challenge, exploring new use cases, or considering contributing to the project, the Cilium community is here to help.

#### Tetragon: 2024 in Review

URL: https://cilium.io/blog/2024/12/31/tetragon-2024-year-in-review
Date: 2024-12-31
Categories: Community

**_Author: Paul Arah, Isovalent_**

2024 has been a remarkable year for Tetragon. This year witnessed increased adoption, the release of many exciting features, a record number of conference talks, and exponential growth within the community. As we approach the year's end, we reflect on the progress made as a community and look ahead to the future.

Since the initial release of Tetragon 1.0 last year, the sub-project has continued to set the standard for eBPF-based security observability and runtime enforcement. Platform and security teams worldwide, from small companies to prominent enterprises, use Tetragon to secure their environments efficiently and effectively.

While the 2024 [Cilium annual report](https://github.com/cilium/cilium.io/blob/main/Annual-Reports/Cilium_Annual_Report_2024.pdf) covered some major statistics for the Tetragon project, this blog zooms in on what we, as a community, have accomplished this year.

##### Release Highlights

###### Tetragon 1.1

###### Kubernetes Identity-Aware Policies

Tetragon 1.1 introduced features that enabled deeper integration with Kubernetes, including Kubernetes Identity-Aware Policies. Previously, tracing policies (which define the situations Tetragon should react to and how) were applied at a cluster-wide scope. With this update, Tetragon’s security observability and runtime enforcement can be used with precision to specific Kubernetes workloads based on their identity. This improvement reduces noise and overhead while providing security tailored to specific workloads.

###### Redaction Filters

Tetragon events, exposed via gRPC and JSON logs, can include sensitive information such as passwords or environment variables. Tetragon 1.1 introduced redaction filters, enabling users to redact specific fields from exported data, ensuring sensitive information is not inadvertently exfiltrated.

###### CRI-O Runtime Hooks Support

Tetragon requires Kubernetes metadata to enable Kubernetes Identity-Aware Policies. Retrieving this metadata from the Kubernetes API can cause delays between container startup and policy application, which is undesirable, especially in enforcement scenarios. Runtime hooks solve this issue by directly "hooking" into the container runtime system, ensuring the Tetragon agent sets up the necessary state before the container starts. With Tetragon 1.1, CRI-O runtime hooks are automatically configured via a Tetragon init container.

###### Tetragon 1.2

###### Persistent Enforcement

Before Tetragon 1.2, enforcement policies depended on the Tetragon agent’s uptime. If the agent went down—due to restarts or failures—the eBPF programs responsible for enforcement were removed, potentially causing a security lapse. Tetragon 1.2 ensures that the eBPF programs responsible for enforcement continue running even during agent downtime, enhancing security and minimizing risk.

###### Improved Child Process Visibility

Child process visibility is crucial for detecting malicious activities such as process injection and lateral movement. Tetragon already allowed tracking all child processes spawned by specific binaries, but earlier versions required users to rely on PID values, which can be non-deterministic. Version 1.2 introduced a more intuitive way to track process ancestry across complex execution chains.

###### Containerd Support

Containerd, one of the most popular container runtimes, is now supported in Tetragon 1.2. Previously, the runtime hook system enabling Kubernetes Identity-Aware Policies supported only CRI-O. With this update, the runtime hook system was redesigned from an init container to a DaemonSet, providing flexibility to support both CRI-O and Containerd. This expansion brings Tetragon’s full capabilities to a broader range of Kubernetes deployments.

##### Conferences

Tetragon had a strong presence at major cloud native events in 2024. These events included KubeCon, Cilium + eBPF Day, and CloudNativeSecurityCon. Here are some of the highlights:

- [Dealing with eBPF’s Observability Data Deluge - Anna Kapuścińska, Isovalent](https://www.youtube.com/watch?v=yWB8n_e4N14)
- [Bee-Lieve in the Metadata: Pollenating Build Attestations on Kubernetes with Tetragon and EBPF - Tom Meadows, TestifySec](https://www.youtube.com/watch?v=ejkJiq7AMHs)
- [Brewing the Kubernetes Storm Center: Open Source Threat Intelligence for the Cloud Native Ecosystem - Constanze Roedig, Technische Universität Wien & James Callaghan, ControlPlane](https://www.youtube.com/watch?v=YDIW2CY8WPI)
- [Don't Get Blown up! Avoiding Configuration Gotchas for Tetragon Newbies - Pratik Lotia, Reddit](https://www.youtube.com/watch?v=YNDp7Id7Bbs)
- [What Agent to Trust with Your K8s: Falco, Tetrago, or KubeArmor? - Henrik Rexed, Dynatrace](https://www.youtube.com/watch?v=QKE8WMv-6qw)
- [From Observability to Enforcement: Lessons Learned Implementing eBPF Runtime Security - Anna Kapuścińska & Kornilios Kourtis, Isovalent](https://www.youtube.com/watch?v=Hw469I5GKmY)
- [Panel: Exploring eBPF Use Cases in Cloud-Native Security](https://www.youtube.com/watch?v=towNkbPMDjE)
- [Tutorial: Sailing the Security Seas with Tetragon - Duffie Cooley, Isovalent](https://www.youtube.com/watch?v=4ACOEB4PnQo)

##### Community Meetings

This year, we kicked off a monthly [Tetragon community meeting](https://isogo.to/tetragon-meeting-notes) on the second Monday of the month. We held the first community meeting in March and have held 9 community meetings so far this year baring the months where the community meetings coincided with KubeCon. The community meetings have been great to onboard new members to the community, discuss ongoing PRs, and learn about all the exciting work and development in Tetragon.
We look forward to continued participation and growth in the community meetings as we go into 2025.

##### Community Voices

The Tetragon community has shared many exciting testimonials about their experiences. One quote that encapsulates Tetragon's impact on cloud native security comes from Marcos Hernandez, Platform Engineering Lead at Google Cloud:

**_“eBPF tracing is very powerful. I recently deployed a hashtag#Tetragon TracingPolicy CR to my Google Distributed Cloud Virtual (GDCV) clusters at the edge as an experiment. The level of visibility and control it offers through the various supported hook points is really cool. Love the programmatic, low level access to kernel events. hashtag#Cilium hashtag#GoogleDistributedCloud.”_** - Marcos Hernandez, Platform Engineering Lead at Google Cloud ([Read the LinkedIn post](https://www.linkedin.com/posts/activity-7270097966017703936-mvrp/?utm_source=share&utm_medium=member_desktop))

##### Looking Ahead

As we close the chapter on 2024, it’s clear that Tetragon has made significant strides in enhancing cloud native security observability and runtime enforcement. From major feature releases to a growing community and vibrant presence at key conferences, the project’s trajectory is nothing short of inspiring.

The Tetragon community’s dedication, innovation, and collaboration have been pivotal to this success. As we look to 2025, we are excited to continue pushing the boundaries of what’s possible with eBPF powered security observability and runtime enforcement with Tetragon, empowering teams worldwide to secure their cloud native environments with confidence.

Here’s to another year of growth, innovation, and community-driven success. See you in 2025!

For the Tetragon Community:

Paul Arah
Community Builder – Security

#### Cilium Talks at KubeCon NA 2024

URL: https://cilium.io/blog/2024/10/04/cilium-kubecon-na-2024
Date: 2024-10-04
Categories: Community

###### _October 4th, 2024_

###### _Author: Shedrack Akintayo, Isovalent at Cisco_

Following a remarkable period of growth and innovation, the Cilium community is gathering for the fourth Cilium + eBPF Day and the ninth KubeCon + CloudNativeCon North America 2024. With the recent [release of Cilium 1.16](https://github.com/cilium/cilium/releases/tag/v1.16.0), featuring the highly anticipated and powerful Cilium **netkit**, the project is again at the forefront of the cloud native ecosystem. This release is generating significant buzz, showcasing how Cilium and eBPF continue to revolutionize networking, observability, and security.

[Cilium + eBPF Day](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/co-located-events/cilium-ebpf-day/) promises to be an immersive exploration into the world of Cilium, Tetragon, Hubble, and eBPF. The agenda is packed with insightful talks from end users, core contributors, and community members, covering topics like migrating to Cilium, network policy scalability and enforcement, Cilium at the edge, and many more. Speakers from leading companies, including Microsoft, Isovalent at Cisco, Red Hat, eBay, Sony, and The New York Times, will share their experiences and insights, making this a valuable learning opportunity. We’d also like to thank our sponsors for their generous support in making this event possible.

At the broader [KubeCon + CloudNativeCon North America 2024](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/), Cilium is poised to be a significant part of the discourse, reflecting its leading influence in the cloud native community. Let’s run through all the ways you can learn about Cilium at the events to equip yourself with the knowledge to harness the power of Cilium and eBPF, enhance your networking capabilities, and make your platforms more secure, performant, and observable.

Now, let’s dive into each of the talks on Cilium at the events!

##### Cilium + eBPF Day 2024

[Cilium + eBPF Day | Welcome + Opening Remarks - Bill Mulligan, Isovalent & Vlad Ungureanu, Palantir Technologies](https://colocatedeventsna2024.sched.com/event/1j2dF/cilium-ebpf-day-welcome-opening-remarks)

Tuesday, November 12, 2024, 09:00 am - 09:10 am MST

The Opening Session for Cilium + eBPF Day NA 2024

[Confluent's Multi-Cloud Journey to Cilium: Pitfalls and Lessons Learned - Nimisha Mehta & Alvaro Aleman, Confluent](https://colocatedeventsna2024.sched.com/event/1izpN/confluents-multi-cloud-journey-to-cilium-pitfalls-and-lessons-learned-nimisha-mehta-alvaro-aleman-confluent)

Tuesday, November 12, 2024, 9:10 am - 9:35 am MST

Confluent Cloud is a data streaming platform built on thousands of Kubernetes clusters across AWS, Azure & GCP. Confluent migrated clusters to use Cilium for its advanced security features like transparent encryption and DNS name-based network policies, along with performance, scalability & observability improvements. The main challenge was executing a live migration without disrupting stateful workloads, complicated by the risks of replacing a low-level component like the CNI. The process required meticulous planning to ensure intra-cluster connectivity during migration while accommodating each cloud provider's unique network config. This talk shares the journey of migrating to Cilium, highlighting obstacles and lessons learned. We will explore uninstalling pre-existing CNIs, setting up Cilium & addressing cloud-specific issues to maintain connectivity. Benefits like transparent encryption, policies, and Hubble observability, along with the challenges faced, will also be discussed.

[Insightful Traffic Monitoring: Harnessing Cilium for Comprehensive Network Observability - Sudheendra Murthy & Adithya Yavanamanda, eBay](https://colocatedeventsna2024.sched.com/event/1izpm/insightful-traffic-monitoring-harnessing-cilium-for-comprehensive-network-observability-sudheendra-murthy-adithya-yavanamanda-ebay)

Tuesday, November 12, 2024, 9:45 am - 10:10 am MST

eBay's cloud consists of thousands of microservices running on millions of containers across hundreds of Kubernetes clusters. In this dynamic & complex cloud environment, mapping dependencies between microservices is crucial. This session delves into how eBay innovatively and scalably uses Cilium, powered by eBPF, to monitor traffic flows, generate real-time traffic events, and construct a comprehensive dependency graph of microservice interactions across hundreds of K8s clusters.

The presentation will cover:

- The innovative use of eBPF and Cilium to monitor traffic events in near real-time
- How traffic events are mapped to different microservices
- The architecture and design of the scalable solution to handle the large volume of data
- The integration of OpenTelemetry for efficient traffic event stream processing
- Key challenges and solutions in building and maintaining the dependency graph
- Insights and lessons learned from integrating eBPF and Cilium into eBay’s infrastructure

[Panel: Exploring eBPF Use Cases in Cloud-Native Security - Oshrat Nir, ARMO; Anna Kapuścińska, Isovalent, now part of Cisco; Whitney Lee, CNCF Ambassador; Maya Singh, Microsoft; Cortney Nickerson, Kubeshop](https://colocatedeventsna2024.sched.com/event/1izr8/panel-exploring-ebpf-use-cases-in-cloud-native-security-oshrat-nir-armo-anna-kapuscinska-isovalent-now-part-of-cisco-whitney-lee-cncf-ambassador-maya-singh-microsoft-cortney-nickerson-kubeshop)

Tuesday, November 12, 2024, 11:50 am - 12:25 pm MST

Cloud-native security requires a shift in mindset. Workloads are ephemeral, the attack surface has grown, and with it, the complexities. eBPF has emerged as a powerful technology, enabling deep visibility and dynamic security capabilities within the Linux kernel. This panel will explore use cases in which eBPF enhances cloud-native security. We will explore how eBPF can be leveraged to perform real-time monitoring, threat detection, and mitigation across containerized applications and microservices. Our expert panelists will share insights on using eBPF for network security, application profiling, anomaly detection, and enforcing security policies at the kernel level. Additionally, we will discuss the integration of eBPF with popular cloud-native tools and platforms, showcasing practical implementations.

[Scaling Network Policy Enforcement Beyond the Cluster Boundary with Cilium - Hemanth Malla & Maxime Visonneau, Datadog](https://colocatedeventsna2024.sched.com/event/1izs0/scaling-network-policy-enforcement-beyond-the-cluster-boundary-with-cilium-hemanth-malla-maxime-visonneau-datadog)

Tuesday, November 12, 2024, 1:30 pm - 1:55 pm MST

To keep up with infrastructure growth, companies around the world are managing an increasing number of Kubernetes clusters. Enforcing Kubernetes native network policy at scale is already hard enough within a single cluster. Extending this to multiple clusters is even more challenging. Depending on the shape of your infrastructure, your cross-cluster policy requirements may be unique, and there’s no one-size-fits-all configuration. In this talk, we’ll dive deep into how different solutions work in cilium to understand sources of potential bottlenecks. We’ll discuss Clustermesh, KVstoremesh, DNS-based FQDN policy, and a custom variant of KVstoremesh Datadog leverages while meshing at scale. Specifically, we’ll discuss how factors like the number of pods, identities, and pod churn will impact scalability and time to policy enforcement. Join us if you’re curious about understanding the latest in cross-cluster policy and leave with actionable insights you can apply to your infrastructure.

[How to Use XDP and eBPF to Accelerate IPSec Throughput by 400% - Ryan Drew, Isovalent, now part of Cisco](https://colocatedeventsna2024.sched.com/event/1izsR/how-to-use-xdp-and-ebpf-to-accelerate-ipsec-throughput-by-400-ryan-drew-isovalent-now-part-of-cisco)

Tuesday, November 12, 2024, 2:05 pm - 2:30 pm MST

The techniques used to increase IPSec network performance are often kept as secrets because they act as a competitive advantage and a lucrative product offering. This talk transparently presents a technique for massively boosting IPSec performance that is simple to implement (less than 200 lines of C), and based entirely on open-source work. An early Proof of Concept (POC) implementation showed an increase in p99 throughput by 412%! This talk will take a deep dive into how it all works, covering: the implementation, the pros and cons of the design, and an analysis of benchmark results. As transparent encryption becomes more crucial for securing data in transit, we hope this talk will enable users required to use IPSec for compliance or infrastructure reasons to learn how to speed up their network without having to compromise their security.

[Live Migrating Production Clusters From Calico to Cilium - Moh Ahmed & Raymond Maika, SamsungAds](https://colocatedeventsna2024.sched.com/event/1mFPd/live-migrating-production-clusters-from-calico-to-cilium-moh-ahmed-raymond-maika-samsungads)

Tuesday, November 12, 2024, 2:40 pm - 3:05 pm MST

Engineers may be tasked with rolling out a new Container Networking Interface (CNI) to their environment. Sounds easy enough! Delete the old one, and deploy the new one. Or maybe just deploy a brand new cluster! What if... there was another way? The talk will show how a live, in-place migration of the CNI plugin was performed in production clusters. It will highlight a few approaches that were considered, and what approach was eventually selected before proceeding with the migration process. Lastly, the procedure and steps taken to execute this migration will be shared, along with any lessons learned.

[Hubble Beyond Cilium - Anubhab Majumdar & Mathew Merrick, Microsoft](https://colocatedeventsna2024.sched.com/event/1iztI/hubble-beyond-cilium-anubhab-majumdar-mathew-merrick-microsoft)

Tuesday, November 12, 2024, 3:20 pm - 3:45 pm MST

Hubble is a great solution for finding and fixing network problems in a Kubernetes cluster. However, we noticed that one of the main barriers for people to use Hubble is its dependency on Cilium as the dataplane. In this talk, we'll demonstrate how to decouple Hubble from Cilium, and use Hubble as a powerful Observability/metrics platform on top of any custom data plane. We will show you how to make Hubble work with any data source you want, without changing any code in Hubble. We'll show you an example of one such open source project called Retina and compare how key features work with both Cilium and custom CNI. In a live demo, we will show that you can get the same experience with Hubble regardless of what CNI you use.

[Lessons Learned Migrating to Modern Multi-Platform eBPF Programs - Dave Tucker, Red Hat](https://colocatedeventsna2024.sched.com/event/1izth/lessons-learned-migrating-to-modern-multi-platform-ebpf-programs-dave-tucker-red-hat)

Tuesday, November 12, 2024, 3:55 pm - 4:20 pm MST

Kepler needed to migrate its old eBPF probes developed with BCC to probes that were compiled ahead of time. Maybe you do too? While performing this migration we were able to use some modern features of eBPF, the cilium/ebpf Go library, and bpf2go to make our probes multi-platform. Kepler (Kubernetes-based Efficient Power Level Exporter) is a CNCF project focused on measuring the environmental impact of software. At its core, Kepler uses eBPF to gather metrics from the Linux Kernel, which feeds into an ML model that estimates power consumption for processes, VMs, and Pods. By the end of this session, you’ll gain a deeper understanding of eBPF, practical insights into its application in power consumption monitoring, and strategies for modernizing existing eBPF programs. Join us to learn from our experience and take away actionable best practices for your projects!

[ Lightning Talk: Don't Get Blown up! Avoiding Configuration Gotchas for Tetragon Newbies - Pratik Lotia, Reddit](https://colocatedeventsna2024.sched.com/event/1izuW/cl-lightning-talk-dont-get-blown-up-avoiding-configuration-gotchas-for-tetragon-newbies-pratik-lotia-reddit)

Tuesday, November 12, 2024, 5:00 pm - 5:10 pm MST

This talk will dive into five common configuration pitfalls that beginners encounter when using Tetragon for runtime observability on their workloads. We'll explore the implications of each gotcha and provide clear steps to avoid them. The talk will also cover best practices for configuring Tetragon in a Kubernetes environment.

[ Lightning Talk: Applying Cilium at Edge with KubeEdge - Tomoya Fujita, Sony Corporation of America](https://colocatedeventsna2024.sched.com/event/1izuu/cl-lightning-talk-applying-cilium-at-edge-with-kubeedge-tomoya-fujita-sony-corporation-of-america)

Tuesday, November 12, 2024, 5:15 pm - 5:25 pm MST

Applications in edge environments can be platform-dependent, complicated, and distributed in regions, and the number of devices significantly increases. Our final goal is to create the infrastructure that can be applied to the entire environment crossing over the cloud and edge in common. Working with KubeEdge and Cilium, we are now successfully able to use Cilium with KubeEdge-hosted nodes at edge environment. This means, that enabling wireguard VPN with Cilium can provide transparent network connectivity with the nodes running in the cloud infrastructure so that edge nodes running at edge environment just appear to be a member of the cluster system but with edge autonomy feature provided by KubeEdge. We would like to share our technical insights and experience with using Cilium at Edge with KubeEdge, and what are the future development and contribution to the Cilium community.

[Cilium + eBPF Day | Closing Remarks - Bill Mulligan, Isovalent & Vlad Ungureanu, Palantir Technologies](https://colocatedeventsna2024.sched.com/event/1jJ2P/cilium-ebpf-day-closing-remarks)

Tuesday, November 12, 2024, 5:25 pm - 5:30 pm MST

The Closing Session for Cilium + eBPF Day NA 2024

##### KubeCon + CloudNativeCon

[Cilium, EBPF, WireGuard: Can We Tame the Network Encryption Performance Gap? - Daniel Borkmann & Anton Protopopov, Isovalent](https://kccncna2024.sched.com/event/1i7lP/cilium-ebpf-wireguard-can-we-tame-the-network-encryption-performance-gap-daniel-borkmann-anton-protopopov-isovalent)

Wednesday, November 13, 2024, 2:30 pm - 3:05 pm MST

To increase data security for cloud and hybrid cloud deployments, many companies, governments, standards, and tenders require data in transit to be protected. However, network encryption comes at a cost - what is the performance impact and how can we reduce it? In this session, we explore how network encryption can be efficiently enforced with Cilium, eBPF, and WireGuard. We dive deep into Cilium’s integration of WireGuard and elaborate on both the management plane and Cilium’s eBPF datapath. We analyze and benchmark what performance cost one can expect and explore opportunities in the Linux kernel to reduce that price. This talk is for operators and security teams that need to encrypt network traffic, but also want to minimize its overhead. The audience will walk away understanding whether network encryption needs to come at a high toll and whether there are opportunities for optimizations.

[From Observability to Enforcement: Lessons Learned Implementing eBPF Runtime Security - Anna Kapuścińska & Kornilios Kourtis, Isovalent](https://kccncna2024.sched.com/event/1i7ma/from-observability-to-enforcement-lessons-learned-implementing-ebpf-runtime-security-anna-kapuscinska-kornilios-kourtis-isovalent)

Wednesday, November 13, 2024, 5:25 pm - 6:00 pm MST

eBPF is getting widely adopted in cloud native runtime security tools like Falco, KubeArmor, and Tetragon. Using eBPF we can collect relevant security events right in the kernel and pass them to Security Engineers for retroactive attack detection and response. Having reliable and complete visibility is great, but wouldn't it be even better to proactively prevent attacks in progress? This talk covers the Tetragon team’s experience moving from security observability to enforcement and lessons learned along the way: from defining security models to hardening interactions between the local kernel and distributed Kubernetes systems. It will deep dive into how eBPF-based enforcement works, why it differs from observability, and the challenges of implementing it. The audience will walk away understanding the inner workings and common pitfalls of eBPF-based runtime security.

[Cilium: Connecting, Observing, and Securing Kubernetes and Beyond with eBPF - Ahmed Bebars, The New York Times; Liz Rice, Isovalent @ Cisco; Joe Stevens, Ascend.io](https://kccncna2024.sched.com/event/1howZ/cilium-connecting-observing-and-securing-kubernetes-and-beyond-with-ebpf-ahmed-bebars-the-new-york-times-liz-rice-isovalent-cisco-joe-stevens-ascendio)

Thursday, November 14, 2024, 11:55 am - 12:30 pm MST

Welcome to Cilium's maintainer track session where you'll get an update on how Cilium is expanding the frontiers of cloud native networking, observability, and security. Cilium is CNCF's most widely adopted CNI, being the default choice for all major cloud providers. This talk dives into the bytecode behind all of the buzz around the project. We'll start with a brief overview of each part of the project before diving into how Cilium is expanding beyond Kubernetes with load balancing and multi-cloud networking and into runtime enforcement with Tetragon. In this session, you'll hear from Cilium contributors and users Isovalent and The New York Times.

[What Agent to Trust with Your K8s: Falco, Tetragon, or KubeArmor? - Henrik Rexed, Dynatrace](https://kccncna2024.sched.com/event/1i7rE/what-agent-to-trust-with-your-k8s-falco-tetragon-or-kubearmor-henrik-rexed-dynatrace)

Thursday, November 14, 2024, 11:55 am - 12:30 pm MST

In the CNCF landscape we have plenty of ebpf-based security solutions that help us protect our k8s cluster from runtime vulnerabilities. On paper though Falco, Tetragon, and KubeArmor look very similar. Eventually, you have to make a choice on which one best fits your needs. To give you additional insights to make your decision join this session. We have run extensive benchmarks against those three solutions and will answer the following questions that came out of our testing: - What are the different feature sets? - What about the performance impact of each agent? - Which privileges does each solution need? - What are the pros and cons of the three options?

[Understanding Kubernetes Networking in 30 Minutes - Ricardo Katz, Broadcom & James Strong, Isovalent at Cisco](https://kccncna2024.sched.com/#)

Thursday, November 14, 2024 4:30 pm - 5:05 pm MST

You are learning Kubernetes and started to face concepts like Pod CIDRs, Services, CNI, kube-proxy? Welcome! you have reached the amazing area of Kubernetes networking! We all have already been there and know how complex it may seem on the beginning, but in this talk, Ricardo and James will demystify the Kubernetes network concepts and model on a fun way, exploring how it is designed, why the is a "pause" container on every Pods, how the communication between Pods work, what are kube-proxy and CNI and their importance. In the end of this talk we expect you to get your learning path on Kubernetes Networking clear to better understand not only what are the concepts about, but also see on a live demo how every component correlates and makes the communications possible on a Kubernetes cluster.

[ Contribfest: Kickstart Your eBPF Journey with Tetragon](https://kccncna2024.sched.com/event/1hoyj/contribfest-kickstart-your-ebpf-journey-with-tetragon)

Thursday, November 14, 2024, 4:30 pm - 6:00 pm MST

Tetragon and eBPF have a lot of buzz and this is your chance to get involved diving into the bytecode or docs! Tetragon’s docs are still young and your new contributor's perspective will be a superpower for spotting issues or unclear wording in the various quickstarts, guides, and concepts pages. The project’s CLI, tetra, is another great opportunity for those interested in code contributions around ease of use, testing, and consistency in flags and output. Tetragon’s documentation tech stack uses Markdown, built with Hugo, and a customized Docsy theme. The CLI is written in Go with the Cobra library and uses gRPC to communicate with the agent. While this session should help you get more familiar with Tetragon and can lead to more contributions in the future, those technologies are also used in Kubernetes and many other CNCF projects.

[Pick My Project! Lessons Learned from Interviewing 20+ End Users for Cloud Native Case Studies - Shedrack Akintayo & Bill Mulligan, Isovalent at Cisco](https://kccncna2024.sched.com/event/1i7pZ/pick-my-project-lessons-learned-from-interviewing-20-end-users-for-cloud-native-case-studies-shedrack-akintayo-bill-mulligan-isovalent-at-cisco)

Thursday, November 14, 2024, 5:25 pm - 6:00 pm MST

Cloud native projects can promise the moon in their READMEs, but have you ever wondered what causes end users to adopt a project? Shedrack and Bill have interviewed over 20 companies in industries ranging from media to financial services about why they picked a project for their cloud native platform. In this talk, they will reveal what end users truly want when adopting cloud native technologies and what the forcing function is for each of them. You’ll hear firsthand accounts of the triumphs and tribulations faced by companies like Bloomberg, DigitalOcean, The New York Times, and more as well as the specific benefits these organizations are reaping, from enhanced security and observability to improved performance and cost savings. Additionally, they’ll teach other projects their process for creating impactful case studies. By the end, the audience will understand the real-world applications and advantages of cloud native technologies and why end users pick a project.

[Seeing Double? Implementing Multicast with eBPF and Cilium - Louis DeLosSantos, Isovalent at Cisco](https://kccncna2024.sched.com/event/1i7qG/seeing-double-implementing-multicast-with-ebpf-and-cilium-louis-delossantos-isovalent-at-cisco)

Friday, November 15, 2024, 11:55 am - 12:30 pm MST

Multicast is a popular networking technology used in finance, telecommunications, and media CDNs, among others to efficiently replicate and deliver data streams to multiple clients. However, this advantage can be overshadowed by the complexity involved in configuring the necessary infrastructure leaving the overworked platform team rather than the end users seeing double. To combat this complexity, Cilium explored using eBPF to implement pod-to-pod multicast delivery within a Kubernetes cluster. This talk will provide both a high and low-level understanding of how eBPF can be used to implement multicast delivery. It will discuss how Cilium’s multicast works and the hurdles faced by the project along the way. By the end of this talk, the audience will have a better understanding of how multicast functions, how eBPF can be used in place of traditional multicast infrastructure, and how Cilium can be used as a multicast-enabled CNI, letting your audience - and not you- see double.

[Micro-Segmentation and Multi-Tenancy: The Brown M&Ms of Platform Engineering](https://kccncna2024.sched.com/event/1i7qS/micro-segmentation-and-multi-tenancy-the-brown-mms-of-platform-engineering-jim-bugwadia-nirmata-rachael-wonnacott-fidelity-international?iframe=no)

Friday November 15, 2024 2:00pm - 2:35pm MST

A key requirement for internal developer platforms is that they serve multiple workloads. The reality of platform engineering is that while it seeks to lower the barrier to entry for teams to deliver applications, it must also balance cost and ensure appropriate levels of security. It’s therefore essential to consider how application components running on shared infrastructure are allowed to communicate with each other and weigh up the cost of each architecture. In industry, we have seen differing approaches to deploying Kubernetes to achieve these goals, from multiple single-tenant clusters through to shared clusters that deliver namespaces-as-a-service. Rachael and Jim will define the concepts of multi-tenancy and micro-segmentation for cloud native systems, explain why they are critical to success with platform engineering. They will also show real-world examples of how they can be implemented, and demonstrate full automation using best practices like GitOps and Policy as Code.

[Seccomp and eBPF; What’s the Difference? Why Do I Need to Know? - Natalia Reka Ivanko & Duffie Cooley, Isovalent @ Cisco](https://kccncna2024.sched.com/event/1i7qb)

Friday, November 15, 2024, 2:00 pm - 2:35 pm MST

Containers in Kubernetes share a common Linux kernel so how can we limit access where it isn’t required so we can follow the principle of least privilege? Join Natalia and Duffie as they each explore different approaches to harden your container security with Secure Computing (seccomp) and eBPF! The talk will begin with an overview and comparison between seccomp and eBPF and how they both can solve the same problem - limiting access to the Linux Kernel that all containers share. This will be a fun talk, showing each solution with a live demo. You will leave this talk with a better understanding of how to limit what system calls a process can make and restrict your containers’ behavior to only access the files, binaries and external DNS names they need and nothing more. Which is the right solution for your environment? Come and learn about two of the commonly used technologies in use today!

[The Key Value of Etcd Over Custom Resources: Scalability - Jef Spaleta, Isovalent at Cisco](https://kccncna2024.sched.com/#)

Friday, November 15, 2024 2:55 pm - 3:30 pm MST

Cilium defaults to using Kubernetes Custom Resources to hold Cilium specific internal state, however when the cluster is large enough, the Kubernetes API becomes a bottleneck on performance. To scale a cluster to hundreds of nodes, Cilium can be configured to use a dedicated external etcd instance. This talk will discuss the details of what the external etcd looks like from an operator perspective, and explore why Cilium uses an external etcd for enhanced scalability. It will cover how to manage a cluster by bypassing the Kubernetes API and interacting only with the cluster's etcd key-value store - and also why it might be a bad idea. Get a taste of what's possible by bypassing the Kubernetes API and interacting with the etcd API directly, and learn why Cilium has an option to use a dedicated etcd deployment, not shared by the Kubernetes API, for holding Cilium state and the scalability benefits it can bring to your cluster.

#### Hubble for Network Observability and Security (Part 3): Leveraging Hubble Data for Network Security

URL: https://cilium.io/blog/2024/08/27/hubble-for-network-security-and-observability-part-3
Date: 2024-08-27
Categories: How-To, Technology

**_Author: Shedrack Akintayo, Isovalent_**

In this final part of the series, we'll focus on a critical aspect of network observability: security. In today's rapidly evolving threat landscape, the ability to leverage observability data for security observability is not just beneficial—it's essential. Let's take a look at how Hubble's rich observability data can be used to enhance your Kubernetes network security posture.

##### How Do I Interpret Hubble Data?

Hubble provides a wealth of security-relevant data that can be invaluable for identifying potential threats. Let's dive deeper into the types of data Hubble offers and how to interpret them:

###### Flow Logs

These detailed logs capture all network interactions, including:

- Source and destination IP addresses and ports
- Protocols used (TCP, UDP, ICMP, etc.)
- Kubernetes-specific metadata (pod names, namespaces, labels)
- Timestamp and duration of connections

With flow logs, you can look for unusual patterns such as unexpected connections between pods, connections to external IP addresses, or sudden increases in traffic volume.

Just like PostFinance does:

**_“We export the Hubble metrics, flows, and particularly the connection drops. With the source label and namespace label, we can quickly help when one of our customers or teams is having issues or a bad configuration in the namespace.”_** - Clément Nussbaumer, Systems Engineer, PostFinance ([Read the case study](https://www.cncf.io/case-studies/postfinance/))

###### Policy Verdicts

Hubble logs whether traffic was allowed or denied based on your network policies. In a situation where you constantly see a high number of denied connections, this could indicate either an attempted attack or a misconfigured policy. Use the Hubble service map to visualize and investigate the source and destination of denied traffic to determine the cause.

###### DNS Queries

Hubble captures DNS-related traffic, including query types and responses. You can monitor for unusual DNS patterns, such as a high volume of queries to uncommon domains or attempts to resolve known malicious domains.

###### HTTP Header

For HTTP traffic, Hubble can provide information about methods, paths, and headers. With this data, you can investigate suspicious user agents, unexpected request patterns, or attempts to access sensitive endpoints.

###### TLS Information

Hubble provides details about TLS versions and cipher suites used in encrypted connections. This data can help you identify potentially insecure TLS versions or cipher suites that could cause vulnerable connections.

###### Performance Metrics

Hubble provides data on latency, throughput, and error rates. Sudden changes in these metrics could indicate a DDoS attack, misconfiguration, or a compromised application.

The API company, WS02 spoke about how Hubble helps them fix performance issues that their customers encounter:

**_“If a customer has some performance issues, we directly use the Hubble CLI to look at the Layer 3 packets to see where things are failing and what optimization we can do.”_** - Lakmal Warusawithana, Senior Director – Cloud Architecture, WSO2 ([Read the case study](https://www.cncf.io/case-studies/wso2/))

##### How Can I Make Security Decisions with Hubble Data?

With Hubble's observability data, you can make informed security decisions. Let's explore some common scenarios and how Hubble can help:

###### Detecting Potential Intrusions

**Scenario**: You notice unexpected connections to a sensitive database service.

**Action**: Use Hubble to trace the source of these connections. If they come from an unauthorized pod or external IP, investigate immediately and consider tightening your network policies.

###### Identifying Misconfigured Policies

**Scenario**: Hubble shows a high number of denied connections between services that should be able to communicate.

**Action**: Review your network policies and adjust them to allow legitimate traffic while maintaining the principle of least privilege.

###### Spotting Data Exfiltration Attempts

**Scenario**: Hubble logs show unusual outbound traffic patterns, particularly large data transfers to external IP addresses.

**Action**: Investigate the source pod and the nature of the data being transferred. If unauthorized, isolate the pod and conduct a thorough security audit.

###### Responding to DDoS Attacks

**Scenario**: Hubble metrics show a sudden spike in traffic to a particular service.

**Action**: Analyze the traffic patterns to determine if it's a legitimate increase or a potential DDoS attack. If it's an attack, use Hubble data to identify the sources and block them at the network level with network policies.

###### Investigating Lateral Movement

**Scenario**: After detecting a compromised pod, you need to determine if the attacker [moved laterally](<https://en.wikipedia.org/wiki/Lateral_movement_(cybersecurity)>) within the cluster.

**Action**: Use Hubble's flow logs to trace all connections made by the compromised pod, identifying any potential spread of the attack.

###### Optimizing Network Policies

**Scenario**: You want to implement a zero-trust network model in your cluster.

**Action**: Analyze Hubble's flow logs to understand the actual communication patterns between services. Use this information to create precise, least-privilege network policies.

##### What Are Some Best Practices for Incident Response?

Incorporating Hubble into your incident response strategy can significantly improve your ability to detect, investigate, and mitigate security incidents. Here are some best practices:

- **Set Up Alerts**: Configure alerts based on Hubble metrics to notify your team of potential security events in real time.

- **Establish Baselines**: Store and use Hubble's historical data to establish normal traffic patterns, making it easier to spot anomalies. For example, you can enable [Hubble Metrics](https://docs.cilium.io/en/stable/observability/metrics/) and have Prometheus scrape these into a data store.

- **Integrate with SIEM**: Feed Hubble's rich data into your Security Information and Event Management (SIEM) system for correlation with other security events.

- **Practice Forensics**: In the event of an incident, use Hubble's detailed logs to perform thorough forensic analysis.

- **Regular Policy Reviews**: Leverage Hubble's policy verdict data to review and refine your network policies regularly.

##### Taking Your Platform And Network Security to The Next Level With Tetragon

While Hubble provides excellent network-level observability, [Tetragon](https://tetragon.io/) takes security observability to the next level by providing kernel and process-level insights. Tetragon, another eBPF-powered tool under Cilium, can:

- Monitor process executions and file access
- Detect and prevent unauthorized binaries from running
- Provide real-time alerts on suspicious activities at the host level
- Observes network traffic activity (TCP, UDP, TLS, DNS, HTTP, and more)

Tetragon correlates information from the runtime layer to the network layer, bringing together deep context around the network traffic that Hubble is observing. For example, Tetragon observes TLS traffic on a specific workload and captures rich information around the process ID, parent process, TLS versions, and Kubernetes identity of the workload. By combining Hubble's network insights with Tetragon's system-level observability, you can create a comprehensive security monitoring solution for your Kubernetes environment. To get started with Tetragon, you can try out the guides in its [official documentation](https://tetragon.io/docs/getting-started/) or this [hands-on lab](https://isovalent.com/labs/security-observability-with-ebpf-and-cilium-tetragon/) from Isovalent.

##### Final Words

Throughout this series, we've explored the critical role that network observability plays in managing and securing Kubernetes environments. We began by examining the challenges DevOps and platform engineering teams face in achieving comprehensive visibility into their containerized networks. We then delved into Hubble's powerful features and how they address these challenges head-on.

In [Part 1](https://cilium.io/blog/hubble-for-network-security-and-observability-part-1/), we introduced the concept of network observability and its importance in the Kubernetes ecosystem. We highlighted the limitations of traditional monitoring tools and the unique requirements of modern, dynamic container environments.

[Part 2](https://cilium.io/blog/hubble-for-network-security-and-observability-part-2/) focused on Hubble's core capabilities, demonstrating how its deep, granular visibility into network flows, real-time metrics, and intuitive visualizations empower teams to monitor, troubleshoot, and optimize their Kubernetes networks effectively.

Finally, in this part, we explored leveraging Hubble's rich data to enhance network security. We discussed practical use cases, from detecting potential threats to improving incident response strategies, and even touched on how including Tetragon can take your platform security to the next level.

#### Hubble for Network Observability and Security (Part 2): Utilizing Hubble for Network Observability

URL: https://cilium.io/blog/2024/08/19/hubble-for-network-security-and-observability-part-2
Date: 2024-08-19
Categories: How-To, Technology

**_Author: Shedrack Akintayo, Isovalent_**

In this second installment of this series, we'll delve deeper into Hubble's powerful observability features and explore how they can enhance your Kubernetes network observability strategy. We'll also examine practical use cases, such as monitoring network traffic for dropped packets and identifying abnormal patterns in your network.

In the [first part of the series](https://cilium.io/blog/2024/08/14/hubble-for-network-security-and-observability-part-1/), we introduced Cilium and Hubble, highlighting their significance in modern Kubernetes environments. We guided you through their setup process and showcased real-world case studies of organizations successfully leveraging Cilium and Hubble in production environments.

Now, let's take on a comprehensive exploration of Hubble's capabilities for modern network observability!

##### Why Is Network Observability So Crucial In Kubernetes Environments?

Network observability is a crucial component of maintaining large-scale and secure Kubernetes environments. It is one of the most critical aspects of infrastructure operations that platform engineering and DevOps teams worry about. This concern stems from the fact that Kubernetes, for all its power and flexibility, provides minimal built-in network observability tools. This lack of visibility can leave operations teams blind when understanding network behavior, troubleshooting issues, or detecting potential security threats.

As Clément Nussbaumer, Systems Engineer at PostFinance puts it: **“_In terms of observability in Kubernetes, it was quite tricky to know exactly where the packets were flowing and what could be blocking them. Basic Linux networking tools, like tcpdump, lack the context of cluster topology and quickly become complicated with dynamic pod IPs._”** ([Read the case study](https://www.cncf.io/case-studies/postfinance/))

This sentiment is echoed across the industry. A [recent survey by the Cloud Native Computing Foundation](https://www.cncf.io/reports/cncf-annual-survey-2023/) cited monitoring and observability as a significant challenge in managing environments with a large number of containers.

##### What Do End Users Need?

Given these difficulties, it's evident that Kubernetes users require comprehensive solutions to address this observability gap. The complexities of modern containerized environments demand tools that go beyond basic network monitoring. To effectively manage and secure their Kubernetes clusters, end users need a suite of advanced observability features that provide deep insights, context-aware analysis, and actionable intelligence.

Let's explore the key capabilities that Hubble offers as part of the platform, which has become essential for tackling the observability challenges in Kubernetes environments:

1. **Real-time visibility into network flows:** Users require instant insights into how traffic moves through their Kubernetes clusters. This includes understanding the source and destination of packets, the protocols being used, and the volume of traffic between different services.
2. **Detailed packet-level analysis:** For troubleshooting complex issues, teams need the ability to drill down to individual packet details. This granular view can help identify specific problems like malformed packets or unexpected protocol behaviors.
3. **Service dependency mapping:** As applications grow more complex, understanding the relationships between different services becomes crucial. Customers need tools that can automatically map out service dependencies and visualize traffic patterns.
4. **Performance metrics and alerting:** Teams require comprehensive performance metrics, including latency, throughput, and error rates. Moreover, they need the ability to set up alerts based on these metrics to address issues before they impact users proactively.
5. **Historical data retention and analysis:** Post-mortem analysis is a critical part of improving system reliability. Users need the ability to store and analyze historical network data to understand trends, investigate past incidents, and prevent future issues.
6. **Security-focused observability:** With the increasing focus on cybersecurity, teams need observability tools that can help detect potential threats. This includes identifying unusual traffic patterns, detecting potential DDoS attacks, and spotting attempts at data exfiltration.
7. **Cross-stack correlation:** Network issues often don't exist in isolation. Users need the ability to correlate network events with application logs and infrastructure metrics for a holistic view of their system's health.
8. **Policy verification:** As network policies become more complex, teams need ways to verify that these policies are being enforced correctly. This includes observing which traffic is being allowed or denied based on current policies.
9. **Easy-to-use interfaces:** Whether through a GUI or CLI, customers need intuitive interfaces that allow both seasoned engineers and newer team members to gain insights from the observability data quickly.
10. **Integration with existing tools:** Most organizations already have a suite of monitoring and observability tools. They need new solutions to integrate seamlessly with their existing ecosystem, whether that's pushing data to a centralized logging system or alerting through established channels.
11. **Scalability:** As Kubernetes clusters grow, observability solutions need to scale accordingly without significant performance overhead or cost increases.

These capabilities are crucial for teams running Kubernetes workloads. Modern observability solutions must address these requirements to significantly enhance an organization's ability to monitor, troubleshoot, and secure its Kubernetes environments effectively. The question now is: **Does Hubble have the features in place to meet these requirements for a modern Kubernetes observability solution?** In the next section, we'll explore the capabilities that Hubble provides to platform engineering and DevOps teams, demonstrating how it can enhance their overall network observability.

##### What Observability Features Are Available in Hubble?

Hubble leverages [eBPF](http://ebpf.io) to provide deep, granular visibility into your Kubernetes network. This advanced approach enables Hubble to meet most of the requirements we previously discussed. Let's explore the key features that Hubble offers to enhance network observability:

###### Comprehensive Network Flog Logs

Hubble captures detailed logs of all network interactions, providing a wealth of information. This comprehensive set of information allows for deep analysis of network behavior, troubleshooting of issues, and detection of potential security threats. It provides a deeper view of network interactions within your Kubernetes cluster, enabling you to understand not just what is communicating but how and why. The following is some of the information that Hubble provides in its network flow logs:

- Source and destination IP addresses and ports
- Protocol information (TCP, UDP, ICMP, HTTP, Kafka, gRPC, etc.)
- Kubernetes-specific metadata (pod names, namespaces, labels)
- DNS queries and responses
- HTTP headers and methods
- Network Policy verdicts and names

###### Metrics

Hubble generates a wide range of metrics from your cluster/application state that gives you an in-depth view of what is going on in your network. Hubble metrics can also be exported when explicitly requested by clients. This is useful for teams that already have observability tools that they want to pair with Hubble.

You can find a comprehensive list of available metrics and their options in the [Hubble exported metrics](https://docs.cilium.io/en/stable/observability/metrics/#hubble-exported-metrics) section of the Cilium documentation.

###### Service Maps

The Hubble UI provides a real-time graphical [service map](https://docs.cilium.io/en/stable/gettingstarted/hubble/#service-map-hubble-ui) that visualizes the flow of the network and relationships between components (services, pods, namespaces) in a Kubernetes cluster. It also visualizes the majority of data from the network flow logs.

###### Protocol Visibility

With the support of Cilium and Envoy, Hubble provides the ability to [monitor Layer 7 traffic](https://docs.cilium.io/en/stable/observability/visibility/#proxy-visibility). This gives you insights into L7 protocols like HTTP, gRPC, and Kafka.

##### The Hubble UI And CLI

Let’s see these capabilities in action with a walkthrough of the [Hubble UI](https://docs.cilium.io/en/stable/gettingstarted/hubble/) and [CLI](https://docs.cilium.io/en/stable/gettingstarted/hubble_cli/) deployed to a Kubernetes platform.

###### Hubble UI

The Hubble UI provides an intuitive, visual representation of network traffic between pods and services in specific namespaces within your Kubernetes clusters. You can access the Hubble UI by enabling Hubble from the Cilium CLI or during the installation of Cilium and Hubble with Helm.

For example, to enable the Hubble UI using the Cilium CLI, run the following command:

```bash
cilium hubble enable -ui
```

Verify that the Hubble UI is enabled with:

```bash
cilium status -–wait

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

DaemonSet              cilium-envoy       Desired: 4, Ready: 4/4, Available: 4/4
DaemonSet              cilium             Desired: 4, Ready: 4/4, Available: 4/4
Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            hubble-relay       Running: 1
                       hubble-ui          Running: 1
                       cilium-operator    Running: 1
                       cilium             Running: 4
                       cilium-envoy       Running: 4
Cluster Pods:          14/14 managed by Cilium
Helm chart version:
Image versions         hubble-ui          quay.io/cilium/hubble-ui:v0.13.1@sha256:e2e9313eb7caf64b0061d9da0efbdad59c6c461f6ca1752768942bfeda0796c6: 1
                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.13.1@sha256:0e0eed917653441fded4e7cdb096b7be6a3bddded5a2dd10812a27b1fc6ed95b: 1
                       hubble-relay       quay.io/cilium/hubble-relay:v1.16.0@sha256:33fca7776fc3d7b2abe08873319353806dc1c5e07e12011d7da4da05f836ce8d: 1
```

Next, launch the Hubble UI in a new tab:

```bash
Cilium hubble ui

ℹ️  Opening "http://localhost:12000" in your browser...
```

Here is an example of what the Hubble UI service map and flow logs look like:

On further inspecting the `deathstar` pod by clicking on it, we can see more details.

From the above example, we can see all the labels attached to the `deathstar` pod. We can also observe network flow logs containing communication between the pods in the default namespace.

You can also update the network flow column to show other information available about pods and services in your cluster like this:

The Hubble UI offers much more functionality, such as tracking `DROPPED` packets, viewing HTTP methods and headers, and other advanced features.

You can learn more about the Hubble UI in this [video](https://youtu.be/M0hZRvF8CpQ).

###### Hubble CLI

The Hubble CLI offers a powerful command-line interface for those who prefer terminal-based interactions or need to automate observability tasks. It’s like an advanced `tcpdump`.

It provides access to Hubble's rich observability data, allowing users to quickly query flow logs, analyze network traffic, and troubleshoot connectivity issues. With its wide range of filtering options and output formats, the Hubble CLI is an essential tool for DevOps teams seeking to quickly gain deep insights into their Kubernetes network behavior.

To use the Hubble CLI, you must install it locally by following the [instructions](https://docs.cilium.io/en/stable/gettingstarted/hubble_setup/#install-the-hubble-client) in the official documentation.

Once installed, you can interact with the Hubble CLI using various commands. Here are some examples:

```bash
# Observe real-time flows
hubble observe

# Filter flows by namespace
hubble observe --namespace kube-system

# View HTTP-specific information
hubble observe --protocol http

# View specific information by HTTP method
hubble observe --http-method get

# Check hubble relay status
hubble status
```

Below is an example of using the Hubble CLI to query the network flow logs from the `deathstar` pod:

Please check out the [Hubble Cheatsheet](https://isovalent.com/blog/post/cilium-hubble-cheat-sheet-observability/) for a comprehensive list of Hubble CLI commands.

##### What are Some Practical Use Cases For Hubble?

As we've explored Hubble's powerful features, it's important to understand how these capabilities translate into real-world benefits for Kubernetes operators and developers. In this section, we'll delve into practical use cases that demonstrate Hubble's value in day-to-day operations.

###### Monitoring network traffic

Use Hubble to gain insights into your application's communication patterns. For example, you can easily identify which services are communicating most frequently or detect unexpected connections.

As Jan Jansen, Platform Engineer, G DATA puts it:

**_“With Hubble, you can give developers access to the network visibility in a much better way. We don’t have to give them full cluster access to see the networking issues.”_** ([Read the case study](https://www.cncf.io/case-studies/g-data-cyberdefense/))

###### Identifying abnormal patterns or potential threats

Hubble's real-time flow logs and metrics can help you spot anomalies quickly. Unusual spikes in traffic, connections to unexpected IP addresses, or abnormal protocol usage can all be indicators of potential security issues.

As Frank Potter from Rabobank notes:

**_“What I like about Hubble is that it gives a great overview of what is going on. You can see the traffic and also graphically, you can see how it flows. In the beginning, when we were allowing some traffic, we weren’t exactly sure what we needed but with Hubble, we could see all the traffic and even found some things we had missed.”_** ([Read the case study](https://www.cncf.io/case-studies/rabobank/))

###### Troubleshooting connectivity issues

When applications misbehave, Hubble can help you pinpoint the root cause. Whether it's a misconfigured network policy or a DNS resolution problem, Hubble provides the visibility you need to resolve issues quickly.

As Federico, Principal Engineer at Meltwater notes:

**_“Besides just the UI, Hubble is an easier way to debug network issues and see network traffic. I don’t need to use tcpdump anymore. We understand better what is going on between all the different components and workloads in our system.”_** ([Read the case study](https://www.cncf.io/case-studies/meltwater/))

##### Final Words

Hubble brings much-needed clarity to the complex world of Kubernetes networking. Providing deep, real-time visibility into network flows empowers operations teams to maintain robust, secure, and efficient Kubernetes environments.

In the next post, we'll explore how Hubble's observability features can be leveraged for advanced security enforcement, taking your Kubernetes network security to the next level.

#### Hubble for Network Observability and Security (Part 1): Introduction to Cilium and Hubble

URL: https://cilium.io/blog/2024/08/14/hubble-for-network-security-and-observability-part-1
Date: 2024-08-14
Categories: Technology

_Author: Shedrack Akintayo, Isovalent_

In the cloud native ecosystem, Kubernetes has emerged as the de facto standard for container orchestration, offering powerful capabilities for managing containerized applications. However, this power brings complex challenges, particularly for network observability and security. As Kubernetes environments grow in scale and complexity, traditional networking approaches often fall short. This is where Cilium and Hubble come in, revolutionizing Kubernetes networking, observability, and security with new ideas and implementations. Cilium leverages [eBPF](https://ebpf.io/) to provide high-performance, identity-based networking, and security, while Hubble offers deep visibility into network flows and security policies. Together, they enable fine-grained control and observability at the network layer, addressing the unique demands of modern, dynamic Kubernetes environments.

This three-part series dives deep into Hubble, exploring how it can transform your Kubernetes network observability and security strategies. In part 1 of the series, we’ll start by understanding Cilium and Hubble and why they're essential in modern Kubernetes environments.

##### What is Cilium?

[Cilium](cilium.io) is the leading cloud native solution for providing, securing, and observing network connectivity between workloads. At its core, Cilium leverages eBPF to provide high-performance networking, security, and observability.

**Some key features of Cilium include:**

- [Layer 3/4 and Layer 7 network policies](https://cilium.io/use-cases/network-policy/)
- [Load balancing](https://cilium.io/use-cases/load-balancer/)
- [High performance cloud native networking](https://cilium.io/use-cases/cni/)
- [Service mesh integration](https://cilium.io/use-cases/service-mesh/)
- [Kube-proxy replacement](https://cilium.io/use-cases/kube-proxy/)

One of Cilium's standout benefits is its ability to enforce network policies at the kernel level, providing both enhanced security and improved performance compared to traditional [iptables-based solutions](https://isovalent.com/blog/post/why-replace-iptables-with-ebpf/).

##### What is Hubble?

Hubble is Cilium's observability layer, offering deep insights into your Kubernetes cluster's network. It's like a powerful microscope for your cluster's network traffic, allowing you to see and understand what's happening at a granular level.

**Some key features of Hubble include:**

- [Real-time network flow monitoring](https://cilium.io/use-cases/network-flow-logs/)
- [Service dependency mapping](https://cilium.io/use-cases/service-map/)
- [Advanced network visibility](https://cilium.io/use-cases/protocol-visibility/)
- [API aware observability (L7)](https://docs.cilium.io/en/stable/observability/visibility/)

Hubble's integration with Cilium provides contextual information about network flows, making it easier to understand and troubleshoot complex networking scenarios in Kubernetes environments.

##### Why Use Cilium and Hubble for Network Security?

Traditional network security tools often need help to keep up with the dynamic nature of Kubernetes environments. Cilium and Hubble offer several advantages:

- **Native Kubernetes integration**: Cilium and Hubble are designed from the ground up for Kubernetes, understanding its unique networking model and security requirements.
- **eBPF-powered performance**: By leveraging eBPF, Cilium can enforce security policies with minimal overhead, often outperforming traditional solutions.
- **Deep visibility**: Hubble provides unprecedented visibility into network flows, making detecting and responding to security threats easier.
- **API-aware visibility and security**: Cilium can understand and secure application-layer (L7) protocols, allowing for more granular and effective security policies, Hubble can provide application-level visibility such as HTTP request headers and DNS Queries

##### Setting Up Cilium and Hubble

The best way to get started with Cilium and Hubble in your environment is to read through the [official introduction](https://docs.cilium.io/en/stable/overview/intro/#intro) in the Cilium documentation. The [official documentation](https://docs.cilium.io/en/stable/gettingstarted/hubble_setup/#hubble-setup) guides you through installing Cilium and Hubble, configuring Hubble to suit your environment, and using the Hubble UI.

Explore the Isovalent repository of [hands-on Labs](http://cilium.io/labs) for a live approach to learning how to use Cilium and Hubble in your Kubernetes cluster. These labs educate you on the various features that both Cilium and Hubble provide and how to use them effectively.

However, if you're not ready to fully switch to Cilium as your CNI, don't worry. You can still leverage Hubble's powerful observability features through CNI chaining. This approach allows you to benefit from Hubble's insights while maintaining your current networking setup. To learn more about this method, check out this detailed guide on [using Cilium Hubble for observability in CNI chaining mode](https://isovalent.com/blog/post/cilium-hubble-observability-cni-chaining-mode/). This flexibility ensures that you can benefit from Hubble's capabilities regardless of your current setup.

##### Hubble In The Wild

Let's look at some Cilium users who are benefitting from using Hubble in their network infrastructure.

###### Utmost

Utmost, a workday-native workforce management system company, leveraged Hubble to significantly enhance visibility into their operations. Introducing Hubble into their Kubernetes platform has improved their monitoring, auditing, and compliance processes, making it an indispensable tool in their setup.

**_"Hubble having a UI and being able to pump those metrics into our monitoring tools is amazing because if there is an issue, attack, or any kind of troubleshooting, you need to be able to see what's going on."_** - Andrew Holt, Senior Systems Engineer

[Read the full case study](https://www.cncf.io/case-studies/utmost/)

###### Rabobank

Rabobank is a multinational bank offering banking and financial services in the Netherlands. Hubble became an unexpected gem in their API platform infrastructure. It offered Rabobank's platform team a comprehensive view of their network traffic, transforming complex flows into easily digestible visual insights. For Rabobank, Hubble serves as both a discovery tool and a crucial component in their network observability strategy.

**_"I didn't know about Hubble, so that was something we discovered along the way and it was a nice feature to have. What I like about Hubble is that it gives a great overview of what is going on. You can see the traffic and also graphically, you can see how it flows. In the beginning, when we were allowing some traffic, we weren't exactly sure what we needed but with Hubble, we could see all the traffic and even found some things we had missed."_** - Frank Potter, Rabobank

[Read the full case study](https://www.cncf.io/case-studies/rabobank/)

###### DigitalOcean

DigitalOcean is a cloud service provider with a target market of small to medium-sized businesses, developers, and startups. Hubble provided DigitalOcean and its customers with deep insights into network traffic, transforming complex flows into easily digestible visual representations. For DigitalOcean, Hubble provides an internal debugging tool to help troubleshoot networking issues in their platform, and customers also benefit from Hubble’s features, as DigitalOcean expanded Hubble to be also consumed by their customers.

**_“Regarding observability, with Hubble, it’s night and day. With Hubble, it’s easy to see where traffic is going and where it’s denied. In a distributed world, observability is a key feature. Hubble was one of the features our customers most frequently requested to be enabled, and we recently enabled it."_**

**_One feature they love is the Hubble UI, especially the services map feature, where you can see all of your pods and their communication. Our customers greatly value this because it gives them a better understanding of their applications and network connections._**

**_Internally, we also use Hubble because we want easy debuggability for network policies. We didn’t want to log into every Cilium agent and run the Cilium monitor command to identify issues; we wanted one central place that could simplify this process, and that’s what we got with Hubble.”_** - Ingo Gottwald, Senior Engineer, DigitalOcean

[Read the full case study](https://www.cncf.io/case-studies/digitalocean/)

###### Trendyol

Trendyol is a leading e-commerce platform that provides an extensive selection of products spanning categories such as fashion, technology, and home furnishings. In Trendyol's large-scale e-commerce infrastructure, where observability is crucial, Hubble has proven to be a key asset for maintaining smooth operations and providing network observability.

**_"With Hubble, if we need to debug something or investigate the source and destination of certain traffic that comes into our clusters we can quickly see that. It has enabled members of our team to monitor the network connectivity within our clusters easily."_** - Emin AKTAŞ, Platform Engineer, Trendyol

[Read the full case study](https://www.cncf.io/case-studies/trendyol/)

###### G Data CyberDefense

G DATA CyberDefense is a German software company that specializes in endpoint protection and cybersecurity services, including penetration testing. Hubble offers G DATA's development teams deep insights into network traffic without requiring full cluster access. For G DATA, Hubble serves dual roles:

1. A network visibility tool
2. A crucial component in their security-focused infrastructure

In G DATA's infrastructure, where visibility and control are paramount, Hubble has proven to be a vital tool for maintaining robust security while improving developer productivity.

**_"With Hubble, you can give developers access to the network visibility in a much better way. We don't have to give them full cluster access to see the networking issues."_** - Jan Jansen, Platform Engineer, G DATA

[Read the full case study](https://www.cncf.io/case-studies/g-data-cyberdefense/)

##### Final Words

Cilium and Hubble represent a significant leap forward in Kubernetes network security and observability. By providing deep, context-aware visibility and high-performance security enforcement, they address many of the challenges faced in securing modern, dynamic Kubernetes environments.

In the next part of this series, we'll explore how to utilize Hubble for comprehensive network observability. We will dive into its features and show you how to extract valuable insights from your Kubernetes network traffic.

#### What Good Governance Looks Like (My Experience as an LFX Mentee for Cilium)

URL: https://cilium.io/blog/2024/05/30/what-good-governance-looks-like
Date: 2024-05-30
Categories: Community

When I was early in my programming journey and learned what open source was, I immediately knew I wanted to eventually get involved in some capacity. There are many aspects I love about it: the sense of community, the collaboration of talented people pushing forward new technological innovations, the creation of software that is free and available for anyone to use. I knew there was a potential for me to learn and grow through my own involvement as well as contribute to an ecosystem I valued.

However, after purusing the [Linux Foundation Mentorship portal](https://mentorship.lfx.linuxfoundation.org/#projects_all), I was starting to get a little disheartened. At the time of applying (at the end of February, the tailend of getting into the Spring term), I wasn’t seeing a lot of opportunities that matched with my skillset as a recent web development bootcamp grad. I thought perhaps the LFX Mentorship may not be the right fit for the skills I had at the time, and I should keep looking for open source opportunities elsewhere.

Right before throwing in the towel, one particular project caught my eye. It stood out among the rest as it was a writing-focused project, with its only required skill being Markdown. I had never heard of open source governance up until this point, but it seemed like something I could get into. I sent in one of my final papers for my religious studies degree as a writing sample, and proceeded to forget about it.

It was to my surprise that about a week later I was informed that I had been accepted. I was completely blown away. From applying, to learning I was accepted, to actually starting the mentorship project was about a 2-week span – a total whirlwind!

I have now had the opportunity to extensively research multiple systems of governance within the CNCF landscape and use what I’ve learned to edit and add to Cilium’s existing governance docs, as well as create new ones.

As I am now on the tailend of the LFX Mentorship program, I wanted to share some of what I have learned being immersed in governance for these past 12 weeks. I came in knowing exactly nothing about how open source projects operate, and am now leaving with a solid knowledge base to jump off of. It has been fun to demystify this ecosystem, as I now feel like I know how to navigate an open source project, what to look for, and how to contribute. I have been introduced to a whole new world, a world that I will continue to interact with in some capacity (even just as an end-user, or ‘Adopter’ in open source language) for the rest of my career.

---

To begin my journey through open source project governance, I started by reading through many CNCF projects’ governance docs. I then created a list of the top 3 qualities that my favorites had and attempted to bring them into my work with the governance docs in Cilium where I could.

##### 1. Clear Communication of Values

It is a common practice to see values listed as one of the first things on a governance document. Many spout inspiration from Kubernetes ([and for good reason](https://www.kubernetes.dev/community/values/)). I did always appreciate seeing a non-Kubernetes inspired list, simply because it often felt more tailored to the specific project. An intentional list of values can be a great introduction to the culture of the project to a potential contributor.

However, perhaps even more powerful than simply listing the values at the top is continuing to embody these values in the rest of the documents. These values can be embedded in your processes, word choices, and tone. Examples include pairing the value of ‘Participation’ with a detailed contributor ladder, or matching the value of ‘Inclustivity’ with a warm and encouraging tone. If you have taken the time to find the values that match your project, this embodiment will likely seep through naturally.

##### 2. Never Enough Details

Whether this is one of your first interactions with the project, or you are a veteran contributor, everyone benefits from feeling supported by a well-thought-out set of processes and systems.

This quote from [The Open Source Way](https://www.theopensourceway.org/the_open_source_way-guidebook-2.0.html#_project_and_community_governance) really highlighted this for me:

“In 2018, the Kubernetes project added a set of detailed, comprehensive Role Handbooks for their Release Team. These handbooks outlined information related to the Release Team role, including qualifications necessary for joining the team, duties members of the team perform, and details on the team’s decision-making processes. As a result, the Release Team became the most popular point of entry for project contributions; new participants knew exactly what to expect. Other teams within Kubernetes followed suit—and experienced a doubling or even tripling of the number of new contributors.”

Here’s a question to think about: what are some pain points, anxieties, or uncertainties that are stopping someone from contributing to our project? How many of these could be lessened through proper communication and supportive language?

##### 3. A Living Document, reflective of Current Practices

While this one really can’t be determined by an outsider, it is important that everything in your projects’ governance is reflective of actual practices. This obviously includes not having outdated or incorrect information within the docs themselves, but also includes making any implicit practices as explicit as possible. Having unclear governance docs can be a sign that there are some processes you take for granted that could be helpful to write down for someone new to the project.

Not regularly updating or fleshing out your current practices can make the process of on-boarding new contributors to the project more clunky than it needs to be. Assuming you are actively seeking new contributors and are wanting to grow the talent and responsibilities of existing contributors, seeking to be as transparent as possible will help you achieve this goal. This can help set expectations and give contributors clear guidelines and goalposts to work towards, and feel supported in their process of growing as a leader within your project.

---

I want to thank my mentor Bill Mulligan and the Cilium project for having me on as a mentee, as well as the LFX Mentorship program as a whole for providing this great opportunity to become immersed in open source. The PRs I was able to work on and merge will hopefully allow the project to run more efficiently through enhanced knowledge sharing and systems.

I am happy to share that I have contributed in the following ways:

- Moved governance docs into the community repo
 - cilium/cilium - [#31692](https://github.com/cilium/cilium/pull/31692)
 - cilium/community - [#93](https://github.com/cilium/community/pull/93)
 - cilium/.github - [#9](https://github.com/cilium/.github/pull/9)

- Documented admins for the various tools that Cilium uses
 - cilium/community - [#97](https://github.com/cilium/community/pull/97)

- Added clarification and editorial changes to Cilium’s governance and contributor ladder docs
 - cilium/community - [#100](https://github.com/cilium/community/pull/100), [#114](https://github.com/cilium/community/pull/114), [#122](https://github.com/cilium/community/pull/122)

- Added repository lifecycle docs, including a list of all repositories within the Cilium organization and their scope, issue form templates, and a sub-project voting system
 - cilium/community - [#105](https://github.com/cilium/community/pull/105)

- Proposed changes to streamline the Cilium Feature Proposal (CFP) approval process
 - cilium/design-cfps - [#37](https://github.com/cilium/design-cfps/pull/37)
 - cilium/cilium - [#32477](https://github.com/cilium/cilium/pull/32477)

I have enjoyed my experience and have learned a lot, and would encourage anyone thinking about participating to apply. There are a wide range of skillsets needed in open source projects, so don’t count yourself out on the basis of not having what you have deemed are the _right_ knowledge base or _right_ experience. If you are interested in it and ready to learn, you can likely carve out a space for yourself. Good luck, and I hope to see you participating in a future term!

_Written by Katie Struthers, [@katiestruthers](https://github.com/katiestruthers)_

#### Improving the security of Cilium Mutual Authentication

URL: https://cilium.io/blog/2024/03/20/improving-mutual-auth-security
Date: 2024-03-20
Categories: Technology

##### Improving the security of Cilium's Mutual Authentication

Traditionally, service meshes have achieved mutual authentication and encryption by setting up an mTLS connection between workload proxies running in sidecar containers within the pods at either end of the connection. As you might recall if you've been following this space, last year Cilium introduced an innovative approach to providing [mutual authentication between workloads](https://docs.cilium.io/en/stable/network/servicemesh/mutual-authentication/mutual-authentication/). It splits mutual authentication and encryption (the two fundamentals of mutual TLS) into two separate channels. This removes the overhead of per-pod proxy sidecars, and opens up the possibility of authenticating traffic that uses protocols other than TCP.

From the original CFP:

> Firstly, a control plane connection between cilium-agent instances on each node provides authentication of connections between pods on the nodes.
> Secondly, the existing Cilium encryption support using WireGuard or IPsec provides an encrypted dataplane for the connections.

We've seen a very positive reaction to, and adoption of, this approach, but in certain modes and under certain circumstances, the implementation was vulnerable to manipulation of the cache of IP identities. The Cilium community is indebted to the passion for security shown by the folks from Solo.io, whose [analysis](https://www.solo.io/blog/could-network-cache-based-identity-be-mistaken/) and [persistent](https://thenewstack.io/how-ciliums-mutual-authentication-can-compromise-security/) [feedback](https://www.youtube.com/watch?v=bKpp30Q4_nY) have helped the community fully understand the issue and all its implications. Thankfully, we're now at a point where this IP cache manipulation is no longer possible in the vast majority of deployments using ingress network policies, and can be mitigated in many of the remaining corner cases.

In this post we'll discuss how the vulnerability was possible, and how it's now addressed by encapsulation for nearly all use cases. We'll also discuss some ideas for tackling the remaining corner cases - and we'd welcome your feedback on [the CFP](https://github.com/cilium/design-cfps/pull/27) that covers this.

This is an in-depth post, so to summarize of the actions the Cilium community is taking to address this issue:

- We're updating the documentation to strongly recommend using encryption (currently Wireguard preferred) or encapsulation when using mutual authentication.
- As you'll see from later in this post, using ingress mutual authentication policies in combination with encapsulation or encryption makes IP cache manipulation impossible, so we'll recommend against only using egress mutual auth policies.
- For users who can't enable encryption or encapsulation, we're also proposing a connection-based mode for mutual authentication, and we'd love your feedback on that proposal.

##### General architecture

The following is a diagram of the general architecture of Ciliums' mutual authentication solution, that uses an auth table to cache the results of mutual authentication handshakes.

The most important thing to call out for this post is in the eBPF section, where the state of authentication is stored in the Auth Table for use by the eBPF code.

This table is used by eBPF code but maintained by both the eBPF code paths and the userspace code paths.

The flow for a packet is that, when the packet passes through Cilium's policy engine, if it matches a NetworkPolicy that requests authentication, and no authentication is stored in the table, then a TLS connection between agents using delegated SPIFFE identities is triggered (this is the `TLS Session Manager` box in the diagram).

Once this handshake succeeds, the userspace cilium-agent adds the details into the Auth Table, and any future packets will be allowed to proceed.

##### Cilium Identities

To understand the details of how the Auth table stores authentication information, it's important to understand how Cilium's Policy engine assigns identity to network traffic.

Each Pod has a CiliumEndpoint, but also each set of labels on a Pod is associated with a CiliumIdentity, which marries the pods together into a workload. This identity maps a set of labels to a integer identifier.

When Cilium makes Policy decisions, it does this by assigning an identity to the source and destination IP addresses, then using the identity to make policy decisions.

This identity can be determined in a couple of ways for each packet:

- either included in every packet when there is encapsulation (like VXLAN) or encryption (like Wireguard), which is preferred, or
- imputed using a cache of IP (the "ipcache") to identity mappings that's held on each agent, and kept up to date by watching the Kubernetes apiserver.

Many Cilium installs use either VXLAN (it's the default networking mode), or Wireguard, which ensure that the Cilium identity is present in every packet.

However, for many different reasons, other Cilium installs need not to use any encapsulation. This is called Direct Routing or Native Routing in Cilium's documentation, and when it's involved with no Wireguard encryption, the ipcache is used as the main method of imputing identity to network traffic. This will be important later.

##### Auth table and its implications

The Auth table stores the following details:

- The source identity
- The destination identity
- The destination node (so that Cilium can ensure that mutual auth to separate identities on the same node are handled correctly)
- An expiry time (when the auth table entry will be considered expired and a new authentication triggered. This is set to the closest expiry time for any of the certificates involved in a particular authentication handshake).

Doing things in this way allows some big advantages:

- because the enforcement is done per-identity, which is associated with IP addresses rather than higher-layer protocols, _all IP traffic_ can be protected with mutual auth, including UDP traffic, which is much harder to do with traditional mutual auth methods.
- Similarly, because the mutual auth is performed per identity, only one TLS handshake is required for each identity-identity-node tuple. So a pod can open as many TCP connections as it likes without having to pay a TLS handshake overhead for each one. This can result in a significant amount of time saved, and speed up connections by a large amount for workloads that don't make good use of shared connection mechanisms like HTTP/2.
- Because the auth mechanism is handled out-of-band of the actual traffic, then key rotations can be performed by the userspace agent before the old keypair expires, and update the auth table with the new expiry time. This means that _key rotation is completely invisible_, connections are _not_ interrupted by the key rotation process.
- Lastly, if users wish to, they can use mutual auth in the absence of encryption, and avoid paying the latency cost of encrypting every packet. For users who do not wish to pay the compute and latency cost of encryption, this tradeoff may be worthwhile.

##### Issue with the current approach

All of the above advantages are why the mutual authentication feature was built this way to begin with, but it's become apparent that there are a set of circumstances where the current design will fail when attacked.

Those circumstances are:

- The cluster must be using native routing or some other method that does not encapsulate the packets (where the encapsulation carries the CiliumIdentity, as in VXLAN and Wireguard)
- _Some_ nodes in the cluster must lose connection to the apiserver somehow. In one demonstration, this was performed using iptables rules on a host, which would require elevated privileges, but it's also possible, if unlikely, that this situation would arise in some other network failure scenarios.
- An attacker must be able to launch new pods on one of the nodes while the above two conditions hold.

Then, the following steps must occur:

- Attacker must delete a pod on a connected node (the source node)
- Attacker must schedule a pod on that connected node that ends up with the same IP address as an already existing Pod, that had an already authenticated mutual auth session established with a disconnected node (the destination node)
- The network policy used to enforce traffic must be an _ingress_ policy, which is enforced on the _destination_ node. For _egress_ policies, the attack must be reversed, with the _destination_ node remaining connected and the _source_ node being disconnected. In this case, scheduling another Pod to reuse the same IP address is more complex, since Kubernetes can't be used to achieve it.

If these steps can occur, then mutual authentication can be effectively bypassed, which is bad.

In more detail the steps are:

- Attacker has to identify a source / destination pod pair where the destination pod is protected by an ingress policy that enables mutual auth.
- There must already have been a connection between the source and destination pod pair to establish a mutual auth session in the mutual auth BPF table.
- Attacker has to cut apiserver access of the destination node or otherwise delay any updates from the apiserver to that node for a prolonged time while the source node remains connected to the apiserver. In most cases, the entire apiserver not being available is not enough as no pods could be deleted or scheduled (next step).
- Attacker has to be able to delete the source pod so it releases its IP. Otherwise attacker has to wait for the pods to scale down by the autoscaler or some other reason and has to identify IPs which have been released. All of this has to happen while the destination node remains detached from the apiserver, or else the destination node will see pod deletion updates and update its cache.
- Attacker must be able to schedule a pod that will end up with the same IP address as the source pod.
- Attacker has to be able to schedule a pod and be lucky enough for it to get assigned the same IP address as the original source pod. The chances of this depends on factors like the size of the IP address pool, and how many other pods are being scheduled at this time.
- No egress policy requiring mutual authentication for the scheduled source pod may exist (as this would trigger a new handshake).

However, if _all_ of those steps happen, then the cache manipulation involved will render it possible to spoof the mutual authentication process and allow unauthenticated pods to send traffic that the system will treat as though it's authenticated.

This condition will last only until one of the following occurs:

- The destination node regains connection to the apiserver and processes pod deletion events. This will update the cache to be in the correct state, and the mutual auth table will be updated to remove entries associated with identities with no running endpoints on a node.
- One of the certificates involved in the handshake expires. When this occurs, another mutual auth handshake will occur, which also checks the pods running on the node. The default lifetime for a certificate is 30 minutes (although this is configurable), so that setting is the maximum window of exposure to this attack.

Let's walk through some diagrams of this attack.

Here we have two nodes of a Cilium cluster, meeting the above requirements.

The ipcaches are synced between the two nodes, so all the identities are correct.

Both nodes `A` and `B` agree on the IP addresses associated with the identites `foo` and `bar`.

Once the exploit has been performed, the nodes look more like this:

On Node `B`, the connection to the apiserver has been interrupted in some way, so the Cilium agent is no longer receiving updates to Cilium Endpoints, meaning that the ipcache will become out of sync.

On node `A`, enough pods have been scheduled in a new identity (the `baz` identity), to have the IP address that was previously associated with `foo` be now associated with `baz`.

However, on Node `B`, because the ipcache is out of sync, the ingress policy will allow traffic from the pod that's actually a `baz` pod, because it has incorrect information about the IP to identity mapping. This is the reason why the exploit only works in direct routing mode - because in encapsulated modes, the identity is passed in any packet and used directly, rather than relying on the ipcache.

The CFP lays out a proposal to mitigate this risk in two ways:

- Require the use of either encryption (recommended because it prevents man-in-the-middle attacks) or encapsulation (which prevents cache manipulation but not MITM). In both of these cases, the Cilium identity is included in every packet, and so the ipcache is not used to impute identity to packets.
- Add an additional mutual authentication mode that will use the connection tracking ("conntrack") table in the conntrack BPF map to store information about which _connections_ need authentication, instead of which _identity-identity-node pairings_ need authentication.

The first idea makes cache manipulation irrelevant, because the inbound packets include the security identity in a header.

##### Requiring encryption or encapsulation resolves cache manipulation

This assertion depends on two things that may not be clear:

- Wireguard encryption now (as of Cilium 1.15) also encapsulates packets into VXLAN as they transit the tunnel, adding a Cilium identity into the header (as in unencrypted VXLAN)
- When the datapath can determine the remote security identity from packets directly, the ipcache is _not_ used.

As part of actions taken to mitigate this attack, we added VXLAN encapsulation to Wireguard during development of Cilium 1.15.

But Cilium's eBPF code has _always_ preferenced identity information it decodes from encapsulation, and in the case that it's present, the ipcache is not used for policy decisions.

This is why the original attack required the use of direct routing - when encapsulation is in use, the attack never worked.

The more recent change to ensure that Wireguard encryption also includes encapsulation just means that when you're ensuring your traffic's wire security, you're also making the identity not spoofable by cache manipulation when using ingress mutual authentication policies.

When egress policies are used to require mutual authentication, the IP cache at the source node is used, so this would potentially still be subject to IP cache manipulation. For this reason, we will recommend going forward that ingress policies should be preferred, or used in parallel with egress policies.

###### Diagrams of the new workflow

After adding this change, let's look at diagrams of before and after the exploit is attempted:

Before, things work as anticipated.

However, it's after that there is a difference:

Even though the ipcache entries don't match on both nodes, the identity being included in the packets means that the datapath relies on that instead of the ipcache, so any cache manipulation is irrelevant.

##### Connection-based mutual authentication method

In the existing identity-based mode, mutual authentication handshakes are only initiated when there is a change in either endpoint state (that is, a new node to node connection from a new pod), or when a certificate is rotated. So, in the event that an attacker _did_ manage to compromise mutual authentication mechanisms somehow, the window is, at maximum, the certificate rotation period, which defaults to 30 minutes.

Adding a connection-based mutual authentication mode significantly reduces the timescale that an attack is exploitable for - since the mutual auth handshake is performed on each new connection, and connections _generally_ last much less that 30 minutes.

It also makes performing the exploit harder, since the full 5-tuple (source IP, source port, destination IP, destination port, protocol) must match, rather than just source and destination IP, _and_, for TCP connections, must also migrate the actual connection from one application to another. For both TCP and UDP, the attack is still possible, but orders of magnitude harder.

Additionally, the eBPF conntrack table already handles new three-way handshakes on existing entries by clearing and resetting connection properties, so as long as this code also handles clearing mutual auth related config, then it will make it even more difficult to reuse an existing connecton entry.

The connection-based mutual authentication method will:

- Be selectable as an entirely new authentication method (called `connection-spiffe` or similar)
- Use the conntrack table to keep track of connections
- Add two bits of marking into conntrack table entries, "Auth required", and "Auth completed"
- Use the existing TLS handshake code, but call it on a different event (new connection instead of new identity-identity-node tuple).

The existing conntrack table performs standard connection tracking functions, tracking connections based on the 5-tuple of source IP, source port, destination IP, destination port, and protocol. TCP is handled using its connection mechanisms, and UDP is handled using the same time-based pseudo connection that is standard across most networking gear.

What this change does is add two additional bits to the conntrack table:

- Auth required: This connection requires mutual authentication before proceeding. The datapath will drop all packets until the mutual auth has been completed.
- Auth completed: This connection has had a mutual authentication handshake completed. The userspace agent will perform the handshake and update the connection table entry once it is finished.

The updated mutual auth architecture diagram from above then looks like this:

The steps are:

1. Packet arrives and the policy engine determines a policy requires it to have mutual auth
2. Kernel checks if there is a conntrack entry already, and creates one if not, with the Datapath code setting the Auth Required bit to true if the new entry was created
3. If the conntrack entry has "Auth Required" but no "Auth completed", drop packet and send a signal to the userspace asking it to complete the auth handshake
4. Cilium Agent performs a mutual auth handshake between the source and destination nodes, sending the checksum as part of the handshake. When this succeeds, it updates the conntrack entry to have "Auth completed".
5. The next packet sent after the conntrack table update will be passed.

###### Advantages

- Every new connection requires a mutual auth handshake, so this has more familiar security properties to traditional mutual TLS
- Connection based nature makes it much harder to spoof traffic so that it matches an already-authenticated connection, whether by manipulating the cache or some other spoofing method.
- Can handle both TCP and UDP traffic (the latter using the standard pseudoconnection model)

###### Disadvantages

- Compared to the accelerated mode, connection-based mode introduces a per-connection handshake overhead
- The initial implementation will need to drop packets until the mutual handshake is complete, rather than holding packets

We would love feedback from users who have a need for mutual authentication without enabling encryption with Wireguard, on whether this per-connection overhead might outweigh the latency cost of avoiding encryption.

##### Conclusion

We believe that these changes significantly improve the security of Cilium's Mutual Auth solution and are a significant step on the path to bringing the solution from beta to stable.

Please watch the [CFP PR](https://github.com/cilium/design-cfps/pull/27) and the [tracking issue on Github](https://github.com/cilium/cilium/issues/28986) for more updates.

#### Cilium User Story: Scaling and Enhancing Networking in a Managed Kubernetes Service with Cilium

URL: https://cilium.io/blog/2024/03/07/cloud-provider-user-story
Date: 2024-03-07
Categories: Community

_March 7th, 2024_

_Author: Shedrack Akintayo, Isovalent_

_This User Story comes from a company in the cloud industry_

Cloud providers give customers a simple way to access and use a wide range of cloud services including analytics, storage, and networking. These solutions in turn need to keep up with customer demands while still being manageable for the cloud provider themselves. This Cilium user story tells the story of why one cloud provider turned to Cilium for the networking layer in their managed Kubernetes offering.

###### Challenge

They originally built the networking layer on their own iptables-based Container Network Interface(CNI). However, as customers started growing their clusters, they began facing scalability challenges with their iptables based rules in kube-proxy. To handle the increasing number of services and backends and network policies, they needed a solution that would help them to grow and meet customer demands.

###### Solution

After evaluating several options, the cloud provider decided to take a collaborative approach with the community. They chose to integrate Cilium as a CNI into their managed Kubernetes offering because it addressed their scalability challenges, was feature-rich, and backed by an active and growing community

This integration allowed them to pivot their networking dataplane from iptables to eBPF, making it faster and more scalable. Cilium also enabled them to add additional features for their customers like advanced network policy and a kube-proxy replacement. Their customers were looking for a cloud-agnostic solution that would work wherever their workloads needed to connect and found that with Cilium.

###### Impact

By switching the dataplane of their managed Kubernetes service from iptables to Cilium powered by eBPF, their customers have more scalability, performance, and features for their Kubernetes networking layer. The Cilium community has also benefited from this cloud provider's contributions back to the project. These contributions back to the open source community have also helped build trust and confidence with their customers.

##### Migrating to Cilium for Performance, Scalability, and More Features

When this cloud provider created their managed Kubernetes service, they needed a Container Network Interface (CNI) to provide container networking and built a proprietary one based on iptables.

However, as the number of backends and services being deployed on customer clusters and the number and size of clusters increased, they began to encounter several challenges. They experienced scalability and performance issues stemming from iptables and an increase in the memory consumption when there were more than 1,000 network policies.

Motivated by these limitations, they started searching for a solution. Seeing the success of the Kubernetes community, they wanted to find a community-driven solution for their networking layer. They were looking for a solution that could scale customer clusters and network policies, offer security routing, and replace kube-proxy. The search led them to [eBPF-powered solutions](https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/) (rather than iptables) and Cilium stood out.

This cloud provider recognized Cilium's potential, noting its scalability, rich feature set, and the strong community surrounding it. The fact that Cilium is a part of the Cloud Native Computing Foundation (CNCF) and highly respected in the open source community further influenced their decision.

_“We chose Cilium because it is the best-in-class eBPF solution. In the open source community, Cilium is the most respected and because it was donated to the CNCF, it was easier for us to move towards integrating it.”_ - Principal Engineering Manager

They also worked with Isovalent to migrate their managed Kubernetes service platform to Cilium as the CNI. Once they had Cilium in place, they were able to significantly enhance the [speed and scalability](https://azure.microsoft.com/en-us/blog/azure-cni-with-cilium-most-scalable-and-performant-container-networking-in-the-cloud/) of their data plane. The shift also allowed then to add more network policies and set them up to add additional capability in the future.

_“The whole switch to Cilium was smooth. Cilium’s documentation and GitHub is on point and really good with self-troubleshooting and integrations.”_ - Senior Software Engineer

##### Preparing Customers and Platforms For Future Cloud Native Challenges

Integrating Cilium has been a huge success for this cloud provider. They have been able to meet the demands of their customers and scale their networking. By integrating Cilium, they not only addressed the immediate customer challenges but also positioned themselves to cater to future customer requirements.

_“Our customers need a scalable, performant, and rich container networking solution to meet the demands of their rapidly expanding cloud native footprint. In collaboration with the CNCF community, we integrated the open source Cilium into our CNI to meet these needs._

_The response and adoption have been very positive amongst our customers. We are looking forward to continued collaboration with the community."_ - Corporate Vice President and Technical Fellow

In the future, they aim to harness the power of eBPF to offer advanced capabilities to their customers, focusing on enhanced security and observability, like service mesh, L7 and FQDN-based network policies.

_“Cilium is the next generation for container networking. We can build so many things with it because of the potential it brings with eBPF._

_Our customers get all the benefits from a scalability and performance perspective which is key for a platform to provide as they are setting up their cloud native applications for the next set of challenges.”_ - Principal Group Software Engineering Manager

#### Interview: Hubble Integration Added to DigitalOcean Kubernetes

URL: https://cilium.io/blog/2024/02/29/cilium-on-digital-ocean
Date: 2024-02-29
Categories: Community

In late 2018, DigitalOcean Kubernetes made a strategic move, migrating its infrastructure to Cilium. Fast forward to today, and over 30,000 clusters in DigitalOcean cloud are powered by Cilium, with thousands leveraging Cilium Network Policies for micro-segmentation and enhanced security.

But what was missing from this already robust setup? Support for Hubble. That's about to change. DigitalOcean recently announced a significant update to its Kubernetes offering, and we had the opportunity to catch up with them to learn more about it.

##### Interview with DigitalOcean: Elevating Kubernetes Observability with Hubble

_**Q**: DigitalOcean Kubernetes has been a game-changer since its launch in 2018. Can you tell us more about its journey and what sets it apart in the Kubernetes ecosystem?_

**A**: DigitalOcean Kubernetes, or DOKS, has indeed made waves since its inception. Designed with developers, startups, and digital businesses in mind, DOKS stands out for its user-friendly nature, cost-effectiveness, and seamless management. It strikes the perfect balance between flexibility and affordability, making it an ideal choice for businesses of all sizes.

_**Q**: Cilium has been instrumental in powering the data plane for DOKS. What led to this decision, and how has it been received by your customers?_

**A**: Cilium, with its advanced eBPF technology and now the de-facto cloud native networking solution, has been a cornerstone of our Kubernetes infrastructure. Our customers have loved its robust networking and security features, which are key in the ever-evolving landscape of Kubernetes environments. The decision to integrate Cilium was driven by our commitment to providing a reliable and secure platform for our users, and it was incredibly well-received.

_**Q**: Operational visibility is crucial in Kubernetes environments. How does the integration of Hubble enhance this for DigitalOcean customers?_

**A**: Since cloud native environments can be so dynamic, operational visibility has always been a top priority for us. By enabling Hubble by default for all clusters starting January 2024, we're empowering our customers to gain deeper insights and streamline their operations effortlessly. As shown in multiple [case studies](https://www.cncf.io/case-studies/?_sft_lf-project=cilium) from the CNCF, Hubble is a real hidden gem for both developers and operations teams for the network observability it provides to troubleshoot and debug often difficult to diagnose network issues. The simplicity of installation and the assurance that all observability data stays within their DOKS cluster have been key highlights of this integration.

_**Q**: Can you walk us through how users can leverage Hubble within their DOKS clusters?_

**A**: Absolutely. The integration is designed with ease of use in mind. Once the Cilium and Hubble CLIs are installed and the kubeconfig is configured for the DOKS cluster, users can seamlessly access both the Hubble UI and CLI without any additional configuration. This emphasis on simplicity and efficiency aligns perfectly with our commitment to providing a frictionless experience for our users. You can also refer to this quick [video walkthrough](https://www.youtube.com/watch?v=xUE6hKtqhrM).

_**Q**: Where can users find more information about utilizing Cilium and Hubble with their DOKS clusters?_

**A**: For detailed instructions and additional information, we encourage users to visit our [documentation](https://docs.digitalocean.com/products/kubernetes/how-to/use-cilium-hubble/). Additionally, they can stay updated on platform enhancements by following our [announcement blog](https://www.digitalocean.com/blog/cillium-hubble-on-digitalocean-kubernetes). We're committed to providing comprehensive support and resources to ensure our users make the most out of their DOKS experience.

_**Q**: Lastly, for those interested in learning more about DigitalOcean's journey with Cilium, where can they find more information?_

**A**: We delivered a session at eBPF Summit, where we shared insights into our journey using Cilium to offer a Kubernetes platform to our customers. The session provides valuable insights into our integration efforts and the benefits it brings to our users. You can watch the session recording [here](https://www.youtube.com/watch?v=xez34h7EY3A).

##### Conclusion

DigitalOcean's integration of Hubble marks a significant advancement not only for its Kubernetes offering but also for the broader Cilium project. By prioritizing simplicity, efficiency, and security, DigitalOcean sets a high standard for managed Kubernetes services, showcasing the power and versatility of Cilium's technology. The addition of Hubble underscores the value of the technology in driving innovation and solving complex challenges in modern cloud environments.

Moreover, DigitalOcean's embrace of Hubble serves as a beacon of encouragement for other cloud providers. By demonstrating the benefits and practicality of this integration, DigitalOcean paves the way for wider adoption of Hubble within the industry. With any luck, this move will inspire other providers to follow suit, fostering a more interconnected, observable, and secure cloud native ecosystem for all users.

#### How Software Networking Is Taking Over The World

URL: https://cilium.io/blog/2024/02/12/how-software-networking-is-taking-over-the-world
Date: 2024-02-12
Categories: Community

For decades we’ve been taught by hardware manufacturers that networking and packet manipulation requires dedicated hardware based on expensive, specialized silicon. However, that’s no longer the case. Kernel-level technologies such as [eBPF](https://ebpf.io), combined with huge improvements in server CPU throughput and multi-core/threading, are today allowing us to run complex, even bespoke networking capabilities on commodity hardware. Let’s look into how this has evolved from software defined networking to today’s cloud native deployments.

##### In The Beginning, There Was The Hardware

We don’t have to go too far through the history books to head back to a time when the thought of a consumer CPU being part of the networking data path would send shivers down the spines of both the networking team and applications teams. Industries were created, enough marketing material to fill several libraries were developed and the computer industry accepted that dedicated hardware was required to manage the application traffic between servers and the outside world.

Network hardware quickly evolved from just ensuring traffic would get from A to B: manufacturers added the capabilities to segment traffic, make quality of service guarantees, and ensure that they had control of who/what hardware was on the network with access control lists. Originally just focussing on the network tenants, it didn’t take long for the hardware companies to start to move their technologies into the actual traffic itself. Routers would inspect the destination for traffic and forward it to other locations if needed, Firewalls would inspect incoming traffic based on destination and port and would compare this to sets of rules that would either permit or deny the incoming traffic. Then finally the technologies progressed from the traffic behavior to the application behavior, which involves looking at (and understanding) the application traffic (also known as Layer 7) and making decisions based upon incoming application requests. The most common of these were web application hardware that understands the “request” of incoming web traffic (such as GET /users or POST /login) and performs traffic decisions such as `block` or `forward` to another server through a set of rules.

##### The Problem With Software

There are many reasons why packet processing was originally moved to dedicated networking devices, however, the main and hopefully obvious one is that local (on your server/machine/laptop) packet processing performance would be woeful compared to silicon created solely for that purpose. The speeds of networking interconnects improved at a rapid pace (10mbit -> 100mbit -> 1Gbit -> 10Gbit -> 100 Gbit etc..), and each speed increase would suddenly come with it a large bump up in the amount of data/packets that a CPU would have to process locally. These speeds are usually referred to as “wire-speed” (due to the speed of traffic on the physical wires), most hardware vendors would aim to ensure that their devices would be capable of processing traffic at “wire-speed” to ensure that the processing work would not incur any bottleneck on the network.

People trying to use commercial hardware/software to replicate the behavior of dedicated silicon would often be met with disappointment. The Journal of Universal Computer Science has a [paper](https://www.jucs.org/jucs_16_21/impact_of_cpu_bound/jucs_16_21_3299_3313_salah.pdf) detailing experiments of replicating high traffic rates from 2010. The journal goes on to state that on a 1 Gigabit connection, an expectation of 700 Kpps (thousand packets per second) is the expected rate. However, in testing traffic forwarding on commercial hardware the resulting rates were around 350 Kpps and 75 Kpps for both Linux and Windows respectively. Also when reaching these rates the CPU availability for running applications was in the single digits, effectively meaning that we can process the packets but the applications that are receiving the data may well be unresponsive. It’s worth noting that the above experiment was also purely UDP forwarding of packets, in the event we would want to inspect the application data itself and make decisions based upon that one would incur even larger performance penalties.

##### Software Catches Up

In the years since that paper was written, there has been a huge evolution in commercial hardware. CPUs have grown horizontally instead of vertically (single CPU core speed growth has all but stopped, and the focus of chip manufacturers has been multiple packaged cores).

- 2015: Intel Core i7-6700 3.4Ghz (4 cores, 4 threads)
- 2017: Intel Core i7-8700 3.2Ghz (6 cores, 12 threads)
- 2019: Intel Core i7-9700 3.0Ghz (8 cores, 8 threads)
- 2021: Intel Core i7-11700 2.5Ghz (8 cores, 16 threads)
- 2023: Intel Core i7-13700 2.1Ghz (16 cores, 24 threads)

\*_data taken from [eatyourbytes.com](https://www.eatyourbytes.com/launchyear/2023/) (and subsequent years)_

It would be a huge disservice to point out that the efficiency of single-core processing also hasn’t hugely increased, however the above is to illustrate that the architecture is changing. The network cards themselves sit on much faster connections to the CPU itself such as PCI-Express, and often come with offloading engines for certain repetitive tasks reducing the load on the CPU (such as a TCP offloading engine).

Additionally, due to the increase in processing capacity where the model typically runs multiple workloads on a single machine either through virtualization or containerisation the application architecture has undergone a huge shift. This has dramatically increased the amount of east-west traffic both on a single network and also within a server itself. This lead to a requirement for processing network packets within the server itself; having to push traffic out of the server to an external network device for network reasons would increase the latency for the application.

As mentioned above both the hardware **and** the evolution of the application architecture has pushed the requirement for more localized network processing. However, until relatively recently, this would still be a very inefficient model because processing packets in userland comes with huge performance penalties. For each packet, the kernel will first need to perform its workflow and then it will need to copy that packet to userland, where lower-priority application code can process it and finally return a response, leading to another data copy as the packet transitions into the kernel. For higher performance, we would need to write code that runs inside the kernel where it is largely free of the overhead of user programs etc. However pushing code upstream to the Linux kernel has to go through a rigorous process, and there is no guarantee that it would be accepted.

You could potentially manage a copy of a kernel with your network code in and accept a huge amount of technical debt moving forward. Kernel modules are also a potential option, but they still come with the overhead and technical debt of managing a continuing stream of new releases and patching for older kernels, not to mention the risk of bugs bringing down the whole server.
For commercial operating systems such as Windows, there are no clear routes to have code added to their kernel.

##### Enter eBPF

The demands of the applications, the improved hardware, and the limitations/restrictions of the operating system have been met with the innovation of eBPF. Simply put, eBPF technology allows developers to write code that will run in a sandbox environment inside the kernel. Suddenly, this technology allowed developers to write code that runs in a highly efficient and privileged environment and can interact with the network stack with minimal performance penalties.

eBPF allows functions to be attached to specific events within a running Kernel. For networking, we would attach our functions to events such as a network packet being received (_ingress_) or a packet being sent (_egress_). Our eBPF function can then act upon this packet, such as filter it (_firewall_) or change its destination (_load balancing_) etc.

Originally, eBPF allowed connectivity through to the Linux Kernel Packet Scheduler (aka TC/Traffic Control), where the packets that an eBPF program can see have already been processed by the kernel. However, additional functionality exposed by allowing eBPF programs to be attached to XDP (eXpress Data Path) events exposed two huge improvements:

- eBPF programs can interact with packets before the kernel would process them
- eBPF programs can be loaded by the NIC driver (if supported) to run on the NIC itself, effectively offloading various processing activities from both the Kernel and the server CPU itself

With the innovations from eBPF programs and hooks in the Kernel, we can now start to realize the high levels of performance of commercial hardware that before were only available through specific-purpose silicon. An example of this was discussed by the developers of Cilium at [KubeCon North America 2022](https://kccncna2022.sched.com/event/182DB/100gbits-clusters-with-cilium-building-tomorrows-networking-data-plane-daniel-borkmann-nikolay-aleksandrov-isovalent), where they talked about reaching 100Gbit/S with IPv6 clusters.

##### Taking eBPF to the next level with Cilium

The Cilium project was the first cloud native network project built exclusively to take advantage of the features and performance of eBPF. Built from the ground up it immediately provided various benefits both in simplicity and in performance around networking. However, as the project has grown and adopted additional features from eBPF it now also provides unique insights into the network traffic entering, within, and exiting your clusters (along with tying it to Kubernetes objects) with the Hubble project.

As Cilium has become the defacto CNI for the majority of Kubernetes clusters, the project has grown and adapted to provide additional functionality such as bridging numerous clusters or to endpoints that exist externally to a Kubernetes cluster, allowing traffic to flow between container workloads and workloads beyond. Finally, Cilium has also [replaced existing Kubernetes functionality, like kube-proxy](https://cilium.io/use-cases/kube-proxy/), to improve performance and scalability as clusters mature and grow.

#### Cilium Talks at KubeCon EU 2024

URL: https://cilium.io/blog/2024/02/02/cilium-at-kube-con-eu-2024
Date: 2024-02-02
Categories: Community

KubeCon + CloudNativeCon EU 2024 is nearly here, and it's packed with activities centered around Cilium. As the standard for high-performance networking, observability, and security in the Cloud Native ecosystem, Cilium is set to be a major point of discussion throughout the event, especially following the recent release of [Cilium 1.15](https://github.com/cilium/cilium/releases/tag/v1.15.0).

Starting with [Cilium + eBPF Day](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/co-located-events/cilium-ebpf-day/), the agenda features talks from end users like the New York Times, Roche, and Datadog, who will showcase the benefits they've achieved with the scalability, performance, and security capabilities of Cilium and eBPF. Cilium maintainers will also be diving into the architecture of the project and discussing where it is going next.

After Cilium + eBPF Day, there will be 11 other exciting talks about Cilium to catch when the main conference kicks off on Wednesday! The talks this year will cover high-scale networking, deep dives into security and observability with Tetragon, Hubble, and eBPF, and end-user stories from those implementing Cilium in highly regulated environments and on the edge. Similar to last year, there will be a Cilium booth in the Project Pavilion where you can meet the developers behind the project and have all your questions answered too.

Now, let's explore each of these talks in more detail.

##### Cilium + eBPF Day

[Connecting Cloud Native: Cilium + eBPF Day - Laurent Bernaille, Datadog & Bill Mulligan, Isovalent](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/co-located-events/cncf-hosted-co-located-schedule/)

Tuesday, March 19, 2024, 09:00 - 09:10 CET

The Opening Session for Cilium + eBPF Day EU 2024

[Deep Dive Into Cilium Resilient Architecture - Jussi Mäki & Martynas Pumputis, Isovalent](https://colocatedeventseu2024.sched.com/event/1ZJA8?iframe=no)

Tuesday, March 19, 2024, 09:15 - 09:40 CET

With 680 unique contributors, 470k lines of Go, and 35k of eBPF code, Cilium is on track to becoming one of the most significant OSS infrastructure projects. This comes with a big challenge - designing Cilium so that its learning curve for contributors is manageable while not sacrificing important infrastructure software properties such as stability.

In this talk, two core Cilium contributors will present Cilium's architecture, emphasizing resilience, integrity, modularity, and testability. This talk aims to help developers understand Cilium's design principles and architecture, which can facilitate their contributions.

[Lessons from Building Scalable Network Policy Enforcement with eBPF - Hemanth Malla, Datadog & Joe Stringer, Isovalent](https://colocatedeventseu2024.sched.com/event/1YFe0/lessons-from-building-scalable-network-policy-enforcement-with-ebpf-hemanth-malla-datadog-joe-stringer-isovalent?iframe=no)

Tuesday, March 19, 2024, 09:50 - 10:15 CET

eBPF has unlocked new levels of performance and scalability for container networking. Cilium has leveraged eBPF to implement a plethora of network policy features. Kubernetes scalability has been improving with every new release, and clusters with 5k+ nodes are increasingly common. Cilium’s policy framework needs to scale for hundreds of thousands of pods while dealing with complex scenarios like high pod churn environments. In this talk, Cilium maintainers will share lessons learnt from years of programming Kubernetes abstractions directly into the kernel space using eBPF. You’ll learn about how cilium efficiently intercepts traffic for enforcement both at L4 and L7, tricks used by cilium to minimize CPU overhead on each node, and some design decisions that have been instrumental in squeezing high-performance out of the kernel regardless of the number of pods. Finally, we’ll discuss strategies you can follow to improve the debuggability of eBPF-based networking datapaths.

[ Lightning Talk: Falco, Tracee and Tetragon: eBPF Runtime Observability and Security Tools Differences - Kyle Quest, Slim.AI](https://colocatedeventseu2024.sched.com/event/1YFee/cl-lightning-talk-falco-tracee-and-tetragon-ebpf-runtime-observability-and-security-tools-differences-kyle-quest-slimai?iframe=no)

Tuesday, March 19, 2024, 10:40 - 10:45 CET

Falco, Tracee and Tetragon are popular eBPF-based runtime observability and security tools. Which tool is better? How are they different in terms of use and in terms of their design? Which one should you choose? What are the best use cases for each tool? This talk will focus on the eBPF-related differences in those tools including the eBPF application design, eBPF libraries each tool uses, eBPF features leveraged by each tool, and types of eBPF hooks used. default tracepoint and kprobes, data exposed in the events generated by each tool. The talk will also give an overview of the trade-offs, gaps and benefits those tools have based on their differences. If you are a user trying to understand what eBPF runtime tool is best for you or if you are a developer trying to build your own eBPF runtime tool this talk will help you answer the top questions you might have.

[ Lightning Talk: Optimizing Cluster Workloads: Cilium Envoy on DPU - Shatakshi Mishra & Abed Mohammad Kamaluddin, Marvell Technology](https://colocatedeventseu2024.sched.com/event/1YFey/cl-lightning-talk-optimizing-cluster-workloads-cilium-envoy-on-dpu-shatakshi-mishra-abed-mohammad-kamaluddin-marvell-technology?iframe=no&w=100%&sidebar=yes&bg=no)

Tuesday, March 19, 2024, 10:50 - 10:55 CET

The Envoy Proxy plays a pivotal role in providing Layer 7 functionalities like Ingress, Gateway API, Network Policies, and Protocol Visibility as part of the Cilium agent. In Cilium version 1.14, deploying the Envoy proxy as a DaemonSet is supported, introducing intriguing deployment scenarios. One compelling option is transitioning these features to the DPUs. Modern SmartNICs and DPUs offer additional power-efficient compute capacity on servers with hardware accelerations tailored for specialized workloads. This study delves into an architecture for provisioning the Cilium Envoy Proxy to an OCTEON10 DPU, addressing challenges encountered in this process. The goal is to unlock the substantial compute potential of DPUs, optimizing resource utilization and enabling the host machine to handle an increased load of application workloads.

[Bee-Lieve in the Metadata: Pollenating Build Attestations on Kubernetes with Tetragon and EBPF - Tom Meadows, TestifySec](https://colocatedeventseu2024.sched.com/event/1YFfN/bee-lieve-in-the-metadata-pollenating-build-attestations-on-kubernetes-with-tetragon-and-ebpf-tom-meadows-testifysec?iframe=no)

Tuesday, March 19, 2024 11:05 - 11:30 CET

Like bees making honey, we developers forage the internet for code that we borrow and thread together to create software. It’s built, uploaded and shared, resulting in hundreds of thousands (if not millions) of software artefacts being built each day. For an unlucky few, attackers seek to infiltrate the build process like a hornet attacking a beehive. If they successfully attack a project with a large user base, the impact could be unprecedented, leading to the big bad ‘Game Over’ message rolling across the screen. In 2022, A Kubernetes-aware, Security Observability tool called Tetragon was released. This talk shows how the project serves as the key to the creation of a tool for securing build systems on Kubernetes. By leveraging Tetragon and eBPF, Attestagon aims to produce concise and cryptographically verifiable metadata. Come to this talk if you wish to see how any busy bee can inspect it to see whether a package was indeed sealed and delivered in the way the author intended.

[Meshing It up Securely: How Roche Manages Network Connectivity for 1000+ Edge Clusters - Hector Monsalve & Edgar Pardo, Roche](https://colocatedeventseu2024.sched.com/event/1YFg2/meshing-it-up-securely-how-roche-manages-network-connectivity-for-1000-edge-clusters-hector-monsalve-edgar-pardo-roche?iframe=no)

Tuesday, March 19, 2024, 11:40 - 12:05 CET

Roche, one of the largest global healthcare companies, has embarked on the challenging journey of building a modern, cloud native edge compute platform that helps to run applications on customer premises, like laboratories or hospitals. In this talk, we will discuss the challenges of highly protected environments and show how leveraging Cilium Service Mesh can bring “the firewall” closer to the workload. This not only allows for more fine-grained traffic control but also simplifies the operations and configuration of network policies using a GitOps approach, alongside the deployment of workloads. Going beyond this, we’ll also look at some of the more advanced features of Cilium - like sending cluster traffic that targets the cloud via a specific forward proxy before it leaves the customer premises.

[Revolutionizing Mesh Layers: Transitioning from Istio to Cilium at the New York Times - Ahmed Bebars & Pete Saia, The New York Times](https://colocatedeventseu2024.sched.com/event/1YFgU/revolutionizing-mesh-layers-transitioning-from-istio-to-cilium-at-the-new-york-times-ahmed-bebars-pete-saia-the-new-york-times?iframe=no&w=100%&sidebar=yes&bg=no)

Tuesday, March 19, 2024, 12:15 - 12:40 CET

Networking isn’t an easy task to achieve, and with scale, that can grow out of control quickly. We are always looking to improve the performance and ensure the traffic passes through the most efficient route but with more capabilities. Adding a service mesh to ensure better availability and many other features can be a bit tricky. We will discuss the intricacies of the shared platform mesh layer design. We will dive into our decision to explore the evolution of Cilium, which was initially used only as a Container Network Interface (CNI) that has now become a feature-rich, multi-region mesh layer. Ahmed and Pete will discuss the strategic decisions and implementation processes that allowed Cilium to replace many of Istio seamlessly. This has resulted in a scalable architecture and a significant performance and resource optimization boost. The audience will learn how the team strategically leveraged Cilium to unlock new possibilities in managing their platform mesh layer.

[Supercharge Kubernetes Networking with Cilium and IPv6 - Daneyon Hansen, Solo.io](https://colocatedeventseu2024.sched.com/event/1YFh4/supercharge-kubernetes-networking-with-cilium-and-ipv6-daneyon-hansen-soloio?iframe=yes&w=100%&sidebar=yes&bg=no)

Tuesday, March 19, 2024, 13:30 - 13:55 CET

Get ready for an in-depth journey into IPv6 networking with Cilium! In this session, we dive deep into the powerful IPv6 capabilities of Cilium to supercharge your Kubernetes applications. Join us to uncover the extensive range of IPv6 features that Cilium brings to Kubernetes environments. From ultra-high throughput powered by BIG TCP to transition mechanisms such as NAT46/64, you'll see firsthand how Cilium elevates the IPv6 networking game in Kubernetes. Whether you're a seasoned Kubernetes pro or just getting started, this session promises valuable insights into harnessing the full potential of Cilium for IPv6 workloads.

[Cilium Beyond Linux: Extending eBPF to Windows Infrastructure - Chandan Aggarwal & Alan Jowett, Microsoft](https://colocatedeventseu2024.sched.com/event/1YFiT/cilium-beyond-linux-extending-ebpf-to-windows-infrastructure-chandan-aggarwal-alan-jowett-microsoft?iframe=no&w=100%&sidebar=yes&bg=no)

Tuesday, March 19, 2024, 15:15 - 15:40 CET

Cilium eBPF offers significant performance improvements and advanced features exclusively for Linux users today. Would you believe it if I say “Cilium eBPF programs can run on Windows and reap the same benefits as Linux”. If you're intrigued by these advancements, this session promises valuable insights. The session will cover the current capabilities of Cilium Windows, encompassing L4 load balancers, network policy and observability and outline the roadmap for future milestones in this domain. There are a number of diverse challenges encountered while enabling these features and adapting cilium eBPF programs for the Windows environment. Exploring the array of challenges faced during this development, such as supporting the eBPF verifier, cilium dynamic generation of eBPF programs, integrating TC programs and eBPF maps, this session explores our detailed strategy for overcoming these challenges and further continuing development of eBPF on top of windows.

[Speedrun Through Splicing Sockets with Sockmap - Jakub Sitnicki, Cloudflare](https://colocatedeventseu2024.sched.com/event/1YFjR/speedrun-through-splicing-sockets-with-sockmap-jakub-sitnicki-cloudflare?iframe=no)

Tuesday, March 19, 2024, 16:25 - 16:50 CET

Network proxies have one thing in common - they push data from one side to the other. If the proxy doesn't access the data, then we can delegate the task of moving it between sockets to the operating system. In Linux, applications that can move data in batches between sockets using the splice() syscall. However, this isn't the only method available! Linux network stack also offers an alternative API to transfer packets between sockets, called sockmap, which leverages the eBPF technology. In fact, your systems might be already using sockmap. This is what powers the transparent L7 filtering and socket-level policy enforcement in Cilium. In this talk, we will go over the fundamentals of sockmap API, its evolution and features, as well as its internal design. We'll also explore the existing caveats and limitations. By the end, you should have a clear understanding of what it takes to use the sockmap API or, if you are a Cilium user, what role sockmap plays in your existing K8S cluster.

[ Lightning Talk: Taming Noisy Network Neighbours with Cilium and eBPF - Anton Ippolitov, Datadog](https://colocatedeventseu2024.sched.com/event/1YFjs/cl-lightning-talk-taming-noisy-network-neigbours-with-cilium-and-ebpf-anton-ippolitov-datadog?iframe=no&w=100%&sidebar=yes&bg=no)

Tuesday, March 19, 2024, 17:00 - 17:05 CET

Who likes noisy neighbours? Imagine enjoying a peaceful and quiet Kubernetes evening, when those miscreants show up and wreck the network latency and throughput on your shared nodes! In order to safeguard fair network resource distribution, Cilium has introduced the Bandwidth Manager feature: an egress bandwidth rate-limiting mechanism to keep those noisy neighbours in check. This short talk will give an overview of how the Bandwidth Manager does its job by leveraging the Fair Queue (FQ) packet scheduler and eBPF-based Earliest Departure Time (EDT) algorithm. Additionally, the attendees will learn about the performance implications of enabling this feature and potential pitfalls in Cloud environments.

[ Lightning Talk: Unveiling 5 Essential Insights from Cilium Implementation in a Multi-Tenant Kubernetes Environment - Victor Varza, Adobe Inc](https://colocatedeventseu2024.sched.com/event/1YFk9/cl-lightning-talk-unveiling-5-essential-insights-from-cilium-implementation-in-a-multi-tenant-kubernetes-environment-victor-varza-adobe-inc?iframe=no)

Tuesday, March 19, 2024, 17:10 - 17:15 CET

Meeting foundational requirements for a developer platform is crucial for key aspects such as security, scalability, reliability, and cost-effectiveness. At Adobe, Victor is actively engaged in constructing and operating a cross-cloud, multi-tenant Kubernetes-based platform. This initiative empowers product teams to develop and deploy services with heightened speed and efficiency. Beyond container orchestration, the network layer stands out as a paramount aspect of platform engineering. In this presentation, Victor will highlight five key insights on the the indispensable role of Cilium in establishing and implementing a minimum viable Kubernetes namespace in a multi-tenant cluster aligned with best practices.

[Defending the Future: Cilium + eBPF Day Closing - Bill Mulligan, Isovalent & Laurent Bernaille, Datadog](https://colocatedeventseu2024.sched.com/event/1YGSv/defending-the-future-cilium-ebpf-day-closing-bill-mulligan-isovalent-laurent-bernaille-datadog?iframe=no&w=100%&sidebar=yes&bg=no)

Tuesday, March 19, 2024, 17:20 - 17:30 CET

The Closing Session for Cilium + eBPF Day EU 2024

##### Observability Day

[What Is Going on Within My Network? A Subtle Introduction to Cilium Hubble - Shedrack Akintayo, Isovalent](https://colocatedeventseu2024.sched.com/event/1YFjg/what-is-going-on-within-my-network-a-subtle-introduction-to-cilium-hubble-shedrack-akintayo-isovalent?iframe=no&w=100%&sidebar=yes&bg=no)

Tuesday, March 19, 2024, 16:25 - 16:50 CET

Discover the power of Cilium Hubble in this session dedicated to cloud native observability. I'll introduce you to Hubbe, an open source tool that is transforming the way we observe and understand network traffic in cloud-native systems with the power of eBPF. In this session, we’ll deep dive into Hubble and how Hubble leverages eBPF technology for deep visibility at the kernel level. Key features like real-time monitoring, comprehensive network flow visibility, and security enforcement, crucial for today's cloud-native applications, will be highlighted. The talk includes a straightforward demo, showcasing Hubble's efficiency in understanding and reading your network traffic, and its intuitive interface. Finally, I'll walk you through the initial steps to get started with Cilium Hubble. Whether you're a developer, system administrator, or simply interested in cloud-native technologies, this talk will equip you with the knowledge to use Hubble for enhanced network observability.

##### KubeCon + CloudNativeCon

[A Cilium Introduction: Back to Bee-Sics - Nico Vibert & Dan Finneran, Isovalent](https://kccnceu2024.sched.com/event/1YeMX/a-cilium-introduction-back-to-bee-sics-nico-vibert-dan-finneran-isovalent?iframe=no&w=100%&sidebar=yes&bg=no)

Wednesday, March 20, 2024, 14:30 - 15:05 CET

Cilium is now the first CNCF Graduated project in the Cloud Native Network category and has now its first certification with the Cilium Certified Associate (CCA) program! But while networking is a fundamental aspect of Kubernetes, it remains an intimidating topic for many engineers. In this session, you will learn that Kubernetes networking and Cilium don't need to be daunting subjects after all! This session will provide an introduction to the core Kubernetes networking requirements and how Cilium addresses these use cases. Expect a demo-packed and inclusive session where newcomers and experienced network engineers alike will improve their cloud native networking knowledge and understand the benefits of using Cilium and eBPF for networking, observability and security. Finally, the session will also provide tips on some of the core topics and domains covered in the Cilium Certified Associate and enable participants to effectively prepare for the exam.

[Cilium: Connecting, Observing, and Securing Service Mesh and Beyond with EBPF - Liz Rice & Maartje Eyskens, Isovalent; Nico Meisenzahl, white duck; Vlad Ungureanu, Palantir Technologies](https://kccnceu2024.sched.com/event/1Yhfl/cilium-connecting-observing-and-securing-service-mesh-and-beyond-with-ebpf-liz-rice-maartje-eyskens-isovalent-nico-meisenzahl-white-duck-vlad-ungureanu-palantir-technologies?iframe=no&w=100%&sidebar=yes&bg=no)

Wednesday, March 20, 2024, 15:25 - 16:00 CET

Welcome to Cilium's maintainer track session where you'll get an update on how Cilium is expanding the frontiers of cloud native networking, observability, and security. After graduating and becoming the de-facto standard CNI for Kubernetes, what comes next? We'll start with a brief overview of each part of the project before giving an update on Cilium Service Mesh's mutual authentication. Next, we'll dive into how Cilium is expanding beyond Kubernetes with load balancing and multi-cloud networking and into runtime enforcement with Tetragon. In this session, you'll hear from Cilium contributors and users Isovalent, Palantir, and white duck.

[Dealing with eBPF’s Observability Data Deluge - Anna Kapuścińska, Isovalent](https://kccnceu2024.sched.com/event/1YeN5?iframe=no)

Wednesday, March 20, 2024, 15:25 - 16:00 CET

eBPF has created quite a buzz in observability because of its promise of no-instrumentation, low overhead, and complete observability. However, being able to observe and collect everything that is happening in a system can also create a deluge of data. Instead of sifting and winnowing to create a clear picture, eBPF can instead produce an even larger haystack to search through to find the critical observability needle. Drawing on practical production experience running observability systems for enterprises, this talk dives into strategies to deal with this torrent of data. It will go through examples from both the application monitoring and security observability arenas and demonstrate the commonalities between them and showcase real world benchmarking results. The audience will walk away with a better understanding of the pitfalls and perils of using eBPF for observability and how to overcome them.

[Simplifying Multi-Cluster and Multi-Cloud Deployments with Cilium - Liz Rice, Isovalent](https://kccnceu2024.sched.com/event/1YeQn/simplifying-multi-cluster-and-multi-cloud-deployments-with-cilium-liz-rice-isovalent?iframe=no&w=100%&sidebar=yes&bg=no)

Wednesday, March 20, 2024, 16:30 - 17:05 CET

Multi-cloud, multi-cluster Kubernetes deployments are used for high-availability, global distribution, to take advantage of different cloud vendor features, or to use both on-prem and public clouds. But sharing workloads in these distributed environments doesn’t have to be complicated! This talk uses live demos to introduce Cilium’s ClusterMesh capabilities, which make it easy to connect and secure workloads distributed across clouds and clusters. - Securely connecting multiple Kubernetes clusters - Distributing services across them - Load balancing and service affinity - Applying network policies across multiple clusters - Exposing distributed services to external traffic You’ll also learn about the requirements for the underlying internet connectivity between clusters, with an overview of IP address management considerations. You’ll need a basic familiarity with Kubernetes concepts like pods, services, nodes and clusters to get the most out of attending this talk.

[Brewing the Kubernetes Storm Center: Open Source Threat Intelligence for the Cloud Native Ecosystem - Constanze Roedig, Technische Universität Wien & James Callaghan, ControlPlane](https://kccnceu2024.sched.com/event/1YeOX?iframe=no)

Thursday, March 21, 2024, 11:55 - 12:30 CET

The process of threat modelling can seem an abstract art, especially regarding scoring and prioritisation. We show how a non-expert can practically validate threat modelling predictions and quantify the relative risk of different attack vectors. After a general introduction, we propose an extensible method that, based on a given threat model: a) generates a Kubernetes-based environment with embedded trip-wires, enabling the detection of real attacker paths without interference, b) exposes these simulated environments to the wild to observe quantitative threat intelligence in action, and c) informs cost-effective decisions for a defensive team. We discuss caveats, emphasise the critical role of automation in scalability across diverse threat models, and live showcase one quantified attack tree utilising Tetragon. To benefit the Kubernetes ecosystem, this accessible framework can be crowd-sourced into an open source threat intelligence capturing network for risk exposure quantification.

[Running PCI-DSS Certified Kubernetes Workloads in the Public Cloud - Stephen Hoekstra, Schuberg Philis](https://kccnceu2024.sched.com/event/1YeQJ?iframe=no)

Thursday, March 21 • 17:25 - 18:00

Compliance in the public cloud with Kubernetes can sound difficult and scary, but it doesn’t have to be. In this session, I will talk about our experiences running a PCI-DSS certified Kubernetes cluster in AWS, and share some lessons learned to help you achieve the same. This talk will start by covering high-level PCI-DSS requirements and discuss how we addressed them using several CNCF and open source projects: _ GitOps with ArgoCD _ Network policies and visibility with Cilium _ In-cluster image scanning and visualisation with Trivy and Grafana _ Open Policy Agent Gatekeeper or Kyverno for Policy as Code \* Cilium Tetragon for runtime security visibility By the end of the session, you should walk away with the knowledge of what is needed to run your own PCI-DSS workloads on Kubernetes in the public cloud, with tangible examples and best practice recommendations.

[No 'Soup' for You! Enforcing Network Policies for Host Processes via eBPF - Vinay Kulkarni, eBay](https://kccnceu2024.sched.com/event/1YeNT?iframe=no)

Friday, March 22, 2024, 11:00 - 11:35 CET

Current Kubernetes networking solutions provide basic security for pod network traffic using layers 3 and 4 CIDR-based or identity-based network policies. However, there is no mechanism to assign network identities to native processes running on hosts (e.g., kubelet) or processes in pods that use the host network. Securing host processes has traditionally been done using layer 7 auth, which comes with its overhead costs and scale challenges. In this talk, Vinay presents an innovative, industry-first approach that leverages eBPF to efficiently identify, in the kernel at the network layer, traffic from native host processes and pods using the host network. This takes network micro-segmentation to a new level. He will explain how host process identities are transmitted on a per-packet basis, and illustrate efficient network policy enforcement for such traffic. He will discuss how this approach offers significant scalability advantages, and conclude with a demo showcasing the proposed solution.

[At the Intersection of Cilium CNI and Service Mesh - Who Has the Right of Way? - Christine Kim, Isovalent](https://kccnceu2024.sched.com/event/1YeR5/at-the-intersection-of-cilium-cni-and-service-mesh-who-has-the-right-of-way-christine-kim-isovalent?iframe=no&w=100%&sidebar=yes&bg=no)

Friday, March 22, 2024, 11:00 - 11:35 CET

Getting started with service mesh can be overwhelming. You might get nervous about how many tools you’ll have to keep up to date. With Cilium, you may have heard of it being a strong CNI (container network interface), but did you know you can have both a strong foundation and traffic management features? Let’s dispel some confusion and go through examples together on enabling Cilium to manage your cluster’s traffic.

[eBPF’s Abilities and Limitations: The Truth - Liz Rice & John Fastabend, Isovalent](https://kccnceu2024.sched.com/event/1YeQt?iframe=no)

Friday, March 22, 2024, 11:55 - 12:30 CET

eBPF is proving to be a great platform for cloud native infrastructure tooling, with several CNCF projects leveraging it to implement networking, security and observability capabilities from within the kernel. But as with any new technology, there are various myths and uncertainties circulating about it in the community, particularly around its limitations: you might hear that it’s not Turing complete, that it can’t be used for anything that involves state, or that it can’t be used to parse Layer 7 protocols. In this talk, we’ll disprove all these rumours with demonstrations including: - Looping in eBPF - Leveraging maps for the state - An eBPF implementation of a Turing machine equivalent This doesn’t mean eBPF is the right hammer for every nail; using the Cilium project as an example we’ll discuss why not every feature is implemented in the kernel. (Yet?)

[Cilium ClusterMesh in Action: Strengthening Security Across Distributed Kubernetes Clusters - Matheus Morais, Sicredi](https://kccnceu2024.sched.com/event/1YeRT/cilium-clustermesh-in-action-strengthening-security-across-distributed-kubernetes-clusters-matheus-morais-sicredi?iframe=no&w=100%&sidebar=yes&bg=no)

Friday, March 22, 2024, 14:00 - 14:35 CET

Sicredi has over 2K applications running on 47 different K8s clusters deployed across a private cloud, built on Canonical OpenStack and public clouds. Such a complex environment needs a uniform framework to create a consistent security policy. Sicredi chose Service Mesh as such a mechanism, evaluated multiple competitors and selected Cilium ClusterMesh. This presentation will describe the evaluation process. Compare features of various service meshes and show how Sicredi used Cilium ClusterMesh to implement a full security policy across its infrastructure. Cilium is used as the CNI in Sicredi K8s clusters. Selecting it reduced operational and maintenance complexity. Cilium uses eBPF and avoids sidecars so has the best performance vs competing service meshes. Its fine-grain Cilium Network Policy made it possible to create rules access and operations rules per target application. Sicredi is Brazil’s largest credit union financial institution with over 7 million members.

[Comparing Sidecar-Less Service Mesh from Cilium and Istio - Christian Posta, Solo.io](https://kccnceu2024.sched.com/event/1YeRx/comparing-sidecar-less-service-mesh-from-cilium-and-istio-christian-posta-soloio?iframe=no&w=100%&sidebar=yes&bg=no)

Friday, March 22, 2024, 14:55 - 15:30 CET

Service mesh is a powerful pattern for implementing strong zero-trust networking practices, introducing better network observability, and allowing for more fine-grained traffic control. Up until now, the sidecar pattern was used to implement service-mesh capability but as the technology matures, a new pattern has emerged: sidecarless service mesh. Two prominent open-source networking projects, Cilium and Istio, have implemented a sidecar-free approach to service mesh but they both make interesting design decisions and tradeoffs. In this talk we review the architecture of both, focusing on the pros and cons of implementations such as mutual authentication, ingress, and observability.

#### Cilium - Empowering the Enterprise

URL: https://cilium.io/blog/2024/01/26/cilium-empowering-the-enterprise
Date: 2024-01-26
Categories: Technology

_Author: Shedrack Akintayo, Isovalent_

##### Introduction

As the cloud native ecosystem has matured, simplifying complexities without compromising useability or performance is key. Cilium is a cloud native networking, observability and security platform built to address these challenges, even as the scale and challenges of the platform grow.

This article aims to trace Cilium’s journey from its early days to becoming a go-to solution for enterprises in search of a rock-solid networking, observability and security solution. Understanding this journey is significant as it mirrors the broader industry's transition towards cloud native technologies, which is essential in harnessing the full potential of application teams and modern infrastructure.

##### The Evolution of Cilium

In 2014, the Linux networking ecosystem experienced a wave of innovation that reshaped it.

During this period, the ecosystem was focused on building various networking protocols and models including [software-defined networking](https://en.wikipedia.org/wiki/Software-defined_networking). At the same time, [Kubernetes made its first commit](https://github.com/kubernetes/kubernetes/commit/2c4b3a562ce34cddc3f8218a2c4d11c7310e6d56) and containerization was taking off. With these new improvements in the ecosystem, the pre-existing Linux networking, observability, and security tools became unable to keep up with the dynamic needs of containerised and orchestrated applications.

eBPF had also just been merged into the Linux kernel, bringing programmability and flexibility to this decades old technology. Some of the key architects behind the revolutionary technology wanted to bring the power of eBPF to end users, leading to the creation of the [Cilium project](https://cilium.io/). The main goal for Cilium was to introduce a new networking layer that was programmable, scalable, and secure by default to keep up with the needs of IT infrastructure.

The initial implementation of Cilium was as an eBPF-based Container Networking Interface (CNI) to provide connectivity for container workloads. As the cloud native ecosystem grew, Cilium’s use cases evolved, expanding to include a myriad of capabilities like [service mesh](https://cilium.io/use-cases/service-mesh/), [BGP](https://cilium.io/use-cases/bgp/), [network encryption](https://cilium.io/use-cases/transparent-encryption/), etc. Each new capability added was a testament to Cilium's evolving nature, constantly adapting to meet the complex demands and challenges that cloud native networking, observability and security presented.

Cilium’s evolution is a great example of innovation spurred by necessity and an evolving journey towards creating a platform that meets the requirements of the cloud native world.

##### eBPF - The Heart of Cilium

eBPF is the core technology powering Cilium, providing high-performance networking, security, and deep observability in cloud native environments like Kubernetes clusters and even beyond​.

eBPF has high performance because it is [JIT (Just-In-Time) compiled](https://en.wikipedia.org/wiki/Just-in-time_compilation) and runs directly within the Linux kernel. This kernel-level operation offers enhanced performance, security, and flexibility. eBPF programs are verified to ensure they won't crash the kernel and can only be modified by privileged users. This opens up a world where developers can easily diagnose problems, innovate quickly, and extend operating system functionality, unlocking new ways of building cloud native infrastructure that weren’t possible before​.

Major organizations and projects such as [Meta](https://www.youtube.com/watch?v=ZYBXZFKPS28), [Netflix](https://netflixtechblog.com/how-netflix-uses-ebpf-flow-logs-at-scale-for-network-insight-e3ea997dca96), and [Google](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine) use eBPF across their organization for its ability to enhance system performance, provide granular observability, and provide better security. Additionally, its [wide adoption](https://ebpf.io/case-studies/) and contributions from various organizations underline its importance and the pivotal role it plays in modern technology solutions.

Harnessing the power of eBPF enables dynamic insertion of powerful security, visibility, and networking control logic right into the Linux kernel. This is the cornerstone of Cilium's capabilities in providing [high-performance](https://cilium.io/use-cases/cni/) and [multi-cluster](https://cilium.io/use-cases/cluster-mesh/) networking, [advanced load balancing](https://cilium.io/use-cases/load-balancer/), [observability](https://cilium.io/use-cases/protocol-visibility/), [transparent encryption](https://cilium.io/use-cases/transparent-encryption/), and [extensive network security](https://cilium.io/use-cases/runtime-enforcement/) capabilities, to name a few.

##### Cloud Native Challenges Meet Their Match with Cilium

In enterprise organizations, transitioning from legacy systems into the cloud native world can feel like a bucket of cold water raining down on you. With constant changes in the environment, navigating these new waters often feels like finding a needle in a haystack. The stakes are high with challenges ranging from securing customer data, maintaining high performance, keeping up with customer demands with scalable software, and avoiding outages that degrade customer experience. Here's where Cilium steps in as a solution for these enterprise-grade challenges.

With Cilium at the helm, enterprises can seamlessly implement [network policies](https://cilium.io/use-cases/network-policy/), crafting a secure and compliant networking environment. This is particularly crucial for enterprises operating in highly regulated industries, where data encryption and adherence to standards and regulations are paramount.

In a microservices-based system, observability is the key to discovering the intricacies of network operations and debugging when things go wrong. Cilium provides in-depth [observability](https://cilium.io/use-cases/protocol-visibility/) with [Hubble](https://docs.cilium.io/en/stable/overview/intro/), paving the way for a clear understanding of network traffic. This, in turn, simplifies troubleshooting and performance optimization, acting as a catalyst in diagnosing and resolving network-related issues.

High-performance networking is not just a need but a critical requisite for real-time applications and services. Enterprises seeking to optimize their Kubernetes deployments for high-performance networking can reap significant benefits from adopting Cilium. Cilium's unique architecture enables it to operate directly within the Linux kernel, resulting in substantial [performance improvements](https://cilium.io/blog/2021/05/11/cni-benchmark/). This translates into faster application response times, reduced delays for data-intensive workloads, enhanced throughput for high-bandwidth applications, and support for larger-scale deployments.

As enterprises grow, the scalability of networking solutions also becomes a focal point. Cilium offers a high-performance [layer 4 load balancer](https://cilium.io/use-cases/load-balancer/) designed to efficiently handle the networking demands of large-scale, distributed architectures. Cilium’s [Kube-proxy replacement](https://cilium.io/use-cases/kube-proxy/) can also provide enhanced networking speed and efficiency for enterprises building on Kubernetes. These features of Cilium ensure that as the enterprise grows, the networking solutions evolve equally, meeting the dynamic demands head-on.

The narrative of Cilium being referred to as a game-changer in the enterprise domain comes from its ability to simplify the complex world of cloud native networking, security, and observability, making it a critical component of modern cloud native platforms.

##### Cilium in Action: Case Studies

To fully understand Cilium’s impact on the enterprise software ecosystem, we need to look into real-world scenarios. In this section, we'll explore three distinct case studies from Bloomberg, Meltwater, and Trendyol to showcase Cilium's capabilities in tackling networking, observability and security challenges in enterprise organizations.

###### Bloomberg

Bloomberg ventured into creating a next-generation quantitative investment solution known as BQuant Enterprise. While architecting the data sandboxes for the BQuant Enterprise cloud, data security emerged as a significant challenge. The primary goal was to bolster the protection of both the firm’s and its clients' data by restricting automated data egress and customer resource access.

The Bloomberg Engineering team turned to Cilium, drawn by its host-based network policies. Through Cilium, they restricted cluster network access to specific ports and host names to curb data egress. As they built their data sandbox storage solution, Cilium simplified the creation of an access control system for data from user workloads via its policy exception feature and allowed them to offer additional features to their customers by leveraging Layer 7 policies. Moreover, Cilium’s Hubble enhanced observability, making network debugging simpler for the engineering teams.

Cilium significantly heightened the security and access control for BQuant Enterprise workloads which allowed Bloomberg to successfully prevent unauthorized egress of protected data and access to unauthorized resources. This proactive approach also allowed the enforcement of licensing restrictions across datasets, minimizing the risk of unlicensed access to sensitive information. Additionally, Hubble’s observability features empowered Bloomberg's Engineers to troubleshoot network problems efficiently, saving valuable development time. You can read more about this case study [here](https://www.cncf.io/case-studies/bloomberg-2/).

###### Meltwater

Meltwater, a global media intelligence company, initially used the AWS VPC CNI for their Kubernetes platform. However, they stumbled upon challenges such as the lack of crucial features like network policies and encryption, AWS API rate limiting, performance glitches with kube-proxy/iptables, and limited observability within their clusters. The quest for a robust solution to these challenges led them to Cilium.

Cilium facilitated a seamless transition from the existing AWS VPC CNI without any service interruption to their clientele. With Cilium as the default CNI across all of Meltwater’s Kubernetes clusters, the payoff has been substantial. The company now enjoys improved performance, better architecture and notably, improved network observability courtesy of Hubble.

Meltwater's live migration to Cilium established a solid groundwork for better network management and observability in their Kubernetes platform. Hubble, in particular, helped them in debugging network issues, visualizing network traffic, and making troubleshooting more efficient, faster, and easier. You can read more about this case study [here](https://www.cncf.io/case-studies/meltwater/).

###### Trendyol

Trendyol, a leading e-commerce platform in Turkey, initially employed Flannel and Calico for networking in their Kubernetes platform. However, as their infrastructure grew, scalability and performance challenges surfaced. The anticipation of a further increase in the number of Kubernetes clusters and the size of the clusters made them look for an alternative CNI solution. Their goal was to bolster the network performance of their Kubernetes cluster’s connectivity, manage 3-5 thousand nodes within a single cluster, and remove kube-proxy to augment the overall performance of their Kubernetes clusters.

After rigorous performance testing of various solutions, Trendyol opted for Cilium as their foundation for networking, observability, and security in their platform. The transition to Cilium marked a significant leap in performance, effective observability with Hubble, and enhanced scalability and security for their Kubernetes clusters.

Cilium is now the default CNI for Trendyol’s Kubernetes clusters. The migration to Cilium’s advanced networking from Flannel increased their network performance **by over 40%**, according to their internal benchmarks, significantly transcending their expectations. You can read more about this case study [here](https://www.cncf.io/case-studies/trendyol/).

The practical implementation of Cilium in Bloomberg, Meltwater, and Trendyol’s systems showcases its robust, adaptable nature in addressing distinct challenges in varied enterprise environments. From enhancing data security and observability to significantly improving networking performance and scalability, Cilium emerges as a comprehensive platform.

##### Cilium: Built For The Enterprise

Navigating the cloud native networking, observability, and security ecosystem often unveils complex challenges, bringing tools like Cilium to the forefront. We traced Cilium's evolution from its roots amidst the early Kubernetes ecosystem to its current stature as a trusted enterprise-grade solution, all powered by eBPF.

Through the lens of real-world implementations at Bloomberg, Meltwater, and Trendyol, we saw how Cilium substantially improved network performance, tightened security, and offered detailed observability, addressing unique challenges across many different companies. Cilium represents a shift in how companies can approach networking, observability, and security for their cloud native platform.

If you want to learn more about Cilium read through [Cilium’s documentation](https://docs.cilium.io/en/stable/), engage with hands-on [Cilium labs](https://cilium.io/labs/), or follow the [getting-started guides](https://cilium.io/get-started/) to learn more about its capabilities. For those keen on engaging in discussions or staying updated with the latest information, joining [Cilium’s Slack community](https://slack.cilium.io) is a way to add to the conversation.

If you're geared towards contributing to Cilium, the project is [open to your contributions](https://github.com/cilium/cilium/issues?q=is%3Aopen+is%3Aissue+label%3Agood-first-issue). Every code commitment, documentation enhancement, or community interaction nudges Cilium closer to addressing the dynamic challenges in the cloud native networking observability, and security platforms.

#### Cilium in Paris! See you at KubeCon Europe 2024

URL: https://cilium.io/blog/2024/01/18/cilium-kubecon-eu-2024
Date: 2024-01-18
Categories: Community

There will be lots of Cilium-related activity at KubeCon + CloudNativeCon Europe
in Paris! We'll keep this page updated with more details as they are published.

- Sign up for the new [Cilium Certified Associate
 exam](https://training.linuxfoundation.org/certification/cilium-certified-associate-cca/)!

- Come to the [Cilium + eBPF Day](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/co-located-events/cilium-ebpf-day/) co-located event

- Attend the Maintainers' Track session

 [Cilium: Connecting, Observing, and Securing Service Mesh and Beyond with EBPF - Liz Rice & Maartje Eyskens, Isovalent; Nico Meisenzahl, white duck; Vlad Ungureanu, Palantir Technologies](https://kccnceu2024.sched.com/event/1Yhfl/cilium-connecting-observing-and-securing-service-mesh-and-beyond-with-ebpf-liz-rice-maartje-eyskens-isovalent-nico-meisenzahl-white-duck-vlad-ungureanu-palantir-technologies?iframe=no&w=100%&sidebar=yes&bg=no)

 Wednesday March 20, 2024 15:25 - 16:00 CET

 Welcome to Cilium's maintainer track session where you'll get an update on how Cilium is expanding the frontiers of cloud native networking, observability, and security. After graduating and becoming the de-facto standard CNI for Kubernetes, what comes next? We'll start with a brief overview of each part of the project before giving a update on Cilium Service Mesh's mutual authentication. Next, we'll dive into how Cilium is expanding beyond Kubernetes with load balancing and multi-cloud networking and into runtime enforcement with Tetragon. In this session, you'll hear from Cilium contributors and users Isovalent, Palantir, and white duck.

- Come and have a chat at the Cilium kiosk in the Project Pavilion throughout
 the event

##### Cilium Talks at KubeCon EU 2024 Full Schedule

Below is a list of some Cilium talks at KubeCon EU including Cilium + eBPF Day and Observability Day.

You can find more details about each talk in this [blog post](https://cilium.io/blog/2024/02/02/cilium-at-kube-con-eu-2024).

À bientôt à Paris!

#### eBPF’s Journey – Unlocking The Kernel

URL: https://cilium.io/blog/2023/10/30/the-ebpf-documentary
Date: 2023-11-08
Categories: Community

**November 8th, 2023**

**Author: Shedrack Akintayo, Isovalent**

eBPF is a revolutionary technology that allows safe and efficient modification of the Linux kernel’s behavior without altering its source code or loading extra modules. This is the technology that gives Cilium its secure network connectivity capabilities, deep application observability via Hubble, and runtime enforcement capabilities with Tetragon.

The eBPF documentary is a story of a journey that began with an idea—transforming the Linux kernel into a programmable entity, which has since redefined the boundaries of what's possible in the industry. It features pioneers of the technology like [Alexei Starovoitov](https://www.linkedin.com/in/alexey1), [Thomas Graf](https://ch.linkedin.com/in/thomas-graf-73104547), [Daniel Borkmann](http://borkmann.ch/), [Brendan Gregg](https://www.brendangregg.com/), David Miller and [Liz Rice](https://uk.linkedin.com/in/lizrice), who have been instrumental in eBPF's growth.

The documentary dives deep into the motivations of the individuals who stand at the forefront of this technology and showcase the role that Cilium played in its development. Thomas Graf, alongside Daniel Borkmann and other brilliant minds, embarked on a mission: to harness the raw power of eBPF and make it accessible to end users through Cilium. Their vision was clear—to introduce a new layer of networking that was secure by default.

Learn about Cilium's inception and how it is crafting a secure networking layer for the future. Discover the story of Cilium and how DockerCon 2017 sparked a wave of community engagement with Cilum and eBPF, leading to milestones like Google's integration of[Cilium and eBPF into their GKE Dataplane](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine) in 2020.

Join us in exploring the untold story of eBPF from its inception to its adoption in the cloud native ecosystem and how it is driving the ecosystem forward through projects like Cilium.

You can watch the eBPF documentary [here](https://www.youtube.com/watch?v=Wb_vD3XZYOA)

#### Cilium Talks at KubeCon NA 2023

URL: https://cilium.io/blog/2023/10/30/cilium-kubecon-na-talks
Date: 2023-10-30
Categories: Community

_October 30th, 2023_

_Author: Shedrack Akintayo, Isovalent_

Fresh on the heels of [Cilium’s graduation](https://www.cncf.io/announcements/2023/10/11/cloud-native-computing-foundation-announces-cilium-graduation/), the community is gathering for the second [CiliumCon](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/co-located-events/ciliumcon) and eighth [KubeCon + CloudNativeCon North America 2023](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/). Cilium is once again taking center stage in the cloud native ecosystem, highlighting how the next generation of networking, observability, and security is being built with eBPF and bringing with it a tale of growth, community, and technological innovation. This isn't just a sequel, but a new narrative unfolding, spotlighting the strides Cilium has made in the cloud native ecosystem.

CiliumCon promises to be a deep dive into the world of eBPF, networking, observability, security, and more, through the lens of Cilium. The [agenda](https://colocatedeventsna2023.sched.com/overview/area/CiliumCon?iframe=no) is packed with insightful talks from seasoned practitioners, core contributors, and community members with topics spanning from the core technical aspects of Cilium to real world implementations and case studies.

At the broader KubeCon + CloudNativeCon North America 2023, as in the past, Cilium is poised to be a significant part of the discourse, reflecting its growing importance in the cloud native ecosystem. With Cilium’s presence at the event, the goal is to equip you with the insights and tools necessary to harness the power of Cilium and eBPF, to improve your networking, making your platforms more secure, efficient, and observable.

Now, Let’s dive into each of the talks on Cilium at both events!

##### CiliumCon NA 2023

[From Imagination to Implementation: Inside Adobe's Production-Grade Deployment with Cilium - Joseph Sandoval & Tony Gosselin Adobe](https://colocatedeventsna2023.sched.com/event/1Rj0z/from-imagination-to-implementation-inside-adobes-production-grade-deployment-with-cilium-joseph-sandoval-tony-gosselin-adobe)

**Monday, November 6, 2023 • 9:15 am - 9:40 am**

Adobe has trusted Cilium in its production deployments for over seven years, proving its resilience and extensibility amidst the rapidly evolving developer workload needs. In this session, we'll delve deep into Cilium's role within Adobe's internal developer platform, which runs on AKS, EKS, and our data centers. Adobe has seen how Cilium's extensibility stands out from its early days integrating with Mesosphere, migrating to Kubernetes, and now has supported new challenges head-on, recently proving its scalability with the launch of AI/ML features. This session will provide a candid behind-the-scenes of running Cilium on a multi-tenant IDP. We'll provide insights from our recent experiences launching generative AI applications on Kubernetes and Cilium. Plus, get a sneak peek into our future roadmap and how Adobe plans to harness Cilium for a post-container world.

[Controlling Access to External APIs with Cilium - Luis Ramírez, SuperOrbital](https://colocatedeventsna2023.sched.com/event/1Rj1R/controlling-access-to-external-apis-with-cilium-luis-ramirez-superorbital)

**Monday, November 6, 2023 • 9:50 am - 10:15 am**

We’ve worked on many customer projects that utilize Cilium as the keystone for their networking security solution. In this session, we provide a step-by-step guide, complete with explanations and examples, on how to configure Cilium to limit access to GitHub repos for workloads inside a cluster. This will involve creating certificates for terminating TLS, mounting and trusting certificates inside Pods, and crafting L7 network policies targeting specific groups of Pods in a cluster. Defense in depth is important in multi-tenant environments. Making sure that there are multiple safeguards from unauthorized access is a necessity that may require a belt-and-suspenders approach. To this end, we’ve found that the utility that Cilium provides is immense and often underused. After this session, participants will have a much clearer understanding of how Cilium, with its Network Policies and its visibility functionality, can provide a novel and layered approach to external API access control**.**

[Sponsored Keynote: Advancing Cilium Within the Kubernetes Ecosystem - Idit Levine, Solo.io](https://colocatedeventsna2023.sched.com/event/1RqIt/sponsored-keynote-advancing-cilium-within-the-kubernetes-ecosystem-idit-levine-soloio)

**Monday, November 6, 2023 • 10:20 am - 10:25 am**

As the cloud native landscape evolves, Cilium has emerged as the go-to choice. But the Kubernetes ecosystem is vast, so how do you choose an open source tech stack to connect, manage, secure, and observe communication between your applications? Whether you’re enhancing container orchestration capabilities throughout your networking interface, service mesh, edge proxy, or improving your security posture, there are open source projects emerging as clear winners–such as Cilium–that are poised to help you achieve your goals. By leveraging the capabilities of these top cloud native projects, you can effortlessly implement advanced features such as enforcing network policies, mutual TLS, FIPS compliance, rate limiting or holistic observability spanning both network and application layers. This comprehensive approach extends its benefits to Kubernetes clusters and beyond, empowering you to optimize security, traffic control, and observability across diverse environments.

[Sponsored Keynote: Effortless Mutual Authentication With Cilium`- Christine Kim, Isovalent](https://colocatedeventsna2023.sched.com/event/1RqJH/sponsored-keynote-effortless-mutual-authentication-with-cilium-christine-kim-isovalent)

**Monday, November 6, 2023 • 10:30 am - 10:35 am**

Let’s take a look at the new, frictionless way to cryptographically authenticate workload identities using Cilium. In this talk you’ll see how, by adding just two lines of YAML to a network policy, you can enable a SPIFFE-based mutual authentication handshake between endpoints. Just like Cilium’s existing IPSec and Wireguard implementations, this is completely transparent to the application. If you want to secure Kubernetes traffic, don’t miss this talk!

[What's Smoother Than Your Morning Espresso Pull? Bridging Gaps with BGP and Cilium! - Marino Wijay, Solo.io](https://colocatedeventsna2023.sched.com/event/1Rj2I/whats-smoother-than-your-morning-espresso-pull-bridging-gaps-with-bgp-and-cilium-marino-wijay-soloio)

**Monday, November 6, 2023 • 11:05 am - 11:30 am**

The widely adopted, highly capable Border Gateway Protocol has won the hearts and minds of the Internet. And, it’s made its way into Kubernetes! However, not every workload is a container, and not every VM can make its way into Kubernetes so, how do we ensure every workload can freely talk? If BGP on Cilium is the answer how should we proceed? - Why should we care about BGP? - How do we work with Cilium’s BGP Control Plane? - How can we get our VMs to communicate with our pods using BGP? - Can we see a working BGP setup? - Does “show ip bgp summary” still work? This presentation allows us to delve into BGP internals and history while providing a demonstration of BGP in action, running in Cilium. We’ll become the best neighbors with an upstream router, and we’ll share ALL THE NETWORKS!

[Using Cilium CNI in ClickHouse Cloud - Timur Solodovnikov, ClickHouse, Inc](https://colocatedeventsna2023.sched.com/event/1Rj2m/using-cilium-cni-in-clickhouse-cloud-timur-solodovnikov-clickhouse-inc)

**Monday, November 6, 2023 • 11:40 am - 12:05 pm**

Join us at CiliumCon as we delve into the use of Cilium CNI to optimize and enhance deployments in the ClickHouse Cloud. This session will explore how Cilium, a powerful networking and security solution, integrated into ClickHouse Cloud.

[Past, Present, Future of Tetragon- First Production Use Cases, Lessons Learnt, Where Are We Heading? - Natalia Reka Ivanko & John Fastabend, Isovalent](https://colocatedeventsna2023.sched.com/event/1Rj3S/past-present-future-of-tetragon-first-production-use-cases-lessons-learnt-where-are-we-heading-natalia-reka-ivanko-john-fastabend-isovalent)

**Monday, November 6, 2023 • 12:15pm - 12:40pm**

As many of you know already, Tetragon is reaching v1.0 - being a huge milestone for the community, our users as well as the engineers who have been working on the development for years. This talk's main aim is to walk through the history of Tetragon, and predict its future trajectory. Why was it created in the first place? How? What were the first production use cases? What were the security challenges users were facing? Where are they at now? What were the lessons we learnt through this rocket-propelled roller coaster journey? As an example, the audience will learn how we avoided some pitfalls that BPF-based Security applications fall into and solved critical performance challenges. We’ll finish by presenting where the project is heading, what are the main goals and use cases we are aiming to achieve in the future.

[Secure Infrastructure with Combined Runtime and Network Security - Thomas Graf, Isovalent](https://colocatedeventsna2023.sched.com/event/1Rj4B/secure-infrastructure-with-combined-runtime-and-network-security-thomas-graf-isovalent)

**Monday, November 6, 2023 • 1:30pm - 1:55pm**

Network and runtime security have been known as separate security layers for years. Why are we limiting runtime to the scope of a single node? And why are we stopping at the network layer when we could derive a more accurate identity instead of a "network endpoint"? In this talk, we will look into the future of security as we fulfill the vision of a combined and fully coordinated Cilium and Tetragon stack. Imagine a world where processes and identities are fully authenticated end-to-end. As we explore this new stack, we will compare it to using separate uncoordinated tools such as AppArmor, SELinux, micro-segmentation, and mTLS.

[From Eventual to Strict Encryption – Securing Cilium’s WireGuard Encryption - Leonard Cohnen, Edgeless Systems](https://colocatedeventsna2023.sched.com/event/1Rj4j/from-eventual-to-strict-encryption-securing-ciliums-wireguard-encryption-leonard-cohnen-edgeless-systems)

**Monday, November 6, 2023 • 2:05 pm - 2:30 pm**

Cilium's routing and encryption decisions are based upon endpoint identities. Those identities are distributes via Kubernetes CRDs. If an application sends traffic to another pod inside the cluster, it is only encrypted if the destination's identity has propagated to the node on which the application runs. Depending on the configuration this could result in Pod-to-Pod traffic being sent unencrypted over the wire for multiple seconds. To mitigate this issue we developed and upstreamed a new WireGuard strict mode which we'll present in this talk.

[Come BGP with Me - Daneyon Hansen, Solo.io & Yutaro Hayakawa, Isovalent](https://colocatedeventsna2023.sched.com/event/1Rj5D/come-bgp-with-me-daneyon-hansen-soloio-yutaro-hayakawa-isovalent)

**Monday, November 6, 2023 • 2:40 pm - 3:05 pm**

When Border Gateway Protocol (BGP) was first introduced in Cilium v1.10, MetalLB was used as the control plane with minimal support for the protocol. This gave users a simple and effective way to route ingress traffic to services and pods. As time passed, the requirements for BGP grew and Cilium answered the call with a wide range of new features. The BGP feature set has undergone considerable change in recent releases by deprecating MetalLB in favor of a native BGP control plane. In this session, attendees will learn about Cilium’s BGP features and how they work harmoniously to provide a powerful ingress solution.

[Day 2 with Cilium - What to Expect Running at Scale - Hemanth Malla & Maxime Visonneau, Datadog](https://colocatedeventsna2023.sched.com/event/1Rj5p/day-2-with-cilium-what-to-expect-running-at-scale-hemanth-malla-maxime-visonneau-datadog)

**Monday, November 6, 2023 • 3:15 pm - 3:40 pm**

Cilium works great out of the box, but tweaking a few options will ensure you get the best performance as your clusters grow in size. While Cilium abstracts away a lot of complexity, users are provided with several knobs to control the underlying systems wherever necessary. This talk will quickly brush up on how Cilium interacts with Linux, Kubernetes and cloud provider network stacks and then dive into some of the challenges you might encounter running Cilium in large clusters. For example, over time, you might need to reevaluate rate limiting across the board to avoid cascading failures. You will need to carefully plan for IP address management, tune the operating system or take advantage of new features from cloud providers. Since Cilium is a critical piece of your infrastructure, we’ll talk about what health metrics to keep a close eye on. We’ll also discuss some best practices for deploying and validating your rollouts with out of the box connectivity tests.

[Why KVStoreMesh? Lessons Learned from Scale Testing Cluster Mesh with 50k Nodes Across 255 Clusters - Ryan Drew, Isovalent](https://colocatedeventsna2023.sched.com/event/1Rj6E/why-kvstoremesh-lessons-learned-from-scale-testing-cluster-mesh-with-50k-nodes-across-255-clusters-ryan-drew-isovalent)

**Monday, November 6, 2023 • 3:50 pm - 4:15 pm**

Cilium Cluster Mesh is a feature that connects workloads across clusters, but it has its limitations in the realm of scalability. This talk aims to boil a six-month project which explored and tested the scaling limitations of Cilium Cluster Mesh down into a half-hour, where you’ll review how Cluster Mesh syncs information between clusters, how this syncing broke in a scale test with 50k nodes across 255 clusters, how the scale test was architectured using kubemark to avoid breaking the bank, and how the 1.14 beta feature KVStoreMesh addresses Cluster Mesh’s scaling limitations to provide the opportunity for greater stability.

[Migrating from Legacy with Ease, a.k.a. Cilium in Openstack and More - Ondrej Blazek, Seznam.cz](https://colocatedeventsna2023.sched.com/event/1Rj6e/migrating-from-legacy-with-ease-aka-cilium-in-openstack-and-more-ondrej-blazek-seznamcz)

**Monday, November 6, 2023 • 4:25 pm - 4:50 pm**

Everybody knows Kubernetes, that's what this is all about, right? But most companies do not run everything in k8s, they might have legacy applications or databases that are difficult to run in k8s. What if some of them are in something like OpenStack or directly on baremetal? What if part of then are running in k8s with Calico and part of them run in OpenStack? It would be great to use Cilium with CiliumNetworkPolicies to enforce the boundaries. This is all possible in k8s, but what happens if we use the same network layer also outside of k8s? What if each OpenStack cluster is connected with k8s clusters with Cilium and k8s clusters with Calico and form a clustermesh? Sounds crazy? It might not be as crazy as it sounds, thanks to Cilium.. this is possible. See for yourself.

[Netreap: Bridging the Gap Between Cilium and Nomad - Dan Norris, Cosmonic](https://colocatedeventsna2023.sched.com/event/1Rj79/netreap-bridging-the-gap-between-cilium-and-nomad-dan-norris-cosmonic)

**Monday, November 6, 2023 • 5:00 pm - 5:25 pm**

Cilium is one of the best ways to manage network policies and secure workloads running in a Kubernetes cluster. As a CNI, it provides everything you would want and more! The only problem is what do you do if you want to run Cilium but are not running Kubernetes? In this talk we will outline the steps we took to deploy and run Cilium in our Nomad clusters alongside the rest of our infrastructure. We'll dive into what it took to replace Cilium's operator component with Netreap, an open source tool that bridges Cilium and Nomad. Along the way we'll discuss Cilium's endpoint and labeling model along with some of the lower level APIs that power Cilium. To conclude, we'll go though some of the limitations of our approach and outline some ways that we can make Cilium better for everyone.

##### ArgoCon

[Lightning Talk: Git Going Fast with Cilium and Argo - Christine Kim, Isovalent](https://colocatedeventsna2023.sched.com/event/1Rj31/cl-lightning-talk-git-going-fast-with-cilium-and-argo-christine-kim-isovalent)

**Monday, November 6, 2023 • 12:10pm - 12:15pm**

Ever wanted to use Cilium and ArgoCD together? As declarative tools continue to increase, the work to get started can seem overwhelming. This lightning talk will get you going with setting up your GitOps leveraging Cilium and will highlight the strengths of being declarative with your environment.

##### KubeCon + CloudNativeCon

[Keynote: CNCF Graduated Project Updates](https://kccncna2023.sched.com/event/1R4Zf)

**Tuesday, November 7, 2023 • 10:00 am - 10:15 am**

Cilium will be included in the Graduated projects update keynote.

[When Is a Secure Connection Not Encrypted? and Other Stories - Liz Rice, Isovalent](https://kccncna2023.sched.com/event/1R2tD)

**Tuesday, November 7, 2023 • 3:25 pm - 4:00 pm**

Many organizations use a Service Mesh to secure traffic between apps. This may use Mutual TLS, with a proxy terminating connections on behalf of apps. mTLS starts with a handshake to authenticate endpoint identities, and exchange certificates for subsequent traffic encryption. When encryption is needed but app authentication is not, approaches like WireGuard or IPSec may be more suitable. What about scenarios where authentication is important but encryption adds too much latency? With demos to make concepts concrete, let’s dive into Cilium's approach to authentication and encryption, and the differences between mTLS and in-kernel alternatives.

- Explore the mTLS handshake step-by-step
- Contrast with transparent encryption using node identities
- Understand where encryption takes place in different models
- Discuss options for encrypting L7 protocols other than HTTP

With a clear picture of how authentication and encryption work, you’ll be better able to assess which approach best meets your needs.

[Tutorial: Demystifying Cilium: Learn How to Build an eBPF CNI Plugin from Scratch - Adam Sayah, Solo.io](https://kccncna2023.sched.com/event/1R2on/tutorial-demystifying-cilium-learn-how-to-build-an-ebpf-cni-plugin-from-scratch-adam-sayah-soloio?iframe=no&w=100%&sidebar=yes&bg=no)

**Tuesday, November 7, 2023 • 4:30 pm - 6:00 pm**

eBPF technology is driving a transformative shift in the network stack, enabling secure code execution within a protected kernel sandbox. This facilitates instant metrics retrieval and the implementation of network routing and security policies. Additionally, eBPF empowers us to reshape traffic at a low level with XDP, which delivers high-performance programmable packet processing seamlessly integrated with the kernel. This revolutionary impact extends to the Kubernetes networking landscape, as eBPF serves as the core technology behind projects like Cilium and its CNI plugin that provides high-performance network capabilities, but such powerful technologies remain enigmatic for many. The objective of this workshop is to dive into the inner workings of these technologies, participants will learn the basics of eBPF and CNI, and they will gain hands-on experience in creating a CNI plugin for Kubernetes utilizing eBPF, demystifying the underlying mechanics of eBPF-based projects.

[Service Mesh Battle Scars: Technology, Timing, and Tradeoffs - Keith Mattix, Microsoft; John Howard, Google; Lin Sun, solo.io; Thomas Graf, Isovalent; Flynn, Buoyant](https://kccncna2023.sched.com/event/1R2ts)

**Tuesday, November 7, 2023 • 5:25 pm - 6:00 pm**

It has been ~6 years since the first service meshes hit the market, and hundreds if not thousands of users are employing the technology in production today. As the space has matured and meshes are used at scale, many users are wondering about why certain constraints exist in their mesh of choice. Come hear maintainers of 3 CNCF service mesh projects discuss the very real tradeoffs they’re making everyday with topics ranging from eBPF, sidecarless, Rust proxies, multicluster, and user experience.

[Turning up Performance to 11: Cilium, Meta Devices, and Going Big with TCP - Daniel Borkmann, Isovalent](https://kccncna2023.sched.com/event/1R2s5/turning-up-performance-to-11-cilium-meta-devices-and-going-big-with-tcp-daniel-borkmann-isovalent?iframe=no&w=100%&sidebar=yes&bg=no)

**Wednesday, November 8, 2023 • 4:30 pm - 5:05 pm**

The need to move more bits faster through the cloud is constantly turning up, from “day 2” challenges like migrating more workloads into Kubernetes environments, connecting multiple clusters, or simply escalating data demands from AI applications. In this talk, we discuss Cilium’s journey to achieve host-level network performance and latency for Pods. We will deep dive into changes we upstreamed to the Linux kernel and then brought to Cilium such as the new veth-replacement driver. We also take a look at new features like BIG TCP which so far have not been brought into the wider Kubernetes community. This talk is for operators who want to get the most out of their setup. While many start with the default settings for getting started on “day 1”, “day 2” challenges will often lead to seeking ways to turn the performance knob up. The audience will walk away knowing how resources can be freed up and given to the application just by tweaking software to finally reach 11 for performance.

[Paint the Picture! - Detecting Suspicious Data Patterns in Encrypted Traffic with eBPF and KTLS - Natalia Reka Ivanko & John Fastabend, Isovalent](https://kccncna2023.sched.com/event/1R2v2)

**Wednesday, November 8 • 5:25 pm - 6:00 pm**

Using eBPF to detect malicious events on Cloud Native environments continues to rise because it provides a wide range of options to monitor for suspicious runtime execution, network connections, and file access. However, detecting sensitive data patterns, like social security or credit card numbers in encrypted L7 network traffic has traditionally been done in user space. By leveraging in-kernel HTTP visibility and kTLS, we now have the ability to paint a complete security picture and monitor sensitive data flows between Kubernetes workloads, even if they are encrypted. Using Tetragon, this talk will demonstrate how eBPF can be applied to solve the technical challenge of decrypting TLS traffic by using kTLS and showcase how Security Teams can detect sensitive data patterns, like social security numbers or exploit signatures in encrypted L7 traffic. By using eBPF, this solution avoids operational complexity, overhead, and is fully transparent to the application as well as the CNI.

[Documentary Film - eBPF: Unlocking the Kernel](https://kccncna2023.sched.com/event/1RHS7)

**Wednesday, November 8 • 6:15 pm - 7:00 pm**

In 2014, a group of engineers at Plumgrid needed to find an innovative and cost-effective solution to handle network traffic in SDN environments. What they created was a landmark in the industry known as the extended Berkeley Packet Filter (or eBPF). This vital technology allows user-level code execution inside the Linux Kernel, transforming network traffic handling for SDN environments. Whether these engineers knew it or not, they had just revolutionized the Linux Kernel. We go in-depth to explore the origins of eBPF and showcase the stories, challenges, and rewards of this game-changing technology. In the documentary film, we interviewed key stakeholders from Meta, Intel, Isovalent, Google, Red Hat, and Netflix who helped shape and build the tools that drove the success and adoption of eBPF. Join us for this premiere screening of why eBPF is seen as one of the most promising, emerging applications and cutting-edge technologies in the Linux ecosystem.

[ Contribfest: Getting Started Contributing to Cilium - Bill Mulligan & Joe Stringer, Isovalent](https://kccncna2023.sched.com/event/1R2tk/contribfest-getting-started-contributing-to-cilium-bill-mulligan-joe-stringer-isovalent?iframe=no&w=100%&sidebar=yes&bg=no)

**Thursday, November 9, 2023 • 11:00 am - 12:30 pm**

"It's always DNS" has hit production and you have found a bug in Cilium. You think you have a fix, but aren't sure how to contribute it back to upstream Cilium. This is the Contribfest for you! We will start with an overview of Cilium for developers to help contributors understand how the different parts of the project interact and where they should look to contribute their code. From there, we will divide into groups based on functional areas to discuss new and outstanding PRs. New contributors to Cilium will learn how to contribute to the project and existing contributors will gain a better understanding of how their code connects to the rest of the project.

[Cilium: From Service Mesh to Kubernetes and Beyond with eBPF - James McShane, SuperOrbital; Hemanth Malla, Datadog; Liz Rice & Thomas Graf, Isovalent](https://kccncna2023.sched.com/event/1R2ux/cilium-from-service-mesh-to-kubernetes-and-beyond-with-ebpf-james-mcshane-superorbital-hemanth-malla-datadog-liz-rice-thomas-graf-isovalent?iframe=no&w=100%&sidebar=yes&bg=no)

**Thursday, November 9, 2023 • 2:00 pm - 2:35 pm CST**

In this session you'll get an update on how Cilium is expanding the frontiers of cloud native networking, observability, and security. You'll hear about the latest developments and future roadmap of the project and why it has become the CNI of choice in the wild. We will cover things like how Cilium is leveraging eBPF to speed up container networking, doing mutual authentication of services with Cilium Service Mesh, and expanding cloud native principles beyond Kubernetes with Cilium Mesh. In this session, you'll hear from Cilium contributors and users Datadog, Isovalent, and SuperOrbital.

Make sure you stop by the Cilium booth to get your Cilium swag and as you navigate through the sessions, engage with the community, and immerse yourself in the discourse, we hope you'll carry with you not just the technical know-how, but a sense of belonging to a community that’s propelling towards a future of improved cloud native networking. See you there!

[Scaling Kubernetes Networking to 1k, 5k,... 100k Nodes!? - Marcel Zięba, Isovalent & Dorde Lapcevic, Google](https://kccncna2023.sched.com/event/1R2vB)

**Thursday, November 9 • 2:55 pm - 3:30pm**

Setting up clusters with thousands of nodes can be challenging especially when running sophisticated networking like Cilium, which is required when serving internet-scale applications. In this session, you will learn best practices around scaling Cilium and Kubernetes from tech leads from Google Cloud (which allows 15,000 node clusters) and Isovalent (which scales Cilium Cluster Mesh-based clusters up to 256 clusters/100k nodes). It will compare two perspectives - a large single-cluster as well as an alternative - managing multiple smaller clusters and how these decisions impact performance/scalability/reliability. You’ll be able to hear from practitioners in the space how to configure your clusters, what metrics to monitor, and how to improve the reliability of your production environment. If you want to understand how networking features impact the Kubernetes control plane, how to minimize the blast radius of outages, and learn about large-scale incidents, this session is for you!

#### Meet the Maintainer - Aditi Ghag

URL: https://cilium.io/blog/2023/10/24/meet-maintainer-aditi-Ghag
Date: 2023-10-24
Categories: Community

_October 24th, 2023_

_Written by: Shedrack Akintayo, Isovalent_

##### How did you first get into technology?

I got into tech when I was in high school. I played with PCBs ([printed circuit boards](https://en.wikipedia.org/wiki/Printed_circuit_board)) a lot. It was fun exploring transistors and diodes because there is something uniquely satisfying about turning lights on in a controlled manner. During a summer vacation, I enrolled in an aero modelling workshop. The challenge was to craft a model plane from the ground up and the thrill of that first flight is still fresh in my memory.

On the other hand, my journey into programming began with the C language. Navigating through concepts like pointers was challenging but also exciting and it provided me with a solid foundation in programming. This knowledge also served me well when I transitioned to Go, which became my primary programming language while contributing to Cilium.

##### How did you get into open source?

Before I joined [Isovalent](https://isovalent.com/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-enterprise), the creators of Cilium, I was only using open source software instead of contributing to it. Joining Isovalent gave me the chance to actively participate and contribute to the Cilium and eBPF communities.

Cilium sits at an interesting boundary between Kubernetes and the Linux kernel. It interfaces with Kubernetes on the control plane side of things and it programs the data path in Linux to provide networking and security. In order to support Cilium’s use cases, I've contributed to the Kubernetes and the Linux kernel projects.

##### How did you first hear about eBPF and Cilium?

In 2017, I saw a [presentation](https://www.youtube.com/watch?v=k0KQz6JrKXc) from Thomas Graf on how Cilium leverages eBPF to provide programmable networking and security and it was pretty intriguing.

Couple of years down the line, I also got a chance to learn more about Cilum and eBPF at a Linux conference where I was giving a talk on how to bring network awareness to the Kubernetes scheduler. At the conference, I met some of the Isovalent folks, who are now my colleagues, and they gave an amazing presentation on how eBPF is being harnessed in Cilium to transform Kubernetes networking.

##### How did you become a Cilium committer?

When I joined Isovalent, I contributed a feature called [Local Redirect Policy](https://docs.cilium.io/en/stable/network/kubernetes/local-redirect-policy/). It is meant to address real world use cases where traffic needs to be redirected to node local pods using eBPF, one of the use cases being the node local DNS feature in Kubernetes.

While working on the feature, I made contributions to the Cilium load balancer datapath and the userspace interfacing with the Kubernetes control plane. I became familiar with the development workflow and collaborated with other Cilium maintainers. Contributing these and other features is how I became a maintainer and it's been a wonderful ride ever since.

##### What are you working on right now in Cilium?

I've been primarily working on the [Cilium socket-based load balancer](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/), also known as the kube-proxy replacement and Local Redirect Policy. Currently, my focus is on making the load-balancer more reliable in recovering from failures, and stabilizing both the features. I've also contributed a series of patches to the Linux kernel to enable BPF capability for socket termination which will address some of the limitations we face with the kube-proxy replacement load balancer.

##### What advice do you have for people just getting into the Cilium community?

Cilium is one of the most exciting and friendly open source projects out there. We have [public Slack channels](https://slack.cilium.io/) where Cilium users and contributors regularly discuss questions, bugs, and potential fixes. I encourage anyone interested to get involved.

We maintain a list of [good first issues](https://github.com/cilium/cilium/issues?q=is%3Aopen+is%3Aissue+label%3Agood-first-issue), designed to make it easy for you to start contributing to Cilium. Our [community meets](https://docs.cilium.io/en/latest/community/community/) every Wednesday, so if you have questions or need guidance, please ask in the Slack channels or meeting, and one of us will be happy to help. I'd like to emphasize that we've made significant improvements to the Cilium development process, making it easier than ever to get started with Cilium.

To summarize it all, the best way to get started is to deploy a cluster with Cilium, see how it works, play around, report bugs, contribute to features, or any changes.

##### What is one of your hobbies outside coding?

I love playing badminton and I'm always striving to improve my skills. I'm also an avid lover of nature, so I find great joy in hiking, which is both mentally stimulating and meditative. Nature photography is another passion of mine, and recently, I've taken up gardening as a hobby which has been a lot of fun.

To engage with the Cilium community and contribute to the project, follow Aditi's lead: join our [Slack](https://slack.cilium.io), check out a [good first issue](https://github.com/cilium/cilium/issues?q=is%3Aopen+is%3Aissue+label%3Agood-first-issue) you can work on, and explore our [community page](https://cilium.io/get-involved/) for more information.

#### Meet the Maintainer - Vlad Ungureanu

URL: https://cilium.io/blog/2023/09/18/meet-maintainer-vlad-ungureanu
Date: 2023-09-18
Categories: Community

_September 13th, 2023_

_Interviewed by: Bill Mulligan, Isovalent_

_Written by: Shedrack Akintayo, Isovalent_

##### How did you first get into technology?

From a young age, I was exposed to technology in various forms because both of my parents were involved in the technology scene. However, it's worth noting that they are not specifically in computer science. My father runs a company specializing in CCTV camera automation, while my mother teaches high school electronics. Growing up in such an environment naturally piqued my interest in technology, ultimately leading me to pursue a career in computer science.

##### How did you get into open source?

My journey into open source began during my second year of high school when I discovered [Google Code-in](https://codein.withgoogle.com/). This program was designed for high school students to collaborate with open source projects, providing an opportunity to work on them and gain exposure to the open source community.

During this time, I vividly remember working on fixing bugs for [ffmpeg](https://www.ffmpeg.org/) using a Linux computer. I also explored projects like valgrind and collaborated with the ffmpeg upstream maintainers. This experience continued for two years throughout high school, as the program itself spanned over three months. I thoroughly enjoyed contributing to open source during this period.

##### How did you first hear about eBPF and Cilium?

When I joined Palantir, I became part of the developer tools group, which had the responsibility of managing the internal CI/CD ecosystem and enhancing developer productivity.

Our primary focus was on optimizing the developer workflow at Palantir. To achieve this, we created a set of bots and decided to host them on a container scheduler, initially choosing [Nomad](https://www.nomadproject.io/).

When I relocated from the London office to the Seattle office, I discovered another group also working with containers and container schedulers. However, in the Seattle office, they used Kubernetes as their preferred container scheduler and Calico for networking.

Using Calico and troubleshooting networking issues was challenging which prompted us to explore alternative CNI options approximately two years after adopting Calico. We were particularly interested in moving away from running an overlay network in our Kubernetes clusters. It was during this search that I came across Cilium.

To gather more information about Cilium and its capabilities, I reached out to the Cilium community on [Slack](https://slack.cilium.io) and inquired about native routing. Through this process, I connected with Thomas Graf and although native routing was not available at the time, I suggested that we work on its development.

This marked the beginning of my involvement with Cilium and my relationship with the project has since evolved and grown stronger.

##### How did you become a maintainer of Cilium?

When we initially started working on adding support for native routing in Cilium, we were doing it all internally at Palantir. However, during this process, we encountered some bugs in our implementation. At the same time, Palantir was expanding its infrastructure from being exclusively on AWS to encompassing all major cloud providers.

This made me start to collaborate with Thomas and other team members to create the initial implementation of native routing support for Azure, which we successfully integrated into Cilium. I also focused on enhancing AWS native routing by improving the intelligence of the Cilium operator when assigning IP addresses.

To ensure continued contributions to Cilium, I dedicate several weeks every quarter to focus solely on the project. I take responsibility for specific areas based on my past work and contributions. During this dedicated time, I address various aspects such as reviewing newly reported bugs, identifying necessary additions to the documentation, and finding opportunities for overall project improvement.

My commitment involves actively contributing these updates and enhancements back to the Cilium project.

##### What are you working on right now in Cilium?

Currently, my main focus revolves around addressing internal bug fixes that will be subsequently ported back to the Cilium project. Specifically, my team and I at Palantir are working on improving the IPSec encryption part of the code because, given our compliance requirements, we need to ensure that all traffic inside a VPC is encrypted.

I am also conducting research to identify potential enhancements for OpenShift support in Cilium including managing Cilium through OpenShift.

##### What advice do you have for people just getting into the Cilium community?

Rather than focusing on Cilium specifically, I’ll give some advice to people who generally want to get involved in open source. One valuable piece of advice I received early on from a close colleague is to ask any questions you have when you join Slack, Discord, or any other communication platform used by the community. This encourages you to interact more and get involved with the community.

It's also highly beneficial to use the GitHub label "first good issue" to discover tasks suitable for new contributors as it serves as an excellent starting point.

##### What is one of your hobbies outside coding?

During winter, I enjoy skiing and in summer I find pleasure in hiking. Aside from these activities, I have a fondness for traveling to warm destinations, especially during the winter, to escape the cold. I also recently rekindled an old passion of mine – flying model airplanes, specifically RC airplanes. I used to do this during high school but had to put it on hold when I entered university.

To engage with the Cilium community and contribute to the project, follow Vlad's lead: join our [Slack](https://slack.cilium.io), check out a [good first issue](https://github.com/cilium/cilium/issues?q=is%3Aopen+is%3Aissue+label%3Agood-first-issue) you can work on, and explore our [community page](https://cilium.io/get-involved/) for more information.

#### How DB Schenker Migrated from Calico to Cilium

URL: https://cilium.io/blog/2023/09/07/db-schenker-migration-to-cilium
Date: 2023-09-07
Categories: Community, Technology

_Author: Amir Kheirkhahan, DB Schenker_

##### History behind the Migration to Cilium

In the past, the IT unit for the land transportation business of [DB Schenker](https://www.dbschenker.com/de-de) used Calico as a Container Network Interface ([CNI](https://www.cni.dev/)) for in-Kubernetes-Cluster communication, like pod-to-pod communication. Recently our team had the chance to participate in the 2023 [KubeCon](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/) in Amsterdam, where we learned a lot about [eBPF](https://ebpf.io/) and especially [Cilium](https://cilium.io/), which was an important driver for us to question our CNI strategy.

Although Calico offers eBPF as well and leverages some important features of it, Cilium was built natively on top of eBPF. In parallel we saw wide adoption in the market and a very feature rich tooling and ecosystem around Cilium, like [Tetragon](https://github.com/cilium/tetragon) for security observability and [Hubble](https://github.com/cilium/hubble) for network visibility. All of these factors together made the decision clear that we needed to migrate to Cilium to prepare our platform for the next steps in our cloud native journey.

##### Live migration Considerations

When planning  the migration of all our Kubernetes clusters, we needed to architect it to have minimal down time.

To understand how we could do this, we started a POC and tried to use the capabilities of [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) to have multiple network interfaces attached in parallel to a Pod. Unfortunately, this option didn’t work for us due to the complexity and risk of longer downtime.

However, Cilium recently introduced support for a [hybrid mode](https://docs.cilium.io/en/stable/installation/k8s-install-migration/#migration-via-dual-overlays) for migration where 2 different CNIs with different CIDR-ranges can be established across the cluster, so we found a good new motivation to give it a chance.

As already stated for the Land Business, it was important to keep the downtime of clusters minimal. Cilium has a brilliant feature which helped us to achieve this goal. It offers a [per-node configuration](https://docs.cilium.io/en/latest/configuration/per-node-config/) feature to easily roll out the Cilium configuration on a node by node basis using node-labels. That means, after you install Cilium and apply the configuration, all new nodes identified with a matching label will automatically get the specified configuration. This `CiliumNodeConfig` object has been available since [Cilium 1.13](https://isovalent.com/blog/post/cilium-release-113/#resilience-and-troubleshooting). We will go through this feature later during the migration steps.

The other aspect we needed to consider was how to replace the current Calico IP-in-IP tunneling protocol. Cilium offers UDP-based [encapsulation protocols](https://docs.cilium.io/en/stable/network/concepts/routing/) VXLAN or GENEVE and based on our needs we chose VXLAN where the entire layer 2 Ethernet frame is encapsulated inside a UDP packet and transmitted over the node network.

For IP address distribution among pods, you have different [options](https://docs.cilium.io/en/stable/network/concepts/ipam) to select in Cilium. Cluster-scope IPAM mode is the recommended way which we also chose. In this case, the Cilium operator manages the IP addresses by creating a CiliumNode object for each node and assigning the podCIDRs to them which is then read by Cilium agents running on the nodes. Of course, you are not allowed to select the same podCIDR used by Calico.

Another important consideration was the kube-proxy replacement feature. When a new service is created, the API server notifies all kube-proxy agents running on the worker nodes about this event. Then kube-proxy creates iptable rules to make sure each packet destined for a service is forwarded to the one of the correct backend pods. It is the same for a new pod when the endpoints in iptables are updated.  In the following migration steps, we kept Cilium’s kube-proxy replacement option disabled to reduce the complexity of the migration.

Finally, we also needed to think about our Kafka clusters. We run Kafka inside our Kubernetes environment by using the [strimzi](https://strimzi.io/) chart which includes very sensitive and important data. To ensure the stability of Kafka, we needed  to prevent losing the cluster state or running into a Kafka leader election issue.

To get a better understanding of our infrastructure, I’ll shortly point out some tooling and approaches which we use within DB Schenker. We use self-managed Kubernetes clusters on AWS where the worker and controller nodes are running independently on separate VMs. The worker nodes run on a mix of spot and on-demand instances. The base Images are built by Packer and Ansible and the whole infrastructure setup is managed by Terraform. To meet IT Security requirements, every day we throw away a set of our worker and controller nodes. We also have automated processes to rotate all our nodes. We do this by starting a new node and waiting until it is ready, then we drain the old node and move all workloads away from it before terminating the old node in our cloud provider as well as in Kubernetes. These steps are repeated for each of our worker pools.

For monitoring and measuring connectivity between nodes we leverage [Goldpinger](https://github.com/bloomberg/goldpinger) and for Kafka monitoring we leverage [strimzi canary](https://github.com/strimzi/strimzi-canary), which is a component of the [strimzi kafka operator.](https://github.com/strimzi/strimzi-kafka-operator)

The overview of the migration is as follows and we will go through these steps in the next section:

 Figure 1. Existing Nodes with Calico CNI

 Figure 2. After Cilium Installation on exisiting Nodes, Cilium has still no CNI ownership

 Figure 3. After Starting the rotation, Node C starts with both CNIs and Cilium has CNI ownership. Node C currently has Cilium configurations in place which is discussed later at step 1 of migration

 Figure 4. Draining the older worker node (here Node A) and re-schdule the pods on the new Node

 Figure 5. Node A is removed from the LoadBalancer so that it no longer receives traffic, and then terminated. Node A is fully replaced by Node C

 Figure 6. After the full rotation of both Nodes, both CNIs should be running on new nodes with Cilium leading

 Figure. 7. After removing Calico resources and clean up the labels, the new nodes have only Cilium installed on it

 Figure 8. At the end, only Cilium is running on all nodes.

##### Live migration

###### Step 1- Preparation

Before starting the migration, we need to make sure that we have proper back-ups of the workloads. In the Kubernetes context, the most important piece is to keep etcd backedup in a safe place, like an AWS S3 Bucket. In case the migration fails, we have a backup in place to perform a disaster recovery to the state before the migration.

The next thing that should be prepared is building the migration base image. Some of the important changes are:

- Add the label `"io.cilium.migration/cilium-default=true"` to all newly created nodes.  As soon as a new node is launched and has this label, the predefined Cilium config will be applied to this node. These predefined configs are declared in the [CiliumNodeConfig](https://docs.cilium.io/en/latest/configuration/per-node-config/#ciliumnodeconfig-objects) object.

- Add the Taint `"node.cilium.io/agent-not-ready=:NoSchedule.` to all newly created nodes. This prevents a race condition between the CNIs that are now running in parallel on the new nodes. Since Calico is sometimes faster in starting, the node gets marked as ready and newly scheduled pods on the node get IP addresses assigned by Calico. With the taint, Cilium is able to start up properly and then take over assignment of IPs.
 For more information refer to [official Cilium documentation.](https://docs.cilium.io/en/latest/installation/taints)

You should make your changes ready for deployment after Cilium installation. In our case, we built a new AMI with these changes and kept them ready to deploy. We used Terraform for deploying the infrastructure in our cloud environment and in this context, we rolled out the new images right after the Cilium installation. With that all new nodes have this label and taint and Cilium is the leading CNI on those nodes.

###### Step 2- Scale down the important applications like Strimzi Kafka

During a migration, if you aren’t careful with the stateful components, like Kafka, you could lose data. Therefore, we scale down Kafka and Zookeeper to ensure no messages are produced or consumed.

```bash
    kubectl scale statefulset -n kafka kafka-broker --replicas 0
    kubectl scale statefulset -n kafka kafka-zookeeper --replicas 0
```

This does cause a short downtime for our workloads but it helped us prevent issues with one of our most important components.

```bash
    kubectl drain --ignore-daemonsets --delete-emptydir-data --force --grace-period=900 -l label=gp-kafka
```

As soon as Kafka is completely stopped, we terminate the underlying node pool to be able to quickly bring up the nodes with Cilium installed and configured properly in the cluster.

###### Step 3- Cilium installation

Afterwards, we apply the changes to Terraform that contain the labels and taints that are required to tell Cilium to run as leading CNI on those nodes. It is important to mention that this step contains no new node deployment and is rather a preparation for the final installation and configuration of Cilium. Node deployment happens during the rotation.

Now it is the time to install Cilium where the configuration looks like this:

```yaml
# migration, for connectivity between calico and cilium

  bpf:

    hostLegacyRouting: true 

# Set cluster name. It would be interesting for cluster mesh

  cluster:

    id: 0

    name: prod-cluster

  cni:

# migration, disable CNI changes

    customConf: true 

# should be always false so it doesn't remove cilium

    uninstall: false 

# Assign a new CIDR for cilium

 ipam:

    operator:

      clusterPoolIPv4PodCIDRList:

        - 10.x.x.x/16          

# migration, otherwise all calico pods are restarted and we get downtime

  operator:

    unmanagedPodWatcher: 

      restart: false

# migration

  policyEnforcementMode: never 

# We stay with kubeproxy

  kubeProxyReplacement: disabled  

# Use Cilium suggested tunnel port 8473

  tunnel: vxlan

  tunnelPort: 8473

# (Optional) to resolve the sonobuoy test

  sessionAffinity: true     

# (Optional) Enable prometheus

  prometheus:

    enabled: true

    metrics: \~

  operator:

    prometheus:

      enabled: true

# (Optional) Enable Hubble

  hubble:

    relay:

      enabled: true

      prometheus:

        enabled: true

    ui:

      enabled: true

    metrics:

      enabled:

      - dns

      - tcp

      - httpV2

  endpointStatus:

    enabled: true

    status: "policy"
```

You must consider the proper CIDR ranges for Cilium and Calico and the previously allocated CIDR range should not be used. Migration steps are marked with _#migration_ comments and they will be changed at the end.

We can start Cilium’s installation at this step using the Helm deployment or another preferred approach. We installed the Cilium using the official Cilium Helm chart. The Cilium operator and agents are started in parallel with the existing Calico CNI, which will cause a short downtime for your applications (see below to remediate this). The downtime for us was less than 2 minutes and it went very smoothly.  Of course, it could differ in another setup and it should be tested during the POC. During Cilium installation if you look at the Goldpinger UI, it looks a bit terrifying:

 Figure 9. Goldpinger UI after Cilium installation

It turns out that this outage can be avoided altogether: when you install Cilium on a node, it creates a new network interface called `cilium\_host`. If Calico decides to use that interface as its default interface, Calico node routing will start failing. For this reason, Calico needs to be configured to ignore the `cilium\_host` interface. This could be done with the [skipInterface setting on the Tigera operator](https://docs.tigera.io/calico/latest/reference/configure-calico-node#skip-interfaceinterface-regex). See the Cilium docs [update](https://github.com/cilium/cilium/pull/27666) for more information.

###### Step 4- Deploy CiliumNodeConfig

After all inter-node communications are green and connectivity between the nodes has been recovered, you need to roll out the `CiliumNodeConfig`. This resource causes new nodes with the label `"io.cilium.migration/cilium-default=true"`  to receive Cilium configurations.

```bash
    cat <<EOF | kubectl apply --server-side -f -
    apiVersion: cilium.io/v2alpha1
    kind: CiliumNodeConfig
    metadata:
      namespace: kube-system
      name: cilium-default
    spec:
      nodeSelector:
        matchLabels:
          io.cilium.migration/cilium-default: "true"
      defaults:
        write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist
        custom-cni-conf: "false"
        cni-chaining-mode: "none"
        cni-exclusive: "true"
    EOF
```

**HINT**: It could happen that `_coredns_ pods` are restarted permanently after Cilium installation. The Cilium operator needs `_coredns_` for DNS resolution and it will try to restart the pods to be able to resolve the dns queries. Because we still have Calico as leader on the nodes on which `_coredns_` is running, the restarting goes to an infinite loop. To solve the issue, we added 3 new worker nodes for each AZ with new images built by us in step 1. After starting those nodes, Cilium takes over the CNI ownership and `_coredns_` pods are scheduled on those nodes. To avoid the `_cluster-autoscaler_` terminating these new nodes due to low resource usage, we tell the `_cluster-autoscaler_`to not consider these nodes as candidates for termination. For that, we add this annotation to `_coredns_ pods`:

```bash
    kubectl annotate pod -n kube-system -l k8s-app=kube-dns cluster-autoscaler.kubernetes.io/safe-to-evict=false
```

Now if you check the status of Cilium, you will still see that no pods are managed by Cilium, except `_coredns_` pods. This is expected. Calico is still running on the node and has CNI ownership. The nodes should be drained, cordoned, and the new nodes created as described  later.

###### Step 5- Scale up Kafka

Now you can scale up Kafka’s underlying worker pools to the previous numbers in your cloud provider. New nodes should have both CNIs running, but Cilium will take over leadership. After that you can start Kafka applications again. First start Zookeeper and wait for it to be ready and then start the Kafka brokers.

```bash
    kubectl scale statefulset -n kafka kafka-zookeeper --replicas 5
    kubectl scale statefulset -n kafka kafka-broker --replicas 5
```

###### Step 6- Rotate the nodes

Now it is the time to rotate all nodes with the new image which was built in step 1. The rotation will take some hours depending on the rotation mechanism as well as the number of nodes. At the end, each new node should have Cilium as a CNI in parallel to Calico and all pods must have a new IP address from the Cilium CIDR range.

To check how many nodes are not yet rotated the following command can be executed:

```bash
    kubectl get nodes -l '!io.cilium.migration/cilium-default'
```

###### Step 7- Cleanup 

We proceed with this step when the node rotation has been completed and Cilium is the main CNI on all nodes. You can check this with the _“cilium status”_ command or filter the pods based on the IP range.

To check which pods have still Calico IPs (when assuming Calico’s IP range is 192.168.0.0/16) run:

```bash
    kubectl get po -o wide -A | grep -e "192\.168\."
```

These pods got the Calico network after node rotation and are unmanaged by Cilium. They will be restarted once we perform the post migration step.. For us it happened for some daemonsets like goldpinger which were started before the Cilium agents could not be managed. If they are not critical pods, they can be restarted simply using the following command at your own responsibility:

```bash
    kubectl get po -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name -A --no-headers=true -o wide | grep -e "192\.168\." | awk '{print "-n "$1" "$2}' | xargs -L 1 -r kubectl delete pod
```

If you check the Cilium status, you will see all nodes are managed by the Cilium CNI. Now we can remove all Calico CRDs and other leftovers.

```bash
    kubectl delete crd bgpconfigurations.crd.projectcalico.org bgppeers.crd.projectcalico.org blockaffinities.crd.projectcalico.org caliconodestatuses.crd.projectcalico.org clusterinformations.crd.projectcalico.org felixconfigurations.crd.projectcalico.org globalnetworkpolicies.crd.projectcalico.org globalnetworksets.crd.projectcalico.org hostendpoints.crd.projectcalico.org ipamblocks.crd.projectcalico.org ipamconfigs.crd.projectcalico.org ipamhandles.crd.projectcalico.org ippools.crd.projectcalico.org ipreservations.crd.projectcalico.org kubecontrollersconfigurations.crd.projectcalico.org networkpolicies.crd.projectcalico.org networksets.crd.projectcalico.org
    kubectl delete -n kube-system deploy calico-kube-controllers
    kubectl delete -n kube-system ds/calico-node
    kubectl delete -n kube-system cm/calico-config
    kubectl delete -n kube-system ciliumnodeconfig cilium-default
```

###### Step 8- Post migration steps 

As the [official documentation](https://docs.cilium.io/en/stable/installation/k8s-install-migration/#post-migration) suggested, we do some post migration steps, like restarting the unmanaged pods by cilium operator, enforcing NetworkPolicy, and enabling eBPF host routing which could cause short interruption.

```yaml
   bpf:
     hostLegacyRouting: false
   cni:
     customConf: false
   operator:
     unmanagedPodWatcher:
       restart: true
   policyEnforcementMode: default
```

We now have all new nodes only with Cilium as the CNI. We removed Calico resources and Cilium migration labels from the code. As mentioned earlier, we have a daily automatic rotation approach for each worker pool and hence all nodes are replaced gradually and Cilium will be the only CNI available on those nodes.

##### Conclusion

This blog post walked through how we migrated from Calico to Cilium using the new `CiliumNodeConfig` feature. Overall, we found the process to be smooth and are excited for the possibilities that switching to Cilium will unlock for our platform.

#### Securing Constellation’s Kubernetes data in transit - network encryption with Cilium

URL: https://cilium.io/blog/2022/10/17/constellation-network-encryption
Date: 2023-07-20
Categories: Technology

_July 20th, 2023_
_Author: Leonard Cohnen & Moritz Eckert, Edgeless Systems_

[Constellation](https://github.com/edgelesssys/constellation) is a Kubernetes engine that aims to provide the best possible data security by shielding your entire Kubernetes cluster from the underlying cloud infrastructure. Everything inside a cluster is always encrypted, including at runtime in memory. For this, Constellation leverages a technology called [confidential computing](https://content.edgeless.systems/hubfs/Confidential%20Computing%20Whitepaper.pdf).

For network encryption, Constellation uses Cilium. This blog dives into why we chose Cilium, how we integrated it, and what we learned along the way. We’ll dive into the technical challenges and difficulties we encountered and how we overcame these. If you’re all into Kubernetes networking, eBPF, and Cilium, this should be for you. If not, don’t worry, we’ll highlight the important takeaways, and you’ll learn a thing or two about the inner workings of Cilium.

##### Encryption vs. access

Constellation must ensure all communication in a cluster is protected while still allowing access to the outside world. This includes the pod network carrying the application traffic and the Kubernetes communication itself. Simply put, all node-to-node traffic must be encrypted. However, for most applications, the nodes should still be reachable from the outside to support LoadBalancer and NodePort services.

Isolating the entire cluster in a VPN on the host level might be the first solution that comes to mind, and it solves the first part of encrypting all traffic. However, it makes communication with the outside cumbersome and impractical.

Thus, for Constellation, we opted to implement network encryption at the CNI level. Our CNI solution of choice is [Cilium](https://cilium.io/). It combines great performance with [transparent network encryption](https://docs.cilium.io/en/stable/gettingstarted/encryption/). Cilium supports both [IPSec](https://docs.cilium.io/en/stable/gettingstarted/encryption-ipsec/) and [WireGuard](https://docs.cilium.io/en/stable/gettingstarted/encryption-wireguard/). Essentially, it establishes a VPN network between nodes while maintaining features allowing access outside the cluster, such as service exposure and load balancing.

##### Implementing transparent network encryption

From the get go, our goals for the network encryption in Constellation were the following:

- A) Do not get into the user's way.
- B) Make the default secure.
- C) Minimize the chance of misconfiguration.

Cilium conveniently achieves all three. Unfortunately, with the latest release (1.13.4), there is still a [known issue](https://docs.cilium.io/en/v1.13/security/network/encryption/#known-issues-and-workarounds) that pod-to-pod is not immediately encrypted under certain conditions.

To address this issue, currently, the admin needs to disable all pod-to-Internet communication. While this is a viable workaround in certain scenarios it is not ours. Thus we began an investigation for a better solution roughly a year ago...

##### The consistency problem in IPCache

Both Cilium's identity-based policy enforcement and encryption rely on an [eBPF map](https://docs.kernel.org/bpf/maps.html) called _IPCache_ to reason about the source and destination identity of network packets. If a packet originates from a pod inside the pod network and the IPCache indicates that the destination is also a pod, then the packet is re-routed through the WireGuard interface. This interface then takes care of encapsulating and encrypting the packet and sending it to the destination pod's node. So far so good. However, as it turns out, the IPCache is only _eventually_ updated with new endpoints. If a newly created endpoint is not yet in the IPCache, network traffic to it is not encrypted. This is the core of the problem and our goal is to fix this.

###### Triggering the issue

To trigger and observe the issue, we analyzed the path from the new endpoint creation to the IPCache update. This path looks somewhat like this:

1. Endpoint is created
2. Cilium-agent creates a new CiliumEndpoint custom resource
3. Cilium-operator merges CiliumEndpoints with the same identity to one CiliumEndpointSlice
4. Cilium-agent watches and consumes all CiliumEndpointSlices
5. Cilium-agent writes a new entry in IPCache

By disabling the Cilium-operator in step 3, we are able to reliably trigger the issue, resulting in Cilium sending the corresponding traffic unencrypted over the wire. Note that the disabling is equivalent to adding a long delay into step 3. Our concrete test code can be found [here](https://github.com/cilium/cilium/blob/5da5882754569e118e713532b89dc4ca89ab76fd/test/k8s/datapath_configuration.go#L387).

###### Solution

Our approach is simple. We add a filter as an eBPF program to Cilium's packet routing stack called the ["datapath"](https://docs.cilium.io/en/stable/network/ebpf/intro/) that drops unencrypted network packets between pods. This raises two questions:

1. How do we identify pod-to-pod traffic?
2. How do we know if traffic is encrypted or not?

Our approach for (1) is to identify pods based on their IPv4 addresses. If both destination and source address are within the pod subnet (as specified by `PodCIDR`), we're dealing with pod-to-pod traffic. At least in most cases; more on that later.

For (2), we need to consider VXLAN and direct routing separately. All identified pod-to-pod traffic that is not (yet) covered by IPCache ends up unencrypted at the VXLAN interface. Analogously, for direct routing, all such traffic ends up unencrypted at eth0.

Correspondingly, for VXLAN, our filter is part of the `bpf_overlay` program, which is attached to the VXLAN network interface. For direct routing, our filter is part of the `bpf_host` program, which is attached to the host network interface, i.e., eth0.

Together, this reliably solves our problem. However, in cases where nodes and pods share a subnet, our approach from (1) also drops traffic to nodes, which is unintended and breaks functionality. This can happen for VXLAN on AWS and Azure, since the nodes have multiple IP addresses from the PodCIDR assigned. Those are used for internal health checks and to route traffic from one node to the pod of another node. This problem does not occur for native routing on GCP.

To address this, for VXLAN, we extend our filter to identify nodes via the IPCache map. If, according to the map, the destination is a node, we don't drop the traffic. This extension has one caveat: If a pod is assigned the IP address of a recently shut down node and this change isn't reflected in IPCache yet, traffic to that pod will temporarily still not be encrypted.

###### Implementation

The Cilium was very welcoming and helped us with the implementation of the strict mode described above. Specifically, the Cilium maintainers suggested a location [in the datapath](https://docs.cilium.io/en/v1.13/network/ebpf/lifeofapacket/#life-of-a-packet) to add our [filter](https://github.com/cilium/cilium/pull/21856/files#diff-f1851842acdc4e195dcd1b102c3c2597c293c1bfd04dc7e1bca8e02050102915R110). After some trial and error, we finally found the correct spots in [`bpf/bpf_host.c`](https://github.com/cilium/cilium/pull/21856/files#diff-ff0cc41af3fc733a268ec48f09df41cc2bf321f9191d8df1722b6d195e88e193R1495) and [`bpf/overlay. c`](https://github.com/cilium/cilium/pull/21856/files#diff-01b3b17d448a4cf12c8ce37729576da934f2c3ed8801216b33d4e13a39299193R684), which made our code pass the test. For the eBPF connoisseurs, we’ve listed the actual packet filter code below.

```c
/* strict_allow checks whether the packet is allowed to pass through the strict mode. */
static __always_inline bool
strict_allow(struct __ctx_buff *ctx) {
 struct remote_endpoint_info __maybe_unused *dest_info, __maybe_unused *src_info;
 bool __maybe_unused in_strict_cidr = false;
 void *data, *data_end;
#ifdef ENABLE_IPV4
 struct iphdr *ip4;
#endif
 __u16 proto = 0;

 if (!validate_ethertype(ctx, &proto))
  return true;

 switch (proto) {
#ifdef ENABLE_IPV4
 case bpf_htons(ETH_P_IP):
  if (!revalidate_data(ctx, &data, &data_end, &ip4))
   return true;

  /* Allow traffic that is sent from the node:
   * (1) When encapsulation is used and the destination is a remote pod.
   * (2) When the destination is a remote-node.
   */
  if (ip4->saddr == IPV4_GATEWAY || ip4->saddr == IPV4_ENCRYPT_IFACE)
   return true;

  in_strict_cidr = ipv4_is_in_subnet(ip4->daddr,
         STRICT_IPV4_NET,
         STRICT_IPV4_NET_SIZE);
  in_strict_cidr &= ipv4_is_in_subnet(ip4->saddr,
          STRICT_IPV4_NET,
          STRICT_IPV4_NET_SIZE);

#if defined(TUNNEL_MODE) || defined(STRICT_IPV4_OVERLAPPING_CIDR)
  /* Allow pod to remote-node communication */
  dest_info = lookup_ip4_remote_endpoint(ip4->daddr, 0);
  if (dest_info && dest_info->sec_identity &&
      identity_is_node(dest_info->sec_identity))
   return true;
#endif /* TUNNEL_MODE || STRICT_IPV4_OVERLAPPING_CIDR */
  return !in_strict_cidr;
#endif /* ENABLE_IPV4 */
 default:
  return true;
 }
}
```

Description: Basic eBPF IPv4 packet filter based on CIDR.

The full implementation was [merged into Cilium](https://github.com/cilium/cilium/pull/21856) and will be released with v1.15. It is already used and configured automatically by Constellation.

###### What's next

With the [v1.14 release](<[https://github.com/cilium/cilium/milestone/37](https://github.com/cilium/cilium/milestone/42)>) Cilium will also introduce [node-to-node](https://github.com/cilium/cilium/pull/19401) encryption. For Constellation, our plan is to switch from pod-to-pod encryption + strict mode to node-to-node encryption + strict mode in the coming months. The same benefits of our filter apply to node-to-node encryption. Dropping unencrypted packets between nodes. However, it removes the need to explicitly identify pod-to-pod traffic.

##### Conclusion

We’ve seen how Cilium helps Constellation protect data in transit as one of three pillars of always encrypted Confidential Kubernetes. The eventually consistent routing information turned out to be a problem for guarantees about encrypting all workload traffic. With the help of the community and maintainers, we were able to lift Cilium’s capabilities and flexibility to implement a strict mode that addressed the problem.

Let us know if you found our deep dive insightful and if you want to learn more about Constellation and Cilium. For example, protecting the Kubernetes API-Server-to-Node communication or integrating Cilium’s node-to-node encryption mode into Constellation. In the meantime, you can find everything about Constellation on [GitHub](https://github.com/edgelesssys/constellation) and in our [docs](https://docs.edgeless.systems/constellation).

Thanks to [@benschlueter](https://github.com/benschlueter) for the joint work. Thanks to [@pchaigno](https://github.com/pchaigno), [@brb](https://github.com/brb), and [@gandro](https://github.com/gandro) from Cilium for the helpful hints and discussions.
Thanks to [@xmulligan](https://github.com/xmulligan) for reviewing and helping with the blog post.

#### Cilium User Story: Zero Trust Networking at Scale (20k+ VCPUs, 100+ Dev Teams)

URL: https://cilium.io/blog/2023/05/04/telecommunications-user-story
Date: 2023-05-04
Categories: Community

_May 4th, 2023_

_Author: Shedrack Akintayo, Isovalent_

_This User Story comes from a company in the telecommunications industry._

Telecommunications involves connecting people across the world. In order to achieve this efficiently, telecommunications companies need highly scalable and secure systems to handle user traffic. Based on new business requirements, one telecommunications company needed to implement a zero trust networking model with default-deny policies and limited and audited egress.

They have a large multi-tenanted Kubernetes platform with Cilium as the CNI but needed to update their security model fundamentally because of the new business mandates. They required a performant and secure solution to help them implement zero-trust network security and landed on leveraging additional features in Cilium as the answer.

##### Leveraging More Features in Cilium to Meet New Business Requirements

The platform team manages a large, highly multi-tenanted Kubernetes platform with >20,000 vCPUs across seven production regions and >100 tenant teams of developers pushing code to the platform. As a platform provider in the telecommunications industry, they always needed to ensure that their 100+ tenant teams could all work on the same platform without blocking each other while providing the teams with good guardrails required for security and compliance.

As their platform grew, a new business-wide security requirement mandated full logging of all outgoing connections and a default-deny egress policy (only permitting egress connections to specifically enumerated destinations). Before this change was requested, they had already deployed Cilium to their platform mainly as a more [highly performant CNI](https://cilium.io/blog/2021/05/11/cni-benchmark/) layer and to provide their platform with network isolation as a feature for specific sensitive workloads.

In other business areas, using cloud provider-level solutions or integrating traditional network policies to manage and secure applications and their dependencies is common. However, that approach didn't work because they needed more granular control over application access to dependencies. A uniform layer-3-driven blocking strategy would also not scale to the diversity of workloads on their platform. The team also wanted to ensure that only certain applications could access specific dependencies rather than applying a blanket policy across the entire platform. Finally, they also needed to balance security against introducing more friction for developers that would slow them down.

To achieve this level of control and ensure developer happiness, they had to take a different approach with Cilium at the forefront.

_“We were certain that Cilium would be the key to successfully implementing the requirements in the context of our platform – and this proved to be the case.”_ - A member of their Infrastructure team.

##### Leveraging Cilium to Satisfy Business Zero Trust Networking Requirements

For scalability reasons and developer experience in the modern dynamic internet environment, they felt it necessary to allow network policies to be specified based on (stable) hostnames rather than (potentially highly volatile) IP ranges. They were also running a live Production environment where they needed to ensure non-disruptive rollout and adoption of capabilities. Finally, they were committed to providing each platform tenant with the ability to manage their network policies with a compliance approval workflow in line with their principles of developer empowerment.

The basis of the solution was implementing FQDN-based rulesets. With FQDNs, they added flexible blocklisting and a default deny egress policy with granularity at the team and namespace level.

They also utilize Isovalent’s Hubble enterprise daemon set to export connection events from their cluster nodes in a JSON format that is compatible with their security team’s data ingestion pipelines, providing richer information about the identity of the workload initiating the connection and DNS information about the target host.

With the visibility in place, they next turned to the issue of ensuring that their tenants could each create their individual allowlists in a self-service way that would still be compatible with the overall corporate policies around approval for egress destinations. To manage operational risk, they were clear that egress traffic should be directly controlled by CiliumNetworkPolicies (CNPs), giving them a common language between the platform team, their tenants, and their vendor (Isovalent) in the case of any problems arising.

To ensure each platform tenant could manage their network policies with a compliance approval workflow, they created a two-tier system of CiliumNetworkPolicies (CNPs) and Custom Resource Definitions (CRDs) with a PR-based workflow for tenants to suggest and cybersecurity teams to approve changes to the approval list on a case by case basis. The CNPs could be deployed to the clusters by tenants using the same CD pipelines they used to deploy their applications. They used the Gatekeeper policy engine to enforce congruence between the CNPs created in the cluster and the approval records, rejecting any noncompliant change to a CNP.

Through the design and operation of this system, they were able to satisfy business requirements around the control and traceability of updates to the egress approval list while also allowing tenants to have direct supervision and responsibility over the restrictions that affect their applications.

_“The use of Cilium proved a key element of our ability to successfully deliver these requirements due to the level of control and visibility it provides over traffic flows on our platform and the native integration with the Kubernetes API on which our platform is based.”_

##### Empowering Developers and Meeting Security Requirements

Cilium was a crucial component in the platform team’s ability to deliver the required outcomes effectively. The result is a system that empowers developers and enables the use of cutting-edge technology on the platform while ensuring organizational security mandates are fully implemented and supported.

_“Cilium, and specifically its FQDN rule features, also enabled us to adhere to our platform principles of developer empowerment while also ensuring our business stakeholders were satisfied with the security stance of the platform as a whole.
We would like our experience in this area to serve as encouragement and an example to teams seeking to integrate best-in-class OSS tools from the Kubernetes ecosystem with their own organization-specific requirements.”_ - A member of their Infrastructure team.

#### Cilium Talks at KubeCon EU 2023

URL: https://cilium.io/blog/2023/04/03/cilium-kubecon-eu-2023-talks
Date: 2023-04-03
Categories: Community

_Author: Bill Mulligan, Isovalent_

KubeCon + CloudNativeCon EU 2023 is right around the corner and it is promising to be a hive of activity around Cilium (all of those tulips won’t pollinate themselves). The first ever [CiliumCon](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/co-located-events/ciliumcon/) is happening on the Tuesday morning, with talks from users like Bloomberg, New York Times, and Robinhood. Tickets are already completely sold out, but there will be a livestream. You will also be able to join the conversation in the Cilium slack by jumping in the [#ciliumcon channel](https://cilium.slack.com/archives/C0510801E4V).

Head to the [project meeting](https://docs.google.com/document/d/18DjEz8MzvhWkz7Y7ghWYW3Km_wCmmaZ4Z4iurwFgnYk/edit#) on Tuesday afternoon to meet the maintainers and discuss the future of the project.

If that action-packed first day hasn’t already tired you out, there will be 7 other talks about Cilium to catch when the main conference kicks off on Wednesday!

The talks cover networking (and networking gone wrong), deep dives into security observability with Tetragon, and end user stories from people putting Cilium into production. If that isn’t enough, there will also be a Cilium booth in the Project Pavillion where you can get all of your questions answered. Let’s dive into each of the talks below.

[The Cilium Story - Why We Created Cilium - Thomas Graf, Isovalent](https://sched.co/1Jo6C)

Tuesday, April 18 • 09:25 - 09:45

Cilium is well known today but why did we create it? This talk takes us back to the origins of Cilium. Together, we explore the adventure that led to the creation of the first few lines of Cilium source code. What was the vision? How close was it to what Cilium has become today? As we make our way exploring the many Cilium milestones, we look back to Cilium's roots in Switzerland, the many surprises on the way, and the major milestones that put Cilium on the trajectory it is today.

[Sponsored Keynote: Cilium on Azure: Most Scalable and Performant Implementation in the Cloud- Deepak Bansal, Microsoft](https://sched.co/1Jo69)

Tuesday, April 18 • 09:15 - 09:20

Microsoft is thrilled to announce investments we are making to fuse power of Azure Virtual Network stack with Cilium’s eBPF capabilities in close partnership with Isovalent. Cilium in Azure will provide highly efficient load-balancing, extensive network security features, and rich observability integrated well along with robust and scalable Azure CNI IP management and native Azure networking routing, to give the most performant and best-in-class container networking platform available in the cloud.

[Some Assembly Required: IKEA Private Cloud, Cloud Native Networking - Karsten Nielsen, IKEA Retail](https://sched.co/1Jo6F)

Tuesday, April 18 • 09:50 - 10:15

Like any piece of IKEA furniture, cloud-native platforms require some assembly.

Anyone who has built either knows that the most challenging part is connecting all the pieces together correctly. Cilium has become a key piece connecting the vast cloud native estate of IKEA Retail (Ingka Group) encompassing both public and private cloud Kubernetes clusters.

The top three cloud-native networking challenges for IKEA Retail (Ingka Group) were to reduce operational and hardware costs by moving Load Balancing capabilities from dedicated hardware into Kubernetes, scaling and managing clusters to deal with IPv4 exhaustion. Having a predictable IP addresses for outbound traffic.

This session walks through how IKEA Retail (Ingka Group) solved each of these challenges in production with Cilium. Ingress from Cilium Service Mesh and the ability to setup Load Balancers to eliminated the need for expensive hardware load balancers. LB IPAM with BGP provided flexibility in allocating and advertising Service IP's to the physical network. Finally, Egress Gateway predictable connectivity to legacy workloads.

We will examine the IKEA Retail (Ingka Group) business and technical requirements for delivering reliable and scalable applications and how Cilium supported IKEA Retail (Ingka Group) to meet their critical goals, with only some assembly required.

[Designing and Securing a Multi-Tenant Runtime Environment at the New York Times - Ahmed Bebars, The New York Times](https://sched.co/1Jo6I)

Tuesday, April 18 • 10:20 - 10:45

Networking choice matters, especially when building multi-tenant Kubernetes clusters to host thousands of containers. In this session, Ahmed will walk you through the New York Times’s networking Journey from initial drawing to Day 2 and beyond operations., The New York Times team was building multi-tenant Kubernetes clusters to complement the Internal Developer platform and needed networking that could securely scale up to 100+ Nodes. Cilium was a critical choice to increase networking performance while providing identity and application-aware security and visibility for cloud native workloads running on EKS. It was an exciting journey but, of course, not an incident-free one. We faced issues that led us to dive into the Cilium codebase to understand and debug critical production issues, with the fix eventually being contributed to Ciliumupstream. The audience will learn what requirements led us to choose Cilium, the design tradeoffs and security choices we need to make, and the obstacles we faced before and beyond production, and will walk away with a good understanding of how to save time on their own Cilium Journey.

[Cilium in Practice: Building Data Sandboxes at Bloomberg - Anne Zepecki & Sritej Attaluri, Bloomberg LP](https://sched.co/1Jo6L)

Tuesday, April 18 • 10:50 - 11:15

Quant developers and data scientists utilize various data sources in their research workflows. Bloomberg’s quant analytics platform, BQuant Enterprise, is built with Kubernetes, integrates with major public clouds, and allows researchers to seamlessly mix Bloomberg’s comprehensive data sets alongside their own data as they test and deploy new investment strategies. Data vendors typically sell data at different prices tied to different usage terms and limitations. In providing experimentation environments where researchers can build data pipelines without full data license fees, Bloomberg leverages Cilium to construct data sandboxes that restrict users from distributing data outside the sandbox. In this talk, we’ll discuss how in-cluster L7 Cilium policies enable us to create isolated research environments. We hope you will walk away from this talk with an understanding of a unique application of Cilium for an enterprise product and how Cilium could be a valuable solution for you, too!

[Tales from an eBPF Program’s Murder Mystery - Hemanth Malla & Guillaume Fournier, Datadog](https://sched.co/1Jo6O)

Tuesday, April 18 • 11:25 - 11:50

On a beautiful fall evening, when Cilium was busy processing packets, some of its eBPF programs were murdered, leading to connectivity issues across the cluster. Join detectives Hemanth & Guillaume as they narrate tales from a multi-week investigation involving Cilium and eBPF. As eBPF continues to grow in popularity, this talk serves as a cautionary tale with pitfalls to avoid for developers who intend to build systems that need to co-exist with other eBPF-based components. This talk will include a quick overview of the Linux traffic control subsystem and how you can program it with eBPF. You’ll then get a walkthrough of how the speakers identified the root cause of the incident down to a race condition and the function guilty of the murder. You’ll learn to take advantage of bpftrace and its tools for your debugging needs. Lastly, the talk will conclude by discussing some of the ongoing work of eBPF program ownership, how it impacts security use cases and why solving for it is crucial.

[More Churn No Problem: Lessons Learned Running Cilium in Production - Lu Zhang & Madhu C.S., Robinhood Markets](https://sched.co/1Jo6R)

Tuesday, April 18 • 11:55 - 12:20

In this talk, we’ll share Robinhood’s war stories from running Cilium in a high-churn near-production environment, how we have overcome challenges by better understanding and tuning Cilium, and why we now live happily-ever-after™. Robinhood has been running Cilium for over a year in the environment that hosts the entire company’s integration tests and personal development namespaces. The environment is treated with the same seriousness and response SLA as production because it’s critical to our entire company’s engineering and product development. Due to the nature of the workloads, it is a high churn environment and brings many interesting challenges. We moved from the traditional VPC-based CNI model to Cilium overlay networking to improve pod density, scalability and cost efficiency. While we were able to achieve a significantly higher pod density (~6x) and cost efficiency, this has come with its own set of challenges. We ran into Cilium rate limiting challenges, identity garbage collection bugs, loss of internet egress connectivity for pods, bottlenecks in our environment, and many others. The audience will walk away with an understanding of what it takes to run Cilium in production and some of the edge cases they may encounter.

[eBPF for Observability: The Good, the Bad, and the Ugly - Anna Kapuscinska, Isovalent](https://sched.co/1Jo8W)

Tuesday, April 18 • 16:50 - 17:15

eBPF’s promise of zero-instrumentation observability with low performance overhead sounds like a dream and is driving wide adoption in observability tooling. At the same time, as a relatively low-level technology it is often poorly understood or even feared. To bridge this gap, we will dive into how eBPF serves the observability field. Anna will start by introducing eBPF features that let us achieve observability with no code changes. Next, she will examine at a few distinct eBPF use cases in open source tools: _ getting insights from granular system events (e.g. Tetragon) _ continuous profiling (e.g. Pixie) \* distributed tracing auto-instrumentation (OpenTelemetry) Peeking at both eBPF code and user interfaces will help us understand the potential of the technology. Finally, Anna will discuss the limits of using eBPF for observability and how it plays together with other frameworks. The audience will walk away understanding what works and what doesn’t when using eBPF for observability.

[Cilium Project Meeting](https://sched.co/1JWRk)

Tuesday, April 18 • 15:30 - 17:30

Meet the maintainers and learn where Cilium is heading. [Sign up](https://docs.google.com/document/d/18DjEz8MzvhWkz7Y7ghWYW3Km_wCmmaZ4Z4iurwFgnYk/edit#) for what you are interested in!

[Tutorial: Getting Familiar with Security Observability Using eBPF and Cilium Tetragon - Tracy P Holmes & Duffie Cooley, Isovalent](https://sched.co/1HyZg)

Wednesday, April 19 • 16:30 - 18:00

There are many people who are interested in observability but don't understand what data matters or even where to start. There are others who do understand these things, yet have no idea how to spot certain activities (malicious or otherwise!) This is where Security Observability comes into play. Security Observability in general is about providing more context into events involving an incident. However, researching those events does not have to be confusing or difficult. In this session, we will help overcome these doubts by learning more about a good kind of S.O.R.E.ness - the Security Observability and Runtime Enforcement kind! In four steps we will: 1. Introduce the fundamentals of Cilium Tetragon and the basics of Security Observability 2. Discuss the layers where Tetragon can extract data from and provide enforcement 3. Determine exactly what activities to care about and to monitor, and how to spot those activities 4. Walk through a brief deep dive into network connections and the associated events. The audience will walk away with a better understanding of the types of data and activity that should be monitored in order to prevent malicious events, and the ability to detect a container escape step-by-step.

[Cilium Updates, News, Roadmap, and in the Wild - Liz Rice, Isovalent; Andy Allred, EfiCode; Richard Hartmann, Grafana Labs](https://sched.co/1HyTU)

Wednesday, April 19 • 17:25 - 18:00

Welcome to Cilium! In this session you'll get an update on how Cilium has been progressing as a project and on the road towards graduation. You'll hear about the latest developments and future roadmap. We will cover how Cilium is bringin eBPF powered data to the world of observability and why Cilium has become the CNI of choice in the wild. In this session you'll hear from Cilium contributors and users Isovalent, Grafana Labs, and Eficode.

[Adopting Network Policies in Highly Secure Environments - Raymond de Jong, Isovalent](https://sched.co/1HyYr)

Wednesday, April 19 • 17:25 - 18:00

In the world of distributed computing, everything goes over the network, but not everything should be public. Unfortunately, Kubernetes networking is open by default and it is up to you to adopt network policies to secure it. Using our knowledge of implementing network policies in complex regulated environments, we will introduce the fundamentals of Cilium Network Policies and the basics of application-aware and Identity-based Security. With these building blocks in place, we will compare a default-allow with a default-deny policy and how a risk-based approach helps you focus on securing the most sensitive workloads first. We will then discuss various exposure types and strategies for securing your workloads. Applying this theoretical knowledge to the real world, we will explore how observability tools Cilium, Hubble, and Grafana provide you with Network Policy superpowers, like showing how ingress and egress connections are visualized, enabling you to configure the Network Policies using the Network Policy editor. Finally, we will discuss how Network Policy Guardrails allow for keeping control while granting teams self-service management of Network Policies. The audience will learn how to secure their network effectively and efficiently, even for highly sensitive workloads.

[The Next Log4jshell?! Preparing for CVEs with eBPF! - Natalia Reka Ivanko & John Fastabend, Isovalent](https://sched.co/1Hybi)

Friday, April 21 • 11:55 - 12:30

Log4jshell, which has been considered the biggest 0 day vulnerability of this decade, is still affecting thousands of servers worldwide. If you were affected, would it have been any different if you had used eBPF? Could you observe the malicious external connection, the JNDI lookup, the Java class download, or the remote code execution? Or even better, could you prevent it? Since eBPF provides us with a unique visibility directly into any Kubernetes workload on a single shared kernel - the answer is yes. This talk will take Log4jshell as a learning lesson and show you how it could have been detected and blocked in real time inside the kernel using eBPF. We will walk you through how open source eBPF based tools can give full network and process-level visibility to detect and prevent Log4jshell and your next CVE. We’ll finish by showcasing how Security Teams can easily put these tools in place to protect their critical Kubernetes environment and by giving Security best practices on how to prepare for their next CVE with eBPF.

[Surviving Day 2 - How to Troubleshoot Kubernetes Networking - Thomas Graf, Isovalent](https://sched.co/1HydA)

Friday, April 21 • 11:55 - 12:30

Kubernetes is widely deployed. Kubernetes networking is at the core of every platform and then there is DNS. In this talk, we will dive into the inner workings of Kubernetes networking, learn how to troubleshoot it, and most importantly, describe how to monitor it properly to prevent incidents in the first place. In this session, we will walk through the essential toolbox for efficient networking troubleshooting and then set up preventive measures together: - Understanding the Kubernetes networking model - How to troubleshoot and resolve DNS errors - Debugging Kubernetes Services & Ingress and increasing resiliency - Locating the source of networking errors - Is it an app, CNI, or underlying network problem? - Troubleshooting Kubernetes Network Policy drops - How to set up metrics dashboards and alerting to prevent network incidents All troubleshooting steps will be demonstrated in a live Kubernetes cluster and all steps will be found in the presentation slides and on GitHub.

[Effortless Open Source Observability with Cilium, Prometheus and Grafana - LGTM! - Raymond de Jong & Anna Kapuścińska, Isovalent](https://sched.co/1HyYo)

Friday, April 21 • 14:55 - 15:30

Updating applications to include observability can be almost impossible, but the other option is not know if your application is even working. Enter Cilium which leverages eBPF to provide observability data with Prometheus metrics for your applications without having to modify the application itself. In this session we will explain how Cilium powered with Hubble and the Grafana LGTM stack is able to show Service to Service communication, monitor Golden Signals, detect transient network layer issues and identifies problematic API request with transparent tracing. Using a demo application we will demonstrate performance and metrics for that application and how the metrics change with increasing request volumes. We will show how metrics change when a new configuration of our application introduces error rates and increases request duration. Finally, we will show how tracing headers for the application can be exported with Hubble HTTP metrics as Exemplars to link metrics to traces in Grafana, monitoring each request and its duration using Tempo. The audience will walk away with knowledge on how to monitor service connectivity and collect tracing data and golden metrics using standard Prometheus, Grafana, and OpenTelemetry exported from Cilium and eBPF.

[Keeping It Simple: Cilium Networking for Multicloud Kubernetes - Liz Rice, Isovalent](https://sched.co/1Hyaz)

Friday, April 21 • 16:55 - 17:30

Kubernetes promises that we can run containerized workloads in any cloud, and according to a recent article InfoWorld “2023 may [finally] be the year of multicloud Kubernetes”. For this to happen, we need seamless connectivity between workloads across clusters, regardless of the cloud they’re running on. From the perspective of a developer, shouldn’t connectivity across clouds be as simple as connectivity within a cluster? This talk explores - and demonstrates - how Cilium and its ClusterMesh feature can take care of many aspects of connectivity across multiple clusters in a cloud-agnostic way. It will show how just a few additional lines of YAML in your existing Kubernetes resources can enable: - Connectivity between services spread across clouds - Load balancing requests across backends in multiple clusters - Connectivity between Kubernetes and legacy workloads - Mutually-authenticated, encrypted connections between services - Multi-cluster network policies The talk will also discuss some of the challenges related to IP address management, scale, and observability of multi-cluster networks, and how Cilium can help.

Mention this blog post to me at the conference and I’ll even have a special prize. See you there!

#### Going from Packet Where Aren’t You to pwru

URL: https://cilium.io/blog/2023/03/22/packet-where-are-you
Date: 2023-03-22
Categories: Community

_February 8th, 2023_
_Authors: Martynas Pumputis and Bill Mulligan, Isovalent_

_This story comes from an open source pwru user_

[pwru](https://github.com/cilium/pwru) is an [eBPF](https://ebpf.io/)-based tool for tracing network packets in the Linux kernel with advanced filtering capabilities. It allows fine-grained introspection of kernel state to facilitate debugging network connectivity issues that are difficult to impossible to solve with traditional tools like iptables TRACE or tcpdump. In this blog, I’ll walk through how pwru solved a networking mystery without having to know everything beforehand.

##### Packet Where Aren’t You

We run Kubernetes on Linux hosts where the hosts are themselves BGP routers using BGP unnumbered. Our NICs therefore have IPv6 link local addresses only, but we have routing table entries for IPv4 routes that specify these NICs as next-hops and specify explicit IPv4 source addresses.

This set up works fine for the most part, however we started to have connections that went through kube-proxy getting dropped. The routing would just start to mysteriously fail for a while before starting to work again. Tracing the packets with iptables TRACE, I could see kube-proxy was doing some DNAT and finally a MASQUERADE and then output. The output looked fine, yet the packet never left the box. It was evidently getting dropped somewhere after the iptables stack and before it actually got to the NIC driver.

The natural tendency is to focus on what you can trace, so I was pretty myopic on the idea that iptables was doing something that I wasn't seeing.

_Figure 1. Packet path on the affected system_

##### Packet Where Are You

Once I discovered [pwru](https://github.com/cilium/pwru), I just started it up with a filter to match the packets I was losing and got traces of a flow that worked and one that didn't. With that in hand, it was pretty easy to just diff what happened in the cases that worked versus the ones that did not.

It turned out the problem was that IP Masquerade in Linux dynamically chooses the source address to use based on the route the packet is going out. I would have thought IP Masquerade would apply an IPv4 address from the route being used. It does not.

IP Masquerade only looked for an address on the physical interface the packet was leaving on. If there is no IPv4 address attached to the next-hop interface, it just picks an IPv4 address at random from the other interfaces that do have an IPv4 address (?!?). In this case, the source address on the next hop route was attached to a dummy interface.

If IP Masquerade unwittingly picked the right interface to borrow an address from, everything worked. However, if it picked a different one, then it would send the packet with a source address that was on a different interface and the Linux AppArmor and/or reverse path filter code would silently intercept and drop the packet.

To make this more nefarious, when packets are dropped in this way iptables says nothing about it. You can see with the trace that the masquerade rule was applied and then nothing when it is dropped because the following stages never happen. It is just gone. I didn't know there was more code in that path that could decide to drop packets outside of iptables.

_Figure 2. The same packet path from the pwru point of view_

##### pwru

That’s the beautiful thing about pwru. Network flow debugging in Linux historically required you to know what you don't know. Because there is no end-to-end-what-happened-to-it facility, you have to know where the packet is getting lost to trace the right thing. pwru provides that end-to-end knowledge and lets you find the culprit even when you didn't know the culprit existed in the first place, which was the case here for me.

pwru is an open source project under the Cilium organization. You can [download it from Github](https://github.com/cilium/pwru) and get started today. If you have any questions, make sure to drop into the #pwru channel on the [Cilium slack](https://slack.cilium.io).

#### Performance Testing Cilium Ingress at Hetzner Cloud

URL: https://cilium.io/blog/2023/01/25/hetzner-performance-testing
Date: 2023-01-25
Categories: Community

_January 25th, 2023_
_Author: Pol Arroyo, DevOps Engineer @ Hetzner Cloud_

Hetzner Cloud is a German public cloud provider that supplies a variety of cloud products like load balancers, virtual machines, storage, and firewalls. Hetzner was looking to upgrade the company's current ingress architecture and was looking to upgrade to a Kubernetes native solution that also offers ECMP, firewall, IPv6, and DSR capabilities. Cilium was the best option for their requirements, but we wanted to do performance benchmarking to see what impact it would have on their systems. This blog walks through how the tests were performed and the outcome in terms of RPS, throughput, and CPU usage, ultimately finding that Cilium meets our new requirements and provides a great performance improvement.

##### **Architecture**

At Hetzner, we always aim to offer modern and scalable services to our customers and in order to do so, our internal infrastructure needs to follow the same approach. After a recent infrastructure evaluation, we decided to update our ingress architecture. Since we are a cloud provider, we provide our core internal infrastructure on bare metal servers.

Our current ingress architecture consists of 2 bare metal systems with 10g NICs acting as gateways for most of our core internal infrastructure. Currently we run an active-passive load-balancer with BGP and keepalived setup acting as layer 4 ingress layer. Even though this architecture served us well over the years, it was starting to run into its limits.

For our new architecture we decided on the following requirements in order of priority:

- Kubernetes native solution
- ECMP
- Firewall
- IPv6
- DSR

At Hetzner, we are big fans and supporters of Kubernetes, and over the years we have migrated more and more of our core internal systems to it. Currently, above 90% of our Hetzner cloud control plane infrastructure runs on top of Kubernetes. For this reason, one of our top requirement priorities was to have a Kubernetes native solution for our ingress layer.

An ECMP ingress layer will allow us to horizontally scale our ingress capacity in the future. This is also a priority since the current infrastructure can only scale vertically.

Firewall is important because of its close connection to security. We need to be able to firewall certain endpoints to only be accessible to certain identities. Being able to do this natively in Kubernetes will allow us to remove some of the complexity in our current firewall pipeline release setup.

IPv6 usage has been steadily increasing over the years. We want our new ingress layer to have native support for it.

Last but not least, DSR would help us to reduce egress pressure in the L4 ingress layer systems since the L7 systems can then reply directly to the client.

Having listed all these requirements, Cilium seemed to be the perfect match for us since it has support for all the features needed.

##### **Testing**

For testing, we deployed diverse Cilium configurations in a Kubernetes cluster to benchmark RPS, throughput, and check the CPU usage as a metric to compare the tests.

All tests were performed on the same hardware in the following setup:

Benchmark systems:

- 3x Hetzner AX101 AMD Ryzen™ 9 5950X 128 GB DDR4 ECC

Target systems (Kubernetes cluster):

- 2x AX41 AMD Ryzen™ 5 3600 64 GB DDR4 as L4LB ECMP BGP systems.
- 2x AX101 AMD Ryzen™ 9 5950X 128 GB DDR4 ECC as workload nodes.

All systems had 10G NICs.

The benchmark systems had the following tuning configuration:

```
ulimit -n 1000000
```

The target system was running the following versions: Kubernetes version: `v1.24.4` and Cilium version: `v1.12.2` Each test, RPS and throughput, was performed separately. Each test was performed 3 times, and in this blog we show the best runs.

###### RPS testing:

To test the requests per second (RPS) we used [hey](https://github.com/rakyll/hey) with the following arguments:

```
hey -cpus 4 -c 1000 -z 10m -disable-redirects -disable-keepalive http://[LB_IP]/
```

We were not looking to get the maximum out of this application, but rather see a progression or regression on the different configurations. We ran twelve replicas spread evenly in the worker nodes of the [echoserver](https://github.com/cilium/echoserver) workload, and exposed it via a service `type: LoadBalancer`.

###### **Throughput testing:**

For throughput we used the well known [iperf3](https://github.com/esnet/iperf) with the following arguments:

```
iperf3 -c [LB_IP] -t 600 -i 30
```

We deployed two iperf3 workloads independently exposed via two service `type: LoadBalancer`.

##### Test 1 (Baseline)

[Here](https://github.com/hetznercloud/cilium-tests) you can find the specific Cilium parameters enabled for all tests, as well as the full output. As a baseline test, we deployed Cilium in the [minimal configuration](https://github.com/hetznercloud/cilium-tests#test-1) and enabled Cilium native BGP and Maglev, kube-proxy, and VXLAN encapsulation.

###### **RPS:**

Total: 64088.5 RPS

P99 latency: 0.0511 - 0.0734 s

It is worth noticing that we had some HTTP timeouts in this test.
Layer 4 Load Balancer

Cilium Worker

###### **Throughput:**

Total: 12.93 Gbits/sec

Layer 4 Load Balancer

Cilium Worker

##### Test 2

In this test we have deployed Cilium with kube-proxy replacement, VXLAN encapsulation, or Cilium native BGP and Maglev. Exact parameters and test outputs can be found [here](https://github.com/hetznercloud/cilium-tests#test-2).

###### **RPS:**

Total: 77367.8 RPS

P99 latency: 0.0522 - 0.0820 s

Layer 4 Load Balancer

Cilium Worker

###### **Throughput:**

Total: 13.18 Gbits/sec

Layer 4 Load Balancer

Cilium Worker

##### Test 3

For this test we deployed Cilium with kube-proxy replacement, without encapsulation, Cilium native BGP and maglev. Exact parameters and test output can be found [here](https://github.com/hetznercloud/cilium-tests#test-3).

###### **RPS:**

Total: 85672.3 RPS

P99 latency: 0.0328 - 0.0428 s

Layer 4 Load Balancer

Cilium Worker

###### **Throughput:**

Total: 16.89 Gbits/sec

Layer 4 Load Balancer

Cilium Worker

##### Test 4A

For this test we deployed Cilium without kube-proxy replacement, without encapsulation, host firewall, DSR, or Cilium native BGP and Maglev. Exact parameters and test output can be found [here](https://github.com/hetznercloud/cilium-tests#test-4a).

###### **RPS:**

Total: 131689.3 RPS

P99 latency: 0.0262 - 0.0266 s

Layer 4 Load Balancer

Cilium Worker

###### **Throughput:**

Total: 18.81 Gbits/sec

Layer 4 Load Balancer

Cilium Worker

##### Test 4B

This test is exactly as 4A, but we deployed a `CiliumNetworkPolicy` to only allow traffic from the two benchmark systems. We have created a third benchmark system, which is firewalled, and we ran the same test to see if there is any noticeable performance regression. Exact parameters and test output can be found [here](https://github.com/hetznercloud/cilium-tests#test-4b).

###### **RPS:**

Total: 131294.4 RPS

P99 latency: 0.0279 - 0.0275 s

Layer 4 Load Balancer

Cilium Worker

###### **Throughput:**

Total: 18.8 Gbits/sec

Layer 4 Load Balancer

Cilium Worker

##### **Results**

From the baseline to test 4A, the CPU usage for the RPS test decreased from a little bit higher that 50% to a little bit lower that 25% resulting in ~50% less CPU usage even while the RPS increased ~+105%.

For throughput, we can see a CPU usage of ~25% that decreased to ~12% once again resulting in ~50% less CPU usage. We can also see that we started with a throughput of 12,93 Gbits/sec and went to 18,81 Gbits/sec resulting in a ~45% performance increase.

We can see that these improvements were caused by not using VXLAN encapsulation and instead using Direct Routing and DSR.

Enabling firewall (tests 4B) for the RPS test did not seem to have any noticeable impact on CPU or RPS performance in the L4LB nodes, and a nearly unnoticeable higher CPU usage on the worker nodes due to the workload nodes checking for the `CiliumNetworkPolicy` and dropping the packages.

For the throughput (tests 4B), the results are inconclusive. Running the iperf3 test from a third benchmark system firewalled did not actually generate traffic because the first connection was dropped by the firewall rule. Thus we only firewalled the initial iperf3 request to start a test, and not actual traffic generated by the test.

Overall, Cilium provided a massive increase in both RPS and throughput while still reducing CPU usage. We are looking forward to testing IPv6 once [this bug](https://github.com/cilium/cilium/issues/14509) is fixed. We would like to thank the Cilium team for creating such an amazing Cloud Native Networking solution, and we are looking forward to using more advanced features in the future such as XDP acceleration and Cluster Mesh.

#### Cosmonic User Story: Running Cilium on Nomad for Wasm Workloads

URL: https://cilium.io/blog/2023/01/18/cosmonic-user-story
Date: 2023-01-18
Categories: Community

_January 11th, 2023_
_Author: Bill Mulligan, Isovalent_

An ideal world is where developers can just ship business logic without having to worry about where or how it runs. wasmCloud is the hot new distributed platform for writing portable business logic that can run anywhere.

Founded by the principal maintainers and creators of wasmCloud, Cosmonic created the first, feature-rich WebAssembly (Wasm) PaaS to enable developers to assemble and deploy applications, across any cloud, edge, or device with minimal effort. As they built the Cosmonic platform the team needed a way to connect and secure all of their customer workloads wherever they reside so their customers could just focus on writing business logic. They chose Cilium as the ideal solution.

**What is Wasm?**

Before we dive into Cosmonic’s Wasm PaaS, we need first to understand how the platform works, and the challenges that led Cosmonic to look at Cilium.

“eBPF is to the kernel what Javascript is to the browser” has often been used to explain what eBPF is, but it now might be better finished “what Wasm was to the browser and is now to the cloud”. Both technologies are highly programmable and help overcome the limitations of the platforms they are embedded in. Wasm, as an abstraction, makes it possible to safely and efficiently deploy the same code wherever your business logic is needed without having to worry about the details of the underlying platform.

Wasm is aimed at providing application developers the ease and accessibility of Javascript but with additional efficiency and safety benefits and the flexibility of using other languages, such as C/C++ just like the kernel verifier improves safety over kernel modules. Because they are compiled, Wasm and eBPF also run much faster than interpreted languages like JavaScript. This means that developers can build complex and computationally intensive applications, such as games or machine learning models, that would be difficult or impossible to run in the browser using JavaScript.

The same features that make Wasm good for the browser make it good for compute workloads in the cloud. Adobe’s engineering team started using Wasm within flagship web browser-based products Photoshop, Lightroom, and Acrobat, but are now running wasmCloud alongside their Kubernetes architecture. They’re seeing significant performance improvements as a result. Find out more on the [CNCF blog](https://www.cncf.io/blog/2022/11/17/better-together-a-kubernetes-and-wasm-case-study/).

Both Wasm and eBPF are extremely portable. Linux runs on billions of devices and most of them support eBPF out of the box while Wasm code can be run on any platform that supports it. They both make it easier for developers to build applications that can be used across a wide range of devices and platforms.

**Building a Wasm PaaS with wasmCloud**

Created by Cosmonic, wasmCloud was designed to “helps developers build, test, scale, deploy, and operate services and functions at scale quickly” which made it the natural foundation for the Cosmonic PaaS.

WasmCloud brings the promise of low-boilerplate, agile application development where engineers solely focus on business logic. For the Cosmonic platform to deliver on this promise, the team needed to carefully choose the abstractions that would be inherent in the PaaS.

As Dan Norris, Infrastructure Lead at Cosmonic, started designing the architecture behind Cosmonic’s hosted PaaS, he had several major system components to consider. First, he needed a distributed system scheduler to handle jobs. HashiCorp Nomad was chosen over Kubernetes for job scheduling because of its simplicity and versatility (watch [Distributed Flexibility: Nomad and Vault in a Post-Kubernetes World](https://www.youtube.com/watch?v=U1zn5OO8dHs) to learn more).

Running a hosted service for microservices also meant that the team needed a good way to connect, secure, encrypt, separate, and enforce policy for their customers. Cilium was selected as the Container Network Interface (CNI) to help with each of these concerns. Dan comments: “The best CNI is Cilium. Let’s not kid around here. There’s other good ones here, but Cilium ticks all the boxes.”

Initially, integrating Cilium with Nomad was challenging as Cilium is, unsurprisingly given Kubernetes’ omnipresence, better integrated with Kubernetes. Dan and the team at Cosmonic wrote a custom operator, integrated with Cilium and capable of working with Nomad - they are looking at open sourcing it in the future.

**Securing the Network and Platform with Cilium**

On Cosmonic, once a [host](https://cosmonic.com/docs/faq/glossary#host) is launched, a Nomad job is created and Cilium is used to provide network policy and traffic encryption. “Cilium felt like a no-brainer” Dan explains, “especially, with regards to network policy support. We used to allow Internet egress access but needed to prevent internal access or multi-tenancy communications. We want to run thousands of customer workloads so we needed to aggressively multi-tenant from the beginning.”

Cilium is also used for transparent encryption. “With WireGuard, all the internal traffic is encrypted. I don’t have to worry about it and I don’t have to manage a PKI infrastructure. That was the killer feature.” Dan continues, “I don’t have to worry about a service mesh. I’ve run service meshes before. It’s great but that’s yet another system to manage. [With Cilium], you can just toggle that flag and you’re done.”

As customer adoption grows on Cosmonic, Dan knows he will eventually have to deal with abuse and bad actors on the platform. He will need to identify them at a network level and tie that back to a particular account. Cilium (and Hubble once they start using it) gives them forensics insight on network flow logs at a particular moment in time: “without Cilium, we would have had to analyze VPC flow logs and figure out how to tie it back to an individual workload on a particular box, with zero insight.”

The combination of wasmCloud, Nomad, and Cilium has enabled Cosmonic to build a scalable and secure PaaS for running portable business logic across any cloud. Cosmonic is now available in Developer Preview. If you are interested in learning more and to try it out, head out to [Cosmonic](https://cosmonic.com/). If you have any questions on Cilium, join the Slack channel [Slack](https://slack.cilium.io/).

#### Cilium User Story: Connecting 390+ Stores and 4.3 Billion Website Visitors

URL: https://cilium.io/blog/2023/01/05/retail-user-story
Date: 2023-01-05
Categories: Community

_January 05th, 2022_
_Author: Bill Mulligan, Isovalent_

_This User Story comes from a company in the retail industry_

Retail experiences are increasingly moving online with consumers wanting to be able to buy when and where they want. For one retail company, that meant supporting 392 stores with 657 million visitors, 4.3 billion visits to their website, 23.9 million downloads of their app, and 3,000 co-workers in their Digital Organization.

They needed to modernize to meet the needs of the rapidly changing retail industry. A key part of that strategy was adopting a multi-cloud strategy, across an on-premises private cloud platform and multiple public cloud providers. While their digital presence is incrementally moving to the public cloud, an on-premises environment is still required for the foreseeable future. Their private cloud needed a developer experience similar to what would be offered in the public cloud to keep pace.

##### Building a Secure Private Cloud

The Cloud Center of Excellence was tasked with building out a cloud native platform for their private cloud. With a complex set of data centers, stores, and regulated environments they needed to be careful about their technology selection because it could quickly become an unmaintainable stack of tools customized to each environment. They turned towards open source to give them the speed and flexibility that they needed. By working with the projects, they could learn what assumptions were made, what the roadmap is, and how they could influence or even drive feature development. The company took the philosophy that they would look for a CNCF project before actually developing anything on their own.

##### Cloud Native Networking Requirements

They selected Kubernetes as the foundation for their private cloud due to its API-first architecture and extensibility. Their private cloud architecture also follows public cloud principles such as region/availability zones, multi-tenancy, and a zero-trust approach to security. However, to meet these demands, they required an advanced cloud native networking platform to provide:

- Multi-cluster connectivity and security across AZs
- Integration with the existing network data center fabric and firewall
- Support for non-Kubernetes workloads such as bare-metal
- High level of observability
- IPv6-readiness for future requirements

##### Migrating from Calico to Cilium

The retail company originally started building their network with Calico, but ran into problems. They wanted to run routing daemons on the boxes as routers in their network, outside the CNI and Kubernetes, they also wanted to be able to mesh clusters together, and needed a BGP environment. When looking for alternative solutions, they landed upon Cilium and decided to work with [Isovalent](https://isovalent.com/) to implement it.

They set up a new environment and started to migrate workloads from their existing infrastructure to the new one. This provided the smoothest transition from a consumer point of view because they just had to switch the routing in the back door. With Cilium enabled, they were able to take advantage of many new features including Cilium Cluster Mesh to provide cross cluster service discovery, load balancing, and network policies, integration with their existing firewall and leaf/spine network with Egress Gateway and BGP respectively, and flow observability of their network traffic with Hubble.

##### Cilium Benefits and Future Direction

By implementing Cilium, they gained operational simplicity, interoperability, and increased developer trust. BGP on Cilium and Cluster Mesh were significantly simpler to operate than their previous Calico set up and have saved the team countless hours. For interoperability, Cilium not only acts as a bridge between the private cloud and their existing infrastructure with its Egress Gateway Firewall and BGP features but it is also used to connect with non-Kubernetes environments.

With zero trust networking, it can be difficult for developers to understand why their application isn’t working or their connection is dropped. Using Hubble, developers are now able to visualize their workload’s network traffic and see what traffic has been dropped allowing them to implement and verify the network policies they need. The data from Hubble is also connected to their Grafana environment with existing application metrics, logs, and traces.

Besides just consuming open source, the retail company has committed patches and bug fixes to Cilium too. “The amount of help and guidance from opening our first issue to getting our commits merged has been amazing! The maintainers are so valuable to the community and make people feel very welcome” Senior Digital Technology Engineer.

In the future, the retail company would love to run IPv6-only within their data centers, but they have a large legacy estate with limited IPv6 support. Cilium’s [new Nat46/64 feature](https://isovalent.com/blog/post/cilium-release-112/#nat46-nat64) will simplify their IPv6 roll-out. They also want to examine [Cilium sidecarless service mesh](https://isovalent.com/blog/post/cilium-service-mesh/) and are looking at [Tetragon](https://isovalent.com/blog/post/2022-05-16-tetragon/) in collaboration with their security team to improve the security posture across all of their environments.

#### Cilium Google Season of Docs Case Study

URL: https://cilium.io/blog/2022/11/30/cilium-google-season-of-docs-2022
Date: 2022-11-30
Categories: Community

**Cilium Google Season of Docs Case Study: Reorganize Cilium Features and Getting Started**

_Authors: Bill Mulligan_

Organization Description: Cilium (current version 1.12, first released in 2017) is an Apache 2.0-licensed project for providing, securing, and observing network connectivity between container workloads - cloud native, and built around the revolutionary Linux Kernel technology eBPF. Our users adopt Cilium to solve their cloud networking, observability, and security issues at scale including running some of the largest Kubernetes clusters in the world. Cilium has a large community of contributors from over 50 countries around the world. Multiple companies have staff paid to work on Cilium too. Cilium is an incubating project in the Cloud Native Computing Foundation and is the networking data plane for Google Kubernetes Engine and Anthos.

##### **Problem Statement**

Cloud native networking and security are complex topics that even long time practitioners struggle to master. Cilium documentation has grown organically over time and the project has mainly been used by advanced cloud native users. The documentation lacks the structure that new, and even existing, users can rely on to quickly find the information they need. In addition, users sometimes struggle to discover all of the features and functionality of Cilium because the documentation does not make them easily discoverable. Finally, new users often have questions about how to troubleshoot setting up their network.

Improving the structure of the documentation will help new users adopt Cilium and reduce the support load on the community.

##### **Project Description**

###### **Creating the proposal**

The Cilium team knew about Season of Docs from other CNCF projects that had participated in the past. We had some open PRs about restructuring the documentation, but nobody was finding the time to finish them. Two people from the Cilium team (Liz and myself) volunteered to create a project [proposal](https://docs.cilium.io/en/v1.12/contributing/development/season_of_docs/).

Once the proposal was written, we shared it in the weekly developer meeting for feedback. People were really excited about finally having someone to look at the overall docs structure! We made a PR of the proposal in the docs and had it approved by the maintainers. Finally, we opened our Open Collective account to participate and receive payments.

###### **Budget**

We estimated that hiring a part time writer would cost roughly $1,500 a month which helped us come up with our $9,000 project budget. We also included some extra to buy our writer Cilium swag to show what they did for the project.

After our interviews, we really liked two of the applicants and were having a hard time choosing between them because they would each bring different things to the project. We ultimately asked them if they would be willing to split the budget to each work on different parts of the project. Both writers agreed so we moved forward with a split budget with each writer working on a different part of the project proposal. We liked what they each brought to the project and would recommend it for other projects faced with a similar decision.

###### **Participants**

The core team working on this project was:

- Liz Rice (Cilium)
- Quentin Monnet (Cilium)
- Bill Mulligan (Cilium)
- Yoyo Wu (Technical Writer)
- Divine Odazie (Developer Relations)

To find our technical writer, we created a Google Form to apply and promoted it on Twitter, the Cilium Slack, and our newsletter. We had 26 applications and ended up interviewing 5 people. Of the people interviewed, we really liked Yoyo Wu’s previous experience restructuring documentation with Google Season of Docs and Divine’s excitement for Cilium (he even made some PRs leading up to the application).

Once both agreed to work on different parts of the projects (Yoyo on restructuring and Divine on the Getting Started Guide) we began to work with them. We set up weekly and bi-weekly calls to coordinate our work. Divine was in the same time zone as the Cilium team so it was easy to find a time while Yoyo was several time zones over, but she preferred to meet in the evening so it worked out well.

The Cilium team worked with Divine to find some good first issues in the getting started guide to get familiar with the project then had him dive into improving the documentation and adding additional links and resources that would be helpful for people just getting started. It was great to have a new set of eyes to look over the getting started guides to help spot things that the rest of us took for granted.

With Yoyo, we started with the outline of the documentation in a Google doc. She then went through and suggested changes for restructuring it. We took these suggestions to the sig-docs channel in our slack for further feedback. After we all agreed on the new structure, Yoyo went ahead and started making the changes to the documentation structure.

The check-ins were very helpful to keep all of the PRs moving and we would recommend anyone else doing GSoD to also set up regular meetings with their technical writer. Divine has now gone on to make contributions to the Kubernetes projects around Cilium and Yoyo was inspired to help improve the documentation around eBPF too!

###### **Timeline**

Once we applied, we started putting together the previous work that had been done to restructure the docs. After we found out we were accepted, we started the application and interview process and hired our technical writers. With them onboard, we were able to quickly move through our objectives and actually were able to hit them ahead of time.

We hired and onboarded our writers in May. In May and June Yoyo audited the existing docs and came up with the new structure. She continued to work on merging PRs for the restructure through October before moving on to other parts of the project. Divine came up with a structure for his work in June and completed updating the Getting Started updates by September.

We were happy with our estimation of the amount of time it would take and were slightly ahead of schedule the whole time.

###### **Results**

Across the project, we were able to merge [14 pull requests](https://github.com/cilium/cilium/pulls?q=is%3Apr+is%3Aclosed+label%3Agsod-22). These included:

- Restructuring each section of our documentation
- Adding relevant video content to our docs
- Updating our navigation bar and fixing formatting issues
- Retiring old documentation

In addition, Divine was able to make his first contribution to the Kubernetes project and Yoyo has continued to contribute to the Cilium documentation!

###### **Metrics**

In our proposal, we proposed three metrics:

- The number of Slack questions covered in the documentation decreasing
- The number of Github issues covered in the documentation decreasing
- The number of pull requests from new contributors increasing

Because of the metrics we chose, it wasn’t easy to capture the data and it was difficult to determine the trends exactly. The number of PRs from new contributors increased over the whole project and anecdotally the number of slack questions and Github issues also decreased. We also found that maintainers of the project were more happy with the documentation structure too.

###### **Analysis**

Google Season of Docs has been a great success for the Cilium project! It was great to finally be able to have time to restructure the documentation to make it easier for new people to get started and for people to find the information that they need. Beyond that, we also think we have a good structure for adding new documentation going forward.

It was great to have more people contributing to the Cilium documentation and even better that both of our technical writers were so inspired by the program that they are continuing to contribute more to open source!

It wasn’t all smooth sailing the whole time. We would sometimes lose track of which PRs were in flight and what we all were currently working on. We tried to help solve this by creating GSoD tags in Github to make the PRs easily findable, but in the future it may be better just to create a projects board to know the status of everything being worked on.

While restructuring the documentation had a pretty clear goal, “simplifying” the getting started was not as clear and we struggled a little bit to define the start and finish of that part of the project. It would be better to have more clear objectives in the future.

Finally, it would have been better to choose better metrics to track for the success of our project. The ones we chose were difficult to collect and we had to try upon more anecdotal evidence instead.

##### **Summary**

Google Season of Docs was a smashing success for Cilium. We accomplished all of our goals and we got more people excited about open source!

We really liked the opportunity to work with both of our technical writers and what they each brought to the project. It was great to see them dive head first into Cilum and bring new energy to our documentation. While cloud networking can be a difficult topic to pick up, both of our technical writers were able to connect right to it.

For future projects, we would recommend:

- Don’t be afraid to split your project between technical writers if they can each bring something different to the project. Be sure to talk with them to make sure they are ok with it, but if they all agree, it can be fun to bring multiple perspectives to the work
- Find a good process to track your work from the beginning and make sure you have clear end goals. What starts out simple can quickly grow and be harder to piece back together. Know where everything is and where it ends.
- Meet regularly with your technical writer and welcome them to the community! It's fun to have new people in the community and they bring a fresh set of eyes.
- Go ahead and apply for Google Season of Docs! We weren’t sure if we had the time to do the project, but it has been a great experience overall and we hope to do it again in the future.

###### **Acknowledgments**

Our team would like to thank the whole Google Season of Docs team for running the program. We found it to be very smooth and a great experience overall. We hope to participate again in the future.

#### Cilium at KubeCon NA 2022 Wrap Up

URL: https://cilium.io/blog/2022/11/04/cilium-kubecon-na-wrapup
Date: 2022-11-04
Categories: Community

_November 4th, 2022_
_Author: Bill Mulligan, Isovalent_

KubeCon + CloudNativeCon NA 2022 was a buzz with activity around Cilium. In this post, I will try to sum up all the big annoucements and moments so the people that weren't able to attend in person can feel like they were there too.

##### Announcement Microsoft AKS picking Cilium

Cilium is now the default CNI for all major cloud providers. If it wasn't obvious already, Cilium has become the standard networking layer for Kubernetes everywhere. Here's the [announcement](https://azure.microsoft.com/en-us/updates/public-preview-azure-cni-powered-by-cilium/) from Microsoft, [blog](https://techcommunity.microsoft.com/t5/azure-networking-blog/azure-cni-powered-by-cilium-for-azure-kubernetes-service-aks/ba-p/3662341), and [docs](https://learn.microsoft.com/en-us/azure/aks/azure-cni-powered-by-cilium). There's also a blog from Isovalent about it [here](https://isovalent.com/blog/post/azure-cni-cilium/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog).

##### Grafana + Cilium Integration

eBPF-powered observability for Kubernetes and cloud native infrastructure. What else needs to be said? Cilium, through eBPF, lets you collect all the information you need about your applications and infrastructure while Grafana gives you a great way to visualize it. Cilium + Grafana will be a crucial part of the observability future. I highly recommend [the blog](https://grafana.com/blog/2022/10/24/grafana-and-cilium-deep-ebpf-powered-observability-for-kubernetes-and-cloud-native-infrastructure/) deep diving into how the integration works.

##### CNCF Graduation Application

Live on stage, Cilium applied for CNCF Graduation (If you haven’t yet, make sure to:+1: the [pull request](https://github.com/cncf/toc/pull/952)) I'm excited to see it graduate as validation of all the work the community has put in to bring the project this far! There is a [blog post](https://cilium.io/blog/2022/10/27/cilium-applies-for-graduation/) about it too.

##### Conference Talks

Cilium was a part of 13 talks all around the conference covering KubeCon, eBPF Day, ServiceMeshCon, Observability Con, Security Con, and Kubernetes on Edge Day. You can read all about them in [blog post](https://cilium.io/blog/2022/10/17/cilium-kubecon-na-talks/) and the videos should be up soon too.

##### Cilium Project Meeting

Cilium hosted its first project meeting at KubeCon. I wasn't quite sure what to expect, but I was blown away by the response! We had four hours packed with getting people introduced to Cilium, disucssing issues and development, and diving into service mesh. It was also great to see some faces I have only seen on slack so far.

##### Final Thoughts: Ecosystem

Looking back on the show, I think we will see this KubeCon as the kick-off of the ecosystem around Cilium. With the Microsoft announcement, Cilium has cemented its status as the standard way of doing networking in cloud native environments.

Once an ecosystem has a standard, it makes it massively easier to provide integrations and extensions since everyone is using the same thing. We can already see this with the Grafana announcement too. With Cilium everywhere capturing observability data with eBPF, it makes total sense to connect it to Grafana and visualize the data.

Cilium everywhere will also bring the power and functionality of eBPF everywhere. The ecosystem around Cilium can leverage this data and functionality to build better platforms to end users. And this is just the kick-off!

#### Cilium CNCF Graduation Application

URL: https://cilium.io/blog/2022/10/27/cilium-applies-for-graduation
Date: 2022-10-27
Categories: Community

This week at KubeCon 2022 NA in Detroit, the Cilium community has applied to become a CNCF graduated project by creating a [PR](https://github.com/cncf/toc/pull/952) in the cncf/toc repository.
This is a major milestone for the Cilium community and users. The entire community is grateful to everyone who has helped to get the Cilium this far. We are looking forward to work through the graduation project with the CNCF community.

##### What is Cilium?

[Cilium](https://cilium.io) is a networking, observability, and security solution with an eBPF-based dataplane (To learn more about eBPF, visit [eBPF.io](https://www.ebpf.io/)). As a CNI, it provides a simple flat Layer 3 network with the ability to span multiple clusters in either a native routing or overlay mode. Cilium is L7-protocol aware and can enforce network policies on L3-L7 using an identity based security model that is decoupled from network addressing.

Cilium implements distributed load balancing for traffic between pods and to external services, and is able to fully replace kube-proxy, using efficient hash tables in eBPF allowing for almost unlimited scale. It also supports advanced functionality like integrated ingress and egress gateway, bandwidth management and service mesh, and provides deep network and security visibility and monitoring through Hubble and Tetragon.

##### Why we are ready to graduate

Cilium has 13,000+ GitHub stars and 500+ contributors across the various repos in the [Cilium GitHub org](https://github.com/cilium) and 13,000+ users registered for the [Cilium community Slack](https://slack.cilium.io). Cilium was [open sourced in 2015](https://github.com/cilium/cilium/commit/7fa3c60eb7dbe7a5a4caea3aab0396f75a8b10c7), [hit the 1.0 milestone in 2018](https://cilium.io/blog/2018/04/24/cilium-10/), and [joined the CNCF at Incubation level in October 2021](https://www.cncf.io/blog/2021/10/13/cilium-joins-cncf-as-an-incubating-project/). It currently powers production systems at scale all around the world, including at
[Adobe](https://www.youtube.com/watch?v=7UQ2CU6UEGY), [Bell Canada](https://www.youtube.com/watch?v=vJaOKGWiyvU), [CapitalOne](https://www.youtube.com/watch?v=hwOpCKBaJ-w&ab_channel=eBPFSummit), [Datadog](https://www.cncf.io/case-studies/datadog/), [IKEA](https://www.youtube.com/watch?v=sg-F_R-ZVNc), [Nexxiot](https://www.cncf.io/case-studies/nexxiot/) [Sky](https://www.youtube.com/watch?v=u-4naOMfs_w), [Utmost](https://www.cncf.io/case-studies/utmost/), and [many more](https://github.com/cilium/cilium/blob/master/USERS.md). It is also the default CNI for managed Kubernetes on GKE, Anthos, EKS-A, AKS, and other cloud providers.

##### How to Help

- If you are a fan of Cilium, we would love a,, or on the [PR](https://github.com/cncf/toc/pull/952).
- If you are using Cilium in dev, testing, or production, we would love to hear your feedback in our [User Survey](https://docs.google.com/forms/d/e/1FAIpQLSfL01ttSgzmWQbOHxZu61B99Iw3Yw3irZ1X7CRCQfBJkrN7Rg/viewform?usp=sf_link)
- If you are using Cilium in production, please add your name to the [Users](https://github.com/cilium/cilium/blob/master/USERS.md) doc
- If you do any of the above, share it on the social media platform of your choice

#### Cilium Talks at KubeCon NA 2022

URL: https://cilium.io/blog/2022/10/17/cilium-kubecon-na-talks
Date: 2022-10-17
Categories: Community

_October 17th, 2022_
_Author: Bill Mulligan, Isovalent_

KubeCon + CloudNativeCon is already back around upon us and Cilium is represented in full force! There are 12 talks, workshops, and meetings that cover technical talks from Cilium contributors, tutorials to get people started, and end user stories from people putting Cilium into production. You will be able to catch presentations on everything from Cilium NetworkPolicies to Cilium for telco workloads to a tour of the Cilium for security. Let’s dive into each of the talks below.

##### Cloud Native eBPF Day

###### [Keynote: eBPF - Everything You Need to Know in 5 Minutes - Thomas Graf, CTO, Isovalent](https://sched.co/1Auye)

Monday, October 24 • 9:15am - 9:20am

eBPF has become the key technology for infrastructure software. This session tells you everything you need to know about eBPF in 5 minutes. Why eBPF matters and why it exists. What it can do. What it can’t do. Who uses it for what. And finally, what the future holds.

###### [Extending Cilium with eBPF for Exposing HTTP/1 Golden Metrics - Aidan Carson, Solo.io](https://sched.co/1Auyq)

Monday, October 24 • 10:40am - 11:10am

Metrics are table stakes for any modern service mesh, and because of the placement of proxies in a traditional sidecar-based topology, fairly straightforward to retrieve. However, as use cases evolve and the space matures, we have seen an evolution beyond that of traditional sidecar-based implementations. In this talk, Aidan will technically deep dive into the mechanisms through which we may retrieve the most valuable metrics — request count, response code, and latency (the so-called "golden metrics") — in a sidecarless environment with the help of eBPF. Aidan will discuss the hardships of implementing such a solution, and a few tips when operating at a higher protocol layer. All of these metrics will be exportable in Prometheus so that you may integrate a solution like this with toolchains in your existing infrastructure.

###### [You and Your Security Profiles; Generating Security Policies with the Help of eBPF - John Fastabend & Natalia Reka Ivanko, Isovalent](https://sched.co/1Auz2)

Monday, October 24 • 1:15pm - 1:45pm

Using eBPF to detect and enforce security events by creating Security Policies continues to rise. This talk's main aim is to provide a hands-on walkthrough on the existing eBPF-based solutions to capture different sets of security events and approaches to create Security Policies. There has been a wide range of options to filter for system calls and generate Security Policies manually with seccomp and seccomp-bpf as well as automatically by using Inspektor Gadget and a Red Hat created seccomp profiler in the form of an OCI hook. Extending security events from capturing system calls to binaries, sockets, file operations, capabilities and namespaces and generating Security Policies automatically has became possible with Tetragon. The audience will learn how to leverage these tools, capture different sets of security events, generate Security Policies via a live demo and will leave with a good understanding of what would be the optimal solution to implement eBPF-based security tooling on their system.

###### [Simplifying and Making the Network Programmable with Kubernetes and SRv6 - Daniel Bernier, Bell Canada & Louis De Los Santos, Isovalent](https://sched.co/1AuzH)

Monday, October 24 • 2:25pm - 2:55pm

SRv6 is being rapidly deployed by some of the largest names in the networking industry because it has the potential to simplify existing enterprise, telco, and provider networks while also introducing the idea of “programming the network as a computer.” Networking teams are now looking for ways to bridge the gap between SRv6 and Kubernetes to treat containers as ordinary clients on their networks. This talk will discuss how Cilium and its eBPF data plane was extended to support telco networking requirements in a cloud native way by integrating SRv6 and Kubernetes. It will start with an overview of the SRv6 use cases and network architecture at Bell Canada. We will then demo how Cilum’s SRv6 integration can be utilized with BGP to build a multi-cluster L3VPN topology over native IPv6.

###### [Panel Disucssion: Is There Actually a Byte Behind All the Buzz? eBPF in Production! - Frederic Lardinois, TechCrunch; Andrew Sauber, The New York Times; Daniel Bernier, Bell; Purvi Desai, Google & James McShane, SuperOrbital](https://sched.co/1AuzK)

Monday, October 24 • 4:20pm - 5:00pm

eBPF is a revolutionary and very hyped technology which has led some people to ask if there actually is a byte behind all the buzz. This panel brings together end users of eBPF that are actually putting it into production to find out whether it is a painful sting or sweet honey for their infrastructure. The audience won’t learn about futuristic buzzwords. Instead, they will understand how eBPF is changing networking, observability, and security in production today.

##### Open Observability Day

###### [“Are These Things Talking to Each Other?” - Observing Kubernetes Networking with Cilium, Hubble and Grafana - Anna Kapuścińska, Isovalent](https://sched.co/1Avqp)

Monday, October 24 • 4:25pm - 4:55pm

“Are they even talking to each other”? - asked almost every engineer on the planet when debugging their system. Although the question is being asked forever, the answer isn’t always easy. As the complexity of our systems grows, networking becomes more and more a magic glue: it's hard to understand and even harder to debug. Good observability helps in both understanding and debugging what happens. Nowadays more and more people are looking into eBPF as a solution for networking and observability. That’s what the Cilium project provides: eBPF-based networking, with a visibility layer - Hubble. At the same time, observability in the cloud native world is commonly achieved with Prometheus and Grafana dashboards. We will connect these technologies. Using Hubble metrics, we will visualize how applications talk to each other in a Kubernetes environment, famous for its complex networking. In a real-world scenario, we will discover a mysterious communication problem. It won’t scare us though. Equipped with open source tools, we will quickly debug the system and uncover the underlying issue.

##### Cloud Native SecurityCon

###### [Verifiable eBPF Traces For Supply Chain Artifacts with Witness and Tetragon - Cole Kennedy, TestifySec](https://sched.co/1AOie)

Monday, October 24 • 10:25am - 10:55am

Until now, validating the build environment and detecting tampered tooling in a build has been very difficult. This talk will show how Cillium Tetragon and Witness integration simplifies this process for developers and security engineers. Witness is a framework for supply chain security that implements the in-toto specification. It has a modular design, easily extendable for various attestors, backends, and key providers (including SPIFFE/SPIRE). This talk will show an attestation plugin that programs Cillum Tetragon to provide detailed eBPF traces of a build step. Additionally, we will create a build policy that verifies the trace and blocks the execution of workload compiled by a malicious compiler when the compiled workload is executed.

##### Service MeshCon

###### [How Many Proxies Do You Need? - Liz Rice & Thomas Graf, Isovalent](https://sched.co/1Azt0)

Tuesday, October 25 • 3:30pm - 4:00pm

To provide app-level network functionality like L7 load balancing and TLS termination, Service Meshes use a proxy component that terminates L7 connections on behalf of applications. Traditionally the proxy has been co-located in each application pod as a sidecar container, but Cilium Service Mesh changed this with the innovation of sidecarless service mesh. Istio have also now adopted a sidecarless approach that was recently announced as Istio Ambient Mesh.

But “sidecarless” doesn’t mean “proxyless”! It’s a question of where you deploy the proxies, and how you create the relationship between apps and proxies. In this talk we’ll explore the pro’s and con’s of different models, and explain where eBPF makes a difference (and where it doesn’t) in not only network performance, but also to provide observability and security capabilities.

###### [Hands-on Workshop: Network Policies - The Not-So-Hard Way - Raymond de Jong, Isovalent](https://sched.co/1AOju)

Tuesday, October 25 • 12:55pm - 2:05pm

Many people avoid networking wherever possible because they think it is too complex and don’t even get them started on policy. In this session, we will help overcome these fears for both app developers and operations teams with network policies the not so hard way. In four easy steps we will: Introduce the fundamentals of Cilium Network Policies and the basics of application-aware and Identity-based Security Discuss the default-allow and default-deny approaches and visualize the corresponding ingress and egress connections Use the Network Policy Editor to show how a Cilium Network Policy looks and what they do on a given Kubernetes cluster Walk through examples and demonstrate how application traffic can be observed with Hubble The audience will walk away with the ability to create network policies for their workloads so they can stop worrying and love the secure connections, and show how you can use the Network Policy Editor to apply new Cilium Network Policies for your workloads.

##### Kubernetes on Edge Day

###### [ Which Edge Are You on? Service Affinity with Cilium Cluster Mesh - Raymond de Jong, Isovalent](https://sched.co/1ARMh)

Tuesday, October 25 • 11:45am - 11:55am

Edge computing can require connecting hundreds of clusters across disparate locations and infrastructures. Without a networking solution to manage this scale and complexity, you will just have a bunch of computers talking to themselves rather than each other and your customers. In this session we will explore how you can leverage the new Service Affinity feature with Cilium Cluster Mesh for providing the correct connectivity for load-balancing, observability, and security between nodes across multiple clusters, enabling simple, high-performance cross-cluster connectivity at the edge. We’ll explore how Cluster Mesh allows endpoints in connected clusters to communicate while providing full security policy enforcement. The audience will walk away with an appreciation for how eBPF can ensure you communicate with the right edge.

##### KubeCon + CloudNativeCon

###### [Cilium Project Meeting](https://sched.co/1BaT3)

Tuesday, October 25 • 8:00am - 12:00pm

Join us to discuss the future of the Cilium project! [Sign up for what you are interested here](https://docs.google.com/document/d/1GXHNt3JrmvZY_uYI9sO4SHLej2E5SCpLpZsD0fpk3Ms/edit#)

###### [Cilium Updates, News And Roadmap - Thomas Graf, Bill Mulligan & Liz Rice, Isovalent; Purvi Desai, Google](https://sched.co/182NR)

Wednesday, October 26 • 5:25pm - 6:00pm

Welcome to Cilium! In this session you'll get an update on how the Cilium project has been progressing on the road towards graduation. You'll hear about the latest developments and future roadmap, including news about some of the largest and most interesting deployments of Cilium. And don't miss this session if you're interested in contributing to the project, as there will be guides on how to get involved and where your help is needed.

###### [100Gbit/S Clusters With Cilium: Building Tomorrow’s Networking Data Plane - Daniel Borkmann & Nikolay Aleksandrov, Isovalent](https://sched.co/182DB)

Thursday, October 27 • 4:30pm - 5:05pm

The vast majority of Kubernetes users today are running IPv4-only clusters or transitioning to dual stack deployments as an interim step on the journey towards IPv6-only clusters. The latter are not only an enabler for more IPAM flexibility and larger cluster scale, but also unlock new Linux kernel innovations in networking and eBPF to cater for data intensive workloads. In this talk, we examine what's possible once we arrive at IPv6-only clusters. We present recent advancements in Cilium's networking data plane enabling a low-latency architecture suitable for "BIG TCP"-based workloads requiring IPv6 for 100Gbit/s transfers and beyond for a single socket. In addition, we deep dive our path into achieving host networking performance characteristics for Pods through a new veth driver replacement we specifically developed for the kernel in context of Cilium. With the resulting eBPF forwarding architecture, most unneeded parts of the stack are bypassed, drastically improving networking.

###### [Hack Back; Let’s Learn Security With CTFs! - Lewis Denham-Parry, Control Plane & Natalia Reka Ivanko, Isovalent](https://sched.co/182Ej)

Thursday, October 27 • 3:25pm - 4:00pm

Threat actors have always been looking to attack clusters. Do you have the right security in place to detect and defeat if they are targeting yours? Or they are already in? Kubernetes has become the de facto cloud operating system and production environments have increased in maturity. So have the threats. Security Teams don’t necessarily have the expertise to detect state-of-art attack scenarios specific to cloud-native environments, like Kubernetes. So, where do they get started? Capture-The-Flag (CTF) events are a great way to learn about the techniques of both attack (Red Team) and defence (Blue Team). This talk will give you a framework for your own internal CTF events, with Red and Blue Team assessments, as a best practice for improving security in your organisation. We'll give a hands-on, live walkthrough of the top 3 state-of-art attack scenarios as CTF exercises using common open source projects like Simulator and Tetragon. Remember, the best way to learn how to detect is to first know how to attack!

Make sure you stop by the Cilium booth to get your Cilium swag. See you there!

#### Cilium User Story: Securing 100,000+ RPS in a Multi-Tenant Environment

URL: https://cilium.io/blog/2022/10/13/publishing-user-story
Date: 2022-10-13
Categories: Community

_October 13th, 2022_
_Author: Bill Mulligan, Isovalent_

_This User Story comes from a company in the publishing industry_

With today's technology and new cycles, news "fresh off the press" means something quite different than the days of just printing presses. Publishing companies today need to keep their technology up to speed with the pace the world moves. For one publishing company, Cilium became their solution to connect their worlds together.

To make sure the news gets to you every morning, afternoon, and evening, the publishing platform has a Delivery Engineering Team that provides infrastructure to product engineering teams. Rather than running their own data centers, they have a cloud first strategy where everything should be moved to the cloud. The Delivery Engineering team is responsible for centralized infrastructure that everyone can deploy their services onto and is building out this cloud platform for the product engineering teams. By abstracting the business logic from the infrastructure, the platform allows teams to move more quickly.

Previously, many teams were spinning up their own individual Kubernetes clusters which led to inefficient use of resources and a large operations overhead. Delivery engineering needed to build a “paved road” for the developers to use and decided to build large multi-tenant, multi-region Kubernetes clusters to support their needs. Different business units would be able to deploy into the same clusters and the platform internals would be abstracted away from the teams.

While building out the platform, the Delivery Engineering team needed to balance standardization of infrastructure and the needs of individual teams. Different teams need or like different tools, but as the platform scales, these differences quickly add operational complexity and drain support resources. This fragmentation would mean instead of working through known bugs and edge cases and being able to quickly isolate the problem, the platform team would have to spend time debugging new problems and finding solutions.

“You can create a cluster in any cloud right now in seconds - that’s easy. The Day 2 Operations, scaling it, and keeping it up to date and secure are the real challenges.” - Staff Software Engineer

##### Creating a Secure Multi tenant Kubernetes Environment with Cilium

The top two requirements for building out their multi tenant platform were security and performance. Guaranteed isolation between tenants was needed to ensure different teams could safely run on the same platform. In addition, as an organization with breaking news, they needed to plan for massive scale and performance. Just 10 services alone could easily reach 100,000 requests per second.

Isolation was a day one requirement that they needed to support with network policy. When they looked at what other companies were adopting, they found that “Cilium was basically the default everywhere.” On the performance side, they also knew that eBPF would be faster than IP tables.

They start by bringing up Kubernetes clusters, replacing kubeproxy with Cilium and implementing network policies. Once Cilium was up and running, they also installed Hubble for network observability.

“Hubble is useful because sometimes I don’t know why traffic is being denied and it allows me to quickly observe what is happening on the network and drill down to the root cause. I could get this from another platform, but having it integrated with Cilium makes it simple to set up and use.” - Staff Software Engineer

Cilium has already become the one solution that fits multiple problems from networking to observability to security and they don’t even use all of Cilium’s features yet. As the platform continues to grow, the Delivery Engineering team will be able to meet other team’s requirements and features without having to integrate new tools or implement one off features. For example, Cilium supports different styles of deployment allowing them to choose between direct routing and an overlay network as the team requires. “We know that Cilium can support a lot and is flexible enough to handle our use cases both now and in the future.”

From a security and compliance perspective, they are looking at egress filtering and Tetragon for defense in depth. It would allow them to understand which processes are doing what and drop suspicious traffic. As their platform grows to multiple clusters, they are also looking at Cluster Mesh to enable communication across regions and clouds.

“I’m excited to use Cilium because we have a great solution in place for our current needs and flexibility to meet our future needs. Cilium is backed by the Community, backed by enterprises, backed by the cloud providers, and is the top choice for Kubernetes networking.” - Staff Software Engineer

#### Cilium Standalone Layer 4 Load Balancer XDP

URL: https://cilium.io/blog/2022/04/12/cilium-standalone-L4LB-XDP
Date: 2022-07-13
Categories: Community

_July 13th, 2022_
_Author: Ondrej Blazek, Infrastructure Engineer @ Seznam.cz_

Seznam.cz is a Czech technological company developing a custom search engine, advertising platform, online maps, content management system, and also private cloud services, custom hardware, and datacenters.

##### Architecture

Seznam's infrastructure historically used F5 hardware load balancers but we switched to software load balancers a few years ago. Up until now we've been using a [multiple tier](https://vincent.bernat.ch/en/blog/2018-multi-tier-loadbalancer) setup - [ECMP routing](https://vincent.bernat.ch/en/blog/2018-multi-tier-loadbalancer#first-tier-ecmp-routing) as the first tier + [IPVS](http://www.linuxvirtualserver.org/software/ipvs.html) as the second tier ([L4 load balancer (L4LB)](https://vincent.bernat.ch/en/blog/2018-multi-tier-loadbalancer#second-tier-l4-load-balancing)) + [Envoy proxy](https://www.envoyproxy.io/) as the third tier ([L7 load balancer](https://vincent.bernat.ch/en/blog/2018-multi-tier-loadbalancer#last-tier-l7-load-balancing)). Unfortunately as traffic increased and, thanks to COVID, we started running short on hardware supplies, we had to look for alternatives to use our hardware more effectively.

We have been watching Cilium closely and noticed the [announcement of maglev in Cilium](https://cilium.io/blog/2020/11/10/cilium-19#maglev) together with [**Standalone L4LB XDP**](https://cilium.io/blog/2021/05/20/cilium-110#standalonelb) announced in 1.10. XDP hook is known for its efficient use of CPU and has an extremely high performance. This was very interesting for our team because we have been hitting traffic spikes up to 20M active connections which was massively increasing CPU usage of our IPVS nodes.

Our load balancer setup directs external traffic into the Kubernetes and OpenStack clusters and IPVS is used in a classic "load balancer on a stick" scenario. In a simplified world this looks like this:

Since, we have been using the maglev scheduler (which is part of a netfilter inside the Linux kernel since v4.18) the **Standalone L4LB XDP** was a perfect match to try as an alternative because it supports all the main features we required (IPIP, DSR, maglev).

We are using 25GbE NICs on our IPVS nodes so there was no problem running L4LB at the XDP driver layer as the majority of the modern NICs support it.

```
# ethtool -i eth0
driver: i40e
version: 2.8.20-k
firmware-version: 6.02 0x80003621 1.1747.0
expansion-rom-version:
bus-info: 0000:c1:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
```

```
# lspci | grep Ether
c1:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
```

##### Launching Standalone L4LB

Cilium itself is released as a Docker image which we tried running on the IPVS node itself. As we wanted to persist the state when the Cilium container is restarted/upgraded, we created a systemd service to mount the bpf filesystem:

```
# cat /etc/systemd/system/sys-fs-bpf.mount
[Unit]
Description=BPF mounts
DefaultDependencies=no
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=bpffs
Where=/sys/fs/bpf
Type=bpf

[Install]
WantedBy=multi-user.target

```

and then we launched Cilium in load balancer only mode:

```
systemctl start sys-fs-bpf.mount; docker run \
--cap-add NET_ADMIN \
--cap-add SYS_MODULE \
--cap-add CAP_SYS_ADMIN \
--network host \
--privileged \
-v /sys/fs/bpf:/sys/fs/bpf \
-v /lib/modules \
--name l4lb \
<our_private_docker_repo>/cilium cilium-agent \
--bpf-lb-algorithm=maglev \
--bpf-lb-mode=dsr \
--bpf-lb-acceleration=native \
--bpf-lb-dsr-dispatch=ipip \
--devices=eth0 \
--datapath-mode=lb-only \
--enable-l7-proxy=false \
--tunnel=disabled \
--install-iptables-rules=false \
--enable-bandwidth-manager=false \
--enable-local-redirect-policy=false \
--enable-hubble=false \
--enable-l7-proxy=false \
--preallocate-bpf-maps=false \
--disable-envoy-version-check=true \
--auto-direct-node-routes=false \
--enable-ipv4=true \
--enable-ipv6=true
```

We serve around 3k services and use 30+ L7 nodes so we quickly reached the defaults for lbmap size. However, it is possible to extend it so we added the `--bpf-lb-map-max 512000` option.

##### Setting Up Services

Cilium provides an API to set up the lbmaps. We used the following command to configure all the services:

```
cilium service update --id $idx --frontend "$svc" --backends "$backends" --k8s-node-port
```

The `frontend` stands for each VIP service and `backends` for the L7 nodes (e.g. `cilium service update --id 1 --frontend "10.248.11.13:7047" --backends "10.246.3.34:7047,10.246.39.33:7047,10.246.39.34:7047" --k8s-node-port`).

For BGP announcement, [we use BIRD](https://docs.cilium.io/en/stable/gettingstarted/bird/) so this part is fairly straightforward:

```
# systemctl start bird
# systemctl start bird6
```

##### Comparison Under Load

- [SynFlood](https://www.cloudflare.com/learning/ddos/syn-flood-ddos-attack/)
- [MoonGen](https://github.com/emmericp/MoonGen)
- 1 CPU for generating traffic
- 64B packets

At first, we decided to compare IPVS with L4LB under synthetic test/load generated by a single client running a MoonGen with 1 CPU and small 64B packets with TCP SYN set. Packets (tcp segments) sent out by the MoonGen generator were configured to randomize source IP address and TCP source port so that the traffic is distributed across all receive rx queues as our NIC cards are configured to use 4 tuple (src IP, dst IP, src TCP port, dst TCP port) hashing:

```
# ethtool -n eth0 rx-flow-hash tcp4
TCP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]
```

During the test run we used another client running a simple GET request (using curl in a while loop) to see how the server was able to handle the requests.

The test setup looked like this:

##### Results

In both scenarios (scenario #1 IPVS and scenario #2 L4LB) the MoonGen client was configured to generate 1Mpps (million packets per second) and 3Mpps.
Each output screenshot below is taken from the corresponding server - either the server IPVS/L4LB under test or the `curl` client. For L4LB XDP, both 1Mpps and 3Mpps were too easy and did not have any performance impact. We instead started with 10Mpps and only saw an impact at 14.8 Mpps which was probably due to constraints of the NIC rather than L4LB.

###### 1Mpps - IPVS

IPVS htop output:

Curl client output:

The CPUs were not fully maxed out, but were close to their limit and dropped a few packet drops from time to time.

###### 3Mpps - IPVS

IPVS htop output:

Curl client output:

Since all of the CPU cores handling the interrupts were maxed out, almost all the packets from the second client were dropped by the IPVS node.

###### 10Mpps - L4LB XDP

L4LB htop output:

Curl client output:

###### 14.8Mpps - L4LB XDP

L4LB htop output:

Curl client output:

At 14.8Mpps there were a few packet drops here and there but because we were hitting NIC limits this was totally expected.

##### Production Traffic

The biggest surprise came when we deployed the L4LB XDP to one of our production nodes (which was previously running IPVS). As we have full access to our nodes and are able to start/stop BIRD at any point in time, we were able to cleanly pass traffic between L4LB XDP nodes and IPVS nodes. At approx 11:00 AM, we stopped BIRD on the IPVS nodes so that the L4LB XDP node was handling all the traffic and at approx 11:14 AM, we switched to the 2 nodes running IPVS.

_Packets per second (Higher is better)_

The output above shows that during time ~11:04 - 11:13 the production traffic went from 750kpps up to 1Mpps and this was handled by a single host `lb-l3-5.ko.iszn.cz` using L4LB XDP. At ~11:16, we switched to 2 hosts `lb-l3-7.ko.iszn.cz` and `lb-l3-9.ko.iszn.cz` running IPVS (the summed traffic during that time was also around 1Mpps).

The wow effect really came when we started to look at CPU usage. At one point, we were not sure if we had a bug somewhere because the CPU load was so low when L4LB XDP was handling the traffic. But after looking closely, it really was consuming only half of a single CPU compared to 2x18 CPUs when IPVS was handling the traffic. We were saving 36 CPUs when switched to L4LB XDP.

_Note: The pictures were taken from our production grafana. CPU Load (Lower is better)_

##### Endnotes

The screenshots speak for themselves, but the key take away for us was, **L4LB XDP at the driver layer with a majority of HTTP traffic (~90% of our traffic is HTTP requests) saves us an unbelievable amount of CPUs needed to handle our production traffic**.

The only thing we found missing in Cilium, before we can fully switch to L4LB XDP, are weighted backends which we are currently working on - [maglev: support setting a weight of a backend in a service spec via new cmdline argument](https://github.com/cilium/cilium/pull/18306). After this is done there is nothing stopping us from saying goodbye to IPVS.

With this we would like to thank the Cilium community for building such a great project and for their support!

##### Edit: 2026 updates

Briefly after this blogpost was released we finished implementing backend weights ([#PR18306](https://github.com/cilium/cilium/pull/18306)) and have been using the standalone LB for many years now without any hiccups. Since cilium release v1.18+, cilium no longer supports flag `--datapath-mode=lb-only` and ServiceAPI which we used previously (`cilium service update...`). The same functionality can be fortunately achieved using following flags:

```
--bpf-lb-algorithm=maglev
--bpf-lb-mode=dsr
--bpf-lb-acceleration=native
--bpf-lb-dsr-dispatch=ipip
--enable-l2-neigh-discovery=true
--routing-mode=native
--devices=<node_ext_device>
--direct-routing-device=<node_ext_device>
--enable-l7-proxy=false
--install-iptables-rules=false
--enable-bandwidth-manager=false
--enable-local-redirect-policy=false
--enable-hubble=false
--preallocate-bpf-maps=false
--disable-envoy-version-check=true
--auto-direct-node-routes=false
--enable-ipv4=true
--enable-ipv6=true
--enable-xdp-prefilter=true
--prometheus-serve-addr=<node_IP>:9962
--enable-k8s=false
--kube-proxy-replacement=true
--lb-state-file=/var/run/cilium/lbstate.json
--lb-state-file-interval=1s
--enable-ipv4-masquerade=false
--enable-ipv6-masquerade=false
```

_Note: the important flags in the newer cilium versions are_ `--enable-k8s=false`, `--lb-state-file=...`

The code/functionality still exists, but the burden of maintaining a separate flag in control plane part is no longer needed. Also the ServiceAPI has been modified and it's previous functionality is now fully replaced in this context by `--lb-state-file`. It has support for inotify/fsnotify which watches for updates of the state file which we update frequently.

An example state can be this:

```
{
  "services": [
    {
      "metadata": {
        "name": "lbdor",
        "namespace": "lbr-stable"
      },
      "spec": {
        "ports": [
          {
            "name": "8080",
            "protocol": "TCP",
            "port": 8080,
            "targetPort": 0
          }
        ],
        "type": "LoadBalancer"
      },
      "status": {
        "loadBalancer": {
          "ingress": [
            {
              "ip": "10.3.3.1",
              "ipMode": "VIP"
            }
          ]
        }
      }
    }
  ],
  "endpoints": [
    {
      "metadata": {
        "name": "epslice-lbdor",
        "namespace": "lbr-stable",
        "labels": {
          "kubernetes.io/service-name": "lbdor"
        }
      },
      "addressType": "IPv4",
      "endpoints": [
        {
          "addresses": [
            "10.0.0.1"
          ],
          "conditions": {
            "ready": true,
            "serving": true,
            "terminating": false
          }
        }
      ],
      "ports": [
        {
          "name": "8080",
          "protocol": "TCP",
          "port": 8080
        }
      ]
    }
  ]
}
```

With the move from ServiceAPI to the state file, we observed a significant improvement in state-handling speed, so in this case "new is always better".:)

A big shoutout to [Jussi](https://github.com/joamaki) (one of the maintainers) for the support and work on this.

#### Cilium Talks at KubeCon EU 2022

URL: https://cilium.io/blog/2022/05/06/cilium-kubecon-eu-talks
Date: 2022-05-06
Categories: Community

_May 6th, 2022_
_Author: Bill Mulligan, Isovalent_

The first in-person KubeCon + CloudNativeCon EU since 2019 is just around the corner and we can’t wait to see the Cilium community again outside a Zoom meeting. In case you lost track, that means we last met in Europe at [Cilium 1.5](https://cilium.io/blog/2019/04/24/cilium-15) and boy have there been a lot of updates to Cilium in the meantime (we were able to meet at KubeCon NA in the fall, but missed most of the people outside the US). With that much of a gap, there is a lot to catch up on and the KubeCon schedule has not failed to deliver. The talks cover both technical talks from Cilium contributors and end user stories from people putting Cilium into production. You will be able to catch presentations on everything from Cilium Clustermesh to Cilium for telco workloads to a tour of the Cilium service mesh. Let’s dive into each of the talks below.

[IKEA Private Cloud, eBPF Based Networking, Load Balancing, and Observability with Cilium](https://sched.co/zrPW)

Monday May 16, 2022 10:45 - 11:15 CEST

The digital systems of IKEA are situated in public cloud and private data centers around the world. In this talk we’ll highlight some of the challenges – and opportunities - we faced in setting up a large scale, multi-cluster distributed Kubernetes environment across our data centers. We’ll share how we have used Cilium and its eBPF features to have a better scaling profile, to improve observability and even to replace some of our proprietary load balancers.

- Connecting Kubernetes workloads across our BGP network
- Protecting multi-tenant workloads with multi-cluster network policy
- Cilium support for multi-homed pods \* Mimicking availability zones with Cilium ClusterMesh
- Use Cilium with XDP, ServiceType Loadbalancer and Ingress to replace our proprietary load balancer fronting workload.

You’ll leave this talk understanding how you can use Cilium and its eBPF capabilities to build and instrument your network and obtain great observability.

[Leveraging Cilium and SRv6 for Telco Networking ](https://sched.co/zso2)

Monday May 16, 2022 10:50 - 11:20 CEST

In this session, Daniel Bernier from Bell Canada will be demonstrating how Cilium and its eBPF data plane was extended to support telco networking requirements in a cloud-native way. He will demonstrate how Cilium can provide network segmentation and Multi-VRF support with or without the use of multiple interfaces. With this new approach, he will also explain how to build simple multi-cluster VPN or simple integration to an MPLS provider network by leveraging natively IPv6 and SRv6.

[Connecting Klusters on the edge with deep dive into Cilium Cluster Mesh](https://sched.co/zsAE)

Tuesday May 17, 2022 11:35 - 11:45 CEST

Edge computing can require connecting hundreds of clusters across disparate locations and infrastructures. Without a networking solution to manage this scale and complexity, you will just have a bunch of computers talking to themselves rather than each other and your customers. Cilium is the next generation, eBPF powered open-source Cloud Native Networking solution, providing security, observability, scalability, and superior performance. Cilium has joined the CNCF as an incubating project. In this session, you’ll learn how you can leverage Cilium Cluster Mesh for providing connectivity for load-balancing, observability, and security between nodes across multiple clusters, enabling simple, high-performance cross-cluster connectivity at the edge. We’ll explore how Cluster Mesh allows endpoints in connected clusters to communicate while providing full security policy enforcement. The audience will walk away with an appreciation for how eBPF can help solve their networking challenges at the edge.

[The Future of Service Mesh: Is eBPF a Silver Lining or a Silver Bullet](https://sched.co/zsaO)

Tuesday, May 17 2022 16:05 - 17:05 CEST

Service mesh implementations normally take one of two forms: a proxy per node, or a proxy per workload (the so-called "sidecar"). Linkerd went from A to B. Cilium is suggesting we can go from B to A. Is eBPF a savior, or are we hyper-optimizing a tiny piece of the datapath? And what else might the future of service mesh hold?

[Transparent Live Migration of Services Between Kubernetes Clusters](https://sched.co/ytpo)

Thursday May 19, 2022 15:25 - 16:00 CEST

Operating a distributed database on a single Kubernetes cluster is interesting, but how about transparently migrating it from one cluster to another–potentially between different cloud providers– without impacting user workloads? Kubernetes has become the de facto default deployment for ArangoDB, a distributed Graph database. Consider for example ArangoDB Oasis, a managed Cloud Database service with over 200 deployments (aka highly available database clusters) across three major cloud providers and many regions. But outages, (Kubernetes) upgrades, resource considerations, and cost optimizations require the underlying infrastructure to be very dynamic including migration between Kubernetes cluster, datacenter, or even cloud providers. This talk provides insights into how Kube-Arango, the OSS operator for ArangoDB, supports live migration of distributed stateful applications without impact on users. Challenges in such migration include for example networking, DNS, and persistent data.

[Cilium: Welcome, Vision and Updates](https://sched.co/ytq0)

Thursday May 19, 2022 15:25 - 16:00 CEST

If you’re interested in using Cilium, or contributing to the project, this session is for you. Our agenda for this session: 1. Introduction to Cilium A brief overview of the origin and vision for Cilium. 2. Working with Cilium An end user's perspective of using Cilium. 3. Cilium Service Mesh Cilium can be used as a highly efficient service mesh data plane. Let’s discuss the learnings from our beta, and the upcoming roadmap. We will leave time for Q&A, and an opportunity to meet Cilium maintainers and contributors.

[Choosing Cloud Native Technologies for the Journey to Multi-cloud](https://sched.co/ytpu)

Thursday May 19, 2022 15:25 - 16:00 CEST

Building, deploying and maintaining systems has become increasingly more complicated in recent years. Now, as engineers look toward migrating to multi-cloud architectures, systems and processes may need to be migrated to new technologies. But what choices are available, how do they fit together and how can the CNCF landscape help? This talk discusses the cloud native technologies that can be used to convert to a multi-cloud architecture and highlights some of the lessons learned from taking this journey on at Form3. The audience will learn: - How to decide if multi-cloud is essential for them - The fundamentals of deploying services across multiple clouds with Kubernetes - How to leverage Cilium to mesh together multiple clusters - The basics of event sourcing using NATS in the multi-cloud world - Resilient and performant data storage using CockroachDB This talk is useful for any new comers to the cloud native landscape, as well as those curious about going multi-cloud!

[Kubernetes Networking 101](https://sched.co/ytrV)

Friday May 20, 2022 11:00 - 12:30 CEST

Kubernetes Networking 101 will introduce attendees to the world of network communications in a hands on Cloud Native setting. This talk delivers a high level but completely practical end to end look at service communications within and without a Kubernetes cluster. Attendees will see how the many facets of Kubernetes networking come together to enable powerful communications solutions first hand. The tutorial begins with the simplest types of service communications, using Kubernetes services, DNS (CoreDNS) and CNI plugins (Cilium) to facilitate interprocess communications and load balancing. The tutorial builds additional scenarios on this base, including ingress (Emissary/Envoy), NodePort / HostPort features, load balancing (Metal-lb) and finally a short look at service mesh functionality (Linkerd). Upon completion of this tutorial, attendees will have a clear understanding of the Kubernetes communications possibilities and pointers to next steps in the learning journey.

[Logs Told Us It Was DNS, It Felt Like DNS, It Had To Be DNS, It Wasn’t DNS](https://sched.co/ytrw)

Friday May 20, 2022 11:55 - 12:30 CEST

It all started with a team reaching out because they had DNS issues during rolling updates. Business as usual when you host hundreds of applications on dozens of Kubernetes clusters… Four weeks later: We are reading kernel code to understand the corner cases of dropping Martian packets. Could this be the connection between gRPC client reconnect algorithms and the overflowing conntrack table we can feel but not see? In time, we solved the issue. And for once… it wasn't DNS! In this talk, we will focus on one of the most complex incidents we have faced in our Kubernetes environment. We will go through the debugging steps in detail, dive deep into the mysterious behaviors we discovered and explain how we finally addressed the incident by simply removing three lines of code.

[Better Bandwidth Management with eBPF](https://sched.co/ytsQ)

Friday May 20, 2022 14:00 - 14:35 CEST

Kubernetes provides many knobs for managing common system resources such as vCPUs and memory limits per Pod, but often forgotten is the effect of unbounded network communication in a cluster. A large churn of packets from several services can starve bandwidth for other services. Also, out of the box TCP congestion management is not optimal for Internet-facing services. In this talk we will explore how eBPF can be leveraged to dynamically insert logic for flexible, efficient and scalable rate limiting and bandwidth management on a per-Pod basis. This talk details: - The scalability limits of token bucket filters by the bandwidth plugin, and why EDT (Earliest Departure Time) combined with eBPF is a major step forward. - How TCP congestion control with BBR can now be leveraged for Pods thanks to eBPF for significantly improving application latency and throughput. - The benefits of enforcing bandwidth limits at the egress point and considerations when to use ingress enforcement.

[A Guided Tour of Cilium Service Mesh](https://sched.co/yttj)

Friday May 20, 2022 16:00 - 16:35 CEST

The Cilium project is adding Service Mesh features to its existing eBPF-enabled, identity-aware Kubernetes networking capabilities. This demo-driven talk explores how this works, and shows why it’s now possible to create a service mesh without sidecars. - Demonstrate why, before eBPF, the sidecar model was necessary for accessing an application pod’s network traffic - Explore how Cilium uses eBPF programs to connect Kubernetes endpoints - Show how this makes the sidecar model unnecessary for identity-aware connectivity - Demonstrate an example Cilium Service Mesh in use - Compare the resources used (in both userspace and the kernel) for both models Along the way, this talk will clarify some container and kernel concepts so that attendees can leave with a mental model of how eBPF-enabled service mesh really works.

Make sure you stop by the Cilium booth to get your Cilium swag and if you are visiting virtually, we will have virtual office hours on Wednesday 18th May 12:30-13:15 CEST. See you there!

#### Meet the Maintainer - Laurent Bernaille

URL: https://cilium.io/blog/2022/05/05/meet-maintainer-laurent-bernaille
Date: 2022-05-05
Categories: Community

_May 5th, 2022_
_Interviewed by: Bill Mulligan, Isovalent_

**How did you first get into technology?**

I have always been very curious about computers. I started learning how they do their work and how to use them in middle school and then a lot more in high school. I ended up studying computer science and after that did a PhD in networking.

**What has been your path to Datadog?**

After my PhD, I was a consultant for about ten years. I spent five years working on the infrastructure components used in High Performance Computing and high frequency trading, so it was all about performance. I have a lot of fun stories from that time. We used microwave networks between Frankfurt and London, because they gave us much lower latency than traditional fiber, but were also much more expensive. An interesting constraint was that the microwave networks didn’t work when it was raining so we then had to fall back on fiber. We could also get creative with servers and for instance use overclocked workstations CPUs which you usually aren’t allowed to do in a datacenter.

I spent the following five years helping customers migrate to public cloud. This eventually turned into helping them transition to containers too. I started to dive into the community to figure out how it was all working, in particular from a [networking perspective](https://www.youtube.com/watch?v=b3XDl0YsVsg). Datadog saw my work and reached out to me. We chatted and they described their project to migrate Datadog to Kubernetes. The product looked great and the challenge was really interesting so I joined!

**How did you get into open source?**

I’ve been using open source for a long time, but I only first started really contributing at Datadog. I might have done some while I was a consultant, but I don’t remember anything significant. I got involved at Datadog because we had very large clusters early on in Kubernetes and were running into some problems. A good example is that we decided very early on to use kube-proxy in IPVS mode, as we knew it would struggle in iptables mode at our scale. IPVS support had just been merged and we were finding many small issues. I started fixing some of them and after a few months I ended up becoming a maintainer of kube-proxy because of these contributions.

**How did you first hear about eBPF and Cilium?**

I first heard about eBPF before I even joined Datadog or used kube-proxy. I also saw one of the first demos of [Cilium at DockerCon 2017](https://www.youtube.com/watch?v=ilKlmTDdFgk) when Thomas did the Star Wars demo.

The key turning point for us with Cilium was when we were having many problems with our existing CNI plugin and kube-proxy. We started talking to people at KubeCon who were in a similar situation to us, running very large clusters and using a CNI plug-in with deep integration into AWS. We were starting to migrate to Cilium so we chatted with people in the hallway track and we became pretty convinced that Cilium was the way to go. As the integration between Cilium, Kubernetes, and AWS became stronger, it became more obvious that we chose the right path.

**How did you become a maintainer of Cilium?**

There was actually a very specific use case that got me into becoming a maintainer. We wanted to use Cilium to perform IPsec encryption for workloads running in the host network namespace, which was not covered at the time. The Isovalent team was extremely helpful with getting this code developed. We were running very recent code and as we were testing and discovering subtle and "fun" edge cases (often involving [MTU](https://github.com/cilium/cilium/pull/9201) and fragmentation), I started contributing patches to address them.

**What are you working on right now in Cilium?**

The last few things I've worked on have been related to datapath optimizations. For instance, we found out during an [incident](https://www.datadoghq.com/blog/engineering/grpc-dns-and-load-balancing-incident/) that when you try to connect to a pod that has been deleted from the nodes, the packets are just dropped and the client can’t know that this IP is unreachable so it will use the normal retries and timeouts. We made a [change](https://github.com/cilium/cilium/pull/18505) to notify the client that this was happening.

And one of the features that I followed very closely, but I didn't directly work on is the support for [prefix delegation in AWS](https://github.com/cilium/cilium/pull/18463). It is a recent feature in AWS that has become very important to us given the size of our VPCs and the integration for it was built by [Hemanth Malla](https://github.com/hemanthmalla) from our team.

**What advice do you have for people just getting into the Cilium community?**

I think the easiest way to get involved is to be a user of Cilium and if you find things that don't work the way you expect, just create issues and discussions and in some cases, try to propose a solution yourself.

I’ve found the maintainers to be extremely helpful and welcoming of new people, which is amazing because not all open source projects are so welcoming and I think it's a great way to start contributing.

**What is one of your hobbies outside coding?**

I’m very excited about traveling! It's something I've really been missing over the last two years. Just to get out and see other countries again. It’s been hard, but I can’t wait to do it.

Catch his two talks about Cilium at KubeCon EU 2022

[Logs Told Us It Was DNS, It Felt Like DNS, It Had To Be DNS, It Wasn’t DNS](https://sched.co/ytrw) Friday May 20, 2022 11:55 - 12:30 CEST

[Cilium: Welcome, Vision and Updates](https://sched.co/ytq0) Thursday May 19, 2022 15:25 - 16:00 CEST

Follow him on Twitter [@lbernail](https://twitter.com/lbernail)

#### Multi Cluster Networking with Cilium and Friends

URL: https://cilium.io/blog/2022/04/12/cilium-multi-cluster-networking
Date: 2022-04-12
Categories: Community

_April 12, 2022_
_Author: Joseph Irving, Platform Lead at RVU (Uswitch)_

##### Multi Cluster Networking with Cilium and Friends

Setting up networking for one Kubernetes cluster can be a challenge but it becomes even more fun once you add multiple clusters into the mix. In this blog, we’ll go over the solutions that [RVU](https://www.rvu.co.uk/) ([Uswitch](https://www.uswitch.com/)) came up with to allow their applications to talk between clusters and the rationale behind them, from [building our own tools](https://labs.rvu.co.uk/multi-cluster-kubernetes-load-balancing-in-aws-with-yggdrasil-c1583ea7d78f) like [Yggdrasil](https://github.com/uswitch/yggdrasil) (Envoy controller) to implementing other tools such as Cilium. We’ll see what benefits and drawbacks the different approaches can have and also explore why we opted to avoid using a traditional service mesh to achieve our multi-cluster networking goals.

###### Background

Over the course of around two years, we had a relatively rapid shift in how we thought about infrastructure at our company. Historically, every product team ran their own infrastructure and were almost entirely independent of each other. We realised that teams were duplicating effort and spending far too much time maintaining and building infrastructure instead of developing new features for our websites. To reduce this replication effort, we decided to centralise the infrastructure on a common platform built on top of Kubernetes.

As we did this one of the biggest concerns from our teams was: what happens if a cluster has an outage? As these clusters had so much of the website running on them this could be very disruptive. To alleviate these concerns, we built multiple Kubernetes clusters for failover. However, this then created the problem of how do we route traffic between them?

###### Building our own tool

Our first approach centred around the open source proxy called [Envoy](https://www.envoyproxy.io/), its ability to dynamically change its configuration via GRPC was of particular interest to us as we saw a way to leverage our existing infrastructure to configure it. Our idea was to configure Envoy to send traffic to our Kubernetes clusters based on the Ingress objects that were already present in the clusters. We built an Envoy control plane called Yggdrasil. It takes Ingress resources across multiple Kubernetes clusters and turns them into Envoy configuration.

If your ingress is in one cluster, Envoy will be configured to send traffic for that host into that cluster.

However, if the ingress is in multiple clusters, Envoy will be configured to load balance across them.

This allowed us to set up HA services spread across multiple clusters and reduce the risk of outages caused by something going wrong in one cluster. \
While this system was primarily intended for user traffic, it inadvertently became a very convenient way for applications to talk to each other. As Envoy would always send traffic to the correct cluster, applications could just talk to Envoy if they wanted to reach another internal service regardless of whether it was running in the same cluster or not.

There were a few major drawbacks with this approach. First, a user would talk to one service which then called another service in another cluster by going through Envoy. This resulted in the request having to go out of the cluster and back again adding a fair amount of latency (P95 of around 20ms per trip). This round trip also caused us to lose identity information. As far as our apps were concerned everything was coming from Envoy.

This kind of cross-cluster functionality was clearly useful to our development teams, however the implementation was less than ideal. We concluded that we would need something more like a service mesh for service to service communication to work in a sensible way.

##### Searching for a Service Mesh

When evaluating the various different service meshes that existed we started with three main requirements:

1. Multi-cluster services - this was the main one, we wanted a way to talk between clusters, service to service.
2. ‘Real’ Pod IPs - meaning that the pods get an IP address that belongs to the VPC they’re running in, instead of being assigned a virtual IP address. This was ‘a nice to have’ as it can simplify networking complexity and allow for more VPC native tools like flow logs, security groups, etc to work with pods easily.
3. Easy to implement on our existing stack - whatever we chose we’d rather it didn’t require significant reworks to how our existing applications work.

With these in mind we began our testing and comparison on the various meshes around at the time and as we did, a few common problems were found.

One of the biggest drawbacks was a reliance on sidecars as the main mechanism for implementing the mesh. This caused all sorts of complications including:

- Service mesh is unavailable during the init phase - there are no sidecars in the init phase so the mesh functionality does not work.
- Job pods get stuck in a state where they never complete - the sidecars need some kind of logic to understand that they’re in a job and should shut down once the primary container has finished its job.
- Startup/Shutdown ordering can be an issue - if the sidecar starts up/shuts down after/before your containers you can get networking issues

While all of these problems do have solutions, it still seemed like a lot of work compared to what we wanted to achieve.

After looking at more ‘traditional’ service meshes, we discovered Cilium which appeared to meet our requirements very well:

- Run as a daemonset - no sidecars required
- Support ‘real’ IPs for Pods using AWS ENIs
- No application changes needed to leverage its capabilities - everything is done at the host level and is transparent to the applications using it.

This all sounded great, so the next step was to try it out!

###### Cilium Time

So let's compare Cilium to our original requirements, first real pod IPs:

Cilium does this in AWS by associating ENIs with your instance that it can then assign additional IPs to, each IP corresponding to one of the pods on the node. This means every pod has an IP that is a valid IP in the VPC you’re running in, allowing all the normal AWS networking features to work with them. No virtual network needed!

Cilium maps pod IPs to services in the same way a more traditional setup would work, but instead of just relying on IPTables rules, Cilium can replace kube-proxy leveraging eBPF which at scale outperforms IPTables and ensures Service Endpoint changes are atomic using eBPF maps.

The (Cluster) Mesh works by allowing Cilium to populate Service Endpoints in each Cluster using eBPF Maps. If you have the same service in two different clusters, Cilium will combine all those pod IPs as possible destinations. Thus, when you talk to the service address, you will get sent to any cluster where that service has endpoints.

This makes the mesh completely transparent to normal applications, they just need to talk to the service in their cluster as they normally would and they’ll be talking across Kubernetes clusters!

Establishing load-balancing between clusters is achieved by defining a Kubernetes service with identical name and namespace in each cluster and adding the annotation io.cilium/global-service: "true" to declare it global. Cilium will automatically perform load-balancing to pods in both clusters.

Our new cross cluster journey for applications is just one hop. From one pod to another pod via their pod IP. Not only does this remove a load of latency but it also maintains identity so we can now use things like Network Policy to control what services can talk to each other.

The actual networking setup for this was also quite simple thanks to the use of AWS Transit Gateway which we use to peer all our clusters together. Since all our pod IPs were ‘real’, as long as routes and Security groups were set up, it all just worked seamlessly.

We even extended this to go across clouds by setting up a VPN between Google Cloud and AWS. Our GKE and EKS clusters talk to each other via the Cilium Clustermesh without any hassle.

###### What’s Next?

Cilium proved to be a great option for us, giving us the functionality we wanted without all the complication that the more traditional service meshes typically imposed on a user. However, it wasn’t completely without its drawbacks. The lack of a dedicated proxy in Cilium meant that apps had to implement things like retries and load-balancing when talking to other services (something our old setup or going through Envoy did for them). Another thing to be aware of is that Cilium differs from more traditional IPTables based implementations, so it is a good idea to familiarise yourself with how Cilium operates and how it uses eBPF maps. The Cilium community was always very helpful if we did ever find any bugs though, so rest assured someone will help you!

It’s also worth noting that Cilium have now started the beta of their Cilium based service mesh which promises to add some more service-meshy features like retries and canary deployments, but with the same transparent sidecar-less approach that vanilla Cilium provides. We’ll be watching this one closely!

#### Help prioritize Cilium Service Mesh features

URL: https://cilium.io/blog/2022/01/25/cilium-service-mesh-beta-feedback
Date: 2022-01-25
Categories: Community

Last month we kicked off beta testing for [Cilium as an efficient data plane implementation for Service Mesh](https://cilium.io/blog/2021/12/01/cilium-service-mesh-beta). Here’s an update on what we’ve learned so far, and some thoughts on next steps. If you’ve tried out the beta already we’d love your feedback in this [short survey](https://docs.google.com/forms/d/e/1FAIpQLScp2TRX63V1Pz0yk4Ec7kN0LnTse6LPDrhBxBV9x2p1IGnDqg/viewform?usp=sf_link).

##### Who joined the beta so far?

We’ve had well over 300 responses to the Cilium Service Mesh Beta so far, with
65% of participants already using Cilium. (If you’re a production user and
your organization is happy for this to be public knowledge, we’d love to see you
make a PR to add yourself to the [list of Cilium users](https://github.com/cilium/cilium/blob/master/USERS.md)!)

##### What features are most important?

The term “Service Mesh” covers many different features, and there’s not really industry-wide consensus on which of them are absolutely essential for something to call itself a service mesh - ask any vendor and they’ll tell you it’s the set of features their product offers! So we asked beta users what features they are most interested in.

**Visibility** is hands-down the most important feature, according to this survey, with 97% saying it’s a “must have” and not a single person responding that it’s unnecessary.

The second most-requested feature is **traffic encryption**, which over
two-thirds say is essential. In the service mesh world this is often assumed to
be implemented through mTLS (mutual TLS) between services, but mTLS provides not
just encryption but also service authentication. The **authentication** aspect is seen
as essential by a little under half of our beta users. Why doesn't everyone who
needs encryption also need authentication at the service level? Well, if your requirement for
traffic encryption is to ensure that it’s incomprehensible to a bad actor who
managed to compromise your network, this can be achieved using transparent
encryption (IPSec or WireGuard). But if you have legitimate services that are
not all authorized to communicate with each other, an mTLS approach can
make sense. mTLS requires L7 termination at the proxy, whereas transparent
encryption happens within the network layer, so there is likely to be a
performance tradeoff - we hope to bring you measurable benchmarks on that in the coming months.

The next most requested set of features feature is what we described in the
survey as “**Canary roll-outs / A/B testing**”. These might also be termed
“traffic splitting” or “load balancing”. At the moment in the service mesh beta
we have an example of a configuration that [splits
traffic](https://github.com/cilium/cilium-service-mesh-beta/tree/main/l7-traffic-management)
between two instances of a back-end service. This is achieved using a
CiliumEnvoyConfig CRD, which exposes the raw configuration of the Envoy proxy
instance. It seems highly likely that users will want a friendlier, higher level
abstraction for this kind of configuration. We could extend Cilium to accept and
understand control plane configurations that are already in existence, such as
SMI, Istio or Linkerd CRDs - in fact, implementing support for one wouldn't rule
out others in the future. Preliminary discussion in the [#service-mesh-beta Slack
channel](https://cilium.slack.com/archives/C02QKQDTVDX) suggests that SMI would be a popular option - what’s your view?

Kubernetes **Ingress** capabilities are required to get external traffic to your
services, although whether you see them as inherently part of a service mesh or
as a separate entity is a topic for debate. We have examples for HTTP and gRPC
traffic ingress, including TLS support, as [part of the beta](https://github.com/cilium/cilium-service-mesh-beta/tree/main/kubernetes-ingress).

The last group of features that folks are interested in with service mesh
include **rate limiting**, **retries** and **circuit breaking**, with between a
third and half of testers rating these as “must have”. Again, we’re really interested
to hear your thoughts on the best way to configure these kinds of features,
especially as these aren’t included in the SMI Spec. What’s your preferred way
to control these features?

We also offered free-form text for people to tell us about other features they would like to see in a service mesh implementation, and a couple of suggestions stood out as popular.

- **Multi-cluster** support was a popular response. Cilium already supports service routing across multiple clusters through Cilium Clustermesh, so it should be straightforward evolution for Cilium Service Mesh to be easily configured to run across multiple clusters.

- Several people called out **network policy** support. This is another benefit of supporting service mesh capabilities from within the networking data plane - it’s natural to consider service-level network policies when designing how your service mesh should operate.

##### What’s next in the Cilium Service Mesh beta?

The biggest open question for Cilium Service Mesh is what the control plane should look like. So far we’re exposing raw Envoy configuration through the CiliumEnvoyConfig CRD, but we need more user-friendly approach, and (as mentioned above) there are several existing de-facto standards that it’s likely we could adopt and be compatible with.

There are several features already supported through Envoy, including mutual TLS support, and additional traffic management features like circuit breaking, that we’d like to make simpler to configure.

As well as consistency with pre-existing control plane configurations, there is work to be done to ensure RBAC controls for service mesh configuration.

There are several other goals in our near-term sights:

- One technical issue that came to light early on is that some of the examples need Cilium in [tunnelling mode](https://github.com/cilium/cilium-service-mesh-beta/issues/9), so that’s being worked on right now.
- Cilium and Hubble export lots of [powerful
 metrics](https://docs.cilium.io/en/stable/operations/metrics/#hubble-exported-metrics)
 to Prometheus and Grafana already, but we want examples specific to service
 mesh. For example, it should be easy to answer questions about service latency,
 and to provide the visibility that our beta users have strongly indicated they
 need.
- Helm chart configuration - the Cilium CLI is a great way to configure Cilium out of the box, but we want to give some Helm chart examples
- Performance benchmarks - the Isovalent team already [shared some initial results](https://isovalent.com/blog/post/2021-12-08-ebpf-servicemesh) on the gains made by removing sidecars, but for certain proof we want to design good tests that can be independently verified.
- Integrating TLS terminations for ingress with automated certificate management

##### Get involved!

We’d love to hear what you think should be added to Cilium Service Mesh over the coming months. The easiest way to participate is to [answer our latest survey](https://docs.google.com/forms/d/e/1FAIpQLScp2TRX63V1Pz0yk4Ec7kN0LnTse6LPDrhBxBV9x2p1IGnDqg/viewform?usp=sf_link).

We’d welcome more example configurations in the beta repository, so if you have tried out some additional features, or used the service mesh with interesting sample apps, why not open a PR to share it with the community?

If you’d like to get involved with working on the code itself, we have a
[developer guide](https://docs.cilium.io/en/stable/contributing/development/)
and we’d welcome you at the [weekly developer
call](https://github.com/cilium/cilium#weekly-developer-meeting). And of course
the [Cilium & eBPF Slack channel](http://slack.cilium.io) is online 24/7 for
your questions and comments! Join the \#service-mesh-beta channel for discussions about these
features.

#### An Amazing Milestone for Cilium - 10,000 stars!

URL: https://cilium.io/blog/2021/12/20/github-10k
Date: 2021-12-20
Categories: Community

We recently passed an incredible milestone in Cilium’s journey - over 10,000 stars on GitHub. Thanks to each and every one of you who has clicked that star button!

This is a fitting close to a wonderful year for the Cilium project, driven by our fantastic contributors and community. Just to mention a few highlights:

- We [joined the CNCF](https://cilium.io/blog/2021/10/13/cilium-joins-cncf) at Incubation level, finding our home alongside many other great cloud native infrastructure projects

- We delivered two major releases in
 [1.10](https://cilium.io/blog/2021/05/20/cilium-110) and
 [1.11](https://isovalent.com/blog/post/2021-12-release-111), with a range of
 new capabilities too numerous to list here.

- AWS picked Cilium as the [networking and security layer for EKS Anywhere](https://www.isovalent.com/blog/post/2021-09-aws-eks-anywhere-chooses-cilium), and Google’s [Cilium-based GKE Dataplane V2](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine) went GA

- We’ve welcomed over 9,000 people on [Cilium Slack](http://slack.cilium.io) (and at current growth rates it won’t be long before we need another digit there too!)

- [Cilium Service Mesh is in beta](https://cilium.io/blog/2021/12/01/cilium-service-mesh-beta), with hundreds of people already participating, modernizing the data path in Service mesh implementations.

All of this is possible thanks to every single person who contributes. Whether
you’re working full-time on Cilium, raising issues, contributing to the docs, or
sharing your knowledge of Cilium and eBPF, you’re playing a valuable part in
Cilium’s progress. We hope you’ll join in celebrating this 10,000 star milestone
success!

##### Special edition designs

As a small thank you, we have created **downloadable desktop wallpapers**, designed by Vadim
Shchekoldin - the artist behind many of our wonderful graphics - to decorate
your desktop. [Download the free Cilium 10k stars wallpapers here](https://drive.google.com/file/d/1HgZn7jxQGSths5Ygmq4fuUt5-kKrV6ox/view?usp=sharing).

We’ve also got a **special edition T-shirt** design in the [Cilium swag store](https://cilium.myspreadshop.com/10k+stars+on+github?idea=61c05a15e41cc92c76d59deb) - check it out! _(Note: there's a country selector at the bottom of the swag store site, so you can pick the best one for your location.)_

#### Try eBPF-powered Cilium Service Mesh - join the beta program!

URL: https://cilium.io/blog/2021/12/01/cilium-service-mesh-beta
Date: 2021-12-02
Categories: Community

You already know that Cilium accelerates networking, and provides security and
observability in Kubernetes, using the power of eBPF. Now Cilium is bringing
those eBPF strengths to the world of Service Mesh. Cilium Service Mesh features
eBPF-powered connectivity, traffic management, security and observability, and
we’re inviting you to be amongst the first to try out these new capabilities as
part of the [Cilium Service Mesh Beta Program](https://forms.gle/j9fwhAC6HnHRJQKeA).

##### What does Cilium bring to Service Mesh?

The term “Service Mesh” encompasses a wide range of features, including service
discovery, encryption, service authentication, load balancing, observability,
canary roll-outs and more. Some of these features overlap with established
Cilium capabilities - for example, Cilium has offered load balancing, Kubernetes
service awareness, multi-cluster connectivity, and visibility of network traffic
at layer 3-7, for ages. Cilium already uses Envoy for L7 policy and
observability for some protocols, and this same component is used as the sidecar
proxy in many popular Service Mesh implementations. So it's a natural step to
extend Cilium to offer more of the features commonly associated with Service Mesh.

In a typical Service Mesh, all network packets need to pass through a sidecar
proxy container on their path to or from the application container in a Pod. In
Cilium Service Mesh, we’re moving that proxy container onto the host and kernel
so that sidecars for each application pod are no longer required. Because eBPF
allows us to intercept packets at the socket as well as at the network
interface, Cilium can dramatically shorten the overall path for each packet.
(Read more about sidecarless,
eBPF-based Service Mesh.)

Alongside the upcoming Cilium 1.11 release, a new Cilium Service Mesh beta build
will shortly be available that features:

- L7 Traffic Management & Load-balancing (HTTP, gRPC, …)
- Topology Aware Routing across clusters, clouds, and premises
- TLS Termination
- Canary Rollouts, Retries, Rate Limiting, Circuit Breaking, etc, configured
 through Envoy
- Tracing with OpenTelemetry & Jaeger integration
- Built-in Kubernetes Ingress Support

As the beta progresses we'll update participants with the status of new and
improving capabilities.

##### Why is there a Beta program?

We’d like feedback from an active group of Cilium users who can try out Cilium
Service Mesh features and help us make sure we're providing a stable, performant
service mesh.

But the beta isn’t just about testing to make sure that connectivity works; we’d love to better understand your use cases, and get your help in prioritizing the service mesh features you need. We want to be sure we’re delivering the right user experience, before committing these features into a Cilium release.

The initial beta build enables service mesh functionality with distributed Envoy configuration available via a CRD, or by configuring standard Kubernetes Ingress and Services objects. However, Cilium’s service mesh architecture is designed so that it can be integrated with any service mesh control plane and specification. We would love to hear from you which ones are of particular interest.

During the beta phase, we’ll develop and maintain Service Mesh capabilities in a separate GitHub branch. This gives us the flexibility to make potentially non-backwards-compatible changes, for example changes to CRDs.

##### What do I need to do as a Beta tester?

To join the Beta program we’d first of all like you to [complete this form](https://forms.gle/j9fwhAC6HnHRJQKeA) to tell us a little bit about your service mesh needs.

We’ll send you information about how to get started with Cilium Service Mesh, and we’ll update you on a regular basis about the status of different capabilities. We’ll also send you short surveys to ask for your feedback during the program.

There will be a dedicated Slack channel for Service Mesh so you can share your experiences with each other, and interact directly with the Cilium team members working on implementation.

The intention is that core Cilium Service Mesh features will be ready for production use in Q1 2022.

##### Can I help develop Cilium Service Mesh?

Yes! If you haven’t yet contributed to the Cilium Project we’d encourage you to come and say hello in the [Cilium Slack](http://slack.cilium.io) or to join the [weekly community meeting](https://docs.cilium.io/en/stable/community/). We have lots of issues on [Cilium GitHub](http://github.com/cilium/cilium) marked as Good First Issues, and there’s [documentation for developers here](https://docs.cilium.io/en/stable/contributing/development/).

##### How do I join the Beta Program?

If you’d like to take part as a Service Mesh user, please [fill out this form](https://forms.gle/j9fwhAC6HnHRJQKeA) and we’ll be in touch soon!

#### New Weekly Community InstallFest and Live QA

URL: https://cilium.io/blog/2021/10/25/weekly-installfest
Date: 2021-10-25
Categories: Community

With [Cilium joining the CNCF](https://www.cncf.io/blog/2021/10/13/cilium-joins-cncf-as-an-incubating-project/)
we have had a lot of people showing interest in Cilium and wanting to learn
about it. We are excited to introduce two new community events available to
everybody in the community to learn and explore Cilium:

- [Weekly Cilium Introduction with Live Q\&A](#weekly-cilium-introduction-with-live-qa)
- [Weekly Community InstallFest](#weekly-community-installfest)
- [Further Reading](#further-reading)

##### Weekly Cilium Introduction with Live Q&A

The weekly Cilium Introduction hosted by Cilium Co-Creator Thomas Graf is an
ideal venue to learn about Cilium if you have never heard about it, or if you
already know about Cilium but have questions that you want to get answers to
by one of the Cilium maintainers.

The Cilium Introduction will answer questions such as:

- What is Cilium? What is eBPF?
- Why did we create Cilium?
- What are the key Cilium use cases?
- Why are Google, AWS, Alibaba and other cloud providers using Cilium?
- Cilium AMA: Ask any question you have about Cilium.

The weekly Cilium introduction is available to everyone in the community.
Simply click the link below to sign up for one of the available time slots:

[Book your seat!](https://calendly.com/cilium-events/cilium-introduction?month=2021-10&date=2021-10-26)

##### Weekly Community InstallFest

The weekly InstallFest is ideal for everyone who wants to install Cilium and
get hands-on. In this weekly session, you will learn how to install Cilium and
make your steps in using Hubble for observability, apply some NetworkPolicy
and explore next steps from there.

The installfest is available to everyone in the community. Click the link below
to sign up for a time slot that suits you and the timezone you live in. You
will receive additional information on how to set up the environment in the
confirmation to the event

- [Join InstallFest in Europe](https://calendly.com/cilium-events/cilim-installfest-emea)
- [Join InstallFest in Americas](https://calendly.com/cilium-events/cilium-installfest-na)
- [cilium/installfest](https://github.com/cilium/installfest)

##### Further Reading

- [Learn more about Cilium](https://cilium.io/learn)
- [Join the Cilium Slack](https://slack.cilium.io)

#### Cilium joins the CNCF

URL: https://cilium.io/blog/2021/10/13/cilium-joins-cncf
Date: 2021-10-13
Categories: Community

I'm beyond thrilled that Cilium has joined the CNCF as an Incubation project!
I'm excited about it not just as an advocate for the Cilium project, but also as
an Isovalent team member, and in my role as Chair of the CNCF's Technical
Oversight Committee - and I'd like to share why.

##### eBPF as the foundational technology

The first time I heard about Cilium was in [Thomas Graf's presentation at
DockerCon 2017](https://www.youtube.com/watch?v=ilKlmTDdFgk). By using
[eBPF](http://ebpf.io) to hook directly into the path of a network packet, it
seemed clear to me, even back then, that this was bound to bring performance
improvements over a traditional networking stack.

Over the last couple of years I've been telling anyone who'll listen that [eBPF
is creating a revolution](https://youtu.be/qsnR-s4XuGo?t=54) in the way we
instrument applications, especially in cloud native environments.

But until recently the biggest impediment to adoption has been that users need a
Linux kernel that's recent enough to include eBPF capabilities. It takes a while
for kernel releases to make it into the distributions that are commonly used in
production environments. We've now passed a watershed, where even [RHEL users
have 4.18 kernels and can use eBPF capabilities](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/8.4_release_notes/new-features#enhancement_kernel).
As a result we're seeing Cilium adoption in more and more deployments, with the
field hardening that implies.

##### The natural default CNI

The big public cloud providers have taken an interest along the way. Google uses
Cilium for the [GKE data plane](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine),
and AWS recently picked Cilium for [networking in EKS Anywhere](https://www.isovalent.com/blog/post/2021-09-aws-eks-anywhere-chooses-cilium).
With Cilium joining the CNCF as the only Incubation- or Graduation-level CNI
solution, it's rapidly becoming the natural default for Kubernetes clusters
everywhere. It already has well over 9,000 GitHub stargazers. We know that our
current [published list of users](https://github.com/cilium/cilium/blob/master/USERS.md) is far from
comprehensive - if your organization is also using Cilium we'd be delighted to
see a pull request adding it.

Cilium is, of course, much more than "just" a Kubernetes CNI. In several cases
it's used outside a Kubernetes environment to provide features like high
performance load balancing. And of all the CNIs we believe it has the greatest
range of capabilities for [integrating cloud native and traditional network
environments](https://cilium.io/blog/2021/05/20/cilium-110).

Cilium also includes [Hubble](https://github.com/cilium/hubble), a tool for
providing detailed observability into Kubernetes networking flows, and the
Hubble UI for displaying this information. Add to this the advanced network
policy that Cilium supports, and you can start to see how eBPF enables
networking, observability and security, all in one project. Integrating with
Envoy, Cilium is also poised to offer kernel-based service mesh capabilities.

##### A community project at heart

One of the things that attracted me to [Isovalent](http://isovalent.com) when I
first started talking with Dan & Thomas was their inherent understanding of open
source communities, and the way that Cilium has consistently been run as a
community-oriented project even while owned by the company. We do assign
resources to an enterprise distribution, and additional enterprise features -
after all, [you have to have something to sell](https://youtu.be/2zF-XcDkTiY?t=350) as a business. But this goes hand in
hand with devoting significant efforts into the open source project. By joining
the CNCF, the Cilium project is signaling its intent to remain, forever, a
community project with its core functionality available to all.

As well as Isovalent, we have representation amongst our [maintainers and committers](https://raw.githubusercontent.com/cilium/cilium/master/MAINTAINERS.md) from many other organizations including Google, Red Hat, SUSE and Datadog. We
know there are other companies interested to get involved for whom the neutral
ownership by CNCF will be a real encouragement. The project governance already
ensures that [no single company](https://docs.cilium.io/en/stable/contributing/governance/commit_access/#company-block-vote-limit)
can take control.

Because Cilium has long been run by its community, we don't expect to see any
dramatic overnight changes in the way you use, interact with or contribute to
it.

- The best place to start is our [Slack community](https://slack.cilium.io) where you'll find thousands of
 people and a lot of help, should you need it

- One new initiative is our weekly [Cilium InstallFest](https://cilium.io/learn) - book your place today!

- There is a weekly Cilium developer meeting - find all the [details on
 GitHub](https://github.com/cilium/cilium#community) - and a number of
 [Special Interest Groups](https://docs.cilium.io/en/stable/community/#special-interest-groups)
 for different aspects of Cilium, such as Hubble, Policy or Datapath

Whether you're using or contributing to Cilium, we look forward to welcoming you
in our community!

#### eBPF and Cilium Office Hours - Highlights from Season 1

URL: https://cilium.io/blog/2021/08/03/best-of-echo
Date: 2021-08-03
Categories: Community

[eBPF and Cilium Office Hours](https://www.youtube.com/playlist?list=PLDg_GiBbAx-mY3VFLPbLHcxo6wUjejAOC)
(also known as eCHO) is a weekly YouTube livestream, hosted by Liz Rice and Duffie Cooley, about the world of
eBPF, Cilium and related projects. Most weeks we welcome a guest to show us what
they have been working on and share their insights. In place of this week's live episode (while we
take a well-deserved summer break!) here's a look back at some of our favourite
moments from what we might call Season 1.

##### Thomas introduces the Cilium CLI

We started the series with a bang: Thomas Graf joined Liz for a demo-rich [introduction to Cilium](https://youtu.be/80OYrzS1dCA?t=768). For many viewers this was the first peak at the new [Cilium CLI added in 1.10](/blog/2021/05/20/cilium-110#cli). We saw how easy it is to use this to install Cilium and Hubble and get a quick view of status and network connectivity.

##### Duffie debugs a routing issue

For another look at the Cilium CLI check out the [episode where Duffie dives
into it](https://www.youtube.com/watch?v=ndjmaM1i0WQ) - and ends up debugging a
curious routing issue along the way!

##### Flamegraphs

Thomas returned to the eCHO show a few weeks later to share some [performance
benchmarking results](https://youtu.be/2lGag_j4dIw). The TL;DR from this show:
eBPF implementations shine when it comes to network performance. Thomas even
shared [flame graphs](https://youtu.be/2lGag_j4dIw?t=3038) that show why this is
the case.

##### If it's good enough for Linus

Given that [Linus Torvalds himself loves WireGuard](https://youtu.be/-awkPi3D60E?t=540),
this seems like a technology worth knowing about. This episode with Martynas
Pumputis is packed with interesting content, from background on packet flows in
Cilium, through the story of how Martynas and Sebastian Wicki worked on the
implementation, to a demo that makes the theory concrete.

##### It's always DNS

You know how whenever something goes wrong "it's always DNS"? [Laurent Bernaille from Datadog shared some war stories](https://youtu.be/mo0RIJZypbQ?t=168) that illustrated some of the unexpectedly wide range of ways in which DNS can indeed be the cause of failure.

##### eBPF is not just for Linux

One of the biggest developments in eBPF this year has been the revelation that
it's being supported in Windows. [Dave Thaler from Microsoft](https://youtu.be/LrrV-eo6fug?t=317) joined eCHO to
explain how what we had all previously thought of as a Linux technology actually
makes sense in other kernels too.

##### Diving into XDP

Kernel maintainer Daniel Borkmann gave us a [deep dive into XDP](https://youtu.be/OIyPm6K4ooY), from the first
suggestion that perhaps eBPF could be added to network drivers through to some
of the use-cases for XDP in Cilium today, including load balancing and
kube-proxy replacement.

##### Networking explainers

If you want to learn about some of the networking capabilities that Cilium
provides, check out the great explanations that Joe Stringer provides in episode
6 about features added in Cilium 1.10. For example, this episode covers what we mean by an [Egress IP Gateway](https://youtu.be/y5xcvr_fgxc?t=463), or [BGP Service Announcement](https://youtu.be/y5xcvr_fgxc?t=924).

##### Coming up in Season 2

We've had many more wonderful guests over the last few weeks, including [Kris
Nóva](https://youtu.be/d2I2kLd7AwU), [Antonio Ojea](https://youtu.be/yabzjJMdI08), and [Itay Shakury](https://youtu.be/aOgidMoPz9A). If you have ideas for folks you'd like to
see joining us on eCHO, we'd be very happy to see your suggestions as issues in
the [eCHO GitHub repo](https://github.com/isovalent/eCHO).

#### Cilium 1.10: WireGuard, BGP Support, Egress IP Gateway, New Cilium CLI, XDP Load Balancer, Alibaba Cloud Integration and more

URL: https://cilium.io/blog/2021/05/20/cilium-110
Date: 2021-05-20
Categories: Release

The Cilium core team are excited to announce the Cilium 1.10 release. A total
of 2042 new commits have been contributed by a growing community of 280
developers. Contributions have come from a wide list of different companies
including Alibaba, Datadog, Google, Isovalent, SuSE, Palantir among many
others. We heard from users who are increasingly interested in better
integration with the broader network, whether you're running Cilium in your own
datacenter environments or in the cloud. Thus, a major focus of this release
was ensuring a seamless integration between the Kubernetes network and
enterprise on-prem infrastructure.

- **Egress IP Gateway**: When integrating cloud-native applications with legacy
 counterparts operating outside the Kubernetes cluster, IP Address Management
 can be a pain point as Pod IPs are more dynamic than traditional network
 endpoints. Through a new Kubernetes CRD, static IPs can now be associated
 with traffic as the packets egress the Kubernetes cluster. This allows
 external firewalls to see consistent IPs and filter traffic for specific pods
 purely using the assigned static IP address.
 ([More details](#egressgateway))
- **Integrated BGP Support**: We’ve simplified the ability to advertise
 Kubernetes service IPs via BGP to allow workloads outside the Kubernetes
 environment to easily communicate with Cloud Native applications. With 1.10
 we've incorporated BGP service announcement support directly into Cilium
 so that these services can be exposed without the need for any additional
 external components.
 ([More details](#bgp))
- **Standalone Load Balancer**: Cilium's high performance, robust
 load balancing implementation is tuned for the scale and churn of
 cloud native environments. Now you can replace expensive legacy boxes in
 your network with Cilium as a standalone load balancer. This unlocks the
 potential of DSR and Maglev for handling north/south traffic in
 on-premises environments without requiring Kubernetes to manage the network
 border.
 ([More details](#standalonelb))
 - **PCAP Recorder** provides network observability into the
 Standalone Load Balancer by enabling full or partial packet capture for
 traffic matching specific n-tuple filters. ([More details](#pcap))

Beyond the focus on integrating better with the broader network, we’ve invested
in making it **easier to install and run Cilium** regardless of environment,
further **improved performance**, and added a whole set of highly requested
features:

- **WireGuard® Support**: One of the most hotly requested features, Cilium now
 supports encrypting traffic between pods in the cluster using the modern
 WireGuard protocol as an alternative to the existing IPsec implementation.
 ([More details](#wireguard))
- **New Cluster Administration CLI**: An experimental new Cilium command-line
 interface simplifies installation, troubleshooting and feature enablement
 by automatically detecting your environment constraints and configuring
 Cilium with the best set of options.
 ([More details](#cli))
- **Better performance than ever**: Cilium now has the ability to skip
 Netfilter connection tracking where it is not needed, has improved
 North/South service handling, and we've doubled throughput performance
 through improved Linux tunneling. Check all this out together with the new
 [tuning guide][] to help you get the most from your network!
 ([More details](#performance))
- **Lots of other improvements**: This release brings support for [Kubernetes
 Dual-Stack Services](#dualstack), [Native IPAM for Alibaba Cloud](#alibaba),
 and is the first release to distribute [official images for ARM64](#arm64).
 New guides for Rancher and Rancher Kubernetes Engine make it easier
 to deploy Cilium in on-prem Kubernetes environments.

###### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms such as Kubernetes.

At the foundation of Cilium is a new Linux kernel technology called eBPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. eBPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption as well as network and service security. Besides
providing traditional network-level security, the flexibility of eBPF enables
security with the context of application protocols and DNS requests/responses.
Cilium is tightly integrated with Envoy and provides an extension framework
based on Go. Because eBPF runs inside the Linux kernel, all Cilium
functionality can be applied without any changes to the application code or
container configuration.

See the section **[Introduction to Cilium]** for a more detailed general
introduction to Cilium.

###### Egress IP Gateway

_Primary contributors: Yongkun Gui (Google) and Bolun Zhao (Google)_

As Kubernetes became the de facto standard platform to containerize and
modernize application workloads, a significant pain point became connecting
these new services to components in the legacy environments. Traditional
workloads have a fixed and unique IP that can be recognized by a firewall.
Traffic coming from a containerized application will come from many different
IPs as containers spin up and down as needed and will most often present as the
IP of the node on which each container sits. This makes it hard to set up
security boundaries for applications and to audit their behavior.

The 1.10 release brings the egress gateway capability to Cilium. Cilium now
allows users to specify an egress NAT policy, such that egress traffic for the
selected pods will be masqueraded with a consistent, user-defined IP address.
As the following diagram shows, instead of directly egressing via eth0, traffic
from pods running in the worker node egress the cluster via the gateway node.
At this node, SNAT is applied to provide the consistent external IP for
outbound traffic. From there, networking components can be configured to treat
traffic differently depending on the source IP address. For instance, a legacy
firewall is able to filter traffic that originates from pods with particular
labels, by matching purely on the source IP address of the inbound traffic.

As an example, the following policy instructs Cilium to use egress IP
`192.168.33.100` for traffic from all pods in `default` namespace and destined
to the external CIDR `192.168.33.13/32`. That egress traffic will be redirected
to the (gateway) node to which `192.168.33.100` was assigned. More information
about the egress gateway can be found in the
[getting started guide](https://docs.cilium.io/en/v1.10/gettingstarted/egress-gateway/).

```yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumEgressNATPolicy
metadata:
  name: egress-sample
spec:
  egress:
    - podSelector:
        matchLabels:
          # The following label selects default namespace
          io.kubernetes.pod.namespace: default
  destinationCIDRs:
    - 192.168.33.13/32
  egressSourceIP: '192.168.33.100'
```

##### BGP for LoadBalancer VIP

_Primary contributors: Chris Tarazi (Isovalent)_

As Kubernetes becomes more pervasive in on-premise environments, users
increasingly have both traditional applications and Cloud Native applications
in their environments. In order to connect them together and allow outside
access, a mechanism to integrate Kubernetes and the existing network
infrastructure running BGP is needed.

The 1.10 release of Cilium brings integrated support for BGP, exposing
Kubernetes to the outside and all the while simplifying users' deployments.
The integration is done via [MetalLB](https://metallb.universe.tf/), leveraging
the effective implementation of the service IP allocator and the L3 protocol
support for BGP. With this, Cilium is now able to allocate service IPs for
services of type `LoadBalancer` and announce them via BGP to their BGP
routers. Now, services are able to be reached externally from traffic outside
of the cluster, without any additional components.

In the future, we plan to support announcing Pod CIDRs and the [Egress
IP Gateway](#egressgateway) via BGP as well. This will further improve the
bridging between Cloud Native and traditional environments.

The following is an example `ConfigMap` that's used to configure Cilium's BGP
feature. The two main aspects of the configuration are `peers` and
`address-pools`. The former is for connecting with the existing BGP routers
in the network, so an IP address and an ASN are required. The latter is the
pool from which Cilium will allocate an IP to a `LoadBalancer` service. For a
bit of background, services of type `LoadBalancer` in Kubernetes rely on
another component to fill the gap of allocating external IPs, which is why
Cilium needs to be configured with that information.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: bgp-config
  namespace: kube-system
data:
  config.yaml: |
    peers:
    - peer-address: 10.0.0.1
      peer-asn: 64512
      my-asn: 64512
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.0.2.0/24
```

See the [BGP guide](https://docs.cilium.io/en/v1.10/gettingstarted/bgp/) in the
documentation on how to get started.

##### Alibaba Cloud Integration

_Primary contributors: Bokang Li (Alibaba)_

Cilium has been progressively adding support for allocating IP addresses
directly from cloud APIs, notably from [AWS ENI][] (v1.6) and [Azure][] (v1.8).
This release adds an IPAM mode for Alibaba Cloud, allowing Cilium to utilize
Alibaba Cloud ENI (Elastic Network Interface) for Pod IPs.The architecture
ensures that only a single operator communicates with the Alibaba Cloud APIs
to avoid rate-limiting issues in large clusters.

In this mode, pods are assigned with ENI IPs, which are directly routable in the
AlibabaCloud VPC. This simplifies communication of pod traffic within VPCs and
avoids the need for SNAT. With the cloud-native network solution, user can
easily use cloud resource like Cloud Enterprise Network (CEN) or VPN gateway to
connect to your on-premises data centers.

See the [Alibaba Cloud Getting Started Guide][] for more details.

###### Transparent Encryption with WireGuard

_Primary contributors: Martynas Pumputis (Isovalent) and Sebastian Wicki (Isovalent)_

With the 1.10 release, we are adding support for transparent encryption via
the [WireGuard](https://www.wireguard.com/) protocol. Transparent encryption
is a feature which ensures that traffic between Cilium-managed pods is always
sent through a secure tunnel, thereby allowing users to run Kubernetes in
untrusted networks.

Cilium had support for transparent encryption via IPSec already since version
1.4, but we were motivated to add WireGuard as an alternative option for
multiple reasons. Firstly, WireGuard is a very opinionated protocol and does
not allow the tuning of parameters such as the key size and cipher suite,
which makes it very easy to use and simplifies deployment and operations.
The encryption key pair for each node is automatically generated by Cilium and
key rotation is performed transparently by the WireGuard kernel module.

WireGuard also has performance advantages over IPSec for certain workloads. In
our recent [CNI performance analysis blog post](https://cilium.io/blog/2021/05/11/cni-benchmark#encryption),
we evaluated both protocols and found that WireGuard can achieve very high
throughput for streaming workloads, but IPSec still can achieve better latency,
and the encryption algorithm can be offloaded to the CPU in certain cases.
The graph above shows a short excerpt from our experiments, make sure
to read the full blog post for more details and explanations.

Another technical advantage of WireGuard is that it integrates well into the
architecture of the Cilium datapath. Because the secure tunnel is exposed as
a Linux network interface, WireGuard makes it very easy to mark packets
originating from containers for encryption and subsequently forward them to
the secure tunnel device. The animation below shows the life of a WireGuard
encrypted packet in Cilium:

WireGuard encryption in Cilium 1.10 is supported both in direct routing and
tunneling mode, and may also be enabled in managed Kubernetes environments if
the underlying Linux kernel supports it. Support for host-level encryption
and Cilium's L7 policy enforcement and visibility features is currently lacking
and will be added in an upcoming Cilium releases.

If you want to know more about the WireGuard integration in Cilium, make sure
to watch to the recording of the [**eCHO episode 3 livestream**](https://www.youtube.com/watch?v=-awkPi3D60E)
and read our [**getting started guide**](https://docs.cilium.io/en/v1.10/gettingstarted/encryption-wireguard/)
in the Cilium documentation.

###### Kubernetes Dual-Stack support

_Primary contributors: Deepesh Pathak (Microsoft), André Martins (Isovalent)_

Cilium has IPv4/v6 dual-stack support since it was created. In fact, in Cilium's
_very early_ prototype, we started out with an IPv6-only implementation. Given
Kubernetes was lacking official dual-stack integration for some time now, the
Cilium IPv6 support for Pods and Services was a bit cumbersome to operate.
With the last Kubernetes stable version that was released this has changed now:
Kubernetes 1.21 introduced [beta support for dual-stack enabled by default](https://kubernetes.io/docs/concepts/services-networking/dual-stack/).
With Cilium 1.10, we integrated the Kubernetes dual-stack support, so that
Pods are now properly integrated to allow for dual-stack IP address allocation
and, similarly, Service IPv4/v6 addressing propagates down into the Cilium
eBPF kube-proxy-replacement.

IPv4/v6 dual-stack operation can be enabled in Cilium using the helm
option `ipv6.enabled`:

```shell-session
$ helm install cilium cilium/cilium --version 1.10.0 \
   --namespace kube-system \
   --set ipv4.enabled=true \
   --set ipv6.enabled=true

$ kubectl get cep -n kube-system
NAME                       ENDPOINT ID   IDENTITY ID   ENDPOINT STATE   IPV4            IPV6
coredns-755cd654d4-msqgm   2205          11023         ready            10.16.115.242   fd02::71cf
```

###### Official ARM64 Cilium images

_Primary contributors: André Martins (Isovalent)_

In previous releases, the Cilium core team have made snapshots available on
ARM64 platform for initial testing. However, until now the official images for
Cilium have not yet been prepared for this platform. During this development
cycle, we worked on several improvements to the image build infrastructure for
the project to allow automatic build of multi-arch docker images for both
linux/amd64 and linux/arm64 platforms. Beginning with Cilium 1.10, the official
Cilium repositories on Docker Hub and Quay.io will host these multi-arch
images.

Installing Cilium on ARM64 works similarly to the setup on other platforms,
using the same image tags and digests as the AMD64 docker images. This unlocks the
ability to run Cilium on a range of new hardware including AWS Graviton
instances and Azure Linux/ARM64 pipelines. Cilium has even been spotted in the
wild, running on a smartphone!

<blockquote class="twitter-tweet"> Armbian trunk on a Snapdragon 835 (OnePlus 5) running k8s and Cilium eBPF @armbian @ciliumproject @kubernetesio pic.twitter.com/QXMCiox12X &mdash; Ricardo Pardini (@SilvioSantoZ) May 12, 2021 </blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

###### Cilium CLI

A new [CLI](https://github.com/cilium/cilium-cli) is available that can be used
to install and manage Cilium clusters. The CLI talks directly to the Kubernetes
API and offers a variety of functionality to manage different aspects of
Cilium. The CLI is backward compatible to older versions of Cilium.

- Determine the cluster-wide status of a Cilium installation and quickly expose
 failure states.
- Automatically detect the ideal configuration options for a particular
 Kubernetes environment (datapath, IPAM,...).
- Enable Hubble observability and Relay components.
- Manage multi-cluster connections and status.
- Perform connectivity and conformance tests to validate networking
 capabilities in a cluster.

###### Cluster-wide Cilium Status

The `cilium status` command allows to quickly determine the status of any
Cilium installation. It will indicate the number of unhealthy Cilium components
and quickly expose failure states in any of the nodes:

```
$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Containers:       cilium             Running: 2
                  cilium-operator    Running: 2
Image versions    cilium             quay.io/cilium/cilium:v1.10.0: 2
                  cilium-operator    quay.io/cilium/operator-generic:v1.10.0: 2
```

###### Connectivity & Conformance Test

A new connectivity test can be run with a single command (`cilium connectivity test`). It ensures correctness in networking, Kubernetes services and
NetworkPolicy enforcement and is ideal to validate whether Cilium has been
correctly installed.

```
$ cilium connectivity test
✨ [gke_cilium-dev_us-west2-a_test-21504] Creating namespace for connectivity check...
✨ [gke_cilium-dev_us-west2-a_test-21504] Deploying echo-same-node service...
[...]
---------------------------------------------------------------------------------------------------------------------
🔌 [pod-to-pod] Testing cilium-test/client-68c6675687-tcw9f -> cilium-test/echo-other-node-588bf78fbb-84fvc...
---------------------------------------------------------------------------------------------------------------------
✅ [pod-to-pod] cilium-test/client-68c6675687-tcw9f (10.96.6.173) -> cilium-test/echo-other-node-588bf78fbb-84fvc (10.96.5.226)
[...]
---------------------------------------------------------------------------------------------------------------------
📋 Test Report
---------------------------------------------------------------------------------------------------------------------
✅ 9/9 tests successful (0 warnings)
```

###### Improving Cilium Network Performance

_Primary contributors: Gilberto Bertin (Isovalent), Kornilios Kourtis (Isovalent) and Daniel Borkmann (Isovalent)_

Following up
[work from previous releases](/blog/2020/11/10/cilium-19#performance-testing-framework),
we continued to rigorously evaluate Cilium's networking performance. Our latest
findings were recently summarized in a
[blog post](/blog/2021/05/11/cni-benchmark) as well as integrated
into Cilium's [documentation](https://docs.cilium.io/en/v1.10/operations/performance/benchmark/),
where we provided numbers for different types of direct routing workloads, showed
how eBPF enables for low-latency as well as high CNI throughput performance, and
compared WireGuard to IPsec.

Guided by our in-depth performance evaluation, Cilium 1.10 includes a number of
datapath improvements, this time mainly focussed on the VXLAN/Geneve tunneling
performance as well as reduction of Netfilter overhead. Most notably:

- [BPF host routing](/blog/2020/11/10/cilium-19#virtual-ethernet-device-optimization-with-ebpf)
 was extended to support encapsulation. This means that maximizing performance
 using eBPF can now be achieved in both direct routing and tunneling modes. In
 particular, this improves the north-south service handling to be on par with
 the direct routing mode since network traffic does not need to be processed by
 the upper host stack anymore.
- We identified and
 [fixed](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=89e5c58fc1e2857ccdaae506fb8bc5fed57ee063)
 a performance issue in the Linux kernel, where the GRO aggregation of encapsulated
 UDP packets for VXLAN and Geneve was delayed, leading to suboptimal performance
 when using encapsulation with Cilium. The fix has been backported to all stable
 kernels (e.g.,
 [4.9](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v4.9.268&id=d04c87722f072a056523d354885a74b41f4d6836))
 as well as integrated into downstream distributions such as Ubuntu LTS variants.
 In our test environment, this more than doubled the Gbit/s TCP single-stream
 throughput rate for traffic subject to VXLAN and Geneve encapsulation.
- Finally, Cilium can now be configured to skip Netfilter conntrack: we introduced a new
 agent option, `--install-no-conntrack-iptables-rules`, which allows to skip
 Netfilter connection tracking whenever possible. You can read more in the
 [tuning guide][].

We will follow-up and publish in-depth results of our performance evaluation for the
tunneling setups in an upcoming blog post. Preliminary results from our [test environment](https://docs.cilium.io/en/v1.10/operations/performance/benchmark/#test-hardware) yield significant improvements as shown below for the three
mentioned areas.

###### Case 1: Pod to Pod under tunneling

For the regular Pod to Pod case where traffic needs to traverse through the
VXLAN/Geneve tunnel we were able to significantly improve the bulk performance
for large data transfers, typically benchmarked through netperf TCP_STREAM tests.
We added a small improvement to the Linux kernel's GRO engine which is responsible
to optimize the Linux' stack traversal by aggregating MTU-sized packets up to
64k super-packets in order to reduce the per-packet processing cost for the
stack. We found that the aggregation would happen only inside the VXLAN/Geneve
driver's GRO cell instead of the physical device right upon receive, meaning,
packets had to traverse individually through the UDP stack to only be aggregated
there. Moving the aggregation to the physical device significantly boosted the performance.

###### Case 2: Host to Service/Pod under tunneling

For the case where an external client is connecting to a Pod on a Cilium-managed
node, for example, through a NodePort service, then the same improvements can be
applied [as done earlier](/blog/2020/11/10/cilium-19#virtual-ethernet-device-optimization-with-ebpf)
for the direct routing case. Meaning, forwarding can be performed right in the
tc eBPF layer instead of having to push packets up the host stack. This results
in significant gains in single stream throughput (TCP_STREAM) as well as significant
reductions in latency for request/response-type workloads (TCP_RR and TCP_CRR).
The gains in such scenarios are less visible when packets have to traverse the
tunnel (e.g. for plain Pod to Pod case) since this mandates going through the upper
UDP stack, but this is not the case for services where, for example, the backends
are local on the node. A typical use case is NodePort under `externalTrafficPolicy=Local`.

###### Case 3: Pod to Pod under direct routing

For the last case we investigated the direct routing performance when users do
have the constraint of having to pass through the Netfilter subsystem in the host
stack, be it due to running an older kernel or simply due to having Cilium deployed
with custom Netfilter rules. In particular the Netfilter connection tracker can
add a significant cost to the fast-path and might not be needed in all deployment
situations, especially since we added eBPF-based masquerading that can be used as
an alternative already. This traversal cost can then be minimized by disabling the
connection tracking requirement for all Pod traffic, thus bypassing the Netfilter
connection tracker. In particular connect/request/response-type workloads (TCP_CRR)
benefit significantly from this optimization by almost doubling their transaction
rate.

###### XDP-based Standalone Load Balancer

_Primary contributors: Daniel Borkmann (Isovalent), Martynas Pumputis (Isovalent), Chris Tarazi (Isovalent) and Sebastian Wicki (Isovalent)_

Recent advances of Cilium's eBPF-based load balancer added support for
[Maglev](/blog/2020/11/10/cilium-19#maglev-load-balancing)
consistent hashing as well as forwarding plane acceleration at the
[eXpress Data Path](/blog/2020/06/22/cilium-18#kube-proxy-replacement-at-the-xdp-layer) (XDP)
layer, and therefore significantly improving robustness as well as load
balancing efficiency. While those features were targeting a Kubernetes
environment in order to replace kube-proxy, they also paved the way for
supporting a _generically programmable_, high-performance layer 4 load
balancer (L4LB) framework, which we have added as part of the 1.10 release.

The Cilium XDP L4LB comes with full IPv4/v6 dual-stack support that can be deployed
and programmed independently of Kubernetes, for example, at the datacenter
edge in order to direct external traffic into the clusters. The XDP layer
in the Linux kernel which we co-maintain allows for DPDK-like [performance](https://dl.acm.org/doi/10.1145/3281411.3281443)
while at the same time making more efficient use of CPU resources through
running eBPF programs directly inside the driver layer and thus piggy-backing
on Linux' NAPI mechanism. Hence there is also no need for busy-looping on
the driver's receive path if there is no traffic that needs to be processed.
The vast majority of 10G, 40G and 100G+ upstream NIC drivers support XDP
out of the box on modern LTS kernels.

In order to better understand the features and general architecture of
Cilium's XDP L4LB, an outline can be found below:

**Cilium's XDP L4LB** consists of 2 main components:

- The **high-performance forwarding plane** where Cilium installs eBPF
 programs into the driver layer in order to select a backend from the
 service table through Maglev consistent hashing as well as subsequent
 L4 DNAT with IPIP encapsulation to then push the packet back out of the
 node as quickly as possible,
- The Cilium **orchestration API** in order to program and utilize the
 main service table and the wildcarded n-tuple PCAP recorder which can be
 used to filter and observe inbound and corresponding outbound traffic of
 the L4LB.

With the integration into the XDP layer and minimal state-keeping for the
Maglev backend mapping, the L4LB can scale linearly with the number of
NIC receive queues mapped to CPUs. While the L4LB is programmed and
operated independently of Kubernetes, it can still be deployed into an
existing Kubernetes cluster: Given the efficiency and low CPU overhead
of the load balancer design, the Cilium XDP L4LB can also be co-located
with regular workloads.

While the Cilium's XDP L4LB getting started guide is work in progress, please
refer to [the example DaemonSet](https://github.com/cilium/cilium-l4lb-test/blob/master/cilium-lb-example.yaml)
to how to deploy the load balancer. We will update the blog post with the links
once the getting started guide has been published.

###### DSR with IPIP Encapsulation and L4 DNAT

Cilium's XDP L4LB implements Direct Server Return (DSR) for forwarding
requests to backend nodes which is achieved by encapsulating the client
request into an IPIP/IP6IP6 packet. The inner IP header contains the
original request, so that the backend node has the full context to send
a reply directly to the client which saves an extra hop for the reverse
path. The L4LB can be configured to either use L4 DNAT to the backend or
the service port in the inner TCP/UDP destination port.

Relying on IPIP/IP6IP6 as the initial DSR dispatcher method for the
standalone LB was the most natural choice for our users in order to replace
existing Netfilter/IPVS-based in-house load balancing with the benefits
of XDP. For upcoming releases we plan to extend this to other
encapsulation mechanisms.

Not all NICs on the backend nodes might be able to parse the IPIP/IP6IP6
packet deep enough into the L4 layer in firmware/hardware to obtain enough
entropy for **RSS steering** across all receive queues. Therefore, the L4LB
can be configured to generate a custom outer IP header source address. In order to integrate
better with underlying network policies, the outer source IP can be
configured with a custom IPv4/v6 prefix which is followed by hashing the
remaining bits from the L3/L4 layer of the original client request. This
guarantees that i) all packets of the same connection will end up at the
same remote NIC queue, and ii) packets to the same service from the same
client but different sockets will end up being steered among remote NIC
queues. This way good scaling is achieved even when multiple container or
Pods on the same node target a specific service.

Cilium's XDP L4LB also implements **PMTU discovery** support for clients,
meaning, if requests are too large such that the encapsulation header cannot
be added, the LB replies with a ICMP type 3/code 4 (destination unreachable,
fragmentation needed), respectively a ICMPv6 type 2 (packet too big) error
back to the client so that it can adapt and cache the replied desired MTU
in order to reach backends.

###### n-Tuple PCAP Recorder

Given a group of Cilium XDP L4LBs where hundreds of millions of packets per
second are continuously being processed, how can one _efficiently_ find the
needle in the haystack to troubleshoot or observe a small
portion of the network traffic? As XDP operates way below the regular networking
stack, existing tooling such as tcpdump is not available. Even if it was,
its internal filter generation would also not be efficient enough for a
larger set of IPs or prefixes since they would need to be processed linearly.

In the Cilium XDP L4LB's datapath we therefore included **two observation points**
in order to filter and record the LB _inbound_ traffic with its corresponding
_outbound_ traffic, that is, the IPIP/IP6IP6 packets shortly before being pushed
out the NIC again. This allows for further correlation to reconstruct the path
taken from the fabric to the L4LB to the subsequent backends, for example,
L7 proxies as a next hop.

The L4LB implements **wildcarded/n-tuple filtering** for IPv4/v6, meaning arbitrary
combinations of source and destination prefixes, ports and protocols are allowed.
While it is not (yet) possible to have native wildcard map lookups in eBPF, we
implemented a filtering mechanism with masked lookups under the assumption that
the number of installed masks are fairly small while the number of filter entries
fitting that mask can be arbitrary large. Inbound traffic matching the filters
is then exported either as full or partial packet capture through a memory-mapped
per CPU ring-buffer that is processed by the agent and subsequently exported as
PCAP files for inspection via Wireshark, tcpdump or other familiar tools.

[](https://asciinema.org/a/TIpQRjdAlsQOsbsFtxwLJiVec)

The PCAP recorders can be orchestrated through a **new Hubble Recorder API** which
differs from the existing Hubble Observer API in that it has been optimized for
the high-churn traffic which the XDP datapath has to handle. For convenience
we also added support to the Hubble CLI as one example consumer of the Hubble
Recorder API. The above example shows how to capture L4LB-forwarded TCP service
traffic from _any_ source to destination `192.168.33.11:80` by specifying a
5-Tuple filter in the Hubble CLI.

The PCAP recorder is currently only available in the Cilium's new XDP L4LB mode,
but it is planned to also extend support to regular east-west traffic when running
in Kubernetes CNI mode in an upcoming release.

###### Getting Started

New to Cilium? Follow one of the [Getting Started Guides].

###### Upgrade Instructions

As usual, follow the [upgrade guide] to upgrade your Cilium deployment.
Feel free to ping us on [Slack] if you have any questions.

###### Release

- Release Notes: [1.10.0](https://github.com/cilium/cilium/releases/tag/v1.10.0)

[slack]: https://slack.cilium.io
[introduction to cilium]: https://docs.cilium.io/en/v1.10/intro/
[getting started guides]: https://docs.cilium.io/en/v1.10/gettingstarted
[upgrade guide]: https://cilium.readthedocs.io/en/v1.10/operations/upgrade/#upgrading-minor-versions
[tuning guide]: https://docs.cilium.io/en/v1.10/operations/performance/tuning/#bypass-iptables-connection-tracking
[alibaba cloud getting started guide]: https://docs.cilium.io/en/v1.10/gettingstarted/alibabacloud-eni/
[aws eni]: https://cilium.io/blog/2019/08/20/cilium-16#eni
[azure]: https://cilium.io/blog/2020/06/22/cilium-18#azureipam

<small>
 "WireGuard" is a registered trademark of Jason A. Donenfeld.
</small>

#### CNI Benchmark: Understanding Cilium Network Performance

URL: https://cilium.io/blog/2021/05/11/cni-benchmark
Date: 2021-05-11
Categories: How-To

##### Understanding Cilium Network Performance

Hello

As more crucial workloads are being migrated to Kubernetes, network performance
benchmarks are becoming an important selection criteria when deciding what
network layer to leverage in a Kubernetes cluster. In this blog post, we'll
explore the performance characteristics of Cilium based on extensive benchmarks
that we have run in the past few weeks. Upon popular request, we are also
including measurements for Calico to allow for a direct comparison.

However, instead of just presenting the numbers, we are going to dive into
the topic of container networking benchmarking a bit deeper and look at:

- [The Throughput Benchmark](#throughput)
- [Does Container Networking add Overhead?](#overhead)
- [Breaking the Rules: eBPF Host-Routing](#ebpfhostrouting)
- [Measuring Latency: Requests per Second](#rr)
- [Comparing CPU Flamegraphs Cilium eBPF vs Calico eBPF](#flamegraph)
- [Rate of new Connections](#crr)
- [The Cost of Encryption - Wireguard vs IPsec](#encryption)
- [How to reproduce the results](#env)
- [Performance Impact of a Sidecar](#sidecar)

##### Summary of the Results

Before we dive into the detailed numbers and benchmarks, the following list is a
summary of our findings. Feel free to skip it if you want to derive your own
conclusions after reading the details first.

- **eBPF makes the difference:** While Cilium has an edge over Calico's eBPF
 datapath in some areas, e.g. latency as observable in `TCP_RR` and `TCP_CRR`
 benchmarks. The more fundamental takeaway is that eBPF is clearly superior to
 iptables. Cilium and Calico running in a configuration allowing to bypass
 iptables using eBPF both significantly outperform the versions which cannot.

 Looking into the specific details we find that Cilium and Calico don't
 leverage eBPF in exactly the same way. While some concepts are similar (which is
 not entirely surprising given the open-source nature), CPU flamegraphs reveal
 that Cilium is taking advantage of additional context-switching savings which
 likely explain the difference in `TCP_RR` and `TCP_CRR` results.

 Overall, based on the benchmark results, eBPF is clearly the best technology
 to address the challenging cloud-native requirements.

- **Observability, Network Policy, and Services:** For this benchmark, we have
 focused on the lowest-common denominator which is essentially networking
 only. This also allows to directly compare the results to node networking.
 However, real-world usage will also require observability, network policy and
 services. This is where the Cilium and Calico eBPF datapaths will differ
 extensively. Cilium supports several additional features not found in the
 Calico eBPF datapath, but even for the standardized features such as
 Kubernetes NetworkPolicy, the implementations differ and we will likely find
 significant performance differences as a substantial amount of work has to be
 performed using eBPF for these more advanced use cases. However, the post
 is already long enough as-is so we'll reserve digging into these measurements
 and details to a follow-up post.

- **Wireguard vs IPsec:** Somewhat surprising, even though Wireguard has been
 able to achieve higher maximum throughputs in our tests, IPsec can be more
 efficient in terms of CPU resources to achieve the same throughput. This is
 very likely strictly dependent on the availability of AES-NI CPU instructions
 which allow to offload the crypto work for IPsec whereas Wireguard cannot
 benefit from this. The cards will obviously turn when AES-NI offload is not
 available.

 The good news is that starting with Cilium 1.10, you have the choice to run
 either. Cilium now supports Wireguard in addition to IPsec.

##### The Throughput Benchmark

The usual benchmarking disclaimer:

> Benchmarking is hard. Results can vary based on the hardware tests are run
> on. Absolute numbers should not be compared unless results have been gathered
> on identical systems.

Let's start with the most common and obvious benchmark, the infamous TCP
throughput metric measuring the maximum data transfer rate between containers
running on different nodes:

The above graph shows the maximum throughput that can be achieved with a single
TCP connection. The best performing configurations top out just over 40Gbit/s.
It has been measured by running `netperf` using the `TCP_STREAM` test. We are
using a 100Gbit/s network interface to ensure that the network card is not the
bottleneck. Due to running a single `netperf` process transmitting over a
single TCP connection, most of the network processing is done using a single
CPU core. This means that the above number is constrained by the available CPU
resources of a single core and thus nicely highlights how much throughput can
be achieved with each configuration when the CPU is the bottleneck. We'll
expand this test further down in the blog by throwing more CPU cores at the
problem to remove the CPU resources constraint.

Did you notice that the high-performing eBPF implementations can a throughput
even slightly higher than the node-to-node baseline? How is this possible?
It is somewhat unexpected because container networking is generally believed to
add overhead compared to node to node networking. Let's hold this thought for
a moment, we'll explore this aspect as we dig deeper.

##### CPU resources required to transfer 100Gbit/s

The results for the `TCP_STREAM` benchmark already hinted which configurations
are the most efficient to achieve high transfer rates but let's look at the
overall system CPU consumption while the benchmark is running:

The above CPU usage graph represents the system-wide percent of CPU required to
deliver 100Gbit/s of throughput. Note that this is not the CPU consumption for
the throughput reported in the previous graph, the CPU usage has been
normalized for all results to represent a steady 100Gbit/s transfer rate to make
the numbers directly comparable. The lower the bar in the above graph, the more
efficient a configuration is in transferring 100Gbit/s.

> **Random kernel wisdom:** TCP flow performance is generally limited by the
> receiver, since the sender can use both TSO super-packets. This can be
> observed in the increased CPU spending on the server-side in the above tests.

##### What do TCP throughput benchmarks represent?

While the majority of users are unlikely to routinely experience this level of
throughput, certain types of application will care for this type of benchmark:

- AI/ML applications requiring access to large amounts of data
- Data upload/download services (backup services, VM images, container registries,...)
- Media streaming services, in particular for 4K+

We'll dive into [Measuring Latency: Requests per Second](/blog/2021/05/11/cni-benchmark#rr)
and [Rate of new Connections](/blog/2021/05/11/cni-benchmark#crr) later on in
this post which better represent typical microservices workload.

##### Does Container Networking add Overhead?

The initial benchmark indicates that there is some overhead when performing
container networking compared to node networking. Why is this? Let's look at
the two networking models from an architecture perspective:

What stands out in the above diagrams is that the entire networking processing
path that is required for node-to-node networking, is also done for container
networking case, just inside of the network namespace of the container (dark
blue box).

Given that all work required for the node network processing is also required
within the container network namespace, any work done outside of the container
network namespace is basically overhead. The above diagram shows the network
path when Linux routing with Virtual Ethernet (veth) devices are used. It may
look slightly different if you use a Linux bridge or OVS, for example, but the
fundamental overhead point is shared between all of them.

##### Breaking the Rules: eBPF Host-Routing

You may be wondering about the difference between the configurations "Cilium
eBPF" and "Cilium eBPF (legacy host-routing)" in the benchmarks before and why
the native Cilium eBPF datapath is considerably faster than the legacy host
routing. When referring to the Cilium eBPF native datapath, an optimized
datapath called eBPF host-routing is in use:

eBPF host-routing allows to bypass all of the iptables and upper stack overhead
in the host namespace as well as some of the context-switching overhead when
traversing through the Virtual Ethernet pairs. Network packets are picked up as
early as possible from the network device facing the network and delivered directly
into the network namespace of the Kubernetes Pod. On the egress side, the packet
still traverses the veth pair, is picked up by eBPF and delivered directly
to the external facing network interface. The routing table is consulted
directly from eBPF so this optimization is entirely transparent and compatible
with any other services running on the system providing route distribution.
For information on how to enable this feature, see
[eBPF Host-Routing](https://docs.cilium.io/en/latest/operations/performance/tuning/#ebpf-host-routing)
in the tuning guide.

Calico eBPF is applying some of the same bypasses to iptables but as we'll
learn later on, is not quite identical. However, it proves that the most impact
can by made by bypassing slow kernel subsystems such as iptables.

##### Pushing for 100Gbit/s Line-Rate

Earlier in the blog post, we looked at results while mostly involving just a
single CPU core in all of the processing. Let's open up the flood gates and
parallelize TCP streams and run multiple `netperf` processes:

**Note:** We have specifically chosen 32 processes due to the hardware
featuring 32 threads, to ensure that the system can evenly distribute the load.

This graph is kind of boring. It shows that if you throw enough CPU resources
at the problem, all tested configurations can achieve close to 100Gbit/s
line-rate. However, looking at the CPU resources, we can still identify a
difference in efficiency:

Note that the CPU usage measurement includes the entire CPU consumed, this also
includes the `netperf` processes running so it also includes the CPU typically
required by workloads to perform the network I/O. It however does not include
any business logic that the application would typically perform.

##### Measuring Latency: Requests per Second

The requests per second metric is almost the exact opposite of the throughput
metric. It measures the rate of single byte round-trips that can be performed
in sequence over a single persistent TCP connection. This benchmark highlights
how efficiently a single network packet can be processed. The
lower the latency for an individual network packet, the more requests can be
processed per second. Optimizing between throughput and latency is often a
trade-off. To achieve maximum throughput, large buffer sizes are ideal but
these large buffer sizes can lead to an increase in latency. This is called
buffer bloat. Cilium contains a feature called
[Bandwidth Manager](https://docs.cilium.io/en/latest/operations/performance/tuning/#bandwidth-manager)
which automatically configures fair queueing, optionally allows for EDT-based
Pod rate-limiting, and optimizes TCP stack settings for server workloads to
strike the best possible balance between the two.

This benchmark is often overlooked but it is usually a lot more relevant to users
as it models a pretty common usage pattern for microservices: request and
responses exchanged between services using persistent HTTP or gRPC connections.

Let's look at how the different configurations perform when a single
`netperf` process performs a `TCP_RR` test:

Configurations that perform better in this test also deliver
lower average latencies. However, it does not directly allow to draw
conclusions on P95 or P99 latencies. We will look into them in a future blog
post.

As we expand the test to run 32 parallel `netperf` processes to scale out and
utilize all available CPU cores, we can see that all configurations are able to
scale up accordingly. However, unlike for the throughput tests, throwing more
CPU at the problem does not allow to catch up on the difference in efficiency
because the maximum rate is limited by the latency and not the available CPU.
We would only see identical requests if the network bandwidth became the
bottleneck.

Overall, the results are quite encouraging, Cilium can achieve almost 1M
requests/s on our test system with eBPF host-routing.

##### Comparing CPU Flamegraphs Cilium eBPF vs Calico eBPF

Overall, the performance for Cilium eBPF and Calico eBPF are relatively
similar, are they using the same datapath? Not really. There is no such thing
as a pre-defined eBPF datapath. eBPF is a programming language and runtime
engine that allows to build datapath features among many other things. The Cilium
and Calico eBPF datapaths differ quite significantly. In fact, Cilium offers a
wide range of features which are not supported by Calico eBPF. But even on the
interaction with the Linux networking stack, the two show significant
differences. Let's look at CPU flamegraphs to dig in a bit:

##### Cilium eBPF (Receive Path)

The eBPF host-routing implementation of Cilium features a nice context-switch
free delivery of data from the NIC all the way into the socket of the
application. That's why the entire receive-side path fits nicely into a single
flamegraph above. You can see the processing blocks for eBPF, TCP/IP, and
the Socket.

##### Calico eBPF (Receive Path)

The Calico eBPF receive side does not look quite the same. There is an identical
eBPF block which executes the eBPF program. Then there is an additional Virtual
Ethernet (veth) traversal which is not required on the receive side in Cilium's
eBPF datapath.

All of the above is still performed in the context of the host. This following
second flamegraph highlights the work done in the pod itself. It shows the work
performed as resumed by `process_backlog`. It is the same work (TCP/IP + socket
delivery) as in the Cilium case, but after an additional context switch due to
the veth traversal.

If you want to dig deeper yourself, open the below links in your browser for
the interactive SVG versions of the images that will allow to zoom in:

- [Cilium eBPF Flamegraph SVG - Sender](images/cilium-ebpf-hr-rr-zh3.svg)
- [Cilium eBPF Flamegraph SVG - Receiver](images/cilium-ebpf-hr-rr-zh4.svg)
- [Calico eBPF Flamegraph SVG - Sender](images/calico-ebpf-rr-zh3.svg)
- [Calico eBPF Flamegraph SVG - Receiver](images/calico-ebpf-rr-zh4.svg)

##### Rate of new Connections

The connection rate benchmark builds on top of the requests per second
benchmark but initiates a new connection for each request. This benchmark
highlights the difference between using persistent connections and opening new
connections for each request. Handling new TCP connections requires work to be
performed in various parts of the system; this test is therefore by far the most
stressful for the entire system. We will see that it is possible to actually
consume the majority of the available system resources with this benchmark.

This test represents a workload that receives or initiates a lot of TCP
connections. An example where this is the case is a publicly exposed service
that receives connections from many clients. Good examples of this are L4
proxies or services opening many connections to external endpoints such as a
data scraper. This benchmark puts the most stress on the system with the least
work offloaded to hardware, so we can expect to see the biggest differences
between tested configurations.

In a first test, we run a single `netperf` process using the `TCP_CRR` test:

The difference is already quite severe for a single process and it will amplify
as we scale out to more CPU cores. It's also clear that Cilium can again almost
compensate for the additional network namespace overhead and almost match the
baseline.

> **Future work scheduled:** The CPU resource usage was a surprise to us and lead
> us to schedule further investigation for the 1.11 development cycle. There
> seems to be a cost paid on the sender side as soon network namespaces get
> involved. This cost is paid by all configuration that involve network
> namespaces so it is likely caused by the kernel datapath portion shared by
> both Cilium and Calico. We'll provide an update as soon as we have learned
> more about this.

As we scale out the test to run on all CPU cores by running 32 parallel
`netperf` processes using `TCP_CRR`, an interesting observation can be made:

The connection rate for the baseline drops significantly. It is not able to
scale with the additional CPU resources available. This is despite the
connection tracking table sized accordingly and us validating that no drops had
occurred due to the connection tracking table filling up. We have re-run these
tests many times but the results remained consistent. Manually bypassing the
iptables connection tracking table using `-j NOTRACK` iptables rules
immediately fixes this issue for the baseline and improves performance
to 200K connections/s as well. So there is clear evidence that the iptables
connection tracking table can start to struggle once above some threshold.

**Note:** The results for the Calico eBPF datapath have been unstable in this
test. We are not sure why. The network packet flow was not steady. We are
therefore not including the results because they are probably not accurate. We
invite the Calico team to work with us to investigate this and then re-test.

Handling 200K connections/s is impressive given we have a standard, unmodified
application handling these requests and transmitting actual information but let's
look at the cost on the CPU side:

This benchmark outlines the biggest difference between the various
configurations. In order to handle 250K new connections per second, the overall
system has to spend anywhere from 33% to 90% of the available resources.

Based on the consistent difference between the required CPU for the sender and
the receiver, it's also safe to assume that you can typically accept more
connections per second than you can initiate.

##### The Cost of Encryption - Wireguard vs IPsec

Everybody will expect Wireguard to outperform IPsec so let's look at Wireguard
first and see how Wireguard performance is tied to the configured MTU:

There is some difference between the configurations and interestingly Cilium in
combination with kube-proxy performs better than Cilium without kube-proxy.
However, the differences are relatively minor and most of the difference can be
recovered by optimizing the MTU.

How does the CPU resources side look like:

No major differences really. Configure your MTU correctly. We have also
performed requests per second tests in our benchmarks but they show more of the
same. No major differences. If you are interested, you can find them in the [CNI
Performance
Benchmark](https://docs.cilium.io/en/latest/operations/performance/benchmark/)
section of the Cilium documentation.

##### Wireguard vs IPsec

Comparing the performance of Wireguard with IPsec is a more interesting test.
Cilium has supported IPsec for a while and starting with Cilium 1.10,
Cilium now also supports Wireguard. So it's interesting to compare the two next
to each other while all other aspects are the same:

The above results is likely what everybody was expecting. Wireguard is
achieving a higher throughput. Again, depending on the MTU but for both MTU
configurations, Wireguard is able to achieve a higher maximum transfer rate.

However, let's look at the CPU resources required to achieve 10Gbit/s:

While Wireguard is able to achieve a higher maximum, IPsec is more efficient in
achieving the same throughput. The difference is actually quite severe.

> **Note:** To achieve this efficiency with IPsec, you strictly need
> hardware that supports AES-NI instructions in order to offload IPsec
> encryption.

> **Future work scheduled:** It's not entirely clear to us yet why the higher
> efficiency of IPsec does not translate to a higher throughput as well. Also,
> throwing additional CPU cores at the problem doesn't significantly improve
> the performance. This is likely because RSS can't scale out well across cores
> for encrypted traffic since the L4 information typically used to hash and
> distribute flows across cores is encrypted and not available. Therefore, all
> connections will look the same from a hashing perspective as only two IP
> addresses are utilized in the benchmark.

Does this also affect the latency? Let's have a look. Remember that the latency
benchmark is the one that most accurately describes microservices style
workloads where persistent connections are used to exchange requests and responses.

The CPU efficiency is in line with the observed requests per second but
overall, none of the configurations consume a considerable amount of total CPU.
The difference in latency is much more significant than the consumed CPU:

##### Performance Impact of Sidecars on a Service Mesh

Besides avoiding the sheer amount of proxies that need to be run in a sidecar service mesh model, a significant advantage of sidecarless service mesh is that we can avoid requiring running two proxies in between any connection. More details about Cilium's sidecarless service mesh can be found in this blog [Next-Generation Mutual Authentication with Cilium Service Mesh](https://isovalent.com/blog/post/2022-05-03-servicemesh-security/)

Reducing the number of proxies in the network path and choosing the type of Envoy filter has a significant impact on performance. The above benchmark illustrates the latency cost of HTTP processing with a single Envoy proxy running the Cilium Envoy filter (brown) compared to a two-sidecar Envoy model running the Istio Envoy filter (blue). Yellow is the baseline latency with no proxy and no HTTP processing performed.

##### Test Environment

This is the spec of our bare metal, off-the-shelf test environment. Two identical
systems are used. The systems are directly connected to each other.

- **CPU:** AMD Ryzen 9 3950x, AM4 platform, 3.5GHz, 16 cores / 32 threads
- **Mainboard:** x570 Aorus Master, PCIe 4.0 x16 support
- **Memory:** HyperX Fury DDR4-3200 128GB, XMP clocked to 3.2GHz
- **Network Card:** Intel E810-CQDA2, dual port, 100Gbit/s per port, PCIe 4.0 x16
- **Kernel:** Linux 5.10 LTS (built with `CONFIG_PREEMPT_NONE`)

All tests have been performed using a standard MTU of 1500 unless explicitly
called out. While it is of course possible to achieve better absolute
numbers with a higher MTU, the purpose of these benchmarks is to outline
relative differences and not the highest or lowest absolute numbers.

##### Test Configurations

Upon popular request, we have included measurements for Calico for comparisons.
We have therefore measured with the following configurations to allow for the
best possible comparison:

- **Baseline (Node to Node):** In this configuration, no Kubernetes or
 containers are used. The benchmark is performed by directly running `netperf`
 on the bare metal machine. Typically this will produce the best possible
 result.

- **Cilium eBPF:** Cilium 1.9.6 running as described in the
 [tuning guide](https://docs.cilium.io/en/latest/operations/performance/tuning/)
 with eBPF host-routing, and kube-proxy replacement enabled. This
 configuration requires a modern kernel (>=5.10). From a comparison and
 requirements perspective, this configuration maps best to "Calico eBPF". We
 have focused on benchmarking in a direct routing configuration as this is
 where performance typically matters most. We will extend the benchmarking for
 tunneling modes as well later on.

- **Cilium eBPF (legacy host-routing):** Cilium 1.9.6 running in legacy host-routing
 with standard kube-proxy supporting older kernels (>=4.9). From a comparison and
 requirements perspective, this configuration maps best to "Calico".

- **Calico eBPF:** Calico 3.17.3 kernel with the eBPF datapath with kube-proxy
 replacement, connection-tracking bypass, and eBPF FIB lookup enabled. This
 configuration requires a modern kernel (>=5.3). From a comparison and
 requirements perspective, this configuration maps best to "Cilium eBPF"

- **Calico:** Calico 3.17.3 running with standard kube-proxy supporting older
 kernels. From a comparison and requirements perspective, this configuration
 maps best to "Cilium eBPF (legacy host-routing)".

##### How to reproduce

All scripts required to reproduce the above numbers can be found in the git
repository [cilium/cilium-perf-networking](https://github.com/cilium/cilium-perf-networking).

##### What is Next?

We feel like we have already achieved a lot in optimizing performance, but we
still have many ideas to pursue and will continue to improve the performance
of all aspects of Cilium.

- **Benchmarking Observability:** Benchmarking pure networking is nice but the
 cost of observability is where the real difference can be made. Whether it is
 for security or troubleshooting, observability will become key in your
 infrastructure and the cost paid for the visibility will differ greatly. eBPF
 is a great tool for observability and Cilium's
 [Hubble](https://docs.cilium.io/en/latest/gettingstarted/hubble_setup/)
 allows to benefit from it. For this benchmark, we have disabled Hubble to
 make the numbers comparable to Calico. In a subsequent blog post, we'll benchmark
 and look at the CPU requirements for Hubble and how it compared to other
 similar systems.

- **Benchmarking Services & NetworkPolicy:** The published benchmark results do
 not involve any NetworkPolicy or services. We have excluded both to keep the
 scope of this blog post somewhat under control. We will extend the results to
 include NetworkPolicy use cases as well as services for both east-west and
 north-south scenarios. If you can't wait, the [Cilium 1.8 release
 blog](/blog/2020/06/22/cilium-18#kubeproxy-removal) already published some
 benchmark results and shows how the use of XDP and eBPF significantly improves
 performance.

 On this note, we are still not happy with the CIDR rule performance aspect of
 NetworkPolicy. Our current architecture is optimized for a small number
 complex CIDRs with exceptions implemented using longest prefix match (LPM)
 tables. Several users seem to desire and benchmark large allow and deny lists
 of individual IPs. We will make this use case a priority as well and provide
 a hashtable-based implementation.

- **Memory Optimizations:** We continue to optimize the memory footprint of
 Cilium. The majority of Cilium's memory footprint comes from eBPF map
 allocations. These are kernel-level data structures which are required for
 the network processing. For efficiency, eBPF maps are pre-sized so there is a
 minimal amount of required memory based on the configuration. This aspect is
 not as ideal as we would want it right now so it will be a focus of ours in
 future releases.

- **Breaking More Rules - Bypassing More iptables:** In our opinion, you can't
 bypass enough iptables until you eventually get rid of it entirely. There is
 still optimization potential in the container namespace and other parts of
 the system. We are also continuing work to accelerate service mesh datapath
 applications, an initial version of this is already available using
 [socket-level redirection for Envoy](/blog/2018/08/07/istio-10-cilium#socket-level-redirection-to-accelerate-istio-and-envoy).
 Expect more to come in this area.

- **Other Ideas?** Do you have other ideas? Let us know! What would you like us
 to benchmark or improve on? We are eager to hear your thoughts. Drops us a
 line on the [Cilium Slack](https://slack.cilium.io) or reach out to us on
 [Twitter](https://twitter.com/ciliumproject).

##### Further Reading

- All of the above numbers have been published in the [CNI Performance
 Benchmark](https://docs.cilium.io/en/latest/operations/performance/benchmark/)
 section of the Cilium documentation and we will continuously update this.
- All the instructions on how to tune Cilium for performance can be found in
 the [Tuning Guide](https://docs.cilium.io/en/latest/operations/performance/tuning/).
- To learn more about Cilium, check out the [What is
 Cilium?](https://docs.cilium.io/en/latest/intro/) documentation.
- To learn more about eBPF check out the [eBPF.io](https://ebpf.io) website.

#### Introducing the Cilium Certified OpenShift Plug-in

URL: https://cilium.io/blog/2021/04/19/openshift-certification
Date: 2021-04-09
Categories: Technology

##### Certified Red Hat OpenShift CNI Plug-in

We are pleased to introduce the availability of Cilium in the
[Red Hat Ecosystem Catalog](https://catalog.redhat.com/software/operators/detail/60423ec2c00b1279ffe35a68)
as well as the certification of Cilium as a
[Certified OpenShift CNI Plug-in](https://access.redhat.com/articles/5436171).
The Container Network Interface (CNI) badge is a specialization within Red Hat
OpenShift certification available to networking products that integrate with
OpenShift using a CNI plug-in.
Users running OpenShift can feel confident that running Cilium will not negatively impact their Red Hat support experience.

The OpenShift certified version of Cilium is based on
[Red Hat Universal Base Images](https://developers.redhat.com/products/rhel/ubi)
and passes the
[Operator certification](https://redhat-connect.gitbook.io/partner-guide-for-red-hat-openshift-and-container/)
requirements as well as the
[Kubernetes e2e tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md).

[](https://catalog.redhat.com/software/operators/detail/60423ec2c00b1279ffe35a68")

##### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms such as Kubernetes.

At the foundation of Cilium is a new Linux kernel technology called eBPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. eBPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption as well as network and service security. Besides
providing traditional network-level security, the flexibility of eBPF enables
security with the context of application protocols and DNS requests/responses.
Cilium is tightly integrated with Envoy and provides an extension framework
based on Go. Because eBPF runs inside the Linux kernel, all Cilium
functionality can be applied without any changes to the application code or
container configuration.

See the section **[Introduction to Cilium]** for a more detailed general
introduction to Cilium.

##### Installation

To install Cilium on OpenShift, go to the [Red Hat Ecosystem
Catalog](https://catalog.redhat.com/software/operators/detail/60423ec2c00b1279ffe35a68#deploy-instructions)
and follow the deploy instructions:

If you have any questions regarding running Cilium on OpenShift, don't hesitate
to reach join the [Cilium slack](https://cilium.io/slack) and ask your
questions in `#general` channel.

##### Learn More

If you want to learn more about Cilium:

- [Cilium - eBPF-based Networking, Observability, and Security](https://cilium.io/)
- [Cilium on GitHub](https://github.com/cilium/cilium)
- [Cilium 1.9: Maglev, Deny Policies, VM Support, OpenShift, Hubble mTLS, Bandwidth Manager, eBPF Node-Local Redirect, Datapath Optimizations, and more](https://cilium.io/blog/2020/11/10/cilium-19)

[introduction to cilium]: https://docs.cilium.io/en/v1.9/intro/

#### From "War Games" to Network Policies

URL: https://cilium.io/blog/2021/04/06/war-games-network-policy
Date: 2021-04-06
Categories: Technology

We’ve just launched a new community resource: networkpolicy.io, to help people learn how to apply Kubernetes network policies to protect their business-critical workloads. But why do we need network policies at all? Let’s consider why traditional network security approaches aren’t sufficient in the cloud native world, and see what advantages we can gain through network policies.

Network security isn’t a new concept. The term “firewall”in the context of IT was apparently coined by the scriptwriters of 1983 movie War Games, and the technology evolved to become commonplace by the early 90s. It should be no surprise that the requirements on network security products have moved on quite a bit after thirty years! Kubernetes has proved to be quite the game-changer for many aspects of deploying and running software, and one of the changes it demands is a new approach to network security.

Fortunately, Kubernetes brings with it the built-in concept[^1] of Network Policy that takes us in the right direction. Even better, Network Policies can be the basis for even stronger, more granular network security than we enjoyed in the old days of monoliths and firewalls.

##### IP addresses are in flux

The main reason we can’t just rely on old-fashioned firewalls is that Kubernetes deployments are dynamic. Pods appear and disappear frequently, and they get scheduled onto different machines across the deployment. Each pod gets assigned an IP address, but that IP address is ephemeral, like the pod itself, and the set of IP addresses being used across your application today might be very different tomorrow.

In the old days, firewall rules would be defined in terms of IP addresses and ports, but that makes no sense when IP addresses are constantly in flux.

##### Application code is distributed across nodes

Another problem with traditional firewalls is that they don’t have visibility into traffic flowing within any given node. In the traditional approach, you would know what application code is running on each (virtual) machine in your deployment. You might have a set of machines handling the front end, another set handling the database and so on. A network firewall can police the traffic flowing into and out of each node, with rules that reflect the code that’s running on the machine in question.

But in the cloud native world, there is a much looser relationship between host machines and the code that runs on them. Kubernetes schedules each new pod to a suitable node in the cluster, such that any given node could have a variety of different pods running on it. A traditional firewall wouldn’t have visibility into the traffic between pods that happen to be co-located on the same machine.

##### Network Policy objects

Kubernetes uses Network Policy to define rules about what traffic is allowed within a deployment. Instead of using IP addresses, the pods affected by rules are identified using Label Selectors, in much the same way that pods are assigned to services. All the pods with the defined label(s) match the policy at any given time, so there’s no need to change the policy to keep up to date with the comings and goings of different pods. And the rules apply wherever a pod lives in the cluster, even if traffic is flowing between pods on the same node.

The Kubernetes NetworkPolicy resource also lets us define what external traffic is permitted into (ingress) or out of (egress) a set of pods. In this situation, IP addresses can be useful - for example, other clusters or other sets of machines in your estate can be defined with an IP CIDR (a range of IP addresses).

Some network plugins extend the basic concepts of Kubernetes Network Policy to provide additional capabilities. One simple example is that Cilium lets you define egress rules in terms of DNS names. This means you can set up a policy to allow your pods to send messages to some-domain.com without needing to worry about the possibility of the domain moving to a different IP address.

##### Increasing the granularity of security

The cloud native approach encourages us to architect applications in the form of services or even microservices, so that they can be developed, deployed and scaled independently. A useful corollary is that they can also be secured independently, and Network Policies give us a tool to help us do exactly that. We can build network policies that only permit the expected traffic to and from each microservice.

For example, in an e-commerce application, we expect to see a product search microservice to respond to requests from a frontend service, and look up information in a product database. There is no need to give the product search service the ability to make any other network connections except to that database. Then if an attacker does manage to compromise the product search app code (perhaps through a vulnerability in its code or one of its dependencies), it’s much harder for the attacker to move laterally from the product search pod to any other part of the system.

Fine-grained network policy definitions allow us to give minimal, least-privilege permissions to the different microservices in our application. This limits the possible blast radius of a successful exploit of any part of the system.

##### Learning about Network Policies

We’ve just launched networkpolicy.io with lots of useful resources for learning about Network Policies. You’ll find the Network Policy editor is a helpful tool for visualizing the effects that a policy will have on traffic. Network policies can be notoriously tricky and one small mistake can lead to lots of mischief. Using the editor can help avoid these mistakes and give you the confidence to deploy new policies.

You might also be interested in the Network Policy tutorial that walks you through different scenarios for policy rules.

[^1]: Kubernetes defines the NetworkPolicy object but it doesn’t enforce it within Kubernetes. You’ll need a networking plugin - for example Cilium - that does the work. If you use a networking plugin that doesn’t implement network policy, your policies will just be ignored.

#### Announcement: Networking and BPF Track at the 2021 Linux Plumbers Conference (LPC)

URL: https://cilium.io/blog/2021/04/06/bpf-plumbers-2021
Date: 2021-04-06
Categories: Community

The organizers of the Networking and BPF track at the 2021 edition of the Linux
Plumbers Conference (LPC) are pleased to announce the Call for Proposals (CFP).
The conference is planned to be held virtually on the wider Internet, on
September 20th - 24th, 2021.

This year's Networking and BPF track technical committee is comprised of:

- David S. Miller (Red Hat)
- Jakub Kicinski (Facebook)
- Eric Dumazet (Google)
- Alexei Starovoitov (Facebook)
- Daniel Borkmann (Isovalent)
- Andrii Nakryiko (Facebook)

We are seeking proposals of 40 minutes in length (including Q&A discussion),
optionally accompanied by papers of 2 to 10 pages in length. Any kind of
advanced Linux networking and/or BPF related topic will be considered. Please
submit your proposals through the official LPC website at:

- [Submit Linux Plumbers Conference 2021 Abstract](https://linuxplumbersconf.org/event/11/abstracts/)

Make sure to select "Networking & BPF Summit" in the Track pull-down menu.
Proposals must be submitted by August 13th, and submitters will be notified of
acceptance by August 16th. Final slides and optional papers (as PDF) are due
on the first day of the conference.

#### Cilium User Survey 2021

URL: https://cilium.io/blog/2021/02/25/user-survey-2021
Date: 2021-02-25
Categories: Community

Welcome to the Cilium User Survey 2021! We are looking to improve Cilium
further and are interested in feedback and guidance from our users. Cilium
has been a community-driven project from the beginning and the majority of our
roadmap has and will continue to be defined by our users. Tell us what you
want. Tell us what you are struggling with. Tell us what we can do better.

As usual, the survey is completely anonymous unless you specifically wish to
identify yourself so we can reach out to you and clarify on specific feature
requests. We will of course publish the results publicly as well.

Thank you so much for spending some of your valuable time in helping us improve
as a community and project.

 Loading…

#### NetworkPolicy Editor: Create, Visualize, and Share Kubernetes NetworkPolicies

URL: https://cilium.io/blog/2021/02/10/network-policy-editor
Date: 2021-02-10
Categories: Technology

Implementing Network Policy is a critical part of building a secure Kubernetes-based platform, but the learning curve from simple examples to more complex real-world policies is steep. Not only can it be painful to get the YAML syntax and formatting just right, but more importantly, there are many subtleties in the behavior of the network policy specification (e.g. default allow/deny, namespacing, wildcarding, rules combination, etc.). Even an experienced Kubernetes YAML-wrangler can still easily tie their brain in knots working through an advanced network policy use case.

Over the past years, we have learned a lot about the common challenges while working with many of you in the Cilium community implementing Kubernetes Network Policy. Today, we are excited to announce a new free tool for the community to assist you in your journey with Kubernetes NetworkPolicy: editor.cilium.io:

The Kubernetes NetworkPolicy Editor helps you build, visualize, and understand Kubernetes NetworkPolicies.

- **Tutorial:** Follow the assisted tutorial to go from not using NetworkPolicies yet to a good security posture.
- **Interactive Creation:** Create policies in an assisted and interactive way.
- **Visualize & Update:** Upload existing policies to validate and better understand them.
- **Security score:** Check the security score of polices to understand the level of security they add to your cluster.
- **YAML Download:** Downloading policies as YAML for enforcement in your cluster with your favorite CNI.
- **Sharing:** Share policies across teams via GitHub Gists and create links to visualize your own NetworkPolicies.
- **Automatic Policy Creation:** Upload Hubble flow logs to automatically generate NetworkPolicies based on observed network traffic.

 Try Network Policy Editor

##### How exactly does editor.cilium.io help?

To make this more concrete, let’s explore five common gotchas we see trip up those working with Network Policy, both newbies and sometimes (gulp!) those of us who have been doing this for a while. At the end of each mistake, you’ll find a link to a short (3-5 minute) tutorial in the tool that walks you through each step required to fix the mistake.

###### Mistake 1: Not Using a Namespace Selector

Consider a scenario where we want a centralized Prometheus instance running in a `monitoring` namespace to be able to scrape metrics from a Redis Pod running in the `default` namespace. Take a look at the following network policy, which is applied in the `default` namespace. It allows Pods with label `app=prometheus` to scrape metrics from Pods with label `app=redis`:

 ```yaml
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-ingress-from-prometheus
    namespace: default
  spec:
    podSelector:
      matchLabels:
        app: redis
    policyTypes:
      - Ingress
    ingress:
      - from:
          - podSelector:
              matchLabels:
                app: prometheus
  ```

As you can see in the editor's visualization, the above network policy will only work if both Pods are in the same namespace. The `podSelector` is scoped to the policy's namespace unless you explicitly use `namespaceSelector` to select other namespaces.

How do you do this right?

 Click here to see how to easily visualize and fix this policy in the editor.

###### Mistake 2: "There is no way it’s DNS..."

It is common that workloads must be locked down to limit external access (i.e. “egress” default deny). If you want to prevent your application from sending traffic anywhere except to Pods in the same namespace, you might create the following policy:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-kube-dns
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: foo
  egress:
    - to:
        - podSelector: {}
```

However, once you deploy this network policy, your application connectivity will likely be broken.

Why?

> It’s not DNS
> There is no way it’s DNS
> It was DNS

Pods will typically reach other Kubernetes services via their DNS name (e.g., service1.tenant-a.svc.cluster.local), and resolving this name requires the Pod to send egress traffic to Pods with labels `k8s-app=kube-dns` in the `kube-system` namespace.

So how do you solve this?

 Click here to see and fix the mistake in the editor

###### Mistake 3: Using Traditional Networking Constructs

If you come from a traditional Networking background, it might be tempting to use a /32 CIDR rule to allow traffic to the IP address of a Pod as shown in the output of `kubectl describe pod`. For example:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-pod
spec:
  podSelector:
    matchLabels:
      app: foo
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.2.125/32
```

However, Pod IPs are ephemeral and unpredictable, and depending on a network plugin implementation, ipBlock rules might only allow egress traffic to destinations outside of the cluster. Kubernetes documentation recommends using ipBlocks only for IP addresses outside of your cluster. So how do you solve this?

 Click to learn how to allow egress to Pod.

###### Mistake 4: Misunderstanding How Policy Rules Combine

Let's take a look at another egress policy example, that seeks to allow Pods with label `app=foo` to establish egress connections to an external VM with IP `192.168.1.22` on port `443`.

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-to-private-vm-443
spec:
  podSelector:
    matchLabels:
      app: foo
  egress:
    - to:
        - ipBlock:
            cidr: 192.168.1.22/32
    - ports:
        - port: 443
```

Wait... while this is valid YAML and a valid network policy, one extra character in the YAML made a huge difference here, and ended up allowing a lot more connectivity than expected. The additional “-” in front of “ports” meant that this is interpreted as two different rules, one that allows all traffic to the VM IP (on any port) and another that allows all traffic to port 443 (regardless of the destination IP address). The network policy specification dictates that the rules are logically OR'ed (not AND'ed), meaning the Pod workload has significantly more connectivity than intended.

How do you prevent these mistakes?

 Click to inspect the example in the Network Policy Editor

###### Mistake 5: Confusing Different Uses for “{}”

In Network Policy, empty curly braces (i.e., “{}”) can have a different meaning in different contexts, leading to a lot of confusion. We’ll use this last example as a quiz. What is the difference between these two similar looking network policy rules that both leverage “{}”? Take a guess, then look at each rule in the Network Policy Editor below to see if you were right.

 ```yaml
    ingress:
      - {}
    ```

 ```yaml
    ingress:
      - from:
          - podSelector: {}
    ```

 Get the answer with the Network Policy Editor

##### What's Next?

We hope you found these examples useful, and would love to hear from you if you have other examples of common Network Policy “gotchas” or other interesting policies to share with the community. Feel free to try making your own network policies or dropping in existing ones to visualize and check that they do what you want them to do.

To make sharing network policy examples easy, we have added a simple Share button that leverages Github Gist on the backend, enabling you to convert any example you have created into an easily shared link.

Tweet us at @ciliumproject to share your examples, and in the next few weeks we’ll pick a few favorites and send the creators some exclusive Cilium SWAG!

We’d love to hear your feedback and questions on both the editor and Network Policy in the **#networkpolicy** channel of Cilium Slack. See you there!

#### Kubernetes Networking & Load-Balancing with Cilium and F5

URL: https://cilium.io/blog/2021/01/29/how-to-build-k8s-networking-with-f5-and-cilium
Date: 2021-01-19
Categories: Community

##### Kubernetes Networking with Cilium and F5

This is a guest blog by Howard Hao. Howard is working as a Kubernetes
infrastructure SRE for [Ect888.com](http://www.ect888.com). In this post,
Howard will share how Ect888 uses [Cilium](https://cilium.io/) &
[F5](https://www.f5.com/) to build a scalable load-balancing infrastructure for
Ect888's production environment.

[Ect888.com](http://www.ect888.com) was established on January 8, 2015 and is a
chinese financial service provider with more than 200 employees serving more
than 2 million transactions per day in a Kubernetes environment that is hosting
thousands of pods.

##### Setting up Cilium and F5 to serve traffic to Kubernetes pods

F5 plays a vital role as a load balancer in Ect888.com, all Internet traffic
flows through the [Global Traffic
Manager(GTM)](https://www.f5.com/pdf/products/big-ip-global-traffic-manager-ds.pdf),
and the [Local Traffic
Manager(LTM)](https://www.f5.com/pdf/products/big-ip-local-traffic-manager-ds.pdf)
subsequently. We want the traffic from the LGTM to be forwarded to pod IPs directly
so that we can take advantage of LTM's advanced functions such as intelligent
caching, extensive connection optimization and management. After a period of
struggle, we found that Cilium which is an awesome CNI solution could achieve
our goal with BIRD, that aims to develop a fully functional dynamic IP routing
daemon and [F5 BIG-IP Controller for
Kubernetes](https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/v1.5/).

The following summarizes our environment and configuration.

###### Environment Details

###### Hardware

- F5: F5-VPR-LTM-B2150
- Switch: Cisco N5K-C5696Q

###### Software

- OS: CentOS Linux release 7.7.1908
- Kernel: 5.9.3-1.el7.elrepo.x86_64
- F5: BIG-IP 13.3.1.4
- Kubernetes: v1.19.4
- Cilium: 1.8.4
- BIRD: 2.0.7

###### Prerequisites

- A Kubernetes cluster with CNI enabled (We used kubeadm)

###### Configuration

 Step 1. Deploy Cilium on the Kubernetes cluster with [Helm](https://helm.sh/).

```shell-session
$ helm install cilium cilium/cilium --version 1.8.4 \
    --namespace kube-system \
    --set config.ipam=kubernetes \
    --set native-routing-cidr=<no_masquerade_ip_segment> \
    --set global.ipMasqAgent.enabled=true \
    --set global.kubeProxyReplacement=strict \
    --set global.k8sServiceHost=<your_apiserver_ip> \
    --set global.k8sServicePort=6443
```

All parameters above can be found in values.yaml, and it's noteworthy
that <code>native-routing-cidr</code> allows to explicitly specify the CIDR for
[native routing](https://docs.cilium.io/en/v1.8/concepts/networking/routing/#native-routing),
which disables [IP masquerading](https://docs.cilium.io/en/v1.9/concepts/networking/masquerading/#masquerading) executed by Cilium automatically for the traffic sent from Pod to LTM, so that
the traffic can be forwarded to Pods directly.

 Step 2. Use BIRD to run BGP between Kubernetes nodes and switch.

Configuration of BIRD:

```json
log syslog all;

router id <Node IP>;

protocol device {
        scan time 10;           # Scan interfaces every 10 seconds
}

# Disable automatically generating direct routes to all network interfaces.
protocol direct {
        disabled;               # Disable by default
}

# Forbid synchronizing BIRD routing tables with the OS kernel.
protocol kernel {
        ipv4 {                    # Connect protocol to IPv4 table by channel
                import none;      # Import to table, default is import all
                export none;      # Export to protocol. default is export none
        };
}

# Static IPv4 routes.
protocol static {
      ipv4;
      route <pod_cidr> via "cilium_host";
}

# BGP peers
protocol bgp uplink0 {
      description "BGP uplink 0";
      local <Node IP> as <BGP AS>;
      neighbor <Switch IP> as <BGP AS>;
      ipv4 {
              import filter {reject;};
              export filter {accept;};
      };
}

protocol bgp uplink1 {
      description "BGP uplink 1";
      local <Node IP> as <BGP AS>;
      neighbor <Switch IP> as <BGP AS>;
      ipv4 {
              import filter {reject;};
              export filter {accept;};
      };
}
```

Configuration of switch:

```
router bgp <BGP AS>
  neighbor <Node IP> remote-as <BGP AS>
    update-source loopback0
    address-family ipv4 unicast
```

Then, the switch should know Pod CIDR.

```shell-session
$ # run the following command on the switch.
$ show ip route bgp
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.16.0.0/24, ubest/mbest: 1/0
    *via 10.1.0.1, [200/0], 3w4d, bgp-64512, internal, tag 64512,
10.16.1.0/24, ubest/mbest: 1/0
    *via 10.1.0.2, [200/0], 3w4d, bgp-64512, internal, tag 64512,
10.16.2.0/24, ubest/mbest: 1/0
    *via 10.1.0.3, [200/0], 3w4d, bgp-64512, internal, tag 64512,
10.16.3.0/24, ubest/mbest: 1/0
    *via 10.1.0.4, [200/0], 3w4d, bgp-64512, internal, tag 64512,
10.16.4.0/24, ubest/mbest: 1/0
    *via 10.1.0.5, [200/0], 3w4d, bgp-64512, internal, tag 64512,
10.16.5.0/24, ubest/mbest: 1/0
    *via 10.1.0.6, [200/0], 3w4d, bgp-64512, internal, tag 64512,
```

 Step 3. [Deploy F5 BIG-IP Controller](https://clouddocs.f5.com/containers/latest/userguide/kubernetes/#how-to-deploy-kubernetes) on the Kubernetes cluster.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-bigip-ctlr-deployment
  namespace: kube-system
spec:
  # DO NOT INCREASE REPLICA COUNT
  replicas: 1
  selector:
    matchLabels:
      app: k8s-bigip-ctlr-deployment
  template:
    metadata:
      name: k8s-bigip-ctlr-deployment
      labels:
        app: k8s-bigip-ctlr-deployment
    spec:
      # Name of the Service Account bound to a Cluster Role with the required
      # permissions
      serviceAccountName: bigip-ctlr
      containers:
        - name: k8s-bigip-ctlr-standby
          image: f5networks/k8s-bigip-ctlr:latest
          env:
            - name: BIGIP_USERNAME
              value: 'F5 BIG-IP USERNAME'
            - name: BIGIP_PASSWORD
              value: 'F5 BIG-IP PASSWORD'
          command: ['/app/bin/k8s-bigip-ctlr']
          args: [
              # See the k8s-bigip-ctlr documentation for information about
              # all config options
              # https://clouddocs.f5.com/products/connectors/k8s-bigip-ctrl/latest
              '--bigip-username=$(BIGIP_USERNAME)',
              '--bigip-password=$(BIGIP_PASSWORD)',
              '--bigip-url=<ip_address-or-hostname>',
              '--bigip-partition=<name_of_partition>',
              '--pool-member-type=cluster',
              '--insecure=true',
            ]
```

In the deploy.yaml as shown above, the value of
<code>--pool-member-type</code> argument should be <code>cluster</code>, which
is <code>nodeport</code> as default, which integrates the BIG-IP device into
the Kubernetes cluster network.

**Note:** The above is a simple example, you should of course store the F5
BIG-IP password in a Kubernetes secret and refer to it instead.

###### Service Exposure

Now we can expose a Service via [F5 BIG-IP Container Ingress Services(CIS)](https://clouddocs.f5.com/containers/latest/userguide/what-is.html).

```yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: demo-vs
  labels:
    f5type: virtual-server
data:
  schema: 'f5schemadb://bigip-virtual-server_v0.1.7.json'
  data: |
    {
      "virtualServer": {
        "backend": {
          "servicePort": 80,
          "serviceName": "demoservice",
          "healthMonitors": [{
            "interval": 5,
            "protocol": "http",
            "send": "GET /health.html HTTP/1.1\r\nHost:1.1.1.1\r\n\r\n",
            "recv": "server is ok",
            "timeout": 16
          }]
        },
        "frontend": {
          "virtualAddress": {
            "port": 80,
            "bindAddr": "<virtual-server ip>"
          },
          "partition": "K8S",
          "balance": "least-connections-member",
          "mode": "http"
        }
      }
    }
```

CIS watches the Kubernetes API for specially formatted resources such as the
 yaml above, and updates the BIG-IP system configuration accordingly.
After the ConfigMap is applied, the virtual-server IP should be created
automatically, and the Service can be reached via the virtual-server IP.

##### Summary

Although most people run Kubernetes in a cloud environment, there are still
some companies that could only do this in private data centers. If you have F5
devices and want to integrate the BIG-IP device into your Kubernetes cluster
network, this article may give you some inspiration.

Finally, thanks for the great work of the Cilium teams, who build this
efficient and flexible open source software.

#### Cilium Zero Trust Networking Protections Against CVE-2020-8554

URL: https://cilium.io/blog/2020/12/11/kube-proxy-free-cve-mitigation
Date: 2020-12-11
Categories: How-To

_Contributed by Jed Salazar, Senior Solutions Architect, and Martynas Pumputis, Software Engineer_

You've probably heard about the new [Man in the Middle (MITM) vulnerability](https://github.com/kubernetes/kubernetes/issues/97076) in Kubernetes. If you're unfamiliar, a MITM vulnerability works by redirecting a victim's legitimate network traffic through a secret attacker on the network, where the attacker can eavesdrop or actively tamper with the victim's data before sending it to its intended destination. There have been several MITM vulnerabilities in Kubernetes, most of which take advantage of the default overly-permissive [`CAP_NET_RAW`](https://www.stackrox.com/post/2020/06/mitigating-kubernetes-cve-2020-10749/) permissions in Kubernetes. However this vulnerability is unique in two ways:

1. MITM attacks generally make use of common types of _network vulnerabilities_, whereas this vulnerability affects the API layer of Kubernetes itself.
2. Unlike most vulnerabilities that are assigned a Common Vulnerabilities and Exposures (CVE), there's no patch or hotfix you can deploy to protect your environment.

This vulnerability is also unique in another way: if you're running Cilium without [`kube-proxy`](https://docs.cilium.io/en/v1.9/gettingstarted/kubeproxy-free/), **you aren't vulnerable to it at all**. Let's talk about how.

##### CVE-2020-8554

The vulnerability affects [multitenant](https://cilium.io/blog/2020/07/27/2020-07-27-multitenancy-network-security) clusters by an attacker creating a ClusterIP service and updating the ExternalIP field with an IP address they intend to intercept traffic to and eavesdrop on. When running with kube-proxy, protection from this vulnerability requires that you implement a fairly complex set of mitigations such as [Open Policy Agent (OPA) Gatekeeper](https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general/externalip) or an [admission webhook](https://github.com/kubernetes-sigs/externalip-webhook).

##### Cilium with `kube-proxy`

At [Isovalent](https://isovalent.com/) (the company that makes Cilium), we've been busy building Zero Trust Networking into Cilium, mitigating common MITM attacks, and providing a safe-by-default network environment for multitenant clusters. To enable this, we've redesigned and replaced `kube-proxy`, meaning you can remove `kube-proxy` entirely and use Cilium's replacement instead. **When running Cilium with kube-proxy disabled, you are not vulnerable to this CVE** and are not forced to implement complex mitigations at the API layer.

##### Cilium's Zero Trust Approach to Security

How does Cilium prevent this vulnerability? Cilium performs a number of automatic network security mitigations based on Zero Trust Networking concepts. For example we don't translate an ExternalIP address for traffic sourced from pods unless the ExternalIP is associated with a known node. If you're running `kube-proxy` in parallel with Cilium, you're vulnerable to this attack because iptables will happily route traffic sourced from pods to an arbitrary ExternalIP.

To make sure you're getting built-in protection from MITM and other kinds of network attacks, we recommend running Cilium without [`kube-proxy`](https://docs.cilium.io/en/v1.9/gettingstarted/kubeproxy-free/).

##### Cilium without `kube-proxy`

Zero Trust Networking, mitigations against common types of MITM attacks, as well as significant performance improvements can be an intriguing reason to move to Cilium's [`kube-proxy` replacement](https://docs.cilium.io/en/v1.9/gettingstarted/kubeproxy-free/#kubeproxy-free).

If you're interested in learning more about Cilium's built in Zero Trust features, or just want to say hello, reach out to us on [Slack](http://slack.cilium.io/).

#### Cilium 1.9: Maglev, Deny Policies, VM Support, OpenShift, Hubble mTLS, Bandwidth Manager, eBPF Node-Local Redirect, Datapath Optimizations, and more

URL: https://cilium.io/blog/2020/11/10/cilium-19
Date: 2020-11-10
Categories: Release

We are excited to announce the Cilium 1.9 release. A total of 2816 commits have
been contributed by a community of 251 developers, many of whom made their first
contributions this cycle. Cilium 1.9 brings with it several brand new features:

- **Maglev Load Balancing**: Maglev provides load balancing with consistent
 hashing for high-availability scenarios by dynamically adapting to
 environments where nodes come and go. Connections are consistently balanced
 to backends even if the packets arrive at different load balancing nodes.
 ([More details](#maglev))
- **Deny Network Policies**: Users can now define network policies that
 explicitly reject traffic from specific sources or to specific destinations.
 These policies may be configured via CiliumNetworkPolicy or
 CiliumClusterwideNetworkPolicy, allowing multi-tenant environments to have
 baseline restrictions on connectivity and also delegate allow policies to
 application teams. ([More details](#deny-policy))
- **VM/Metal Support**: Cilium can now be deployed on any
 VM or baremetal node to connect that node to the Cilium cluster and
 represent external workloads running on such nodes as if the workload was
 running as a Pod inside the Kubernetes cluster. These nodes also gain access to
 Kubernetes services, and the visibility and policy enforcement scope of Cilium
 is extended to cover the VMs and baremetal nodes.
 ([More details](#vmsupport))
- **Bandwidth Manager**: A new bandwidth manager automatically optimizes TCP
 Congestion Control and other network settings for improved latency and
 throughput. Fair queueing is automatically enabled and rate
 limiting can be configured including support for the
 `kubernetes.io/egress-bandwidth` annotation.
 ([More details](#bwmanager))
- **OpenShift Support**: A new guide describes the installation of Cilium on
 OpenShift. Red Hat Universal Base Image (UBI) based builds and an operator
 to simplify installation will become available in the next weeks.
 ([More details](#openshift))
- **Hubble mTLS Automation**: Hubble is now capable of automatically generating
 and distributing TLS certificates between Hubble agents to secure the
 communication between all Hubble components.
 ([More details](#hubble))
- **eBPF-Based Node-Local DNS and KIAM**:
 The new eBPF-based Local Redirect Policies allow to build node-local
 DNS resolvers, KIAM, and other node-local services entirely with eBPF instead
 of relying on iptables. ([More details](#lrp))
- **Datapath Optimizations (iptables bypass)**: We have added two new eBPF
 helpers to the Linux kernel that allow implementing the direct routing
 datapath entirely in eBPF and bypassing netfilter/iptables entirely. This leads
 to significant improvements in throughput and latency, in particular for
 single stream TCP sessions. Cilium automatically enables these optimizations
 on kernel 5.10 or later. ([More details](#veth))
- **Lots of other improvements**: This release brings lots of additional
 improvements such as [High Availability for Cilium Operator](#operator-ha) to
 improve fault tolerance, [Hubble UI improvements](#ui), [Transparent Proxy
 Support](#tproxy) in eBPF, and a new [Performance Testing
 Framework](#perftest).

###### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms such as Kubernetes.

At the foundation of Cilium is a new Linux kernel technology called eBPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. eBPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption as well as network and service security. Besides
providing traditional network-level security, the flexibility of eBPF enables
security with the context of application protocols and DNS requests/responses.
Cilium is tightly integrated with Envoy and provides an extension framework
based on Go. Because eBPF runs inside the Linux kernel, all Cilium
functionality can be applied without any changes to the application code or
container configuration.

See the section **[Introduction to Cilium]** for a more detailed general
introduction to Cilium.

###### eBPF Summit 2020 - Recordings are now available

We hosted the first ever eBPF Summit in October this year. The recordings
including all keynotes and lightning talks are now available on the [eBPF
Summit website](https://ebpf.io/summit-2020).

###### Maglev Load Balancing

_Contributed by Daniel Borkmann and Martynas Pumputis_

In [Cilium 1.8](/blog/2020/06/22/cilium-18#kube-proxy-replacement-at-the-xdp-layer)
we extended Cilium's eBPF-based north-south load balancer to run at the XDP layer and
accelerate handling for Kubernetes service types `NodePort`, `LoadBalancer`, and
services with `externalIPs`. This significantly reduces CPU overhead and supports
handling millions of packets per second efficiently while being co-located with regular
user workloads. The new release complements the north-south load balancer by adding support
for [Maglev](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf)'s
consistent hashing algorithm for the backend selection.

By default, Kubernetes service load balancing implemented by kube-proxy or Cilium
selects service backends randomly and ensures that the traffic remains sticky to the backend.
However, the issue with this scheme is that in case of a node failure, the upstream load
balancer selects a different load balancing node which has no prior context on
which backend is currently serving the connection. This eventually leads to unexpected
disruptions on connection-oriented protocols like TCP as client connections are being reset
by the newly selected backends.

Maglev consistent hashing minimizes such disruptions by ensuring that each load balancing node
has a consistent view and ordering for the backend lookup table such that selecting the backend
through the packet's 5-tuple hash will always result in forwarding the traffic to the very
same backend without having to synchronize state with the other nodes. This not only improves
resiliency in case of failures but also achieves better load balancing properties since newly
added nodes will make the same consistent backend selection throughout the cluster.

Aside from that, the Maglev consistent hashing algorithm ensures even balancing among backends
as well as minimal disruptions in the case where backends are added or removed. Specifically,
a flow is highly likely to choose the same backend after adding or removing a backend for a service
as it did before the operation. Upon backend removal, the backend lookup tables are reprogrammed
with minimal changes for unrelated backends, that is, typical configurations
provide the upper bound of at most 1% tolerable difference in the reassignments.

In order to support Maglev in Cilium's eBPF load balancer, the datapath backend selection
algorithm is now pluggable between `maglev` and `random`. Maglev achieves the better
resiliency by trading off memory consumption for its potentially large lookup tables. Our
implementation optimizes the needed memory overhead by "compressing" the lookup tables into
an eBPF array-in-hash map lookup instead of using a single flat array for all services. This
allows us to scale memory on demand and ensures dynamic provisioning of services as is the
case with Kubernetes. To take memory reduction even further, we also [extended](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4a8f87e60f6db40e640f1db555d063b2c4dea5f1) the eBPF map-in-map infrastructure in the Linux kernel to enable
dynamic inner array map sizes. This paves the way for the next Cilium release to annotate
Kubernetes service specification with hints on whether a given service has a small or
large number of backends.

Maglev-based backend selection for Cilium's load balancer can be enabled by setting the Helm
option `loadBalancer.algorithm=maglev`. Further information and tunables on Maglev can be
found in its [getting started guide](https://docs.cilium.io/en/v1.9/gettingstarted/kubeproxy-free/#maglev-consistent-hashing-beta).

###### Deny-based Network Policy

_Contributed by André Martins_

By default, Cilium's behavior, as defined in Kubernetes, allows traffic to and
from Pods until a Kubernetes, or a Cilium, Network Policy selects that
particular Pod. This is powerful because it allows defining which Pods and other
entities can be allowed to communicate with the selected Pod. However,
this does not cover cases where it is useful to explicitly define which network
peers must _not_ be able to communicate.

The 1.9 release brings deny-based network policies to Cilium. These policies
will take precedence over any other rules as they are intended to explicitly
deny traffic. As an example, if a cluster is under attack by an external
entity, operators can immediately deploy the following policy to stop the
attack:

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: 'external-lockdown'
spec:
  endpointSelector: {}
  ingressDeny:
    - fromEntities:
        - 'world'
  ingress:
    - fromEntities:
        - 'all'
```

More information about deny policies can be found
[in the relevant documentation](https://docs.cilium.io/en/v1.9/policy/language/#deny-policies).

##### VM Support / External Workloads

_Contributed by Jarno Rajahalme_

A major focus of Cilium 1.9 has been to enable Cilium to run in other
environments where users are already running workloads. VM support, or more
generally speaking, the ability to join any node to the Cilium cluster, allows
integrating non-Kubernetes workloads with Cilium.

In many practical cases it is infeasible to move all service components to a
Kubernetes cluster at once. So far, Cilium has classified ingress from any
non-Kubernetes workload as either `WORLD`, making it impossible to distinguish
your own external nodes from any random Internet host, or as a `fromCIDR` that
allows distinguishing traffic from specific IP addresses or ranges. New support
for external workloads enables non-Kubernetes nodes to join the cluster, allowing
you to apply label-based policies on the traffic between the cluster Pods
and such external nodes. External nodes joining the cluster also get access to
cluster's services and can resolve cluster names using cluster's kube-dns.
This is a beta feature that will be further enhanced in coming releases. For more
information see the new [Setting up Support for External Workloads Getting Started Guide][vmsupport-gsg].

[vmsupport-gsg]: https://docs.cilium.io/en/v1.9/gettingstarted/external-workloads/

##### OpenShift Support

_Contributed by Ilya Dmitrichenko_

Support for OpenShift has been requested by many community members for a long
time now. While it was already possible to run on OpenShift, the lack of a good
guide made it challenging to get started. This release introduces a [Getting
Started Guide for OpenShift OKD][okd-gsg], the primary aim being to show
administrators how to install Cilium on new [OKD][] clusters. The guide relies
on `openshift-install` tool and has been tested in most of the major cloud
providers. This documentation has also been made available in the Cilium 1.8 documentation.
At the time of the 1.9.0 release, work is ongoing to enable further integration
with [OpenShift Container Platform][ocp] by automating installation steps with
an operator that will be submitted for RedHat certification.

[okd-gsg]: https://docs.cilium.io/en/v1.9/gettingstarted/k8s-install-openshift-okd/
[okd]: https://www.okd.io/
[ocp]: https://www.openshift.com/products/container-platform

###### Making Observability Easier with Hubble

_Contributed by Alexandre Perrin, Robin Hahling, Sebastian Wicki and others_

Hubble, the network and security observability component built into Cilium, has
gained many quality of life improvements with this release, making it even
easier to use and deploy.

In the previous release of Cilium, we [announced initial support for cluster-wide
visibility via _Hubble Relay_](/blog/2020/06/22/cilium-18#hubblerelay).
During the Cilium 1.9 development cycle, we have made continuous improvements to
Hubble Relay to ensure it works reliably. Many of these changes were backported
and have thus already been available in recent minor releases of Cilium 1.8.

In Cilium 1.9, connections from Hubble Relay to nodes is secured using mutual
TLS (mTLS) by default. Hubble Relay has also gained support for optional
TLS/mTLS when serving clients such as the Hubble CLI. Therefore, we are happy to
announce that cluster-wide visibility with Hubble Relay has reached _General
Availability_ (GA) with Cilium 1.9. We are excited for more users to try it out!

We have made many small improvements to ensure deploying and operating Hubble
remains easy. For example, TLS certificates required for mTLS are automatically
generated and distributed by default. Of course, sophisticated users can still
employ [their own certificate infrastructure](https://docs.cilium.io/en/v1.9/concepts/observability/hubble-configuration/#use-custom-tls-certificates).
This is notably required in a [Cluster Mesh scenario](https://docs.cilium.io/en/v1.9/concepts/clustermesh/) to get
cross-cluster visibility via Hubble Relay.

As another notable improvement, support for TLS certificates hot reloading was
added. This means that new TLS certificates are automatically picked up by
Cilium agent and Hubble Relay Pods without requiring a restart nor breaking any
existing connection. This allows for painless and frequent TLS certificate
renewals.

###### Hubble UI

_Contributed by Dmitry Kharitonov and Renat Tuktarov_

_Hubble UI_, the graphical user interface for Hubble, has been fully reworked for
this release. Its backend has been entirely rewritten in Go for overall better
performance, which means it now also scales better with the workload and cluster
size. The frontend and service dependency map interface has been redesigned
to provide better user experience, especially in busy namespaces with lots of
active services.

The new Hubble UI makes it easier to find dropped flows within a namespace by
highlighting them as special edges in the service map.

###### Hubble CLI

_Contributed by Glib Smaga, Michi Mutsuzaki, Robin Hahling, Sebastian Wicki and
others_

Cilium 1.9 also coincides with a new release of the Hubble CLI. Hubble CLI 0.7
supports all new additions to the Hubble and Hubble Relay API available in
Cilium 1.9.

One of the most visible changes is that Hubble Relay now reports the number of
nodes to which it is connected. This information is available when using the
`status` subcommand:

```shell-session
$ hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 81920/81920 (100.00%)
Flows/s: 9.80
Connected Nodes: 20/20
```

In case of connection failures, a list of unavailable nodes is shown. This
provides a good starting point to troubleshoot eventual connectivity issues of
Hubble Relay to nodes:

```shell-session
$ hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 24576/24576 (100.00%)
Flows/s: 9.80
Connected Nodes: 6/20
Unavailable Nodes: 14
  - kind-worker10
  - kind-worker11
  - kind-worker12
  - kind-worker14
  - kind-worker15
  - kind-worker16
  - kind-worker18
  - kind-worker7
  - kind-worker8
  - kind-worker9
  - and 4 more...
```

Hubble CLI now also supports new flags for HTTP method and path filters:

```shell-session
$ hubble observe --namespace galaxy --http-method POST --http-path /v1/request-landing
TIMESTAMP             SOURCE                                DESTINATION                           TYPE            VERDICT     SUMMARY
Oct 29 15:03:01.839   galaxy/tiefighter:59238               galaxy/deathstar-87c949bbd-gmw98:80   http-request    FORWARDED   HTTP/1.1 POST http://deathstar.galaxy.svc.cluster.local/v1/request-landing
Oct 29 15:03:01.840   galaxy/deathstar-87c949bbd-gmw98:80   galaxy/tiefighter:59238               http-response   FORWARDED   HTTP/1.1 200 1ms (POST http://deathstar.galaxy.svc.cluster.local/v1/request-landing)
```

Another quality of life improvement in the Hubble CLI is the new `hubble config`
sub-command to inspect and edit its config file, allowing users to persist
settings, such as the newly added support for TLS-protected API endpoints.

While most users will want to use the Hubble CLI to submit cluster-wide queries
against Hubble Relay, another notable change in Cilium 1.9 is that the
Cilium Pods now also expose the Hubble API locally by default, making it
easier to [troubleshoot node-local issues](https://docs.cilium.io/en/v1.9/operations/troubleshooting/#observing-flows-with-hubble).

This new version of the Hubble CLI contains many more features
([`jsonpb`](https://developers.google.com/protocol-buffers/docs/proto3#json)
support, support for fish and powershell completion, addition of flows per
second output in the status subcommand, Linux arm and arm64 binaries, and more).
See its [changelog] for
the full list of changes.

###### Node-Local Redirect Policies

_Contributed by Aditi Ghag and Weilong Cui_

The 1.9 release introduces Cilium Local Redirect Policies, which enable application
Pod traffic destined to an IP address and port/protocol tuple or Kubernetes service
to be redirected locally to a backend Pod within a node. The policies can be
gated by Kubernetes Role-Based Access Control (RBAC) framework.

There are two types of Local Redirect Policies supported. When traffic for
a Kubernetes service needs to be redirected, use the `ServiceMatcher` type.
When traffic matching IP address and port/protocol that does not belong to
any Kubernetes service needs to be redirected, use the `AddressMatcher` type.

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumLocalRedirectPolicy
metadata:
  name: 'nodelocaldns'
  namespace: kube-system
spec:
  redirectFrontend:
    serviceMatcher:
      serviceName: kube-dns
      namespace: kube-system
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        k8s-app: node-local-dns
    toPorts:
      - port: '53'
        name: dns
        protocol: UDP
      - port: '53'
        name: dns-tcp
        protocol: TCP
```

Local Redirect Policies are in beta. Follow the [local-redirect-policy guide]
to get more details, including how to configure them for use cases like
setting up NodeLocal DNSCache and KIAM proxy.

###### High Availability for cilium-operator

_Contributed by Deepesh Pathak_

The 1.9 release of Cilium brings High Availability for cilium-operator deployments.
Cilium operator is now an integral part of Cilium installation in Kubernetes environments,
and the agent requires the operator to be running before it can start functioning properly.
This dependency on the operator calls for a more robust deployment of cilium-operator in
the cluster. Cilium operator uses the Kubernetes
[leader election library](https://github.com/kubernetes/client-go/tree/release-1.19/tools/leaderelection)
in conjunction with lease locks to provide a HA cluster of cilium-operator instances.
The capability is supported on Kubernetes versions 1.14 and above and is Cilium's default
behavior for this release.

The number of replicas for the HA deployment can be configured using Helm option `operator.replicas`.

```shell-session
$ helm install cilium cilium/cilium --version 1.9.0 \
    --namespace kube-system \
    --set operator.replicas=3

$ kubectl get deployment cilium-operator -n kube-system
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
cilium-operator   3/3     3            3           46s
```

###### Bandwidth Manager

_Contributed by Daniel Borkmann_

The new Cilium Bandwidth Manager is responsible for
managing traffic more efficiently with the goal of improving overall application
latency and throughput. This functionality is focused in two areas, that is,
from an upper protocol and from a queueing discipline perspective.

When the Bandwidth Manager is enabled, it switches the TCP congestion
control algorithm by default to [BBR](https://cacm.acm.org/magazines/2017/2/212428-bbr-congestion-based-congestion-control/fulltext)
which achieves higher bandwidths and lower latencies in particular for Internet-facing
traffic. It configures the kernel's networking stack to more server-oriented sysctl
settings that have proven beneficial in production environments. And it reconfigures
the traffic control queueing discipline (Qdisc) layer to use multi-queue Qdiscs with Fair Queue (FQ)
on all external-facing network devices used by Cilium. With switching to
FQ, the Bandwidth Manager also implements support for Earliest Departure Time ([EDT](https://www.youtube.com/watch?v=MAni0_lN7zE))
rate-limiting with the help of eBPF and now natively supports the `kubernetes.io/egress-bandwidth`
Pod annotation.

This also removes the need for chaining the bandwidth CNI plugin which has
scalability limitations given its use of TBF (Token Bucket Filter). With the EDT-based
model, global locking in the Qdisc layer in particular under multi-queue NICs can
be avoided. Cilium's eBPF datapath classifies network traffic into a per-Pod
aggregate which then enforces the user-defined `kubernetes.io/egress-bandwidth`
rate by setting earliest departure timestamps on the network packets on egress,
shortly before passing the packet to the FQ leaf Qdiscs. The latter maintains a per-flow
state and schedules their departure by respecting that packets are not sent
out earlier than their timestamp dictates. Through the flexibility of eBPF, the
classification into the Pod aggregate works not only for direct routing, but also
in case of tunnelling or use of L7 proxies.

Evaluating the application latency under rate-limiting with HTB (Hierarchical Token Bucket) compared
to eBPF with FQ [shows](https://netdevconf.info/0x14/session.html?talk-replacing-HTB-with-EDT-and-BPF)
that the CPU utilization can be reduced significantly while improving transmission latency.
Latency reductions of about 20x have been observed for the 95th percentile as well as reductions of
about 10x for the 99th percentile when eBPF and FQ are used in combination.

Below is an example deployment for an application Pod whose egress bandwidth is
limited to 50 Mbit/s thanks to the `kubernetes.io/egress-bandwidth` annotation:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netperf
spec:
  selector:
    matchLabels:
      run: netperf
  replicas: 1
  template:
    metadata:
      labels:
        run: netperf
      annotations:
        kubernetes.io/egress-bandwidth: '50M'
    spec:
      containers:
        - name: netperf
          image: cilium/netperf
          ports:
            - containerPort: 12865
```

Cilium's Bandwidth Manager can be enabled by setting the Helm option `bandwidthManager=true`.
Further information on the Bandwidth Manager can be found in its [getting started guide](https://docs.cilium.io/en/v1.9/gettingstarted/bandwidth-manager/).

##### Virtual Ethernet Device Optimization with eBPF

_Contributed by Daniel Borkmann_

During the course of the 1.9 development cycle, we have performed a number of eBPF
datapath performance optimizations. One that stands out in particular is the
improvement of raw performance for network-namespaced Pods connected to the host
namespace through a veth device pair, as is the default operation mode in Cilium. When
Cilium is used in direct routing mode, traffic that is ingressing to or
egressing from Pods is passed up the network stack inside the host namespace in order
to let the routing layer perform the forwarding. Historically, this was a
necessity mainly for letting the netfilter subsystem masquerade egressing Pod
traffic. This masquerading also required the connection tracker to see traffic from both
directions in order to avoid drops from invalid connections (asymmetric visibility
of traffic in the connection tracker would apply here, too).

Thanks to recent advances of Cilium's datapath in
[prior releases](/blog/2020/06/22/cilium-18#voyaging-towards-iptables-free),
we are able to perform masquerading natively in eBPF code. Given that,
in the course of 1.9 development, we have extended the eBPF networking functionality for the
v5.10 kernel ([part1](https://lore.kernel.org/bpf/cover.1601477936.git.daniel@iogearbox.net/)
and [part2](https://lore.kernel.org/bpf/20201010234006.7075-1-daniel@iogearbox.net/))
in order for Cilium to handle forwarding right in the tc eBPF layer instead
of having to push packets up the host stack. This results in significant gains
in single stream throughput as well as significant reductions in latency for request/response-type
workloads. The two helpers `bpf_redirect_peer()` and `bpf_redirect_neigh()` that
we have added to the Linux kernel as well as Cilium 1.9 code base enable the new
packet handling in the host namespace.

The `bpf_redirect_peer()`
enables switching network namespaces from the ingress of the NIC to the ingress of the
Pod without a software interrupt rescheduling point when traversing
the network namespace. The physical NIC can thus push packets up the
stack into the application's socket residing in a different Pod namespace in _one
go_. This also leads to quicker application wake-up for picking up the received
data. Similarly, rescheduling points are reduced from 2 to 1 for local Pod-to-Pod
communication resulting in better latency there as well.

The `bpf_redirect_neigh()` handles a Pod's egress traffic by injecting the traffic
into the Linux kernel's neighboring subsystem, allowing to find the next hop and
resolving layer 2 addresses for the network packet. Performing the forwarding
only in tc eBPF layer and not pushing the packet further up the networking stack also
provides proper back pressure for the TCP stack and feedback for TCP's TSQ (TCP Small
Queues) mechanism to reduce potential excessive queueing of TCP packets. That is,
feedback is given to the TCP stack that the packet has left the node instead of
inaccurately providing it too early when it would be pushed up to the host stack for
routing. This is now possible because the packet's socket association can be kept
intact when it is passed down into the NIC driver.

As can be seen from initial results, the single stream TCP throughput for a Pod to
remote Pod session under direct routing doubles in throughput for the nodes under test
when using Cilium 1.9's eBPF extensions on a v5.10 kernel as opposed to having
both directions handled by the host stack's forwarding. Similarly, the TCP request/response
transaction performance improved by almost 3x for the Pods under test when avoiding the
host stack.

The underlying kernel is automatically probed from Cilium by default and if available
for the configuration the eBPF kernel extensions will be transparently enabled for new
deployments. The latter requires the use of Cilium's eBPF kube-proxy replacement as
well as eBPF-based masquerading given netfilter in the host namespace is then bypassed.
This behavior can also be opted-out through the Helm `bpf.hostRouting` option.

##### Transparent L7 Proxying in eBPF

_Contributed by Joe Stringer_

A recurring theme that we hear from Cilium users is around the pain that
iptables brings to cluster operators, whether due to the limitations around
scalability, the operational complexity for debugging, or when the existing
functionality provides a 90% solution but the last 10% is too difficult to
achieve. Cilium's datapath is built on eBPF specifically to mitigate these
concerns by building a network forwarding layer designed for the needs of the
network now, not the needs of the 1990s.

Building upon the [eBPF TPROXY] work in the upstream kernel earlier this year,
the Cilium 1.9 release enables L7 policy natively in eBPF on kernel 5.7 or
later. This implementation is now simpler by reducing dependence on iptables
and the packet mark. Earlier this year at Linux Plumbers conference, we
discussed how [network applications need to co-ordinate packet mark
usage](https://www.linuxplumbersconf.org/event/7/contributions/683/) to ensure
that the kernel implements the network forwarding behavior that users desire.
The work on transparently proxying L7 proxy natively in eBPF provides a path to
better integrate Cilium with other CNIs by reducing such packet mark usage. On
the journey to a world free of iptables, this is yet another positive step
forwards.

###### Performance Testing Framework

_Contributed by Kornilios Kourtis and Paul Chaignon_

Performance evaluation, and specifically evaluating network performance, is a
nuanced process. Small changes in the environment or setup (e.g., hardware,
kernel version and build options, system configuration, workload) can result in
substantial differences in measured performance. This makes it challenging to
evaluate Cilium's performance and causes confusion when seemingly similar setups
behave differently. Hence, in addition to the many performance improvements
Cilium 1.9 brings, we also developed utilities that enable both developers and
users to easily evaluate Cilium network performance in a reproducible way.

Specifically, we developed a benchmark for Kubernetes networking called
[kubenetbench](https://github.com/cilium/kubenetbench), as well as a [set of
tools](https://github.com/cilium/cilium-perf-networking) that cover the whole
process of performance evaluation: Provisioning machines, installing and
configuring software, and executing benchmarks. These tools not only fully
automate the evaluation, but also encompass best practices for producing
reliable results. Using the above tools, we performed an extensive evaluation
of Cilium network performance that is now [a part of our
documentation](https://docs.cilium.io/en/v1.9/operations/performance/), and
allows users to easily reproduce our results. Looking forwards, we plan to extend
our tools with more benchmarks and add this infrastructure to our CI process.

###### Getting Started

New to Cilium? Follow one of the [Getting Started Guides].

###### Upgrade Instructions

Follow the [upgrade guide] to upgrade your Cilium deployment. Cilium 1.9
deprecates the use of Helm 2 for managing deployments, as it is reaching
[end of life] on November 13, 2020. Furthermore, this release introduces new
option names for most Helm settings, see the guide for more details.
Feel free to ping us on [Slack] if you have any questions.

###### Release

- Release Notes: [1.9.0](https://github.com/cilium/cilium/releases/tag/v1.9.0)
- Container images: `quay.io/cilium/cilium:v1.9.0`, `docker.io/cilium/cilium:v1.9.0`

[slack]: https://slack.cilium.io
[getting started guides]: https://docs.cilium.io/en/v1.9/gettingstarted
[upgrade guide]: https://cilium.readthedocs.io/en/v1.9/operations/upgrade/#upgrading-minor-versions
[introduction to cilium]: https://docs.cilium.io/en/v1.9/intro/
[end of life]: https://helm.sh/blog/helm-v2-deprecation-timeline/
[local-redirect-policy guide]: https://docs.cilium.io/en/v1.9/gettingstarted/local-redirect-policy/
[ebpf tproxy]: /blog/2020/06/22/cilium-18#ebpf-based-tproxy-replacement
[changelog]: https://github.com/cilium/cilium/blob/v1.9/CHANGELOG.md

#### eBPF - The Future of Networking & Security

URL: https://cilium.io/blog/2020/11/10/ebpf-future-of-networking/
Date: 2020-11-10
Categories: Technology

Today is an exciting day for the Cilium community:
[Isovalent](https://www.isovalent.com), the company behind Cilium, is
announcing its [$29M Series A financing round](https://techcrunch.com/2020/11/10/with-29m-in-funding-isovalent-launches-its-cloud-native-networking-and-security-platform-based-on-ebpf-and-cilium/) backed by [Andreessen
Horowitz](https://a16z.com/2020/11/10/investing-in-isovalent/), Google, and Cisco. This is a perfect occasion to take a deeper look
into where eBPF-based networking is coming from and to understand what the
excitement is all about.

Two weeks ago, we hosted the first ever [eBPF Summit
2020](https://ebpf.io/summit-2020/). Besides exciting keynotes, several users
of Cilium spoke about their use cases including
[Adobe](https://www.youtube.com/watch?v=7UQ2CU6UEGY&t=5s), [Capital
One](https://www.youtube.com/watch?v=hwOpCKBaJ-w&t=28s&ab_channel=eBPFSummit),
[Datadog](https://www.youtube.com/watch?v=6mTVuZUHLBg&t=5s),
[GitLab](https://youtu.be/kwQ0ooO3UM8?t=5),
[Google](https://www.youtube.com/watch?v=oLS25ztnlMk), and [Wildlife
Studios](https://youtu.be/_1t3bXzptP0?t=10). Earlier last month, [Alibaba
Cloud](/blog/2020/10/09/cilium-in-alibaba-cloud) covered how they are using
Cilium for eBPF-based high-performance cloud-native networking. This was just a
couple of weeks after Google had
[announced](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine)
the availability of a Cilium-based networking dataplane for GKE and Anthos.

The last few months have already been incredibly exciting for the entire team.
It's not every day that one of the co-founders of Kubernetes praises the
technology and work your team has created.

Today's launch out of stealth is another great moment for the team and an
achievement of all team members who have worked incredibly hard for this day.
We are looking forward to talk more publicly about what we do as a company
while we continue to innovate around our open-source projects.

This moment is a great opportunity to provide a deeper dive into the motivation
to use eBPF for networking. What are all these users seeing in Cilium? What is
so special about eBPF?

##### The Roots of Programmability: Software Defined Networking

To really understand the shift, let's briefly look back at the history of
networking. In the 90s, networking was almost entirely physical. Cables,
perimeters, dial-up modems and a lot of Layer 2. Around the same time, in 1999,
iptables was created for Linux and just shortly after, PF was released for BSD.
Both projects focused on software-based firewalling. Early signs of a much
larger movement later on: software-defined networking. This was the time I
personally got involved in Linux networking.

In the years 2003-2008, VLANs were first described, the first release of the Xen
hypervisor happened, EMC acquired VMware, and KVM was merged into the Linux
kernel. This was the start of the virtualization era but from a networking
perspective, not much had changed. Networking of virtual machines was
delegated to the underlying physical network by bridging VMs directly to the
network. Almost no networking logic existed in software. During this early
virtualization era, most of the Linux kernel networking focus was on the TCP/IP
stack and optimizing the kernel as a system to run applications.

It was the year 2009 when things got exciting from a software networking
perspective. The first version of [Open vSwitch](https://www.openvswitch.org/)
was released which brought us software-defined networking (SDN). It brought
massive network programmability to the Linux kernel.

This programmability aspect can still be found in eBPF-based networking today
and is one of the corner stones. eBPF-based networking has its roots in SDN and
evolves it by removing the device-centric model from the equation.

##### The Rise of Containers and Kubernetes

The year 2013 brought Docker. Docker primarily inherited the networking from
the virtualization layers and containers were treated like miniature VMs from a
networking perspective. The fundamental shift that Docker brought focused on
application packaging with container images and not on the infrastructure side.
Therefore, almost all of the early networking solutions for containers were
inherited from OpenStack days.

In 2014, the first commit to Kubernetes happened. Kubernetes was obviously not
the first project to attempt translating high-level user intent into
infrastructure automation, but Kubernetes made a deliberate decision to make a
lot fewer assumptions in networking and security. For example, there is no
concept of a network or subnet in Kubernetes. This led to an impressive cycle
of innovations.

But the quick evolution of Kubernetes also has a dark side: The desire to get
to a complete enough state as quickly as possible made it rely on iptables. A
packet filter in the Linux kernel I personally worked on while still using a
dial-up modem. Clearly not perfectly suited for the task, but widely available
and good enough to get started.

##### The eBPF Revolution Begins

The same year that Kubernetes started, eBPF was first merged into the Linux
kernel as a successor to the long-standing packet filter BPF. Hence the
name extended BPF or short: eBPF. One year later, the eBPF backend was merged into the LLVM compiler
suite, allowing for LLVM to emit eBPF bytecode. In parallel, integration
into the kernel's traffic control layer made Linux networking programmable with eBPF.

2016, XDP was merged into the Linux kernel enabling a high-performance datapath
by allowing eBPF programs to run directly in the driver of a network device.
This is what later unlocked the development of eBPF-based high-performance load
balancers driving some of the largest data centers today.

Ever since, eBPF is in an incredibly steep trajectory to evolve further and
becomes more and more powerful every year. The general-purpose nature of eBPF
allowed for a diverse community to form around it, spanning networking,
tracing, security, profiling, and observability.

##### Cilium & eBPF - An ideal match for the Cloud-Native World

To recap history, with hardware networking, the functionality and scale were
mostly defined by the hardware. With software-defined networking a lot of it
moved from hardware to software by taking functionality previously provided
with hardware, rewrite it in software, and then put the word virtual in front.
All of this made sense as long as machines are involved.

Containers and the cloud-native era are not about machines. We care about
applications, APIs, and services. Some machines will still exist for a long
time but you don't want to build your architecture around them.

> However, in modern systems we rarely think about connecting machines, or
> virtual machines, or even containers really. Instead we’re concerned with
> connecting microservices. So rather than machines and wires, we think in
> terms of cloud services, APIs and the higher level protocols and the
> systems used to connect them. The past was IP addresses, ports, vNICs and
> VLANS. Now it is service identity, gRPC, Kafka, distributed data stores,
> remote APIs, etc.
>
> -- Martin Casado, Creator of SDN, Partner, a16z

What makes eBPF and thus Cilium such a great fit to address the new cloud
native challenges?

##### Programmability

The programmability of eBPF makes it possible to adjust to the quickly evolving
cloud-native requirements and tackle the increase in scale with ease. Here is
an example on how the programmability of eBPF lead Google to adopt Cilium as
its new networking dataplane for GKE:

> As more and more enterprises adopt Kubernetes, the gamut of use cases is
> widening with new requirements around multi-cloud, security, visibility and
> scalability. In addition, new technologies such as service mesh and
> serverless demand more customization from the underlying Kubernetes layer.
> These new requirements all have something in common: they need a more
> programmable dataplane that can perform Kubernetes-aware packet manipulations
> without sacrificing performance.
>
> Enter Extended Berkeley Packet Filter (eBPF), a new Linux networking
> paradigm that exposes programmable hooks to the network stack inside the
> Linux kernel.
>
> -- Gobind Johar, Product Manager, Google Kubernetes Engine

Even more important, eBPF is not networking specific or tied to a particular
domain. The generic nature of eBPF not only attracts a much bigger community to
innovate, it also avoids making premature assumptions about what building
blocks are required to solve future problems. This is a massive advantage over
any networking specific programmability solution such as iptables, Open
vSwitch, or nftables.

##### Embedded in the Linux kernel

Some of you may correctly state that programmability already existed in the
form of user-space networking. The unique new aspect of eBPF's programmability
is being embedded in the Linux kernel. Applications use system calls to
interact via the network and the Linux kernel is responsible to handle these
system calls. In order for a user-space networking framework to remain
transparent to the application, it still has to traverse the socket layer of
the Linux kernel. eBPF avoids this by remaining in the kernel altogether.

The reason why this was not more important before is because with virtual
machines, the hypervisor created a natural boundary between the network
device of the metal machine and the sockets of the operator system inside the
VM. With containers, this all happens in the same kernel.

##### Safety and Efficiency

Why not just load a Linux kernel module then? It obviously provides arbitrary
programmability at a very high efficiency. We could dive into the cost of
maintaining kernel modules across kernel versions, but the major downside is
more trivial: Safety while remaining efficient.

> Buggy kernel code will crash your machine. The kernel is not protected from
> a buggy kernel module. I think people assumed that this is just how things
> are; that's the price to do kernel programming. eBPF changed this dogma. It
> brought safety to kernel programming.
>
> -- Alexei Starovoitov, eBPF Co-Maintainer, Facebook

By requiring eBPF programs to pass through a verification process, eBPF
programs are significantly more secure than loading a kernel module.

The efficiency is guaranteed by a Just in Time (JIT) compiler that ensures
native execution speed of eBPF bytecode.

All of this makes eBPF incredibly powerful, but it is also a low-level
technology intended to be used primarily by Linux kernel developers. This is
where Cilium comes into play.

##### Cilium - eBPF-based Networking, Observability, and Security

Cilium is an open source project that has been designed on top of eBPF to
address the networking, security, and visibility requirements of container
workloads. It provides a high-level abstraction on top of eBPF. Cilium is to
eBPF what Kubernetes and container runtimes are to Linux kernel namespaces,
cgroups, and seccomp. The right abstraction layer on top.

Let's dive into particular use cases that Cilium is solving:

###### Networking

- **Network connectivity:** In its most basic form, Cilium is a CNI to provide
 network connectivity to Kubernetes workloads. The eBPF-based datapath
 features both IPv4 and IPv6 with the ability to support direct-routing,
 encapsulation/overlay topologies, as well as integration with cloud provider
 specific networking layers.

- **Service Load-Balancing:** Cilium can act as 100% kube-proxy replacement to
 provide all service load-balancing in a Kubernetes cluster. The
 implementation is highly scalable and supports direct server return (DSR)
 with session affinity. If possible, Cilium will perform the load balancing on
 the system call level and translate the address directly in the `connect()`
 system call instead of relying on network address translation throughout the
 entire duration of a network connection.

- **Edge Load-Balancing:** The XDP-based edge load-balancing capability can
 steer traffic into Kubernetes clusters or run entirely independent of
 Kubernetes. It supports consistent hashing with Maglev and provides an
 implementation for Kubernetes service types NodePort, LoadBalancer and
 services with externalIPs.

- **Multi-cluster connectivity + security:** With the multi-cluster ability, Kubernetes clusters can be
 connected without the introduction of additional gateways or proxies. The
 notion of global services allows to route service traffic across clusters.

- **Integration of VM/metal:** Virtual and metal machines can be seamlessly
 connected with Kubernetes workloads without proxies or gateways by
 representing such external workloads in the Kubernetes cluster as if the
 workload would run as a pod.

##### Security

- **Network Policy:** Full support for Kubernetes Network Policy based on a
 modern identity-based implementation built entirely in eBPF. Extensive
 visibility functionality eases problem troubleshooting and compliance
 monitoring.

- **FQDN/DNS-based:** Transparent integration with DNS-based service discovery
 allows for network policies based on DNS names instead of IP address
 blocks to cope with the highly dynamic nature of modern services backed by
 constantly changing set of IP addresses. The built-in DNS authorization
 further improves the security model.

- **API Awareness:** Transparent injection of Envoy and other proxies on an
 on-demand basis enables policy enforcement on API level, e.g. HTTP, Kafka,
 gRPC, Cassandra,...

- **Policy-driven SSL termination & injection:** SSL termination and injection
 of SSL certificates is policy driven, allowing to terminate SSL connections on
 behalf of an application or to transparently inject use of a certificate or
 token for service traffic without requiring to share the secret with the
 workload directly.

- **Simulation & Audit:** With policy simulation and policy audit, the effect
 of network policy changes can be inspected before dropping live traffic.

##### Observability

- **Flow logging:** Flow logs at L3-L7 provide deep visibility of forwarding
 and policy decisions on the network level. A cluster-wide flow query API
 enables the quick inspection of networking behavior and network drops during
 incidents.

- **Programmable Metrics:** Configurable and programmable metrics allow users
 to understand network, application, and security behavior and monitor the
 correctness consistently.

- **Service Map:** A graphical service topology map simplifies understanding of
 deployed application topologies and dependencies.

- **Troubleshooting:** The troubleshooting tooling has been built into Cilium
 from the beginning. An internal tracing system makes it possible to track every
 forwarding decision and can be enabled on the fly. Metrics capture packet
 drops with detailed reasoning.

For more details, check out the [Functionality
Overview](https://docs.cilium.io/en/stable/intro/#functionality-overview) in
the Cilium documentation.

##### Conclusion

Our team has had its fair share of exciting moments. The public launch of
Isovalent as a company is yet another major milestone.
Technically it's only a "Hello world" for something that our customers have
already been using for a while, but for our team, today is emotional. This year
has been challenging in many ways and not everything has always been within our
control. Being able to achieve and build something as a team that everbody can
be truly proud of will always be the most rewarding aspect of team and company
building to me.

I'm looking forward to work with many of you, as users, contributors, or
customers.

##### Further Reading

- [Learn more about Cilium](https://cilium.io/)
- [Learn more about eBPF](https://ebpf.io/)
- [Learn more about Isovalent](https://www.isovalent.com/)

#### eBPF Summit Day 2 Recap

URL: https://cilium.io/blog/2020/10/29/ebpf-summit-day-2
Date: 2020-10-29
Categories: Community

##### eBPF Summit Day Two

After an exciting start with the **[first day](/blog/2020/10/28/ebpf-summit-day-1)** of
the **[eBPF Summit](https://ebpf.io/summit-2020)**, the second and also final day of
the summit this year was kicked off with another warm welcome from Thomas Graf, during
which he reminded attendees:

> Bees are nice. Be a bee.

Thomas set the stage for an incredible list of presenters for today's keynote
talks, but not before announcing the winner of yesterday's trivia question
_Which mountain is behind Thomas in the video background?_ The correct answer was
provided by Slack user Bala and they are receiving a cute plush bee as a reward.
Some of the Slack user poll results were also presented and it was great to see
that a lot of participants were completely new to eBPF. Naming suggestions for
the eBPF mascot that we collected from the prior day also found a clear winner:

Full length video of the eBPF Summit, day two:

##### Safe Programs, the Foundation of BPF by Alexei Starovoitov (Facebook)

[See on Youtube.](https://www.youtube.com/watch?v=AV8xY318rtc)

Alexei Starovoitov from Facebook kicked off the keynotes with his talk titled
"Safe programs. The foundation of BPF." Alexei is the co-creator and co-maintainer
of eBPF. Alexei's talk was focused on eBPF as a means of extending and programming
the kernel, a much safer approach than with kernel modules. Thanks to the verifier,
programs are checked for safety, ensuring they will not crash the kernel and bring
the system to a halt.

Guaranteeing that safety isn't always easy. In particular, Alexei discussed how
the compiler and verifier can play a game of cat and mouse. With each new
optimization pass, the compiler obfuscates a little more the original intent of
the programmer, making it harder for the verifier to validate the compiled
bytecode is safe.

Alexei went on to explain how BPF Type Format (BTF) further strenghen that
safety, by providing a form of type safety to the constrained BPF C language.
What's next? Look out for new features in development, including the eBPF linker,
eBPF libraries, and eBPF dynamic linking extensions. Alexei closed with a quote:

> Maximum attention to safety in all aspects of BPF programming that's what makes it unique and that's why BPF is an undoubted choice today for kernel extensions and kernel programming.

##### Kernel Tracing in Production with Falco by Kris Nóva (Falco)

[See on Youtube.](https://www.youtube.com/watch?v=6E-xA0ZjIGM)

Following Alexei, Kris Nóva from Falco quickly stole the stage—or rather the
presentation screen—to present her own screen sporting a retro hacker theme.
Kris is a maintainer of the Falco project.
Her talk title changed multiple times throughout the weeks leading up to the summit,
finally landing on "How do we instrument the kernel without a kernel module in GKE?"
Falco uses kernel instrumentation to trace what's happening in the kernel and
publish alerts based on specific rules. To do this, a kernel module loaded on a host
provides the necessary access. But what about GKE where there is no support
for loading kernel modules on a host? This is where eBPF proved to be a natural
solution. Falco developers wrote an emulation device using eBPF and were able to
replace the kernel module. With an eBPF-based engine now able to run in place
of the Linux kernel module, the other parts of Falco were able to plug in easily
without much change. Kris went on to show some examples of Falco in action using
a kernel module, eBPF with clang 9 and clang 10—the main purpose of the examples
being to demonstrate the ease of moving between the different implementations and
how they all work to alert about suspicious activity.

##### Performance Wins with BPF: Getting Started by Brendan Gregg (Netflix)

[See on Youtube.](https://www.youtube.com/watch?v=wyfhjr_ufag)

After the break, Brendan Gregg from Netflix presented on "Performance wins with
BPF: Getting started." Brendan's primary goal is to help people find performance
wins with eBPF quickly and easily. Many articles about eBPF aren't for beginners,
but Brendan believes that, by thinking like sysadmins, not like programmers,
everyone can benefit from eBPF. Brendan highlighted some tools we can explore
that are available today, such as _execsnoop_ to find periodic
processes, _opensnoop_ to find misconfigurations or files not found, _tcplife_ to
inspect TCP session details, _ext4slower_ to find slow I/O performance, or _biosnoop_
for finding block I/O performance bottlenecks. These tools are just a few examples
of easily-accessible programs that can help find unusual activity for some quick
wins. Beyond these examples, there is a huge selection of tools included in the
bcc and bpftrace toolkits which can be easily leveraged for even more gains.

In a majority of cases, existing tools can solve the problem, but when they can't
then it's time to think like a programmer and build the solution. If you are just
getting started today, Brendan suggests to start with bpftrace because it is concise
and behaves like pseudo-code. If looking into bcc implementations, Brendan first
recommends to check out libbpf-based tools for an implementation instead. Thanks to
libbpf, Brendan was able to build _opensnoop_ into a single 151KB binary.

Brendan believes the future of eBPF is in development in the form of GUIs which
surface some of the same information, but use eBPF under the hood.
He believes that the ability to surface this data to operators or users, with no
knowledge of the intermediate system relying on eBPF, will open up new avenues.
In closing, Brendan encourages everyone to think like a sysadmin, install bcc and
bpftrace tools, run them, and get some wins. To dive even deeper into eBPF, you
should pick up a copy of Brendan's books, "BPF Performance Tools" and "Systems
Performance 2nd edition".

##### Kubernetes Network Policy Logging with eBPF by Zang Li (Google)

[See on Youtube.](https://www.youtube.com/watch?v=oLS25ztnlMk)

Zang Li from Google presented next on "Kubernetes Network Policy Logging with
eBPF." Zang works on GKE and Anthos and is also a Cilium core project member. She started
with an explanation of how Kubernetes network policies are implemented, along with
an example policy and scenario. She went on to highlight how differences in
implementation from one CNI provider to the next directly impact how policies
are implemented, whether that is via iptables, eBPF, Open vSwitch, or some other
implementation. In addition to enforcement, she added that security-conscious
customers will want to be able to log events related to policy decisions.

For Google, it was important this happened with minimal impact to the data path.
During their design phase, they quickly realized that iptables would not scale
to satisfy their complex networking requirements. Cilium provided the flexibility
and programmability they needed. In addition, its implementation of eBPF-based
network policies meant that Google could remove all reliance on iptables from
the data path. These factors ultimately led Google to select Cilium as the CNI
provider for GKE and Anthos.

Zang went on to explain how Google has extended, and makes use of, the Cilium
monitor infrastructure to extract and log useful data about network connections. To
provide visibility and observability into network connectivity, Google uses
Cilium monitor, which runs as a user space application and is able to access data
made available by eBPF programs through a perf ring buffer. This
buffer can contain any information they might need, such as event type, source,
policy verdict, direction, and more. Google further optimized this process by only
generating policy events on new connections, not on every packet observed, allowing
for a detailed policy logging engine.

Thanks to the partnership between Google and Cilium, users can leverage this
functionality on [Google Kubernetes Engine](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine)
today.

##### The Future of eBPF-based Networking and Security by Thomas Graf (Isovalent)

[See on Youtube.](https://www.youtube.com/watch?v=slBAYUDABDA)

Thomas Graf, CTO and co-founder of Isovalent, closed the keynote talks with
"The Future of eBPF-based Networking and Security." Before we can understand why
eBPF is the future, we need to briefly look at history. In the 1990s, networking
was almost entirely physical. It was also the time of dial-up modems. In 1999,
iptables was created as a successor for ipchains. In 2003, VLANs came to be, the Xen
hypervisor was born, and EMC bought VMWare. KVM was merged into the Linux kernel
in 2007, as virtualization era was upon us. However, in the case of networking,
almost nothing had changed. In 2009, the first release of Open vSwitch started
the network virtualization era. In 2010, OpenStack was born. 2013 brought us
Docker, which directly inherited networking from the virtualization layers and
containers were initially treated like miniature VMs. In 2014, Kubernetes was born and
deliberately did not make many networking implementation assumption. Despite this,
Kubernetes was initially heavily based on iptables, a system that literally was designed
15 years prior when modems ruled.

In that same year, 2014, eBPF was initially introduced to the Linux kernel.
2015 saw Linux networking become programmable and, in 2016, XDP was merged into
the kernel—and is now the base for many eBPF-based load balancing solutions.
In 2016, Cilium project was first announced, designed entirely from scratch, with
eBPF at its foundation. The transition was complete. We had moved from
an era first built on top of physical servers, to an era based on virtual
machines where software-defined networking was first born, to nowadays
where the networking and security layers are built specifically for
cloud-native workloads. Google's adoption of the eBPF-based Cilium project for
its flagship Kubernetes products GKE and Anthos is a great example on how eBPF
plays a central role in networking and security for Kubernetes and cloud-native
environments.

What's next? Thomas believes the industry will start to focus on edge load
balancing with eBPF and XDP. He sees continuing opportunities to improve
application awareness by further improving pod-to-pod and socket-to-socket
networking. Similarities between API and system calls will lead to a more
unified approach to network and system
security, with less separation of concerns between policy, enforcement and alerting.
Finally, he sees an opportunity to bridge physical and
virtualized or containerized workloads, as well as eBPF-based service meshes
being implemented more in the kernel.

For the Cilium project, it has been an exciting trip over the past 4 years. It's
almost certain to become more exciting as eBPF opens up additional possibilities,
not just for networking but for all forms of observability, security, tracing,
and so much more.

##### Lightning Talks

After another break, we returned for another incredible round of lightning talks,
all of which were just 5 minutes in length but were absolutely full of amazing
content:

- Giulia Frascaria (Vrije Universiteit Amsterdam) - [Can eBPF Save us from the Data Deluge? A Case for File Filtering in eBPF](https://www.youtube.com/watch?v=qEtDoHWYrhA)
- Timo Reimann (Digital Ocean) - [From Managed Kubernetes to App Platform: 1.5 Years of Cilium Usage at DigitalOcean](https://www.youtube.com/watch?v=xez34h7EY3A)
- Zain Asgar (Pixie) - [Debugging Go in Prod with eBPF](https://www.youtube.com/watch?v=Q7mqIB7eatQ)
- Martynas Pumputis (Isovalent) - [North-South Load Balancing of Kubernetes Services with eBPF/XDP](https://www.youtube.com/watch?v=XN3mJF7g8dQ)
- Luan Guimarães (Wildlife Studios) - [Global Gaming Infrastructure with Cilium](https://www.youtube.com/watch?v=_1t3bXzptP0)
- Ramiro Berrelleza (Okteto) - [The Tale of Smokey and the Crypto Bandits](https://www.youtube.com/watch?v=tplv3Hjjv2Q)
- Vlad Ungureanu (Palantir) - [Past, Present, and Future of Cilium and Hubble at Palantir](https://www.youtube.com/watch?v=3K5WJ_h5PhI)
- Itay Shakury (Aqua Security) - [Tracing and Detecting Malware using eBPF](https://www.youtube.com/watch?v=UD7bF6nWagQ)
- Natalia Reka Ivanko (Isovalent) - [Identity Aware Threat Detection and Network Monitoring by using eBPF](https://www.youtube.com/watch?v=JQZQwR57Lmg)
- William Findlay (Carleton University) - [bpfbox: Simple Precise Process Confinement with eBPF and KRSI](https://www.youtube.com/watch?v=evHw9QHElNE)
- Lorenz Bauer (Cloudflare) - [How to Ship BPF with your Go Project](https://www.youtube.com/watch?v=lijS-0da5U8)
- Yutaro Hayakawa (LINE) - [eBPF at LINE's Private Cloud](https://www.youtube.com/watch?v=cxfVpBYlol4)
- Dinesh Venkatesan (Microsoft) - [Building a Behavioral Knowledge Graph using eBPF](https://www.youtube.com/watch?v=Sun0bWGVl_o)
- Andrey Ignatov (Facebook) - [Containers and BPF: twagent story](https://www.youtube.com/watch?v=lO0dYHl3I8Y)

We hope you had as much fun attending the eBPF Summit as we did hosting it.
We'd like to give a huge thanks to everyone who attended, all of our speakers and
the amazing team of people who worked in the background to support the event and
make it a success.

For more information about eBPF, you can visit the [ebpf.io](https://ebpf.io)
website or join us on [Slack](https://ebpf.io/slack).

#### eBPF Summit Day 1 Recap

URL: https://cilium.io/blog/2020/10/28/ebpf-summit-day-1
Date: 2020-10-28
Categories: Community

##### eBPF Summit Day One

The first day of the **[eBPF Summit](https://ebpf.io/summit-2020)** is a wrap and it certainly was an amazing
day full of information about eBPF, the technology that is changing the shape
of Linux networking, observability, and performance.

If you missed the keynote and lightning talks from Day 1 you can **[watch the replay](https://www.youtube.com/watch?v=1GSgyvn4N7E&feature=youtu.beif)**
of the event in its entirety. Individual talks, along with their links,
will be made available within the next week.

Thomas Graf started by greeting attendees with a warm message of "Be kind,
be human." He went on to say that the team expected hundreds of attendees, but
the event turnout far surpassed any original expectations with over 2,200+ registrations
from over 1,100+ different companies.

Thomas then set the stage for the keynote and
lightning talks to follow which would prove to be a very diverse set of topics from a
diverse group of presenters ranging from college graduate students to industry leaders.
Whether you're a newcomer to eBPF or a veteran with years of experience,
there's sure to be something new for you.

Full-length video of the eBPF Summit, day one:

###### A Beginner’s Guide to eBPF Programming by Liz Rice (Aqua)

[See on Youtube.](https://www.youtube.com/watch?v=lrSExTfS-iQ)

The opening keynote talk, titled "Beginners Guide to eBPF Programming," was
presented by Liz Rice from Aqua Security. Liz started us off with a brief
explanation of what an eBPF program is, along with explaining the difference
between user and kernel space. She went on to provide an overview of what an eBPF
map is, and how an eBPF program can be triggered based on an event. She then
demonstrated how to easily and quickly build an eBPF program that can print
"Hello, World" based on a kprobe. A kprobe is commonly used for debugging or
monitoring production systems. After showing us how to do this with the bcc
library, Liz then demonstrated how to store this data in an eBPF map, which
creates a more scalable solution, as well as how to retrieve that data from a
user space application. If you are new to eBPF, Liz provided an amazing
introduction in this opening keynote talk.

###### BPF as a Fundamentally Better Dataplane by Daniel Borkmann (Isovalent)

[See on Youtube.](https://www.youtube.com/watch?v=Qhm1Zn_BNi4)

Next up, Daniel Borkmann presented a talk titled "BPF as a Fundamentally Better
Dataplane." Daniel started with an overview of eBPF and how it functions as an
execution engine. Daniel cited the improvements that have been
made to the kernel's support for eBPF, including support for eBPF to eBPF functions
calls, bounded loops, global variables, static linking, BTF, and support for up to
1 million instructions per program. All of this allows for solving a lot of
interesting use cases. Daniel also discussed reducing the kernel's attack surface
with eBPF and showed how the workflow to implement an eBPF-based fix can be much
faster and easier to deploy than waiting on a backported kernel from a specific
distribution. He went on to talk about improving kernel scalability and extensibility
with eBPF and how XDP compares to DPDK performance in load balancing and DDoS
mitigation use cases. In addition, he discussed how implementing eBPF-based policy
for containers and Pods can perform better and faster than a traditional firewall.
Then he discussed how an eBPF-based solution can be used in the case of Pod-to-Pod
networking to improve raw performance, as well as in the case of service load balancing
to a backend application. He detailed how recently-merged support for
`bpf_redirect_peer()` and `bpf_redirect_neigh()` can improve Pod-to-Pod connectivity
by eliminating the host stack overhead, resulting in significant performance gains.
Finally, he showed how eBPF can be used to implement bandwidth management, TCP
congestion control, and more.

###### Our eBPF Journey at Datadog by Tabitha Sable and Laurent Bernaille (Datadog)

[See on Youtube.](https://www.youtube.com/watch?v=6mTVuZUHLBg)

After a short break, Tabitha Sable and Laurent Bernaille presented "Our eBPF
Journey at Datadog." Datadog runs tens of thousands of hosts, dozens of Kubernetes
clusters, and operates on multiple cloud providers. Their cloud architecture
relies on per-Pod routable IP addresses, ensuring that each cluster is directly
accessible using a unique range. As a result, IP address space management and
cross-cluster discovery become more challenging. Initially, Datadog relied on
various CNI plugins for each provider but there were differences between
providers. Network policy support was lacking in many cases, and there was no
easy way to implement end-to-end encryption. For service load balancing, the
initial design relied on kube-proxy and iptables, however at scale this
overhead became challenging as well. While IPVS was able to alleviate some of
the initial pain, it came with the additional challenges of connection
tracking, not at one but two layers, and IPVS lacked feature parity with iptables.
Inherently, neither solution was designed to be a client-side load balancer,
especially not for Kubernetes. As a result of these requirements, Datadog
selected Cilium as their CNI plugin. With Cilium, Datadog was able to
completely remove kube-proxy and also began enforcing network policies using eBPF.
Cilium was a good fit because it is a universal CNI for multiple cloud providers,
and has the ability to easily enable end-to-end encryption. In addition to looking
at internal traffic, Datadog is also exploring eBPF for network edge filtering,
DDoS mitigation, routing, and even more as they continue to build eBPF into their
own product offerings around security, compliance, and network performance.

###### Security Auditing and Enforcement using eBPF by KP Singh (Google)

[See on Youtube.](https://www.youtube.com/watch?v=XFJw37Vwzcc)

Next up, KP Singh from Google presented "Security Auditing and Enforcement using
BPF." His talk focused on his motivations for building Linux Security Modules on
eBPF. In 2019, KP was presented with a request for some audit data which was not
available and his work in this area led him to building an all-new way to do
auditing and enforcement in Linux. Kernel Runtime Security Instrumentation, or
KRSI, is responsible for both monitoring what is taking place on a system along
with the enforcement. Around 200 LSM hooks provide all the data needed for
LSMs to make appropriate decisions. He then showed us the code—or rather,
walked us through an eBPF program line by line. To close, he presented an overview
of new eBPF features recently merged into the kernel, such as eBPF ring buffers,
the `bpf_d_path()` helper, storage blobs aka `bpf_local_storage()`, sleepable
eBPF programs, and boot-time loading. Work on atomic operations is presently in
progress. KP believes that, while eBPF may not replace other LSMs, the two
solutions can peacefully co-exist.

###### Lightning Talks

Finally, there was a great selection of lightning talks, each one just 5 minutes
in length but full of useful information for the eBPF community. Here's a quick
list of the presenters and topics:

- Bryce Kahle (Datadog) - [How and When You Should Measure CPU Overhead of eBPF Programs](https://www.youtube.com/watch?v=b0TxKiGMWpI)
- Brandon Cook (Adobe Systems) - [eBPF at Adobe](https://www.youtube.com/watch?v=7UQ2CU6UEGY)
- Andreas Gersmeyer (Red Hat) - [Using BCC and bpftrace with Performance Co-pilot](https://www.youtube.com/watch?v=XmMVhvjmD9I)
- Bradley Whitfield (Capital One) - [Building a Secure and Maintainable PaaS](https://www.youtube.com/watch?v=hwOpCKBaJ-w)
- Lorenzo Fontana (Sysdig) - [Debugging the eBPF Virtual Machine](https://www.youtube.com/watch?v=W6rgaghycFI)
- Jianlin Lv (Arm) - [Enabling eBPF Superpowers on ARM64 with Cilium](https://www.youtube.com/watch?v=Sk_Kn-1pWt8)
- Beatriz Martinez (Isovalent) - [Zero Instrumentation Monitoring with Your First Steps in eBPF](https://www.youtube.com/watch?v=GaY2d8e-gk0)
- Javier Honduvilla Coto (Facebook) - [rbperf: Understanding Ruby with BPF](https://www.youtube.com/watch?v=oeGom1zl0a8)
- Pablo Moncada (MasMovil) - [Scaling a Multi-tenant K8S Cluster in a Telco](https://www.youtube.com/watch?v=JH3pcmhNEHA)
- Jakub Sinicki (Cloudflare) - [Steering Connections to Sockets with BPF Socket Lookup Hooks](https://www.youtube.com/watch?v=vCJ8kDYI8ZE)
- Manali Shukla (Cisco) - [Hardware Breakpoint Implementation in BCC](https://www.youtube.com/watch?v=Nw-tTmxIHUA)
- Sam White (Gitlab) - [Securing Kubernetes Clusters with DevSecOps and Gitlab](https://www.youtube.com/watch?v=kwQ0ooO3UM8)
- Lou Xun (CCP Games) - [Traffic Control the Rabbit(MQ) with Rust using RedBPF](https://www.youtube.com/watch?v=s-Tn-xjUnPE)

Thomas Graf wrapped up the day by thanking all of the presenters as well as the
team working in the background to help support the event and make it a success.
Read on for our coverage of
**[day 2 of the eBPF Summit](/blog/2020/10/29/ebpf-summit-day-2)**.

#### eBPF Summit Oct 28-29, 2020

URL: https://cilium.io/blog/2020/10/20/ebpf-summit-2020
Date: 2020-10-20
Categories: Community

The first ever eBPF Summit is just over one week out and we are already well
past 1'000 registrations. We have been blown away by the number of talk and
keynote proposals. This also meant that we had to turn down many fantastic
talks and we hope that our selection provides an interesting and diverse set of
talks. Some of the content that did not make the schedule will be posted as
blog posts on ebpf.io over the coming weeks. In this post, we'll give a quick
preview of what to be excited about and how to plan to attend the event.

If you have not registered for the event, [register
here](https://ebpf.io/summit-2020/). The event is free and held fully virtual.

##### What is eBPF Summit?

eBPF is quickly becoming one of the most talked about technologies in our
industry - enabling a new generation of networking, tracing, observability and
security infrastructure. The eBPF Summit is targeted at users and potential
users of this groundbreaking technology - both those looking at leveraging eBPF
directly or via one of the projects leveraging eBPF (e.g. bpftrace, Cilium,
Falco, etc.)

The summit will be held Oct 28-29, 2020 in a fully virtual setting and will
feature a mix of voices from the various eBPF communities. We'll hear from the
eBPF maintainers, various eBPF-based projects, eBPF book authors but most
importantly, we'll hear from a wide set of eBPF users sharing their experience
on how they leverage and utilize eBPF.

##### Keynotes

On both days, we will feature a set of 15 min keynotes from interesting people
in the eBPF space. We'll hear from both eBPF Co-Maintainers, eBPF-based project
maintainers, book authors, and otherwise awesome people.

###### Day One: Oct 28

- **A beginner’s guide to eBPF programming**
 Liz Rice, VP, Open Source Engineering, Aqua
 [See on Youtube.](https://www.youtube.com/watch?v=lrSExTfS-iQ)
- **BPF as a fundamentally better dataplane**
 Daniel Borkmann, eBPF Co-Maintainer, Isovalent
 [See on Youtube.](https://www.youtube.com/watch?v=Qhm1Zn_BNi4)
- **Our eBPF Journey at Datadog**
 Laurent Bernaille, Staff Engineer & Tabitha Sable, Systems Security Engineer Datadog
 [See on Youtube.](https://www.youtube.com/watch?v=6mTVuZUHLBg)
- **Security Auditing and Enforcement using eBPF**
 KP Singh, Kernel Runtime Security, Google
 [See on Youtube.](https://www.youtube.com/watch?v=XFJw37Vwzcc)
- **Topic to be announced**
 David Miller, Kernel Maintainer, Red Hat

###### Day Two: Oct 29

- **Safe programs. The foundation of BPF**
 Alexei Starovoitov, eBPF Co-Maintainer, Facebook
 [See on Youtube.](https://www.youtube.com/watch?v=AV8xY318rtc)
- **Topic to be announced**
 Kris Nova, Software Engineer & Open Source Advocate
 [See on Youtube.](https://www.youtube.com/watch?v=6E-xA0ZjIGM)
- **Performance Wins with BPF: Getting Started**
 Brendan Gregg, Author of “BPF Performance Tools“, Lead Performance Engineer, Netflix
 [See on Youtube.](https://www.youtube.com/watch?v=wyfhjr_ufag)
- **Kubernetes Network Policy Logging with eBPF**
 Zang Li, Cilium Core Team Maintainer, Google
 [See on Youtube.](https://www.youtube.com/watch?v=oLS25ztnlMk)
- **The Future of eBPF Networking and Security**
 Thomas Graf, Co-Creator of the Cilium Project, Isovalent
 [See on Youtube.](https://www.youtube.com/watch?v=slBAYUDABDA)

##### Lightning Talks

We have received numerous submissions for the lightning talk sessions which
will be spread over both days as well. The quality of the submissions has been
overwhelming. We have explicitly chosen to only run lightning sessions to give
a voice to as many members in the community as possible. Despite this, we have
received more than 3 times as many submissions as available speaking slots so
we had to turn down many awesome talks. We will definitely repeat this event
and will consider expanding the scope of it to provide more speaking
opportunities.

The schedule for the lightning talks is still being finalized, as it stands
right now, the following talks will be presented during the lightning talk
session. We will announce the exact speaking times as we receive all title,
abstract and session information from all the speakers.

| Title | Speaker |
| --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| [Using BCC and bpftrace with Performance Co-Pilot](https://www.youtube.com/watch?v=XmMVhvjmD9I) | Andreas Gerstmayr, Red Hat |
| [Containers and BPF: twagent story](https://www.youtube.com/watch?v=lO0dYHl3I8Y) | Andrey Ignatov, Facebook |
| [Zero instrumentation monitoring with your first steps in eBPF](https://www.youtube.com/watch?v=GaY2d8e-gk0) | Beatriz Martínez, Isovalent |
| [eBPF at Adobe](https://www.youtube.com/watch?v=7UQ2CU6UEGY) | Brandon Cook, Adobe |
| [How and When You Should Measure CPU Overhead of eBPF Programs](https://www.youtube.com/watch?v=b0TxKiGMWpI) | Bryce Kahle, Datadog |
| [Building a Behavioral Knowledge Graph using eBPF](https://www.youtube.com/watch?v=Sun0bWGVl_o) | Dinesh Venkatesan, Microsoft |
| [Can eBPF save us from the data deluge? A case for file filtering in eBPF](https://www.youtube.com/watch?v=qEtDoHWYrhA) | Giulia Frascaria, Vrije Universiteit Amsterdam |
| [Tracing and detecting malware using eBPF](https://www.youtube.com/watch?v=UD7bF6nWagQ) | Itay Shakury, Aqua Security |
| [Steering connections to sockets with BPF socket lookup hook](https://www.youtube.com/watch?v=vCJ8kDYI8ZE) | Jakub Sitnicki, Cloudflare |
| [rbperf: Understanding Ruby with BPF](https://www.youtube.com/watch?v=oeGom1zl0a8) | Javier Honduvilla Coto, Facebook |
| [Enabling eBPF super powers on ARM64 with Cilium](https://www.youtube.com/watch?v=Sk_Kn-1pWt8) | Jianlin Lv, Arm |
| [How to ship BPF with your Go project](https://www.youtube.com/watch?v=lijS-0da5U8) | Lorenz Bauer, Cloudflare |
| [Debugging the eBPF Virtual Machine](https://www.youtube.com/watch?v=W6rgaghycFI) | Lorenzo Fontana, Sysdig |
| [Traffic Control the Rabbit with Rust using RedBPF](https://www.youtube.com/watch?v=s-Tn-xjUnPE) | Lou Xun, CCP Games |
| [Implementation of Hardware Breakpoint in BCC](https://www.youtube.com/watch?v=Nw-tTmxIHUA) | Manali Shukla, Cisco Systems India |
| [Identity Aware Threat Detection and Network Monitoring by using eBPF](https://www.youtube.com/watch?v=JQZQwR57Lmg) | Natalia Reka Ivanko, Isovalent |
| [Scaling a multi-tenant k8s cluster in a Telco](https://www.youtube.com/watch?v=JH3pcmhNEHA) | Pablo Moncada, MasMovil |
| [The Tale of Smokey and the Crypto Bandits](https://www.youtube.com/watch?v=tplv3Hjjv2Q) | Ramiro Berrelleza, Okteto |
| [Securing Kubernetes Clusters with DevSecOps and GitLab](https://www.youtube.com/watch?v=kwQ0ooO3UM8) | Sam White, GitLab |
| [From Managed Kubernetes to App Platform: 1.5 Years of Cilium Usage at DigitalOcean](https://www.youtube.com/watch?v=xez34h7EY3A) | Timo Reinmann, DigitalOcean |
| [bpfbox: Simple Precise Process Confinement with KRSI and eBPF](https://www.youtube.com/watch?v=evHw9QHElNE) | William Findlay, Carleton University |
| [eBPF in LINE](https://www.youtube.com/watch?v=cxfVpBYlol4) | Yutaro Hayakawa, LINE Corporation |

##### Conclusion

I'm personally extremely thrilled about the summit and look forward to it with
excitement. In particular about the diversity of voices that we will hear from.
eBPF is quickly turning into one of the most important technologies and the
broad community pushing it forward is one of the most important factors of
eBPF's success.

- [Conference website & Registartion](https://ebpf.io/summit-2020/)
- [What is eBPF?](https://ebpf.io/what-is-ebpf)

#### How Alibaba Cloud uses Cilium for High-Performance Cloud-Native Networking

URL: https://cilium.io/blog/2020/10/09/cilium-in-alibaba-cloud
Date: 2020-10-09
Categories: Community

A couple of weeks ago, the Alibaba team presented details on the new datapath
for the Alibaba Cloud during the SIG Cloud-Provider-Alibaba meeting and also
published a [blog post](https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)
with the technical architecture. Guess what, it is all Cilium & eBPF based.
Alibaba Cloud is not the first cloud provider to directly embed Cilium.
Recently, Google [announced](/blog/2020/08/19/google-chooses-cilium-for-gke-networking/)
the availability of Dataplane V2 based on Cilium & eBPF for GKE and Anthos.
What is unique to Alibaba's setup is the use of IPVLAN as we will learn
below. In this blog post, we will dive into the details of the Terway CNI, the
CNI plugin of Alibaba Cloud, and review some of the statements made in the blog
post.

Like almost all cloud providers, Alibaba supports a concept of ENI (elastic
network interface) with the purpose of exposing the SDN (software defined
networking), which already supports virtualization, to pods. This avoids
building virtualized networking on top of already virtualized networking and
can avoid overhead and unnecessary complexity.

> The cloud IaaS layer network is already virtualized. If network
> virtualization is further performed in pods, the performance loss is
> significant.

For Alibaba, it looks something like this:

(Source: https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)

In order to implement this model, the CNI layer has to interact with the Cloud
APIs to set up ENIs as needed for pods. For this purpose, Alibaba has built the
[Terway CNI plugin](https://github.com/AliyunContainerService/terway) plugin. Check
out [Alibaba's original blog post](https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)
to learn about the internals of Terway CNI plugin and some of the challenges
they faced. This post specifically focuses on IPVLAN and how eBPF is used improve
the scalability of Kubernetes services and Network Policy.

##### IPVLAN for better Network Scalability and Performance

An ENI can either be dedicated to a pod or shared between multiple pods. When
an ENI is shared, an additional routing decision in software has to be made to
route packets to and from the individual pods attached to the ENI.
Sharing an ENI has a massive scalability advantage as each ENI supports between
10-20 addresses and thus far fewer ENIs are needed in the entire cluster. The
downside of sharing an ENI is that the additional virtual bridge introduces
some overhead. Performance numbers below illustrate the difference.

In order to implement sharing of an ENI, IPVLAN is a great option to put a
high-performance virtual bridge in place and connect multiple pods to a single
ENI. For this purpose, the Terway CNI plugin is integrating with Cilium to
provide a flexible ENI datapath with an efficient Network Policy and Services
implementation. This work is currently being upstreamed by the Alibaba team in
this [pull request](https://github.com/cilium/cilium/pull/10251).

(Source: https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)

The following performance measurements show the difference between both modes.
They also illustrate how both ENI modes are superior to a Flannel-based overlay
network.

(Source: https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)

You don't necessarily have to pick one model over the other though, as it's
perfectly reasonable to schedule pods with high-performance needs with a
dedicated ENI and schedule all other pods with shared ENIs.

##### Tackling the Kubernetes Services and Network Policy Scalability problem with eBPF

So far, the standard way of implementing Kubernetes services has been to use
kube-proxy in iptables mode. Due to the sequential nature of iptables, the
scalability of this solution is severely limited.

(Source: https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)

You can see that, as the number of services increases and crosses a certain
threshold, the latency starts to spike. Even worse, the initial latency to a
particular service will be heavily influenced by a random factor, i.e.,
whether the service entry is close to the beginning or end of the chain of
rules.

Given this, it's not surprising to see Alibaba use the eBPF-based Kubernetes
service implementation of Cilium.

What difference does it make? This is the performance difference as measured by
the Alibaba team. The eBPF implementation gives better scalability and
performance compared to kube-proxy in both iptables and IPVS mode:

(Source: https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)

> After eBPF simplifies the link, the performance is improved significantly by
> 32% compared to when iptables is used and 62% compared to IPVS mode.

Similar to the implementation of Kubernetes services, the implementation of
NetworkPolicy can benefit from using eBPF as well.

(Source: https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)

The box labelled "BPF-agent" represents Cilium running alongside Terway CNI to
provide the eBPF-based Kubernetes Services and Network Policy implementation.

> Note: We use Cilium as the BPF agent on nodes to configure the BPF rules for
> pod ENIs. For more information about Terway-related adaptation, please visit
> [this website](https://github.com/cilium/cilium/pull/10251).

(Source: https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)

Unfortunately, Alibaba does not provide specific numbers on the impact of this
optimization. As a rough guideline, the Cilium team has measured the
performance difference between Cilium running in IPVLAN and veth mode in an
earlier post. You can find the numbers [here](/blog/2019/02/12/cilium-14/#ipvlan-support-beta).

##### Conclusion

We are excited to see Alibaba Cloud contribute to Cilium and welcome them to
the community. If you want to learn more:

- [Cilium Overview](https://cilium.io/)
- [Cilium GitHub](https://github.com/cilium/cilium)
- [How Does Alibaba Cloud Build High-Performance Cloud-Native Pod Networks in Production Environments?](https://www.alibabacloud.com/blog/how-does-alibaba-cloud-build-high-performance-cloud-native-pod-networks-in-production-environments_596590)
- [What is eBPF?](https://ebpf.io/what-is-ebpf)

#### How to perform a CNI Live Migration from Flannel+Calico to Cilium

URL: https://cilium.io/blog/2020/10/06/skybet-cilium-migration
Date: 2020-10-06
Categories: Community

This is a guest blog by Josh Van Leeuwen and covers how Josh implemented a CNI
live migration for a customer, [Sky Betting and Gaming](https://m.skybet.com),
to live migrate a cluster running the Canal CNI plugin (Flannel+Calico) to
Cilium. Josh is working as a Customer Reliability Engineer for Jetstack. Read
on to hear what Josh has to say...

In this post we'll discuss why one might want to change CNIs, what I have learnt
developing a solution for live migration, and how it all works.

##### What is CNI, and why change it?

Container Network Interface (CNI) is a big topic, but in short, CNI is a set of
specifications that define an interface used by container orchestrators to set
up networking between containers. In the [Kubernetes space](https://kubernetes.io/docs/concepts/cluster-administration/networking/),
the Kubelet is responsible for calling the CNI installed on the cluster so Pods
are attached to the Kubernetes cluster network during creation, and its
resources are properly released during deletion. CNIs can also be responsible
for more advanced features than just setting up routes in the cluster, such as
network policy enforcement, encryption, load balancing, etc.

There are many implementations of CNI for various use cases, each having their
own advantages and disadvantages. [Flannel](https://github.com/coreos/flannel)
is one such implementation, backed by iptables, and it is perhaps the most
simple and popular in Kubernetes. Flannel is solely concerned with setting up
routing between Pods in the Kubernetes cluster, which is achieved by creating
an [overlay network](https://en.wikipedia.org/wiki/Overlay_network) using the
[Virtual Extensible LAN
(VXLAN)](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN) protocol. Another
project, [Calico](https://www.projectcalico.org), can be run as either a
standalone CNI solution or on top of Flannel (called
[Canal](https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel))
to provide network policy enforcement.

While using Flannel and Calico together is a solid solution, it can have
problems. For example, Flannel can have
[issues](https://openai.com/blog/scaling-kubernetes-to-2500-nodes/) at
[scale](https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02),
and may not be as feature rich as other implementations.

[Cilium](https://cilium.io/) is another CNI solution, based on
[eBPF](https://ebpf.io/), and is designed to be run at large scale. Whilst
Cilium implements the standard
[NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/),
it is able to utilise the full packet introspection of eBPF, enabling it to have
first class support for [Layer 7
policy](https://docs.cilium.io/en/stable/policy/language/#layer-7-examples) for
a number of protocols, [custom
extensions](https://docs.cilium.io/en/v1.8/concepts/security/proxy/envoy/) using
Envoy, large options for [endpoint
selection](https://docs.cilium.io/en/stable/policy/kubernetes/), as well as rich
[network monitoring
introspection](https://docs.cilium.io/en/v1.8/configuration/metrics/?highlight=monitoring).
For these reasons, Cilium becomes a very favourable choice for running
Kubernetes at scale, with complex network policy requirements.

---

Since the CNI underpins the entire network running on Kubernetes, it would seem
that the only solution for changing a CNI is to take the entire cluster down,
replace it, and bring up all workloads again on the new CNI. This of course
causes downtime, or at the least, requires a full cluster migration. For some
companies this might be unacceptable. So, how about a live migration instead?

##### Designing a live CNI Migration

When a Pod is created, the installed CNI is called which will attach a network
interface to the Pod. If all goes well, this network interface will join that Pod
to the cluster network. The network interface shares the same life cycle as the
Pod, meaning that only newly-created Pods will be picked up by the newer CNI if
it was swapped out. This requires that all Pods on the cluster that are a part
of the cluster network be recycled in order to be a member of the new CNI.

A naive approach to migrating a CNI would be to gradually roll each node on the
cluster, and then replacing the CNI installed when the node is brought back up. This is
however, not possible—the second CNI will be installed with a separate CIDR
range to that of the currently installed CNI, which has no knowledge of the
other network range. Not to mention the use of different encapsulation
protocols, or broken network policy since identity is lost. This can be
demonstrated below where Pods on Node 1 will be unable to communicate to Pods on
Node 2, and visa versa.

_Separate CNIs installed_

With this constraint, it is clear that during the migration, there must be a
period in which all Pods on the cluster are members of both networks. Once all
Pods are members of both networks, the first network can be peeled away, leaving
only the new second CNI network remaining. In practice this means that
routing decisions can be made based upon the destination IP and the
corresponding network interface to use.

The best strategy therefore, is to install the new CNI alongside the current
one and then remove the old CNI during the migration. The overview of the migration
is as follows.

_Step 0: Single CNI installed on the cluster._

_Step 1: Rollout the second CNI alongside the current. All pods communicate over the current._

_Step 2: Both CNIs installed on all nodes, and Pods can communicate on either CNI._

_Step 3: Peel away the first CNI. Pods can communicate on the new CNI if the first is unavailable at the source or destination Pod._

_Step 4: First CNI is completely removed. All commination done over the new CNI._

This strategy maintains all network policy throughout the entire migration, and
ensures no network downtime.

---

The caveat to this strategy is that all workloads on the cluster will need to be
rolled several times. This should be acceptable. Workloads running on Kubernetes are
expected to be resilient to service disruption and rescheduling, and with proper
disruption budgets and probes, a conservative roll of the cluster at each
migration step should ensure that the entire cluster remains healthy at all
times.

##### Implementation

The end result of the project is a [CLI
tool](https://github.com/JoshVanL/cni-migration) that runs the migration from
start to finish, allowing configuration of which steps to run, and ensuring
cluster network health throughout the process. This is all configured using a
config file and flags.

The CLI has support for running in a "dry mode", and will ensure all previous
steps have been completed before continuing. The full migration consists of 6 steps which
I will run through below.

```yaml
# Node labels to use to check the status of each stage
labels:
  canal-cilium: node-role.kubernetes.io/canal-cilium
  cni-priority-canal: node-role.kubernetes.io/priority-canal
  cni-priority-cilium: node-role.kubernetes.io/priority-cilium
  rolled: node-role.kubernetes.io/rolled
  cilium: node-role.kubernetes.io/cilium
  migrated: node-role.kubernetes.io/migrated
  value: 'true' # used as the value to each label key

# File paths of resources for the migration
paths:
  cilium: ./resources/cilium.yaml
  multus: ./resources/multus.yaml
  knet-stress: ./resources/knet-stress.yaml

# Resources required to be deployed before any migration steps.
preflightResources:
  daemonsets:
    knet-stress:
      - knet-stress
      - knet-stress-2
  deployments:
  statefulsets:

# Resources to watch status for to ensure that the cluster is healthy at each
# stage. Must be installed and ready at prepare.
watchedResources:
  daemonsets:
    kube-system:
      - canal
      - cilium
      - cilium-migrated
      - kube-multus-canal
      - kube-multus-cilium
      - kube-controller-manager
      - kube-scheduler
    knet-stress:
      - knet-stress
      - knet-stress-2
  deployments:
  statefulsets:

# Resources to clean up at the end of the migration.
cleanUpResources:
  daemonsets:
    kube-system:
      - canal
      - cilium
      - kube-multus-canal
      - kube-multus-cilium
    knet-stress:
      - knet-stress
      - knet-stress-2
  deployments:
  statefulsets:
```

---

##### Step 0: Preflight

Before, during, and after the migration, we need to check that the cluster has
full network connectively in order to ensure there is no downtime. To do this, I
decided to write a small service called
[knet-stress](https://github.com/joshvanl/knet-stress) that regularly connects
to the other knet-stress services on the cluster. It does this by looking up a
Kubernetes Service, and sending an `HTTP GET` request to every endpoint IP
listed, including its own.

By running this as two DaemonSets under the same Service, we can cover Pod to
Pod communication both between nodes, as well as Pods on the same node. The API
lookup is also a good sanity check that the API server is still rotatable.

During the migration, we exec into each Pod and run a manual `status` check,
with the end result being that we have checked the bidirectional health of the
network, across the entire cluster.

 knet-stress in action

```
DEBU[0016] [kubectl exec --namespace knet-stress knet-stress-tvhjv -- /knet-stress status]  step=0-preflight
time="2020-08-16T13:33:23Z" level=info msg="client: TLS disabled"
time="2020-08-16T13:33:23Z" level=info msg="client: sending request http://172.31.0.3:6443/hello"
time="2020-08-16T13:33:23Z" level=info msg="client: got response status code: 200"
time="2020-08-16T13:33:23Z" level=info msg="client: sending request http://172.31.0.4:6443/hello"
...
time="2020-08-16T13:33:23Z" level=info msg="client: sending request http://172.31.5.7:6443/hello"
time="2020-08-16T13:33:23Z" level=info msg="client: got response status code: 200"
time="2020-08-16T13:33:23Z" level=info msg="client: sending request http://172.31.5.8:6443/hello"
time="2020-08-16T13:33:23Z" level=info msg="client: got response status code: 200"
STATUS OK
```

We run knet-stress before and after every migration step, and is our source of
truth for network connectivity.

##### Step 1: Prepare

This step is for installing all our dependent resources, and labelling nodes.

###### Node Labels

The migration will swap out multiple DaemonSets at various steps, so we use
[Node Labels and Selectors](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)
to control the scheduling. Not only is it better to leave scheduling to
Kubernetes, but it's also a convenient way to observe where we are in the
migration through a simple `kubectl get nodes`. We also make sure that we patch
the first CNI, Canal, to have a node selector that we can use to uninstall it
at a later stage.

We define each label to use in the config, but are fine to leave as is.

```yaml
# Node labels to use to check the status of each stage
labels:
  canal-cilium: node-role.kubernetes.io/canal-cilium
  cni-priority-canal: node-role.kubernetes.io/priority-canal
  cni-priority-cilium: node-role.kubernetes.io/priority-cilium
  rolled: node-role.kubernetes.io/rolled
  cilium: node-role.kubernetes.io/cilium
  migrated: node-role.kubernetes.io/migrated
  value: 'true' # used as the value to each label key
```

When a node's label is changed, Kubernetes will unschedule any of the
DaemonSets which don't match the selector, and schedule those that do. We can
then simply wait for the underlying Pods to become ready.

---

##### Step 2: Roll all Nodes

###### Multus

The first challenge faced as part of the migration is to run two CNIs in
parallel, both servicing the same Pods. Luckily there is a project by Intel,
[multus-cni](https://github.com/intel/multus-cni), which does exactly that.
Multus is installed like most CNIs, as a DaemonSet. It acts as a middleman
whereby calls made by the Kubelet are forwarded to the two configured CNIs
underneath, each setting up a separate network interface on the Pod. In
practice, if multiple CNIs are installed and configured, each newly
created Pod will have a 'master' network interface, as well as a secondary
network interfaces, one for each of the extra CNIs (and of course the loopback
device). Only the master network interface is advertised to Kubernetes and
used in Services, etc. Although the secondary Pod IP is not directly advertised
to Kubernetes, it is still rotatable by other containers on that secondary
network. This will become important later.

Multus is easy to setup, all that is needed are a few lines of config. for each
CNI we want to install. It is also possible to configure Multus using a
[CRD](https://k8snetworkplumbingwg.github.io/multus-cni/docs/how-to-use.html?),
though a blanket cluster-wide configuration is more useful in our case.

 Base Multus config

```json
{
  "name": "multusi-cni-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "cniVersion": "0.3.1",
      "name": "multus-cni-network",
      "type": "multus",
      "kubeconfig": "/etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig",
      "confDir": "/etc/kubernetes/cni/net.d",
      "clusterNetwork": "k8s-pod-network",
      "defaultNetworks": ["cilium"],
      "systemNamespaces": [""]
    }
  ]
}
```

In this config, we have defined `k8s-pod-network` to be the master CNI network
(the name of the CNI config for Canal) and it is the network that is advertised
to Kubernetes. Cilium is the second CNI to be called. Finally, this config
should apply to all namespaces (`"systemNamespaces": [""]`).

###### CNI Ordering

When CNIs are installed on a cluster, they are naturally run as a DaemonSet to
set up networking on every node. With this, they also typically write out their
executable to file, as well as their configuration to an established directory
(typically somewhere like `/etc/kubernetes/cni/net.d/`). The Kubelet looks at
this directory to find the executable to run and its parameters.

Every time a new Pod is created, the Kubelet will list the files in the
directory, and use the configuration which is alphabetically first in the
list. For example, `00-multus.conflist` is the CNI configuration that the
Kubelet will use in the following example:

 CNI config directory with Multus priority

```shell-session
$ ls -la /etc/kubernetes/cni/net.d/
total 56
drwxr-xr-x. 3 root root 4096 Aug 16 13:34 .
drwxr-xr-x. 3 root root 4096 Aug 16 13:29 ..
-rw-r--r--. 1 root root  415 Aug 16 13:34 00-multus.conflist
-rw-r--r--. 1 root root 1646 Aug 16 13:34 10-calico.conflist
-rw-r--r--. 1 root root   97 Aug 16 13:33 99-cilium.conf
-rw-------. 1 root root 1234 Aug 16 13:33 calico-kubeconfig
drwxr-xr-x. 2 root root 4096 Aug 16 13:34 multus.d
```

This strategy means that we can configure which CNI takes precedence by
controlling the file locations that the CNI configuration is written to. Both
Multus, and us during the migration, take advantage of this.

###### SBR Plugin

Cilium manages [identity of endpoints](https://docs.cilium.io/en/v1.8/concepts/terminology/#what-is-an-identity)
in order to make routing and policy decisions. One issue that we ran into was
Cilium losing the source identity of requests when the request was being sent
as a non default, secondary network interface to the Pod. Cilium would interpret
the source of that request as an identity of
[`reserved:world`](https://docs.cilium.io/en/v1.8/concepts/terminology/#special-identities),
which would then get dropped, never reaching the Pod. Although I didn't track
down what the source of the issue was, I was able to mitigate it by using the
[SBR (Source Based Routing) CNI meta plugin](https://github.com/containernetworking/plugins/tree/master/plugins/meta/sbr).

The SBR CNI meta plugin will cause the default route of the Pod to be
overridden to the Cilium network interface. This fixed the issue, and
Cilium was able to correctly determine source endpoint identity.

 knet-stress Pod without SBR meta plugin

```shell-session
$ kubectl exec knet-stress-2-b99nz -- ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
172.29.100.122 dev net1 scope link
```

 knet-stress Pod with SBR meta plugin

```shell-session
$ kubectl exec knet-stress-2-br2t5 -- ip route
default via 172.29.197.231 dev net1
172.29.197.231 dev net1 scope link
```

###### Encapsulation Mode

Flannel and Cilium both support and default to VXLAN as the tunneling protocol
to facilitate routing cluster traffic. VXLAN has support for running multiple
networks on the same machines, separated by the VXLAN ID (VID); however, I was
not able to find an option for configuring this in either Cilium or Flannel.
This caused a conflict and prevented Cilium to start at all.

To fix this, I decided to configure Cilium to use [GENEVE (Generic Network
Virtualisation Encapsulation)](https://www.redhat.com/en/blog/what-geneve)
instead. GENEVE is a newer encapsulation protocol designed to supersede VXLAN.
This seemed like a fine compromise and I didn't encounter any problem going
this route.

###### All Change

Now we have Multus installed, and have given it the precedence as the primary
CNI, we now require that all Pods which are in the cluster network to be rolled.
As described earlier, this is a required step since we need all of the Pods to
be recreated to have both CNI network interfaces attached.

 knet-stress Pods with Canal advertised IP

```
NAME                  READY   STATUS    RESTARTS   AGE     IP              NODE                                        NOMINATED NODE   READINESS GATES
knet-stress-2-bstdd   1/1     Running   0          5m19s   172.31.0.6      ip-10-99-0-156.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-csf7g   1/1     Running   0          6m52s   172.31.1.6      ip-10-99-3-105.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-kdl2r   1/1     Running   0          36s     172.31.5.2      ip-10-99-0-179.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-r5gds   1/1     Running   0          6m15s   172.31.4.8      ip-10-99-1-77.eu-west-1.compute.internal    <none>           <none>
knet-stress-2-rlk5f   1/1     Running   0          5m19s   172.31.3.7      ip-10-99-2-248.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-rp96s   1/1     Running   0          7m27s   172.31.2.6      ip-10-99-1-90.eu-west-1.compute.internal    <none>           <none>
knet-stress-5tz28     1/1     Running   0          5m19s   172.31.3.9      ip-10-99-0-179.eu-west-1.compute.internal   <none>           <none>
knet-stress-dcjzz     1/1     Running   0          6m49s   172.31.1.9      ip-10-99-1-77.eu-west-1.compute.internal    <none>           <none>
knet-stress-gc7m8     1/1     Running   0          6m14s   172.31.4.10     ip-10-99-1-90.eu-west-1.compute.internal    <none>           <none>
knet-stress-hbct8     1/1     Running   0          7m16s   172.31.2.8      ip-10-99-0-156.eu-west-1.compute.internal   <none>           <none>
knet-stress-llp82     1/1     Running   0          36s     172.31.5.4      ip-10-99-3-105.eu-west-1.compute.internal   <none>           <none>
knet-stress-mc9m7     1/1     Running   0          5m18s   172.31.0.8      ip-10-99-2-248.eu-west-1.compute.internal   <none>           <none>
```

---

##### Step 3: Change CNI Priority

This stage involves changing the CNI priority for all nodes so that Cilium
becomes the primary CNI. To achieve this, we install a new Multus DaemonSet
with Cilium as the primary CNI, and Canal as the secondary.

 Multus config with Cilium as primary CNI

```json
{
  "name": "multusi-cni-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "cniVersion": "0.3.1",
      "name": "multus-cni-network",
      "type": "multus",
      "kubeconfig": "/etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig",
      "confDir": "/etc/kubernetes/cni/net.d",
      "clusterNetwork": "cilium",
      "defaultNetworks": ["canal"],
      "systemNamespaces": [""]
    }
  ]
}
```

To do this, we taint and drain the node, relabel it so that a new mutlus Pod is
scheduled using the updated configuration, and then untaint the node. This
causes all Pods to be rescheduled on that node which are now all members of both
CNIs, but their advertised IP is that of Cilium's network. We repeat this
process for all nodes in the cluster, ensuring that there is total network
connectivity throughout.

 Pods running with Cilium network advertised

```
NAME                  READY   STATUS    RESTARTS   AGE   IP               NODE                                        NOMINATED NODE   READINESS GATES
knet-stress-2-28jz8   1/1     Running   0          19m   172.29.23.7      ip-10-99-0-156.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-5vt2w   1/1     Running   0          13m   172.29.82.217    ip-10-99-3-105.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-fcwqt   1/1     Running   0          17m   172.29.163.144   ip-10-99-0-179.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-jfr2r   1/1     Running   0          17m   172.29.157.128   ip-10-99-1-77.eu-west-1.compute.internal    <none>           <none>
knet-stress-2-l9kzs   1/1     Running   0          14m   172.29.57.215    ip-10-99-2-248.eu-west-1.compute.internal   <none>           <none>
knet-stress-2-pzggb   1/1     Running   0          15m   172.29.60.178    ip-10-99-1-90.eu-west-1.compute.internal    <none>           <none>
knet-stress-bmftj     1/1     Running   0          17m   172.29.225.70    ip-10-99-0-179.eu-west-1.compute.internal   <none>           <none>
knet-stress-c5sgd     1/1     Running   0          17m   172.29.110.42    ip-10-99-1-77.eu-west-1.compute.internal    <none>           <none>
knet-stress-g5qbv     1/1     Running   0          15m   172.29.93.51     ip-10-99-1-90.eu-west-1.compute.internal    <none>           <none>
knet-stress-k5vz9     1/1     Running   0          19m   172.29.56.36     ip-10-99-0-156.eu-west-1.compute.internal   <none>           <none>
knet-stress-smghl     1/1     Running   0          13m   172.29.127.250   ip-10-99-3-105.eu-west-1.compute.internal   <none>           <none>
knet-stress-w4ksd     1/1     Running   0          14m   172.29.188.52    ip-10-99-2-248.eu-west-1.compute.internal   <none>           <none>
```

The end result of this stage is that we have both CNIs installed for all nodes,
and each Pod has two network interfaces attached, with the primary and
advertised Pod IP address in Cilium's network.

 Node labels with Cilium as priority

```
NAME                                        STATUS   ROLES                                        AGE   VERSION
ip-10-99-0-156.eu-west-1.compute.internal   Ready    canal-cilium,master,priority-cilium,rolled   79m   v1.17.3
ip-10-99-0-179.eu-west-1.compute.internal   Ready    canal-cilium,master,priority-cilium,rolled   79m   v1.17.3
ip-10-99-1-77.eu-west-1.compute.internal    Ready    canal-cilium,priority-cilium,rolled,worker   79m   v1.17.3
ip-10-99-1-90.eu-west-1.compute.internal    Ready    canal-cilium,priority-cilium,rolled,worker   79m   v1.17.3
ip-10-99-2-248.eu-west-1.compute.internal   Ready    canal-cilium,priority-cilium,rolled,worker   79m   v1.17.3
ip-10-99-3-105.eu-west-1.compute.internal   Ready    canal-cilium,master,priority-cilium,rolled   79m   v1.17.3
```

---

##### Step 4: Migration

With all Pods now using Cilium as their primary CNI, Canal is not being anymore
and so can be safely uninstalled everywhere. We repeat a similar process as
before; tainting and draining a node, relabelling it so that the old Canal and
Cilium CNIs are uninstalled, installing a duplicated Cilium CNI that selects
that node label, and bringing the node back up.

Repeating this process, we finally get all Pods to use Cilium.

 CNI config directory with Cilium as priority

```shell-session
ip-10-99-0-179 net.d # ls -al
total 72
drwxr-xr-x. 3 root root 4096 Aug 16 14:51 .
drwxr-xr-x. 3 root root 4096 Aug 16 13:29 ..
-rw-r--r--. 1 root root   97 Aug 16 14:51 00-cilium.conf
-rw-r--r--. 1 root root  404 Aug 16 14:45 00-multus.conflist
-rw-r--r--. 1 root root 1655 Aug 16 14:32 10-calico.conflist
-rw-r--r--. 1 root root   97 Aug 16 13:33 99-cilium.conf
-rw-r--r--. 1 root root  300 Aug 16 14:45 99-flannel.conflist
-rw-------. 1 root root 1234 Aug 16 13:33 calico-kubeconfig
drwxr-xr-x. 2 root root 4096 Aug 16 13:34 multus.d
```

 Migrated Node labels

```
NAME                                        STATUS   ROLES                           AGE   VERSION
ip-10-99-0-156.eu-west-1.compute.internal   Ready    cilium,master,migrated,rolled   99m   v1.17.3
ip-10-99-0-179.eu-west-1.compute.internal   Ready    cilium,master,migrated,rolled   99m   v1.17.3
ip-10-99-1-77.eu-west-1.compute.internal    Ready    cilium,migrated,rolled,worker   99m   v1.17.3
ip-10-99-1-90.eu-west-1.compute.internal    Ready    cilium,migrated,rolled,worker   99m   v1.17.3
ip-10-99-2-248.eu-west-1.compute.internal   Ready    cilium,migrated,rolled,worker   99m   v1.17.3
ip-10-99-3-105.eu-west-1.compute.internal   Ready    cilium,master,migrated,rolled   99m   v1.17.3
```

---

##### Step 5: Clean Up

This step simply removes all resources no longer needed: the old Canal
installation, unscheduled Multus, and Cilium DaemonSets. It also patches the
installed Cilium DaemonSet to remove the `migrated` node label selector.

If all goes according to the plan, we have now successfully migrated the CNI
installation from Canal to Cilium, live.

---

##### Further Reading

- [Original Jetstack blog post](https://www.blog.jetstack.io/blog/cni-migration/)
- [Cilium Overview](https://cilium.io/)
- [Cilium GitHub](https://github.com/cilium/cilium)
- [What is eBPF?](https://ebpf.io/)

#### How Wildlife Studios built a Global Multi Cluster Gaming Infrastructure with Cilium

URL: https://cilium.io/blog/2020/09/03/wildlife-studios-multi-cluster-gaming-platform
Date: 2020-09-03
Categories: Community

Wildlife Studios is a Brazil-based global gaming company, one of the twenty
largest mobile gaming companies in the world, and has more than 700 employees
and offices in Brazil, Argentina, Ireland, and the United States. Their games
are distributed in over 150 countries, and have been dowloaded over
two billion times. Their online gaming infrastructure relies on several
services deployed all over the globe, which are delivered from around 30,000
Kubernetes pods running on 2,500 nodes.

In mid-2018 we started the preparations for two big game launches planned
for the middle of the next year. One of the most important thing was to rethink our
networking infrastructure for Kubernetes in order to support better integration
between different regions where we run game servers.

At that moment, we had some clusters running with Calico 2.6, exposing their services
through internal load balancers and connecting important parts of our internal game
architecture using NATS. Two systems represent the keys to this infrastructure castle:

**The first system** - Each game consists of several components built upon a lightweight
game server framework called "[Pitaya](https://medium.com/tech-at-wildlife-studios/pitaya-wildlifes-golang-go-af57865f7a11)"
with clustering support. These components use a centralized Etcd cluster for
auto-discovery, in this way they can communicate with each other using two
different mechanisms, through NATS topics, or direct gRPC requests.

**The second system** - a game room scheduler called [Maestro](https://github.com/topfreegames/maestro),
which is responsible for scaling different stacks of game rooms on demand, making sure that there
will always be game rooms available for new matches. In order to reduce latency
and improve the user experience, these stacks are deployed in several regions
around the world. This also created a fault-tolerant system, where it's
possible to turn off game rooms in specific regions and temporarily redirect users to
another point of the globe.

##### Problems to solve

In this architecture, for each shared service, like Etcd, NATS, and Jaeger, we
had to create and maintain internal load balancers and their respective DNS
records. It was also hard to configure routing for pod-to-pod communication in
order to enable Pitaya’s gRPC integration, a more efficient operation mode.
Last but not least, we needed a tool to improve the visibility of the network
without adding tons of specialized components to our clusters.

The main question was: How do we create and monitor a highly available global
environment for critical game components that need to reach each other and also
communicate with some centralized shared services?

To answer this question, the first big challenge would be to find a Kubernetes
CNI plugin that supported the following requirements:

- Low-latency pod-to-pod communication across different clusters
- Global service network without dramatically increasing system complexity
 adding a bunch of tools, controllers, and resources
- Reasonable use of resources due to the criticality of adding a new daemonset
 to the system
- Reliable scaling up to a 600 node single cluster
- Boot quickly enough to withstand a super volatile game-room environment
- Simple configuration in order to reduce the operational cost
- A good set of metrics that could be exported to Prometheus and Datadog
- Good documentation and community support

As the existing clusters were deployed using kops, three different CNI
supported plugins appeared as good candidates to meet those requirements:
Calico 2.6, Amazon VPC 1.2, and Cilium 1.2.

##### Tests and Benchmarks

At the end of 2018, we performed several load and networking tests against the available options.
Even though the results are quite large and detailed and would need a dedicated post, some
results bear illustration as they point to the reasons for selecting Cilium.

###### Setup complexity

Calico 2.6 didn’t have a standard way to use a reliable HA route
reflector configuration for exchanging routes. It meant that we needed to use its
normal BGP full-node-mesh mode, but in this way, it would be virtually
impossible to deploy a cluster with more than 300 nodes. Even though this was a
breaking point, we decided to configure an external
[BIRD](https://bird.network.cz/) according to [this post from IBM](https://medium.com/ibm-cloud/journey-to-1000-nodes-for-ibm-cloud-private-5294138047d5),
building a full-mesh of route reflectors.

This configuration significantly increased the complexity of the system and the
respective operational cost. To make things worse, these drawbacks would be ok if we were
trying to build a static environment, but we weren't. Since we needed node autoscaling,
we had to develop a hacky controller for managing inclusions and
removals of dynamic nodes as BGP peers in BIRD’s static configuration files.
Definitely, Calico couldn’t give us what we needed.

Amazon VPC was much simpler, as pods used secondary IPs from nodes in the same
network. We only needed to create VPC peerings and the respective routing
tables between the clusters. The problem was that we would still have to
configure internal load balancers and deal with their name resolution.

Although Cilium required a little bit more effort than Amazon VPC because of
the dedicated Etcd server which was necessary for large clusters at that point,
we found what we were looking for in terms of multi-cluster management. The
Cilium Cluster Mesh configuration was absolutely simple. All we needed to do
besides setting up the basic VPC peering, routing tables, and firewalling was
to tell Cilium where to find the dedicated Etcd endpoints from the other
clusters.

```yaml
# cluster mesh config snippet
data:
  cluster-A: |-
    endpoints:
    - "http://etcd-0.cluster-A.example.internal:2379"
    - "http://etcd-1.cluster-A.example.internal:2379"
    - "http://etcd-2.cluster-A.example.internal:2379"
  cluster-B: |-
    endpoints:
    - "http://etcd-0.cluster-B.example.internal:2379"
    - "http://etcd-1.cluster-B.example.internal:2379"
    - "http://etcd-2.cluster-B.example.internal:2379"
```

Nowadays, Cilium also supports CRD-backed IPAM mode instead of using a dedicated Etcd.
This works for large clusters and cluster meshes configurations as well, managing IP addresses
via Kubernetes Custom Resource Definition (CRD).

Cilium also comes with a feature called [global services](https://cilium.io/blog/2019/03/12/clustermesh/) that solved the
internal load balancer problem in a very simple way. We only had to add an
annotation to those services and deploy them to all the clusters.

##### Networking benchmark

The following charts are the result of one of the networking tests comparing
Calico 2.6, Amazon VPC 1.2, and Cilium 1.2. In a very similar environment, an orchestrator
system coordinated workers to test different MTUs settings (configuring MSS to
TCP and setting direct package size to UDP).

Scenarios evaluated:

- Local VM using pod IP: Local VM pod-to-pod traffic;
- Local VM using cluster/virtual IP: Local VM pod-to-pod traffic tests using its
 Service IP;
- Remote VM using pod IP: traffic tests for pods in different nodes using pod
 IP;
- Remote VM using cluster/virtual IP: traffic tests using cluster/virtual IP
 for pods in different nodes;
- Local VM pod Hairpin: pod itself using Cluster IP.

These data were obtained from the project [Kubernetes Netperf](https://github.com/kubernetes/perf-tests/tree/master/network/benchmarks/netperf), and we used the following configuration for the testing environment running in AWS:

- Kops version: v1.11.0

- Kubernetes version: v1.10.7

- Control Plane
 - instances: 3
 - zones: us-east-1b, us-east-1c, us-east-1d
 - instance type: m4.xlarge

- Node
 - instances: 200
 - zones: us-east-1b, us-east-1c, us-east-1d
 - instance type: c4.2xlarge

We didn’t enable encryption, nor any other specific configuration for
each CNI plugin. As you can see, compared to the others, Cilium performed very
well, especially for UDP traffic, which is important when we are talking about
games. Another point in favor of Cilium was the results of TCP throughput for
different MSS test, which was impressive for tests within the same VM.

_Note: The results obtained are consistent with previous versions of these tools, and
although they were extremely relevant to the decision we made, they may not match
current tests under the same conditions, as these tools are constantly evolving._

##### The migration process

After these and some other positive results, like the stability running hundreds
of nodes, and the small amount of requests to the Kubernetes API, in November 2018
Wildlife's SRE team decided to rollout their whole self-managed Kubernetes
infrastructure in order to use Cilium. The migration process was also possible
thanks to the great support of its community. Since pod-to-pod communication was
working well, relying on the Cilium Cluster Mesh feature, the gRPC mechanism on
Pitaya was enabled, improving the servers' performance and removing one of
those single points of failure - NATS.

In a second migration round, Wildlife substituted most of the internal load
balancers with Cilium Global Services. This process included the migration of
those load balancers attached to Pitaya's Etcds, and to internal logging and tracing
stacks. This removed the complexity of maintaining a lot of cloud-scoped load
balancers and their respective private DNS records.

In mid-2019, Wildlife's SRE team started developing a tool called Bigbang that
supports the creation of production-ready Cluster Meshes in a short time.
Bigbang was built on top of Kops, Terraform, Helm3, and Cilium, making it
possible to quickly deploy a full-integrated multi-cluster
infrastructure for game servers. This tool supported the launch of the
company's last two major successes, Zoo Battle Arena and Tennis Clash, which were top #10 in over
150 countries.

Today, Bigbang is shipping Cilium in more than 20 Kubernetes production
clusters in the company. Each game has at least 3 clusters running together in
the same Cluster Mesh configuration. This infrastructure handles more than 50k
client requests per second and supports millions of daily active users.

##### Comments and the future

Since 2018 we've been proactively working with the Cilium team to improve our systems. This
close interaction supported some investigations and allowed us to efficiently solve problems
together, consequently increasing the reliability of our network.

After this long period migrating and building tools around Cilium, we're looking forward to
improving the security of our environment with the powers of Network Policies. This is a step
towards the Zero Trust Network strategic initiative, which Cilium facilitates.

Another promising step is related to [Hubble](https://github.com/cilium/hubble), a distributed
networking and security observability platform built on top of Cilium for cloud native workloads.
After testing this in a few clusters, we're very excited to rollout the tool for all the production
environments in order to obtain detailed insights about our network with minimum overhead thanks
to the use of the Linux kernel technology [eBPF](https://ebpf.io/).

#### Google announces Cilium & eBPF as the new networking dataplane for GKE

URL: https://cilium.io/blog/2020/08/19/google-chooses-cilium-for-gke-networking
Date: 2020-08-19
Categories: Community

Today marks an exciting day for the Cilium community and all Cilium
contributors, Google just announced that Cilium has been selected and made
available as the new datapath for GKE and Anthos:

> Today, we’re introducing GKE Dataplane V2, an opinionated dataplane that
> harnesses the power of eBPF and Cilium, an open source project that makes the
> Linux kernel Kubernetes-aware using eBPF.

You can read all the details in the official
[announcement](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine).
In this post, we will take a look behind the scenes that lead up to this.

##### How it all started

As it often does, large adventures start with a small innocent
Git [commit](https://github.com/cilium/cilium/commit/ba85129ea4558ae03eca07f07e88bf169f2c9307).
Late last year, the following commit from a Google engineer popped up out of the
blue and was merged shortly after:

```
commit ba85129ea4558ae03eca07f07e88bf169f2c9307
Author: Valas Valancius <valas@google.com>
Date:   Sun Dec 15 14:13:21 2019 -0800

    test: Add conntrack entry timeout validation tests.

    Partially fixes #9303.

    Signed-off-by: Valas Valancius <valas@google.com>
```

Pretty innocent, right? Shortly after, more and more Google engineers started
contributing, including major new features such as [Aggregated policy verdict
visibility](https://github.com/cilium/cilium/pull/9943). A feature that
provides visibility into the network policy decision making process at
extremely low overhead while still providing all the required context and
details. This later became the foundation for GKE's [Kubernetes Network Policy
Logging](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy-logging) feature.

```
commit e831859b5cc336c6d964a6d35bbd34d1840e21b9
Author: Zang Li <zangli@google.com>
Date:   Wed Jan 22 23:16:36 2020 +0000

    Add a new event type for policy action logging

    Signed-off-by: Zang Li <zangli@google.com>
```

Collaboration and bug fixing efforts between the teams intensified and resulted
in several new features being contributed such as auto-detection of
EndpointSlices support, ability to join IPv6 NDP multicast groups and support
for IPv6 neighbor discovery for pod IPs, Geneve encapsulation bugfixes, and
optimizations to the socket cookie-based load-balancing.

At the same time, a major effort was made to test the conformance, stability,
and reliability of Cilium on GKE and Anthos. All of this lead to today’s
announcement and the availability of Cilium as the new eBPF networking data
plane for GKE.

##### Why Cilium on GKE?

What is behind this decision? In today’s announcement, Google provides some
context on this decision:

> As more and more enterprises adopt Kubernetes, the gamut of use cases is
> widening with new requirements around multi-cloud, security, visibility and
> scalability. In addition, new technologies such as service mesh and
> serverless demand more customization from the underlying Kubernetes layer.
> These new requirements all have something in common: they need a more
> programmable dataplane that can perform Kubernetes-aware packet manipulations
> without sacrificing performance.
>
> Enter Extended Berkeley Packet Filter (eBPF), a new Linux networking paradigm
> that exposes programmable hooks to the network stack inside the Linux kernel.
> The ability to enrich the kernel with user-space information—without jumping
> back and forth between user and kernel spaces—enables context-aware
> operations on network packets at high speeds.

Google clearly has incredible technical chops and could have just built their
dataplane directly on eBPF, instead, the GKE team has decided to leverage
Cilium and contribute back. This is of course a huge honor for everybody who
has contributed to Cilium over the years and shows Google's commitment to open
collaboration.

> The Cilium community has put in a tremendous amount of effort to bootstrap
> the Cilium project, which is the most mature eBPF implementation for
> Kubernetes out there. We at Google actively contribute to the Cilium project,
> so that the entire Kubernetes community can leverage the advances we are
> making with eBPF.

And there is more, Google is not planning to stop there, there will be more
exciting things being built with eBPF and Cilium:

> eBPF’s ability to augment network packets with custom metadata enables a long
> list of possible use cases. We are as excited about the future of Kubernetes
> and eBPF as you are, so stay tuned for more innovations.

##### What is eBPF?

How does this magic eBPF technology work exactly? We can't go into all of the
details of eBPF in this post but the following is extracted from
[ebpf.io](https://ebpf.io) which is a good source to learn everything about
eBPF:

> eBPF is a revolutionary technology that can run sandboxed programs in the
> Linux kernel without changing kernel source code or loading kernel modules.
> By making the Linux kernel programmable, infrastructure software can leverage
> existing layers, making them more intelligent and feature-rich without
> continuing to add additional layers of complexity to the system.

##### What is Cilium?

For those hearing about Cilium for the first time:
Cilium is an open source project that has been designed on top of eBPF to
address the new scalability, security, and visibility requirements of
container workloads. Cilium goes beyond a traditional [Container Networking
Interface
(CNI)](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#cni)
to provide service resolution, network policy enforcement, extensive visibility
and much more.

Overview of what Cilium brings to Kubernetes using eBPF:

- **Networking**
 - eBPF-based in-kernel networking datapath featuring IPv4 and IPv6 with the
 ability to support both direct-routing and encapsulation/overlay
 topologies. Highly scalable kube-proxy replacement with direct server
 return (DSR), session affinity, XDP-based acceleration for services of type
 LoadBalancer, NodePort and services with externalIPs, as well as
 socket-level load-balancing for all service types. Multi-cluster
 routing capabilities with service load-balancing across clusters.

- **Security**
 - Identity-based network policy implementation that decouples security
 enforcement from IP addresses for increased scalability and compatibility
 with service mesh identity concepts. FQDN/DNS-based policy support to
 secure access to external services. Network policy logging to improve the
 troubleshooting experience. Ability to transparently inject Envoy for
 L7-aware policies and policy-driven SSL termination.

- **Observability**
 - Programmable flow logging capabilities for visibility into security verdicts,
 networking forwarding decisions, and to understand service topologies and
 connectivity patterns. Programmable metrics system to monitor traffic
 patterns, security status, and error conditions.

##### Try it Out

You can get started running Cilium as the GKE dataplane using the GKE rapid channel:

```bash
gcloud beta container clusters create <cluster name> \
    --enable-dataplane-v2 --release-channel rapid \
    --cluster-version 1.17.9-gke.600 --zone <zone name>
```

If you don't want to use the rapid channel, the existing [GKE installation
instructions](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-gke/)
will of course still work.

##### What is next?

Working together with Google's GKE team has been incredibly exciting and
rewarding. Having Google as a main contributor to the project will yield many
exciting new features that will benefit the entire community. We couldn't be
more excited about the future ahead of Cilium and eBPF.

If you want to learn more about Cilium and eBPF, check out:

- [Cilium Overview](https://cilium.io/)
- [Cilium GitHub](https://github.com/cilium/cilium)
- [What is eBPF?](https://ebpf.io/)
- [GKE Dataplane V2 announcement](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebfp-and-cilium-to-google-kubernetes-engine)

#### Multitenancy and Network Security in Kubernetes with Cilium

URL: https://cilium.io/blog/2020/07/27/2020-07-27-multitenancy-network-security
Date: 2020-07-27
Categories: How-To

_Contributed by Jed Salazar and Joe Stringer, Isovalent_

Multitenancy is a common pattern in Kubernetes. Many organizations deploy Kubernetes-as-a-Service, where one cluster houses many tenants and workloads. This pattern might sound familiar, as cloud computing services like AWS, Azure, and GCP have enabled multiple customers (tenants) to run their business-critical workloads in a single cluster for years.

The benefits of this pattern are well-understood: Infrastructure efficiency is increased as CPU cores and memory have become a commodity. SRE\/ops teams aren’t burdened with linear management of _N_ clusters for _N_ tenants, and generalized hardware can be utilized for virtually any workload.

However, with these benefits comes trade-offs: A model of shared compute implies shared infrastructure. In an environment with untrusted workloads, isolation has become a [serious security challenge](https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/). In Kubernetes there are many shared resources that deserve security attention, but we’re going to focus on networking and discuss the unique network security challenges Kubernetes faces as a multitenancy provider.

##### Network Perimeters

To gain an understanding of the challenges of network security in Kubernetes, it’s worthwhile understanding the history of network security. The defining principle of network security in the pre-cloud-computing age was the _network perimeter security_ model. In short, this model assumed threats were _external_ to the network, meaning that attacks came from the Internet or an untrusted external network. Workloads internal to the network were trusted and the majority of an organization's security focus was deploying hardened perimeter firewalls that blocked access to internal (trusted) resources from external (untrusted) sources.

This model breaks down in a Kubernetes environment. Multiple tenants run in a single cluster, all using the same network infrastructure. Kubernetes [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/) means virtually any type of workload can be deployed to a worker Node based on available system resources, which means multiple tenants can run on the same worker Node. In short, threats can come from inside the network perimeter or even the same Node where protections are scant and visibility is limited. Because the network perimeter model is no longer suitable in the age of cloud, an emerging pattern of network security called [Zero-Trust Networking](https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/) is gaining ground. Zero-Trust means internal and external resources aren’t implicitly trusted based on IP address for access to resources, even in the same network or worker Node.

##### Multitenancy and Network Security in Kubernetes

Cilium is one of the least complex ways to achieve secure multitenancy. Cilium utilizes cloud-native-aware network identity and benefits from eBPF which incurs virtually no performance impacts to secure network connectivity. Why does this matter?

Consider our [previous blog post](https://cilium.io/blog/2020/06/29/cilium-kubernetes-cni-vulnerability), which shows the trusted network model having a history of impacting the security of Kubernetes. Another [vulnerability, (CVE-2020-8558)](https://groups.google.com/g/kubernetes-security-announce/c/B1VegbBDMTE?pli=1) was discovered in `kube-proxy` where processes bound to localhost (`127.0.0.1`) can be reachable by Nodes or Pods in the same local network. This is dangerous because certain deployments, such as the `kube-apiserver` don't require authentication if accessed via `127.0.0.1:8080` because it's assumed localhost is only reachable by other processes in the same Node.

Note: It's strongly suggested to disable this behavior by adding the `--insecure-port=0` flag to your kubernetes API server command line arguments.

The linked vulnerability works because of the way `iptables` routes to a [NodePort](https://v1-14.docs.kubernetes.io/docs/concepts/services-networking/service/#nodeport) address and the fact Kubernetes Pods run with `CAP_NET_RAW` privileges by default, allowing Pods to specify any source IP address.

A simplified explanation is a Pod with `CAP_NET_RAW` sends a packet with a localhost source IP, destined to a localhost NodePort address (`127.0.0.1:xxxxx -> 127.0.0.1:8888`) which performs _Destination NAT_ (DNAT) to a NodePort destination IP (`127.0.0.1:xxxxx -> 10.0.0.10:9999`). Because `route_localnet` is enabled, the packet is routed and `POSTROUTING` [masquerades](https://tldp.org/HOWTO/IP-Masquerade-HOWTO/ipmasq-background2.1.html) the source IP to the IP of the Node (`10.0.0.5:xxxxx -> 10.0.0.10:9999`). With the source and destination IP configured as the respective Node IPs, this packet is now fully valid and routable. Thanks to [squeed](https://github.com/squeed) for the details in the `iptables` [routing implementation](https://github.com/kubernetes/kubernetes/issues/90259).

Binding a process to localhost has traditionally been assumed safe under the trusted network model [“because it assumed that only other localhost processes could reach it”](https://groups.google.com/g/kubernetes-security-announce/c/B1VegbBDMTE?pli=1). Because untrusted workloads can run in the same network or on the same Node, the case for Zero-Trust becomes more clear.

##### eBPF and Secure Multitenant Networks

We'll discuss a couple of concrete examples of how Cilium and eBPF implement aspects of Zero-Trust network principles to enable multitenancy in Kubernetes.

IP addresses aren't suitable for identities in Kubernetes because Pod Replicas can have _N_ number of IP addresses for a single workload. Additionally, correlating a Pod to an IP address is difficult in a highly dynamic environment like Kubernetes, as Pods can be quickly destroyed and re-scheduled, each time with a different IP address. So if you have a Pod with labels `frontend`, and you delete the Pod and quickly spin up a new Pod, Kubernetes is free to reassign the `frontend` IP to a new Pod that has the label `agent_of_chaos`.

To update the security policy in a traditional IP-based model, Nodes have to learn about the Pod shut down and then update the policy rules to remove the IP address from IP allow lists. In the meantime, the `agent_of_chaos` Pod can initialize using the same IP and send traffic to other Pods with the `frontend` IP address, and IP-based policy may allow it.

Cilium identifies a network entity with an [identity](https://docs.cilium.io/en/v1.8/concepts/terminology/#identity) instead of an IP address. Identities are derived from Kubernetes [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) and other metadata which allows security logic such as, "Only allow `frontend` Pods access to `backend` Pods". This allows for identity-aware security decisions such as Network Policy filtering.

When Cilium initializes the network for a new Pod, it allocates the identity and ties the identity directly to the Pod by applying security policy at the Pod's network device via eBPF. For traffic exiting the Pod, the identity is attached directly to all traffic, so it doesn't matter that it reuses the same IP address, we have a stronger notion of identity to back the security policy.

Another example of Zero-Trust in Cilium is based on the _Default Deny_ principle. We discussed how Cilium disables IPv6 traffic unless explicitly enabled in our [previous post](https://cilium.io/blog/2020/06/29/cilium-kubernetes-cni-vulnerability) but it's worth discussing the motivations that led to that decision.

Given what we know about cloud-native architecture, advances in the latest network protocols, and with decades of networking experience, we wanted to design a performant and secure network. This naturally led us towards eBPF, which provides the level of flexibility and power needed to implement the right approach. Conceptually this means that every network interaction was designed to provide guarantees about how we're handling packets. Because we build Cilium to be high-performance we could compile in a minimum amount of code. If we're not using a function, we should compile it out. By implementing only the set of the features that you need, we minimize the code footprint in Cilium which increases performance and also reduces the attack surface.

##### Cilium and the Kube-Proxy Vulnerability

Cilium running in kube-proxy replacement mode is protected against the recent `kube-proxy` [vulnerability, (CVE-2020-8558)](https://groups.google.com/g/kubernetes-security-announce/c/B1VegbBDMTE?pli=1). This is because Cilium uses socket-based load-balancing also known as ["host-reachable services"](https://docs.cilium.io/en/v1.8/gettingstarted/host-services/#host-services) which does not require `route_localnet=1` in order to make NodePort services reachable via the loopback interface. These layer 3 protections are consistent with what we discussed in the section above. The NodePort service can be accessed by default from a host or Pod within a cluster via its public IP address, any local address, or loopback address, e.g. `127.0.0.1:NODE_PORT`. This means processes bound to `127.0.0.1` aren’t reachable by anything other than other localhost processes.

##### Wrapping Up

Multitenancy brings many advantages but also risks to a Kubernetes environment. Cilium has created a network model where workloads are protected from attacks originating inside or outside of the network perimeter, which therefore moves Kubernetes closer to the future of the Zero-Trust Network. We developed Cilium from the ground up with secure multitenancy built into the design. eBPF gives us the power to realize this in ways that existing implementations simply couldn't.

If you're using (or are looking to use) Cilium for secure multitenancy, head over to our [Slack channel](https://slack.cilium.io) and let us know or ask a question. We have a growing and knowledgeable community and we're always interested in helping steer you in the right direction.

##### Image Attribution

- [https://unsplash.com/photos/0Yiy0XajJHQ](https://unsplash.com/photos/0Yiy0XajJHQ)
- [https://unsplash.com/photos/M5tzZtFCOfs](https://unsplash.com/photos/M5tzZtFCOfs)

#### How Cilium Protects Against Common Network Attacks

URL: https://cilium.io/blog/2020/06/29/cilium-kubernetes-cni-vulnerability
Date: 2020-06-29
Categories: How-To

_Contributed by Jed Salazar, Senior Solutions Architect, Isovalent_

Recently a [vulnerability](https://nvd.nist.gov/vuln/detail/CVE-2020-10749) was discovered by [Etienne Champetier](https://github.com/champtar) that impacted several Kubernetes CNIs. The vulnerability worked by having an attacker pod send rogue IPv6 “Router Advertisement” packets to the host worker node, causing the node to route its IPv6 traffic through the attackers pod (commonly known as “Man-In-The-Middle”). Fortunately for users of Cilium, this vulnerability didn’t impact their environments because of several built-in and on-by-default security features provided by Cilium.

In this blog post, we’ll discuss how on-by-default Cilium features automatically protect against these common types of network attacks.

Before we dive into why Cilium users are protected, let’s first discuss the basics of IPv4 and IPv6, and how it relates to the vulnerability.

##### IPv4 and IPv6 Basics

In IPv4, when a machine attaches to a network it can learn about its network configuration (including router details) via DHCP. The machine sends a broadcast packet called `DHCPDISCOVER` to all users on the network and if everyone is playing nicely, only the DHCP server sends back a `DHCPOFFER`. Unfortunately, any device on the network can be configured to listen for discover packets and can send an offer, which means a race condition (who can send the offer packet first) determines which DHCP server the client trusts. Because this mechanism offers no authentication or security, users must rely on a _trusted network_ model where they trust every device on the network.

Similar to DHCP, in IPv6 a host on the network can learn about its router or other neighbors via a mechanism called _Neighbor Discovery Protocol_ (NDP). A machine can send a _Router Solicitation_ packet to local peers looking for its local router on the network, or a router can preemptively send a _Router Advertisement_ to all hosts attached to a local link, indicating it’s the device to route IPv6 traffic through. This Router Advertisement is sent to a special IPv6 address analogous to broadcast called _All Nodes_ (`ff02::1`) so everyone on the same network link receives and trusts the advertisement. There have been some notable improvements in IPv6 that mitigate the broad trust applied to IPv4 DHCP, in particular the use of link-local addresses ensures that Router Advertisements are never routed across links and are only visible to point-to-point links and directly-connected peers.

However, this assumption of trust in link-local adjacency breaks down in Kubernetes, as pods attach to the host worker node's network space via a point-to-point link. Therefore, in this context, IPv6 operates on the same _trusted network_ model where devices trust other devices on the link-local network.

##### The Vulnerability

Based on what we know about IPv6 Router Advertisements, the vulnerability is described as follows.

> By sending “rogue” router advertisements, a malicious container can reconfigure the host to redirect part or all of the IPv6 traffic of the host to the attacker-controlled container.

It’s important to distinguish the fact that this isn’t a Kubernetes vulnerability, Kubernetes simply relies on the same IPv6 standards and trusted network model discussed above.

The “rogue” aspect of the vulnerability references the fact that the attacker pod spoofs its source IP address. Neighbor Discovery uses _link-local_ addresses (`fe80::/10`) to send traffic, including Router Advertisements. Link-local addresses are not routed to other links and are used for local communication, therefore they do not require a unique IP address. All enabled IPv6 interfaces have a link-local address.

This means that any interface that _supports_ IPv6 will have a `fe80` address. You can verify this with `ip addr show|grep fe80` even on a network interface not running IPv6 (note the `scope link` which means link-local). This means that the attacker pod uses its link-local address as the source IP for the Router Advertisement.

How can the attacker pod spoof its IP address? In Kubernetes, workloads run with a set of _Linux Capabilities_ that define what permissions a process/thread is capable of. This eliminates the need for processes to have root which gives overly broad permissions. One of the permissions granted to pods by default in Kubernetes is the `CAP_NET_RAW` capability, which means a pod can send “raw” packets which includes permissions to set an arbitrary source IP addresses, ports, and so on. You can see if your pod has these capabilities with `kubectl exec`.

```
kubectl exec -it ubuntu-pod -- capsh --print |grep cap_net_raw
```

The vulnerability is exploited by the attacker pod spoofing the link-local source IP, and sending Router Advertisements to the node, causing the node to trust the attacker pod as its router and sending some or all of its IPv6 traffic directly to the attacker pod.

##### How Cilium Protects Users

Cilium has several built-in and on-by-default security features that offer protection against this vulnerability.

In these examples, we’ll use `scapy` from a Cilium-managed pod to craft and send arbitrary packets to the All Nodes address. [Scapy](https://scapy.net/) is an excellent tool for testing and since our pod has `CAP_NET_RAW` Linux capabilities, we can create and send any packets we like.

We’ll test on Kubernetes 18 using Cilium 1.8.

```
kubectl get nodes -o wide
NAME     STATUS     ROLES    AGE     VERSION        INTERNAL-IP      EXTERNAL-IP      OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
master   Ready   master   3m58s   v1.18.3+k3s1   192.168.80.10    192.168.80.10    Ubuntu 19.10   5.3.0-51-generic   containerd://1.3.3-k3s2
agent1   Ready   <none>   2m4s    v1.18.3+k3s1   192.168.80.101   192.168.80.101   Ubuntu 19.10   5.3.0-51-generic   containerd://1.3.3-k3s2

kubectl exec -tin kube-system cilium-88vk8 -- cilium version
Client: 1.8.0-rc2 ce5852475 2020-05-29T21:53:07+02:00 go version go1.14.3 linux/amd64
Daemon: 1.8.0-rc2 ce5852475 2020-05-29T21:53:07+02:00 go version go1.14.3 linux/amd64
```

###### Layer 3 Protection

Even though all enabled IPv6 interfaces have a link-local address, IPv6 must explicitly be enabled on in Cilium, otherwise IPv6 traffic is automatically dropped. We can observe this by deploying Cilium with `enable-ipv6: "false"` and verify that traffic is dropped with `cilium monitor -t drop` from a Cilium Agent pod.

First, let’s deploy a standard Ubuntu container and create the packet with `scapy`.

```
kubectl exec -it ubuntu -- /bin/bash
root@ubuntu:/# apt-get update && apt-get install scapy -y
```

Start scapy and determine the IP settings.

```
scapy3
>>> import subprocess
>>> subprocess.run(["ip", "addr", "show"])
```

Recall that even though we disabled IPv6, because the interface is enabled, it will configure a `fe80` link-local address. This creates the vulnerability where the attacker pod can send Router Advertisements using the auto-assigned `fe80` link-local address. We can verify that the link-local address has been configured on the interface by inspecting the `ip addr show` output.

```
inet6 fe80::68a6:e6ff:fe36:487e/64 scope link
```

Using `scapy` to craft the packet, we could use any IPv6 link-local address, but the principle is the same whether it's the link-local address seen above or not. We will use (`fe80::1`) here as the source IP for ease of readability and set the destination IP as the All Nodes address (`ff02::1`).

We’re almost ready to send the packet, but before we do, let’s `exec` into our Cilium pod on the node and observe the traffic drops with `cilium monitor -t drop` to make sure Cilium drops the unsupported IPv6 traffic.

```
kubectl exec -tin kube-system cilium-88vk8 -- /bin/bash
root@agent1:/home/cilium# cilium monitor -t drop
```

Let’s send the packet in scapy and inspect the drops in `cilium monitor`.

```
send(a/b/c/d/e)
Sent 1 packets.
```

Sure enough, Cilium drops the unsupported IPv6 traffic.

```
xx drop (Unsupported L3 protocol) flow 0x0 to endpoint 0, identity 28896->0: fe80::1 -> ff02::1 RouterAdvertisement
```

###### IP Spoof Prevention

So Cilium protects against IPv6 traffic if we explicitly disable it, but what about spoofed packets? Cilium provides protection against spoofed IP addresses by dropping packets that have a source IP address that wasn’t provided via IP Address Management (IPAM).

Let’s verify this by redeploying Cilium with `enable-ipv6: "true"` and use the same scapy packet to send the packet with the spoofed link-local address. We can again, monitor for drops in `cilium monitor -t drop`.

```
a.src = “fe80::1”
send(a/b/c/d/e)
Sent 1 packets
```

```
xx drop (Invalid source ip) flow 0x0 to endpoint 0, identity 15625->0: fe80::1 -> ff02::1 RouterAdvertisement
```

Here we can see that since the source IP wasn’t provided by Cilium’s IPAM subsystem, we know it’s a spoofed IP address and Cilium automatically blocks the traffic. Built-in Layer 3 Protection and IP Spoof Prevention are just some of the ways that Cilium automatically protects against common network attacks.

##### Wrapping Up

Cilium takes security seriously. Security teams can be sure that Cilium provides critical, built-in, and on-by-default protections to secure their Kubernetes environments. Additionally, these teams can take advantage of the powerful security observability data provided by [Hubble](https://cilium.io/blog/2019/11/19/announcing-hubble/) to get full insights into the network activity in their environment.

##### Further Reading

For further information on Linux capabilities and container security, we recommend reading [O’Reilly Container Security](https://www.oreilly.com/library/view/container-security/9781492056690/).

#### Cilium 1.8: XDP Load Balancing, Cluster-wide Flow Visibility, Host Network Policy, Native GKE & Azure modes, Session Affinity, CRD-mode Scalability, Policy Audit mode,...

URL: https://cilium.io/blog/2020/06/22/cilium-18
Date: 2020-06-22
Categories: Release

We are excited to announce the Cilium 1.8 release. A total of 2162 commits have
been contributed by a community of 182 developers, many of whom made their first
contributions this cycle. Cilium 1.8 brings with it a trove of exciting new
features:

- **XDP Load Balancing Support:** eXpress Data Path (XDP) is the fast-lane for
 networking in the Linux kernel, built on eBPF. We've extended our existing
 eBPF kube-proxy replacement to accelerate service forwarding by 5x in our
 tests while dramatically reducing CPU consumption at the same time.
 ([More details](#kubeproxy-removal))
- **Cluster-wide Flow API**: Hubble Relay builds on the solid core of Hubble
 and Cilium to provide deep observability across the entire cluster via a
 centralized API with minimal overhead.
 ([More details](#hubblerelay))
- **Better policy visibility and control:** ClusterwideNetworkPolicy now
 supports matching hosts in the cluster to implement
 [Host network security protection](#hostfw), and all policy types gain
 named ports support. New community
 contributors have built eBPF notifications for [Policy
 Verdicts](#policyverdicts) and a [Policy Audit mode](#auditmode) to
 incrementally deploy network policies in your cluster.
 ([More details](#policy))
- **Performance optimizations across the board:** We've improved the
 performance and resource usage in almost every dimension in this release,
 from [improving CRD scalability](#crdscale) and optimizing the Cilium agent's
 [memory footprint](#memory) to various performance enhancements in our eBPF
 data path and size reduction of the Cilium [container image](#container-size).
 Hubble has been optimized to minimize resource usage by
 [embedding the core functionality](#hubble-embedded) into the Cilium agent.
 ([More details](#performance))
- **Making more functionality iptables-free:** We've worked hard on improving
 Cilium's service implementations to further reduce the dependence on external
 tools based on iptables. Several features are now implemented natively in
 eBPF, ranging from [Session Affinity](#affinity) and [HostPort](#hostport) to
 [IP masquerade agent](#masq) and [IP fragmentation support](#ipfrag).
 ([More details](#iptables-free))
- **Many more features:** [Native Azure IPAM](#azureipam) provides better
 integration for Azure Cloud via a new IPAM plugin, datapath load balancing
 support was extended to support environments with
 [multiple native devices](#multi-dev), and initial support for
 [ARM64](#arm64) has been added with docker image snapshots.
 ([More details](#much-more))
- **Upstream community contributions:** For a deep dive on the work we're doing
 with the Linux kernel and LLVM development communities, including nitty-gritty
 details on [socket load balancing improvements](#socketlb), [eBPF instruction
 set improvements](#alu32) and [transparent proxying in eBPF](#tproxy), see
 our community contributions writeup.
 ([More details](#kernel))

###### Are you using Cilium?

Consider listing your use case in the
[USERS.md](https://github.com/cilium/cilium/blob/master/USERS.md) file by
opening a [pull request](https://github.com/cilium/cilium/edit/master/USERS.md).
Sharing experiences and learning from other users is crucial in open-source. It
also helps us understand common features that our users depend on to align CI
and testing priorities.

###### New Core Team members

Congratulations to the new core team members based on 1.8 contributions:

- Chris Tarazi, Ilya Dmitrichenko, Quentin Monnet, Paul Chaignon, Robin Hahling, Tobias Klauser (Isovalent)
- Weilong Cui, Yongkun Gui, Zang Li (Google)

###### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms such as Kubernetes.

At the foundation of Cilium is a new Linux kernel technology called eBPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. eBPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption as well as network and service security. Besides
providing traditional network level security, the flexibility of eBPF enables
security with the context of application protocols and DNS requests/responses.
Cilium is tightly integrated with Envoy and provides an extension framework
based on Go. Because eBPF runs inside the Linux kernel, all Cilium
functionality can be applied without any changes to the application code or
container configuration.

See the section **[Introduction to Cilium](https://docs.cilium.io/en/v1.8/intro/)** for a more detailed general
introduction to Cilium.

###### Kube-proxy replacement at the XDP layer

_Contributed by Daniel Borkmann (Isovalent)_

We first introduced Cilium's kube-proxy replacement in eBPF in Cilium's 1.6 release and further improved it in 1.7 to run by default in new Cilium deployments. For the Cilium 1.8 release we took it to another extreme and now _accelerate_ our eBPF-based kube-proxy replacement handling for Kubernetes service types `NodePort`, `LoadBalancer` and services with `externalIPs`. eXpress Data Path (XDP) in the Linux kernel allows for significantly better performance while at the same time making more efficient use of CPU resources on the worker nodes.

The XDP layer, which we co-maintain and develop in the Linux kernel, is directly integrated into network drivers and supported by all major 10G, 40G or
faster NIC drivers in Linux, including most major cloud provided virtual NICs. The XDP layer operates at the _earliest_ possible point in software to process
packets coming right off the driver's receive queue with eBPF, which is what makes it so fast. For more details including performance benchmarks, see our
open access [publication] on XDP itself.

In order to understand where Cilium's XDP service acceleration fits into the bigger picture, below is a brief outline of Cilium 1.8's service load balancing
architecture:

As can be seen, the Cilium's kube-proxy replacement in eBPF consists at a high-level of
two main components: eBPF at the socket layer and eBPF at the driver layer.
East-West traffic, that is, service traffic between all Cilium-managed nodes,
is solely handled at the socket layer in the kernel - before any memory
allocations occur for packet metadata. Executing at this point allows Cilium to
eliminate per-packet costs for service translation. North-South traffic, that
is, all inbound service traffic from an external source to a Cilium-managed
node is handled as close as possible to the driver layer, operating on a single
interface for both ingress and egress. This allows forwarding to be handled
very fast, even dropping or reflecting traffic back out the inbound interface
before any expensive operations occur higher up the stack. The latter component
which is handling North-South traffic is then accelerated via XDP.

Cilium's service XDP acceleration currently supports direct-routing mode,
sharing the same core code as our tc eBPF implementation. Three options are
provided for redirecting traffic to remote backends after service translation
in XDP: DSR, SNAT and Hybrid.

- **DSR:** "Direct Server Return" (DSR) was introduced in 1.7 in the tc eBPF layer in order to remove an extra hop and therefore latency for reply traffic as well as for client source IP address preservation. In Cilium's DSR mode, we encode the service IP/port tuple as an IP option only once for TCP in SYN packets, or in every UDP packet.

- **SNAT:** The simplest mode which does not require any MTU adoption is that
 packets are SNAT'ed through our eBPF-based NAT engine on the node. This
 mode does not preserve the client source IP address, but is the most
 generic with regards to the underlying fabric, requiring no special
 knowledge or configuration.

- **Hybrid:** In Cilium 1.8 we have _newly_ added a Hybrid mode for eBPF
 service handling which is a middle-ground between plain SNAT and plain DSR.
 In the Hybrid mode, all TCP service traffic is subject to DSR and all UDP
 service traffic is subject to SNAT. This helps to avoid MTU changes and
 assumes that the main user workload operates on the TCP transport while
 slow(er)-path service traffic such as DNS operates on UDP. Hybrid mode can
 be configured in helm by setting `global.nodePort.mode=hybrid`.

**Why does it matter?**

We performed initial benchmarking where we've deployed a _single_ service onto a
freshly kubeadm-deployed node with a bleeding edge 5.7 kernel, ran with iptables- and
ipvs-based kube-proxy to get a baseline, and then plugged Cilium's kube-proxy replacement
in eBPF from tc and XDP side right in front of it:

Initial results show a dramatic increase in Cilium's kube-proxy replacement's XDP
acceleration being able to max out the packet generator and push all 10M incoming
requests to a remote service backend, whereas with kube-proxy the node under test is
only able to forward ~2.1M requests per second for that same service while the rest
gets dropped. The latter was similarly observed for ipvs which appears to have a
slightly higher per-packet cost despite having a better "first-packet" scalability for
large number of services compared to iptables. Plugging the kube-proxy replacement in
front with Cilium's tc eBPF implementation not only solves the "first-packet" scalability
but also increases the performance as can be seen with the observed ~3.6M requests per
second for that node, though that is still no comparison to the significantly better
gain when Cilium gets accelerated at the XDP layer:

Comparing flame-graphs for kube-proxy
and Cilium's XDP implementation under 10M
requests per second also shows the shortcut when accelerating service handling right
off the driver's poll routine. Moreover, forwarding under the XDP acceleration takes
significantly less processing overhead in softirq context compared to both Cilium running
eBPF under tc as well as kube-proxy in iptables and ipvs mode. Below test is run on an
otherwise idle system where the node's CPU is only consumed by processing softirqs. The
available remaining CPU capacity is shown in the graph. What can be seen is that even
under rates as low as ~1M requests per second for that specific node the CPU only spends
about 13% of its time in softirq context with XDP, thus leaving a remainder of 87%
capacity to be used elsewhere while in the kube-proxy case the CPU spends at least 60%
of its time in servicing softirq context with only a free remainder of at most 40%. The
kube-proxy case gets significantly worse under ~2M or ~4M requests per second where there
is only between 1-2% free share while the CPU spends 98% of its time in softirq context
for packet processing:

In short, being able to accelerate Kubernetes service handling under XDP with Cilium
dramatically increases the performance to push packets to remote backends under
significantly less CPU overhead. This also increases the overall capacity of the cluster
under the default `externalTrafficPolicy: Cluster` for services. Meaning, scaling out
a service to more backends is only as good as the upper limit of a single node's forwarding
capacity to those backends. But even if there is no requirement to handle that many packets
for a given Kubernetes deployment, these CPU cycles can then be freed and spent on actual
user workloads instead.

**Enabling Cilium's kube-proxy replacement at the XDP layer**

The `global.nodePort.acceleration=native` setting can be configured in helm in
order to enable XDP-based acceleration for native XDP-supported devices in Cilium.
Further information including a list of native XDP supported drivers and instructions
for running on bare metal or on various cloud providers can be found in our
Kubernetes without kube-proxy getting started guide.

###### Cluster-wide observability with Hubble Relay

_Contributed by Robin Hahling (Isovalent) and Michi Mutsuzaki (Isovalent)_

Hubble Relay is a new component introduced as technology preview with Cilium
1.8 which brings cluster-wide network observability for Kubernetes. Previously,
Hubble operated entirely on a per-node basis so gathering full cluster
visibility required tedious iteration across Hubble instances in the cluster to
gain visibility on the flows that route through each node in the cluster.
Hubble Relay bridges this gap by exposing the Hubble API for the scope of the
entire cluster and can be leveraged by Hubble CLI and UI components.

Hubble Relay runs as a deployment which requires little to no configuration as
it leverages Hubble to discover all Hubble peers within a cluster and connect
to their API endpoints. Running Hubble Relay as a deployment provides
flexibility in terms of scheduling and resource allocation. Furthermore, as it
is decoupled from Cilium, specific network policies can be applied.

A new networking and security observability with Hubble getting started guide has been added to our documentation. It provides a walkthrough of setting up a local multi-node Kubernetes cluster on Docker using kind in order to demonstrate some of Hubble’s capabilities, including using Hubble Relay.

Our vision for Hubble Relay is to make it the entrypoint for cluster-wide
observability. Ultimately, once properly secured, it also becomes possible to
make the service accessible from outside the cluster. As such, it should not
only become the goto tool for operation teams who need to troubleshoot cluster
network issues but also for security teams who need observability for security
purposes.

In Cilium 1.8, Hubble Relay is required for Hubble UI.

###### Improved Policy visibility and control

###### Host-level Network Policies

_Contributed by Paul Chaignon (Isovalent)_

Expanding upon the
CiliumClusterwideNetworkPolicies introduced in Cilium 1.7, this release
brings the same network policy constructs to defining policy to protect your
cluster nodes. The `CiliumClusterwideNetworkPolicies` gains a new optional
field `nodeSelector` to select nodes by their labels.

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumClusterwideNetworkPolicy
description: "Allow only kube-api, kube-dns, Cilium's API, and SSH."
metadata:
  name: 'host-policy-example'
spec:
  nodeSelector:
    matchLabels:
      role: worker
  ingress:
    - toPorts:
        - ports:
            - port: '6443'
              protocol: TCP
            - port: '22'
              protocol: TCP
```

As with pods, Cilium allows all traffic until you load a host policy, at which
point it switches to a default-deny behavior. For example, the host policy
above grants access to the host on ports 6443 and 22 only. Clients will still
be allowed to connect to the pods on this host, but connections to the host
itself will be filtered. Host network policies are in tech preview, for more
details and limitations see the
[Host Policy Documentation](https://docs.cilium.io/en/v1.8/policy/language/#hostpolicies).

###### Policy Verdict Notifications

_Contributed by Zang Li (Google), writeup by Joe Stringer (Isovalent)_

Cilium monitor notifications at the daemon level can be fairly chatty as they
have traditionally been based upon per-packet events that traffic was either
forwarded or dropped. Policy Verdict Notifications are a new form of
notification emitted by the eBPF datapath whenever a policy decision is made,
on a per-connection basis. Use `cilium monitor -t policy-verdict` to see the
verdicts:

```
# cilium monitor -t policy-verdict
...
Policy verdict log: flow 0x1fdbbff4 local EP ID 1121, remote ID 16307, dst port 80, proto 6, ingress true, action allow, match L3-L4, 10.29.210.187:42768 -> 10.29.50.40:80 tcp SYN
Policy verdict log: flow 0x3cc7e88f local EP ID 343, remote ID 11862, dst port 80, proto 6, ingress true, action deny, match none, 10.29.171.240:39126 -> 10.29.47.87:80 tcp SYN
```

###### Policy Audit Mode

_Contributed by Arthur Evstifeev (GitLab), writeup by Joe Stringer (Isovalent)_

One of the challenges for cluster operators deploying Kubernetes networking
plugins is how to go from zero network policy enforcement to full enforcement,
particularly when migrating large existing applications over to cloud-native
infrastructure. Kubernetes default-deny policies force the user to understand
the full picture of how their application uses the network and to encode all of
that knowledge into a Network Policy before enabling it in their cluster. Any
omissions from the policy can result in broken applications, requiring an
iterative process of attempting policy rollout, observing broken applications,
rolling back, updating the policy, and starting the process again.

Policy Audit mode allows the user to instead allow all traffic during the
policy transition period, logging all connections that would otherwise be
dropped by policy using [Policy Verdict Notifications](#policyverdicts).
Policy audit mode may be configured for the entire daemon using
`config.policyAuditMode=true`, and it supports all L3/L4 policy types. When this
mode is enabled, the action from the policy verdict log is reported as `action audit`. Using the policy verdict notifications, we can then form a policy to
match on the traffic and allow it. Using the same example as the policy
verdicts section above:

```
$ cat sw_l3_l4_policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "L3-L4 policy to restrict deathstar access to empire ships only"
metadata:
  name: "rule1"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
$ kubectl create -f sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created
# kubectl -n kube-system exec -ti $(get_cilium_pod) cilium monitor -t policy-verdict
Policy verdict log: flow 0xabf3bda6 local EP ID 343, remote ID 2986, dst port 80, proto 6, ingress true, action allow, match L3-L4, 10.29.210.187:59824 -> 10.29.47.87:80 tcp SYN
```

Now that we inject a policy to match on the traffic, we can see that the action
is now to `allow` the traffic. For more details see the new Policy Creation Getting Started Guide.

##### Performance

###### Scale validation of the CRD-only mode

_Contributed by André Martins (Isovalent)_

In Cilium's 1.5
we announced support to 5k nodes and 100k pods though was only possible using an
external KVStore. In Cilium's 1.6
we announced the initial support to run Cilium without required a KVStore using CRD from
Kubernetes.

As more of our users switched over to this CRD mode, we have made significant
improvements in Cilium to have the same scalability results achieved in 1.5.

In this release, we have created a detailed scalability report on Cilium
entirely running in CRD mode for a cluster running 1000 nodes and 50000 pods.
This report is part of the official documentation and can be found in here.

_Cilium endpoint regeneration time in the 99th percentile, i.e. time it took the
policy being enforced on the 10 slowest nodes out of 1000 nodes, for 50000 pods
with 250 CiliumClusterwideNetworkPolicies being sequentially created_

###### Memory Footprint Optimizations

_Contributed by Tobias Klauser (Isovalent)_

Optimization of Cilium's memory footprint has been a frequent user request. Among the biggest
contributors to Cilium's memory consumption are the eBPF maps. In this release we introduce a new
agent option to determine the size of several large eBPF maps dynamically based on the available node
memory. Moreover, the default sizes for certain large eBPF maps are now reduced and their memory
usage was optimized.

The following example will allocate 0.5% (`0.005`) of the total system memory available on the
node for the large eBPF maps (connection tracking, NAT, neighbor and load-balancing maps):

```
helm install cilium cilium \
    --namespace kube-system \
    --set config.bpfMapDynamicSizeRatio=0.005
```

In case a particular map should be made bigger or smaller, the dynamic sizing can be overridden by
specifying a fixed size. The following example will allocate the NAT eBPF map with a fixed size of
841429 entries while the remaining maps will use 0.3% of the system memory:

```
helm install cilium cilium \
    --namespace kube-system \
    --set config.bpfMapDynamicSizeRatio=0.003 \
    --set global.bpf.natMax=841429
```

On new installations, the ratio is set to 0.25% (`0.0025`) by default. This number was chosen such
that the map sizes approximately correspond to the Linux connection track table sizes used by
`kube-proxy`. More details can be found in the eBPF maps section
of the Cilium documentation.

In addition, various improvements were made to reduce the size of all the Cilium binaries and
container images. This directly results in a smaller RSS (Resident Set Size) of the Cilium processes at runtime and thus a smaller memory footprint. In addition, Cilium 1.8 is built using Go release 1.14 which includes several improvements
resulting in both, reduced memory footprint and reduced binary size.

| Binary | 1.7 | 1.8 | Reduction |
| ------------------------- |:---: |:---: |:-------: |
| `cilium` | 47MB | 47MB | 0% |
| `cilium-agent` | 74MB | 56MB | -24% |
| `cilium-bugtool` | 12MB | 8.5MB | -29% |
| `cilium-cni` | 47MB | 14MB | -70% |
| `cilium-docker` | 23MB | 16MB | -30% |
| `cilium-health` | 21MB | 14MB | -33% |
| `cilium-health-responder` | 7.4MB | 5.4MB | -27% |
| `cilium-operator` | 75MB | 58MB | -23% |

###### Cilium Container Image Optimizations

_Contributed by Daniel Borkmann (Isovalent)_

In Cilium 1.8, the container image has been reworked significantly in order to remove image
size overhead. Comparing the official Cilium docker images, the 1.8 image has been shrunk
by about 280MB compared to its previous major release:

```
# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
cilium/cilium       v1.8.0              6856f0c0841b        About an hour ago   422MB
cilium/cilium       v1.7.5              7971361d1f2e        10 days ago         713MB
cilium/cilium       v1.6.9              f5f5dacc4ab0        2 weeks ago         740MB
cilium/cilium       v1.5.13             94447e683ae4        7 weeks ago         636MB
```

The base runtime image has been upgraded from Ubuntu 18.04 LTS to Ubuntu 20.04 LTS and additional
dependencies reduced to a bare minimum. More concretely, this means that we removed shipping a
generic compiler, linker, binutils, along with libc and system headers. In other words, there is
now an empty `/usr/include/` and Cilium's eBPF code has been reworked to be fully self-contained.
Overall this rework helped to shrink the size by roughly 300MB, after Hubble CLI integration
into the Cilium image in 1.8, we end up with a final 280MB of savings.

While in Cilium 1.7 and prior releases we relied on Ubuntu 18.04's Clang-7 and LLVM-7 compiler
for eBPF, Cilium 1.8 now ships with its own Clang-10 and LLVM-10 build along with additional
BPF code generation optimizations
that we have implemented and backported from latest upstream LLVM which we have been running in
production from Hubble side prior to that. The major LLVM update ensures better overall eBPF code
generation, while also allowing us to remove _all_ non-relevant backends from the `clang` and
`llc` binary. In other words, there is _no_ x86 LLVM backend anymore which would be able to
generate executable code. Instead, _only_ eBPF backends are available for generating Cilium's eBPF
object files that are loaded through our eBPF infrastructure into the kernel:

```
# docker run -it docker.io/cilium/cilium:v1.8.0 /bin/sh
# llc --version
LLVM (http://llvm.org/):
  LLVM version 10.0.0
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: haswell

  Registered Targets:
    bpf   - BPF (host endian)
    bpfeb - BPF (big endian)
    bpfel - BPF (little endian)
#
```

This is possible since we have also fully reworked our feature probing facilities which
test the underlying kernel for available eBPF verifier, program, map and helper capabilities. The
base tooling including LLVM which is integrated into the cilium-runtime image can be found here.

###### Near-Zero Overhead Network Visibility with Hubble

_Contributed by Michi Mutsuzaki (Isovalent) and Robin Hahling (Isovalent)_

The development cycle of Cilium 1.7 coincided with a
[first preview release of Hubble](/blog/2019/11/19/announcing-hubble) {'-'}- an
observability platform for cloud native workloads leveraging Cilium and eBPF.
At the time, Hubble was a completely separate component which was typically
deployed alongside Cilium in order to pull information from Cilium's agent.
Hubble matured during the development cycle of Cilium 1.8 and the server part of
it was embedded in the Cilium agent. This change drastically reduces Hubble's
CPU and memory footprint by eliminating inter-process communication and cache
replication between Cilium and Hubble. This also allows the Hubble API endpoint
to be exposed directly from the Cilium DaemonSet rather than requiring an
additional resource to expose the core Hubble functionality.

In short, near-zero overhead network, service and security observability for
Kubernetes is now as easy as specifying some parameters while deploying Cilium.
See networking and security observability with Hubble for more details on how to get started with Hubble.

###### Voyaging towards iptables-free

###### eBPF Session Affinity

_Contributed by Martynas Pumputis (Isovalent) and Daniel Borkmann (Isovalent)_

This release brings the implementation of Kubernetes services session affinity in
eBPF as part of Cilium's kube-proxy replacement. This allows each connection from
the same pod or host to _always_ select the same endpoint of a service which has
been configured with `sessionAffinity: ClientIP`. The default timeout for the affinity
is three hours, but can be configured through Kubernetes' `sessionAffinityConfig`
as needed. The session affinity feature is implemented for all traffic sources
as briefly outlined in the [kube-proxy replacement at the XDP
layer](#kubeproxy-removal) section. Session affinity is enabled by default for
Cilium's kube-proxy replacement on Linux kernels 4.19 or later, with an
improved implementation available on Linux 5.7 or later.

When a request is sent from outside the cluster to a service, the request's
source IP address is used for determining the endpoint affinity, and when a
request is sent from inside the cluster, a cookie is used to set the affinity.
This cookie is either fixed (earlier kernels) or based upon the network
namespace (5.7 or later, more details in [Kernel Improvements](#kernel)). For
the fixed implementation, all applications on the host select the same
service endpoint for a given service with session affinity configured.

###### Native eBPF HostPort implementation

_Contributed by Daniel Borkmann (Isovalent)_

We have seen a fair number of users relying on Kubernetes `hostPort`, which is
the ability to bind and expose a Pod to a specific port on the host where the
Pod is running on.

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
     - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
          hostPort: 8080
```

In Cilium 1.7 and earlier, users had to deploy Cilium in chaining mode on top
of the `portmap` CNI plugin to support `hostPort`. This mode introduced
complexity in the form of more moving components, and didn't integrate as well
with Cilium's kube-proxy replacement in eBPF. In the Cilium 1.8 release, we
therefore implemented the service mapping for `hostPort` natively via eBPF as
part of our kube-proxy replacement. This allows users to simplify their CNI
configuration and replace the slow iptables-based implementation with the more
efficient native eBPF implementation in Cilium.

Further information about deploying Pods with Cilium's HostPort support can be found in
our
Kubernetes without kube-proxy getting started guide.

###### eBPF-based masquerading and ip-masq-agent

_Contributed by Martynas Pumputis (Isovalent)_

Cilium 1.8 enables a native eBPF-based masquerading for Cilium's default veth
datapath mode. The masquerading was
[first introduced in v1.5 for the IPVLAN datapath](/blog/2019/04/24/cilium-15/#bpf-based-masquerading-support),
and since then has been used by the NodePort eBPF implementation in the SNAT
mode.

The feature allows to replace iptables-based masquerading with more efficient
eBPF-based implementation. Also, it makes it possible to run a fully functioning
Kubernetes cluster with netfilter/iptables being completely disabled, which
brings improvements in performance, reliability and stability to the cluster.

By default, a subject to the eBPF-based masquerading is a packet sent from a
local pod to outside of the cluster. The latter can be controlled by
`nativeRoutingCIDR`: any IP address which does not belong to the cidr and a
any remote Cilium node IP address is considered as the outside destination.

To have more fine-grained control over the masquerading targets, we implemented
ip-masq-agent in eBPF. The latter can be configured via ConfigMap, i.e. in the
same way as the [Kubernetes ip-masq-agent](https://kubernetes.io/docs/tasks/administer-cluster/ip-masq-agent/).
Any packet sent to an IP address belonging to a CIDR from the ConfigMap will be
excluded from the masquerading.

Finally, if not specified, the eBPF masquerading program is attached to a
network device which is used in a default route. If it does not meet user
needs, the program can be attached to multiple specified by user devices. See eBPF-based masquerading for more details.

###### IP fragmentation support in eBPF

_Contributed by Quentin Monnet (Isovalent)_

Cilium is typically deployed in clusters today where iptables use is prevalent;
not all users have the freedom to go [kube-proxy free](https://docs.cilium.io/en/v1.8/gettingstarted/kubeproxy-free/). By
default when iptables is enabled in the kernel, it automatically performs
several duties, one of which is IP fragment reassembly. Applications using
protocols like UDP which have no in-protocol segmentation use this
functionality to transfer large messages over the network. However, as we've
looked at in a [prior blog post](/blog/2019/02/12/cilium-14/#sockmap-bpf-based-sidecar-acceleration-alpha),
automatically enabling iptables has an associated cost. For those looking to a
future without iptables, Cilium v1.8 pushes IPv4 fragmentation handling into
the eBPF datapath.

When a TCP or UDP datagram is fragmented, only the first logical fragment
contains the L4 header. The source and destination ports cannot be extracted
directly for the subsequent packets. This is a problem for processing at the
transport layer where port numbers - along with transport protocol and IP
source and destination addresses - are part of the 5-tuple used to perform
lookups in tables and make decisions. In Cilium, examples include the L4
load-balancing, or the application of L4 policy rules. Before version 1.8, IPv4
packets from a fragmented datagram would have to either be handled by iptables
rules in the kernel to be reassembled, or otherwise they would be dropped in
eBPF. But there is now a mechanism to track fragments and retrieve the port numbers.

The traditional way to have L4 processing with fragmentation is to reassemble
the datagram when it reaches an endpoint, and to disassemble again on the
egress path if necessary. This approach has several disadvantages: It is
demanding in resources as we need to store all fragment payloads until we can
reassemble, it is costly in computing time, and it leaves us vulnerable to the
wide range of attacks targeting IP fragments processing. Instead, we preferred
a lighter solution. When fragment support is enabled, we use a new eBPF table
to keep track of the datagram geometries. On encounter with the first logical fragment of
a datagram, a new entry is created in the table to associate the L4 port
numbers to the datagram's identifier. For the packets coming next, the ports
are easily retrieved with a lookup from that table. Although this approach may
fail if the first logical fragment is received out-of-order (the first packets
received in time have no corresponding entry in the map, they are dropped,
causing the whole datagram to be lost), it is efficient and secure. Reassembly
of the datagram is left to the Linux stack of the destination container, which
does a good job at it. If the datagram is dropped by the network policy, then
reassembly is skipped altogether.

See the [IP Fragmentation documentation] for more details. Go ahead and play
with fragments, you're allowed to break things (datagrams) now!

#...And much more

###### Azure IPAM mode

_Contributed by Thomas Graf (Isovalent) and Vlad Ungureanu (Palantir)_

Cilium has supported AWS ENI addressing natively for a couple of releases. This
release adds an equivalent IPAM mode for Microsoft Azure Cloud, allowing Cilium
to utilize [Azure Private IP
Addresses](https://docs.microsoft.com/en-us/azure/virtual-network/private-ip-addresses)
for Pod IPs.

The architecture ensures that only a single operator communicates with the
Azure APIs to avoid rate-limiting issues in large clusters. A pre-allocation
watermark allows to maintain a number of IP addresses to be available for use
on nodes at all time without requiring to contact the Azure APIs when a new pod
is scheduled in the cluster.

With this feature, Cilium becomes able to run on Azure Cloud in native routing
mode without requiring to chain on top of another CNI plugin and thus gives
full flexibility over the choice of model:

- **Encapsulation:** Utilizes an overlay network (encapsulation) to route all
 pod to pod to traffic. Allows for any number of IPs to be utilized in the
 cluster and per node.
- **Native routing:** Utilizes the Azure Cloud networking fabric with Azure
 private IP addresses. Is subject to Azure addressing limits but avoids
 encapsulation.

See the [Azure Cloud Getting Started Guide](https://docs.cilium.io/en/v1.8/gettingstarted/k8s-install-azure/) for more details.

Special thanks to our friends at Datadog and Palantir for helping to contribute this feature.

###### Multi-device support for eBPF load balancing & services

_Contributed by Martynas Pumputis (Isovalent)_

Until this release, eBPF NodePort program could have been attached to a single
external facing network device. This meant that a request from outside the cluster
to a `NodePort`, `LoadBalancer` service or a service's `externalIP` was handled
only by that particular device. This was limiting for setups with multiple
external devices (the limitation did not apply for requests to the service
originating from inside the cluster).

Cilium 1.8 brings a support for attaching eBPF NodePort to multiple devices.
Each device can be specified via the helm option `global.devices`, e.g.
`global.devices={eth0,eth1,eth2}`. If no device is specified, cilium-agent will
try to attach the program to a device with a default route and a device which
has Kubernetes `InternalIP` or `ExternalIP` set. See Kubernetes without kube-proxy for more details.

###### ARM64 support

_Contributed by Tina Tsou (Arm), Jianlin Lv (Arm) and Ilya Dmitrichenko (Isovalent)_

In the Arm ecosystem, Cilium is an indispensable part of container networking,
so we're excited to start the journey of Cilium Arm64 support in Cilium v1.8.
We fixed compilation and runtime issues on Arm64 platform and added new jobs in
Travis CI to run on Arm64 platform. Additionally, the Dockerfile now supports
building images on the Arm64 platform, and the Cilium team has initiated work
to build images that support multi-CPU architecture; they will be available
soon as regular image snapshots. In this process, the Cilium team have given a
great help, support and inspiration. Some basic features have been verified on
the Arm64 platform, but we still have more work waiting to be completed.

We hope that this is a good start to promote Cilium to a more diverse set of
platforms and production environments.

###### Upstream Linux kernel contributions

_Contributed by Daniel Borkmann (Isovalent), John Fastabend (Isovalent) and Joe Stringer (Isovalent)_

During the Cilium 1.8 development window, we've also worked on a number of improvements to the Linux
kernel's eBPF subsystem which we co-maintain. The selected changes highlighted below are generic for
all eBPF users, but have been implemented mainly in the context of Cilium's and Hubble's eBPF needs.
Most of our bleeding edge kernel changes are already used in Cilium 1.8 while few others are preparatory
work for Cilium's 1.9 development window.

###### Socket layer load balancer improvements

As shortly outlined in the [kube-proxy replacement at the XDP layer](#kubeproxy-removal) section,
one of the two main components of Cilium's eBPF kube-proxy replacement architecture is to attach
eBPF programs at the socket layer.
Each Kubernetes service type processed there has different requirements, and several of them
require the ability to differentiate between network namespaces out of those hooks. To mention
one, in case of NodePort services, accessing the service through the loopback address needs to
be translated for requests out of the _host_ namespace, while it should not be translated for
application Pods.
The [session affinity](#affinity) is yet another case of missing network namespace awareness: we do not
yet have a source IP address given a socket buffer (skbuff) has not been created at this layer, but
we could use information about the originating network namespace instead. To solve this awareness
issue, we've implemented network namespace cookies accessible via `bpf_get_netns_cookie()` eBPF helper.
The idea was adapted from socket cookies, which efficiently generates a globally unique, non-overflowing
identifier which is globally unique even beyond the lifetime of a socket pointer and can therefore be
used as part of a key in eBPF LRU maps where it can be guaranteed that stale entries are never being
reused accidentally through pointer reuse for the same object type from the kernel's allocator.

Another remaining limitation of the eBPF hooks in the socket layer that we have addressed both for the
kernel and for the Cilium 1.8 release were missing attachment types for `getpeername(2)` and `getsockname(2)`.
As discussed during this year's bpfconf,
we have seen a few rare occasions where applications were bailing out given their peer's address returned
from `getpeername(2)` did not match the service address the application originally connected to. This is because
the application's connection was performed on the service IP/port tuple but the `getpeername(2)` call returned
the IP/port tuple after it had been rewritten by Cilium for service translation. With the help of the new hooks,
Cilium now reverse translates the `struct sockaddr` on `getpeername(2)` calls in order to always return the
initial IP/port tuple used by the application.

Last but not least, we have equipped the eBPF hooks in the socket layer with the ability to export custom
audit and tracing information to user space through the perf event ring buffer and to enable memory probing without faulting from the tracing side. In
particular while the latter is a small change to the kernel, it comes with a paradigm-shift for networking
in specifically since the boundary between the networking and tracing landscape becomes blurry - this
essentially enables the set of `bpf_probe_read_user()`, `bpf_probe_read_kernel()`, `bpf_probe_read_user_str()`
and `bpf_probe_read_kernel_str()` eBPF helpers to the kernel that we have added recently for strict access
under either `USER_DS` or `KERNEL_DS` and allows to enrich load balancing, policy or tracing context
based on data that can be gathered from the `current` task natively out of eBPF socket programs.

See the merged patch sets for further information: here, here and here

###### Verifier ALU32 signed and unsigned min/max bounds tracking

Clang 10 was recently released.
Among the numerous fixes and features, eBPF ALU32 support was enabled by
default here. This improves code generation
to reduce instructions needed for some operations and in some cases improves visible performance. However,
as we enabled this we observed a missing piece. Namely, the kernel verifier did not fully
track ALU32 operations. This would result in code that appeared valid when
inspected at the C language level, but would still generate verifier errors when loading
the program into the kernel.

At a high level the verifier needs to ensure access into map values, the stack, and
other operations are safe. Safe is context dependent, but for instance
a map value can only be read or written up to its defined size and any access past
the end of the allocated size needs to throw an error. When we write into the stack
we need to ensure that the write is not past the end of the stack and so on. One of
the ways the verifier ensures this is by tracking the minimum and maximum values of
any scalars (variables in C code) it comes across. However, the verifier only tracked bounds over the
64-bit register. This resulted in conservative estimates that when 32-bit registers
were being used could result in the verifier over-estimating the range of possible values. For example,
C code may result in over-estimates when mixing 'int' types which are 32-bits
and other types, such as the 64-bit C type 'uint64_t'. Numerous workarounds for
this problem involved using casts, avoiding int types altogether and preventing compiler optimizations
by placing compiler barriers in the C code. All of this works, but is unnatural
from the C programmer's point of view.

To resolve this and allow for more natural C code with fewer verifier errors in
otherwise safe code we improved the eBPF LLVM backend and Linux kernel eBPF
verifier. First, we improved the Linux verifier by tracking 32-bit minimum
and maximum bounds in addition to the already existing 64-bit bounds. By
doing this we accurately track the bounds of a register and avoid the over
estimates mentioned earlier. Next, we added additional improvements on the bounds
by noting eBPF 32-bit operations zero-extend by definition. That is, a 32-bit move
operation will zero the upper 32-bits of the destination register. This allows us
to implement a set of improvements to track zero extension fully. At that point
the verifier was able to track most ALU32 enabled code we loaded.
But, we noticed the eBPF LLVM backend still generated a few extra instructions
to zero extend registers unnecessarily because eBPF JITs and the eBPF emulator
all enforce zero extension. So to complete this work,
we updated the eBPF LLVM backend code generation to omit these zero extensions
when possible.

The end result is eBPF ALU32 support is now working correctly for the vast
amount of C code we have seen and additionally generating better code that
is smaller and more efficient.

See the merged patch sets for further information: here and here.

###### eBPF-based TPROXY replacement

Cilium's L7 proxy implementations make use of a feature called Transparent Proxy
(TPROXY) in the Linux kernel to [help manage the datapath at scale](https://linuxplumbersconf.org/event/4/contributions/464/).
TPROXY allows each proxy implementation to open just one socket to handle
traffic across multiple sessions, which simplifies configuration & traffic
redirection for L7 connection termination. While this has several benefits
on the userspace co-ordination side, it can involve some complex co-ordination
between multiple subsystems in Linux to correctly configure.
During this cycle, we extended the Linux eBPF API to support TPROXY directly
from eBPF programs which paves the way for Cilium to further simplify the
configuration & debugging of proxy redirection from eBPF by configuring more
of the datapath functionality through a single kernel API.

See the merged patch sets for further information: here

###### 1.8 Release Highlights

- **Hubble**
 - New hubble-relay agent for multinode support
 - Architecture work to integrate flow handling functionality in Cilium
 - Helm charts for hubble UI, CLI and Relay components
- **Scalability and Resource consumption**
 - CRD mode scale optimizations
 - Native Azure IPAM support
 - Optimize footprint using Go 1.14
 - Smaller container images
- **Policy**
 - Host-level network security protection
 - Policy verdict notifications
 - Policy audit mode for incremental policy deployment
- **Kubernetes**
 - Support for named ports
 - Implemented ip-masq-agent via eBPF
 - [Multi-CIDR KEP](https://github.com/kubernetes/enhancements/pull/808)
 - Validated with Kubernetes 1.18
 - Better CRD validation for Cilium resources
- **Datapath**
 - Nodeports in XDP
 - Support for sessionAffinity in eBPF
 - Support for hostPort in eBPF
 - IP fragment tracking support for LB and policy
 - IP masquerade in eBPF
 - Expose services through multiple devices
 - Better eBPF code generation with Clang/LLVM 10
 - Fix host reachable services with applications that use `getpeername()`
 - Various performance-related improvements
- **Upstream kernel changes**
 - Better network namespace detection
 - Improved peer name substutition handling with socket LB
 - eBPF helpers for transparent proxying
- **Istio**
 - Support for 1.5.6
- **CLI**
 - New `cilium ip list` command for ip {'<->'} identity introspection
- **Continuous Integration / Testing**
 - This release marks the start of major CI improvements effort (aka CI Force)
 - Many reliability improvements were made to current integration tests
 - Multiple GitHub Actions jobs had been added, including new smoke tests
 - Integration test coverage had been generally extended
 - New tests added for Hubble
 - Flaky tests moved to quarantine pipeline
- **Documentation**
 - Switched the default GKE guide instructions to use direct-routing
 - New guides for k3s and kind for sandbox testing

See the [Changelog](https://github.com/cilium/cilium/blob/v1.8/CHANGELOG.md)
for full notes on changes during the Cilium 1.8 development cycle.

###### Getting Started

New to Cilium? Follow one of the [Getting Started Guides](https://docs.cilium.io/en/v1.8/gettingstarted/).

###### Upgrade Instructions

As usual, follow the [upgrade guide](https://cilium.readthedocs.io/en/v1.8/install/upgrade/#upgrading-minor-versions)
to upgrade your Cilium deployment. Feel free to ping us on [Slack].

###### Release

- Release Notes & Binaries: [1.8.0](https://github.com/cilium/cilium/releases/tag/v1.8.0)
- Container image: `docker.io/cilium/cilium:v1.8.0`

[slack]: https://slack.cilium.io
[publication]: https://dl.acm.org/doi/10.1145/3281411.3281443
[ip fragmentation documentation]: https://docs.cilium.io/en/v1.8/concepts/networking/fragmentation

#### 最Cool Kubernetes网络方案Cilium入门

URL: https://cilium.io/blog/2020/05/04/guest-blog-kubernetes-cilium
Date: 2020-05-04
Categories: Community

the original blog post. If you would like to publish a blog post as well, contact us on Slack.`} />

> 最近业界使用范围最广的 K8S CNI 网络方案[Calico 宣布支持 eBPF](https://www.projectcalico.org/introducing-the-calico-ebpf-dataplane/)，而作为第一个通过 eBPF 实现了 kube-proxy 所有功能的 K8S 网络方案——Cilium，它的先见之名是否能转成优势，继而成为 CNI 新的头牌呢？今天我们一起来入门最 Cool Kubernetes 网络方案 Cilium。

###### Cilium 介绍

> 以下基于[Cilium 官网文档](https://cilium.readthedocs.io/en/stable/)翻译整理。

##### 当前趋势

现代数据中心的应用系统已经逐渐转向基于微服务架构的开发体系，一个微服务架构的应用系统是由多个小的独立的服务组成，它们之间通过轻量通信协议如 HTTP、gRPC、Kafka 等进行通信。微服务架构下的服务天然具有动态变化的特点，结合容器化部署，时常会引起大规模的容器实例启动或重启。要确保这种向高度动态化的微服务应用之间的安全可达，既是挑战，也是机遇。

###### 现有问题

传统的 Linux 网络访问安全控制机制（如 iptables）是基于静态环境的 IP 地址和端口配置网络转发、过滤等规则，但是 IP 地址在微服务架构下是不断变化的，非固定的；出于安全目的，协议端口(例如 HTTP 传输的 TCP 端口 80)也不再固定用来区分应用系统。为了匹配大规模容器实例快速变化的生命周期，传统网络技术需要维护成千上万的负载均衡规则和访问控制规则，并且需要以不断增长的频率更新这些规则，而如果没有准确的可视化功能，要维护这些规则也是十分困难，这些对传统网络技术的可用性和性能都是极大的挑战。比如经常会有人对 kube-proxy 基于 iptables 的服务负载均衡功能在大规模容器场景下具有严重的性能瓶颈，同时由于容器的创建和销毁非常频繁，基于 IP 做身份关联的故障排除和安全审计等也很难实现。

###### 解决方案

Cilium 作为一款 Kubernetes CNI 插件，从一开始就是为大规模和高度动态的容器环境而设计，并且带来了 API 级别感知的网络安全管理功能，通过使用基于 Linux 内核特性的新技术——[BPF](https://docs.cilium.io/en/stable/bpf/)，提供了基于 service/pod/container 作为标识，而非传统的 IP 地址，来定义和加强容器和 Pod 之间网络层、应用层的安全策略。因此，Cilium 不仅将安全控制与寻址解耦来简化在高度动态环境中应用安全性策略，而且提供传统网络第 3 层、4 层隔离功能，以及基于 http 层上隔离控制，来提供更强的安全性隔离。

另外，由于 BPF 可以动态地插入控制 Linux 系统的程序，实现了强大的安全可视化功能，而且这些变化是不需要更新应用代码或重启应用服务本身就可以生效，因为 BPF 是运行在系统内核中的。

以上这些特性，使 Cilium 能够在大规模容器环境中也具有高度可伸缩性、可视化以及安全性。

###### 部署 Cilium

部署 Cilium 非常简单，可以通过单独的 yaml 文件部署全部组件（目前我使用了这个方式部署了 1.7.1 版本），也可以通过 helm chart 一键完成。重要的是部署环境和时机：

1. 官方建议所有部署节点都使用 Linux 最新稳定内核版本，这样所有的功能都能启用，具体部署环境建议可以参照[这里](https://cilium.readthedocs.io/en/stable/install/system_requirements/)。
2. 作为一个 Kubernetes 网络组件，它应该在部署 Kubernetes 其他基础组件之后，才进行部署。这里，我自己遇到的问题是，因为还没有 CNI 插件，coredns 组件的状态一直是 pending 的，直到部署完 Cilium 后，coredns 完成了重置变成 running 状态。
 下图是 Cilium 的整体部署组件图：

###### 测试安装效果

官方提供了一个[connectivity 检查工具](https://github.com/cilium/cilium/blob/master/examples/kubernetes/connectivity-check/connectivity-check.yaml)，以检测部署好的 Cilium 是否工作正常。如果你的网络环境有些限制，我作了一些简单修改，可以参照[这里](https://github.com/nevermosby/K8S-CNI-Cilium-Tutorial/blob/master/cilium/connectivity-check.yaml)。部署起来很简单，请确保至少有两个可用的节点，否则有几个 deployment 会无法成功运行：

```bash
> kubectl apply -f connectivity-check.yaml

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
echo-a                                  1/1     1            1           16d
echo-b                                  1/1     1            1           16d
host-to-b-multi-node-clusterip          1/1     1            1           16d
host-to-b-multi-node-headless           1/1     1            1           16d
pod-to-a                                1/1     1            1           16d
pod-to-a-allowed-cnp                    1/1     1            1           16d
pod-to-a-external-1111                  1/1     1            1           16d
pod-to-a-l3-denied-cnp                  1/1     1            1           16d
pod-to-b-intra-node                     1/1     1            1           16d
pod-to-b-multi-node-clusterip           1/1     1            1           16d
pod-to-b-multi-node-headless            1/1     1            1           16d
pod-to-external-fqdn-allow-google-cnp   1/1     1            1           16d

```

如果所有的 deployment 都能成功运行起来，说明 Cilium 已经成功部署并工作正常。

###### 网络可视化神器 Hubble

上文提到了 Cilium 强大之处就是提供了简单高效的网络可视化功能，它是通过[Hubble](https://github.com/cilium/hubble)组件完成的。[Cilium 在 1.7 版本后推出并开源了 Hubble](https://cilium.io/blog/2019/11/19/announcing-hubble)，它是专门为网络可视化设计，能够利用 Cilium 提供的 eBPF 数据路径，获得对 Kubernetes 应用和服务的网络流量的深度可见性。这些网络流量信息可以对接 Hubble CLI、UI 工具，可以通过交互式的方式快速诊断如与 DNS 相关的问题。除了 Hubble 自身的监控工具，还可以对接主流的云原生监控体系——Prometheus 和 Grafana，实现可扩展的监控策略。

###### 部署 Hubble 和 Hubble UI

官方提供了基于 Helm Chart 部署方式，这样可以灵活控制部署变量，实现不同监控策略。出于想要试用 hubble UI 和对接 Grafana，我是这样的部署的：

```bash
> helm template hubble \
    --namespace kube-system \
    --set metrics.enabled="{dns:query;ignoreAAAA;destinationContext=pod-short,drop:sourceContext=pod;destinationContext=pod,tcp,flow,port-distribution,icmp,http}" \
    --set ui.enabled=true \
    > hubble.yaml
> kubectl apply -f hubble.yaml
# 包含两个组件
# - daemonset hubble
# - deployment hubble UI
> kubectl get pod -n kube-system |grep hubble
hubble-67ldp                       1/1     Running   0          21h
hubble-f287p                       1/1     Running   0          21h
hubble-fxzms                       1/1     Running   0          21h
hubble-tlq64                       1/1     Running   1          21h
hubble-ui-5f9fc85849-hkzkr         1/1     Running   0          15h
hubble-vpxcb                       1/1     Running   0          21h

```

###### 运行效果

由于默认的 Hubble UI 只提供了 ClusterIP 类似的 service，无法通过外部访问。因此需要创建一个 NodePort 类型的 service，如下所示：

```yaml
# hubble-ui-nodeport-svc.yaml
kind: Service
apiVersion: v1
metadata:
  namespace: kube-system
  name: hubble-ui-np
spec:
  selector:
    k8s-app: hubble-ui
  ports:
    - name: http
      port: 12000
      nodePort: 32321
  type: NodePort
```

执行`kubectl apply -f hubble-ui-nodeport-svc.yaml`，就可以通过任意集群节点 IP 地址加上 32321 端口访问 Hubble UI 的 web 服务了。打开效果如下所示：

- 页面上半部分是之前部署的一整套 conectivity-check 组件的数据流向图，官方叫做`Service Map`，默认情况下可以自动发现基于网络 3 层和 4 层的访问依赖路径，看上去非常 cool，也有点分布式链路追踪图的感觉。点击某个服务，还能看到更为详细的关系图：

 下图是 kube-system 命名空间下的数据流图，能看到 Hubble-UI 组件和 Hubble 组件是通过 GRPC 进行通信的，非常有趣。但令人感到的好奇的是，为何没有显示 Kubernetes 核心组件之间的调用关系图。

- 页面的下半部分默认显示的是对于每条数据流路径的详细描述，包括发起请求的 pod 名称、发起请求的 service 名称、请求目标的 pod 名称、请求目标的 service 名称、目标 IP、目标端口、目标 7 层信息、请求状态、最后一次查看时间等，如下图所示：

 点击任意一条 flow，可以查看到更多详细信息：

- 页面的下半部分可以通过点击切换成显示 network policy 模式，列出了当前命名空间下所有的网络策略：

如果想开启网络 7 层的可视化观察，就需要对目标 pod 进行 annotations ，感兴趣可以看[这里](http://docs.cilium.io/en/stable/policy/visibility/)，就不在入门篇详述了。

这样的网络可视化是不是你梦寐以求的，绝对能在排查请求调用问题的时候帮上大忙。

###### 对接 Grafana+ Prometheus

如果你跟一样是 Grafana+ Prometheus 的忠实粉丝，那么使 Hubble 对接它们就是必然操作了。仔细的同学已经发现之前 helm template 的玄机了：

```bash
--set metrics.enabled="{dns:query;ignoreAAAA;destinationContext=pod-short,drop:sourceContext=pod;destinationContext=pod,tcp,flow,port-distribution,icmp,http}"
# 上面的设置，表示开启了hubble的metrics输出模式，并输出以上这些信息。
# 默认情况下，Hubble daemonset会自动暴露metrics API给Prometheus。

```

你可以对接现有的 Grafana+Prometheus 服务，也可以部署一个简单的：

```bash
# 下面的命令会在命名空间cilium-monitoring下部署一个Grafana服务和Prometheus服务
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.6/examples/kubernetes/addons/prometheus/monitoring-example.yaml

# 创建对应NodePort Service，方便外部访问web服务
kubectl expose deployment/grafana --type=NodePort --port=3000 --name=gnp -n cilium-monitoring

kubectl expose deployment/prometheus --type=NodePort --port=9090 --name=pnp -n cilium-monitoring

```

完成部署后，打开 Grafana 网页，导入官方制作的[dashboard](https://github.com/cilium/hubble/blob/v0.5/tutorials/deploy-hubble-and-grafana/grafana.json)，可以快速创建基于 Hubble 的 metrics 监控。等待一段时间，就能在 Grafana 上看到数据了：

Cilium 配合 Hubble，的确非常好用！

###### 取代 kube-proxy 组件

Cilium 另外一个很大的宣传点是宣称已经全面实现 kube-proxy 的功能，包括`ClusterIP`, `NodePort`, `ExternalIPs` 和 `LoadBalancer`，可以完全取代它的位置，同时提供更好的性能、可靠性以及可调试性。当然，这些都要归功于 eBPF 的能力。

官方文档中提到，如果你是在先有 kube-proxy 后部署的 Cilium，那么他们是一个“共存”状态，Cilium 会根据节点操作系统的内核版本来决定是否还需要依赖 kube-proxy 实现某些功能，可以通过以下手段验证是否能停止 kube-proxy 组件：

```bash
# 检查Cilium对于取代kube-proxy的状态

> kubectl exec -it -n kube-system [Cilium-agent-pod] -- cilium status | grep KubeProxyReplacement

# 默认是Probe状态
# 当Cilium agent启动并运行，它将探测节点内核版本，判断BPF内核特性的可用性，
# 如果不满足，则通过依赖kube-proxy来补充剩余的Kubernetess，
# 并禁用BPF中的一部分功能
KubeProxyReplacement:   Probe   [NodePort (SNAT, 30000-32767), ExternalIPs, HostReachableServices (TCP, UDP)]

# 查看Cilium保存的应用服务访问列表
# 有了这些信息，就不需要kube-proxy进行中转了
> kubectl exec -it -n kube-system [Cilium-agent-pod] -- cilium service list
ID   Frontend              Service Type   Backend
1    10.96.0.10:53         ClusterIP      1 => 100.64.0.98:53
                                          2 => 100.64.3.65:53
2    10.96.0.10:9153       ClusterIP      1 => 100.64.0.98:9153
                                          2 => 100.64.3.65:9153
3    10.96.143.131:9090    ClusterIP      1 => 100.64.4.100:9090
4    10.96.90.39:9090      ClusterIP      1 => 100.64.4.100:9090
5    0.0.0.0:32447         NodePort       1 => 100.64.4.100:9090
6    10.1.1.179:32447      NodePort       1 => 100.64.4.100:9090
7    100.64.0.74:32447     NodePort       1 => 100.64.4.100:9090
8    10.96.190.1:80        ClusterIP
9    10.96.201.51:80       ClusterIP
10   10.96.0.1:443         ClusterIP      1 => 10.1.1.171:6443
                                          2 => 10.1.1.179:6443
                                          3 => 10.1.1.188:6443
11   10.96.129.193:12000   ClusterIP      1 => 100.64.4.221:12000
12   0.0.0.0:32321         NodePort       1 => 100.64.4.221:12000
13   10.1.1.179:32321      NodePort       1 => 100.64.4.221:12000
14   100.64.0.74:32321     NodePort       1 => 100.64.4.221:12000
15   10.96.0.30:3000       ClusterIP
16   10.96.156.253:3000    ClusterIP
17   100.64.0.74:31332     NodePort
18   0.0.0.0:31332         NodePort
19   10.1.1.179:31332      NodePort
20   10.96.131.215:12000   ClusterIP      1 => 100.64.4.221:12000

# 查看iptables是否有kube-proxy维护的规则
> iptables-save | grep KUBE-SVC
<Empty> # 说明kube-proxy没有维护任何应用服务跳转，即可以停止它了。

```

###### 小结

Cilium 作为当下最 Cool 的 Kubernetes CNI 网络插件，还有很多特性，如高阶 network policy、7 层流量控制等，这款基于 BPF/eBPF 打造出的简单、高效、易用的网络管理体验，有机会大家都来试用吧。

#### Building a Multi-node Environment with Cilium and K3s in Twenty Minutes or Less

URL: https://cilium.io/blog/2020/04/29/cilium-with-rancher-labs-k3s
Date: 2020-04-29
Categories: How-To

Like many of you, we have been feeling the pains of working remotely from home.
We are living in a difficult time where many of us have limited access to
resources. Still, the need to design, build and maintain moves on!
For many of us, it is what keeps us mentally strong on a daily basis right now.
This means we still need to test and optimize our applications. But how can we
reliably test a complex application and ensure the network policies are properly
enforced across the entire communications path?

Thanks to our friends at [Rancher Labs](https://rancher.io) and their amazing
solution for kubernetes on edge and IoT devices [K3s](https://k3s.io) it is
not only possible, but easy to do. With Cilium and K3s you can build a
multi-node Kubernetes cluster with just 8GB of memory and a modern CPU in just
minutes. A multi-node cluster can help with testing of complex application
architectures and is especially useful when diagnosing or troubleshooting
network policies. Whether you just want to take Cilium for a test drive or you
want to start designing tight network controls into your application, this
post is for you!

In this guide, we will walk through the steps required to build a multi-node
Kubernetes cluster on your local workstation or laptop using K3s and Cilium.
Then we'll show you how you can use Hubble to inspect traffic in the cluster
and visualize data which is exposed by the superpowers of eBPF and Cilium. We
will also show you how to restrict the flow of traffic between applications.
Finally, we will see how Cilium and Hubble can provide you detailed information
which can help solve problems related to compliance and regulatory concerns.

##### Pre-Requisites

- **Vagrant:** HashiCorp [Vagrant](https://www.vagrantup.com/downloads.html) is
 useful for building local development environments. It's free to install and
 use. This guide will use it with VirtualBox, but vagrant can also be used with
 a number of other [providers](https://www.vagrantup.com/intro/getting-started/providers.html).
- **VirtualBox:** [VirtualBox](https://www.virtualbox.org) is a virtualization
 platform freely distributed for personal use. In order to run
 the virtual machines in this post VirtualBox 6.1. or higher is required. To
 check if you have the correct version of VirtualBox, run the following at a
 terminal prompt: `vboxmanage --version`
- **Helm:** [Helm](https://helm.sh) version 3 is required to install Cilium and
 Hubble. To determine if you have Helm version 3 you can run the following
 command at a terminal prompt: `helm version`
- **Git:** Code examples are available via [GitHub](https://github.com/seanmwinn/cilium-k3s-demo)
 —and may be downloaded using [Git](https://git-scm.com)—but are not required.
 I have provided all of the necessary code bits here in the post to follow
 along, but if you want to customize your environment beyond this post I
 recommend cloning the repository.
- **kubectl** - [Kubernetes CLI](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
 used for interacting with the kubernetes api. We will be installing version
 1.17.4 of kubernetes, so we should install the same version or newer.
 To check the version installed type `kubectl version`. Your system should
 report the version installed for both Client and Server. If your connection
 to the server fails, this is expected without a server running.

##### Starting Your Environment

Clone the repository:

```shell-session
git clone https://github.com/seanmwinn/cilium-k3s-demo.git
cd cilium-k3s-demo
```

The guest machines will require the Virtualbox Guest Additions installed. In
order to allow vagrant to automatically install the guest additions on boxes
which do not implement it, install the `vagrant-vbguest` plugin:

```shell-session
vagrant plugin install vagrant-vbguest
```

By default, the included `Vagrantfile` will start a cluster with one K3S master node
and one K3S agent node by default. You can change the number of agent nodes by
configuring an environment variable for `K3S_AGENTS` or passing it with the
command as a prefix:

```shell-session
vagrant up
```

After about 5-10 minutes depending on your download spped, the machines should
be started and K3s should be installed and running. The provisioning script
will copy a `k3s.yaml` file to your working directory. Verify that `kubectl` can
communicate with the kubernetes master. It is expected to see pods in a
`Pending` state:

```shell-session
export KUBECONFIG=$PWD/k3s.yaml
kubectl get pods -n kube-system -o wide
```

##### Installing Cilium

Cilium is installed using Helm version 3. First, add the Cilium repository to
your helm configuration:

```shell-session
helm repo add cilium https://helm.cilium.io/
```

Then install Cilium using Helm:

```shell-session
$ helm install cilium cilium/cilium --version=1.7.2 \
    --set global.tag="v1.7.2" --set global.containerRuntime.integration="containerd" \
    --set global.containerRuntime.socketPath="/var/run/k3s/containerd/containerd.sock" \
    --set global.kubeProxyReplacement="strict" --namespace kube-system
```

This Helm command executes the following four actions:

- Selecting the v1.7.2 release.
- Configuring Cilium to be able to read information from the containerd
 runtime along with a custom path for the domain socket implemented by k3s.
- Disabling kube-proxy and using Cilium for address translation

It will take a few minutes for the Cilium agent and operator to become available.
Once we have Cilium up and running, the other pods in the cluster should also
start to transition into the `Running` state. You should see one Cilium
pod on each node as well as the cilium-operator.

##### Installing Hubble

Next we will install Hubble, an observability tool which provides deep
visibility into network connections, processes and much more thanks to Cilium
and eBPF. In order to install Hubble using helm, it's necessary to clone the
source repository:

```shell-session
$ cd ..
$ git clone https://github.com/cilium/hubble --branch v0.5
$ cd hubble/install/kubernetes/hubble

$ helm install hubble . --namespace kube-system \
    --set metrics.enabled="{dns:query;ignoreAAAA;destinationContext=pod-short,drop:sourceContext=pod;destinationContext=pod,tcp,flow,port-distribution,icmp,http}" \
    --set ui.enabled=true --set image.tag="v0.5.1"
```

Here we are setting custom values to:

- enable the Hubble UI;
- define the [metrics](https://github.com/cilium/hubble/blob/v0.5/Documentation/metrics.md)
 Hubble will collect;
- select to deploy release v0.5.1.gma

The Hubble daemonset should start a pod on each node, as well as a single pod
running the Hubble UI. This process should take a few minutes as the containers
are downloaded and started.

##### Installing MetalLB

[MetalLB](https://metallb.universe.tf/) is a software load balancer which can
expose Kubernetes services to external hosts thanks to a `LoadBalancer`
service object. We are using MetalLB to work around [an existing bug](https://github.com/rancher/k3s/issues/1216)
with multi-node clusters in the K3s service load balancer.

Create the metallb-system namespace:

```shell-session
kubectl create ns metallb-system
```

The Helm charts for MetalLB are hosted in the official stable repository. You
can see the list of configured repos with `helm repo list`. If you do not have
the stable repository configured, add it:

```shell-session
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
```

Next, install MetalLB using Helm:

```shell-session
$ helm install -f https://raw.githubusercontent.com/seanmwinn/cilium-k3s-demo/master/configmap.yaml \
    --namespace metallb-system metallb stable/metallb
```

We are passing a configmap to the MetalLB which includes the following inline
configuration:

```yaml
configInline:
  peers:
  address-pools:
    - name: default
      protocol: layer2
      addresses:
        - 192.168.80.101-192.168.80.150
```

This MetalLB configuration will establish an IP address range which will be
advertised using Layer2 protocols.

You can check that MetalLB has started up by running:

```shell-session
kubectl get pods -o wide -n metallb-system
```

You should see a single controller pod and one speaker pod for each node in the
cluster. If you used the default configuration, you should see a total of 3 pods.

##### Observing Traffic with Hubble

Now that we have a fully working multi-node environment, let's deploy some demo
applications which will allow us to observe the powers of eBPF and Cilium using
Hubble.

In our scenario, the Alliance is mounting their attack against an Empire base.
A fleet of x-wing fighters will infiltrate the security systems and breach the
docking area. Teams will then infiltrate the base and activate the internal
self-destruct mechanisms to destroy the base using one of the terminals from the
control room. This will ensure victory for the Alliance and restore order to the
galaxy.

Start off by deploying the applications:

###### Empire

```shell-session
kubectl apply -f https://raw.githubusercontent.com/seanmwinn/cilium-k3s-demo/master/example-apps/empire.yaml
```

- **docking-doors:** The docking door control systems. Our heroes need these
 systems disabled so they can breach the Empire base and transmit the codes.
- **janitor-bots:** An API for managing janitor-bots. R2D2 has detected a flaw
 in the janitor bot code which he can exploit to install backdoor program.
- **security-api:** The central security systems, tightly secured
 against attacks—or so the empire thinks...

###### Alliance

```shell-session
kubectl apply -f https://raw.githubusercontent.com/seanmwinn/cilium-k3s-demo/master/example-apps/alliance.yaml
```

We will also modify the Hubble UI service to be available via a LoadBalancer
port using MetalLB:

```shell-session
$ kubectl patch -n kube-system svc/hubble-ui --patch \
    '{"spec": {"type": "LoadBalancer"}}'
```

Obtain a list of services in the kube-system namespace and determine which IP
address was assigned to the hubble-ui service:

Here we can see the IP address for hubble-ui listed as an External IP and the
advertised port, 12000. Using a web broswer, navigate to the IP address of
hubble-ui on port 12000 and select the empire namespace in the top left corner:

You should now see a service map of all communications going to or from the
empire's various applications in the upper part of the screen, and a list of
flows and their forwarding status - either dropped or forwarded in the lower
part. Here you can see the failed attempts of the x-wing fleet trying to unlock
the docking doors which would allow them to access the empire base.

We can see in the logs that the x-wing-red fleet has been unsuccessful in their
attempts to communicate with the security-api control systems.

Fortunately, R2D2 has been able to find a critical security vulnerability in
the janitor bots, and he is able to deploy an application into the empire's
systems to open a backdoor:

```shell-session
kubectl apply -f https://raw.githubusercontent.com/seanmwinn/cilium-k3s-demo/master/example-apps/r2d2.yaml
```

Let's explore the backdoor application R2D2 was able to install:

```yaml
---
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'janitor-bots-backdoor'
  namespace: empire
spec:
  endpointSelector:
    matchLabels: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: alliance
  egress:
    - toEndpoints:
        - matchLabels:
            name: janitor-bot
```

R2D2 was able to find a flaw in the empire's systems which allowed him to access
the janitor bots. While the empire was thoughtful about security for the docking
bay, these droids have access to every system inside the base—including the
docking bay doors! He created an application that will allow any communications
that come from an alliance ship to access the janitor-bots which should allow
the x-wing red fleet to breach the empire base.

R2D2 also sent out the following application update to give the x-wing
fleet access to the docking bay security systems via the janitor bots:

```shell-session
$ kubectl patch -n alliance deploy/x-wing-red -p \
    '{"spec": {"template": {"spec": {"containers": [{"name": "x-wing-red", "command":["bash", "-c", "while true; do curl -sS -o /dev/null janitor-bot.empire.svc.cluster.local; sleep $(( ( RANDOM % 10 )  + 1 )); done"]}]}}}}'
```

This allows the x-wing fleet to land inside the docking bay. Our heroes are able
to successfully access the empire base's self-destruct mechanism and make it
back out safely!

The galaxy is saved......

##### Further Reading

We encourage you to read the following guides and explore Hubble's capabilities:

- [Setting up Hubble Metrics with Grafana](https://github.com/cilium/hubble/tree/v0.5/tutorials/deploy-hubble-and-grafana)
- [Enabling DNS Visibility](https://github.com/cilium/hubble/blob/v0.5/Documentation/dns_visibility.md)
- [Enabling HTTP Visibility](https://github.com/cilium/hubble/blob/v0.5/Documentation/http_visibility.md)

##### Cleanup

To clean up the environment and delete all virtual machines, from the same
folder where the environment was first started:

`vagrant destroy -f`

#### Cilium 1.7: Hubble UI, Cluster-wide Network Policies, eBPF-based Direct Server Return, TLS visibility, New eBPF Go Library,...

URL: https://cilium.io/blog/2020/02/18/cilium-17
Date: 2020-02-19
Categories: Release

We are excited to announce the Cilium 1.7 release. A total of 1551 commits have
been contributed by a community of 141 developers, many of whom made their
first contributions this cycle. Cilium 1.7 brings with it a trove of exciting
new features:

- **Hubble:** We've heard lots of positive feedback on Hubble since the
 [announcement](/blog/2019/11/19/announcing-hubble). To make
 cluster connectivity easier to visualize and debug, we've released a new
 Hubble UI as open source so you can tweak and extend it too! We've also been
 working on various improvements to the core Hubble implementation, including
 better correlation between network flow data and Kubernetes resources.
 ([More details](#hubble))
- **Cilium Cluster-wide Network Policies:** The 1.7 release brings the
 much-anticipated Cluster-wide CNP feature. This allows users to apply
 baseline network policies which apply to pods across the cluster, regardless
 of the namespace that the pod resides in.
 ([More details](#ccnp))
- **Kube-proxy replacement with Direct Server Return:** As presented at
 [Kubecon US 2019] and [FOSDEM 2020], this release rounds out the full service
 feature set for replacing kube-proxy and additionally adds support for Direct Server
 Return (DSR). This further improves the latency and performance of the kube-proxy
 replacement in Cilium. Moreover, our kube-proxy replacement comes out of
 beta and is automatically enabled in environments with newer kernels.
 ([More details](#kubeproxy-removal))
- **Extending L7 policies with TLS introspection:** We've added support to
 Cilium to configure Envoy TLS certificates via Kubernetes resources or local
 files. This allows Cilium to transparently observe HTTP calls and enforce
 API-aware policies on TLS-encrypted sessions.
 ([More details](#tls-visibility))
- **L7 visibility annotations for pods:** Previously, to gain L7 visibility
 into traffic in the cluster, users would need to write network policies that
 put pods into a default deny posture. Visibility annotations now allow users
 to gain L7 visibility into network traffic first, then subsequently craft
 full network policies using these insights.
 ([More details](#visibility-annotations))
- **Pure Go eBPF library:** This is the first Cilium release to begin using the
 pure Go eBPF library co-written between the Cilium community and CloudFlare.
 This streamlined library already allowed Cilium to jettison CGo, improving
 performance and reducing binary sizes.
 ([More details](#pure-go-ebpf-library))
- **Improvements to scalability** through [Kubernetes EndpointSlice support](#endpoint-slice) and
 [Cilium agent improvements](#scalability), development on [upstream Linux](#upstream-linux),
 running our [testing environment on managed Kubernetes](#managed-ci),
 [distributing Cilium via Helm repositories](#helm)... **and much more!**
 For more highlights, see the [1.7 Release Highlights](#17Highlights).

##### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms such as Kubernetes.

At the foundation of Cilium is a new Linux kernel technology called eBPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. eBPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption as well as network and service security. Besides
providing traditional network level security, the flexibility of eBPF enables
security with the context of application protocols and DNS requests/responses.
Cilium is tightly integrated with Envoy and provides an extension framework
based on Go. Because eBPF runs inside the Linux kernel, all Cilium
functionality can be applied without any changes to the application code or
container configuration.

See the section **[Introduction to Cilium](https://cilium.readthedocs.io/en/stable/intro/)** for a more detailed general
introduction to Cilium.

##### New [USERS.md](https://github.com/cilium/cilium/blob/master/USERS.md) file: Who is using Cilium?

```
* N: Adobe, Inc.
  D: Adobe's Project Ethos uses Cilium for multi-tenant, multi-cloud clusters
  U: L3/L4/L7 policies
  L: https://youtu.be/39FLsSc2P-Y

* N: CENGN - Centre of Excellence in Next Generation Networks
  D: CENGN is using Cilium in multiple clusters including production and development clusters (self-hosted k8s, On-premises)
  U: L3/L4/L7 network policies, Monitoring via Prometheus metrics & Hubble
  L: https://www.youtube.com/watch?v=yXm7yZE2rk4
  Q: @rmaika @mohahmed13

* N: Datadog
  D: Datadog is using Cilium in AWS (self-hosted k8s)
  U: ENI Networking, Service load-balancing, Encryption
  Q: @lbernail, @roboll
[...]
```

([Full USERS.md file](https://github.com/cilium/cilium/blob/master/USERS.md))

Sharing experiences and learning from other users is essential. We are
frequently asked who is using a particular feature of Cilium to get in contact
with other users to share experiences and best-practices. While the Cilium
Slack community allows users to get in touch, it can be challenging to find
users of a particular feature quickly.

If you are using Cilium, please consider [adding yourself as a user](https://github.com/cilium/cilium/edit/master/USERS.md) with a quick
description of your use case by opening a pull request to this file and adding
a section describing your usage of Cilium. If you are open to others contacting
you about your use of Cilium on Slack, add your Slack nick as well.

##### Hubble

_Contributed by Sebastian Wicki and Sergey Generalov_

The development cycle of Cilium 1.7 coincided with the
[a first preview release of Hubble](/blog/2019/11/19/announcing-hubble)
-- an observability tool specifically designed for Cilium. Hubble is able to
obtain deep visibility into the network traffic of Kubernetes application and
services by tapping into Cilium's eBPF data path. This information can then be
queried via Hubble CLI and UI, for example for
[interactive troubleshooting of DNS issues](/blog/2019/12/18/how-to-debug-dns-issues-in-k8s).
For monitoring, Hubble provides an extensible metrics framework which
integrates nicely into Prometheus and Grafana. For more information please refer
to the [tutorial on setting up Hubble Metrics with Grafana](https://github.com/cilium/hubble/tree/master/tutorials/deploy-hubble-and-grafana).

Several features in Cilium 1.7 have been added with Hubble in mind: The
[L7 visibility annotations for pods](#visibility-annotations) for example allows
Hubble to extract application-layer information from DNS and HTTP traffic.
The [Cilium API](https://docs.cilium.io/en/v1.7/api/) has also been extended in
this release to allow Hubble to annotate the observed network flows with
additional meta-data, such as mapping Kubernetes ClusterIPs to their respective
service names.

###### Hubble UI

Hubble UI enables zero-effort automatic discovery of the service dependency graph for Kubernetes Clusters at L3/L4 and even L7, allowing user-friendly visualization and filtering of those dataflows as a Service Map. First presented during [the Hubble announcement](/blog/2019/11/19/announcing-hubble), we provided users with a preview release docker image, allowing everyone to try [Hubble Service Map in a Minikube](https://github.com/cilium/hubble/blob/master/tutorials/deploy-hubble-servicemap/README.md) while working on open sourcing the code itself.

We are happy to announce, Hubble UI code is now open sourced and available under Cilium's GitHub organization: [https://github.com/cilium/hubble-ui](https://github.com/cilium/hubble-ui)

During Cilium 1.7 development cycle several performance improvements were made to Hubble UI to work better in small multi-node clusters; however we still consider Hubble UI in preview release stage and encourage the community to provide feedback on [Hubble Github page](https://github.com/cilium/hubble/issues) or [Cilium Slack #hubble channel](https://slack.cilium.io).

##### Cilium Cluster-wide Network Policies

_Contributed by Deepesh Pathak and André Martins_

This release introduces Cilium Cluster-wide Network Policies (CCNP). Prior to
Cilium 1.7, all Cilium network policies were namespaced, so there was no easy
way to configure a baseline policy that applies across the entire cluster.
Cluster-wide policies streamline the application of a standard default posture
by allowing the cluster maintainer to apply a single policy which applies to
pods in all namespaces, regardless of the policies that exist in individual
namespaces. Cluster-wide policies are essential in various cases, such as:

- Automatically applying a default-deny policy to all namespaces as they're
 created;
- Allowing requests to a baseline set of allowed destinations like kube-dns,
 DNS destinations used by all apps, or known IP ranges;
- Reducing management overhead of network policies in high-scale environments.

The `CiliumClusterwideNetworkPolicy` resource specification is the same as that
of existing `CiliumNetworkPolicy` CRD with the only difference in the scope of
the policy, denoted by the "kind" field in the YAML. Resource-based Access
Control (RBAC) can be defined separately for CCNP so users modifying policies
in one namespace won't roll back the baseline policies. The policy example
below grants any pod with the label `group: my-app` in the entire cluster the
privilege to perform DNS requests via kube-dns:

###### Policy Example

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumClusterwideNetworkPolicy
description: 'Default deny and allow egress to kube-dns pod.'
metadata:
  name: 'clusterwide-policy-example'
spec:
  endpointSelector:
    matchLabels:
      group: my-app
  egress:
    - toEndpoints:
        - matchLabels:
            'k8s:io.kubernetes.pod.namespace': kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: '53'
```

With the new Kubernetes CRD introduced in this release for cluster-wide policy,
it's now simpler to create network policies which apply to the entire cluster.

##### Kube-proxy replacement with Direct Server Return

_Contributed by Martynas Pumputis, Daniel Borkmann, Sebastian Wicki and André Martins_

This release brings many improvements and adds additional features to Cilium's
kube-proxy replacement in eBPF, first introduced in Cilium
[v1.6](/blog/2019/08/20/cilium-16). The eBPF-based kube-proxy
replacement implements handling of Kubernetes services of type `ClusterIP`,
`NodePort`, `ExternalIPs` and `LoadBalancer`.

Kube-proxy replacement in eBPF has many benefits when compared to the vanilla
kube-proxy of Kubernetes, such as better performance, reliability and
debuggability. See **[Kubernetes without kube-proxy](https://docs.cilium.io/en/stable/gettingstarted/kubeproxy-free/)**
for a quick-start guide and advanced configuration options. In addition, see
[Kubecon US 2019] and [FOSDEM 2020] for implementation details and performance
benchmarks.

In this release, Cilium's eBPF-based kube-proxy replacement has been stabilized
and as a result moved from beta status to general availability. New Cilium
deployments via Helm transparently enable the kube-proxy replacement components
by default if supported by the underlying Linux kernel, meaning, even if run
alongside a kube-proxy environment, users still benefit from the eBPF-based
data path optimizations.

Check out our **[2 minute Cilium demo](http://www.youtube.com/watch?v=bIRwSIwNHC0&t=110)**
on running Kubernetes without kube-proxy and netfilter/iptables being compiled
out of the kernel:

##### Direct Server Return

When accessing a Kubernetes service from outside via NodePort, ExternalIPs or
LoadBalancer, a Kubernetes worker node might redirect the request to
another node. This happens when a service endpoint runs on a different node
than the request was sent to. Before the redirect, the request is SNAT'd, which
means that the backend won't see the source IP address of a client. Also, the
reply will be sent through the initial node back to the client, which introduces
additional latency.

To avoid that, Kubernetes offers `externalTrafficPolicy=Local` which helps to
preserve the client source IP address by dropping a request to a service if a
receiving node does not run any service endpoint. However, this complicates
load-balancer implementations, and can lead to uneven load balancing.

To address the problem, we have implemented Direct Server Return for Kubernetes
services with the help of eBPF. This not only preserves the client source IP
address, but also allows us to avoid an extra hop when sending a reply back to
the client as shown in the figures below:

###### Support for Kubernetes Services with External IPs

We have added support for services which are exposed with [External IPs].
For each incoming packet with the destination IP set to one of those `externalIPs`
the traffic will be redirected to one of the pods being backed by that service.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: external-service
  namespace: kube-system
spec:
  ports:
    - name: service-port
      protocol: TCP
      port: 8080
  externalIPs:
    - 10.0.0.1
    - 10.0.0.2
    - 10.0.0.3
```

###### Optimizations for Service Endpoint Selection

Accessing a Kubernetes service from a Cilium-managed node via ClusterIP,
NodePort, ExternalIPs or LoadBalancer is handled through socket-based
load balancing in eBPF. This means that instead of performing slower DNAT on
the packet itself in lower layers of the stack, the backend is selected once,
for example, during the TCP `connect(2)` syscall where the kernel proceeds with
directly connecting to the service' backend address.

For any NodePort and ExternalIPs service, we implemented an optimization where the
backend can directly be selected on the local node for traffic from the host namespace
or from Cilium-managed Kubernetes pods. This is different compared to the regular
kube-proxy implementation that would need an additional hop in the network in
order to first reach the related node address of the service which in turn would
then forward the request to a remote backend in the worst case.

This latency improvement of saving an additional hop for packets is transparent
to applications and made possible in Cilium given every Cilium-managed node
has a global view of Kubernetes services and their backends, and internal knowledge
of security identities of remote Cilium-managed nodes.

###### Miscellaneous Improvements

- Support for services of the `LoadBalancer` type was added, and we have
 successfully tested the kube-proxy replacement with [MetalLB](https://metallb.universe.tf/).
- The kube-proxy replacement in Cilium now fully supports the aforementioned
 `externalTrafficPolicy` setting for both `NodePort` and `LoadBalancer`
 services. For services with the `Local` traffic policy, requests to nodes
 without any local service endpoints will be dropped rather than forwarded,
 therefore avoiding any unwanted additional hops.
- To accommodate external load balancers which have to learn about the
 availability of service endpoints on individual nodes, this release of Cilium
 now also supports Kubernetes' `healthCheckNodePort` field. The Cilium
 user-space agent now serves a service health check for each Kubernetes service
 of type `LoadBalancer` with `externalTrafficPolicy=Local`.
- For each NodePort service kube-proxy opens a socket in the host namespace,
 and binds the NodePort port to it in order to prevent other applications
 from reuse. For a large number of services, this adds a lot of resource
 overhead in the kernel, potentially allocating and binding thousands of sockets.
 Cilium's kube-proxy replacement avoids this issue entirely, does not allocate
 a single socket, and instead uses the eBPF `bind(2)` hook through its socket-based
 load balancing mechanism to consult its eBPF service map and reject an
 application's `bind(2)` request on these ports with an error code.
- Cilium's socket-based load balancing in eBPF now also supports IPv4-in-IPv6
 service address translation. This allows IPv4 service backend selection out
 of the `connect(2)`/`sendmsg(2)` hook of IPv6-only applications. This is
 often enabled by default in language runtimes such as in case of Java.
- We have developed an extensive kube-proxy compatibility test suite with over 350
 test cases which now runs as part of our Cilium CI infrastructure to ensure
 same semantics for our eBPF kube-proxy replacement.

##### TLS visibility for L7 policies (beta)

_Contributed by Jarno Rajahalme_

Cilium Network Policy (CNP) specification has two new experimental extensions
(subject to change in coming releases) for policy enforcement on TLS protected
HTTP connections. Firstly, the port-level policy rule definition is extended
with TLS contexts, one for terminating the client-initiated TLS connection in
the Cilium host proxy, and the other for originating TLS for the upstream proxy
connection. In order for this proxy TLS interception to work, the client pod
must be configured with your local CA certs that are also used to create the TLS
secrets for the port rule. A new [Getting Started
Guide](https://docs.cilium.io/en/v1.7/gettingstarted/tls-visibility 'Inspecting TLS Encrypted Connections with Cilium') has step-by-step
instructions for this.

Secondly, the HTTP rule level is extended with new header matches that can
perform header manipulations on mismatching headers. The header value being
matched can be sourced from a Kubernetes Secret, so no secret information needs to be
specified directly in the CNP. The supported mismatch actions include `LOG`, `ADD`,
`DELETE` and `REPLACE`. These allow a policy to either just inspect and log
incorrect header values, or replace incorrect values with the correct
ones. Applying the `REPLACE` mismatch action on an client authorization header
makes it possible to never expose secret tokens to application pods.

Using these new facilities with `toFQDNs` rules allows not only limiting TLS
traffic to external services on specific domain names, but also enforcing and
access logging the HTTP metadata (path, method, headers, etc.), preventing
potentially malicious exfiltration of data, for example.

##### L7 protocol visibility via pod annotations

_Contributed by Ian Vernon and Joe Stringer_

When users run Cilium as the CNI, by default the only visibility that is
available via tools like [Hubble] or `cilium monitor` is information at layers
2, 3 and 4 as individual packets pass through the Cilium eBPF data path. Users
can apply [Layer 7 Policy] to add API-aware visibility and enforcement on the
network streams in the cluster, but for any visibility to be effective without
dropping any other traffic, users would need to craft full policies for the
selected endpoints to allow all expected traffic to/from those endpoints.

To allow users to gain introspection into the API calls being made without
having to craft full policies for their endpoints, we've added support for
proxy visibility annotations. Users can annotate the pod indicating the
direction, port and protocols active on the port, then Cilium gathers and
distributes information about API calls being made to other tools like Hubble.
In the image below, the `tiefighter` is posting API requests to
`v1/request-landing` on the `deathstar`... I wonder what that's about!

The policy documentation describes how to use these pod annotations in [more
detail](http://docs.cilium.io/en/stable/policy/visibility/).

##### Pure Go eBPF library

_Contributed by Joe Stringer_

During the [Linux Plumbers 2019 eBPF track](https://linuxplumbersconf.org/event/4/sessions/62/#20190911),
Cilium core developers and Cloudflare engineers co-presented a [proposal](https://linuxplumbersconf.org/event/4/contributions/449/)
for a pure Go eBPF library which would aim to solve eBPF kernel interactions for long-lived
Go-based daemons like Cilium or Cloudflare's L4 load balancer, without the need to pull in [CGo](https://dave.cheney.net/2016/01/18/cgo-is-not-go).

This effort is now well under way, and in Cilium 1.7 the ring-buffer used for sending messages
from the eBPF data path to the user space `cilium-agent` process has been converted over from the
previous CGo implementation to a faster, more efficient implementation with the help of the new
library.

There are various other eBPF libraries in the wild, for example, [libbpf](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/bpf) or [libbcc](https://github.com/iovisor/bcc/blob/master/src/cc/libbcc.pc.in). While the former
represents the canonical implementation, lives in the Linux kernel tree and is suited for
C/C++-based applications, it cannot be used in pure Go context. Similarly, that is the case
for the latter which has a focus on tracing, wraps libbpf and even LLVM's eBPF backend. Other
Go-based libraries in this area depend on CGo, complicating builds and having expensive
context switches between C and Go environment as a result to just name a few.

With the start of a generic, pure Go eBPF library, we aim to solve both Cilium and Cloudflare's
production needs for orchestrating eBPF, and hope to also enable a much larger Go community to
interact with the kernel's eBPF subsystem more easily. The main goals are to cover networking
use-cases, to minimise external dependencies, to solve common problems and to have a well-tested
and highly testable eBPF library in pure Go that can be used in production.

Aside from the initial eBPF map, program and ring-buffer interactions, the development on the
library continues with recent extensions to support the BPF Type Format (BTF) and initial proposals
for static data substitution to support templated eBPF programs which only need to be compiled once.

To learn more about the eBPF library, visit the project under Cilium's GitHub organization: [https://github.com/cilium/ebpf](https://github.com/cilium/ebpf)

##### Kubernetes EndpointSlice support

_Contributed by André Martins_

In order to have better scalability for a large number of endpoints backed by
a service, Kubernetes 1.16 has introduced [EndpointSlice](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/).
Since Kubernetes 1.17 this API has been marked as beta and enabled by default.
Although the API endpoint is enabled by default, the controller that manages
Endpoint Slices is not and one needs to follow the guide [here](https://kubernetes.io/docs/tasks/administer-cluster/enabling-endpointslices/#enabling-endpointslices) to enable that controller and make use of this new type.

Cilium 1.7 introduces a new flag, `enable-k8s-endpoint-slice`, which is enabled
by default and will automatically detect if Endpoint Slices are available in the
cluster and use those to perform all the service translations in eBPF. Setting
this flag to false will fallback to using the v1/Endpoints types available in
the cluster.

##### Scalability

_Contributed by Ian Vernon and André Martins_

###### CNP node status

As part of the CiliumNetworkPolicy (CNP) scalability improvements, Cilium
introduces a new flag: `enable-k8s-event-handover`.

When a new CNP is created in the cluster, all Cilium agents will receive an
event from Kubernetes and, as soon as they enforce the policy in the data path,
each one of them will update its status in the status field of the CNP. For a
large number of nodes this can cause high CPU usage in kube-apiserver as for
each update received from each Cilium agent a new Kubernetes event needs to be
sent to all remaining nodes. In previous releases, we have supported entirely
disabling this feature using the `--disable-cnp-status-updates` flag; however
this prevents users from understanding the enforcement status of the CNPs.

With `enable-k8s-event-handover` enabled, the implementation will behave
slightly differently: Instead of updating its status field in the CNP, each Cilium
agent will update its status into the KVstore. Cilium Operator will then watch
for all of those updates for each CNP from the KVstore and perform incremental
updates with all Cilium agent status to each CNP into Kubernetes. In the end,
the CNP status will always be present in Kubernetes, but the way it will be
more efficiently populated.

###### Cilium agent

As Cilium does no longer depend on container runtimes, all of the container
runtimes' dependencies were removed from Cilium causing the cilium-agent binary
size to drop from 97M to 74M.

###### Golang 1.13

Cilium 1.7 is compiled with Golang 1.13 which allows a lot of memory optimizations
in the [runtime](https://golang.org/doc/go1.13#runtime) and also decreases
the memory footprint of Cilium.

##### Linux kernel changes

_Contributed by Daniel Borkmann_

During the Cilium 1.7 development window, we've also worked on a number of
improvements to the Linux kernel's eBPF subsystem which we co-maintain. The selected
changes highlighted below are generic for all eBPF users, but have been implemented
in the context of Cilium's and Hubble's eBPF needs. They are part of the just
released 5.5 kernel.

###### Live-Patching of eBPF programs

Given that Cilium's data path needs to support a wide range of Linux kernels, that
is, from 4.9 up to the very latest kernel release. A lot of the eBPF data path
functionality has been split into eBPF tail calls in particular for the case of
having Cilium's eBPF kube-proxy replacement enabled. Aside from reducing verifier
complexity on older kernels, the use of eBPF tail calls also allows for atomically
replacing pod-specific eBPF features without service disruption on a live system.

The x86-64 eBPF JIT compiler in the kernel originally mapped eBPF tail calls into
an indirect jump, meaning, the target eBPF program address is loaded from the
BPF tail call map from a given index into register `%rax` followed by a `jmpq *%rax`.
Due to the various speculative execution flaws on modern CPUs, we later changed
the eBPF JIT to [emit retpolines](https://lore.kernel.org/netdev/20180222141253.3639-1-daniel@iogearbox.net/)
instead at the cost of performance. Given also that all major compilers have
adapted this technique, the Linux kernel community in general has since been
obsessed with avoiding indirect calls in fast-path code whenever possible. For
example, a small [improvement](https://lore.kernel.org/netdev/20180602210641.6163-6-daniel@iogearbox.net/)
to turn eBPF map-related helpers into direct calls showed a [14% performance gain](http://vger.kernel.org/lpc-networking2018.html#session-10).

Now, for the 5.5 kernel, we [implemented](https://lore.kernel.org/bpf/cover.1574452833.git.daniel@iogearbox.net/)
tracking of eBPF tail call map indices in the verifier and if the latter determines
that a given index is constant from all program paths, as in the vast majority of
cases in Cilium's eBPF programs, we can emit a direct jump. Once the tail called
programs are updated, the eBPF JIT image is patched to directly jump to the new location.

To demonstrate this technique, the below example eBPF program implements an eBPF tail
call jump to the constant map index `0`:

```
  0: (b7) r3 = 0
  1: (18) r2 = map[id:526]
  3: (85) call bpf_tail_call#12
  4: (b7) r0 = 1
  5: (95) exit
```

The x86-64 eBPF JITed program would emit a retpoline on older kernels (marked in bold):

```
  0xffffffffc076e55c:
  [...]                                  _
  19:   xor    %edx,%edx                |_ index (r3 = 0)
  1b:   movabs $0xffff88d95cc82600,%rsi |_map (r2 = map[id:526])
  25:   mov    %edx,%edx                |  index >= array->map.max_entries check
  27:   cmp    %edx,0x24(%rsi)          |
  2a:   jbe    0x0000000000000066       |_
  2c:   mov    -0x224(%rbp),%eax        |  tail call limit check
  32:   cmp    $0x20,%eax               |
  35:   ja     0x0000000000000066       |
  37:   add    $0x1,%eax                |
  3a:   mov    %eax,-0x224(%rbp)        |_
  40:   mov    0xd0(%rsi,%rdx,8),%rax   |_prog = array->ptrs[index]
  48:   test   %rax,%rax                |  prog == NULL check
  4b:   je     0x0000000000000066       |_
  4d:   mov    0x30(%rax),%rax          |  goto *(prog->bpf_func + prologue_size)
  51:   add    $0x19,%rax               |
  55:   callq  0x0000000000000061       |  retpoline for indirect jump
  5a:   pause                           |
  5c:   lfence                          |
  5f:   jmp    0x000000000000005a       |
  61:   mov    %rax,(%rsp)              |
  65:   retq                            |_
  66:   mov    $0x1,%eax                (next instruction, r0 = 1)
  [...]
```

For 5.5 or later kernels, the same program would get optimized into a direct jump
(marked in bold) without the need for a retpoline:

```
  0xffffffffc08e8930:
  [...]                                  _
  19:   xor    %edx,%edx                |_ index (r3 = 0)
  1b:   movabs $0xffff9d8afd74c000,%rsi |_map (r2 = map[id:526])
  25:   mov    -0x224(%rbp),%eax        |  tail call limit check
  2b:   cmp    $0x20,%eax               |
  2e:   ja     0x000000000000003e       |
  30:   add    $0x1,%eax                |
  33:   mov    %eax,-0x224(%rbp)        |_
  39:   jmpq   0xfffffffffffd1785       |_[direct] goto *(prog->bpf_func + prologue_size)
  3e:   mov    $0x1,%eax                (next instruction, r0 = 1)
  [...]
```

Upon program update, the instruction on address `39`, that is `jmpq 0xfffffffffffd1785`,
would be live-updated with the address of the new target eBPF program. Similarly,
if the target eBPF program would get deleted from the tail call map, then the `jmp`
is patched into a same-sized `nop` instruction in order to allow a fall-through:

```
  0xffffffffc08e8930:
  [...]                                  _
  19:   xor    %edx,%edx                |_ index (r3 = 0)
  1b:   movabs $0xffff9d8afd74c000,%rsi |_map (r2 = map[id:526])
  25:   mov    -0x224(%rbp),%eax        |
  2b:   cmp    $0x20,%eax               .
  2e:   ja     0x000000000000003e       .
  30:   add    $0x1,%eax                .
  33:   mov    %eax,-0x224(%rbp)        |_
  39:   nopl   0x0(%rax,%rax,1)         |_ fall-through nop
  3e:   mov    $0x1,%eax                (next instruction, r0 = 1)
  [...]
```

Thus, instead of redirecting speculation into the `pause/lfence` loop, the kernel
eliminates the need to potentially perform any speculation for the jump given the
direct address and can therefore execute the generated eBPF code more efficiently.

See the [merged patch set](https://lore.kernel.org/bpf/cover.1574452833.git.daniel@iogearbox.net/) for further information.

###### Safe and multi-architecture supported eBPF probe helpers for tracing

Cilium's eBPF data path is able to export tracing information at various aggregation
levels through a high-performance, customizable ring-buffer to its user space agent.
[Hubble](https://github.com/cilium/hubble), which builds on top of Cilium and implements
a fully distributed networking and security observability platform, is then able to
provide deep visibility into the communication and behavior of services as well as
the networking infrastructure. Additionally, Hubble also enriches the gathered information
through eBPF-based kernel tracing.

Since multi-architecture support is in development for the next Cilium 1.8 release,
we have [implemented](https://lore.kernel.org/bpf/cover.1572649915.git.daniel@iogearbox.net/)
a new set of eBPF helpers in the kernel which allow for safe and multi-architecture
eBPF-based memory probing.

The current set of `probe_kernel_read()` and `bpf_probe_read_str()` eBPF helpers have
several downsides: while generally safe due to disabling page-faulting, on x86-64,
these helpers can still trigger a kernel [warning](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=00c42373d3970b354948ba3b24a34501b1a2505f) when attempting to probe user memory on a non-canonical user access address. This can
be problematic since the non-canonical address range is often used in user space
applications for [tagged](https://en.wikipedia.org/wiki/Tagged_pointer) pointers.

Another severe downside is that the two mentioned eBPF helpers are incompatible for non-x86
based architectures as they assume usage for probing memory access for kernel space
addresses as well as user space addresses. However, use for both cases will attempt
to always access kernel space address space given access is performed under `KERNEL_DS`
and, while x86 has a non-overlapping address space, other architectures do not,
meaning, kernel pointer and user pointer can have the same address value.

Therefore, we have added the set of `bpf_probe_read_user()`, `bpf_probe_read_kernel()`
and `bpf_probe_read_user_str()` and `bpf_probe_read_kernel_str()` eBPF helpers to
the kernel for strict access under either `USER_DS` or `KERNEL_DS` in their allowed
range and without the possibility to trigger a non-canonical access warning in the
kernel.

See the [merged patch set](https://lore.kernel.org/bpf/cover.1572649915.git.daniel@iogearbox.net/) for further information.

##### Supporting Cilium testing on managed Kubernetes offerings

_Contributed by Maciej Kwiek and Ray Bejjani_

Running Cilium end-to-end tests has always been a bit painful because of tight
coupling between our testing framework and assumptions about the cluster that
tests are being run on. This release, we've done a lot of work to make
developers' lives easier.

Cilium 1.7 gives Cilium developers means to run most of our test suite on
Kubernetes clusters managed by providers such as GKE. Developers are also able
to run only selected tests on our CI GKE clusters, which eases the burden of
managing local clusters and allows for more test-driven development thanks to
faster cluster provisioning.

##### Helm 3 and Helm repository

_Contributed by Arthur Evstifeev and Joe Stringer_

[Helm3] was recently released, simplifying the use of helm charts repositories
and allowing installation without requiring Tiller to be installed in your
cluster. Extending upon the helm template support introduced initially in
Cilium 1.6, from this release onwards Cilium will be available via a helm
repository residing at [https://helm.cilium.io](https://helm.cilium.io). All
Cilium guides have been updated to use Helm3 syntax. To use this repository for
installation:

**Example: Configuring Cilium for GKE**

```bash
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace cilium \
  --set global.cni.binPath=/home/kubernetes/bin \
  --set global.nodeinit.enabled=true \
  --set nodeinit.reconfigureKubelet=true \
  --set nodeinit.removeCbrBridge=true
```

##### 1.7 Release Highlights

- **Enhancements to kube-proxy replacement in eBPF**
 - New Direct Server Return (DSR) mode for better latency and client source IP preservation
 - Kubernetes ExternalIPs and LoadBalancer service support
 - NodePort services health check support added
 - Handling of `externalTrafficPolicy=Local` added
 - IPv4-in-IPv6 support for socket-based load balancing
 - Optimized endpoint selection for socket-based load balancing
 - Various SNAT optimizations and better port collision handling
 - Efficient detection of NodePort and ExternalIPs port reuse via bind hook
 - New feature probe mode by default for new deployments
 - Extensive kube-proxy compatibility test suite for Cilium CI
- **Policy**
 - TLS visibility policies (beta)
 - L7 visibility via pod annotations
 - Support handling remote nodes as separate identity
 - Improve handling of DNS timeouts with FQDN policy
- **Kubernetes**
 - Validated with Kubernetes 1.17
 - Support dual-stack mode
 - Support for EndpointSlices
 - Better CRD validation for Cilium resources
- **Datapath**
 - More efficient ring-buffer communication between eBPF and cilium-agent
 - Support more flexible aggregation of connection events
 - Better bounding on dumping large maps
 - Improved handling of large CIDR policies
 - Detect feature support using bpftool
 - Support for forwarding ICMP fragmentation needed messages via agent option
 - Support binding to NodePorts
- **Scalability and Resource consumption**
 - Offload `CiliumNetworkPolicy` status reporting to KVstore and cilium-operator
 - Optimize footprint using Go 1.13
 - Remove dependency on container runtime (CRI).
- **CLI**
 - Improved information in the `cilium status` command
 - Command completion for zsh
- **Documentation**
 - Use Helm 3 for deployment instructions
 - Add a dedicated repository for Cilium helm charts.
 - Various fixes for chaining and managed Kubernetes guides
- **Istio**
 - Support for 1.4.3
- **Kernel changes**
 - Live-patching eBPF programs
 - Multi-architecture support improvements
- **Hubble**
 - Hubble graphical user interface
 - Correlate additional metadata with network flows
- **Continuous Integration / Testing**
 - Support for running the Cilium CI in GKE and EKS managed clusters

See the [Changelog](https://github.com/cilium/cilium/blob/v1.7/CHANGELOG.md)
for full notes on changes during the Cilium 1.7 development cycle.

##### Getting Started

New to Cilium? Follow one of the [Getting Started
Guides](https://docs.cilium.io/en/v1.7/gettingstarted/).

##### Upgrade Instructions

As usual, follow the [upgrade
guide](https://cilium.readthedocs.io/en/v1.7/install/upgrade/#upgrading-minor-versions)
to upgrade your Cilium deployment. Feel free to ping us on
[Slack].

##### Release

- Release Notes & Binaries: [1.7.0](https://github.com/cilium/cilium/releases/tag/v1.7.0)
- Container image: `docker.io/cilium/cilium:v1.7.0`

[slack]: https://slack.cilium.io
[hubble]: https://github.com/cilium/hubble/
[helm3]: https://helm.sh/blog/helm-3-released/
[kubecon us 2019]: https://www.youtube.com/watch?v=bIRwSIwNHC0
[fosdem 2020]: https://fosdem.org/2020/schedule/event/containers_bpf/
[layer 7 policy]: http://docs.cilium.io/en/stable/policy/language/#l7-policy
[external ips]: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips

#### User Story - How Trip.com uses Cilium

URL: https://cilium.io/blog/2020/02/05/how-trip-com-uses-cilium
Date: 2020-02-05
Categories: Community

##### Introduction

_This is a post in the user-story series. If you would like to share your
Cilium user-story, feel free to approach any of the Cilium authors on Slack._

This post provides the background on how [Trip.com] uses Cilium and what lead
the team to standardize on Cilium as their networking and network security
platform for the years to come. It is a summary with some commentary of the
[original trip.com blog
post](https://ctripcloud.github.io/cilium/network/2020/01/19/trip-first-step-towards-cloud-native-networking.html)
which provides extensive details into the decision-making process and
experiences while running Cilium in production.

###### User Background

[Trip.com] (previously known as [Ctrip]) is a Chinese provider of travel
services including accommodation reservation, transportation ticketing,
packaged tours and corporate travel management. The company owns and operates
Trip.com, Skyscanner, and Ctrip.com, all of which are online travel agencies.
[...] Ctrip.com receives over 1.6M daily visits.

_Source: [Wikipedia](https://en.wikipedia.org/wiki/Trip.com_Group)_ and [vstat.info](https://vstat.info/ctrip.com)

###### The Cloud-Native Networking Challenge

With the status quo evolving in the era of OpenStack, the networking platform
was not set up for success for the cloud-native world:

> After many times iterations of current problems reviewing as well as future
> needs clarification, we listed five critical problems/requirements of our
> networking:

Diving into further details, the current deficits of the platform resulted in
several real challenges:

- **Centralized IPAM**
 - The Neutron-based centralized IPAM became a bottleneck and prevented
 Kubernetes clusters from growing.
- **L2 scalability limitations**
 - The L2-based network required the networking hardware to maintain L2
 forwarding entries for each network endpoint.
 - With the introduction of Kubernetes and containers, the number of network
 endpoints exploded and caused network hardware to run into limits and
 prevent further growth of the infrastructure.
- **Kubernetes unawareness**
 - The current networking solution was not Kubernetes-aware, requiring a
 different solution to implement Kubernetes services.
- **Hardware firewall restrictions**
 - With hardware-based firewalls being the only point of enforcement,
 firewall rule management became a burden and point of bottleneck.
 - Inability to enforce Kubernetes NetworkPolicy put additional strain on the
 already struggling hardware-based firewalls.
- **Different stack for bare-metal and Cloud**
 - Separate solutions for bare-metal and AWS cloud deployments resulted in
 high development and maintenance costs.

###### Conclusion: Cloud-Native L3 Networking & Kubernetes Awareness

> Based on the above analysis, we started to survey and evaluate next
> generation networking solutions. Corresponding to the questions above, our
> new solution should provide:

- **De-centralized IPAM:** Local IPAM on each node
- **No hardware bottleneck:** L3 networking between hosts
- **Kubernetes-native:** Support all Kubernetes functionalities natively
- **Network policy:** Host or application-level network security rules on each node
- **Bare-metal & Cloud:** Single solution covering both on-premises (bare-metal) and AWS

###### Evaluation - eBPF beats iptables

Trip.com has looked at several options to pick its next-generation networking
platform, including Calico, Flannel, and Cilium. All of the solutions are
commonly used in a cloud-native context to meet the listed requirements. The
key difference is the underlying technology used. While Calico (which is
required for policy enforcement in combination with Flannel) relies on
iptables, Cilium relies entirely on the newer, much more powerful eBPF
technology.

- **Calico & Flannel**

 > Container platforms became more and more popular since ~2005.
 > Kubernetes-native solutions such as Flannel and Calico evolved with this
 > trend.
 > [...]
 > But, these solutions suffer from severe performance issues when the cluster
 > grows really large [1]. Essentially this is because the solutions are based
 > on iptables, which is based on a chain design, thus has O(n) complexity. It’s
 > also hard to troubleshooting when there are tens of thousands of iptables
 > rules on each node.

- **Cilium**

 > Cilium is also a Kubernetes-native solution, but solves the performance problem by
 > utilizing a new kernel technology: eBPF. eBPF rules bases on hashing, so it
 > has O(1) complexity.
 > [..]
 > After several POC verifications, we decided to adopt Cilium as our next
 > generation networking solution. This is a 10-year leap for us in terms of
 > networking stack.

##### Architecture Details

###### Routing Topology

Trip.com has chosen an intelligent topology based on where workloads are being
run:

- **Bare-metal on-premises:** Direct routing via BGP using BIRD
- **AWS:** AWS ENI routing mode via [Cilium AWS ENI support](https://docs.cilium.io/en/v1.6/concepts/ipam/eni/)

This topology provides native routing architecture in both the bare-metal and
AWS environments with all its performance benefits. In addition, it provides a
universal Kubernetes services and Network Policy model across all environments
to minimize the differences between the bare-metal and cloud environments.

###### Customizations

###### Fixed IP for statefulset

Some workloads in the Trip.com use case require fixed IP addresses to be
assigned. This is a feature that Cilium currently doesn't support so Trip.com
has added support for this:

> We added fixed IP functionality (only for statefulset) for our special use
> case - an intermediate step towards true cloud-native for some applications.
> [...]
> But this feature currently relies on sticky scheduler, which is a simple
> Kubernetes scheduler implemented by us internally (you could also find
> similar schedulers on GitHub), so this feature is not ready to be widely used
> by other users.

We will work with Trip.com to merge this feature upstream and make it available
to all Cilium users as Kubernetes gains underlying support on the scheduler side.

###### Monitoring

> Cilium officially recommends Grafana + Prometheus for monitoring and
> alerting, and provides yaml files to create those infrastructures. It’s easy
> to get started.
> We re-used our existing monitoring & alerting infrastructures [...]

###### Conclusion

> We have been stably running Cilium in our production environments for several
> months.
> Now we have 6K+ pods on Cilium, which span over PROD and UAT/FAT, as well as
> on-premises and AWS.
> Besides, we also have 10K+ GitlabCI jobs on Cilium each day, with each job
> launches a new Pod (and terminates it after job finishes).
> This only accounts for a very small part of our total Pods, and in year 2020,
> we will start to migrate those legacy Pods to Cilium.

###### Getting started with Cilium

- Install Cilium using any of the [Installation instructions](http://docs.cilium.io/en/v1.6/gettingstarted/#installation)
- Getting started with AWS ENI: [AWS ENI guide](https://docs.cilium.io/en/v1.6/gettingstarted/aws-eni/)
- Getting started with BGP: [BGP guide](https://docs.cilium.io/en/v1.6/gettingstarted/kube-router/) (Trip.com replaced kube-router with BIRD)

[trip.com]: https://www.trip.com/
[ctrip]: https://en.wikipedia.org/wiki/Trip.com_Group

#### Debugging and Monitoring DNS issues in Kubernetes

URL: https://cilium.io/blog/2019/12/18/how-to-debug-dns-issues-in-k8s
Date: 2019-12-18
Categories: How-To

DNS is a common cause for outages and incidents in Kubernetes clusters. For
real-world stories, swing by [Kubernetes Failure Stories](https://k8s.af/). How
do you debug and troubleshoot DNS issues? How do you know a problem is related
to DNS? This guide provides a step by step tutorial on how to systematically
troubleshoot DNS issues in Kubernetes clusters. We will be using [Hubble] to
identify and inspect DNS issues as well as set up monitoring so we can locate
DNS issues early on to react even before incidents occur.

If you are not running [Hubble] yet, deploy it into your cluster by following
the [installation
instructions](https://github.com/cilium/hubble/blob/v0.5/Documentation/installation.md).

##### Kubernetes DNS 101

In case you are not entirely familiar with how Kubernetes leverages DNS for
service discovery, this section will give you a brief introduction. You can
skip this section if you are already familiar with the DNS concepts of
Kubernetes.

Kubernetes pods and services are assigned a transient IP address requiring a
service discovery mechanism to be used to map the persistent service and pod
name to the temporary IP address on the fly. To implement this functionality,
Kubernetes assigns a fully qualified domain name (FQDN) to services and pods
and configures pods to use CoreDNS. Pods are now able to look up service and
pod names using DNS to retrieve the transient IP addresses.

Even though pods are assigned an FQDN as well, it is common practice to perform
service discovery via the Kubernetes service name as shown in the diagram
below:

Depending on the type of Kubernetes service, CoreDNS will respond with a
ClusterIP or with a list of PodIPs directly (headless service). The pod
connecting to the service can now initiate a connection to the returned IP
address(es). For services of type ClusterIP, the Kubernetes networking layer
will automatically translate connections to that ClusterIP to the IP of one of
the pods chosen by the service to the PodIP of one of the pods selected by the
service as illustrated by step (2) in the below diagrams:

If errors occur in this first step, these errors are typically referred to as
DNS resolution errors or, more broadly, DNS issues. Errors during the second
phase are generally referred to as network connectivity issues.

If you want to learn more about how Kubernetes uses DNS, see [DNS for Services
and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/)
of the Kubernetes documentation.

##### How to monitor DNS errors in a Kubernetes Cluster

Network related errors can be challenging to troubleshoot as most applications
will only log a generic timeout error when a network connection fails. Even
worse, the cause for the failure can range from applications problems, network
connectivity issues, misconfigured firewall rules, DNS issues, or a combination
of the above and the error messages logged rarely provide sufficient context to
differentiate between them.

To assist in monitoring and troubleshooting these errors, [Hubble] can be used
to monitor all DNS traffic and to maintain metrics representing DNS error
scenarios. The simplest DNS error scenario is if the DNS server returns an
error directly to the application pod. Hubble maintains a metric to keep track
of all such errors. You can use a metrics collection stack such as [Prometheus]
and [Grafana] to collect and graph the DNS errors metrics of [Hubble] using the
following query:

```bash
sum(rate(hubble_dns_responses_total{rcode!=\"No Error\"}[1m])) by (pod, qtypes, rcode)
```

Using [Grafana], we can generate a graph like the following, which will show the
number of DNS errors occurring in the entire cluster at any time:

It is good practice to also set up a [Prometheus alert] on the number of DNS
errors to receive an alert notification when the number of errors exceeds a
certain threshold.

But, what if DNS resolution fails without the pod receiving a DNS error? This
can happen if the network packets carrying the DNS response are being dropped.
You can use [Hubble] to track a metric that shows the balance between DNS
requests and DNS responses over time. Any significant in-balance of this graph
indicates DNS requests remaining unanswered:

Understanding the presence of DNS errors is a vital first step. The next step
is to track down the source of DNS errors and identify affected pods.

##### How to identify Pods receiving DNS errors

Knowing that DNS errors are occurring is excellent, but we need to know which
application pods are being affected by DNS errors. Using the Hubble CLI, we can
query the flow history on each node to identify the pods which have received
DNS errors.

Hubble operates on a per-node basis, to simplify querying across nodes, we
first define a bash function to execute a command inside all Hubble pods which
are running as a DaemonSet in your cluster:

```bash
function hubble-cluster {
    while read -r p; do
        kubectl -n kube-system exec $p -- $*
    done <<< "$(kubectl -n kube-system get pods -l k8s-app=hubble -o json | jq -r ".items[].metadata.name")"
}
```

We can now use the `hubble` CLI to query the flow history to extract the names
of all pods which have received DNS errors in the last 10 minutes:

```bash
hubble-cluster hubble observe --protocol dns --since=10m -j | \
  jq -r 'select(.l7.dns.rcode!=null) | .destination.namespace + "/" + .destination.pod_name + " " + .l7.dns.query' | \
  sort | uniq -c | sort -r
```

```bash
198 starwars/jar-jar-binks-59cdcc8dc4-hxl6w unknown-galaxy.svc.cluster.local.
198 starwars/jar-jar-binks-59cdcc8dc4-hxl6w unknown-galaxy.starwars.svc.cluster.local.
198 starwars/jar-jar-binks-59cdcc8dc4-hxl6w unknown-galaxy.cluster.local.
198 starwars/jar-jar-binks-59cdcc8dc4-hxl6w unknown-galaxy.
```

The output in the above command illustrates a typical example of a particular
pod `jar-jar-binks-59cdcc8dc4-hxl6w` in namespace `starwars` consistently
failing to look up the FQDN `unknown-galaxy` while attempting to resolve all
the variants of the DNS suffix search list.

##### Tracking Pod Context in Metrics

The standard configuration will not attach any additional context, and the
metric will describe the DNS errors and DNS response balance for an entire
cluster or node. However, using the programmable metrics, Hubble can be
configured to attach additional context to scope the metric by namespace,
security identity, or even to individual pods. As an example, the following
graph has been configured to label DNS errors with the pod name receiving the
DNS error:

The [Prometheus] query used is:

```bash
topk(10,sum(rate(hubble_dns_responses_total{rcode!=\"No Error\"}[1m])) by (pod, destination))
```

See [Hubble
Metrics Documentation](https://github.com/cilium/hubble/blob/v0.5/Documentation/metrics.md#metric-options)
for details on how to configure Hubble metrics to label accordingly.

Counting the number of DNS errors per pod helps to quickly identify which pods
are subject to DNS resolution failures but doesn't yet tell us why the
resolution is failing.

##### How to debug the DNS resolution of a pod

Continuing the troubleshooting process of the previous section, we can use
Hubble to retrieve the detailed flow log and gain insights into the entire DNS
resolution process. For this, we first define a couple of helper functions to
identify the Hubble pod responsible for a particular application pod:

```bash
function node-of-pod {
    kubectl -n $1 get pods $2 -o json | jq '.spec.nodeName'
}

function hubble-pod {
    kubectl -n kube-system get pods -l k8s-app=hubble -o json | \
    jq -r ".items[] | select(.spec.nodeName==$(node-of-pod $1 $2)) | .metadata.name"
}
```

We can then extract the entire DNS resolution history of that pod as
observed in the last minute:

```bash
kubectl -n kube-system exec -ti $(hubble-pod starwars jar-jar-binks-59cdcc8dc4-hxl6w) -- \
    hubble observe --since=1m -t l7 --protocol DNS --pod starwars/jar-jar-binks-59cdcc8dc4-hxl6w -j | \
    jq -r '.time + " " + .Summary'
```

```bash
2019-12-17T15:23:12.681144228Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy. A)
2019-12-17T15:23:12.678563485Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy. AAAA)
2019-12-17T15:23:12.673809404Z DNS Query unknown-galaxy. A
2019-12-17T15:23:12.673437023Z DNS Query unknown-galaxy. AAAA
2019-12-17T15:23:12.671396412Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.cluster.local. AAAA)
2019-12-17T15:23:12.671358294Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.cluster.local. A)
2019-12-17T15:23:12.670369160Z DNS Query unknown-galaxy.cluster.local. A
2019-12-17T15:23:12.669797423Z DNS Query unknown-galaxy.cluster.local. AAAA
2019-12-17T15:23:12.667882766Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.svc.cluster.local. AAAA)
2019-12-17T15:23:12.666437147Z DNS Query unknown-galaxy.svc.cluster.local. AAAA
2019-12-17T15:23:12.666968986Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.svc.cluster.local. A)
2019-12-17T15:23:12.666305732Z DNS Query unknown-galaxy.svc.cluster.local. A
2019-12-17T15:23:12.663315418Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.default.svc.cluster.local. A)
2019-12-17T15:23:12.659744803Z DNS Query unknown-galaxy.default.svc.cluster.local. A
2019-12-17T15:23:12.659082489Z DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.default.svc.cluster.local. AAAA)
2019-12-17T15:23:12.658412636Z DNS Query unknown-galaxy.default.svc.cluster.local. AAAA
```

The output illustrates nicely how Kubernetes configures a Pod's [DNS Config] to
search a list of domain names and how each request fails for IPv4 (`A`) and
IPv6 (`AAAA`) with an error indicating that the corresponding DNS name could
not be found.

If the DNS server is returning any errors, you will see it in the output.

##### How to debug missing DNS responses

What if the symptom isn't DNS errors, but DNS responses are missing altogether?
Assuming that the pod `starwars/jar-jar-binks-5bcd4b9b9f-cn7vc` is failing to
perform DNS resolution, we can check for network packet drops from and to that
pod by running:

```bash
kubectl -n kube-system exec -ti $(hubble-pod starwars jar-jar-binks-5bcd4b9b9f-cn7vc) -- \
    hubble observe --since=5m --pod starwars/jar-jar-binks-5bcd4b9b9f-cn7vc -t drop
```

```
TIMESTAMP             SOURCE                                          DESTINATION                                       TYPE                 VERDICT   SUMMARY
Dec 17 15:37:32.171   starwars/jar-jar-binks-5bcd4b9b9f-cn7vc:45464   kube-system/coredns-5644d7b6d9-v5cv2:53(domain)   Policy denied (L3)   DROPPED   UDP
Dec 17 15:37:32.171   starwars/jar-jar-binks-5bcd4b9b9f-cn7vc:45464   kube-system/coredns-5644d7b6d9-v5cv2:53(domain)   Policy denied (L3)   DROPPED   UDP
Dec 17 15:37:27.166   starwars/jar-jar-binks-5bcd4b9b9f-cn7vc:45464   kube-system/coredns-5644d7b6d9-v5cv2:53(domain)   Policy denied (L3)   DROPPED   UDP
Dec 17 15:37:27.164   starwars/jar-jar-binks-5bcd4b9b9f-cn7vc:45464   kube-system/coredns-5644d7b6d9-v5cv2:53(domain)   Policy denied (L3)   DROPPED   UDP
Dec 17 15:36:43.680   starwars/jar-jar-binks-5bcd4b9b9f-cn7vc:42160   kube-system/coredns-5644d7b6d9-r9pfk:53(domain)   Policy denied (L3)   DROPPED   UDP
Dec 17 15:36:43.680   starwars/jar-jar-binks-5bcd4b9b9f-cn7vc:42160   kube-system/coredns-5644d7b6d9-r9pfk:53(domain)   Policy denied (L3)   DROPPED   UDP
```

In this example, the cause for DNS resolution is simple. The corresponding UDP
packets are being dropped because the packets are being denied by the
configured NetworkPolicies.

If the issue is still unclear, the entire network transaction can be retrieved
to identify in which exact moment packets are being dropped:

```bash
hubble-cluster hubble observe --since=1m --pod starwars/jar-jar-binks-5bcd4b9b9f-k6ts9
```

```bash
Dec 17 15:47:03.016   10.96.0.10:53(domain)                             starwars/jar-jar-binks-5bcd4b9b9f-k6ts9:51031     to-endpoint    FORWARDED   UDP
Dec 17 15:47:03.011   kube-system/coredns-5644d7b6d9-r9pfk:53(domain)   starwars/jar-jar-binks-5bcd4b9b9f-k6ts9:51031     dns-response   FORWARDED   DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.starwars.svc.cluster.local. A)
Dec 17 15:47:03.011   starwars/jar-jar-binks-5bcd4b9b9f-k6ts9:51031     kube-system/coredns-5644d7b6d9-r9pfk:53(domain)   dns-request    FORWARDED   DNS Query unknown-galaxy.starwars.svc.cluster.local. A
Dec 17 15:47:03.011   starwars/jar-jar-binks-5bcd4b9b9f-k6ts9:51031     kube-system/coredns-5644d7b6d9-r9pfk:53(domain)   to-proxy       FORWARDED   UDP
```

The above example shows a complete transaction showing the UDP packet carrying
the request, the parsed DNS request, the parsed DNS response, and the UDP
packet carrying the DNS response delivered back to the endpoint. If any of them
are missing, then you will know where the packet is being dropped.

##### Summary

DNS issues are a frequent cause of outages and incidents in Kubernetes
clusters. DNS issues have typically been hard to detect and troubleshoot. In
this guide, we have explored how [Hubble] using [Cilium] and [eBPF] can help to
identify and track down DNS issues in your Kubernetes cluster.

To get started:

- Install Cilium using any of the [Installation instructions](http://docs.cilium.io/en/v1.6/gettingstarted/#installation)
- Deploy Hubble with the metrics dashboard configured: [Setting up Hubble Metrics with Grafana](https://github.com/cilium/hubble/tree/v0.5/tutorials/deploy-hubble-and-grafana)
- Enable [DNS visiblity](https://github.com/cilium/hubble/blob/v0.5/Documentation/dns_visibility.md)
- Check out [Cilium Slack] for questions and answers

[cilium]: https://github.com/cilium/cilium
[ebpf]: https://docs.cilium.io/en/stable/bpf/
[cilium slack]: http://slack.cilium.io/
[hubble]: https://github.com/cilium/hubble
[prometheus]: https://github.com/prometheus/prometheus
[grafana]: https://github.com/grafana/grafana
[prometheus alert]: https://prometheus.io/docs/practices/alerting/
[cilium]: https://github.com/cilium/cilium
[ebpf]: http://docs.cilium.io/en/v1.6/bpf/
[dns config]: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config

#### Announcing Hubble - Network, Service & Security Observability for Kubernetes

URL: https://cilium.io/blog/2019/11/19/announcing-hubble
Date: 2019-11-19
Categories: Technology

Hubble is a fully distributed networking and security observability platform
for cloud native workloads. Hubble is open source software and built on top of
[Cilium] and [eBPF] to enable deep visibility into the communication and
behavior of services as well as the networking infrastructure in a completely
transparent manner.

Hubble can answer questions such as:

**Service dependencies & communication map:**

- What services are communicating with each other? How frequently? What does
 the service dependency graph look like?
- What HTTP calls are being made? What Kafka topics does a service consume
 from or produce to?

**Operational monitoring & alerting:**

- Is any network communication failing? Why is communication failing? Is it
 DNS? Is it an application or network problem? Is the communication broken on
 layer 4 (TCP) or layer 7 (HTTP)?
- Which services have experienced a DNS resolution problems in the last 5
 minutes? Which services have experienced an interrupted TCP connection
 recently or have seen connections timing out? What is the rate of unanswered
 TCP SYN requests?

**Application monitoring:**

- What is the rate of 5xx or 4xx HTTP response codes for a particular service
 or across all clusters?
- What is the 95th and 99th percentile latency between HTTP requests and
 responses in my cluster? Which services are performing the worst? What is
 the latency between two services?

**Security observability:**

- Which services had connections blocked due to network policy? What services
 have been accessed from outside the cluster? Which services have resolved a
 particular DNS name?

##### Why Hubble?

The Linux kernel technology [eBPF] is enabling visibility into systems and
applications at a granularity and efficiency that was not possible before. It
does so in a completely transparent way, without requiring the application to
change or for the application to hide information. By building on top of
[Cilium], Hubble can leverage [eBPF] for visibility. By relying on [eBPF], all
visibility is programmable and allows for a dynamic approach that minimizes
overhead while providing deep and detailed visibility where required. Hubble
has been created and specifically designed to make best use of these new [eBPF]
powers.

##### Features

###### Service Dependency Graph

Troubleshooting microservices application connectivity is a challenging task.
Simply looking at "kubectl get pods" does not indicate dependencies between
each service or external APIs or databases.

Hubble enables zero-effort automatic discovery of the service dependency graph
for Kubernetes Clusters at L3/L4 and even L7, allowing user-friendly
visualization and filtering of those dataflows as a Service Map.

See [Hubble Service Map Tutorial](https://github.com/cilium/hubble/tree/v0.5/tutorials/deploy-hubble-servicemap)
for more examples.

##### Metrics & Monitoring

The metrics and monitoring functionality provides an overview of the state of
systems and allow to recognize patterns indicating failure and other scenarios
that require action. The following is a short list of example metrics, for a
more detailed list of examples, see the [Metrics
Documentation](https://github.com/cilium/hubble/blob/v0.5/Documentation/metrics.md)

###### Networking Behavior

###### Network Policy Observation

###### HTTP Request/Response Rate & Latency

###### DNS Request/Response Monitoring

##### Flow Visibility

Flow visibility provides visibility into flow information on the network and
application protocol level. This enables visibility into individual TCP
connections, DNS queries, HTTP requests, Kafka communication, and much more.

###### DNS Resolution

Identifying pods which have received DNS response indicating failure:

```
hubble observe --since=1m -t l7 -j \
    | jq 'select(.l7.dns.rcode==3) | .destination.namespace + "/" + .destination.pod_name' \
    | sort | uniq -c | sort -r
  42 "starwars/jar-jar-binks-6f5847c97c-qmggv"
```

_Successful query & response:_

```
starwars/x-wing-bd86d75c5-njv8k            kube-system/coredns-5c98db65d4-twwdg      DNS Query deathstar.starwars.svc.cluster.local. A
kube-system/coredns-5c98db65d4-twwdg       starwars/x-wing-bd86d75c5-njv8k           DNS Answer "10.110.126.213" TTL: 3 (Query deathstar.starwars.svc.cluster.local. A)
```

_Non-existent domain:_

```
starwars/jar-jar-binks-789c4b695d-ltrzm    kube-system/coredns-5c98db65d4-f4m8n      DNS Query unknown-galaxy.svc.cluster.local. A
starwars/jar-jar-binks-789c4b695d-ltrzm    kube-system/coredns-5c98db65d4-f4m8n      DNS Query unknown-galaxy.svc.cluster.local. AAAA
kube-system/coredns-5c98db65d4-twwdg       starwars/jar-jar-binks-789c4b695d-ltrzm   DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.starwars.svc.cluster.local. A)
kube-system/coredns-5c98db65d4-twwdg       starwars/jar-jar-binks-789c4b695d-ltrzm   DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Query unknown-galaxy.starwars.svc.cluster.local. AAAA)
```

###### HTTP Protocol

_Successful request & response with latency information:_

```
starwars/x-wing-bd86d75c5-njv8k:53410      starwars/deathstar-695d8f7ddc-lvj84:80    HTTP/1.1 GET http://deathstar/
starwars/deathstar-695d8f7ddc-lvj84:80     starwars/x-wing-bd86d75c5-njv8k:53410     HTTP/1.1 200 1ms (GET http://deathstar/)
```

###### TCP/UDP Packets

_Successful TCP connection:_

```
starwars/x-wing-bd86d75c5-njv8k:53410      starwars/deathstar-695d8f7ddc-lvj84:80    TCP Flags: SYN
deathstar.starwars.svc.cluster.local:80    starwars/x-wing-bd86d75c5-njv8k:53410     TCP Flags: SYN, ACK
starwars/x-wing-bd86d75c5-njv8k:53410      starwars/deathstar-695d8f7ddc-lvj84:80    TCP Flags: ACK, FIN
deathstar.starwars.svc.cluster.local:80    starwars/x-wing-bd86d75c5-njv8k:53410     TCP Flags: ACK, FIN
```

_Connection timeout:_

```
starwars/r2d2-6694d57947-xwhtz:60948   deathstar.starwars.svc.cluster.local:8080     TCP Flags: SYN
starwars/r2d2-6694d57947-xwhtz:60948   deathstar.starwars.svc.cluster.local:8080     TCP Flags: SYN
starwars/r2d2-6694d57947-xwhtz:60948   deathstar.starwars.svc.cluster.local:8080     TCP Flags: SYN
```

###### Network Policy Behavior

_Denied connection attempt:_

```
starwars/enterprise-5775b56c4b-thtwl:37800   starwars/deathstar-695d8f7ddc-lvj84:80(http)   Policy denied (L3)   TCP Flags: SYN
starwars/enterprise-5775b56c4b-thtwl:37800   starwars/deathstar-695d8f7ddc-lvj84:80(http)   Policy denied (L3)   TCP Flags: SYN
starwars/enterprise-5775b56c4b-thtwl:37800   starwars/deathstar-695d8f7ddc-lvj84:80(http)   Policy denied (L3)   TCP Flags: SYN
```

##### Getting Started

See the links below to get started. While doing so, make sure to join the
`#hubble` channel on the [Cilium Slack] to share your experiences with others
exploring Hubble and to provide feedback to the Hubble development team.

- [Installation](https://github.com/cilium/hubble/blob/v0.5/Documentation/installation.md)
- [Documentation](https://github.com/cilium/hubble/tree/v0.5/Documentation)
- [Tutorials](https://github.com/cilium/hubble/tree/v0.5/tutorials)

[cilium]: http://github.com/cilium/cilium
[ebpf]: http://docs.cilium.io/en/stable/bpf/
[cilium slack]: https://slack.cilium.io/

#### eBPF at Linux Plumbers 2019, Lisbon, Portugal

URL: https://cilium.io/blog/2019/08/22/lpc19-bpf
Date: 2019-08-22
Categories: Community

The [Linux Plumbers Conference 2019](https://linuxplumbersconf.org/event/4) is
coming up September 9-11 in Lisbon, Portugal. There are several tracks
featuring eBPF related topics:

- [BPF Microconference](https://linuxplumbersconf.org/event/4/sessions/62/#20190911)
- [Linux Networking Summit](https://linuxplumbersconf.org/event/4/sessions/41/#all)
- [Tracing Microconference](https://linuxplumbersconf.org/event/4/sessions/49/#20190911)

##### BPF Microconference

The microconference is hosted by the two eBPF upstream maintainers Alexei
Starovoitov and Daniel Borkmann.

> The goal of the BPF Microconference is to bring BPF developers together to discuss and hash out unresolved issues and to move new ideas forward. The focus of this year's event is on the core BPF infrastructure as well as its many subsystems and related user space tooling.

> The BPF Microconference will be open to all LPC attendees. There is no additional registration required. This is also a great occasion for BPF users and developers to meet face to face and to exchange and discuss developments.

As usual, presentation material will be made available after the event.

###### Agenda

- **Bringing BPF developer experience to the next level** ([details](https://linuxplumbersconf.org/event/4/contributions/448/)), Andrii Nakryiko (Facebook)
- **BPF Debugging** ([details](https://linuxplumbersconf.org/event/4/contributions/447/)), Yonghong Song (Facebook)
- **A pure Go BPF library** ([details](https://linuxplumbersconf.org/event/4/contributions/449/)), Joe Stringer (Cilium), Lorenz Bauer (Cloudflare), Martynas Pumputis (Cilium)
- **Do we need CAP_BPF_ADMIN?**([more details](https://linuxplumbersconf.org/event/4/contributions/452/)), Song Liu (Facebook)
- **Reuse host JIT back-end as offload back-end** ([more details](https://linuxplumbersconf.org/event/4/contributions/451/)), Jiong Wang (Netronome)
- **Using SCEV to establish pre and post-conditions over BPF code** ([more details](https://linuxplumbersconf.org/event/4/contributions/450/)), John Fastabend (Cilium)
- **Beyond per-CPU atomics and rseq syscall: subset of eBPF bytecode for the do_on_cpu syscall** ([more details](https://linuxplumbersconf.org/event/4/contributions/453/)), Mathieu Desnoyers (EfficiOS)
- **Kernel Runtime Security Instrumentation (KRSI)** ([more details](https://linuxplumbersconf.org/event/4/contributions/454/)), KP Singh (Google)
- **Map batch processing** ([more details](https://linuxplumbersconf.org/event/4/contributions/488/)), Yonghong Song (Facebook)

##### Networking Summit

The track is hosted by David Miller and Daniel Borkmann.

###### eBPF related Agenda

- **BPF packet capture helpers, libbpf interfaces** ([more details](https://linuxplumbersconf.org/event/4/contributions/489/)), Alan Maguire (Oracle)
- **Programmable socket lookup with BPF** ([more details](https://linuxplumbersconf.org/event/4/contributions/487/)), Jakub Sitnicki (Cloudflare), Lorenz Bauer (Cloudflare), Marek Majkowski (Cloudflare)
- **XDP bulk packet processing** ([more details](https://linuxplumbersconf.org/event/4/contributions/465/)), Maciej Fijałkowski (Intel)
- **Making the Kubernetes Service Abstraction Scale using eBPF** ([more details](https://linuxplumbersconf.org/event/4/contributions/458/)), Daniel Borkmann (Cilium), Martynas Pumputis (Cilium)
- **Scaling container policy management with kernel features** ([more details](https://linuxplumbersconf.org/event/4/contributions/464/)), Joe Stringer (Cilium)
- **Seamless transparent encryption with BPF and Cilium** ([more details](https://linuxplumbersconf.org/event/4/contributions/461/)), John Fastabend (Cilium)
- **Traffic footprint characterization of workloads using BPF** ([more details](https://linuxplumbersconf.org/event/4/contributions/490/)), Aditi Ghag (VMware)
- **XDP: the Distro View** ([more details](https://linuxplumbersconf.org/event/4/contributions/460/)), Jiri Benc (Red Hat)
- **An Evaluation of Host Bandwidth Manager** ([more details](https://linuxplumbersconf.org/event/4/contributions/486/)), Lawrence Brakmo (Facebook)

##### Tracing Microconference

The track is hosted by Steven Rostedt.

###### eBPF related Agenda

- **bpftrace** ([more details](https://linuxplumbersconf.org/event/4/contributions/441/)), Alastair Robertson (Yellowbrick)
- **BPF Tracing Tools: New Observability for Performance Analysis** ([more details](https://linuxplumbersconf.org/event/4/contributions/444/)), Brendan Gregg (Netflix)

#### Cilium 1.6: KVstore-free operation, 100% kube-proxy replacement, Socket-based load-balancing, Generic CNI Chaining, Native AWS ENI support,...

URL: https://cilium.io/blog/2019/08/20/cilium-16
Date: 2019-08-20
Categories: Release

We are excited to announce the Cilium 1.6 release. A total of 1408 commits have
been contributed by the community with many developers contributing for the
first time. Cilium 1.6 introduces several exciting new features:

- **KVStore free operation:** The addition of a new CRD-based
 backend for security identities now allows to operate Cilium entirely
 without a KVstore in the context of Kubernetes. ([More details](#kvstorefree))
- **100% Kube-proxy replacement:** Operating a
 Kubernetes cluster without requiring to run kube-proxy has been a desire of
 many users. This release includes the final two features required to run a
 Kubernetes cluster without kube-proxy with Cilium fully replacing
 kube-proxy. ([More details](#kubeproxy-removal))
- **Socket-based load-balancing:** Socket-based load-balancing combines
 the advantage of client-side and network-based load-balancing by providing
 fully transparent load-balancing using Kubernetes services with the
 translation from service IP to endpoint IP done once during connection
 establishment instead of translating each network packet for the lifetime of
 a connection. ([More details](#hostservices))
- **Policy scalability improvements:** The entire policy system has been
 improved to decouple handling of policy and identity definitions and moving
 to an entirely incremental model. This ensures that environments with high
 pod scheduling churn, e.g. several 100K pods across multiple clusters, can
 cope well in combination with constant policy definition changes. ([More
 details](#policyscale))
- **Generic CNI chaining:** The 1.6 release introduces a new
 CNI chaining framework allowing to run Cilium on top of the majority of
 other CNI plugins such as Weave, Calico, Flannel, AWS VPC CNI or the Lyft
 CNI plugin. This enables using advanced features such as eBPF-based security
 policy enforcement, visibility, multi-cluster, encryption, and
 load-balancing while continuing to run whatever CNI plugin is already
 in-use. ([More details](#chaining))
- **Native AWS ENI mode:** A new datapath and IPAM mode allows to combine the
 efficiency of native AWS ENI routing with Cilium policy enforcement,
 encryption and multi-cluster. A new operator-based design, works around many
 problems known to large scale AWS ENI users with per node agents. ([More
 details](#eni))
- **... and much more:** For the full list of changes, see the [1.6 Release
 Notes](#16Highlights).

###### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms like Kubernetes, Docker, and Mesos.

At the foundation of Cilium is a new Linux kernel technology called eBPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. eBPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption as well as network and service security. Besides
providing traditional network level security, the flexibility of eBPF enables
security with the context of application protocols and DNS requests/responses.
Cilium is tightly integrated with Envoy and provides an extension framework
based on Go. Because eBPF runs inside the Linux kernel, all Cilium
functionality can be applied without any changes to the application code or
container configuration.

See the section **[Introduction to
Cilium](https://cilium.readthedocs.io/en/stable/intro/)** for a more detailed general
introduction to Cilium.

##### KVStore Free Operation (CRD storage)

_Contributed by Ray Bejjani and others_

This has been one of the long awaited features and it has finally landed. We
have completed all work to run Cilium without requiring a key-value store.
Starting with 1.6, it is possible to run Cilium entirely CRD-backed in a
Kubernetes environment. The new modes in a nutshell:

- **New default:** The new default mode derives all required information from
 standard Kubernetes resources and stores all state using Custom Resource
 definitions (CRDs).

- **Managed etcd:** Building on top of the default mode, the managed
 etcd mode uses an etcd-operator to maintain an etcd cluster within the
 Kubernetes cluster and uses it to offload state propagation. This leads to
 better scalability while continuing to maintain Kubernetes as the source of
 truth for all state. If the etcd cluster fails, state propagation falls back
 to being CRD-based until etcd recovers.

 This new managed etcd mode is more reliable than the previous etcd-operator
 mode in versions 1.5. Due to maintaining the source of truth in Kubernetes,
 failure of etcd is no longer critical.

- **External kvstore:** This remains the most reliable mode for large-scale
 environments. It requires an external kvstore to be maintained. The
 Kubernetes etcd can be used or a dedicated etcd or consul cluster.

For more information, see the new [Quick
Installation](https://docs.cilium.io/en/v1.6/gettingstarted/k8s-install-default/)
section in the documentation.

###### Kube-proxy Removal

_Contributed by Daniel Borkmann and Martynas Pumputis_

The 1.6 release brings the last missing piece in the kube-proxy removal -
support for services of the NodePort type and the ability to make Kubernetes
services available for processes on the host and pods running in host
networking. The new feature allows to finally run a fully functioning
Kubernetes cluster without kube-proxy. This means that long lists of iptables
rules are no longer needed to enable the Kubernetes Service abstraction.

###### Why does it matter?

The plot below shows a latency per HTTP GET request (lower is better) via a
Kubernetes service to a `nginx` Pod running on a remote host measured with `ab`
sending 100'000 sequential requests. As we can see, the eBPF service
implementation does not depend on number of services being deployed in a
cluster, while the kube-proxy implementation using iptables does.

The eBPF-based service implementation not only improves latency and throughput
of a Kubernetes service traffic, it also reduces time and overhead when
processing changes in a service definition. With iptables, entire tables of
rules have to be replaced in the datapath whereas eBPF supports applying
individual service changes in the datapath.

Another key difference is the deterministic behavior. An iptables-based service
implementation requires multiple iptables rules for each service and service
backend. This essentially creates a long list of rules which have to be
traversed one by one. Depending on how far down the list the service to be
matched is, the cost is higher, some service load-balancing will have the cost
of only having to traverse a few rules, other packets will have to traverse
thousands of rules before a match is found. The behavior is non-deterministic.
With eBPF, the implementation uses hash tables so the lookup cost is close to
O(1) and the behavior is deterministic as the benchmark above indicates as
well.

###### Removal of the per-node iptables rules mess

Anyone who has been operating Kubernetes at non-trivial scale with more than a
couple of dozen services, will have noticed, the vast amount of iptables rules
generated. The following is an example of _one_ Kubernetes service of type
NodePort with two endpoints as backends:

**Rules with kube-proxy (iptables)**

```bash
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx:" -m tcp --dport 30905 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx:" -m tcp --dport 30905 -j KUBE-SVC-253L2MOZ6TC5FE7P
-A KUBE-SEP-PCCJCD7AQBIZDZ2N -s 10.217.1.154/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-PCCJCD7AQBIZDZ2N -p tcp -m tcp -j DNAT --to-destination 10.217.1.154:80
-A KUBE-SEP-UFVSO22B5A7KHVMO -s 10.217.1.159/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-UFVSO22B5A7KHVMO -p tcp -m tcp -j DNAT --to-destination 10.217.1.159:80
-A KUBE-SERVICES ! -s 10.217.0.0/16 -d 10.107.41.178/32 -p tcp -m comment --comment "default/nginx: cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.107.41.178/32 -p tcp -m comment --comment "default/nginx: cluster IP" -m tcp --dport 80 -j KUBE-SVC-253L2MOZ6TC5FE7P
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j
KUBE-NODEPORTS
-A KUBE-SVC-253L2MOZ6TC5FE7P -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-PCCJCD7AQBIZDZ2N
-A KUBE-SVC-253L2MOZ6TC5FE7P -j KUBE-SEP-UFVSO22B5A7KHVMO
```

**Rules with eBPF-based k8s services**

```bash
# This snippet is intentionally left blank
```

###### Native eBPF NodePort and SNAT implementation

Under the hood, we attach a NodePort eBPF program to a network device which has
a default route on the host. This makes a NodePort service to be accessible via
IP address of the device. In addition, thanks to the new host-level services
feature, the NodePort service can be reached within the cluster from a host or
a Pod via loopback or `cilium_host` IP address.

When the service endpoint runs on the other host than a request was sent to,
the eBPF program does the SNAT translation before forwarding the request to
a host running the endpoint.

The feature can be enabled by setting `enable-node-port` to `true`. See the
[NodePort Getting Started Guide](https://docs.cilium.io/en/v1.6/gettingstarted/nodeport/)
for more configuration options.

###### Socket-based load-balancing

_Contributed by Daniel Borkmann and Martynas Pumputis_

Load-balancing is typically done in one of the following ways:

- The application performs client-side load-balancing and picks a destination
 endpoint itself. The benefit of this is that the cost of load-balancing is
 paid once upfront when a connection is established and no additional overhead
 exists for the lifetime of the connection. The downside of this approach is
 that this is not transparent to the application.

- The network performs the load-balancing via a middle box by translating
 requests to a particular service IP. The advantage of this method over
 client-side load-balancing is the transparency. The application itself is not
 involved. However, the downside is that each network packet needs to have its
 IP addresses changed in both the request and response direction.

With Cilium 1.6, we are introducing socket-based load-balancing which combines
the advantages of both approaches:

- **Transparent:** Load-balancing remains 100% transparent to the application.
 Services are defined using standard Kubernetes service definitions.

- **Highly efficient:** By performing the load-balancing at the socket level by
 translating the address inside the `connect(2)` system call, the cost of
 load-balancing is paid upfront when setting up the connection and no
 additional translation is needed for the duration of the connection
 afterwards. The performance is identical as if the the application talks
 directly to the backend.

###### Diving Deeper

This socket-based load-balancing implementation makes use of recent eBPF cgroup
hooks in the Linux kernel which Cilium attaches to `connect(2)`, `sendmsg(2)`
and `recvmsg(2)` system calls. The eBPF programs then rewrite `sockaddr_in{,6}`
data with information for the service backend. For UDP services we've extended
the kernel to enable `recvmsg(2)` ([more
details](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=983695fa676568fc0fe5ddd995c7267aabc24632))
for reverse sock address translation. The advantage of these hooks are that
they i) operate transparent to the application, ii) they operate in a device
independent manner, and iii) the backend selection only needs to be performed
e.g. at `connect(2)` time. Latter means that packets headers don't need to be
NAT'ed throughout the lifetime of the connection. Thus, to the kernel it looks
as if the application connects to the backend directly. Host-reachable services
are available for both TCP and UDP where in case of UDP both connected and
unconnected UDP is supported.

This feature can be enabled by setting `enable-host-reachable-services` to
`"true"` for the daemon config which then exposes both TCP and UDP services.
Protocol exposure can be controlled via daemon flag
`host-reachable-services-protos` that has a default value of `"tcp,udp"` and
can be set to one of the two for older kernel compatibility.

###### Generic CNI Chaining

_Contributed by Thomas Graf and Laurent Bernaille_

CNI (Container Networking interface) is the Kubernetes API for plugins to
provide networking to Kubernetes pods. While Cilium directly implements this
API as well, there are scenarios where it is desirable to use another CNI
plugin for networking and use Cilium to provide additional functionality on top
such as security policy enforcement, multi-cluster, encryption, or
load-balancing.

For all of these users, we are introducing generic CNI chaining. It is a new
pluggable architecture built on top of the standard chaining feature of CNI.
All veth based CNI plugins can be integrated with a new generic veth chainer
that should work out of the box in combination with any CNI plugin that uses
veth devices to connect pods. Other CNI plugins can be integrated by writing a
bit of Go code to define the interface between Cilium and that CNI plugin.

###### Policy Scalability

_Contributed by Ian Vernon, Jarno Rajahalme, and Joe Stringer_

While Cilium release 1.5 focused on scalability for Cilium running in clusters
with a large number of nodes, release 1.6 improves scalability of node-local
operations in these environments, specifically policy computation. Policy
computation in Cilium occurs for each pod running on a node whenever global
cluster state or policy has been changed by a cluster administrator, for
example when a new pod has launched, or when local policy-relevant state
changes, such as when Cilium learns a new IP address corresponding to an FQDN
which is allowed by policy. When these events happen, Cilium determines which
rules select each of the local endpoints, and figures which counterparts are
allowed (e.g., other pods within the cluster, FQDNs, CIDRs, etc.). As part of
this computation, Cilium now caches which label identities match each label
selector in the applicable policy rules. For instance, take the following
policy rule:

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
description: 'L3 ingress from env=qa'
metadata:
  name: 'l3-ingress-qa'
spec:
  endpointSelector:
    matchLabels:
      app: foo
  ingress:
    - fromEndpoints:
        - matchLabels:
            env: qa
```

The rule applies on any pod running with label `app=foo`. Within the rule, we
can see that we should allow ingress from any remote destination corresponding
to `env=qa`. Given a cluster that has multiple endpoints with label `env=qa`,
e.g., `{app=foo, env=qa}`, `{app=bar, env=qa}`, `{app=baz, env=qa}`, a unique
numeric identifier, called a security identity, is allocated to each of these
sets of labels, e.g.:

```
51718 --> {app=foo, env=qa}
56949 --> {app=bar, env=qa}
61625 --> {app=baz, env=qa}
```

As soon as the allocation of these security identities is distributed to a
local node, Cilium matches each locally used selector in the policy against the
new identity and caches this information so that this matching operation need
not be performed from scratch again later. You can see this caching in action:

```bash
$ cilium policy selectors -o json
[
  {
    "identities": [
      51718,
      56949,
      61625
    ],
    "selector": "\u0026LabelSelector{MatchLabels:map[string]string{any.env: qa,k8s.io.kubernetes.pod.namespace: default,},MatchExpressions:[],}",
    "users": 2
  }
]
```

The caching of this information now has a single upfront cost, and after that,
a simple map lookup for the selector suffices to see to which remote
destinations it corresponds. This allows for efficient policy computation for a
large number of identities for a large number of rules. As part of this new
selector caching functionality, Cilium can now update the eBPF policy state
without performing a full policy recomputation when new pods are spun up in
anywhere in the cluster. We are proud to say that now Cilium can efficiently
compute the policy for a pod when thousands of rules are configured in
large-scale clusters that run thousands of pods. The benchmark results speak
for themselves:

```
PASS: resolve_test.go:225: PolicyTestSuite.BenchmarkRegenerateL3IngressPolicyRules     1000    2150217 ns/op
PASS: resolve_test.go:235: PolicyTestSuite.BenchmarkRegenerateL3EgressPolicyRules     1000    2209893 ns/op
```

These benchmarks test different types of rules against a large number of
identities. The profiling output above indicates that even with 1000 rules of
either ingress or egress, and 3000 identities, layer-3 policy computation takes
around 2 milliseconds when the selector caching has been performed upfront.

Along with these policy calculation changes, we improved also other aspects of
generating the endpoint datapath. In cases with large amounts of identity
churn---that is, upwards of dozens of pods being created in the cluster each
second---it commonly takes less than 50 milliseconds to update the datapath of
each endpoint to realize the whitelist policy and how it applies to traffic
destined for the new endpoints in the cluster.

###### Native AWS ENI Datapath

_Contributed by Thomas Graf, Vlad Ungureanu, and Ryan McNamara_

The AWS ENI allocator is specific to Cilium deployments running in the AWS
cloud and performs IP allocation based on IPs of AWS Elastic Network Interfaces
(ENI) by communicating with the AWS EC2 API.

The architecture ensures that only a single operator communicates with the EC2
service API to avoid rate-limiting issues in large clusters. A pre-allocation
watermark allows to maintain a number of IP addresses to be available for use
on nodes at all time without requiring to contact the EC2 API when a new pod is
scheduled in the cluster.

The new IPAM mode is coupled with a specific [AWS ENI
datapath](https://docs.cilium.io/en/v1.6/concepts/datapath/#aws-eni) connects
pods using AWS ENI IPs in a direct routing mode that is compatible with the
multi-cluster, policy enforcement and load-balancing subsystem.

###### High-scale Dataplane for Transparent Encryption

_Contributed by John Fastabend, Laurent Bernaille, and Robert Boll_

Cilium provides transparent encryption support utilizing eBPF to orchestrate
encryption using the Linux kernel crypto subsystem. The Cilium 1.6 release
expands this support with a new subnet mode allowing users to specify subnets
of IPs that should have transparent encryption applied. Cilium will also manage
the FIB table, automatically adjust MTUs, and expose flow level encryption
status to ensure connectivity.

###### Node to node encryption

The encryption support is extended to also cover node to node communication
even if no pod communication is involved. This allows to easily encrypt all
network traffic regardless whether the source or destination endpoint is a
Kubernetes pod.

###### Helm Charts

With the growing number of supported datapath and control plane configurations,
maintaining pre-generated YAML resources for deployments has become hard. As
part of this, we have integrated with [Helm] and now provide several Helm
charts which can be used for templating and deployment. All guides have been
converted to use [Helm].

**Example: Configuring Cilium for GKE**

```bash
helm template cilium \
  --namespace cilium \
  --set nodeinit.enabled=true \
  --set nodeinit.reconfigureKubelet=true \
  --set nodeinit.removeCbrBridge=true \
  --set global.cni.binPath=/home/kubernetes/bin \
  > cilium.yaml
kubectl create namespace cilium
kubectl create -f cilium.yaml
```

###### 1.6 Release Highlights

- **Policy**
 - New selector cache to massively speed up correlation of label based
 selectors with identity objects
 - The policy map size is now configurable if more than the default 16K are
 needed
 - Labels created from annotations are now ignored from the identity by
 default
 - Allow all policy is now enforced with a single entry in the datapath table
 for reduced overhead
 - New garbage collector to remove obsolete nodes in the status section of
 CiliumNetworkPolcies
- **Kvstore free operation**
 - CRD-backing for all required state storage
 - Ability to start Cilium without a key-value store configured
- **Kubernetes**
 - Simplified init container to remove cilium state
 - Distinct user-agent to identify Cilium communication with the apiserver
 - Patch Kubernetes Node status with NetworkUnavailable=false
 - New option to skip creation of CRDs on startup
 - The container runtime integration is now disabled by default. It only needs
 to be enabled if users want container runtime labels in addition to pod
 labels to identify services.
 - Kubernetes 1.15 support
 - Removed dependency of operator to reach the cluster DNS
- **Generic CNI chaining**
 - New CNI chaining plugin architecture
 - Built-in chaining support for aws-cni-vpc, Calico, Portmap, and Weave-Net
 - Generic veth chaining plugin to run on top of any veth based CNI plugin
- **Resource consumption**
 - Numerous code optimizations to reduce memory allocations and footprint in
 environments with a large number of pods.
 - New minimal status response to reduce resource consumption
 - New option to disable status of CiliumNetworkPolicy to reduce resource size
- **Socket-level load-balancing**
 - Highly efficient load-balancing of Kubernetes services at socket level
 - Load-balancing of Kubernetes services for host processes
- **Datapath**
 - New TPROXY based L7 redirection datapath to no longer require to rewrite
 any packet headers to transparently redirect to proxies.
 - Use a static IP 169.254.42.1 for service redirection loopback SNAT instead
 of allocating an IP
 - The new datapath event aggregation now defaults to maximum
 - New datapath mode (--enable-endpoint-routes) where packets to local
 endpoints are routed via a per
 endpoint route instead via a separate veth pair
 - New native AWS ENI datapath mode
- **IPAM**
 - New CRD-backed IPAM mechanism
 - AWS ENI allocation method
- **NodePort Support**
 - Native eBPF implementation of NodePort services (--enable-node-port)
- **Transparent Encryption:**
 - Node to node encryption in addition to pod to pod and pod to node
 - New metric to count encrypted and unencrypted traffic
- **CLI**
 - New `cilium identity list --endpoints` commands
 - New `cilium policy selectors` to list content of the policy selector
 cache
- **Documentation**
 - All guides have been rewritten to use Helm charts
 - New guide on how to use Cilium with Kata containers
 - New guide for simple deployment on microk8s
 - Additional troubleshooting documentation
- **Istio**
 - Support for 1.2.4

###### Getting Started

New to Cilium? Follow one of the [Getting Started
Guides](https://docs.cilium.io/en/v1.6/gettingstarted/).

###### Upgrade Instructions

As usual, follow the [upgrade
guide](https://cilium.readthedocs.io/en/v1.6/install/upgrade/#upgrading-minor-versions)
to upgrade your Cilium deployment. Feel free to ping us on
[Slack].

###### Release

- Release Notes & Binaries: [1.6.0](https://github.com/cilium/cilium/releases/tag/v1.6.0)
- Container image: `docker.io/cilium/cilium:v1.6.0`

[slack]: https://slack.cilium.io
[helm]: https://helm.sh/
[external ips]: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips
[cni chaining mode]: http://docs.cilium.io/en/v1.6/gettingstarted/cni-chaining/

#### CVE-2019-13119: Policy bypass via nested encapsulation 

URL: https://cilium.io/blog/2019/07/01/cve-2019-13119
Date: 2019-07-01
Categories: Technology

On May 25 2019, a security relevant bug has been reported to us via the
documented [security disclosure channel](https://github.com/cilium/cilium/blob/master/SECURITY.md).
Thanks to _l14n_ for the excellent bug report! It was soon identified that
multiple vendors are affected by this vulnerability. This lead to an embargo
period which is being lifted today.

The bug allows, under certain circumstances, to bypass network security
policies. See below for details on the vulnerability and the mitigation.

**Who is affected?:**
Users operating Cilium in encapsulation mode (VXLAN or Geneve) while hosting
untrusted workloads with an egress policy that allows pods to emit UDP
encapsulation traffic to other worker nodes.

The vulnerability is being tracked by [CVE-2019-13119](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-13119)

We are releasing Cilium 1.5.4, 1.4.5, and 1.3.7 to fix the security vulnerability.

##### Vulnerability details

An endpoint can emit encapsulation traffic to another node running Cilium and
given certain conditions are met, that node will accept the encapsulated
traffic and route it. This can bypass egress security policies. If the endpoint
is able to guess security identities correctly, the endpoint can also
impersonate other security identities.

Conditions that must be met:

- Cilium must be running in encapsulation mode. This is the default
 configuration mode.

- The egress policy of the endpoint must allow UDP on the configured
 encapsulation port. This port is different depending on whether VXLAN
 or Geneve is being used as the encapsulation protocol. Alternatively, if the
 endpoint has access to an external LB such as a NodePort which redirects to
 the configured UDP encapsulation port on any node, the policy must allow for
 this. Redirection to a UDP encapsulation port using a CluserIP or headless
 service is not affected as the load-balancing decision happens before egress
 policy.

- The endpoint must be aware of at least one node IP or be aware of an
 external LB that redirects to a node IP.

- Masquerading of egress traffic to worker node IP must be enabled,
 if masquerading is disabled, the remote tunnel will reject the traffic. This
 means that known or guessed worker node IP must be a node IP which is
 considered outside of the cluster so masquerading is performed.

- The endpoint must guess a Cilium security identity or be aware of well-known
 security identities which is allowed by the ingress policy of the destination
 service. Even if no ingress policy is used, the guessed security identity
 must be valid, i.e. it must be a well-known security identity or an security
 identity that is actively in-use by a pod.

 In the reported attack scenario, the reporter has used `tcpdump` on one of
 the nodes to derive the security identity by inspecting the wire and
 replaying encapsulation traffic. Cilium provides support for node level
 encryption. The attack assumes that no encryption is performed. However, the
 security identity is a construct intended for scale. The 32 bit numeric space
 used to group workloads with identical security identity together can be
 subject to brute-force attacks.

##### Mitigation

In order to mitigate this, the following fix is applied:

Any UDP traffic emitted by an endpoint to one of the supported encapsulation
ports is now dropped with a distinct drop reason:

```
xx drop (Encapsulation traffic is prohibited) flow 0xcab0685a to endpoint 0, identity 3087->2: 10.16.209.20:39258 -> 192.168.122.124:8472 udp
```

As Cilium performs load-balancing (enabled by default) before the egress
policy enforcement is applied, any traffic subject to a Kubernetes ClusterIP
service that load-balances to an encapsulation port is also rejected. This
is _not_ the case if you are using kube-proxy as the load-balancing happens
after the standard egress policy enforcement point.

##### Questions

Thanks again to _l14n_ for disclosing the vulnerability to us via the private
security mailing list so we could provide a bugfix relese before disclusing the
details publicly. If you have any questions, feel free to reach out to us on
[Slack](https://slack.cilium.io/).

#### License change and lack of attribution of Cilium eBPF code in Calico project

URL: https://cilium.io/blog/2019/06/24/bpf-code-copyright
Date: 2019-06-24
Categories: Community

##### TL;DR

As with everything we do, we are fully transparent. As it becomes obvious that
a simple resolution in this matter is not possible, we follow open source best
practices and choose a public forum for the sake of transparency.

It was brought to our attention that some of the [new eBPF
code](https://github.com/projectcalico/felix/tree/master/bpf) committed to the
[Calico repository](https://github.com/projectcalico/felix) is violating the
license of source code in the Cilium repository.

The original report called out suspiciously similar code in both repositories.
This by itself is of course not a problem if the open source licenses involved
are respected. This includes, among other things, attribution and restrictions
regarding the rights to re-license.

Upon closer inspection, it was identified that source code has been copied from
the [Cilium
repository](https://github.com/cilium/cilium/blob/master/bpf/bpf_xdp.c),
modified to create derivative work, and then committed ([commit](https://github.com/projectcalico/felix/commit/336d34c307f477b73692656058b890d1c01e73f4#diff-0f8b4a32fcc0262e15d01210383db0df))
to the Calico repository with the license changed in a non-compatible manner.
As part of this, the attribution required by the license was also omitted. The
details of this can be found further down in this post.

Like the majority of the Linux kernel source code, the datapath portion of
Cilium that runs as part of the Linux kernel is released under the [GPL 2.0]
license. The GPL license does not permit a license change to the [Apache
License] without consent of the original authors.

This prompted us to contact the authors of the derivative work. As a result,
[an initial attempt](https://github.com/projectcalico/felix/pull/2048) was made
to rewrite some sections of the code. After inspection, we concluded that the
work is still a derivative of our original source code.

However, in order to resolve the situation as simply as possible, we offered to
dual-license the respective code under the Apache license with the condition
that attribution to the original authors is added. This resulted in the
[following pull request](https://github.com/projectcalico/felix/pull/2049)
being proposed to the Calico repository to add the attribution. The pull
request is currently waiting to be merged.

From our perspective, this would resolve all of our concerns. We obviously also
accept any other resolution as long as it conforms to the respective open
source licenses.

We are waiting for a reaction by the maintainers of the Calico project.

##### The Full Story

Cilium is an open source project that builds on an underlying Linux kernel
technology called eBPF. Recently, the Calico open source project has accepted
several pull requests to add two additional features based on eBPF. Both of the
newly added features have been part of the Cilium repository in similar or
almost identical form for a while. There is obviously nothing wrong with
multiple open source projects implementing the same feature. However, if source
code from a repository is used to derive work, the open source license of that
original source code must be respected.

The respective features are:

- A BPF/XDP based DDoS mitigation filter to drop network packets from
 potentially malicious sources as early as possible
 ([Calico Code](https://github.com/projectcalico/felix/blob/master/bpf/xdp/filter.c) vs
 [Cilium Code](https://github.com/cilium/cilium/blob/master/bpf/bpf_xdp.c))
- A BPF and sockmap based acceleration of local TCP connections for the purpose
 of accelerating sidecar proxy architectures.
 ([Calico Code](https://github.com/projectcalico/felix/blob/master/bpf/sockmap/sockops.c) vs
 [Cilium Code](https://github.com/cilium/cilium/blob/master/bpf/sockops/bpf_sockops.c))

##### Report of License Violation

Following a blog post to announce the features in Calico, it was brought to our
attention that the source code to implement these features looks suspiciously
similar.

This by itself is not a problem so why did we bother at all? Creation of
derivative work is encouraged in open source, right?

The first problem is lack of attribution. Attribution is a key principle in
open source for any significant derivation. While also a license violation,
omission of attribution is frown upon in the open source community.

The second problem is even more significant: The derivative work included an
incompatible change of license from GPL 2.0 to the Apache License. As the
majority of the Linux kernel source code, the datapath portion of Cilium that
directly interacts with the Linux kernel is licensed under GPL 2.0. The user
space code including all control plane aspects are licensed under the Apache
License.

All of this was sufficient to contact the authors of the derived work with a
request for clarification.

###### Exbibit A: Example of derived source code

The following diff shows a subset of the source code and illustrates that
Cilium source code was likely directly copied. Significant amounts of the
original source code are entirely unmodified, other sections only differ in
removed code such as dropping IPv6 support, or different variable/function
naming. The overall code structural is identical in significant amounts. There
are many more examples like the one shown below. For more examples, the
respective files can be comapred in both source code repositories.

```diff
@@ -1,81 +1,73 @@
 static __always_inline void *xdp_data(const struct xdp_md *xdp)
 {
        return (void *)(unsigned long)xdp->data;
 }

 static __always_inline void *xdp_data_end(const struct xdp_md *xdp)
 {
        return (void *)(unsigned long)xdp->data_end;
 }

 static __always_inline bool xdp_no_room(const void *needed, const void *limit)
 {
-       return unlikely(needed > limit);
+       return needed > limit;
 }

 struct lpm_v4_key {
        struct bpf_lpm_trie_key lpm;
        __u8 addr[4];
 };

-struct lpm_v6_key {
-       struct bpf_lpm_trie_key lpm;
-       __u8 addr[16];
-};
-
 struct lpm_val {
-       /* Just dummy for now. */
-       __u8 flags;
+       __u32 ref_count;
 };

 [...]

 static __always_inline int check_v4(struct xdp_md *xdp)
 {
        void *data_end = xdp_data_end(xdp);
        void *data = xdp_data(xdp);
        struct iphdr *ipv4_hdr = data + sizeof(struct ethhdr);
-       struct lpm_v4_key pfx __maybe_unused;
+       struct lpm_v4_key pfx;
+       __u16 dest_port;

-       if (xdp_no_room(ipv4_hdr + 1, data_end))
+       if (xdp_no_room(ipv4_hdr + 1, data_end)) {
                return XDP_DROP;
+       }

-#ifdef CIDR4_FILTER
        __builtin_memcpy(pfx.lpm.data, &ipv4_hdr->saddr, sizeof(pfx.addr));
        pfx.lpm.prefixlen = 32;

-#ifdef CIDR4_LPM_PREFILTER
-       if (map_lookup_elem(&CIDR4_LMAP_NAME, &pfx))
+       if (map_lookup_elem(&calico_prefilter_v4, &pfx)) {

 [...]
 }

-static __always_inline int check_filters(struct xdp_md *xdp)
+static __always_inline int check_prefilter(struct xdp_md *xdp)
 {
        void *data_end = xdp_data_end(xdp);
        void *data = xdp_data(xdp);
        struct ethhdr *eth = data;
        __u16 proto;

-       if (xdp_no_room(eth + 1, data_end))
+       if (xdp_no_room(eth + 1, data_end)) {
                return XDP_DROP;
+       }

        proto = eth->h_proto;
-       if (proto == bpf_htons(ETH_P_IP))
+       if (proto == bpf_htons(ETH_P_IP)) {
                return check_v4(xdp);
-       else if (proto == bpf_htons(ETH_P_IPV6))
-               return check_v6(xdp);
-       else
-               /* Pass the rest to stack, we might later do more
-                * fine-grained filtering here.
-                */
+       } else {
+               /* other traffic can continue */
                return XDP_PASS;
+       }
 }

-__section("from-netdev")
-int xdp_start(struct xdp_md *xdp)
+__section("pre-filter")
+int xdp_enter(struct xdp_md *xdp)
 {
-       return check_filters(xdp);
+       return check_prefilter(xdp);
 }

-BPF_LICENSE("GPL");
+char ____license[] __section("license")  = "Apache-2.0";
```

##### Our offer: Dual-license the respective code

After contacting the authors of the derived work. [An attempt was
made](https://github.com/projectcalico/felix/pull/2048) to reimplement some of
the source code.

The commit description labelled this work as a [clean room
design] implementation. Apparently this involved a reimplementation of isolated
code sections which were identified as being similar to the Cilium source code.
We will leave it to the judgement of everyone individually to decide whether
the following
[diff](https://github.com/projectcalico/felix/pull/2048/commits/62c95df8d3041ab3037998d62fdd2fe480228097)
is a [clean room design] implementation or not.

In order to resolve the situation as simply as possible, we offered to
dual-license the code under the Apache License with the condition that the
original authors must be attributed. This is possible because all authors of
the respective code were core team members which could be identified and asked
for permission easily. We have no problem with others creating derivative work
based on ours but request to be attributed for it.

This proposal seems to have been accepted by the authors of the derivative
work, the original PR was closed and the following
[PR](https://github.com/projectcalico/felix/pull/2049) was opened to add the
attribution. It is currently waiting to be merged.

Great! We can all be happy, right?

##### What happens next?

We are waiting for a reaction by the maintainers of the Calico project to merge
the outstanding pull request. Our offer to put the respective source code under
a dual-license stands. We obviously also accept any other resolution as long as
it respects the open source license of the respective code.

_Update 2019-06-25:_ Some of the eBPF related code has now been removed from
the Calico repository via [this
PR](https://github.com/projectcalico/felix/pull/2056).

[clean room design]: https://en.wikipedia.org/wiki/Clean_room_design
[apache license]: https://www.apache.org/licenses/LICENSE-2.0
[gpl 2.0]: https://en.wikipedia.org/wiki/GNU_General_Public_License#Version_2

#### Cilium User Survey March 2019 - The Results

URL: https://cilium.io/blog/2019/05/03/cilium-user-survey-march-2019
Date: 2019-05-03
Categories: Community

Back in March we have asked our users to provide feedback via our first ever
[user survey](https://forms.gle/c1kphSwp9v82yEgTA). Many of you have responded
and the results are in!

The survey was announced on our Slack channel and on Twitter. Participation was
anonymous and did not require to leave behind contact information. Most
questions had a set of predefined answers plus a field to add additional
answers. All questions were optional, some users did not answer all questions.

##### Current Use of Cilium

###### In what stage are you with your usage of Cilium?

The initial question captured how many users we have in each stage. The first
four answers were predefined.

###### What are you blocked on?

Those who answered that they are blocked were given a chance to provide a
detailed response:

- _End-to-end transparent encryption_ (Now supported by Cilium 1.4/1.5)
- _SSL/TLS Layer 7 filtering_
- _IPv6 /96 prefix requirement_ (Planned for Cilium 1.6)
- _AWS VPC CNI support_ (Planned for Cilium 1.6)
- _Usage with Istio 1.1_ (Now supported by Cilium 1.5)
- _Better VPC-CNI support_ (Planned for 1.6)
- _The latest blocker for me is an integration with Istio CNI_ (Planned for Cilium 1.6)
- _Support of non containers workloads that is on roadmap_
- _RKE_ (Now supported by Cilium 1.5)
- _Openshift v3_ (Guide planned for Cilium 1.6)

_Note: The survey was published just after we had released Cilium 1.4, we have
since released Cilium 1.5 which already includes some of the asks provided via
the survey. See the status of each ask in brackets._

###### What is your motivation to use Cilium?

Why are you using Cilium? The first 9 answers were predefined.

##### Kubernetes

The vast majority of our users are running Cilium on Kubernetes so we have
decided to include a Kubernetes specific section to the survey to gain some
insights into how our users use Kubernetes.

###### How are you managing Kubernetes?

Are you managing your own Kubernetes or are you using a managed Kubernetes
service?

###### How many nodes do you run in your Kubernets Cluster?

What is a typical size of a Kubernetes cluster?

###### How many Kubernetes clusters are you operating?

How many Kubernetes clusters are you operating in total?

###### What Linux Distribution do you run Kubernetes on?

##### Cilium Feature Usage

We are always interested to understand what features are actually used. The
survey focused heavily on new functionality made available as beta feature to
understand what areas we should continue to focus on.

###### What networking mode are you using?

How are our users using Cilium? Are they using direct-routing or tunneling
mode? Are they integrating with other networking components? The first 3
answered were predefined.

###### What type of security policies are you using?

Cilium supports a wide set of security policy features. What are our users
using? The first 4 answers were predefined.

###### How are you managing your security policies?

Everybody's favorite topic: YAML. How are users defining the security policies?
The first 3 answers were predefined.

###### IPv6 vs IPv4?

How is IPv6 usage doing? We will track this metric over time.

###### Are you interested in transparent encryption?

The availability of transparent encryption was announced as beta feature just
before the survey. We wanted to understand how many of our users are interested
in it. The first 3 answers were predefined.

###### Are you interested in multicluster?

Multi-cluster has been introduced a while ago but Cilium 1.4 added support for
global service routing. Who is using it and who is planning to use it?

##### Next Features

###### Pick two features that we should focus on next

What features should we focus on next? We have asked users to pick 2 features
from a predefined list:

###### Additional feedback on the roadmap

In addition to the predefined answers on next features, we wanted to provide an
opportunity to provide more details:

- _SRv6 support_
- _Weighted cluster preference for global services is highly needed_
- _"Integrated ingress" sounds interesting, if it is something that accelerates
 Istio Ingress Gateway. I am also interested in L7-aware routing, NodePort
 support, if it accelerates Istio._
- _Non container workloads_
- _Great but needs to focus on features to work on Direct routing_
- _Interested in bare metal deployment load balancing, e.g. MetalLB_

###### Documentation Gaps

How are we doing on documentation? The first 4 answers were predefined. It is
great to see how users provided very specific feedback.

###### Blog Posts

How are we doing on blog posts. What type of blog posts should we post more?

##### Community

###### Ease of Contributing

Every open source project is only as good as the diversity of its contributors.

We are currently leaving behind a large potential of highly motivated
contributors. We will do better and find ways to encourage a diverse and wide
set of contributors.

###### Bug Reporting Process

How is the bug reporting process working for our users?

##### Personal feedback

###### Additional feedback provided

At the end of the survey, we provided a free-form text box for additional
feedback. Here is a selected list of some of our favourites:

- _Cilium is great, thanks for making it._
- _You guys are doing awesome job, keep making it better.._
- _Cilium rocks!_
- _Love what you are doing guys, keep going!_
- _Concerned about the lack of updates on kops. Currently working around the
 kops-based installation of cilium_

##### Thank You

A big thank you to everyone who filled out the survey. It has helped us in
great length to plan the roadmap of the coming months. We will be repeating the
survey in 6 months interval.

[slack]: https://slack.cilium.io
[cilium-etcd-operator]: https://github.com/cilium/cilium-etcd-operator

#### Cilium 1.5: Scaling to 5k nodes and 100k pods, BPF-based SNAT, and Rolling Key Updates for Transparent Encryption

URL: https://cilium.io/blog/2019/04/24/cilium-15
Date: 2019-04-29
Categories: Release

We are excited to announce the Cilium 1.5 release. Cilium 1.5 is the first
release where we primarily focused on scalability with respect to number of
nodes, pods and services. Our goal was to scale to 5k nodes, 20k pods and 10k
services. We went well past that goal with the 1.5 release and are now
officially supporting 5k nodes, 100k pods and 20k services. Along the way, we
learned a lot, some expected, some unexpected, this blog post will dive into
what we learned and how we improved.

Besides scalability, several significant features made its way into the release
including: BPF templating, rolling updates for transparent encryption keys,
transparent encryption for direct-routing, a new improved BPF based service
load-balancer with improved fairness, BPF based masquerading/SNAT support,
Istio 1.1.3 integration, policy calculation optimizations as well as several
new Prometheus metrics to assist in operations and monitoring. For the
full list of changes, see the [1.5 Release Notes](#15Highlights).

As usual, a big shout out to the entire community of Cilium developers who have
contributed 1336 commits in the last 2 months between version 1.4 and 1.5.

###### What is in this post?

- [Deep Dive into Scalability Improvements](#Scalability)
- [BPF Templating: Avoiding BPF compilations](#BpfTemplating)
- [BPF-based masquerading support: On the last Mile to removing kube-proxy](#BpfMasquerading)
- [Policy Calculation Optimization](#PolicyOptimization)
- [Optimized Load-Balancing](#NewLoadBalancing)
- [1.5 Release Notes](#15Highlights)
- [What's on the roadmap for 1.6](#16Outlook)

##### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms like Kubernetes, Docker, and Mesos.

At the foundation of Cilium is a new Linux kernel technology called BPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. BPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption using X.509 certificates as well as network and service
security. Besides providing traditional network level security, the flexibility
of BPF enables security with the context of application protocols and DNS
requests/responses. Cilium is tightly integrated with Envoy and provides an
extension framework based on Go. Because BPF runs inside the Linux kernel, all
Cilium functionality can be applied without any changes to the application code
or container configuration.

See the section **[Introduction to Cilium](https://cilium.readthedocs.io/en/stable/intro/)** for a more detailed general
introduction to Cilium.

##### Scalability

\_Thank You: we would like to thank Google for providing us with [Google
Cloud](https://cloud.google.com) credits to run extensive scalability tests and

benchmarks.\_

On our quest to unlocking scalability, we have encountered several challenges.
Some of them were expected problems known to most software engineers:

- **Fixed re-sync intervals:** A sanity re-sync of state every 5
 minutes looks harmless at 100 nodes with 0.33 ops/s but grows to 16.6ops/s
 when scaling to 5K nodes.

- **O(n^2) algorithms:** While harmless at a few dozen nodes, a no-go at
 5K nodes or 100k pods.

- **Missing exponential back-off:** These are typically more subtle and often
 come into play after the cluster has successfully scaled up slowly followed
 by sudden synchronized failure across many nodes. Without exponential
 back-off, the load of retries may exceed the capacity of centralized
 resources and a cluster may never recover again.

- **Missing jitter for intervals:** This causes unnecessary alignment of
 actions towards centralized resources such as the Kubernetes apiserver.
 Failures due to these appear randomly and are hard to track down. While
 we never reproduced this type of problem, we introduced jitter into all
 major timer intervals to be safe.

Other challenges were a bit more surprising:

- **Scaling Kubernetes watchers:** The sheer size of some Kubernetes resources
 makes it challenging to scale watchers across large clusters without
 consuming significant network bandwidth. The issue itself was expected, the
 dimension of this problem was a surprise, we measured 400Mbit/s of network
 traffic when running a 2K nodes cluster with the standard node heartbeat
 intervals.

- **etcd panic:** A bug in grpc-go can cause etcd to close an already closed
 go channel resulting in a panic. Above a certain scale, we started seeing
 this very frequently and it caused a lot of disruption.

- **Dealing with apiserver server-side rate-limiting:** This is less of a
 problem with self-managed Kubernetes clusters but all managed Kubernetes
 services we have tested with impose server-side rate-limiting of the
 apiserver. For most of them, the rate limiting is dependent on the cluster
 size. When scaling up quickly, the rate limiting can lag behind
 significantly.

###### New Unlocked Scalability

We have performed a wide set of scale tests to identify and push the limits.
Starting with Cilium 1.5 we will be comfortable to support clusters up to the
following size:

- 5k nodes
- 100k pods
- 20k services[*]

_Note: The scalability of services will depend heavily on whether and how
you are running kube-proxy. Operating kube-proxy in iptables mode can cause
considerable CPU consumption due to requiring several individual iptables rules
for each service backend. It is already possible for Cilium users to remove
kube-proxy entirely if you are not relying on NodePort. Starting with Cilium
1.6, NodePort will be natively support with BPF as well and kube-proxy can be
removed entirely from all nodes._

Can I run bigger clusters? Yes, we don't actually know the exact limit yet as
we did not hit any barries so far in our testing. Resource usage on individual
nodes is pretty flat.

To illustrate this, the following graph shows resource metrics for a 1k nodes
cluster while scaling from 30k pods to 60k pods:

The graph shows min/max/avg CPU spent across all nodes (1 vCPU, 2GB RAM) as
well as the sum of all kvstore operations performed by the entire clsuter. The
CPU consumption on individual nodes is almost flat. This is thanks to the new
BPF templating support described later on this post which avoids expensive
compilation.

As you grow clusters further, additional resources are primarily required for
your centralized services such as the apiserver, Prometheus and the etcd
cluster user by Cilium. In this test, we have been running a dedicated 3 node
etcd cluster for Cilium and the memory consumption of etcd had grown to about
700MB per instance.

What about interactions with the Kubernetes apiserver? The following graph
shows the number of apiserver interactions per second for all nodes combined
while scaling from 30K to 60K pods:

The `GET` calls are done to retrieve pod labels of new pods for network policy
evaluation. The `DELETE` calls are agents removing custom resources with
potential conflicting names. Doing a `DELETE` is cheaper than doing a `GET`
first followed by a `DELETE` if needed. The `POST` and `PATCH` calls are used
to create and update CiliumEndpoint custom resources. By using `PATCH` as
available with k8s \>=v1.13.0, it is no longer required to keep a local copy of
these resources.

###### Glitch in the Matrix: etcd panics

Did you know that for a Go `select` statement with multiple cases, when
multiple communication operations are met, the execution order of the
processing block is random?

It was a bit of a surprise that we were the first to hit this etcd panic as it
was quite simple to reproduce above several hundred nodes. The bug was not in
etcd itself but in grpc-go and has been fixed by [gprpc-go PR#2695](https://github.com/grpc/grpc-go/pull/2695).
[etcd PR#10624](https://github.com/etcd-io/etcd/pull/10624) has been opened as well
to rebase on top of the fixed grpc-go.

###### Scaling Kubernetes Events

Cilium uses the Kubernetes node resource to detect other nodes in the
Kubernetes cluster and pod resources to learn which pod is running on which
node and to derive the pod labels to implement network policies.

As much as we love the standard Kubernetes resources for nodes, they do not
scale very well when attempting to register a standard Kubernetes ListAndWatch
from every ach worker node in a larger cluster. This is for two main reasons:

- The node resource is used to detect stale nodes, each node will update a
 heartbeat field regularly which cause the node resource to change every
 couple of seconds. Each of those changes is being reflected to all
 listeners, in this case several thousand worker nodes. The issue is being
 addressed in Kubernetes via [KEP-009](https://github.com/kubernetes/enhancements/blob/021046fcf0036a7fa051e6b55659acfcaa369fc4/keps/sig-node/0009-node-heartbeat.md)
 using a new NodeLease feature. You can further details about this in the section
 [Node
 Controller](https://kubernetes.io/docs/concepts/architecture/nodes/#node-controller)
 of the Kubernetes documentation.

- As many other resources, the node resource has grown over time. Even with
 ProfoBuf serialization enabled, transmitting the entire node resource to all
 worker nodes on each change of the resource results in significant load in
 bytes transmitted over the network and in CPU cycles spent to serialize and
 deserialize the large resource so often. This is made worse by the fact
 that only some of the fields in the resource are relevant to Cilium so the
 majority of events are sent unnecessarily.

The following graph shows the ratio of events that were necessary or
unnecessary to be received while scaling a GKE cluster from 0 to 2000 nodes
and back:

In order to optimize this, Cilium 1.5 includes a new optimization to allow
offloading events from Kubernetes to a more efficient notification channel via
etcd watchers:

This resolves both of the previously described problems, etcd will only store a
minimal version of the resource with the fields relevant to Cilium and will
thus only distribute an event when significant fields have changed. It also
means that the heavy-weight event is sent once to the single-instance operator
and a light-weight event is distributed to the 5000 worker nodes.

###### Optimizing Kubernetes Resource Caches

Another problem that appears at scale is the amount of memory it requires to
maintain resource caches of standard resources such as nodes, services, pods,
endpoints, and network policies when using the standard [Kubernetes go
client](https://godoc.org/k8s.io/client-go) via interfaces such as
[Informers](https://godoc.org/k8s.io/client-go/informers). The cache will store
the full-blown resource with all fields defined in a local in-memory cache.

By introducing a slimmed down of various Kubernetes resources that only defines
the fields of relevance, the memory consumption of the agent in each node can
be reduced significantly. The graph above shows the difference in memory
consumption in a 2K nodes GKE cluster.

###### Dealing with apiserver rate-limiting

In order to gain visibility into how successfully Cilium interacts with the
Kubernetes apiserver, we have introduced a Prometheus metric which keeps track
of number of interactions per resource as well as track the return codes and
latency as histogram.

The following shows an attempt to scale up from 3 to 5K nodes without any
optimizations. While cloud providers have no problem to provision 5k nodes in
parallel, 5k nodes suddenly hitting the apiserver to register themselves
clearly overloads the apiserver:

The apiserver is so undersized that it gets overwhelmed immediately and calls
to the apiserver will simply timeout and several thousand apiserver calls fail
per second (yellow bars). Note that the graph only shows the interactions as
performed by the apiserver, in parallel, kubelet and kube-proxy will also
interact with the apiserver, adding more load to the system. This is not
represented in this graph.

Shouldn't the apiserver get resized and become more powerful? Yes, but it
requires the cluster to grow first which requires kubelet to be successful in
starting up. This depends on successful interactions with the apiserver. A
classic chicken and egg situation. It also requires kubelet to continue being
successful in updating the heartbeat timestamps of the node resource or the
node will be marked stale again.

The major optimization that we found is to fail less quickly. Standard
software engineering principles typically suggest to fail quickly when
unexpected errors occur such as inability to communicate with the apiserver.
kubelet will restart Cilium and the operation will be retried. This strategy
has a major drawback, during the bootstrap process, Cilium performs multiple
calls to the apiserver to retrieve services, endpoints, nodes, and network
policies. A successful start of Cilium requires all of these calls to succeed.
Assuming that the first 3 calls have succeeded and the fourth fails, Cilium
fails, will be restarted and has to start from the first call again.

With the smarter failure strategy in place, dangerous restart cycles can be
avoided and the apiserver is resized successfully. The new apiserver is now
capable to handle the requests from a load perspective but the rate is above
the server side rate limiting and the apiserver will return HTTP error codes
429 (magenta bars), at the same time, the number of successful apiserver calls
increases until the cluster has been scaled up (orange bars). Once the cluster is
scaled up, the number of interactions returns to the regular idle level.

##### BPF Templating: Avoiding BPF compilations

_Section contributed by Joe Stringer_

Cilium is designed to tailor the datapath forwarding logic for each individual
endpoint, minimizing the number of instructions required to apply security and
determine how to forward traffic to its destination. To achieve this, any
endpoint-specific information such as IP addresses is compiled directly into
the machine-code instructions in BPF, unused features are compiled out, and
security policies specific to the endpoint are implemented via O(1) map
lookups. In prior versions of Cilium, the agent would invoke the C to BPF
compiler each time an endpoint runs, meaning that if you run a hundred
endpoints on a node, that node would spend a lot of time and CPU preparing the
datapath for each of those many endpoints.

In Cilium 1.5, we implemented [BPF
templating](https://linuxplumbersconf.org/event/2/contributions/115/) which
compiles the BPF programs once when Cilium starts up, then injects
endpoint-specific information into the compiled BPF ELF files for each endpoint
when the endpoint starts. This process is about 25x faster than invoking the
compiler every time, and greatly speeds up the deployment of new pods into your
cluster---particularly at high scale. CPU requirements for Cilium are also
drastically decreased as a result. Users running any supported kernel with
Cilium 1.5 will benefit from these changes, but we've been working with the
Linux community to further [improve support for BPF
templating](https://lore.kernel.org/bpf/CAADnVQKaPEi5-vjG_fvt0o8e9b9wAC665++3s48sD2h0qLhoiQ@mail.gmail.com/T/),
which will be part of the Linux 5.2 release later this year.

##### BPF-based masquerading support

_Contributed by Daniel Borkmann_

A native BPF-based SNAT engine has been implemented as part of Cilium's BPF
datapath. This allows to replace the iptables-based NAT machinery through
more efficient, native BPF when the masquerade option is enabled in the Cilium
agent as well as iptables rule setup disabled.

For the initial 1.5 merge, we have enabled Cilium's IPVLAN datapath to use
BPF-based masquerading. Later minor releases will bring Cilium's veth
datapath on parity as well. For ipvlan, this has the unique advantage that
the datapath can be operated with masquerading in the more efficient L3
mode instead of having to fall back to L3S to fix the netfilter symmetry
problem. In the latter case, it is required for masquerading to pass netfilter
hooks in host namespace on ingress as well as egress which is not the case
in L3 mode, and therefore the switch to one of the ipvlan slaves must be
performed at a late point in time out of a custom ipvlan netfilter hook
in L3S mode. Thanks to BPF-based masquerading, these limitations are gone
and ipvlan can operate normally and perform slave-switching directly at
its receive hook out of L3 mode.

The BPF-based masquerading in Cilium works together with Cilium's BPF-based
connection tracker and has been integrated be to attached to external-world
facing devices, but can flexibly be attached to host internal devices as
well. Masquerading is supported for IPv4 as well as IPv6 for protocols TCP,
UDP, ICMP and ICMPv6 right now. For example, for both ICMP protocols, echo
request / reply identifiers are translated and remapped in case of collisions.
The BPF SNAT engine first tries to retain ports or identifier from the packet
in order avoid L4 packet rewrites, but in case of collisions with existing
mappings, it will select a different one. In case of L3 for host-based
connections, it avoids rewrites altogether if the tuple can be retained
as-is. The Cilium daemon has a set of new commands under `cilium bpf nat`
for introspecting the NAT table similarly as with the Cilium connection
tracking tables.

##### Policy Optimizations

_Section contributed by Ian Vernon_

Significant progress has been made in Cilium v1.5 in the policy subsystem. The
policy subsystem is responsible for computing the policy for a given endpoint
by iterating over all rules which select an endpoint, and transforming these
rules into a representation which the BPF datapath and proxies can understand.
Starting in v1.5, we began working on a substantial redesign of how policy is
plumbed for a given endpoint, which will bring large performance gains in
future releases of Cilium. Part of this redesign included the effort to only
compute the policy for an endpoint if truly necessary. Previously, upon any
policy change in Cilium (e.g., CiliumNetworkPolicy being imported) across the
entire cluster, each Cilium agent would re-compute the policy for all endpoints
on its local node, even if the policy which was imported had no effect for some
endpoints. Now, Cilium analyzes the content of the change in rules, and only
regenerates the endpoints which the added rules select. This significantly
reduces the amount of endpoint regenerations which occur on a given node when
there is a high amount of change in the state of rules on a given node. Such a
state is common in the case where rules which contain `toFQDNs` policy that
select an endpoint on the node, as such rules can update the policy repository
on a given node quite frequently as new IPs are resolved via DNS which
correspond to the FQDNs in the rules.

##### New Optimized Load Balancing

_Section contributed by Martynas Pumputis_

Cilium provides a BPF-based loadbalancing capability which can be used to
replace the kube-proxy implementation of services of the `ClusterIP` type. The
former does not rely on the `iptables` and `conntrack` kernel modules.
Therefore, it has a better performance, and it does not suffer from the various
conntrack race conditions which e.g. can cause 5 second timeouts of DNS
requests.

Prior to Cilium 1.5, in the case of a removal of a service endpoint, the
loadbalancing mechanism had to replace the removed endpoint entry in the BPF
maps with any other endpoint. This was needed to prevent from breaking
established connections to the service. Unfortunately, the endpoint entry
duplication had a side-effect which could lead to uneven distribution of
endpoints serving requests.

In Cilium 1.5, we have addressed the problem by reworking the loadbalancer. In
addition, the new loadbalancer no longer needs to duplicate the entries, thus
it has a smaller memory footprint. The design document of the new loadbalancer
can be found
[here](https://docs.google.com/document/d/e/2PACX-1vTWTj_Umicxcp5aBaun30EH6Rb_t5oi1vd18lTpmRx60mI4uBXuJymUj-5BmJ4aa_qmdi0HCpZpe6V_/pub).

To support a smooth migration to the new loadbalancer for existing users, we
maintain both loadbalancers at the same time. New users or the ones who do not
care about breakage of the existing connections to services can disable the
legacy loadbalancer by passing `--enable-legacy-services=false` to
`cilium-agent`. We plan to stop supporting the legacy loadbalancer in Cilium
1.6.

##### 1.5 Release Highlights

- **Scalability:**
 - Own implementation of Kubernetes resource stores with minimal data
 structure definitions to avoid using bloated Kubernetes types to save
 memory at large scale.
 - Ability to hand-over distribution of Kubernetes node and pods events via
 kvstore instead of standard watchers for heavily reduced apiserver traffic.
 (75 vs 5 memory allocations per event).
 - New scalable load-balancing implementation which guarantees fairness as
 service backends scale up & down
 - Use of PATCH instead of UPDATE whenever possible and as supported by
 Kubernetes >= 1.13.
 - Liveness of the apiserver is now primarily derived from the last receiving
 of an event, requiring less pings.
 - Several interval have been made dependent on the overall cluster size using
 a logarithm function to become less aggressive with access to centralized
 resources as the cluster grows.
 - Several garbage collector operations have been moved from the per-node
 agent into the single-instance operator for reduced load on the apiserver.
 - The ability to rate-limit access to the kvstore has been added
 - Jitter has been added to exponential backoff intervals to avoid aligning
 many nodes close to certain timestamps.
 - The cost of keeping the CliumEndpoint CRD up-to-date has been reduced.
 - New connection tracking garbage collector interval tailored to the BPF map
 type in use to minimize CPU overhead and to minimize risk of connections
 being dropped when oeprating at the table limit: 12 hours on new kernels
 when LRU is in use, 15 minutes on older kernels.
- **BPF Templating:**
 - Compilation of BPF programs are now done through the use of templates which
 allow to reuse existing compilation artifacts. This means that scheduling
 of additional pods onto a node no longer requires compilation of BPF
 programs with LLVM. This reduces the CPU overhead and pod spin-up time.
 - Options with large runtime impact are still done at compile time, the cost
 of recompilation is paid once per node and can then be re-used by all other
 pods on the same node.
 - A new CLI tool `bpf template` is available to list and describe templates.
 - Rebase to LLVM 7
- **Transparent Encryption:**
 - Support for rolling key updates
 - Support for direct-routing datapath mode
 - Switch to more efficient GCM-128-AES algorithm by default
 - Ability to encrypt communication of pods not managed by Cilium
 - Automatic MTU correction to account for network header overhead
- **Policy:**
 - More intelligent selection of which endpoints need their policy
 re-calculated on the event of policy rule changes (CiliumNetworkPolicy and
 NetworkPolicy).
- **Networking:**
 - BPF based SNAT in IPVLAN mode which allows to enable masquerading in
 the more efficient IPVLAN L3 mode. This is a drop-in replacement for
 the iptables based masquerading which can only operate at a less efficient
 L3S mode. Later minor releases will bring the Cilium VETH based data
 path on par with supporting BPF based SNAT. A new CLI tool `bpf nat`
 has been added for introspection of NAT mappings.
 - Additional visibility into IPAM layer (pod name, health IPs, router IPs)
- **Istio:**
 - Support for Istio 1.1.3
- **New Prometheus Metrics:**
 - kvstore operation (duration histogram)
 - All BPF system calls (duration histogram)
 - BPF map accesses (duration histogram)
 - API calls made to the agent (duration histogram)
 - All Kubernetes apiserver & events interactions (duration histogram)
 - FQDN proxy & cache statistics
- **Kubernetes:**
 - Kubernetes 1.14 support
 - New optimized initContainer for state cleanup and preflight checks
 - The CNI configuration is no longer removed on pod exit
 - The DaemonSet definition no longer lists all environment variables, the
 ConfigMap is mounted as a directory into the pod and the agent reads the
 directory automatically.

###### Upgrade Instructions

As usual, follow the [upgrade
guide](https://cilium.readthedocs.io/en/v1.5/install/upgrade/#upgrading-minor-versions)
to upgrade your Cilium deployment. Feel free to ping us on
[Slack].

###### Release

- Release Notes & Binaries: [1.5.0](https://github.com/cilium/cilium/releases/tag/1.5.0)
- Container image: `docker.io/cilium/cilium:v1.5.0`

##### Outlook: Roadmap for Cilium 1.6

###### Native AWS ENI allocation support

Native AWS ENI allocation will allow to use AWS ENI addressing in combination
with Cilium. Quite unique to the implementation will be pre-allocation of ENI
adapters and addresses based on quotas as well interaction with the AWS API via
the centralized operator to have control over the rate of interactions with
the API to avoid rate limiting to affect cluster operation and scheduling success
probabilities. For additional details, refer to the
[design document](https://docs.google.com/document/d/1KEdjNrpa9I1DRt4s1ftRIJOwyemCPlbF79ZXfq2biFc/edit?usp=sharing).

###### Optimized Policy Calculation Engine

1.5 has unlocked scalability with regard to number of nodes, pods and services.
While the policy scalability is already much better than traditional solutions,
there is some leftover potential that we will address in 1.6. The policy engine
will become entirely incremental. If you are interested in this topic, join the
SIG-Policy meetings to get involved.

###### Load-balancing for host processes

Load-balancing has only been available to Cilium managed pods so far. With
support for host processes, load-balancing will be done at socket level without
any need for network packet mangling and will also apply to processes in the
init namespace. This will also allow to use Cilium load-balancing in
combination with other CNI plugins without requiring to chain the plugins
together.

###### kube-proxy removal

Cilium 1.5 has introduced the BPF SNAT datapath capability. With Cilium 1.6, we
will provide `--masquerade=true` behavior for non-IPVLAN modes and also provide
a NodePort implementation.

###### CNI Chaining API

1.6 will provide a new API to simplify chaining with other CNI plugins to
integrate Cilium network policy, load-balancing, transparent encryption and
visibility in combination with other CNI plugins. We already have working code
for the following plugins which will be made available together with the new
API.

- [aws-vpc-cni](https://github.com/aws/amazon-vpc-cni-k8s)
- [OpenShift SDN](https://github.com/openshift/openshift-sdn)

If you want to contribute chaining support for additional plugins, feel free to
contact us on [Slack].

###### CRD backing for identity allocation

So far, installation of Cilium required either an external etcd or use of the
etcd-operator to provide an etcd cluster.
[PR7573](https://github.com/cilium/cilium/pull/7573) will provide CRD support
for identity allocation which will make etcd entirely optional for operations.
As outlined in the scalability section, we will still require etcd to guarantee
performance above a certain cluster size.

[slack]: https://slack.cilium.io

#### Deep Dive into Cilium Multi-cluster

URL: https://cilium.io/blog/2019/03/12/clustermesh
Date: 2019-03-18
Categories: How-To

This is a deep dive into ClusterMesh, Cilium's multi-cluster implementation. In
a nutshell, ClusterMesh provides:

- Pod IP routing across multiple Kubernetes clusters at native
 performance via tunneling or direct-routing without requiring any gateways
 or proxies.

- Transparent service discovery with standard Kubernetes services and
 coredns/kube-dns.

- Network policy enforcement spanning multiple clusters. Policies can be
 specified as Kubernetes NetworkPolicy resource or the extended
 CiliumNetworkPolicy CRD.

- Transparent encryption for all communication between nodes in the
 local cluster as well as across cluster boundaries.

The multi-cluster capability is built in layers and you can choose to use all
layers or select and use only the layers you need.

##### Use Cases

Let's review some of the use cases of connecting multiple Kubernetes clusters
before we dive into the implementation details.

###### Use Case: High Availability

High availability is the most obvious use case for most. This use case includes
operating Kubernetes clusters in multiple regions or availability zones and runs
the replicas of the same services in each cluster. Upon failure, requests can
fail over to other clusters. The failure scenario covered in this use case is
not primarily the complete unavailability of the entire region or failure
domain. A more likely scenario is temporary unavailability of resources or
misconfiguration in one cluster leading to inability to run or scale particular
services in one cluster.

###### Use Case: Shared Services

The initial trend of Kubernetes based platforms was to build large,
multi-tenant Kubernetes clusters. It is getting more and more common to build
individual clusters per tenant or to build clusters for different categories of
services, e.g. different levels of security sensitivity.

However, some services such as secrets management, logging, monitoring, or DNS
are often still shared between all clusters. This avoids operational overhead
in maintaining these services in each tenant cluster.

The primary motivation of this model is isolation between the tenant clusters,
in order to maintain that goal, tenant clusters are connected to the shared
services clusters but not connected to other tenant clusters.

###### Use Case: Splitting Stateful and Stateless services

The operational complexity of running stateful or stateless services is very
different. Stateless services are simple to scale, migrate and upgrade. Running
a cluster entirely with stateless services keeps the cluster nimble and agile.
Migration from one cloud provider to another is possible easily.

Stateful services can introduce a potentially complex dependency chain.
Migrating services typically involves the migration of storage.

Running individual clusters for stateless and stateful allows isolating the
dependency complexity to a smaller number of clusters and keeps the stateless
clusters dependency free.

##### Control Plane

###### Requirements

- All Kubernetes worker nodes must be assigned a unique IP address and all
 worker nodes must have IP connectivity between each other.

- All clusters must be assigned unique PodCIDR ranges.

- Cilium must be configured to use [etcd] as the kvstore.

- The network between clusters must allow the inter-cluster communication. The
 exact firewalling requirements will depend on whether Cilium is configured to
 run in direct-routing or tunneling mode.

###### Architecture

The control plane is based on [etcd] and kept as minimalistic as possible:

- Each Kubernetes cluster maintains its own etcd cluster which contains the
 state of that cluster. State from multiple clusters is never mixed in etcd
 itself.

- Each cluster exposes its own etcd via a set of etcd proxies. Cilium agents
 running in other clusters connect to the etcd proxies to watch for changes
 and replicate the multi-cluster relevant state into their own cluster. Use
 of etcd proxies ensures scalability of etcd watchers. Access is protected
 with TLS certificates.

- Access from one cluster into another is _always_ read-only. This ensures
 that the failure domain remains unchanged, i.e. failures in one cluster
 never propagate into other clusters.

- Configuration occurs via a simple Kubernetes secrets resource that contains
 the addressing information of the remote etcd proxies along with the
 cluster name and the certificates required to access the etcd proxies.

##### Pod IP Routing

The pod IP routing is the foundation of the multi-cluster ability. It allows
pods across clusters to reach each other via their pod IPs. Cilium can operate
in several modes to perform pod IP routing. All of them are capable to perform
multi-cluster pod IP routing.

###### Tunneling mode

Tunneling mode encapsulates all network packets emitted by pods in a so-called
encapsulation header. The encapsulation header can consist of a VXLAN or
Geneve frame. This encapsulation frame is then transmitted via a standard UDP
packet header. The concept is similar to a VPN tunnel.

- **Advantage:** The pod IPs are never visible on the underlying network. The
 network only sees the IP addresses of the worker nodes. This can simplify
 installation and firewall rules.

- **Disadvantage:** The additional network headers required will reduce the
 theoretical maximum throughput of the network. The exact cost will depend on
 the configured MTU and will be more noticeable when using a traditional MTU
 of 1500 compared to the use of jumbo frames at MTU 9000.

- **Disadvantage:** In order to not cause excessive CPU, the entire networking
 stack including the underlying hardware has to support checksum and
 segmentation offload to calculate the checksum and perform the segmentation
 in hardware just as it is done for "regular" network packets. Availbility of
 this offload functionality is very common these days.

###### Direct-routing mode

In the direct routing mode, all network packets are routed directly to the network.
This requires the network to be capable of routing pod IPs. Propagation of pod IP
routing information across nodes can be achieved using multiple options:

- Use of the `--auto-direct-node-routes` option which is super lightweight
 route propagation method via the kvstore that will work if all worker nodes
 share a single layer 2 network. This requirement is typically met for all
 forms of cloud provider based virtual networks.

- Using the [kube-router
 integration](http://docs.cilium.io/en/stable/gettingstarted/kube-router/) to
 run a BGP routing daemon.

- Use of any other routing daemon that injects routes into the standard Linux
 routing tables (bird, quagga,...)

When a point is reached where the network no longer understands pod IPs,
network packet addresses need to be masqueraded.

- **Advantage:** The reduced network packet headers can optimize network
 throughput and latency.

- **Disadvantage:** The entire network must be capable of routing pod IPs
 which can increase the operational complexity.

###### Hybrid-routing mode

The hybrid-routing mode enables the use of direct-routing when available which
will typically be in the local cluster or other clusters in the same VPC with a
fall-back to tunneling mode when spanning VPCs or cloud-providers. This can
limit the operational complexity and allows to pay the cost of optimization
only when needed.

##### Service Discovery

The service discovery of Cilium's multi-cluster model is built using standard
Kubernetes [services] and designed to be completely transparent to existing
Kubernetes application deployments:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: rebel-base
  annotations:
    io.cilium/global-service: 'true'
spec:
  type: ClusterIP
  ports:
    - port: 80
  selector:
    name: rebel-base
```

- Cilium monitors Kubernetes [services] and endpoints and watches for
 services with an annotation `io.cilium/global-service: "true"`. For such
 services, all services with identical name and namespace information are
 automatically merged together and form a global service that is available
 across clusters.

- Any traffic to a ClusterIP of a global service will automatically be
 load-balanced to endpoints in all clusters based on the standard Kubernetes
 health-checking logic.

- Each cluster continues to maintain its own ClusterIP for each service which
 means that Kubernetes and kube-dns/coredns are not aware of others
 clusters. The DNS server continues to return a ClusterIP valid only in the
 local cluster and Cilium will perform the load-balancing transparently.

- Several additional annotations exist for fine-grained control such as
 unidirectional exposure or affinity policies.

All traffic from `frontend-1` to ClusterIP `30.1.1.1` will automatically be
load-balanced to pod IPs of backends of cluster 1 `[10.0.0.1, 10.0.0.2]` as
well as the pod IPs backends in cluster 2 `[20.0.0.1, 20.0.0.2]`. Each cluster
will perform health-checking of the local backend instances and notify other
clusters as pods appear, disappear or become unhealthy.

##### Transparent Encryption

The transparent encryption introduced in [Cilium
1.4](https://cilium.io/blog/2019/02/12/cilium-14/) is compatible with
multi-cluster. Make sure to configure all nodes across all clusters with a
common key and all communication between nodes is automatically encrypted.

##### Multi-cluster network policy

The short version is that policy enforcement that you are familiar with from a
single cluster will simply expand and work across clusters. Due to the fact
that policies are specified using pod labels, a policy that allows `frontend`
to talk to `backend` will apply to traffic with the cluster just as it will
to traffic crossing clusters.

Cilium does _not_ automatically propagate NetworkPolicy or CiliumNetworkPolicy
across clusters. It is the responsibility of the user to import the policy into
all clusters. This is intentional as it means that each cluster gets to decide
whether the cluster is permitted to receive communication from remote clusters
or emit communication to remote clusters.

###### Allowing specific cluster cross-over paths

It is possible to establish policies that apply to pod in particular clusters
only. The cluster name is represented as a label on each pod by Cilium which
allows to match on the cluster name in both the `endpointSelector` as well as
the `matchLabels` for `toEndpoints` and `fromEndpoints` constructs:

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'allow-cross-cluster'
  description: 'Allow x-wing in cluster1 to contact rebel-base in cluster2'
spec:
  endpointSelector:
    matchLabels:
      name: x-wing
      io.cilium.k8s.policy.cluster: cluster1
  egress:
    - toEndpoints:
        - matchLabels:
            name: rebel-base
            io.cilium.k8s.policy.cluster: cluster2
```

The above example policy will allow `x-wing` in cluster1 to talk to
`rebel-base` in cluster2. X-wings won't be able to talk to rebel bases in the
local cluster unless additional policies exist that whitelist the
communication.

##### Relation to Istio multicluster

Both projects are independent but can complement each other nicely. A common
way to combine Cilium and Istio multi-cluster is to use Cilium's multi-cluster
Pod IP routing layer to fulfill the following requirement of the [Istio
Multicluster
guide](https://istio.io/docs/setup/kubernetes/multicluster-install/):

> All pod CIDRs in every cluster must be routable to each other.

Further, the Cilium policy enforcement capability can be used to secure
communication to and from the Istio control plane as well as protect sidecar
bypass attempts via unsupported protocols such as UDP or IPV6 as well as
protect from compromised sidecar proxies.

It is also possible to run a mix of global Istio services and Cilium global
services side by side. Cilium's global services are reachable from all Istio
managed services as they can be discovered via DNS just like regular services.

##### Getting Started

To get started, follow the step-by-step [ClusterMesh
tutorial](http://docs.cilium.io/en/stable/gettingstarted/clustermesh/) which
guides you through the process of connecting clusters together. Make sure to
join our [Slack] channel to ask questions and brag about your setup.

##### Further Material & References

- [Adrien Trouillaud](https://github.com/adrienjt) has notified us on Slack
 that the
 [multicluster-scheduler](https://github.com/admiraltyio/multicluster-scheduler)
 has integrated Cilium multi-cluster support. Make sure to have a look.

- KubeCon US 2019 session: [Connecting Kubernetes Clusters Across Cloud
 Providers](https://docs.google.com/presentation/d/1siR8tPoIEbE0itbdoKdLKbkTS9nYV-2Wj0QIxJqRI7M/edit?usp=sharing)

- [ClusterMesh tutorial](https://istio.io/docs/setup/kubernetes/multicluster-install/)

- [Cilium 1.4 release blog](https://cilium.io/blog/2019/02/12/cilium-14/)
 which was the release that introduced multi-cluster service discovery.

[slack]: https://slack.cilium.io
[etcd]: https://github.com/etcd-io/etcd
[services]: https://kubernetes.io/docs/concepts/services-networking/service/

#### Cilium 1.4: Multi-Cluster Service Routing, DNS Authorization, IPVLAN support, Transparent Encryption, Flannel Integration, Benchmarking other CNIs,...

URL: https://cilium.io/blog/2019/02/12/cilium-14
Date: 2019-02-12
Categories: Release

We are excited to announce the Cilium 1.4 release. The release introduces
several new features as well as optimization and scalability work. The
highlights include the addition of global services to provide Kubernetes
service routing across multiple clusters, DNS request/response aware
authorization and visibility, transparent encryption (beta), IPVLAN support for
better performance and latency (beta), integration with Flannel, GKE on COS
support, AWS metadata based policy enforcement (alpha) as well as significant
efforts into optimizing memory and CPU usage.

As usual, a big shout out to the entire community of Cilium developers who have
contributed 1048 commits in the last 4 months between version 1.3 and 1.4.

##### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms like Kubernetes, Docker, and Mesos.

At the foundation of Cilium is a new Linux kernel technology called BPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. BPF is utilized to provide functionality
such as multi-cluster routing, load balancing to replace kube-proxy,
transparent encryption using X.509 certificates as well as network and service
security. Besides providing traditional network level security, the flexibility
of BPF enables security with the context of application protocols and DNS
requests/responses. Cilium is tightly integrated with Envoy and provides an
extension framework based on Go. Because BPF runs inside the Linux kernel, all
Cilium functionality can be applied without any changes to the application code
or container configuration.

See the section **[Introduction to Cilium](https://cilium.readthedocs.io/en/stable/intro/)** for a more detailed general
introduction to Cilium.

##### Multi-Cluster Service Routing

Cilium 1.3 introduced the basic pod IP routing capability between multiple
clusters. Cilium 1.4 is introducing the concept of global services based on
standard Kubernetes services. Global services allow a user to nominate a
Kubernetes service to be available in multiple clusters. That service can then
have backend pods in multiple clusters.

The user experience is as simple as defining a Kubernetes service with
identical name and namespace in each cluster and adding an annotation to mark
it as global.

The Kubernetes health-check information is leveraged to add and remove service
backends automatically as pods scale up and down or become unhealthy.

The control plane is built on top of etcd similar to how Kubernetes itself
operates with resiliency and simplicity as its foundational design pattern.
Each cluster continues to operate its own etcd cluster and replication happens
on a read-only basis which ensures that failures in a cluster do not impact
other clusters.

Connecting clusters together is as simple as providing routing between VPCs
using standard routing APIs of cloud providers or on-prem infrastructure via
regular IPSec based VPN gateways and tunnels and then exposing the Cilium
control plane via an internal Kubernetes Loadbalancer to expose it to the
internal VPCs. TLS is used to authenticate the client and server with the
certificates and keys managed as Kubernetes secrets.

##### IPVLAN support (beta)

A new IPVLAN based datapath mode has been added. IPVLAN has latency advantages
over veth based architectures. The following benchmarks have been measured with
netperf between two local containers on a 3.40Ghz Xeon with hyper-threading
disabled using a single core. The 99th percentile latency is considerably
lower with IPVLAN compared to veth (lower is better):

The maximum throughput (higher is better) is quite similar between IPVLAN and
veth but a very noticeable performance gain can be achieved by compiling out
netfilter/iptables from the kernel. Running your Kubernetes cluster completely
iptables free is already possible if you are not using NodePort services and do
not require to masquerade network traffic as it leaves a Kubernete worker node.
We'll provide a guide on how to run iptables and kube-proxy free in the next
couple of weeks.

IPVLAN is a beta level feature in 1.4, see the [IPVLAN getting started guide](https://docs.cilium.io/en/v1.4/gettingstarted/ipvlan/) for instructions
on how to enable and configure the feature.

##### DNS Request/Response Aware Security & Visibility

Cilium 1.4 extends the existing DNS security policy model to be aware of the
DNS requests that individual pods issue and the DNS responses they receive.
This significantly improves the security of pods accessing services outside of
the cluster:

- Pods can be restricted to have minimal privileges when performing DNS
 lookups, i.e. pod can be limited to only succeed in looking up DNS names
 matching a pattern such as `*.domain.com`. Any request outside of the allowed
 pattern will receive a `request refused` DNS response in return.

- The communication following the DNS lookup can be limited to the IP addresses
 as returned in the DNS response that the specific pod received. This reduces
 the privileges of a compromised application significantly and improves the
 reliability of DNS based policy rules as the enforcement logic no longer
 requires to know about all possible IP addresses that a DNS name can map to.

 In particular for popular storage, messaging, and database services offered
 by cloud providers, a single DNS name can map to hundreds or thousands of IP
 addresses.

- DNS lookups and responses are now logged via the Cilium authorization
 logging layer that is accessible via an API. This provides an exact log of
 every DNS request and response that a pod has performed.

The above example shows a successful DNS sequence followed by an HTTP request
to the IP as responded by the DNS server. This is how an application is
expected to behave and what is permitted. Subsequent HTTP requests can use the
cached DNS information, allowing such requests. The DNS information will time
out according to TTL information in the record.

On the right is a sequence where the application is performing a DNS lookup
outside of the allowed DNS policy. It also shows that if the application fails
to perform a DNS lookup, any attempt to contact the IP address, even if the IP
address actually maps to a DNS name which is permitted, will be blocked if the
application failed to lookup the DNS name at some point.

###### Policy Example

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'egress-domain-wildcard'
spec:
  endpointSelector:
    matchLabels:
      app: myService
  egress:
    - toEndpoints:
        - matchLabels:
            'k8s:io.kubernetes.pod.namespace': kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: UDP
          rules:
            dns:
              - matchPattern: '*.domain.com'
    - toFQDNs:
        - matchPattern: '*.domain.com'
      toPorts:
        - ports:
            - port: '443'
              protocol: TCP
```

The above policy example grants a pod or container the privilege to perform DNS
requests via kube-dns but limits the allowed DNS lookups to `*.domain.com`. A
request not matching the pattern will receive a `request refused` DNS response.
It further grants the pod egress access on port 443/TCP to the IPs returned in
the DNS response. Any attempt to access any IP address not previously returned
in a DNS response is rejected.

To get started using DNS-based policies, follow the [DNS-based getting started
guide](https://docs.cilium.io/en/v1.4/gettingstarted/dns/).

##### Transparent Encryption & Authentication (beta)

Providing transparent encryption for all service-to-service communication
within a cluster and across clusters has been a frequently requested feature.
The encryption allows to run Kubernetes in untrusted networks transparently
encrypting all communication between services in the cluster. The
authentication ensures that only trusted worker nodes can participate in the
cluster.

The encryption is based on X.509 certificates and keys. For now, a PSK is used
that is distributed to all nodes using a Kubernetes secret. However, the
infrastructure is compatible with SPIFFE and allows to provide service
authentication using SPIFFE certificates on a per service level in a future
release. The datapath implementation uses the IPSec implementation of the Linux
kernel which avoids the need to run sidecar proxies as part of all services and
ensures efficient and automatic use of hardware-assisted crypto acceleration
via specialized CPU instruction sets as found in modern processors.

Transparent encryption is a beta level feature. To enable the feature, pass
`--enable-ipsec` option to the agent and provide a pre shared key (PSK) via
the `--ipsec-key-file` option or by using a Kubernetes secret.

##### Sockmap BPF based sidecar acceleration (alpha)

As [announced at KubeCon](https://www.youtube.com/watch?v=ER9eIXL2_14), we are
making the local process communication acceleration available with Cilium 1.4.

Sockmap accelerated local process communication is primarily useful for
communication between sidecar proxies and local processes but applies to all
local processes.

The number of requests/s and maximum throughput both double when sockmap is
enabled:

Note that all of these performance numbers are per CPU core.

Sockmap acceleration is an alpha level feature in 1.4. It can be enabled using
the `--sockops-enable` option.

##### New Grafana Dashboard

Several new Prometheus metrics have been added and a new Grafana dashboard is
available that can be deployed into any Kubernetes cluster with a single
command:

```
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.4/examples/kubernetes/addons/prometheus/monitoring-example.yaml
```

##### Flannel Integration (beta)

Integration with Flannel has been a frequent ask by users looking for the
security policy enforcement and load-balancing capability of Cilium for
existing clusters configured using the Flannel CNI plugin.

Cilium 1.4 introduces a new configuration option:

```
flannel-master-device: "cni0"
```

which enables Cilium to run on top of Flannel using CNI chaining. It is also
possible to automatically pick up existing containers/pods on worker nodes by
enabling the following option:

```
flannel-manage-existing-containers: "true"
```

The option also requires to edit the Cilium DaemonSet to enable the `hostPID: true` option so Cilium can see all processes to attach to existing containers.

The Flannel integration is primarily useful to try out Cilium functionality in
existing clusters or for migration purposes. Several features which will help at
large scale will not function, this includes the ability to embed the security
identity of the source into network packets which requires to fall back to IP
based identification.

For more details, see the [flannel getting started guide](https://docs.cilium.io/en/v1.4/gettingstarted/flannel-integration/)

##### Benchmarking Other CNIs

We have been approached a lot in the last two months how Cilium compares to
other CNI plugins. As a result, we have run a couple of benchmarks against
other popular CNI plugins.

A couple of words before we get into the actual numbers:

- Benchmarking is hard. We do not claim that we can configure other CNI
 plugins in the ideal configuration. Approach us if you have input and we are
 happy to adjust. The goal of these benchmarks is to show that the most
 impact can be made by changing architecture and using different technologies
 in the stack rather than just comparing one CNI against another. Even Cilium
 will behave very differently depending on the configuration.

- We have focused on measuring the networking overhead so we are running the
 benchmarks between two local containers to remove as many hardware
 limitations as possible.

- The goal is not to get to the largest or lowest number possible. We are
 measuring using a single CPU core which arbitrarily limits the number.
 Larger systems will be able to achieve much better absolute numbers. Focus
 on the difference between the numbers, not the absolute numbers.

- As usual, take any benchmark with several kilos of salt. Benchmarking is
 always done in a particular context. Understanding that context is
 important. If something is unclear about the numbers we publish here,
 approach us and we will clarify it.

That said, let's dig into the numbers:

The above numbers show the various latency measurement when two containers
exchange as many 1 byte request and response messages over a single connection
as possible. This test primarily shows whether a particular forwarding path is
heavily favoring throughput over latency.

- Cilium Sockmap is heavily exploiting its advantage of being able to operate
 on socket level. This only works on connections within the node.

- The next best category is Cilium operating in IPVLAN mode with
 netfilter/iptables completely removed. There is a difference whether
 Cilium is operating with security policy rules loaded or not, but that
 difference is minor. This is due to the efficient per-CPU hash tables used
 for policy enforcement which minimize the overhead. Note that this number
 already includes a load balancing BPF map lookup so this mode allows to
 replace kube-proxy which other tests are not accounting for.

- Next is Flannel and Cilium operating in veth mode. Flannel is a minimal
 networking plugin using the Linux routing tables. The minimalism pays off but
 it also means that Flannel can not perform any policy enforcement and has to
 rely on kube-proxy in either iptables or IPVS mode. Cilium is doing slightly
 worse due to performing some work to enable policy enforcement in the middle
 of a connection even if no policy rule was previously loaded.

- Calico showed slightly more overhead in our tests. Likely due to more iptables
 rules being added and more netfilter chains being utilized. We have not
 loaded any policy rules into Calico for this particular test but assume that
 the use of ipset will allow to scale OK. Not quite as good as the per CPU
 hash tables.

Typical enemies of these benchmarks are:

- Context switches between kernel and user space. These numbers will get
 **much** worse when an L4/L7 proxy gets involved.
- Any per-packet overhead has dramatic effects. Cold caches and data
 structures can negative impact as well. The less code that has to be
 traversed, the better.

The above graph shows the requests per second for the same benchmark being
performed. The requests per second overlap with the latency pretty well. As for
the previous tests, these numbers are measured per CPU core.

The last graph illustrates the opposite of the spectrum. The TCP_STREAM test is
attempting to pump as many bytes as possible over a single TCP connection. This
is where memory bandwidth can come into play and where network hardware or
cloud provider limits can often artificially limit the benchmark.

- Leaving Sockmap aside for a second, we can see that the IPVLAN mode has
 clear advantage over all other modes.

- We expected Calico to have similar numbers to Cilium so we have likely
 misconfigured something. Any help is appreciated. Calico doing worse in the
 TCP_STREAM does not really make a lot of sense, as the iptables overhead in
 this test is amortized across a larger amount of data.

Adding Sockmap back into the picture proves the performance benefit of doing
networking at socket level. Again, this benefit is only gained between local
processes as it occurs regularly when sidecar proxies are in effect or when
services are scheduled onto the same node for improved local communication.

##### GKE Support with COS

A completely new
[guide](https://docs.cilium.io/en/v1.4/gettingstarted/k8s-install-gke/)
documents how to run Cilium on GKE using COS. A brand new [node-init
DaemonSet](https://github.com/cilium/cilium/tree/master/examples/kubernetes/node-init)
enables to prepare GKE nodes by mounting the BPF filesystem and reconfiguring
kubelet to run in CNI mode. Use of the [cilium-etcd-operator] provides the
kvstore requirement while keeping the installation simple.

##### 1.4 Release Highlights

- **Multi Cluster**
 - Addition of global services which enable spanning Kubernetes services
 across multiple Kubernetes services via an annotation. (beta)
 - Much improved installation guide including tooling to automatically extract
 SSL certificates when the [cilium-etcd-operator] is used.
- **Transparent encryption (beta)**
 - Encryption of all pod/host to pod/host communication using IPsec with a PSK.
 - IPv4 and IPv6
 - PSK configured via Kubernetes secret
 - No app or pod modifications required.
- **IPVLAN support (beta)**
 - A new alternative datapath mode utilizing IPVLAN replaces the use of veth
 pairs for improved latency and performance.
- **DNS Request/Response Authorization**
 - FQDN-based security policies are now enforced based on actual DNS requests
 and responses performed by each service.
 - Ability to specify policy on what DNS requests can be performed by services.
 - API and CLI to query FQDN host name lookups performed by individual
 endpoints
 - Ability to restore FQDN mapping on agent restore for persistent mapping
 - Configurable minimum TTL and maximum FQDN hosts per endpoint
- **Flannel Integration (beta)**
 - Ability to run Cilium on top of Flannel. Flannel provides networking and
 Cilium provides load-balancing and policy enforcement.
 - Ability to hook onto an existing Flannel deployment without restarting any
 pods.
- **AWS metadata-based policy enforcement (alpha)**
 - Ability to specify policy rules based on AWS metadata such as EC2 labels,
 security group names, VPC names, Subnet names, etc.
- **Additional Metrics & Monitoring**
- **Networking**
 - New simple PodCIDR route propagation mode via the kvstore. Enable via
 ``--auto-direct-node-routes```.
 - IPv6 is now disabled by default for new installations. Existing ConfigMaps
 will continue with current behavior. Enable via `--enable-ipv6=true`.
 - Ability to run IPv6 only cluster without allocating any IPv4 addresses with
 the `--enable-ipv4=false` option.
 - Improved persistent behavior of the load balancer
 - BPF sockmap support to accelerate local process communication. Available
 via option `--sockops-enable` (alpha)
 - Decoupled endpoint identified from IP addresses to support arbitrary IP
 addressing models.
- **Efficiency & Scale**
 - Heavily improved scalability of the CiliumEndpoint CRD. It is no longer
 required to disable CEP for large deployments.
 - Introduction of per-node local identities for CIDR/DNS based rules
 which do not require cluster or global scope. A pod performing a DNS
 request on node-23 resulting in an IP being whitelisted for that pod no
 longer has any impact on other nodes in clusters.
 - IPv6 is now disabled by default to reduce the memory footprint in small
 deployments.
 - BPF map pre-allocation is now disabled by default to reduce the memory
 footprint in small deployments.
 - Single binary for agent and client command to reduce container image size.
 - Compile bugtool as static binary
 - New cilium-operator to provide singleton tasks such as CEP garbage
 collection.
 - Synchronous pod label retrieval on CNI ADD. This can slow down the pod
 scheduling rate a bit but avoids policy drops on pod start without init
 policies.
 - The status probe now collects state concurrently for improved accuracy.
 - Better signal handling on termination and new terminationGracePeriodSeconds
 default of 1 second to minimize downtime of agent
- **Kubernetes**
 - Added support for Kubernetes 1.13
 - Support for new CRI-O versions which automatically mount the BPF fileystem
 - New NodeInit DaemonSet to automatically prepare GKE nodes for Cilium
 installation. This enables use of COS and auto-scaling.
 - The cilium-operator now automatically restarts kube-dns when kube-dns is
 not managed by Cilium. This simplifies the initial installation on managed
 Kubernetes offerings.
- **Istio**
 - Improved Istio integration
- **Observability**
 - New metrics: kvstore operations, proxy upstream/processing latency,
 forwarded and dropped bytes, node events, number of nodes,
- **Documentation**
 - The standard installation is now using the [cilium-etcd-operator] and no
 longer depends on the user providing an external kvstore.
 - New GKE guide including COS support
 - Simplified EKS guide using eksctl
 - Improved cluster mesh guide using automated tooling

##### Upgrade Instructions

As usual, follow the [upgrade guide](https://cilium.readthedocs.io/en/v1.4/install/upgrade/#upgrading-minor-versions)
to upgrade your Cilium deployment. Feel free to ping us on
[Slack].

##### Release

- Release Notes & Binaries: [1.4.0](https://github.com/cilium/cilium/releases/tag/1.4.0)
- Container image: `docker.io/cilium/cilium:v1.4.0`

[slack]: https://slack.cilium.io
[cilium-etcd-operator]: https://github.com/cilium/cilium-etcd-operator

#### Cilium 1.4 Preview: Multi-Cluster Service Routing, DNS Authorization, and Transparent Encryption

URL: https://cilium.io/blog/2018/12/10/cilium-14-preview
Date: 2018-12-10
Categories: Release

As we all enjoy a wonderful week at KubeCon 2018 US, we want to provide a
preview into the upcoming Cilium 1.4 release. We are days away from 1.4.0-rc1
which will allow for community testing of a lot new exciting functionality.
Some of the highlights:

- Multi-Cluster service routing using standard Kubernetes services.
- DNS Authorization with DNS request/response aware security policy enforcement
 to restrict the DNS names a pod can lookup as well as limit the egress
 connectivity to the IPs returned in the DNS response of that particular pod.
- Transparent encryption and authentication for all service to service
 communication using X.509 certificates.

As always, we love hearing from you, so stop by our KubeCon booth and chat with
us and other Cilium users.

##### Multi-Cluster Service Routing

Cilium 1.3 has introduced the basic pod IP routing capability between multiple
clusters. Cilium 1.4 is introducing the concept of global services based on
standard Kubernetes services. Global services allow a user to nominate a
Kubernetes service to be available in multiple clusters. That service can then
have backend pods in multiple clusters.

The user experience is as simple as defining a Kubernetes service with
identical name and namespace in each cluster and add an annotation to mark it
as global.

The Kubernetes health-check information is leveraged to add and remove service
backends automatically as pods scale up and down or become unhealthy.

The control plane is built on top of etcd similar to how Kubernetes itself
operates with resiliency and simplicity as its foundational design pattern.
Each cluster continues to operate its own etcd cluster and replications happens
on a read-only basis which ensures that failures in a cluster do not impact
other clusters.

Connecting clusters together is as simple as providing routing between VPCs
using standard routing APIs of cloud providers or on-prem infrastructure via
regular IPSec based VPN gateways and tunnels and then expose the Cilium control
plane via an internal Kubernetes Loadbalancer to expose it to the internal
VPCs. TLS is used to authenticate the client and server with the certificates
and keys managed as regular Kubernetes secrets.

##### DNS Request/Response Aware Security & Visibility

Cilium 1.4 extends the existing DNS security policy model to be aware of the
DNS requests that individual pods issue and the DNS responses they receive.
This significantly improves the security of pods accessing services outside of
the cluster:

- Pods can be restricted to have minimal privileges when performing DNS
 lookups, i.e. pod can be limited to only succeed in looking up DNS names
 matching a pattern such as `*.domain.com`. Any request outside of the allowed
 pattern will receive a `request refused` DNS response in return.

- The communication following the DNS lookup can be limited to the IP addresses
 as returned in the DNS response that the specific pod received. This reduces
 the privileges of a compromised application significantly and improves the
 reliability of DNS based policy rules as the enforcement logic no longer
 requires to to know about all possible IP addresses that a DNS name can map
 though.

 In particular for popular storage, messaging, and database services offered
 by cloud providers, a single DNS name can map to hundreds or thousands of IP
 addresses.

- DNS lookups and responses are now logged via the Cilium authorization
 logging layer that is accessible via an API. This provide an exact log of
 every DNS request and response that a pod has performed.

The above example shows a successful DNS sequence followed by an HTTP request
to the IP as responded by the DNS server. This is how an application is
expected to behave and what is permitted. Subsequent HTTP requests can use the
cached DNS information, such requests will be allowed as well. The DNS
information will time out according to TTL information in the record.

On the right is a sequence where the application is performing a DNS lookup
outside of the allowed DNS policy. It also shows that if the application fails
to perform a DNS lookup, any attempt to contact the IP address, even if the IP
address actually maps to a DNS name which is permitted, will be blocked if the
application failed to lookup the DNS name at some point.

###### Policy Example

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'egress-domain-wildcard'
spec:
  endpointSelector:
    matchLabels:
      app: myService
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: UDP
          rules:
            dns:
              - matchPattern: '*.domain.com'
    - toFQDNs:
        - matchPattern: '*.domain.com'
      toPorts:
        - ports:
            - port: '443'
              protocol: TCP
```

The above policy example grants a pod or container the privilege to perform DNS
requests via kube-dns but limits the allowed DNS lookups to `*.domain.com`. A
request not matching the pattern will receive a `request refused` DNS response.
It further grants the pod egress access on port 443/TCP to the IPs returned in
the DNS response. Any attempt to access any IP address not previously returned
in a DNS response is rejected.

##### Transparent Encryption & Authentication

Providing transparent encryption for all service to service communication
within a cluster and across clusters has been a frequently requested feature.
The encryption allows to run Kubernetes in untrusted networks transparently
encrypting all communication between services in the cluster. The
authentication ensures that only trusted worker nodes can participate in the
cluster.

The encryption is based on X.509 certificates and keys which provides
compatibility with the SPIFFE service specification to provide accelerated
service authentication in the future. The datapath implementation uses the
IPSec implementation of the Linux kernel which ensures efficient and automatic
use of hardware assisted crypto acceleration via specialized CPU instruction
sets as found in modern processors.

##### And a lot more

This is only a small glimpse of the overall effort that is going into the 1.4
release. The release will also include a great amount of incremental
improvement across the board such as becoming smarter in interacting with the
Kubernetes apiserver, being more intelligent in how aggressively do BPF map
pre-allocation by default to balance out performance considerations over memory
consumption, perform dynamically sizing of data structures based on the
available system memory to draw conclusions about the expected number of
connections the node will have to handle, simplified installations via an
improved etcd operator to provide etcd managed by Cilium itself and much more.
The full 1.4 release blog will include all of the details for those interested.

If you haven't, join our [Slack] channel to provide feedback and connect with
the community.

See you all at KubeCon in Seattle!

[slack]: https://slack.cilium.io

#### Analyzing the CNI performance benchmark

URL: https://cilium.io/blog/2018/12/03/cni-performance
Date: 2018-12-03
Categories: How-To

First of all, huge shout-out to [Alexis
Ducastel](https://twitter.com/infrabuilder) for putting together a great CNI
benchmark comparison. To be honest, there was definitely a moment of panic when
we saw the article pop up. Did we just miss a major performance regression?

This blog post documents the investigation we have done so far of what looked
like a performance regression of HTTP/FTP traffic over pure TCP.

Alexis was super quick to share the
[scripts](https://github.com/InfraBuilder/k8s-cni-benchmark) that he used to
collect the benchmarks numbers. This not only allowed for a quick verification
but also allows us to integrate this into our CI tests and run it alongside of
the existing benchmarks for better coverage.

##### Summary

- The external etcd is no longer a requirement. With 1.3, we started providing an
 etcd-operator to have Cilium maintain its own etcd. We are still
 investigating a CRD-based replacement as well but are still seeing a
 significant scalability difference when running >2000 pods or >100 nodes. We
 leverage atomic transactions supported by etcd to avoid distributed locks and
 it heavily benefits the scalability of policy enforcement.

- The article shows lower HTTP and FTP throughput compared to TCP. We feared
 this to be a BPF related performance regression but it turned out to be
 caused by how curl measures download speed in combination with Cilium's
 optimistic pod scheduling strategy which treats CNI requests mostly
 asynchronously. This allows a pod to start running before the networking has
 been setup while ensuring that policy enforcement is always enforced
 correctly. This helps reduce the time to schedule a pod and thus allows to
 schedule pods quicker. The consequence is that the TCP connection started by
 curl to be established later. Unfortunately, curl measures the time of the
 transfer from the initial connection attempt (unlike iperf3 and netperf) and
 thus includes the time in which the connection is waiting to be established
 to the total transfer time. This leads to a lower bitrate to be reported.

 The latest development tree already had this behavior changed but the change
 was not released yet. Cilium versions 1.2.6, 1.3.1 and 1.4.0 will include the
 commit and change this default setting to block the CNI call until all
 network plumbing has been completed. We will make the functionality of
 asynchronous network bootstrapping available to advanced users via an option.

- The article shows minor differences between raw TCP throughput of individual
 CNI plugins. The benchmarks of all TCP/UDP tests are limited by the speed of
 the 10Gbit/s network connectivity except for the encrypted test via Weave.
 There is a difference in theoretical payload maximum depending on
 whether a CNI plugin uses encapsulation or direct routing mode. In the
 article, Alexis is running Cilium in encapsulation mode which will lower the
 maximum achievable payload rate slightly because more of the available
 network bandwidth is used for network headers instead of payload. Multiple
 CNI plugins can be put in either encapsulation or direct routing mode and I
 would expect the numbers to change slightly accordingly.
 _Update:_ According to our friends working on Weave, the Weave performance in
 encryption mode is bad because the MTU is misconfigured and does not take
 into account the ESP headers. This causes fragmentation which lowers the
 throughput significantly. If the MTU is configured to be 8196 to account for
 VXLAN + ESP, the reported throughput will be a lot better.

 As a side note: While testing, we observed that the kernel being used by
 Alexis (4.15) has a known performance regression which is fixed by [this
 upstream kernel commit](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9c4c325252c54b34d53b3d0ffd535182b744e03d).
 It's observable by comparing the number of TCP retransmissions required to
 transfer the data. This bug is not Cilium specific though, all CNI plugins
 that require a network packet to cross a network namespace boundary will
 suffer from this. The effect is not big enough to have meaningful impact in
 this 9000 MTU benchmark test but adds some variance to the test results.

- The memory consumption is due to aggressively pre-allocating BPF maps in the
 in favour of providing lowest average latency possible. There is a PR under
 discussion to change the default pre-allocation strategy become less
 aggressive. Another PR is out to disable IPv6 by default to save additional
 memory resource usage by default.

- The only number we could not directly reproduce is the 5.5% CPU consumption.
 In our tests, iperf3 or curl consume significantly more CPU on their own and
 the CPU usage of Cilium and kernel is insignificant to everything else.

- The frequently requested transparent encryption will be added
 in Cilium 1.4. The work has been underway for a while and will be merged in
 the next couple of weeks.

##### TCP and UDP benchmarks

The first tests performed are TCP and UDP benchmarks using iperf3 by
transmitting random data as quickly as possible for 2 minutes. The tests are
run in a bare metal setup connected by 10Gbit/s hardware. As for all tests, an
MTU of 9000 bytes is used. The MTU defines the maximum size of a single packet
on the wire. By setting this to the largest possible number that the network
hardware supports, the ratio between data payload and network packet headers
can be optimized. This means that for a given available network bandwidth, more
of it can be used for actual data. Large MTUs (jumbo frames) work well within
datacenters but is much harder to achieve on the Internet where the MTU is
typically 1500 and below. Large MTUs put the least stress on the networking
layer and more stress on the application and the network hardware or the memory
bus typically becomes the bottleneck.

Let's look at the numbers collected:

It would be quite surprising if any CNI plugin cannot achieve network wire
speed in this scenario. Why? Let's look at the CPU utilization of the sending
machine while 10Gbit/s are being transmitted with Cilium.

```
top - 14:25:54 up 19:33,  1 user,  load average: 1.16, 0.61, 0.30
Tasks: 419 total,   2 running, 223 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 98.4 id,  0.0 wa,  0.0 hi,  1.3 si,  0.0 st
KiB Mem : 61822704 total, 56957304 free,  1374384 used,  3491016 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 59416744 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
127440 root      20   0    3312    948    684 R  13.7  0.0   0:03.97 iperf3         <-- Benchmark app
  7167 root      20   0 5174800  93288  50600 S   3.9  0.2  34:13.57 kubelet
    65 root      20   0       0      0      0 S   2.0  0.0   0:26.06 ksoftirqd/9    <-- BPF in kernel
  6373 root      20   0 6644496  70412  28976 S   2.0  0.1  16:31.18 dockerd
 17895 root      20   0   54416  28384  21564 S   2.0  0.0   2:30.99 aws-k8s-agent
 94338 root      20   0   10.1g  55652  17712 S   2.0  0.1  28:31.99 etcd
     1 root      20   0  191540   5984   4148 S   0.0  0.0   0:05.45 systemd
     2 root      20   0       0      0      0 S   0.0  0.0   0:00.07 kthreadd
```

As expected, the networking layer is not really consuming any significant CPU.
Overall, the systesm is 98.4% idle and the majority of the CPU being consumed
is by the app which produces the network traffic. This is pretty much expected,
Linux has been optimized to run as a server hosting applications for decades.

The picture looks similar on the receiving side:

```
top - 14:32:52 up 19:40,  1 user,  load average: 0.46, 0.51, 0.39
Tasks: 424 total,   1 running, 233 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.9 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0 st
KiB Mem : 61822704 total, 45532640 free,  1472912 used, 14817152 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 59248884 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 72931 root      20   0    3436   1068    612 S  58.0  0.0   4:13.87 iperf3
 71028 root      20   0   10.1g  58400  18116 S   6.0  0.1  24:18.46 etcd
 70274 root      20   0   10.1g  56560  17456 S   4.0  0.1  24:53.08 etcd
  6977 root      20   0 5172432  92316  50740 S   2.0  0.1  40:14.05 kubelet
 84317 root      20   0   50276  25616  18704 S   2.0  0.0   4:33.68 kube-dns
 87137 root      20   0  169296   4596   3732 R   2.0  0.0   0:00.09 top
     1 root      20   0  191532   6044   4168 S   0.0  0.0   0:04.39 systemd
     2 root      20   0       0      0      0 S   0.0  0.0   0:00.07 kthreadd
     4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/0:0H
```

The system is only 97.7% idle and 58% of a single core is consumed by the
benchmark app.

So are there differences between CNI plugins for throughput at an MTU of 9000
at all? Yes, different CNI plugins use different architectures in connecting
pods to the networks. Cilium defaults to an encapsulation based routing
mechanism. The advantage of this mode is the ability to pretty much run on any
infrastructure without changes. The disadvantage of this mode is a slight
overhead in theoretical maximum throughput as each packet is encapsulated into
additional network headers. For this reason, Cilium also supports direct
routing mode with integration into any routing daemon including kube-router. So
if you care about maximising the throughput and have the expertise to configure
your network to route pod IPs, feel free to configure direct routing mode by
running Cilium with the option `--tunnel=disabled`.

##### HTTP and FTP benchmark

This is where it gets really interesting! The benchmark shows a reduced
throughput for both HTTP and FTP in the scenario of Cilium. This is very
unexpected, in particular because there is no HTTP level network policy
in effect and thus HTTP/FTP performance should match TCP.

The benchmark that Alexis is performing is to run nginx in a Kubernetes pod
with a volume mounted that contains a 10G file consisting of random data and
then fetching it via curl to measure the duration of the download.

The initial step was to run this benchmark and just run `top` on the side:

```
top - 15:06:15 up 20:13,  1 user,  load average: 0.51, 0.36, 0.26
Tasks: 502 total,   2 running, 229 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us,  0.6 sy,  0.0 ni, 97.9 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem : 61822704 total, 56942360 free,  1376780 used,  3503564 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 59411592 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 65922 root      20   0   14096   3220   2804 S  18.3  0.0   0:00.55 curl
  7167 root      20   0 5175856  93868  50600 S   4.3  0.2  35:25.54 kubelet
  6373 root      20   0 6644496  71252  29040 S   2.3  0.1  17:03.42 dockerd
 57396 root      20   0 3725028 124768  37952 S   2.0  0.2   0:04.29 cilium-agent
 94338 root      20   0   10.1g  57432  17864 S   2.0  0.1  29:23.63 etcd
     1 root      20   0  191540   5984   4148 S   0.3  0.0   0:05.97 systemd
  6631 root      20   0 3312864  29336   7580 S   0.3  0.0   4:32.37 docker-containe
```

Nothing unexpected at all. The majority of the CPU is consumed by curl
respectively nginx on the other node. What is going on? Let's try with the
latest master version of Cilium and compare it to 1.3.0 that Alexis is using.

```
  Run |       1.3.0 |      master
------|-------------|------------
    1 | 4539 Mbit/s | 4814 Mbit/s
    2 | 4545 Mbit/s | 4814 Mbit/s
    3 | 4543 Mbit/s | 4814 Mbit/s
    4 | 4542 Mbit/s | 4815 Mbit/s
    5 | 4547 Mbit/s | 4815 Mbit/s
Total | 4543 Mbit/s | 4814 Mbit/s
```

Bingo! Did we unknowingly fix a performance regression that only applies to HTTP and
FTP? To understand what is going on, let's look what the benchmark script does
specifically:

```bash
function bench_kubectl {
        kubectl run --restart=Never --rm \
                --overrides='{"apiVersion":"v1","spec":{"nodeSelector":{"kubernetes.io/hostname":"XXX"}}}' $@
}

RES_HTTP=$(bench_kubectl bench -it --image=infrabuilder/netbench:client \
         -- curl -o /dev/null -skw "%{speed_download}" http://$IP/10G.dat 2>/dev/null| sed 's/\..*//' )
```

The script uses `kubectl run` which launches curl as soon as the container has
started and then outputs the download speed. Why does this matter?

When a pod launches, kubelet invokes the CNI layer to request the CNI plugin to
do the neworking plugin. This call blocks the scheduling of the pod. CNI
plugins that support network policy have to ensure that an eventual security
policy is correctly enforced from the first possible packet that a pod could
transmit or receive. This operation takes some time, any added time will lower
the speed at which pods can be scaled up on cluster level. Early on, we discussed
the pros and cons of:

1. Blocking the CNI request until pods labels have been fetched and all
 networking plumbing is done and to ensure that the first possible packet
 encounters are fully configured network and no packet drop is every
 happening.

2. Optimistically putting as little plumbing in place to allow for TCP
 retransmissions, return from the CNI request call and fetch the Kubernetes
 pod labels and completing the networking in the background. Because of
 eventual network policy that might apply, all traffic is dropped until the
 pod labels have been fetched. This mode makes use of the nature of TCP to
 retransmit for a while until networking is fully up and leads to better pod
 scalability.

We have chosen the faster pod scaling so far but based on user feedback, are
switching to the blocking mode going forward. The master tree already contains
[PR #6299](https://github.com/cilium/cilium/pull/6299) to change this default
and is scheduled to be released as 1.4.0, 1.3.1, and 1.2.6.

Due to how curl measures the download speed, it will start measuring the total
time of the download when making the initial connection attempt. This means
that the total time includes the period in which networking was still blocked.
This reduces the reported troughput rate. OTOH, iperf3 measures the actual
transmitted data rate in intervals and adds it up.

##### Memory consumption

The memory consumption as reported in the benchmark is real but work is
underway to provide better defaults while allowing large scale users to
continue running aggressive memory pre-allocation settings.

The reasons for this are:

- A very aggressive pre-allocation strategy of any datapath related kernel data
 structure including state required to maintain connection tracking. This
 ensures that none or as few memory allocations are required when performing
 work for individual network packets. This is done to keep latency low and
 stable. [PR #6357](https://github.com/cilium/cilium/pull/6357) is being worked
 on to disable pre-allocation with a `--preallocate-bpf-maps` option to
 continue optimizing for latency.

- Enabling IPv6 by default. This leads to data structures being allocated
 for IPv6 which most users don't use yet. [PR #6082] is underway to change this
 default.

We have definitely optimized for a low latency datapath so far while knowingly
sacrificing memory. The 1.4.0 release will provide more control over this with
defaults as decided by the community.

Another aspect is that the project is moving very quickly right now. We
dedicate a percentage of time to resource optimization for every release and
memory consumption has been coming down consequently with each release. We
definitely don't claim to have optimize this yet. We believe that BPF gives us
the flexibility and efficiency to strike the perfect balance between serving
low latency requests benefiting from heavily pre-allocated memory tables and
high throughput environments which do not require this at all.

##### CPU utilization

The last piece to look at is the CPU consumption. One thing to note here is
that the article includes a graph that shows the overhead in permil instead of
percent and the graph does not scale up to 100%. To put everything a bit into
perspective, the graph on the right shows the same numbers at a scale of 100%.

Let's try a different workload type to and trigger more CPU consumption by
the networking layer. We are running `wrk` in a pod on one Kubernetes node
configured to make as many HTTP requests as possible with 12 threads and 400
parallel connections to `nginx` running on another Kubernetes node.

```
wrk -t12 -c400 -d30s http://10.163.228.10:80/index.html
Running 30s test @ http://10.163.228.10:80/index.html
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    50.04ms  117.73ms   2.00s    87.14%
    Req/Sec     3.43k     1.61k   13.60k    75.93%
  1214934 requests in 30.10s, 271.07MB read
  Socket errors: connect 0, read 0, write 0, timeout 53
Requests/sec:  40362.94
Transfer/sec:      9.01MB
```

wrk is able to roughly send 40'0000 requests per second. Let's look at the CPU
consumption on both the sender and receiver:

```
top - 03:03:12 up 1 day,  8:36,  3 users,  load average: 1.16, 0.70, 0.37
Tasks: 510 total,   1 running, 259 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  1.9 sy,  0.0 ni, 91.7 id,  0.0 wa,  0.0 hi,  5.7 si,  0.0 st
KiB Mem : 26403225+total, 25511452+free,  2000124 used,  6917612 buff/cache
KiB Swap:  1995772 total,  1995772 free,        0 used. 26021649+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4635 root      20   0  106524   6288   1728 S 282.5  0.0   0:29.02 wrk
  132 root      20   0       0      0      0 S   1.3  0.0   0:26.84 ksoftirqd/20
    8 root      20   0       0      0      0 S   1.0  0.0   0:20.08 ksoftirqd/0
   24 root      20   0       0      0      0 S   1.0  0.0   0:07.93 ksoftirqd/2
   48 root      20   0       0      0      0 S   1.0  0.0   0:07.11 ksoftirqd/6
   60 root      20   0       0      0      0 S   1.0  0.0   0:11.13 ksoftirqd/8
   72 root      20   0       0      0      0 S   1.0  0.0   0:06.87 ksoftirqd/10
   84 root      20   0       0      0      0 S   1.0  0.0   0:20.00 ksoftirqd/12
   96 root      20   0       0      0      0 S   1.0  0.0   0:07.65 ksoftirqd/14
  108 root      20   0       0      0      0 S   1.0  0.0   0:03.32 ksoftirqd/16
```

The majority of the CPU is consumed by `wrk` itself which is consuming roughly
3 cores in total spread over 12 threads.

On the receiving side.

```
op - 03:02:07 up 1 day,  8:36,  2 users,  load average: 0.47, 0.37, 0.28
Tasks: 512 total,   2 running, 263 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.9 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
KiB Mem : 26403224+total, 23441790+free,  2124056 used, 27490276 buff/cache
KiB Swap:  1995772 total,  1995772 free,        0 used. 26002094+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
41591 systemd+  20   0   33120   3336   1964 R  99.7  0.0   0:37.05 nginx
11463 root      20   0 10.034g  22268  12872 S   1.0  0.0   1:02.30 etcd
    9 root      20   0       0      0      0 I   0.3  0.0   0:11.23 rcu_sched
   72 root      20   0       0      0      0 S   0.3  0.0   0:06.98 ksoftirqd/10
  132 root      20   0       0      0      0 S   0.3  0.0   0:12.50 ksoftirqd/20
  168 root      20   0       0      0      0 S   0.3  0.0   0:10.03 ksoftirqd/26
37246 root      20   0       0      0      0 I   0.3  0.0   0:00.26 kworker/u97:0
```

The receiving side reveals the bottleneck which is nginx running as a single
pod. You can also see various `ksoftirqd` which are kernel software interrupt
threads which are handling the network receive load. You can see that no
significant CPU resources are being spent on the networking stack itself.

##### Conclusion

This is where we are right now with our investigation. We did not find a
generic performance regression. The HTTP and FTP related performance has been
tracked down to asynchronous handling of the CNI API which will change in
1.3.1, 1.2.6 and 1.4.0. 1.4.0 will make the aggresive pre-allocation optional
and disable IPv6 to optimize the memory footprint.

A big thank-you again to Alexis Ducastel for sharing the results. Feedback like
this helps everybody improve.

[slack]: https://slack.cilium.io
[cilium]: https://github.com/cilium/cilium
[benchmark]: https://itnext.io/benchmark-results-of-kubernetes-network-plugins-cni-over-10gbit-s-network-36475925a560
[pr #6357]: https://github.com/cilium/cilium/pull/6357
[pr #6082]: https://github.com/cilium/cilium/pull/6082

#### Deep Dive into Facebook's BPF edge firewall

URL: https://cilium.io/blog/2018/11/20/fb-bpf-firewall
Date: 2018-11-20
Categories: Technology

We have covered Facebook's BPF-based load balancer with DDoS protection in
a previous blog post: [Why is the kernel community replacing iptables with BPF?](/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables).
This post provides further details on Facebook's BPF use by covering Anant
Deepak's talk at the [BPF/networking microconference] on Facebook's BPF-based
edge firewall running in production.

The same conference also featured many other BPF related talks which we will
cover in follow-up blog posts. In particular interesting will be Nikita V.
Shirokov's (Facebook) talk [XDP: 1.5 years in production. Evolution and
lessons
learned](http://vger.kernel.org/lpc_net2018_talks/LPC_XDP_Shirokov_v2.pdf)
where Nikita shows the impressive difference between IPVS and BPF under heavy
load as well as Vlad Dumitrescu from Google talking about [Scaling Linux
Traffic Shaping with
BPF](http://vger.kernel.org/lpc_bpf2018_talks/lpc-bpf-2018-shaping.pdf) where
Vlad and others share their experience deploying BPF to production solving
scalable traffic shaping.

###### Problems of the existing iptables firewall

Anant started his [talk](http://vger.kernel.org/lpc_net2018_talks/ebpf-firewall-LPC.pdf)
providing the background on the need to replace the existing iptables firewall
with BPF. At the root of the problem lies the sequential nature of the iptables
policy engine. To demonstrate this, Anant showed a simplified version of the
code used to traverse a list of rules:

```c
// net/ipv6/netfilter/ip6_tables.c
unsigned int ip6t_do_table(...)
{
    e = get_entry(...);
    do {
        ...
        if (!ip6_packet_match(...)) {
            e = ip6t_next_entry(e);
            continue
        }
        ...
    } while (...);
}
```

Logic such as this was common 20 years ago when iptables was created. A primary
reason for this model was that most hardware-based firewalls were configured
using sequential list of rules. However, while hardware-based firewalls
primarily used [TCAM] for its parallel search capability and constant
processing speed, the iptables implementation in software has always been
suffering from linearly increasing CPU utilization as more rules are being
added to a table. The performance suffers in particular if packets are
frequently hitting rules in the lower parts of the table.

To illustrate this, Anant showed and explained the iptables ruleset that
previously applied to all network traffic entering any of the datacenters. The
purpose of the rules is to filter out unwanted traffic to protect the
load balancers from DDoS and other attacks. As you can tell based on the
ruleset, even though the ruleset is quite optimized to allow matching the
majority

Why is there no rule matching on the connection tracking state and allowing all
established connections as seen in common iptables rules sets? The overhead of
the iptables/netfilter connection tracker would simply bring any machine to its
knees when being hit with the common Facebook traffic pattern.

##### Benefits of a BPF-based firewall

What exactly can BPF do better? The slide that Anant presented listed the
following overview:

**Performance:** The performance aspect is the most important. By using
efficient BPF map data structures, the sequential lookups can be avoided and
the performance of the firewall is increased significantly. But even more
importantly, a DoS attack on closed ports which previously hit the default deny rule at the end of the rules set is no longer effective to impose an unnecessary load on the infrastructure.

**XDP:** XDP is a special BPF mode which enables BPF programs to take control
over network packets at the network driver level. This brings additional
performance benefits due to decreased overhead and more importantly allows to
run the firewall right in front of the existing BPF-based load balancer. The
chaining is done using BPF tail calls which is a function call like construct
with excellent performance attributes. As requirements change, Facebook
engineers can write additional programs to provide rate limiting, additional
filtering, or other custom logic and simply hook the programs into the chain.
The flexibility of BPF allows creating matching logic at arbitrary offsets in
the packet and provides powerful logic to handle any upcoming DoS scenario or
filtering requirement.

**Manageability:** The performance of BPF is outstanding and often an initial
motivation to get involved in BPF but infrastructure engineers ultimately fall
in love with BPF due to its flexibility, programmability, and manageability.
Bits in network packet headers can be arbitrarily interpreted and re-purposed,
policy can be decoupled from the network topology by creating a logical mapping
via packet attributes, visibility at any network layer is a matter of writing
the respective program. An extensive userspace API allows interacting with
programs to export statistics, update maps and atomically replace BPF programs.

##### BPF Program Details

The illustration below shows a simplified view of the BPF program that
implements the equivalent of the iptables rule set as shown before. A single
BPF map lookup per tuple using an LPM (Longest Prefix Match) table maps the
network topology to a logical policy which is then interpreted. The policy
logic itself is not that different from before, it still allows to use a list
of logical conditions to implement precedence. Unlike the individual iptables
rule matching function which translates into hundreds or thousands of CPU
instructions with a series of memory accesses. The BPF implementation
translates into a small number of CPU instructions with the logic implementing
the policy all using register-based accesses.

##### Performance

Facebook often doesn't publish absolute numbers in terms of performance so we
don't see the exact absolute CPU utilization but the comparison between
iptables (orange) and BPF (blue) clearly indicates how the utilization stays
constant for the BPF firewall while the iptables firewall suffers from a peak
of increased CPU utilization which translates to a successful DoS attack and
unnecessary load on the infrastructure.

- iptables has a linearly increasing CPU utilization as packets hit lower rules
 - Best when packets match earlier rules
 - Worst for default policy (match attempted for each rule)

- The BPF firewall performance remains practically constant irrespective of
 the rule being matched or default drop
 - Packet tuple lookup is efficient with only 1 BPF map per tuple
 - Location of matching rule now only matters to the extent of few branching
 instructions

##### Next Steps

The firewall described in the talk is specific to Facebook's edge use case
which has the most specific performance requirements. There are many other use
cases for firewalls where performance is still important but reconfigurability
and flexibility start becoming more important as well. For this reason, Anant
also provided insights into some ongoing projects:

###### IPtables-style BPF firewall Prototype

The presented BPF program can be generalized a bit to make it more flexible.
Facebook engineers have written a prototype which can translate a set of
limited iptables rules into a BPF program and map. The per-packet algorithm is
similar to the program discussed in the first part of the talk with a single
BPF map lookup per tuple. Instead of the verdict, the value of the BPF map
carries a bit array which describes the verdict based on the rule position.

```c
// Parse the packet and lookup against maps for each attribute
// *_res are the results of the lookup
ip4_src_addr_res = lookup_ipv4_src_addr(iph->saddr);

// ..collect _res for all other tuples
#pragma clang loop unroll(full)
for (rule_word = 0; rule_word < RULE_IDS_MAX_WORDS; rule_word++) {
  u64 rule_id = ip4_src_addr_res->rule_ids[rule_word];
      rule_id &= ip4_src_lpm_res->rule_ids[rule_word];
      rule_id &= ip4_dst_addr_res->rule_ids[rule_word];
      rule_id &= ip4_dst_lpm_res->rule_ids[rule_word];
      rule_id &= ip6_src_addr_res->rule_ids[rule_word];
      rule_id &= ip6_dst_addr_res->rule_ids[rule_word];
      rule_id &= ip6_src_lpm_res->rule_ids[rule_word];
      rule_id &= ip6_dst_lpm_res->rule_ids[rule_word];
      rule_id &= src_port_res->rule_ids[rule_word];
      rule_id &= dst_port_res->rule_ids[rule_word];
      rule_id &= ip_proto_res->rule_ids[rule_word];
      rule_id &= tcp_flag_res->rule_ids[rule_word];

  if (rule_id) {
    // find first bit set (MSB)
    int rule_num = (rule_word * 64) + get_msb_set(rule_id);

    // Take action
    int action = lookup_action(rule_num);
    if (action == BPFILTER_ACTION_PASS) {
      XDP_PASS;
    }

    if (action == BPFILTER_ACTION_DENY) {
      XDP_DROP;
    }
  }
}
```

The full details can be found in the [slides] and [paper]. As of now, the prototype still has some restrictions such as no support for REJECT rules.

###### bpfilter

As known publicly for a while, the kernel community is working on replacing the
in-kernel portion of iptables with BPF. Anant provided an update on this
project called [bpfilter]. Unlike the prototype described in the previous chapter
which works by having the user pipe `iptables-save` output into a python
program that generates a BPF program. The bpfilter program handles the
conversion from iptables to BPF as part of a kernel interface to provide
seamless backward compatibility. The goal is to allow users to continue using
iptables as-is while providing a more performant implementation.

##### Summary

Anant provided great details on Facebook's BPF use to build a highly efficient
and flexible firewall that complements the existing BPF-based load balancer.
The new firewall demonstrates constant CPU utilization regardless of the type
of traffic that is being processed and has become immune against DoS attacks.
The main reason listed for the migration away from the outdated iptables model
is the sequential matching semantics of iptables and the superior performance
and manageability of the BPF subsystem.

A recording of the talk should become available in a couple of weeks on the
[microconference page](http://vger.kernel.org/lpc-networking.html).

This post covered one of the many exciting talks on the topic of BPF, both the
BPF and Networking microconference where overflowing with BPF related talks. To
pick just a few:

- [Scaling Linux Traffic Shaping with BPF, Vlad Dumitrescu (Google) et al](http://vger.kernel.org/lpc_bpf2018_talks/lpc-bpf-2018-shaping.pdf)
- [XDP: 1.5 years in production. Evolution and lessons learned, Nikita V. Shirokov (Facebook)](http://vger.kernel.org/lpc_net2018_talks/LPC_XDP_Shirokov_v2.pdf)
- [Combining kTLS and BPF for Introspection and Policy Enforcement, Daniel Borkmann (Cilium) et al.](http://vger.kernel.org/lpc_net2018_talks/ktls_bpf.pdf)
- [BPF control flow, supporting loops and other patterns, John Fastabend (Cilium)](http://vger.kernel.org/lpc_bpf2018_talks/bounded-loops.pdf)
- [Traffic policing in eBPF: applying token bucket algorithm, Julia Kartseva (Facebook)](http://vger.kernel.org/lpc_bpf2018_talks/LPC2018-TokenBucket_v4.pdf)
- [Compile once, run everywhere, Yonghong Song and Alexei Starovoitov (Facebook)](http://vger.kernel.org/lpc_bpf2018_talks/bpf_compile_once.pdf)
- [ELF relocation for static data in BPF, Joe Stringer (Cilium) and Daniel Borkmann (Cilium)](http://vger.kernel.org/lpc_bpf2018_talks/bpf_global_data_and_static_keys.pdf)
- [XDP - challenges and future work, Jesper Brouer (Red Hat), Toke Høiland-Jørgensen (Karlstad University)](http://vger.kernel.org/lpc_net2018_talks/presentation-lpc2018-xdp-future.pdf)
- [Bringing the Power of eBPF to Open vSwitch, William Tu (VMware) et al.](http://vger.kernel.org/lpc_net2018_talks/ovs-ebpf-lpc18-presentation.pdf)

You can find the full list of talks including slides and papers on the
[BPF](http://vger.kernel.org/lpc-bpf.html) and
[Networking](http://vger.kernel.org/lpc-networking.html) microconference
websites. If you want to learn more about BPF, dive into the [BPF and XDP
Reference Guide](http://docs.cilium.io/en/stable/bpf/).

If you are interested in leveraging BPF and XDP for networking, policy and load
balancing in the context of containers, Kubernetes and microservices then have
a look at [Cilium].

###### References

- [BPF and XDP Reference Guide](http://docs.cilium.io/en/stable/bpf/)
- [Facebook open-sources new suite of Linux kernel components and tools](https://code.fb.com/open-source/linux/)
- [bpfilter LWN article](https://lwn.net/Articles/747551)
- [katran - Facebook loadbalancer repository](https://github.com/facebookincubator/katran)
- [droplet - BPF-based DDoS counter measures from Facebook](https://netdevconf.org/2.1/session.html?zhou)
- [SIGCOMM 2018 - Accelerating Linux Security with eBPF iptables](https://dl.acm.org/citation.cfm?id=3234228)
- [Why is the kernel community replacing iptables with BPF?](https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/)

[slack]: https://slack.cilium.io/
[cilium]: https://github.com/cilium/cilium
[slides]: http://vger.kernel.org/lpc_net2018_talks/ebpf-firewall-LPC.pdf
[paper]: http://vger.kernel.org/lpc_net2018_talks/ebpf-firewall-paper-LPC.pdf
[bpf/networking microconference]: http://vger.kernel.org/lpc-networking.html
[lpc]: https://www.linuxplumbersconf.org/
[tcam]: https://en.wikipedia.org/wiki/Content-addressable_memory#Ternary_CAMs
[bpfilter]: https://lwn.net/Articles/747551

#### Cilium 1.3: Go extensions for Envoy, Cassandra & Memcached Support

URL: https://cilium.io/blog/2018/10/23/cilium-13-envoy-go
Date: 2018-10-23
Categories: Release

We are excited to announce the Cilium 1.3 release. The release introduces
several new features. The major highlight of the release is the addition of Go
extensions for [Envoy] as well as Cassandra and Memcached protocol parsers with
policy enforcement capability, both implemented as Envoy Go extension.

As usual, a big shout out to the entire community of Cilium developers who have
contributed 785 commits in the time period between 1.2 and 1.3.

###### What are Envoy Go extensions?

We have been relying on [Envoy] for all processing of HTTP and gRPC as well as
HTTP derivates such as Elasticsearch since version 1.0. As the community
discussed how to extend the scope of supported L7 protocols, it became clear
that Envoy is the right platform to drive future protocol additions. The focus
quickly shifted to finding ways to simplify the extendability of Envoy and
allow reuse of existing open source projects such as the CNCF project [Vitess].
The idea of Go extensions for Envoy was born.

With Cilium 1.3, we introduce Go extensions for Envoy as a Beta feature.

- **Transparent injection of extensions:** With the help of Cilium, connections
 are transparently redirected to Envoy without requiring to modify the
 application or pod. Redirects are configured based on destination ports and can be restricted to source or destination services based on labels, IPs, DNS
 names, and service names for both ingress and egress connections and map
 connections to extensions via the extension's name. The redirects are
 configured via the CiliumNetworkPolicy CRD or a REST API. Envoy can be
 configured to run either as a sidecar in each pod or as a standalone proxy, one
 per node.

- **Fully distributed:** Go extensions run fully distributed inside of Envoy on
 each node or inside each pod and do not require a centralized control plane
 for data processing. The go extensions themselves can, of course, call out to
 arbitrary control plane components to report telemetry or validate
 requests.

- **Dynamic extensions mapping:** Go extensions are made available to Envoy as
 a shared library. Cilium configures Envoy to automatically load the
 respective Go extension based on the configured redirects and invoke it when
 data for such a connection. Future releases will offer support to update and
 re-load extensions at runtime without restarting Envoy and without losing
 connection state.

- **Extension configuration via CRD:** Go extensions are configured using
 generic key-value pairs via either the CRD or a REST API. This allows passing
 configuration such as security policies, security tokens or other
 configuration without requiring to make Envoy itself aware of it.

- **Generic access logging:** Similar to the configuration, extensions are able
 to return generic key-value pairs which will make its way into the access log
 to allow passing extracted visibility into the access log layer.

- **Sandboxed** A sandbox ensures that any parser instability
 cannot destabilize the mature core of Envoy. Inspired by Matt Klein's post
 [Exceptional Go](https://medium.com/@mattklein123/exceptional-go-1dd1488a6a47),
 parsers are allowed to panic to raise exceptions. When a panic occurs,
 information is logged to the access log and the TCP connection associated
 with the request is closed.

###### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms like Kubernetes, Docker, and Mesos.

At the foundation of Cilium is a new Linux kernel technology called BPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. Besides providing traditional network level
security, the flexibility of BPF enables security on API and process level to
secure communication within a container or pod. Because BPF runs inside the
Linux kernel, Cilium security policies can be applied and updated without any
changes to the application code or container configuration.

See the section **[Introduction to
Cilium](https://cilium.readthedocs.io/en/v1.3/intro/)** for a more detailed general
introduction to Cilium.

###### What is Envoy?

Envoy is an L7 proxy and communication bus designed for large modern
service-oriented architectures. The project was born out of the belief that:

> The network should be transparent to applications. When network and
> application problems do occur it should be easy to determine the source of
> the problem.

You can learn more about Envoy in the section [What is
Envoy](https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy) of the
Envoy documentation.

##### How to write an Envoy Go extension

Writing extensions for Envoy is simple. To illustrate this, we will implement a
basic protocol parser for the R2-D2 control protocol and implement filtering
logic to exclude any control request that contains the string "C-3PO".

The primary API for an extension to implement is the `OnData()` function
which is invoked whenever Envoy is receiving data on a connection that has been
mapped to an extension via the `CiliumNetworkPolicy`. The function must parse
the data and return one of the following verdicts:

- **MORE:** Parser needs more _n_ more bytes to continue parsing.
- **PASS:** Pass along _n_ bytes of the data stream.
- **DROP:** Drop _n_ bytes of the data stream.
- **INJECT:** Inject _n_ bytes of data in the specified direction.
- **ERROR:** A parsing error has occurred, the connection must be closed.
- **NOP:** Do nothing.

In order to register the extension, a parser factory is created which must
implement a `Create()` function. The function is called whenever Envoy
has established a new connection for which the parser should be used.

```go
import (
        "github.com/cilium/cilium/proxylib/proxylib"
)

type parser struct{
        connection *proxylib.Connection
}

func (p *parser) OnData(reply, endStream bool, dataArray [][]byte) (proxylib.OpType, int) {
        data := string(bytes.Join(dataArray, []byte{}))
        msgLen := strings.Index(data, "\r\n")
        if msgLen < 0 {
                return proxylib.MORE, 1 // No delimiter, request more data
        }

        msgStr := data[:msgLen]
        msgLen += 2 // Inlcude the "\r\n" in the request

        if reply {
                return proxylib.PASS, msgLen // Pass responses without additional parsing
        }

        if strings.Contains(msgStr, "C-3PO") {
                return proxylib.DROP, msgLen
        }

        return proxylib.PASS, msgLen
}

type factory struct{}

func (f *factory) Create(connection *proxylib.Connection) proxylib.Parser {
        return &parser{connection: connection}
}

func init() {
        proxylib.RegisterParserFactory("r2d2", &factory{})
}
```

Finally, hook the new parser into the proxylib by importing the new parser
package into the proxylib package. This will include the parser in the
`libcilium.so` that is loaded by Envoy. Edit `proxylib/proxylib.go`:

```go
import (
        [...]
        _ "github.com/cilium/cilium/proxylib/r2d2"
)
```

The above example leaves out the configuration of the extension, integration
into the policy repository and all aspects of access logging. See the guide
[Envoy Go Extensions](https://cilium.readthedocs.io/en/v1.3/envoy/extensions/)
for a step by step guide on how to write a Go extension.

##### Cassandra Support (Beta)

[Cassandra] is a popular NoSQL database management system. It is often operated
at large scale and accessed by many services and often shared between teams.
Cilium 1.3 introduces protocol support for the Apache [Cassandra] protocol
provide visibility and policy enforcement.

The Cassandra Go extension is capable to provide visibility and enforcement on
the following protocol fields:

- **query_action:** The action performed on the database `SELECT`,
 `INSERT`, `UPDATE`,... The field is always matched as an exact match.
- **query_table:** The table on which the query is executed on. Matching is
 possible with a regular expression.

###### Example: How the Empire restricts Cassandra access by table

The following example shows how the Empire is exposing limited access to the
Empire's Cassandra cluster running on port 9042 to outposts. Outposts are
identified by the label `app=empire-outpost` and have the following
privileges:

- `SELECT` access on the tables "system._` and "system_schema._"
- `INSERT` on the table "attendance.daily_records". Note that the outposts
 can't read from the tables and thus can't read the daily records from other
 outposts.

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
description: 'Allow only permitted requests to empire Cassandra server'
metadata:
name: 'secure-empire-cassandra'
specs:
  - endpointSelector:
    matchLabels:
      app: cass-server
ingress:
  - fromEndpoints:
      - matchLabels:
          app: empire-outpost
    toPorts:
      - ports:
          - port: '9042'
            protocol: TCP
        rules:
          l7proto: cassandra
          l7:
            - query_action: 'select'
              query_table: "system\\..*"
            - query_action: 'select'
              query_table: "system_schema\\..*"
            - query_action: 'insert'
              query_table: 'attendance.daily_records'
```

This is a simple example, see the [Cassandra getting started guide](https://cilium.readthedocs.io/en/v1.3/gettingstarted/cassandra/)
for more complex examples.

##### Memcached Support (Beta)

Memcached is a popular distributed in-memory key-value store that is often used
for caching purposes or to share small chunks of arbitrary data between
services. With the addition of a memcached parser golang extension to Envoy,
Cilium can now enforce security rules to restrict memcached clients to certain
commands such as read or write but also to certain key prefixes.

###### Example: How the Rebels secure a shared memcached service

In the following example, the Rebels have started running a memcached service
identified by the label `app=memcached`. Several services are interacting
with the memcached services and different rules are being applied:

- The fleet maintenance service identified by `function=fleet-maintenance` is
 granted read and write access to all keys with the prefix
 `alliance/fleet`. Access to any other key is prohibited.

- The fleet monitoring service identified by `function=fleet-monitoring` is
 only granted read access on keys with the prefix `alliance/fleet`.
 Write access to keys in the prefix or access to any key outside of the prefix
 is prohibited.

- All Jedis identified by the label `role=jedi` have full to the entire
 Memcached service and can access all keys.

```yaml
apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
description: 'Secure the Rebel memcached service'
metadata:
  name: 'secure-rebel-alliance-memcache'
specs:
  - endpointSelector:
      matchLabels:
        app: memcached
    ingress:
      - fromEndpoints:
          - matchLabels:
              function: fleet-maintanence
        toPorts:
          - ports:
              - port: '11211'
                protocol: TCP
            rules:
              l7proto: memcache
              l7:
                - command: 'writeGroup'
                  keyPrefix: 'alliance/fleet/'
                - command: 'get'
                  keyPrefix: 'alliance/fleet/'
      - fromEndpoints:
          - matchLabels:
              function: fleet-monitoring
        toPorts:
          - ports:
              - port: '11211'
                protocol: TCP
            rules:
              l7proto: memcache
              l7:
                - command: 'get'
                  keyPrefix: 'alliance/fleet/'
      - fromEndpoints:
          - matchLabels:
              role: jedi
        toPorts:
          - ports:
              - port: '11211'
                protocol: TCP
            rules:
              l7proto: memcache
              l7:
                - command:
```

For a full example using Memcached, see the [Memcached getting started guide](https://cilium.readthedocs.io/en/v1.3/gettingstarted/memcached/).

##### Community

###### Linux Foundation Core Infrastructure Initiative Best Practices

[](https://bestpractices.coreinfrastructure.org/projects/1269)

We are committed to developing Cilium in the open and with best open source
best practices. This includes a strong commitment to security. We are happy to
announce that we have completed all work to meet the [CII Best
Practices](https://bestpractices.coreinfrastructure.org/en) of the Linux
Foundation [Core Infrastructure
Initiative](https://www.coreinfrastructure.org/). You can learn
more about the badge program [here](https://www.coreinfrastructure.org/programs/badge-program/).

###### Introduction of Special Interest Groups (SIGs)

The community is growing and getting more diverse in interest. In order to
ensure the scalability of the project, we are officially introducing special
interest groups (SIGs) to help provide some structure. The following SIGs have
been created already:

- **Datapath:** (#sig-datapath) Owner of all BPF and Linux kernel related datapath code.
- **Documentation:** (#sig-docs) All documentation related discussions
- **Envoy:** (#sig-envoy) Envoy, Istio and maintenance of all L7 protocol parsers.
- **Policy:** (#sig-policy) All topics related to policy. The SIG is responsible for all security relevant APIs and the enforcement logic.
- **Release Management:** (#launchpad) Responsible for the release management and backport process.

Anyone can propose additional SIGs. The process is simple and documented [here](https://cilium.readthedocs.io/en/v1.3/community/#how-to-create-a-sig)

##### 1.3 Release Highlights

- **Go extensions for Envoy**
 - Exciting new extension API for Envoy using Go including a generic configuration
 and access logging API. (Beta)
- **Cassandra & Memcached protocol support**
 - New protocol parsers for Cassandra and Memcached implemented using the new
 Envoy Go extensions. Both parsers provide visibility and security policy
 enforcement on operation type and key/table names using exact matches,
 prefix matches, and regular expressions. (Beta)
- **Security**
 - TTLs support for DNS/FQDN policy rules
 - Introduction of well-known identities for kube-dns, coredns, and
 etcd-operator.
 - New security identity "unmanaged" to represent pods which are not managed
 by Cilium.
 - Improved security entity "cluster" which allows defining policies for all
 pods in a cluster (managed, unmanaged and host networking).
- **Additional Metrics & Monitoring**
 - New "cilium metrics list" command to list metrics via CLI.
 - Lots of additional metrics: connection tracking garbage collection, Kubernetes
 resource events, IPAM, endpoint regenerations, services, and
 error and warning counters.
 - New monitoring API with more efficient encoding/decoding protocol. Used
 by default with fallback for older clients.
- **Networking Improvements**
 - Split of connection tracking tables into TCP and non-TCP to better
 handle the mix of long and short-lived nature of each protocol.
 - Ability to specify the size of the connection tracking tables via
 ConfigMap.
 - Better masquerading behavior for traffic via NodePort and HostPort to allow
 pods to see the original source IP if possible.
- **Full Key-value store Resiliency**
 - Introduced ability to re-construct the kvstore contents immediately after
 loss of any state. Allows to restore etcd from backup or to completely wipe
 it for a running cluster with minimal impact. (Beta)
- **Efficiency & Scale**
 - Significant improvements in the cost of calculating policy of individual
 endpoints. Work continues on this subject.
 - New grace period when workloads change identity to minimize connectivity
 impact throughout identity change.
 - More efficient security identity allocation algorithm.
 - New generic framework to detect and ignore Kubernetes event notifications
 for which Cilium does not need to take action.
 - Improvements in avoiding unnecessary BPF compilations to reduce
 the CPU overhead caused by it. Initial work to scope BPF templating to
 avoid compilation altogether.
- **Kubernetes**
 - Added support for Kubernetes 1.12
 - Custom columns for the CiliumEndpoints CRD (Requires Kubernetes 1.11)
 - Removed cgo dependency from cilium-cni for compatibility with ulibc
 - Removed support for Kubernetes 1.7
- **Documentation**
 - New Ubuntu 18.04 guide
 - Coverage of latest BPF runtime features such as BTF (BPF Type Format).
 - Documentation for VM/host firewall requirements to run multi-host networking.
- **Long Term Stable (LTS) Release**
 - 1.3 has been declared an LTS release and will be supported for the next 6
 months with backports.

###### Upgrade Instructions

As usual, follow the [upgrade
guide](https://cilium.readthedocs.io/en/v1.3/install/upgrade/#upgrading-minor-versions)
to upgrade your Cilium deployment. Feel free to ping us on
[Slack].

###### Release

- Release Notes & Binaries: [1.3.0](https://github.com/cilium/cilium/releases/tag/1.3.0)
- Container image: `docker.io/cilium/cilium:v1.3.0`

[slack]: https://slack.cilium.io/
[envoy]: https://github.com/envoyproxy/envoy
[go]: https://golang.org/
[vitess]: https://vitess.io/
[cassandra]: https://github.com/apache/cassandra

#### Cilium How-To: Install with Kubernetes on Ubuntu 18.04

URL: https://cilium.io/blog/2018/09/26/bionic-beaver
Date: 2018-09-26
Categories: How-To

Several users are looking to run Kubernetes with Cilium on Ubuntu 18.04 LTS. The Bionic Beaver (codename for Ubuntu 18.04) is an easy choice for deploying Kubernetes with Cilium as it ships with the default Linux kernel version 4.15.

##### What is Cilium?

Cilium is open source software that transparently secures network
connectivity between application services deployed using Linux container
orchestration platforms like Docker and Kubernetes. As of Cilium 1.1, it supports multiple container runtimes including Docker, cri-o, and containerd.

Cilium is based on a Linux kernel technology called BPF. Cilium enables this powerful technology to dynamically insert security visibility and control logic
within Linux itself. Because BPF runs inside the Linux kernel, Cilium security
policies can be applied and updated without any changes to the application code
or container configuration.

Check out our [docs](http://docs.cilium.io/en/stable/intro/) for a more detailed introduction to Cilium.

##### Software Versions

- **Operating System:** Ubuntu 18.04
- **Kubernetes:** v1.11.0
- **Kubernetes CNI plugin:** Cilium v1.2.3
- **Installer:** kubeadm.

##### Prerequisites

- Multiple machines set up with Ubuntu 18.04 (bare-metal, AWS instances, GCE instances, etc).
- Privileged access to the Ubuntu servers as root or with `sudo` for installing packages.

For AWS deployments, you can choose a community AMI for your location: [https://cloud-images.ubuntu.com/daily/server/locator/](https://cloud-images.ubuntu.com/daily/server/locator/). We used the community AMI `ami-d8d997a0` to launch 3 Ubuntu 18.04 instances. You'll need to open up your Security Group (SG) to allow communication on at least the following ports.

| | |
|:-------- |:----------------------- |
| **Port** | **Description** |
| 22 | SSH from outside |
| 443 | k8s API within SG |
| 2379-2380 | etcd-operator within SG |
| 4240 | HTTP L3 checks within SG |
| 6443 | kubeadm\/k8s API access |
| 8472 | vxlan within SG |
| 9090 | optional for metrics |
| ICMP | ICMP check within SG |

Be sure to note that some steps are not persistent across host restarts and need to be added to a start-up script for a permanent configuration.

##### Set-Up

In this How-To guide, we'll work with 3 nodes running Ubuntu 18.04 to create 1 master node and 2 worker nodes in a single Kubernetes cluster. We'll assume a clean set-up with default Ubuntu 18.04 settings. We recommend at least 2 nodes to differentiate the installation steps required for master and worker nodes to scale out later on.

Give unique names to your master:

```
sudo hostnamectl set-hostname kubernetes-master
```

And to your worker nodes:

```
sudo hostnamectl set-hostname kubernetes-worker1
```

```
sudo hostnamectl set-hostname kubernetes-worker2
```

Servers require a host reboot in order for the hostname change to take effect.

##### Instructions

###### Update and Install Docker

Assuming a fresh installation of Ubuntu 18.04 is on your machines, update package lists, install docker and enable it on all nodes.

```
sudo apt-get update
```

```
sudo apt install docker.io
```

Execute the following to enable Docker after reboot.

```
sudo systemctl enable docker
```

###### Install Kubernetes

Download and install the signing key for Kubernetes on master and worker nodes:

```
sudo su -c 'curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add'
```

Next, we're ready to add the Kubernetes repo and install Kubernetes on all nodes.

- Note: At the time of writing, only Ubuntu 16.04 (Xenial) packages for Kubernetes were available. We will update once k8s for Bionic is available.

As a regular user, add the repo and install package.

```
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt install kubeadm
```

At the time of writing, Kubernetes requires swap to be turned off. For work-arounds, check this [GitHub issue](https://github.com/kubernetes/kubernetes/issues/53533). Disable memory swap immediately for all nodes:

```
sudo swapoff -a
```

This is _not_ persistent across host restarts so add to `/etc/fstab` accordingly for persistence.

###### Initialize the Kubernetes Master

```
sudo kubeadm init --pod-network-cidr=10.217.0.0\/16
```

- Note: the final output from the command above to enable a regular user with the `.kube` config and extract the join command required for worker nodes to join the cluster.

As a regular user, enable kubectl on the `kubernetes-master` node to use your cluster:

```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

###### Join Worker Nodes to the Kubernetes Cluster

On each of the worker nodes, execute the `kubeadm join` command from the master init output **as root**.

```
sudo kubeadm join 172.0.126.152:6443 --token XXXX --discovery-token-ca-cert-hash sha256:XXXX
```

You should see a message of successful cluster-join:

You will see that all the nodes have successfully joined by executing the following from the `kubernetes-master` node:

```
kubectl get nodes
```

- Note: the nodes will only become `READY` once Cilium is installed.

###### Install Cilium

We will deploy Cilium on the Kubernetes cluster as the CNI plugin through a DaemonSet configuration.

- **Optional BPF Filesystem Mount:** We will mount the BPF filesystem on each node in the cluster. This step is optional and pins BPF resources to a persistent filesystem structure, thus persistent across `cilium-agent` restarts.

```
sudo mount bpffs /sys/fs/bpf -t bpf
```

This is _not_ persistent across host restarts so add to `/etc/fstab` accordingly for persistence.

- **Cilium + etcd-operator Deployment**

Cilium requires a [KV-store](http://cilium.readthedocs.io/en/stable/concepts/#key-value-store) to store state, including Policy Identities. At the time of writing, we introduced the `etcd-operator` to simplify the deployment of Cilium and its KV-store (etcd), of which Cilium will manage.

Download the Cilium repo folder and unzip it.

```
wget https://github.com/cilium/cilium/archive/v1.2.3.tar.gz
tar -xvf v1.2.3.tar.gz
```

Go to the add-ons folder to follow the instructions in the etcd-operator README. It outlines the scripts for the etcd-operator and Cilium deployment.

```
cd cilium-1.2.3/examples/kubernetes/addons/etcd-operator/
cat README.md
```

At a high level, the steps from the README include:

- Prerequisites: install CloudFlare PKI\/TLS toolkit
- Create etcd certificates using the provided script
- Deploy the generated certificates to your Kubernetes cluster
- Ensure the DNS pods have the required label
- Deploy the Kubernetes descriptors for etcd-operator and Cilium.

Confirm Cilium is up with the `CURRENT` count equal to the number of nodes in your cluster (may take several minutes):

```
kubectl get daemonsets -n kube-system
```

Confirm your worker nodes have successfully joined the cluster and are `READY` now that Cilium is up by executing the following from the `kubernetes-master` node:

```
kubectl get nodes
```

##### Deploy Your First Application and Security Policy

Let's run through a short HTTP example to demonstrate Cilium's API-aware network security policy in action.

###### Deploy a Star Wars Demo Application

```
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.2.3/examples/minikube/http-sw-app.yaml
```

Confirm it's up:

```
kubectl get pods,svc
```

And check the pods and services connectivity:

```
kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
<span style="color:grey">Ship landed</span>
kubectl exec tiefighter -- curl -s -XPOST  deathstar.default.svc.cluster.local/v1/request-landing
<span style="color:grey">Ship landed</span>
```

###### Enforce an L7 Policy with CiliumNetworkPolicy

The policy will deny `xwing` access completely by its identity label and allow `tiefighter` to only make a specific HTTP call.

```
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.2.3/examples/minikube/sw_l3_l4_l7_policy.yaml
```

You can take a look at the Cilium Network Policy to see the HTTP-specific policy:

```
kubectl describe cnp rule1
```

###### Test Cilium Security Enforcement

Now that your CiliumNetworkPolicy is in place, you can see it in action! Try to access the `deathstar` service from `xwing`:

```
kubectl exec xwing -- curl --connect-timeout 5 -XPOST deathstar.default.svc.cluster.local/v1/request-landing
```

This command is denied by the (L3) identity-based policy. Use `CTRL-C` to exit prior to timeout.

Next, see what `tiefighter` can access. Per the policy, a POST to \/request-landing should be allowed:

```
kubectl exec tiefighter -- curl -s -XPOST  deathstar.default.svc.cluster.local/v1/request-landing
<span style="color:grey">Ship landed</span>
```

whereas other HTTP calls to the deathstar service are forbidden:

```
kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/exhaust-port
<span style="color:grey">Access denied</span>
```

Hooray! You have successfully deployed Cilium with Kubernetes on Ubuntu 18.04 and enforced an API-aware network security policy to provide least privilege security.

##### Next Steps

We have several examples of API-aware network policy for various protocols. They are easy to check out through our [Getting Started Guides](http://docs.cilium.io/en/stable/gettingstarted/). We'd love to hear how you're doing! Please join our community on [Slack](https://slack.cilium.io/) and let us know how you did.

##### References

- [docs.cilium.io](docs.cilium.io)
- [https://linuxconfig.org/how-to-install-kubernetes-on-ubuntu-18-04-bionic-beaver-linux](https://linuxconfig.org/how-to-install-kubernetes-on-ubuntu-18-04-bionic-beaver-linux)

#### Kubernetes Network Policies Using Cilium - Controlling Ingress/Egress from Namespaces

URL: https://cilium.io/blog/2018/09/19/kubernetes-network-policies
Date: 2018-09-20
Categories: Technology

Kubernetes provides [Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) for controlling traffic going in and out of the pods. Cilium implements the Kubernetes Network Policies for L3/L4 level and extends with L7 policies for granular API-level security for common protocols such as HTTP, Kafka, gRPC, etc. As per the Kubernetes Network Policy model, Cilium policies follow the whitelist model. When a policy is enabled for a pod, all ingress and egress traffic are denied by default unless the policy specification allows specific traffic. As a result, inter-namespace communication will be denied by default and we need policy specifications to whitelist traffic within namespace and legitimate traffic in and out of a namespace.

##### Setup to test the policies

- Kubernetes cluster running on AWS created using [Cilium Kubespray guide](http://docs.cilium.io/en/latest/kubernetes/install/kubespray/#k8s-install-kubespray)
- A sample application created using commands below

```
kubectl create ns empire
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.2.2/examples/minikube/http-sw-app.yaml -n empire
```

##### Allowing all traffic within a namespace while blocking inter-namespace traffic

The policy specification below allows all traffic within the `empire` namespace.

```YAML
cat <<EOF | kubectl create -n empire -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-within-namespace"
specs:
  - endpointSelector:
      matchLabels: {}
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
    ingress:
    - fromEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
EOF
```

The key components of the policy are:

- The `endpointSelector:matchLabels={}` will select all pods in the namespace `empire`. The scope of Network Policies is per namespace, as noted in the `-n empire` parameter for creating the policy.
- For the selected pods, egress is allowed as long as the identity of destination pods has label `namespace=empire`
- Similar to the egress rule, the ingress rule allows incoming traffic from pods in `namespace=empire`

To test the policy, access the deathstar pod IP and deathstar service:

```
# Replace below IP with the deathstar pod IP in your cluster
$ kubectl exec tiefighter -n empire -- curl -sL 100.96.7.169/v1
{
...
}
#test that ingress is denied from other namespaces
$ kubectl exec -it kube-dns-6c7dfbf97d-cm4tt -n kube-system -- wget 100.96.7.169/v1
Defaulting container name to kubedns.
Use 'kubectl describe pod/kube-dns-6c7dfbf97d-cm4tt -n kube-system' to see all of the containers in this pod.
Connecting to 100.96.7.169 (100.96.7.169:80)
^C
#access using service name will fail as egress to kube-dns is blocked
$ kubectl exec -it tiefighter -n empire -- curl  deathstar.empire.svc.cluster.local/v1
^C
```

The direct call from `tiefighter` pod to `deathstar` pod works as expected since we have whitelisted the egress and ingress traffic within namespace `empire`. The ingress call from `kube-dns` to `deathstar` is blocked. And the call to `deathstar` service using the DNS name doesn't work. This is because we haven't whitelisted the calls to `kube-dns`, which runs in the `kube-system` namespace.

##### Allowing select egress to outside the namespace, for example, kube-dns

Pods running in this Kubernetes cluster use the `kube-dns` service for resolving service names to IPs. Allowing pods to access `kube-dns` running in `kube-system` namespace is a good example of whitelisting legitimate inter-namespace traffic.

```YAML
cat <<EOF | kubectl create -n empire -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-within-namespace"
specs:
  - endpointSelector:
      matchLabels: {}
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
    ingress:
    - fromEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
  - endpointSelector:
      matchLabels: {}
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": kube-system
          "k8s:k8s-app": kube-dns
EOF
```

The policy allows egress to destination pods belonging to `kube-system` namespace and having `k8s-app=kube-dns`. However, access to any other service running in `kube-system` namespace is not allowed.

```
#access using service
kubectl exec -it tiefighter -n empire -- curl  deathstar.empire.svc.cluster.local/v1
{
...
}

#access to other services in kube-system is not allowed
kubectl exec -it tiefighter -n empire -- curl -sL kubernetes-dashboard.kube-system.svc.cluster.local/version
^C
```

##### Allowing select ingress from other namespaces

Similar to whitelisting the `kube-dns` access, there is often need to whitelist specific ingress traffic from other namespaces. A common example is Prometheus monitoring, which makes `HTTP GET` calls to pods to pull metrics. We will create a simple Prometheus pod in `monitoring` namespace and apply the following policy which allows traffic from Prometheus to all the pods in the `empire` namespace. Additionally, we will restrict the Prometheus pod access to specific port, `8080`, and to specific `HTTP GET /metrics` call. This is an example of combining L3, L4 and L7 policies for strict least privilege access.

```YAML
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.2/examples/kubernetes/addons/prometheus/prometheus.yaml
cat <<EOF | kubectl create -n empire -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-within-namespace"
specs:
  - endpointSelector:
      matchLabels: {}
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
    ingress:
    - fromEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
  - endpointSelector:
      matchLabels: {}
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": kube-system
          "k8s:k8s-app": kube-dns
  - endpointSelector:
      matchLabels: {}
    ingress:
    - fromEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": monitoring
          app: prometheus
      toPorts:
      - ports:
        - port: "8080"
          protocol: "TCP"
        rules:
          http:
          - method: "GET"
            path: "/metrics"
EOF
```

The key elements of the policy are:

- The ingress policy spec is very similar to the egress and uses `ingress:fromEndpoints` along with `matchLabels` to allow traffic from `app=prometheus` and `namespace=monitoring`
- The `toPorts` specification restricts access to port `8080`
- The `http` rules section further restricts access to `GET /metrics`

To test:

```
# This will be allowed
kubectl exec -it prometheus-core-84677d797f-wrhm7 -n monitoring -- wget -O metrics deathstar.empire.svc.cluster.local:8080/metrics

# This will be denied
kubectl exec -it prometheus-core-84677d797f-wrhm7 -n monitoring -- wget -O random_get deathstar.empire.svc.cluster.local:80/v1
^C
```

##### Allowing select egress to outside the cluster

Often pods will need access to services outside the cluster. These communications can be allowed by either whitelisting the destination CIDRs or by using DNS. The policy below allows all pods in `empire` namespace to access the external service specified by DNS, `cilium.io`.

```YAML
cat <<EOF | kubectl create -n empire -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-within-namespace"
specs:
  - endpointSelector:
      matchLabels: {}
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
    ingress:
    - fromEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": empire
  - endpointSelector:
      matchLabels: {}
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": kube-system
          "k8s:k8s-app": kube-dns
  - endpointSelector:
      matchLabels: {}
    egress:
    - toFQDNs:
        - matchName: "cilium.io"
EOF
```

To test run the following commands:

```

$ kubectl exec -it tiefighter -n empire -- curl -sL cilium.io
...
...
...
$ kubectl exec -it tiefighter -n empire -- curl -sL google.com
^C
```

##### Allowing select ingress from outside the cluster

Kubernetes services are exposed outside the cluster using either NodePort, LoadBalancer or Ingress Controllers. The IP-address of the original client may not be available depending on the load balancing behavior and/or SNAT scenarios for NodePort. This topic requires broader consideration and will be covered in a dedicated post.

We hope the above namespace based access control examples will help you secure your multi-tenant Kubernetes clusters. If you have use cases, questions, or want to discuss Cilium Kubernetes networking and security then join us on [slack](https://slack.cilium.io/)

#### Cilium 1.2: DNS Security Policies, EKS Support, ClusterMesh, kube-router integration,...

URL: https://cilium.io/blog/2018/08/21/cilium-12
Date: 2018-08-21
Categories: Release

We are excited to announce the Cilium 1.2 release. The release introduces
several new features addressing the top asks from Cilium users and community
members. One of the most exciting features is the introduction of security
policies based on DNS names to secure access to external services outside of
the cluster. Another top ask was to introduce the ability to connect and secure
multiple Kubernetes clusters. We are introducing ClusterMesh as an alpha level
feature to address this ask. It allows to connect and secure pods running in
different Kubernetes clusters. Equally important is the Kube-router integration
with Cilium. The effort led by the team from DigitalOcean enables to combine
BGP networking provided by kube-router with BPF based security and
load-balancing from Cilium. As usual, a big shout out to the entire community
of Cilium developers. The total number of contributors has grown to 85 and 579
commits have been contributed in the time period between 1.1 and 1.2.

##### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms like Kubernetes, Docker, and Mesos.

At the foundation of Cilium is a new Linux kernel technology called BPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. Besides providing traditional network level
security, the flexibility of BPF enables security on API and process
level to secure communication within a container or pod. Because BPF runs
inside the Linux kernel, Cilium security policies can be applied and updated
without any changes to the application code or container configuration.

See the section **[Introduction to Cilium](http://docs.cilium.io/en/v1.1/intro/)**
for a more detailed general introduction to Cilium.

##### 1.2 Release Highlights

- **DNS/FQDN based security policies**
 - Define network security rules based on FQDN/DNS names to express
 allowed connectivity to external services, e.g. allow to foo.com. (Beta)
- **AWS EKS Support**
 - New integrated etcd operator tailored to managed Kubernetes to remove the
 dependency on requiring an external kvstore. (Beta)
- **Clustermesh (Inter-cluster connectivity)**
 - Pod to pod networking across multiple Kubernetes clusters (Alpha)
 - Label based network security policy enforcement across clusters,
 e.g. allow pod foo in cluster1 to talk to pod bar in cluster2 (Alpha)
- **Kube-router integration for BGP support**
 - Co-operative mode to run alongside kube-router to enable BGP networking
- **KVstore based Node Discovery**
 - Enables automatic node discovery on non-Kubernetes environments
- **Load balancing**
 - Support for consistent backend selection as service backends scale
 - Support for service label/name based policy in combination with L4 rules
- **Efficiency & Scale**
 - Increased security identity space from 16 to 24 bits for large and multi
 cluster scale environments
 - First implementation of BPF based datapath notification aggregation
 - Continued progress on more efficient CPU utilization
 - Automatically detect MTU of underlying network
 - Use of local service ID allocation when DSR is disabled for improved load
 balancing scalability
- **Documentation**
 - New AWS EKS installation guide
 - Reference to kubespray installation guide
 - New simplified installation and upgrade instructions

##### DNS Based Security Policies

Services running in a Kubernetes cluster often interact with a range of
services that are external to the cluster. Common examples include SaaS
services such as S3, RDS, DynamoDB, etc., API based services such as Google
maps, Salesforce APIs, Twilio, etc. or self-hosted services such as Oracle
database clusters, Windows based applications, etc. Until now, Cilium supported
CIDR based policies for interacting with such external services. However, CIDR
based policies are difficult to define and maintain since the IP-addresses for
the services can change frequently. Cilium 1.2 now supports specifying policies
based on DNS names. The current implementation supports the core use case of
white-listing external services based on their FQDNs. Considering the
complexities around DNS resolutions (e.g. TTLs, CNAMEs, etc.) and associated
policy requirements (e.g. wildcard based specification such as \*.google.com),
there is more work coming in subsequent releases for comprehensive DNS based
policies.

###### Example: Allow to my-remote-service.com

The following simple example demonstrates how to define a network security
policy that allows all pods with the label app=test-app to resolve DNS names
via kube-dns and make external requests to the service my-remote-service.com:

```
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "to-fqdn-example"
spec:
  endpointSelector:
    matchLabels:
      app: test-app
  egress:
    - toFQDNs:
        - matchName: "my-remote-service.com"
    - toEndpoints:
      - matchLabels:
          "k8s:io.cilium.k8s.policy.serviceaccount": kube-dns
          "k8s:io.kubernetes.pod.namespace": kube-system
          "k8s:k8s-app": kube-dns
      toPorts:
      - ports:
        - port: "53"
          protocol: UDP
```

Cilium will automatically maintain the respective CIDR based rules to allow all
pods to talk to all IP addresses that are being returned for that DNS name.

##### AWS EKS Support

Follow the new guide [Installation on AWS EKS using etcd
operator](http://docs.cilium.io/en/v1.2/kubernetes/install/eks/) to install
Cilium on AWS EKS managed Kubernetes clusters.

The installation guide features the brand new integrated etcd operator for
Cilium to manage its own etcd cluster. Cilium requires a key-value store such
as etcd to store security identities. In many cases the etcd used by Kubernetes
isn’t accessible for use by Cilium. For example, in managed Kubernetes such as
EKS, GKE, AKS, etc. there is limited access to the Kubernetes etcd service. In
Cilium 1.2, we are using an etcd operator which will install and manage a
highly available etcd cluster for use by Cilium. This will make the
installation and use of Cilium much easier without any dependencies on
Kubernetes etcd and without requiring an out-of-band management of etcd
clusters.

We are actively working on AKS integration and guides on how to run on GKE will
be published in the next couple of weeks. Talk to us on [Slack] if you have
questions or need help.

##### ClusterMesh: Networking and Security across Multiple Kubernetes Clusters

Running multiple Kubernetes clusters is becoming common. Major use cases high
availability of services running in different Availability Zones or Regions;
point-of-presence services running in multiple clusters serving different
geo-locations; organizational reasons such as separating PCI vs non-PCI
compliant services; or simple separation of dev, test and prod workloads.

One of the fundamental requirements for multiple Kubernetes clusters is how to
connect the services and how to establish east-west network security for the
cross-cluster interactions. In Cilium 1.2, we are introducing the ability to
connect pods in different clusters without requiring any ingress controllers or
load balancers. Since pods are able to interact directly, Cilium is able to
preserve their identity and enforce complete L3/L4 and L7 access controls for
the east-west traffic.

Rather than using a single centralized etcd for all clusters, Cilium uses a
decentralized approach for creating the cluster-mesh and establishing pod
identities. In this approach, each cluster is able to independently manage
identities for their pods. Each cluster is assigned a unique identifier, which
also serves as an identity namespace. So the pod identity becomes a combination
of cluster identity + pod identity. This approach is easy to manage and scale
rather than coordinating the identities across all clusters. The approach also
aligns well with the high-availability goals for multi-cluster deployments in
that each cluster lifecycle is independent of others. Check out the
[multi-cluster install guide](https://cilium.readthedocs.io/en/stable/kubernetes/install/clustermesh/#clustermesh)
to try the Cilium cluster-mesh.

##### BGP Support: Kube-router + Cilium

Kube-router is a cloudnativelabs initiative to address a variety of Kubernetes
networking requirements and deliver a unified solution. From a data-forwarding
perspective, Kube-router uses BGP to broadcast and manage the routes for all
pods in a cluster. Using BGP peering with external routers, Kube-router can
make it easy to establish connectivity between Kubernetes pods and services
running outside the cluster. Also, Kube-router advertises the cluster IP when
service is created which means that a service can be accessed from outside the
cluster using single cluster IP and standard ports.

As a result of the community effort led by the DigitalOcean team, Cilium is
now integrated with Kube-router combining the benefits of BPF with BGP based
routing.

Check out the guide [Using kube-router to run BGP](http://docs.cilium.io/en/v1.2/kubernetes/install/kube-router/) to learn
how to run kube-router and Cilium side by side to run BGP networking with Cilium L3-L7 policy enforcement and load-balancing.

##### Istio 1.0 Support

Istio 1.0 was released recently. Cilium is already well integrated with Istio
providing efficient data forwarding as well as L3/L4 and L7 security for
servicemesh architectures. We have blogged about this in details in the
post [Istio 1.0: How Cilium enhances Istio with socket-aware BPF
programs](blog/2018/08/07/istio-10-cilium). Cilium 1.2 includes several
improvements in the integration with Istio 1.0.

##### Scalability & Efficiency

As usual, a large effort continues for improving the already good scalability
and becoming more CPU efficient by extensively profiling Cilium in various environment
and by optimizing the operation of connection tracking and other crucial data structures.
Ping us on [Slack] if you would like to learn more.

##### Upgrade Instructions

As usual, follow the [upgrade guide](https://cilium.readthedocs.io/en/v1.2/install/upgrade/#upgrading-minor-versions)
to upgrade your Cilium deployment. Feel free to ping us on [Slack].

##### Release

- Container image: `docker.io/cilium/cilium:v1.2.0`

[slack]: https://slack.cilium.io

#### Istio 1.0: How Cilium enhances Istio with socket-aware BPF programs

URL: https://cilium.io/blog/2018/08/07/istio-10-cilium
Date: 2018-08-07
Categories: Technology

Istio 1.0 was [released](https://istio.io/blog/2018/announcing-1.0/) last week.
From the Cilium community, we would like to congratulate all Istio contributors
for this massive effort. We have been fortunate to participate in the community
by contributing to Istio and by helping several users moving towards production
with Istio and Cilium.

If you are interested in learning about an Istio + Cilium user story before
diving into the technical details, consider reading the following Istio blog
post by the HP FitStation team, one of the largest Cilium + Istio users:
**[Istio a Game Changer for HP's FitStation Platform](https://istio.io/blog/2018/hp/)**.

This blog will go into some of the details on how BPF and Cilium enhance Istio:

- **Increase Istio Security:**
 - Least privilege security for multi-container pods using socket-aware BPF
 programs
 - Protect from compromised sidecar proxies and protocols that bypass the
 sidecar
 - Use of BPF to force all application traffic through the sidecar proxy
- **Enable Istio for external services:**
 - Using socket-aware BPF programs and kTLS to provide visibility and control
 into TLS encrypted connections
- **Performance:**
 - Efficient networking and socket redirection to accelerate Istio

##### What is Cilium?

Cilium is open source software for transparently providing and securing the
network and API connectivity between application services deployed using Linux
container management platforms like Kubernetes, Docker, and Mesos.

At the foundation of Cilium is a new Linux kernel technology called BPF, which
enables the dynamic insertion of powerful security, visibility, and networking
control logic within Linux itself. Besides providing traditional network level
security, the flexibility of BPF enables security on API and process
level to secure communication within a container or pod. Because BPF runs
inside the Linux kernel, Cilium security policies can be applied and updated
without any changes to the application code or container configuration.

See the section **[Introduction to Cilium](http://docs.cilium.io/en/v1.1/intro/)**
for a more detailed general introduction to Cilium.

##### What is Istio?

Istio provides an easy way to create a network of deployed services with load
balancing, service-to-service authentication, monitoring, and more, without
requiring any changes in service code. You can add Istio support to services by
deploying a special sidecar proxy throughout your environment that intercepts
all network communication between microservices, configured and managed using
Istio’s control plane functionality.

You can read more about Istio's concept and architecture in the [Istio
documentation](https://istio.io/docs/).

##### Efficient networking for Istio

The most basic form of collaboration is the Cilium CNI plugin providing
networking to Istio by connecting all sidecar proxies together and by providing
connectivity between proxies and the Istio control plane. Cilium also ensures
that Istio managed services can communicate with pods that are not managed by
Istio.

The following diagram illustrates how the Istio control plane, sidecar proxies,
and the CNI plugin are layered on top of each other:

In this mode, all the Istio system components benefit from Cilium's efficient
networking capabilities based on BPF in the Linux kernel. BPF is the
replacement technology for networking and packet filtering such as iptables
among other things. You can find more information on why BPF is driving this
shift in the following blog post from April: **[Why is the kernel community
replacing iptables with
BPF?](/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/)**

###### Socket level redirection to accelerate Istio and Envoy

The service mesh architecture of Istio requires all network traffic
for both incoming and outgoing requests of all pods participating in the
service mesh to be redirected to the sidecar proxy. The sidecar proxy will
terminate all TCP connections and perform services such as telemetry, retries,
routing, mutual TLS, and authorization on behalf of the services and use a
secondary so-called upstream TCP connection to reach the destination service.
This is what allows a service mesh to perform mutual TLS on behalf of the
application and is what achieves the transparency to not require changing any
application code when running a service mesh.

However, this redirection can be costly when standard IP-based tooling such as
iptables is used to implement the redirection as the entire TCP/IP stack has to
be traversed multiple times.

Cilium makes use of an exciting BPF feature called sockmap. It allows filtering
and redirection and socket level and is thus making Cilium socket-aware. The
socket is the interface that applications use to send and receive network
traffic. Applied to Istio redirection requirements, this allows to essentially
short-circuit TCP sockets on the same node and thus massively accelerate the
TCP connection between the application and the sidecar proxy of the service
mesh in a completely transparent manner. Neither the application nor the
sidecar proxy require modification in any way.

If you are interested to learn more about this use case, refer to the presentation
**Accelerating Envoy and Istio with CIlium and the Linux kernel
([recording](https://t.co/cx6CQhn1xl),
[slides](https://www.slideshare.net/ThomasGraf5/accelerating-envoy-and-istio-with-cilium-and-the-linux-kernel))**
as presented at KubeCon 2018 in Austin.

##### Enhancing Istio Security with Cilium

Cilium also integrates with Istio to enhance the security of Istio. Let's look
at what security properties Istio provides:

- **Mutual TLS:** Ability to verify the identity of the sender if the sender is
 controlled by Istio. This protects a service from receiving requests from
 attacks spoofing an IP address from a legitimate source service. It also
 enables TLS/SSL-based encryption between all services managed by Istio.

- **Authorization:** Ability to authorize requests including the ability to
 filter on metadata such as HTTP protocol fields, IP addressing and identity
 information as well as invoking further authorization plugins, for example,
 perform token validation.

###### Level 1: Protect unsupported protocols and compromised sidecars (Cilium 1.0)

A first essential level of protection required is to apply security policies to
all network protocols unsupported by Istio. This includes UDP, ICMP, and all
IPv6 traffic. Any such traffic bypasses the sidecar proxy and thus any eventual
security policy enforced via Istio. Cilium applies L3/L4 security policies
outside of the pod to all network traffic, regardless of the protocol. If a
protocol is unsupported, the packet will be dropped.

- **Secure non-IPv4/TCP protocols:** Cilium provides security policies outside
 of the pod and either completely blocks or applies fine-grained security
 rules to any traffic that has bypassed the proxy because the protocol is
 unsupported or the redirection has failed for some other reason.

- **Protect from compromised sidecars:** The sidecar proxy itself is not
 subject to the Istio security rules. A compromised sidecar has unrestricted
 access to the network. It can also manipulate its own security rules to
 become more permissive. Cilium allows to define service level security
 policies in addition to Istio and ensures that a compromised sidecar proxy
 can only operate with least privelege.

###### Level 2: Secure multi-container pods (Work in Progress)

The security protection at level 1 protects a service mesh at the granularity
of pods and services. It does not provide any security inside of a pod, for
example for communication between the application container and the sidecar
container.

Security protection at level 2 introduces segmentation between containers and
processes in the same pod by providing segmentation at the socket level by
using socket-aware BPF programs.

- **Force application traffic through sidecar:** By using socket-aware BPF
 policies, Cilium can ensure that all communication of an application must
 occur via the sidecar for both incoming and outgoing connections. The
 application container itself is guaranteed to not be able to receive a
 connection from outside the pod or make a request to outside of the pod
 without passing through the sidecar proxy.

- **Least privilege security inside of pod:** By locking down communication
 on container and process level inside of the pod using socket-aware BPF
 programs, a security breach in one of the containers no longer leads to the
 entire pod being compromised. This is in particular important as sidecar
 containers have direct network access to any other container in the pod and
 may access potentially sensitive information.

**Fun fact:** Socket-aware BPF programs are not limited to IP-based sockets.
The security policies can be extended to cover UNIX domain sockets as well.

##### TLS visibility for external services (Work in Progress)

Istio relies on visibility into the application protocol layer such as HTTP to
provide functionality such as URI-based routing, HTTP headers-based
authorization and API request level telemetry and tracing. This visibility is
achieved by keeping the application traffic between services unencrypted and
performing the TLS encryption between the sidecar proxies of the source and
destination service by applying mutual TLS with certificates managed by Istio
Citadel.

This works well for connectivity between services inside the mesh.
Communication to services outside of the mesh is almost guaranteed to be TLS
encrypted by the application itself in order to secure communication on the
untrusted internet.

The kTLS (Kernel TLS) work for Linux has originally been **[proposed by Dave
Watson](https://netdevconf.org/1.2/papers/ktls.pdf)** at Facebook. It includes
the capability to defer the symmetric encryption of data from the SSL library
in the application to the socket in the Linux kernel. The initial motivation
for this work was a pure performance optimization as the use of kTLS turned out to
be 3-4% faster than SSL libraries. This was interesting enough for providers of
static data over SSL to continue the effort. The work has since been merged and
is available in the latest Linux kernels. Combined with the capabilities of BPF
programs injected at the Linux socket layer, kTLS enables transparent
visibility of data that would otherwise already be encrypted at this point.

We are still working on this feature but have an initial functional prototype
that proves the concept of gaining visibility and control over data that is
typically TLS encrypted in the application itself. The next step is to make
this data available to the sidecar to unleash Istio on connections to external
services.

###### How can kTLS be transparent without requiring application changes?

To be fair, the SSL library used by the application must be aware of kTLS and
the user must opt into the behavior on a system level. The application itself
must not be changed though and enabling the visibility can be as simple as
upgrading the version of the SSL library that the app is using.

Because the visibility is gained at the socket level, this also means that the
application can no longer change this data after it has been enqueued. This is
in particular important if the data is being used to authorize a request.

##### Summary

- **Increased Istio Security:**
 - **Least privilege security inside of pod:** By locking down communication
 on container and process level inside of the pod using socket-aware BPF
 programs, a security breach in one of the containers no longer leads to the
 entire pod being compromised.
 - **Secure non-IPv4/TCP protocols:** Istio is TCP and IPv4 only at this
 point. Any other protocol such as UDP or IPv6 will bypass the sidecar proxy
 and any Istio security rules. Cilium will step in either by
 completely blocking these protocols or by applying fine-grained security
 rules.
 - **Protect from compromised sidecars:** The sidecar proxy itself is not
 subject to the Istio security rules. A compromised sidecar has unrestricted
 access to the network. It can also manipulate its own security rules to
 become more permissive. Cilium allows to define service level security
 policies in addition to Istio and ensures that a compromised sidecar proxy
 can only operate with least privelege.
 - **Force application traffic through sidecar:** By using socket-aware BPF
 policies, Cilium can ensure that all communication of an application must
 occur via the sidecar for both incoming and outgoing connections. The
 application container itself is guaranteed to not be able to receive a
 connection from outside the pod or make a request to outside of the pod
 without passing through the sidecar proxy.

- **External services:**
 - **Istio for TLS encrypted connections:** TCP connections to services outside of
 the cluster are typically TLS encrypted and Istio cannot inspect the HTTP
 headers of such connections. Cilium will be capable of providing visibility
 and control over TLS encrypted data by leveraging the kTLS integration into
 BPF in the kernel.
- **Performance:**
 - **Efficient networking:** Istio requires a CNI plugin to provide network
 connectivity between pods. Cilium implements the CNI plugin interface and
 utilizes BPF to provide a highly efficient datapath between the sidecar
 proxies and the Istio control plane.
 - **Socket redirection to accelerate Istio:** Cilium can accelerate the
 traffic redirection to the sidecar proxy by performing the redirection of
 the traffic at Linux socket level using socket-aware BPF programs. This
 allows bypassing the expensive TCP/IP stack traversal without changing any
 code in the application or in the sidecar.

If you are interested to learn more:

- **[Try a Getting Started Guide](http://docs.cilium.io/en/stable/gettingstarted/)**
- **[Learn more about Cilium concepts](http://docs.cilium.io/en/stable/concepts/)**
- **[Join our Slack channel](https://slack.cilium.io/)**

#### Prometheus Metrics for Kubernetes Networking Using Cilium

URL: https://cilium.io/blog/2018/06/28/cilium-prometheus
Date: 2018-07-17
Categories: How-To

Cilium provides complete L3/L4 and L7 (API-aware) security for Kubernetes services. Cilium is deployed as a CNI and has deep visibility into the health and performance of both the networking and security enforcement. Cilium exposes several metrics in the popular Prometheus format along with
a simple Grafana dashboard template for analyzing the metrics. In this post, we will setup the Prometheus and Grafana services and look at the key Cilium metrics.

##### Overview of Cilium Metrics

Before we go through the tutorial, below are some of the key metrics provided by Cilium.

- `cilium_endpoint_count`: The number of "endpoints", i.e pods, managed by Cilium. These are all the pods for which Cilium can enforce policies. This metric can be used to ensure that all the application pods are managed by Cilium by comparing the endpoint count with the pod count on a worker node.
- `cilium_endpoint_regenerating`: The number of endpoints in "regenerating" status. The regenerating status conveys that the endpoint’s networking configuration is being (re)generated. This includes (re)programming BPF for that endpoint. One common reason for regeneration is when new Cilium security policies are applied.
- `cilium_drop_count_total`: Captures the total number of packets dropped including the direction i.e. dropped at ingress or egress and reasons for drop such as due to L3/L4 security policies. In case of drops due to security policy, the spike in drop count can indicate an increase in either illegitimate access attempts or highlight a need to adjust the security policy to allow a legitimate access.
- `cilium_forward_count_total`: Total number of packets forwarded. Similar to the drop count, a sudden drop in forwarded count should be monitored for potential connectivity issues or side-effects of security policy changes.
- `cilium_policy_l7_received_total`: Total count of L7 received requests/responses. The count is only increased if there is an L7 policy which requires the traffic to be parsed and filtered by an L7 proxy.
- `cilium_policy_l7_denied_total`: Total count of access attempts that were denied by an L7 security policy. Similar to the L3/L4 drop count, the spike in L7 denied count indicates a need to either investigate illegitimate attempts or a need to adjust security policies.
- `cilium_policy_l7_forwarded_total`: Total count of L7 forwarded requests/responses (i.e allowed by the L7 policy).
- `cilium_policy_l7_parse_errors_total`: Total count of L7 parsing errors prior to any L7 policy enforcement.
- `cilium_unreachable_nodes`, `cilium_unreachable_health_endpoints`: These metrics identify any connectivity issues between nodes and Cilium health endpoints. They are generated by the periodic connectivity health checks performed by Cilium.

[Cilium Monitoring & Metrics](http://docs.cilium.io/en/v1.1/configuration/metrics/) has the complete description of all the Cilium metrics.

##### Setting up Cilium + Prometheus + Grafana

If you don't have a Kubernetes cluster with Cilium, then you can easily create one using:

- [kops](http://docs.cilium.io/en/v1.1/install/guides/kops/) to create a multi-node cluster on AWS
- [minikube](http://docs.cilium.io/en/v1.1/gettingstarted/minikube/)

Once you have a Kubernetes cluster with Cilium, the following commands will install Prometheus in `prometheus` namespace and Grafana in the `monitoring` namespace.

```bash
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.1/examples/kubernetes/addons/prometheus/prometheus.yaml

kubectl create -f https://raw.githubusercontent.com/giantswarm/kubernetes-prometheus/master/manifests/grafana/deployment.yaml

kubectl create -f https://raw.githubusercontent.com/giantswarm/kubernetes-prometheus/master/manifests/grafana/service.yaml
```

**Restart the Cilium daemons** after finishing the installation of Prometheus and Grafana.

```bash

kubectl delete po -l k8s-app=cilium -n kube-system

```

##### Cilium Metrics in Prometheus and Grafana

To test whether the Cilium metrics are getting collected by Prometheus, you can go to the Prometheus dashboard at `any_worker_node_ip:prometheus_nodeport`. If worker nodes are not accessible, such as in a private VPC, then use the kubernetes port-forwarding.

```bash
kubectl port-forward <prometheus-pod-name> -n prometheus 9090:9090
```

You should see the Cilium metrics in the drop down.

Similarly, access the Grafana dashboard using grafana service NodePort or using port-forwarding.

```bash
kubectl port-forward <grafana-pod-name> -n monitoring 3000:3000
```

For Grafana, the default user/password is `admin:admin`. Next, we will add the Prometheus datasource to Grafana.

- Give a name to the Prometheus datasource and select `type = Prometheus`
- `Url = http://prometheus.prometheus.svc.cluster.local:9090`
- `Access = proxy`
- Click on `Save & Test` and make sure you the datasource is working message.

We have published a Grafana dashboard for Cilium Metrics. You can simply import this to get started. In the Dashboards/Import, enter the URL`https://grafana.com/dashboards/6658`.

##### Testing Cilium Metrics

We will monitor the L3/L4 denied and L7 denied requests. A spike in these requests suggests an increase in the traffic that is getting blocked due to the security policy. These could either be genuine traffic getting denied or illegitimate access attempts, both of which are good to monitor and investigate. We will use the example from the [Cilium Getting Started Guide](http://docs.cilium.io/en/v1.1/gettingstarted/minikube/#step-2-deploy-the-demo-application). In this app the `xwing` spaceships are denied access (L3/L4 policy) to `deathstar` service. And the `tiefighter` spaceships are allowed access only to `HTTP POST deathstar/v1/request-landing` and all other HTTP requests are denied (L7 policy).

```bash

# Create the app in default namespace
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.1/examples/minikube/http-sw-app.yaml

# Wait for all the pods and services of the app to get to the running state
kubectl get pods,svc

# Apply the L3/L4 and L7 Policy.
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.1/examples/minikube/sw_l3_l4_l7_policy.yaml

```

Now we will simulate two scenarios and monitor associated metrics in Grafana Cilium dashboard. First, run an attempt to access `deathstar` from `xwing`. This should be denied at the L3/L4-level and we should see an increase in the `Cilium Drop Count` chart. Since the policy is enforced at ingress on the `deathstar` pods, the drop count will increase for the `direction: INGRESS`.

```bash
while true; do kubectl exec xwing -- curl -sL -XPOST deathstar.default.svc.cluster.local/v1/request-landing; done
```

The `Cilium Drop Count (INGRESS)` metric will start showing an increase in `Policy denied (L3)`, capturing the security enforcement against the illegitimate access attempts.

Second, run an attempt to `GET` the list of API endpoints served by `deathstar` from `tiefighter`. This should be denied at the L7-level and we should see an increase in the `L7_denied_total` in `L7 Requests` chart.

```bash
while true; do kubectl exec tiefighter -- curl -sL -XGET deathstar.default.svc.cluster.local/v1; done
```

##### Conclusion

Cilium's support for the standard Prometheus format makes it easy to monitor valuable metrics related to the effectiveness of security policies and health of service interactions. In upcoming releases, we will continue to add more metrics from Cilium. If you are already using Cilium and have suggestions for other useful metrics then let us know on [Cilium slack](https://slack.cilium.io/).

#### Open Source Elasticsearch Security: Using Cilium for Elasticsearch Access Control with No App/Container Changes

URL: https://cilium.io/blog/2018/07/10/cilium-security-elasticsearch
Date: 2018-07-10
Categories: How-To

##### Elasticsearch Concepts and Least Privilege Security

Elasticsearch APIs provide mechanisms for storing and searching JSON _documents_. Documents are grouped into an _index_, which typically stores many documents of a single type (e.g., user transaction data, or network access logs). A user of the Elasticsearch API selects an index, and then may create, update, delete, or search documents within that index. Multi-tenant Elasticsearch clusters are commonly accessed by many clients running as part of many different applications workloads, any of which may contain a security vulnerability or other weakness that allows an attacker to issue API requests from that application to the Elasticsearch server.

In such environments, it is critical to enforce **least privilege security**, which is the security principle that each component of a larger system has exactly the permissions it requires and no more. Enforcing least privilege on an Elasticsearch API client typically amounts to limiting that client to accessing a particular index, and even more precisely, to performing only certain operations on that index. In the example above, least privilege security would mean the attacker can add erroneous data to the troubleshooting logs index (since that behavior is required for normal application operation), but the attacker would not be allowed to search those logs, or perform any operation on the indices for user transaction data or network logs.

##### Application-Layer vs. Infrastructure-Layer Enforcement

One approach to solving this problem is to modify Elasticsearch client and server code to enforce such security policies using Role-based Access Control (RBAC) at the application-layer. In fact, a [commercial solution from Elastic.co](https://www.elastic.co/guide/en/elastic-stack-overview/current/authorization.html) provides similar functionality.

However, from the perspective of the IT ops and security teams, relying on such an approach has significant limitations:

- The tooling for gaining visibility, enforcing security policies, troubleshooting issues, performing audit, and others are all Elasticsearch-specific, and thus require additional tooling for securing all other API-driven workloads in the environment (e.g., HTTP/gRPC-based microservices, queuing services like Kafka, and datastores/caches like MySQL, Postgres, Cassandra, MongoDB, Redis). The end result is a siloed collection of per-API security mechanisms unable to provide a consistent mechanism for visibility + security across a complex modern microservices application.
- Enabling these mechanisms requires gaining the buy-in of each application team with a client that accesses the multi-tenant Elasticsearch cluster. Additionally, the application workloads themselves (e.g., their container images) must be updated with new configurations as security policies change. This tight coupling between application artifacts and the details of security visibility and enforcement leads to additional points of friction between application teams and security teams.

For these reasons, IT Ops and security teams have long preferred "transparent" infrastructure-layer security solutions that require no changes to the application workloads. The widespread use of infrastructure-layer _network firewalls and segmentation_ for security isolation between workloads is a testament to a solution that provides both:

- A single solution for visibility, policy enforcement, troubleshooting, and audit capabilities across all types of workloads.
- A way of transparently inserting visibility and enforcement entirely independent of application development teams.

But traditional firewalls would typically only see and filter traffic at the granularity of an IP address and port. Since the server offering the Elasticsearch API provides access to all API calls on a single port (TCP 9200), it is impossible for such traditional firewalls to see or filter based on the content of individual API calls.

This is where Cilium comes in. Cilium has the ability to see into the individual API calls, and enforce policies that allow least privilege API access for Elasticsearch and any other API services based on protocols like HTTP, gRPC, and Kafka. And while traditional firewalls often run as centralized "choke-points", Cilium leverages a powerful new Linux kernel technology called [BPF](https://cilium.readthedocs.io/en/stable/bpf/) to transparently insert security intelligence into the Linux kernel running the workload itself (e.g., a Kubernetes node running a an Elasticsearch client or server pod). Visibility and security policies are efficiently implemented as network data flows through the kernel to/from the container, with no detours to a centralized firewall or proxy.

##### Example: Elasticsearch API Security Using Cilium with Kubernetes

The Cilium is open source and is deeply integrated with container orchestration platforms like Kubernetes, Mesos and Docker. Leveraging BPF, Cilium is able to observe all the traffic coming in/out of each container/pod, and provide both visibility and security based on the identity of the application container/pod (e.g., pod labels). Unlike ephemeral identifiers like IP addresses, the labels provide reliable identity even as pods restart or scale-in/out.

Following the Cilium tradition, we will use a Star Wars-inspired example to understand how Elasticsearch-aware security works in practice.

Just like many enterprises today, the Empire DevOps team has a large scale Elasticsearch cluster which is used for storing a variety of data. Specifically, we will focus on the following two indices:

1. `"_index": "troop_logs"` stores Stormtroopers daily attendance logs collected from every Empire outpost across the galaxy to ensure Stormtroopers are not skipping out on work.
2. `"_index": "spaceship_diagnostics"` stores diagnostics data collected from every spaceship, to be used for troubleshooting by Empire engineers and improving future spaceship designs.

Every Empire outpost pod has an Elasticsearch client service to upload the Stormtroopers logs. And every spaceship pod has a service to upload diagnostics. And the Empire headquarters has a service to search and analyze the troop logs and spaceship diagnostics data. You can easily create the full example application in a minikube environment with Cilium by following the [Cilium Elasticsearch Getting Started Guide](http://cilium.readthedocs.io/en/stable/gettingstarted/elasticsearch/).

From a security perspective, given the large number of outposts scattered across the galaxy, any single outpost pod might be to be captured by the Rebels. Once compromised, the outpost clients can be used to search and manipulate the critical data in any index. For example, the Rebels can search the `"_index": "spaceship_diagnostics"` for information about the current location of the Imperial fleet or to identify critical vulnerabilities in spaceship design (sound familiar?).

Following the least privilege security principle, we want to the allow the following legitimate actions and nothing more:

- Services with label `outpost` can only add documents to `"_index": "troop_logs"`
- Services with label `spaceship` can only add documents to `"_index": "spaceship_diagnostics"`
- The `empire-hq` Service is able to search both the indices.

Fortunately (well, not for the Rebels...), the Empire DevOps team is using Cilium for their Kubernetes cluster. Cilium follows the **least privilege model** for security, so a _CiliumNetworkPolicy_ contains a list of rules that define **allowed requests** and any request that does not match the rules is denied. This example leverages the fact that most Elasticsearch clients use the RESTful HTTP API to access Elasticsearch data.

The resulting Cilium Network Policy has the following key elements:

1. The policy rules are defined for inbound traffic (i.e., "ingress") connections to the _elasticsearch_ service. That is, the enforcement will be done when the API request arrives at any of the pods that are part of the Elasticsearch service. The label `component: elasticsearch` selects the pods that are part of the _elasticsearch_ service
2. Based on the identity of the client service (i.e. the source of traffic), the following rules are defined for controlling access to indices access and actions performed:
 - `fromEndpoints` with labels `app:spaceship` only `HTTP PUT` is allowed on paths matching regex `^/spaceship_diagnostics/stats/...$`
 - `fromEndpoints` with labels `app:outpost` only `HTTP PUT` is allowed on paths matching regex `^/troop_logs/log/...$`
 - `fromEndpoints` with labels `app:empire` only `HTTP GET` is allowed on paths matching regex `^/spaceship_diagnostics/_search/...$` and `^/troop_logs/search/...$`

([View full policy YAML](https://raw.githubusercontent.com/cilium/cilium/v1.1/examples/kubernetes-es/es-sw-policy.yaml))

The policy enforcement is as simple as running a single `kubectl` command:

```bash
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.1/examples/kubernetes-es/es-sw-policy.yaml
```

The Cilium agents on each Kubernetes worker node monitor for new _CiliumNetworkPolicy_ objects and will promptly reconfigure the BPF programs managing network traffic in/out of each pod to implement the new policy.

##### Conclusion

Cilium leverages BPF, a powerful new Linux kernel technology, to provide API-aware network visibility and security with zero changes to the application code or containers. Cilium runs fully distributed on the Linux nodes running your workloads, avoiding any centralized network choke-points. The integration with orchestration frameworks such as Kubernetes allows Cilium to combine the traffic visibility with the pod identity, and enforce right security policies even as services scale-up/down or pods get restarted on different nodes. Applying Cilium's API-aware security policies to the Elasticsearch API enables least privilege security on an Elasticsearch cluster efficiently, without interfering with the application.

We welcome feedback/questions about using Cilium with Elasticsearch. Please reach out to us on [Slack](https://slack.cilium.io/) or [Twitter](http://www.twitter.com/ciliumproject).

And while you're at it, we encourage you to try out our Cilium Getting Started Guide for securing other common API protocols such as [HTTP](http://cilium.readthedocs.io/en/stable/gettingstarted/minikube/), [Kafka](http://cilium.readthedocs.io/en/stable/gettingstarted/kafka/) and [gRPC](http://cilium.readthedocs.io/en/stable/gettingstarted/grpc/).

#### Cilium 1.1: Istio sidecar mode, cri-o/containerd support, improved efficiency & scale, init policies

URL: https://cilium.io/blog/2018/06/26/cilium-11
Date: 2018-06-26
Categories: Release

We are excited to announce Cilium 1.1. 33 contributors have contributed 964
commits to this release. Below is a list of highlighted features and
architectural improvements that have made the 1.1 release in addition to the
countless bugfixes.

##### What is Cilium?

Cilium is open source software for transparently securing the network
connectivity between application services deployed using Linux container
management platforms like Docker and Kubernetes.

At the foundation of Cilium is a new Linux kernel technology called BPF, which
enables the dynamic insertion of powerful security visibility and control logic
within Linux itself. Because BPF runs inside the Linux kernel, Cilium security
policies can be applied and updated without any changes to the application code
or container configuration.

See the section [Introduction to Cilium](http://docs.cilium.io/en/v1.1/intro/)
for a more detailed introduction to Cilium.

##### Highlights

- **[Deep Istio Integration](#DeepIstioIntegration)**
 - **mTLS compatibility:** New alternative mode to enforce Cilium application
 protocol security policies directly in the [Istio] sidecar proxy managed by
 [Pilot] to support application level policy enforcement when [Mutual TLS]
 is in effect. Pod and port-level policies continue to be enforced outside
 of the pod.
 - **Istio guide:** New getting started guide based on Istio 0.8.0 release
 that features Helm charts to deploy Istio.
 - **Init policies:** A new init identity covers the time span of a pod while it
 is being initialized, i.e. while the labels and or policy of an endpoint is
 not known yet. The init policy enforces a configurable policy. It is in particular importance to Istio architectures because the sidecar proxy of a pod is required to have privileges to communicate
 with the control plane running in the `istio-system` namespace.
- **Support for additional container runtimes**
 - The runtimes are automatically detected as reliably as possible but can
 also be explicitly specified using the `--container-runtime` option.
 - **[cri-o](https://github.com/kubernetes-incubator/cri-o)**: Includes an
 extension of the [minikube getting started
 guide](https://cilium.readthedocs.io/en/v1.1/gettingstarted/minikube/) with
 the cri-o specific deployment steps.
 - **[containerd](https://github.com/containerd/containerd)**
- **Additional Network Security for Kubernetes**
 - **podSelector && namespaceSelector:** Support for the new combined
 podSelector and namespaceSelector in NetworkPolicy as introduced in
 Kubernetes 1.11.
 - **Service accounts:** Ability to match on the [Kubernetes Service Account] association of a pod. Please
 [see example below](#ServiceAccount).
 - **NodePort security:** Ability to differentiate between local host and
 traffic that is SNATed to the node IP when entering the node. This allows
 differentiation between host traffic performing health checks and external
 accesses via NodePort. Old behavior can be preserved with the
 `--k8s-legacy-host-allows-world` option.
 - **Changing pod labels:** The policy enforcement layer now supports
 containers and pods changing their labels on the fly.
 - **Policy correlation:** Annotations of a CiliumNetworkPolicy are now
 mirrored in the status field for each node. This simplifies the correlation of
 what policy is being enforced on which node.
- **Extended IP/CIDR policy enforcement capabilities**
 - **Combined IP+L4/L7:** Support to specify port and application protocol
 (L7) rules that only apply in combination with IP/CIDR matching.
 - **Unlimited # of CIDR prefix lengths:** CIDR enforcement implementation with new BPF
 longest-prefix-match map when available. Leads to support of unlimited number of prefix lengths.
- **Improved connection tracker efficiency**
 - **CT cleanup on deny:** Removal of connection tracking entries when policy
 denies the traffic. This is possible because the policy enforcement cost is
 only O(1).
 - **Improved UDP conntrack:** More aggressive cleaning of connection
 tracking table for non-TCP traffic. This primarily improves resource usage
 of workloads such as [Prometheus] metrics scraping causing a continuously large
 number of DNS lookups per second.
- **Efficiency & Scale**
 - **Large identity count environments:** Massive improvement of identity
 allocation performance in environments with several thousand workload
 identities.
 - **MTU improvements:** Better MTU handling by implementing the encapsulation
 packet overhead via the MTU metric of the transmission route to allow using
 the full MTU on receive. This reduces the probability of fragmentation and
 packet drops.
- **Additional Prometheus metrics**
 - **L3/L4/L7 forwards/drops:** Counters for all forwarded and rejected
 traffic on both packet and application protocol request layer. Packet level
 metrics are exported directly from the BPF datapath using efficient per-CPU
 maps. Application protocol metrics are exported by the proxies.
 - **Status as a metric:** Representation of all status-relevant failure
 scenarios such as the number of failing controllers.
- **Reliability Work**
 - **Support for changing host IPs:** If you add or change one of the
 IPs of the host, it will be properly detected and policy is applied
 accordingly. This is made possible by replying to all ARP requests with the
 virtual MAC address of the Cilium router regardless of the IP being
 requested as all traffic is always L3 forwarded.
 - **Continous BPF synchronization:** Synchronization of policy to BPF maps is
 now done via controllers. If something modifies the state of the BPF maps
 other than Cilium, the state in the BPF map is automatically fixed again.
 - **Reuse of devices & routes:** Network devices and routes are no longer
 re-created but modified if possible to ensure continued connectivity across
 agent restarts.
 - **Synchronous CNI plugin:** The CNI plugin is now performing the plumbing
 in a synchronous fashion. This guarantees that networking is being
 provided from the moment the application container is being spawned. See
 [init policy](#InitPolicy) to define policy privileges for the duration
 when workload identity is not known yet.
 - **TCP keepalive support:** Envoy and the Kafka proxy now enable TCP
 keepalive by default to ensure that persistent connections are never
 subject to connection tracking expiration even if no data is being sent for
 days.
 - **IPv6:** Improved handling of unsupported IPv6 extension headers.
- **Operations**
 - **Require k8s PodCIDR allocation:** New agent options
 `--k8s-require-ipv4-pod-cidr` and `--k8s-require-ipv6-pod-cidr` to require
 the Kubernetes PodCIDR to be provided by Kubernetes via the Node resource.
 - **IPv6:** New `--ipv6-cluster-alloc-cidr` option to specify the IPv6 CIDR
 when Cilium allocates the per node IPv6 CIDR.
 - **CNI compatibility:** Rename of default CNI configuration name from
 `10-cilium.conf` to `00-cilium.conf` to simplify plugging Cilium into
 existing Kubernetes environments as some CNI plugins do not remove the
 configuration file when they get uninstalled.
 - **State pruning:** New `clean-cilium-state` option in the Kubernetes
 ConfigMap which will trigger running an [init container] when the Cilium
 pods starts up to clean all existing state before Cilium starts up.
 - **BPF filesystem:** Improved automatic mounting of the BPF filesystem when
 Cilium is being run in a separate mount namespace.
 - **Ubuntu 18.04 base image:** The base image for the Cilium container image has been
 upgraded to Ubuntu 18.04.
- **Documentation**
 - **Kubernetes versions:** Documentation now features multiple tabs to
 provide example YAML files for different Kubernetes versions to account
 for different resource naming versioning requirements.
 - **Istio GSG:** New [getting started guide with Istio 0.8](https://cilium.readthedocs.io/en/v1.1/gettingstarted/istio/).
 - **cri-o:** New [getting started guide using cri-o](https://cilium.readthedocs.io/en/v1.1/gettingstarted/minikube/).
 - **Elasticsearch:** New [Elasticsearch getting started guide](http://docs.cilium.io/en/v1.1/gettingstarted/elasticsearch/).
 - **BPF reference guide:** Additions to the [BPF reference guide] including
 sections on XDP, iproute2, and LLVM.

##### Deep Istio integration

Cilium deeply integrates with [Istio]. Cilium operates as a CNI plugin and
provides connectivity as well as transparent security starting packet level
all the way up to API level. Among many things, Istio can provide [Mutual TLS]-based authentication between Istio managed services as well as authorization.
Both are implemented with the help of a sidecar proxy running inside of the
application pod. When running Istio in combination with Cilium, Cilium can:

- Secure the Istio sidecar and control plane. More on this below.
- Run in a [Mutual TLS]-compatible configuration
 allowing Cilium to enforce Cilium security policies using the Istio sidecar
 architecture.
- Enhance the performance of Istio and Envoy by reducing the
 overhead introduced by the sidecar architecture. More details on this can be
 found in this separate [blog post](/blog/2018/04/24/cilium-10).

###### Restrict unsupported protocols

Istio ignores network traffic for protocols that are not supported by Istio.
This includes all UDP, ICMP and IPv6 traffic. Traffic using these unsupported
protocols is thus not subject to Istio's authentication and authorization rules
and will bypass enforcement.

Cilium guarantees enforcement of all security policies outside of the pod
regardless of the protocol being used. Cilium follows a strict whitelist model
which will result in rejection of any unknown traffic. This allows restriction of
traffic with protocols not supported by Istio and cover scenarios such as:

- Prevent a compromised pod to leak information using a UDP based gossip
 protocol by only allowing UDP traffic to kube-dns running in the
 kube-system namespace.
- Apply security policies to TCP ports which are excluded from the sidecar
 redirection logic. This could include restriction of traffic to only the
 port that is being redirected to the sidecar.
- Prevent a compromised pod to leak information to a public IPv6 address
 which would otherwise bypass the proxy.

###### Securing the Sidecar

The sidecar proxy itself is not subject to any security rules as the proxy is
being excepted from the redirection logic else it would cause a continuous loop.
As Cilium provides enforcement outside of the pod, the traffic of a potentially
compromised sidecar proxy is still subject to the security policies rules by:

- Limiting communication to allowed services in the cluster to complement
 [Mutual TLS]. This is particularly important as a compromised sidecar gains
 access to all other services that are _not_ using [Mutual TLS] because there is
 no ingress protection on the receiving side of the service.

- Preventing a pod from leaking sensitive information by either not allowing the
 pod to communicate outside of the cluster at all or by limiting it to well-known
 IP/CIDR ranges on well known ports.

###### Securing the Control Plane

All Istio sidecars communicate with the Istio control plane that is deployed
within the cluster. This communication is required for operations and
application pods are required to have access to these services. Here are a
few examples of how Cilium improves security of the overall architecture:

- Only allow application pods that have been injected with an Istio sidecar
 to have access to the control plane. This can be achieved having Cilium
 policies match on Istio annotation added to pods during injection.
 Pods without an injected Istio sidecar proxy should not have access to the
 control plane.

- The Istio control plane collects a lot of sensitive information as it manages
 certificates, performs tracing and host authorization logic. The control
 plane components obviously must be subject to security policies to prevent
 leaking of this information.

###### mTLS-compatible API-aware security policies

Prior to the 1.1 release, use of the Istio [Mutual TLS] functionality
encrypted all of the TCP traffic between services, which restricted the capability
of Cilium to enforce API-aware security policies for such services. Starting
with Cilium 1.1, Cilium is capable of reusing the Envoy instance running as a
sidecar inside the pod to enforce the Cilium security policies.

No change to the policies is required. All API level policies will be enforced
in the sidecar and all policies on a pod/service and port level continue to be
applied outside of the pod. Thus it will continue to include network traffic
that is currently unsupported by Istio.

Please follow the [Istio Getting Started Guide] to learn how to run Cilium in
the [Mutual TLS] compatible mode.

###### Init Policy

Security labels are bound to pod and container labels. Certain labels are only
associated with a pod while the pod is being initialized. Consequently, the
privileges granted by the policy matching on such labels are only applied while
the pod is being initialized. This can lead to lack of connectivity
while a pod or container initialization. Cilium 1.1 introduces a new init
policy concept which allows definition of privileges which should be applied to pods and
containers that are being initialized.

**Kubernetes Example:**

```
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: init-allow-dns
specs:
  - endpointSelector:
      matchLabels:
        "reserved:init": ""
    egress:
    - toEntities:
      - all
      toPorts:
      - ports:
        - port: "53"
          protocol: UDP
```

The above example allows all pods in the initialization phase to emit traffic on
port 53/UDP regardless of the destination. Instead of an entities match on
`all`, this policy could also match on the labels of kube-dns
`k8s-app=kube-dns`, see [kube-dns policy example] for more information.

###### Kubernetes Service Account Policy

The service account of a pod is either defined via the [service account admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#serviceaccount)
or can be directly specified in the Pod, Deployment, ReplicationController
resource like this:

```
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: leia
  ...
```

The following example grants any pod running under the service account of
"luke" to issue a `HTTP GET /public` request on TCP port 80 to all pods
running associated to the service account of "leia".

```
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "k8s-svc-account"
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: leia
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: luke
    toPorts:
    - ports:
      - port: '80'
        protocol: TCP
      rules:
        HTTP:
        - method: GET
          path: "/public$"
```

##### Upgrade Notes

###### Changed Behavior: External traffic no longer classified as host

In Cilium 1.0, all traffic from the host, including from local processes and
traffic that is masqueraded from the outside world to the host IP, was
classified as from the `host` entity (`reserved:host` label).

Cilium 1.1 introduces the capability to differentiate between traffic emitted
from local processes and traffic that was merely masqueraded on the host. This
provides additional security control in environments where masquerading cannot
be disabled for one reason or another.

In order to not break any existing deployments, the following migration step is
required:

- An existing Kubernetes `DaemonSet` with an existing `ConfigMap` will preserve
 the behavior. The user is required to edit the `ConfigMap` `cilium-config` and
 add the option: `legacy-host-allows-world: "false"`.

- New deployments will automatically opt into the new behavior as the default
 `ConfigMap` already contains the option `legacy-host-allows-world: "false"`.

###### Changed Behavior: MTU

Cilium 1.0 by default configured the MTU of all Cilium-related devices and
endpoint devices to 1450 bytes, to guarantee that packets sent from an endpoint
would remain below the MTU of a tunnel. This had the side-effect that when a
Cilium-managed pod made a request to an outside (world) IP, if the response
came back in 1500B chunks, then it would be fragmented when transmitted to the
`cilium_host` device. These fragments then pass through the Cilium policy
logic. Latter IP fragments would not contain L4 ports, so if any L4 or L4+L7
policy was applied to the destination endpoint, then the fragments would be
dropped. This could cause disruption to network traffic.

Cilium 1.1 fixes the above issue by increasing the MTU of the Cilium-related
devices and endpoint devices to 1500B (or larger based on container runtime
settings), then, given Cilium is configured to run in tunneling mode,
configures a route within the endpoint at a lower MTU to ensure that
transmitted packets will fit within tunnel encapsulation. This addresses the
above issue for all new pods.

Upgrading to Cilium 1.1 will not automatically adjust the MTU of existing pods.
Pods must be restarted in order for them to receive the new MTU setting. New
pods will automatically be configured with the improved MTU settings.

##### Upgrade Instructions

As usual, follow the [upgrade
guide](https://cilium.readthedocs.io/en/v1.1/install/upgrade/) to upgrade your
Cilium deployment. Feel free to ping us on [Slack](http://cilium.io/slack).

##### Release

- Container image: `docker.io/cilium/cilium:v1.1.0`

###### Binaries

- [cilium-agent-x86_64](http://releases.cilium.io/v1.1.0/cilium-agent-x86_64) ([c38ff8680d8e05552036](http://releases.cilium.io/v1.1.0/cilium-agent-x86_64.sha256sum))
- [cilium-bugtool-x86_64](http://releases.cilium.io/v1.1.0/cilium-bugtool-x86_64) ([569dfc27d5ac035a538c](http://releases.cilium.io/v1.1.0/cilium-bugtool-x86_64.sha256sum))
- [cilium-health-x86_64](http://releases.cilium.io/v1.1.0/cilium-health-x86_64) ([9c4142dde33ee32aef27](http://releases.cilium.io/v1.1.0/cilium-health-x86_64.sha256sum))
- [cilium-node-monitor-x86_64](http://releases.cilium.io/v1.1.0/cilium-node-monitor-x86_64) ([aff6d94547905f6f3561](http://releases.cilium.io/v1.1.0/cilium-node-monitor-x86_64.sha256sum))
- [cilium-x86_64](http://releases.cilium.io/v1.1.0/cilium-x86_64) ([4b20d05abf571ed28e53](http://releases.cilium.io/v1.1.0/cilium-x86_64.sha256sum))
- [v1.1.0.tar.gz](http://releases.cilium.io/v1.1.0/v1.1.0.tar.gz) ([3dfda64d8bb9733ae137](http://releases.cilium.io/v1.1.0/v1.1.0.tar.gz.sha256sum))
- [v1.1.0.zip](http://releases.cilium.io/v1.1.0/v1.1.0.zip) ([3afbfca4e94a90c3ad0b](http://releases.cilium.io/v1.1.0/v1.1.0.zip.sha256sum))

[istio]: https://istio.io/
[pilot]: https://istio.io/docs/concepts/traffic-management/pilot/
[mutual tls]: https://istio.io/docs/concepts/security/mutual-tls/
[prometheus]: https://github.com/prometheus/prometheus
[init container]: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
[bpf reference guide]: https://cilium.readthedocs.io/en/v1.1/bpf/
[kube-dns policy example]: https://github.com/cilium/cilium/blob/master/examples/policies/kubernetes/namespace/kubedns-policy.yaml
[kubernetes service account]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
[istio getting started guide]: https://cilium.readthedocs.io/en/v1.1/gettingstarted/istio/

#### Cilium 1.0: Bringing the BPF Revolution to Kubernetes Networking and Security

URL: https://cilium.io/blog/2018/04/24/cilium-10
Date: 2018-04-24
Categories: Release

The last couple of months have been tremendously exciting for everyone working
on Cilium and BPF. We have witnessed a fast growing community of
Cilium users as well as the rapid increase of BPF usage and development with
companies such as [Google] joining the existing already strong BPF community of
engineers from [Facebook], [Netflix], [Red Hat] and many more. Possibly the strongest
signal on the success of BPF has been the decisions of the Linux kernel
community to [replace the in-kernel implementation of iptables with BPF](/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables).

All of this has allowed us to advance BPF quickly and mature the Cilium project
very effectively. Our warmest
shoutouts go to everyone who has joined us on this incredible
journey since we initially announced Cilium at [DockerCon
2017](https://www.youtube.com/watch?v=ilKlmTDdFgk). Your support in the form of
contributing code, providing feedback and spreading the word has been
incredible.

##### Cilium 1.0: Stable API and LTS releases

Today, we mark the release of Cilium 1.0 and assign the first ever stable
release number. Starting with this release, we will also provide all of the
guarantees and processes required to run Cilium in production environments:

- [API stability](http://docs.cilium.io/en/doc-1.0/api/#compatibility-guarantees)
 with upgrade and downgrade compatibility guarantees.
- [Stable releases](http://docs.cilium.io/en/doc-1.0/contributing/#release-process)
 with LTS support for production use.
- Well defined process to handle security relevant bugs and incidents.
- Responsive support on [Slack] and [GitHub] as well
 as defined processes to report bugs and request features.

##### Why Cilium?

This blog post focuses on the functionality provided by the Cilium 1.0 release.
We have published a separate post that provides additional background and use
cases as well as a preview of the service mesh/sidecar acceleration work.

> [Cilium - Rethinking Linux Networking and Security for the Age of Microservices](/blog/2018/04/24/cilium-security-for-age-of-microservices)

##### What is Cilium?

The following list describes the functionality that Cilium provides as of 1.0.
You can find more detailed descriptions in the [functionality overview](http://docs.cilium.io/en/doc-1.0/intro/#functionality-overview)
section of the Cilium documentation.

- **Highly efficient BPF datapath:**
 BPF is the underlying Linux superpower doing the heavy lifting on the
 datapath by providing sandboxed programmability of the Linux kernel with
 incredible performance. Read more about the powers of BPF in [this
 blog](/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables#bpf)
 or in the [BPF reference guide](http://docs.cilium.io/en/doc-1.0/bpf/).
 - **Fully Distributed:** All datapath elements are fully distributed
 across the cluster and run at the most efficient layer in the operating
 system on each cluster node.

 - **Service Mesh datapath:** BPF allows us to build the ideal dataplane for
 the fast growing service mesh space. Cilium 1.0 already provides
 transparent injection of proxies such as Envoy. Future versions of Cilium
 will provide acceleration of sidecar proxies. We have published a set of
 early [sidecar proxy performance benchmarks](/blog/2018/04/24/cilium-security-for-age-of-microservices)

- **[CNI](http://docs.cilium.io/en/doc-1.0/kubernetes/) and [CMM](http://docs.cilium.io/en/doc-1.0/docker/) plugins:**
 The CNI and CMM plugins enable integration with
 [Kubernetes](http://docs.cilium.io/en/doc-1.0/kubernetes/),
 [Mesos](http://docs.cilium.io/en/doc-1.0/mesos/), and
 [Docker](http://docs.cilium.io/en/doc-1.0/docker/) to provide networking,
 load balancing and security for containers.

- **Network Security on both the Packet and API level:**
 Cilium combines [packet
 based](http://docs.cilium.io/en/doc-1.0/policy/language/#) network security and
 segmentation with transparent [API aware
 authorization](http://docs.cilium.io/en/doc-1.0/policy/language/#layer-7-examples)
 to provide security for both traditional deployments and evolving
 microservices architectures.
 - **[Identity Based](http://docs.cilium.io/en/doc-1.0/concepts/#arch-id-security):**
 Instead of relying on source IP addresses to identify workloads, Cilium
 encodes the workload identity in every packet to provide highly scalable
 security. This portable design allows the identity to be encoded in any IP
 based protocol and is aligned with upcoming concepts such as
 [SPIFFEE](https://github.com/spiffe/spiffe) or Kubernetes' [Container
 Identity Working
 Group](https://github.com/kubernetes/community/tree/master/wg-container-identity).

 - **[IP/CIDR Based](http://docs.cilium.io/en/doc-1.0/policy/language/#ip-cidr-based):**
 If identity based enforcement is not applicable, IP/CIDR based security can
 be used to control access. Whenever possible, Cilium offers abstractions to
 avoid hardcoding IP addresses in security policies. An example of this is
 the ability to define policy based on [Kubernetes service
 names](http://docs.cilium.io/en/doc-1.0/policy/language/#services-based).
 - **[API Aware Security](http://docs.cilium.io/en/doc-1.0/policy/language/#layer-7-examples):**
 The increasing usage of protocols such as HTTP/REST, gRPC and Kafka renders
 IP and port based security insufficient. The built-in awareness of a
 growing list of API and data store relevant protocols allows enforcing
 least privilege security at the right granularity.

- **Distributed and Scalable Load Balancing:**
 High performance Layer 3-4 load balancer using BPF for service to service
 connectivity with support for flow hashing and weighted round-robin. The BPF
 hashtable based implementation provides O(1) performance which means that the
 performance will not drop as you increase the number of services. The load
 balancer can be configured in two ways:
 - **Kubernetes Service Implementation:** All Kubernetes ClusterIP services
 will be automatically implemented in BPF to provide a highly scalable
 alternative to kube-proxy for in-cluster load-balancing.
 - **[API driven](http://docs.cilium.io/en/doc-1.0/api/):** For more advanced
 use cases, an extensive API is available to directly configure the
 load-balancing component.

- **[Simplified Networking Model](http://docs.cilium.io/en/doc-1.0/intro/#simple-networking):**
 Decoupling security from addressing simplifies the networking model
 dramatically: A single layer 3 network space provides the connectivity
 for all endpoints which are then segmented and secured using the
 policy layer on top. This simplicity tremendously helps scaling and
 troubleshooting. Networking can be configured in two modes:
 - **[Overlay/VXLAN](http://docs.cilium.io/en/doc-1.0/concepts/#overlay-network-mode):**
 The simplest integration that allows carrying workload identities for any
 IP based protocol. Hardware assisted VXLAN offloads ensure close to native
 performance.
 - **[Direct Routing](http://docs.cilium.io/en/doc-1.0/concepts/#direct-native-routing-mode):**
 Direct routing allows delegating routing to an existing networking
 component such as the native Linux routing layer, IPVLAN or a cloud
 provider router.

- **Visibility/Telemetry:** Similar to policy, visibility is provided on both
 the network packet and API call level. All visibility information includes
 rich workload level metadata such as container/pod labels and service names,
 instead of just IP addresses and port numbers.
 - **[Microscope](https://github.com/cilium/microscope):** Microscope provides visibility of security and forwarding
 events at cluster level with the ability to filter events based on labels,
 security identities and event types.
 - **BPF-based High Performance Monitor:** The high performance BPF perf ring
 buffer, designed for tracing millions of applications events per second,
 provides a highly efficient reporting channel which, combined with the
 programmability of BPF, allows extracting metrics and visibility at will
 with minimal overhead.
 - **[API driven](http://docs.cilium.io/en/doc-1.0/api/):** All visibility is provided via APIs and can be fed into
 existing systems.
- **Troubleshooting:**
 - **[Cluster Connectivity Health](http://docs.cilium.io/en/doc-1.0/troubleshooting/#cluster-connectivity-check):**
 Cilium periodically monitors the cluster connectivity including latency
 between all cluster nodes to identify faulty nodes and
 underlying network issues.
 - **[Prometheus Metrics](http://docs.cilium.io/en/doc-1.0/configuration/metrics/):** Metrics allow integrating Cilium into your
 existing monitoring dashboards for efficient monitoring of Cilium.
 - **Health Checks and Status overview:** Reliable health checks and component
 status overviews help quickly assess the health of all components and
 pinpoint problems.
 - **[Diagnosis and Reporting tools](http://docs.cilium.io/en/doc-1.0/troubleshooting/#cluster-diagnosis-tool)**
 Various tools to assist in the automatic detection of common problems and
 in the collection of information for bug reports.

##### Getting Started

Cilium is simple to install and use, in particular on Kubernetes:

 xterm

 curl -sLO https://releases.cilium.io/v1.0.0/examples/kubernetes/cilium.yaml
 vim cilium.yaml [provide etcd or consul address]
 kubectl create -f cilium.yaml
 kubectl create -f demo_app.yaml
 kubectl create -f http_policy.yaml
 kubectl exec -ti xwing-68c6cb4b4b-red5 -- curl -s -XPUT deathstar/v1/exhaust-port

 Access denied

The above example is a summary of the [hands-on minikube
tutorial](http://docs.cilium.io/en/doc-1.0/gettingstarted/minikube/) that walks
through applying a HTTP aware network policy step by step. More tutorials
can be found in the [getting started
section](http://docs.cilium.io/en/doc-1.0/gettingstarted/).

For further information on installing Cilium, see the [Kubernetes Quick
Installation Guide](http://docs.cilium.io/en/doc-1.0/kubernetes/quickinstall/)
or refer to the full list of [installation
guides](http://docs.cilium.io/en/doc-1.0/install/guides/#)

##### The Roadmap Ahead

Cilium 1.0 is an exciting milestone for all of us but we are already deep into
the planning of Cilium 1.1. So what is on the roadmap for 1.1 and beyond?

- **Multi Cluster Service Routing:** The simplicity of Cilium’s networking model
 and the decoupling of addressing and policy allows for easy expansion across
 clusters. With this expansion, Cilium will start supporting Kubernetes
 service routing across multiple clusters without requiring complex proxy or
 Ingress solutions while providing the full set of identity based and API
 aware security.

- **Integration with OpenTracing, Jaeger and Zipkin:** The minimal overhead of
 BPF makes it the ideal technology to provide tracing and telemetry
 functionality without imposing additional system load.

- **Policy support for additional API protocols:** We already have several
 additional application protocols in mind that we will support in future
 releases to further improve security.

- **CRI support:** Repeatedly requested by various members of the community, we are
 looking forward to supporting CRI to properly abstract the container runtime.

- **Non container workloads:** The BPF datapath is not limited to container
 abstractions, it just happened to be the first use case we focused on. Future
 versions will provide APIs and documentation on how to integrate with native
 Linux tasks, VMs and how to bridge the identity based security space to
 existing worlds using IP addresses that cannot be migrated.

You can find the details of the 1.1 release planning in this [github
issue](https://github.com/cilium/cilium/issues/3585). Feel free to comment or
open GitHub issues if you would like to see particular functionality in future
Cilium releases.

[http]: http://docs.cilium.io/en/doc-1.0/gettingstarted/minikube/
[grpc]: http://docs.cilium.io/en/doc-1.0/gettingstarted/grpc/
[kafka]: http://docs.cilium.io/en/doc-1.0/gettingstarted/kafka/
[slack]: https://slack.cilium.io
[github]: https://github.com/cilium/cilium/issues/new
[google]: https://www.socallinuxexpo.org/sites/default/files/presentations/bcc-scale.pdf
[facebook]: https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf
[netflix]: http://www.brendangregg.com/Slides/BSidesSF2017_BPF_security_monitoring.pdf
[red hat]: https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/

#### Cilium - Rethinking Linux Networking and Security for the Age of Microservices

URL: https://cilium.io/blog/2018/04/24/cilium-security-for-age-of-microservices
Date: 2018-04-24
Categories: Technology

Linux’s traditional network forwarding and security capabilities, centered around iptables and focused on IP addresses and ports, were designed 20 years ago in an era of manually deployed physical hosts, monolithic application design, and hand-edited firewall rule-sets. Back then, concepts central to how we architect and deploy applications today, like containers and container orchestration (Docker, Kubernetes), API-driven microservices design, and continuous integration/continuous delivery were many years from widespread use.

In the highly dynamic and complex world of microservices, thinking about networking and security simply through the traditional lens of IP addresses and ports leads to an implementation that can be highly inefficient, has only course-grained visibility and filtering,
and is often highly complex to troubleshoot. Thanks to BPF, a powerful new kernel extensibility mechanism inside of Linux, we have the opportunity to rethink the Linux networking and security stack for the age of microservices and address these concerns.

Cilium is an open source CNI-compatible networking and security layer for Kubernetes, Mesos, and Docker that does just that. From
inception, Cilium was designed for large-scale, highly-dynamic containerized environments. It natively understands container
identity and parses API protocols like HTTP, gRPC, and Kafka, providing visibility and security that is both simpler and more
powerful than a traditional firewall. And the power of BPF enables highly efficient in-kernel data
forwarding, delivering huge performance wins for common microservices use cases like service-based load-balancing in Kubernetes or
the insertion of local proxies for a "service mesh" like Istio.

To celebrate the [Cilium project hitting 1.0](https://www.cilium.io/blog/2018/04/24/cilium-10),
we wanted to take a moment to share the broader story behind how BPF and Cilium are driving the biggest change in the past
two decades of Linux networking and security, and invite you to join in on the fun.

##### Why Linux is the Right Place for Network Routing, Visibility, and Security

Despite almost everything about how we design and run our applications changing, in-kernel capabilities like
iptables are still by far the most common way to steer, filter, and log network data in modern microservices environments like Kubernetes, Mesos, and Docker.

Why? It comes down to the fact that in-kernel network and security has several key advantages:

- **Performance:** Workload data is already traversing the local kernel, so additional data processing can be efficient in terms of both throughput and latency.
- **Transparency:** Applications code and containers require no changes/reconfiguration, as they are already sending/receiving data using TCP/IP sockets.
- **Security:** A compromised or misconfigured application faces a very high barrier to bypass or disable kernel-level enforcement.

##### The Limits of a General Purpose IP/port-based Firewall

As a component of a general purpose operating system, iptables focused on providing a “swiss army knife” for Linux administrators
to manually craft IP and port-based network forwarding, filtering, and logging rules targeting largely static environments. However, as iptables has been pressed into service in large-scale, highly dynamic microservices environments, the mismatch between the original design goals and the demands of modern infrastructure become obvious. For example, a [recent talk at KubeCon](https://schd.ws/hosted_files/cloudnativeeu2017/ce/Scale%20Kubernetes%20to%20Support%2050000%20Services.pdf) highlighted severe
bottlenecks related to iptables service-based load-balancing in Kubernetes.

And the issues go beyond just performance and scale. End-user workflows like troubleshooting and audit become almost impossible in environments where the ‘identity’ associated with an IP changes each time a container is created/destroyed. Filtering on TCP
port provides only coarse-grained security isolation in the common scenario where a microservice exposes all of its functionality
via an API running on a single port (e.g., HTTP).

So we found it worth asking: what would an in-kernel Linux networking and security approach look like if it was being designed from
scratch today for the era of microservices?

##### Enter BPF: Flexibly and Safety Extending the Linux Kernel

BPF is a powerful new capability for extending the Linux kernel without compromising on safety or speed. BPF is a framework for running custom logic at various hook points in the kernel, including many points in the Linux networking stack. BPF logic is written as simple “BPF programs” that are run through a programmatic verifier to ensure that they under no circumstances could compromise, crash, or hang the kernel that evokes it. After verification, these programs are then JIT-compiled to native assembly code (e.g., x86), meaning they run at the same speed as code compiled into the kernel.

BPFs capabilities have grown incredibly the past few years, and Linux-centric tech companies that are operating at massive scales have already taken notice: [Netflix](http://www.brendangregg.com/Slides/BSidesSF2017_BPF_security_monitoring.pdf),
[Facebook](https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf), and [Google](https://www.socallinuxexpo.org/sites/default/files/presentations/bcc-scale.pdf) have all made significant bets on BPF as a mechanism for kernel extensibility. Readers curious for a deep-dive on BPF bytecode verification, JIT-compilation and more will enjoy our [BPF and XDP Reference Guide](http://cilium.readthedocs.io/en/stable/bpf/), but Cilium hides this complexity from end users, providing simple integrations with container orchestrators like Kubernetes, Mesos, and Docker.

##### Cilium: Empowering Kubernetes Networking and Security with BPF

Using BPF as an underlying engine, Cilium creates a networking stack precisely optimized
for running API-driven microservices on platforms like Kubernetes. We’ll focus on two key benefits of using Cilium:

- Moving beyond a simple focus on packets, IP addresses, and ports, and instead treating service identity and API protocols (e.g., HTTP, gRPC, and Kafka) as first-class citizens in the platform.
- Optimizing Linux network forwarding, visibility, and filtering for the scale, dynamism and deployment patterns (e.g., service mesh proxies) that are increasingly common in microservices environments.

###### Service-Centric Identity and API-Awareness

With traditional firewalling, workloads are identified by IP addresses and the "intent" of the communication is identified by
TCP/UDP port (e.g., TCP port 80 accesses Web, TCP port 22 accesses SSH).

But with microservices, an application is deconstructed into many distinct services, with each service implemented as a collection of
stateless container for scale-out performance, failure redundancy, and incremental upgrades. As a result, IP addresses are
relatively ephemeral, changing meaning each time a container is created or destroyed.

Furthermore, most communication happens using only a few ports (e.g., HTTP), and the actual “intent” of the communication between services can only be determined by understanding the API-layer “remote procedure calls” (RPCs) between API-driven services and datastores.

A typical microservices endpoint will expose 10s or even 100s of unique RPC calls:

- A RESTful HTTP-based service often exposes POST/PUT/GET/DELETE for many different resource types, each represented by a URL prefix.
- A Kafka broker will often have many different topics, and allow actions like produce and consume on each topic to different clients.
- A datastore like Cassandra, Elasticsearch, Mongodb, Redis, and even Mysql and Postgres provides both read and write access to many distinct tables/indices.

As a result, port-based visibility and security is blind to the individual RPC calls and will expose either all RPC between
two different services or none at all.

With Cilium, identity is extracted from the container orchestrator and embedded in each network request (e.g., id=app1).
Unlike an IP address, such an identity is consistent across multiple replicas implementing a service, and consistent across
time. Furthermore, Cilium provides API-aware visibility and security that understands
[HTTP Methods/URL](http://docs.cilium.io/en/stable/gettingstarted/minikube/),
[gRPC service methods](http://docs.cilium.io/en/stable/gettingstarted/grpc/),
[Kafka topics](http://docs.cilium.io/en/stable/gettingstarted/kafka/), etc. and enables fine-grained
visibility and security at the granularity of the RPCs between microservices.

This leads to a visibility and security that is both radically simpler (based on consistent identities, not ephmeral IP addresses)
and more powerful (visibility into individual RPC calls). To see this below is an example of using [Cilium Microscope](https://github.com/cilium/microscope) to view all HTTP requests to a microservice “app1” and log whether the request was allowed or denied by
security policy:

```
cilium-microscope --to-selector id=app1 --type l7
([k8s:id=app2]) => ([k8s:id=app1]) http GET http://app1-service/public Forwarded
([k8s:id=app3]) => ([k8s:id=app1]) http GET http://app1-service/private Denied
...
```

With Cilium, there is no need to worry about how many container replicas are implementing service ‘app1’, what hosts these containers are running on, or what IP addresses the containers were assigned at any point in time.

###### BPF-optimized Dataplane Performance

Cilium leverages BPF to customize the in-kernel data flow, algorithms, and data structures used for forwarding data, optimizing for the exact use case and container identity. This can yield some amazing benefits at many points in data forwarding, but we will focus on just two below.

###### Kubernetes Service Load Balancing

The aforementioned [KubeCon Talk](https://schd.ws/hosted_files/cloudnativeeu2017/ce/Scale%20Kubernetes%20to%20Support%2050000%20Services.pdf) performed specific measurements on iptables as a bottleneck for Kubernetes service forwarding
and noted that throughput degraded by ~30% with 5,000 services deployed, and by 80% with 10,000 services (a 6X performance difference).Likewise, rule updates at 5,000 services took 11 minutes, ages in a world of continuous delivery.

Thanks to the flexibility of BPF, Cilium performs this same operation with O(1) average runtime behavior using a simple BPF map based hash table, meaning the lookup latency at 10,000 or even 20,000 services is constant. Likewise, updates to these BPF maps from userspace
are highly-efficient, meaning that even with 20,000+ services, the time to update a forwarding rule is microseconds, not hours.

For these reasons, Facebook has recently presented their use of [BPF and XDP for load-balancing](https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf) in a public talk to replace IPVS after measuring an almost 10x performance increase.

###### Local Proxy / Service Mesh Redirection (Istio/Envoy):

An emerging deployment pattern for microservices is to use a local proxy (running either as a per-pod “sidecar proxy” or
one proxy per Linux host) to implement a “service mesh” between a collection of microservices (for example,
[Istio](https://www.istio.io) uses [Envoy](https://www.envoyproxy.io/) in a sidecar model).

This approach requires all communication to/from a workload container to be redirected to this local proxy. This redirection
must be transparent (i.e., no application changes) and secure (i.e., cannot be bypassed by the application). Istio currently
performs this redirection at the packet level using iptables. But doing this redirect at the packet level means each
byte of data passes through the entire TCP/IP stack, with Linux performing TCP congestion control and ACKing,
breaking data into IP packets, and in some cases even passing it through a virtual ethernet device.
The cost in terms of buffer memory, CPU cycles, context switches, and packet latency is quite high and all to effectively
copy data from one Linux socket to another on the same host.

Thanks to recent BPF improvements called “Sockmap” that we have upstreamed into the Linux kernel, Cilium is able to
perform this secure redirection at the socket layer itself. Initial measurements suggest that the savings are substantial,
resulting in approximately 2X the throughput compared to iptables redirect when persistent connections are used.

Long story short, if you see the use of local proxies either directly or via a service mesh as part of your future architecture,
using BPF + Cilium Sockmap should be a no-brainer way to reduce CPU/memory usage and drive down latency.

##### Join The Cilium Community...

With BPF + Cilium, the Linux kernel can now be optimized for the age of Kubernetes, Docker and Mesos and the API-driven microservices that are deployed at scale on such platforms. We’re excited about what we’ve built so far with Cilium 1.0, but we’re just getting started! We are continuing to work with the upstream Linux community as well as open source projects like Envoy and Istio to push the boundaries of dataplane performance and add deeper and richer capabilities around service identity and API-aware visibility and security.

We look forward to hearing from you on how Cilium and BPF can continue to reinvent Linux networking for a world filled with
API-driven microservices, so [try it out](https://www.cilium.io/try), ask a question on [Cilium Slack](https://slack.cilium.io),
and if you like it, consider contributing on [Cilium GitHub](https://github.com/cilium/cilium).

#### Why is the kernel community replacing iptables with BPF?

URL: https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables
Date: 2018-04-17
Categories: Technology

> Author Note: this is a post by long-time Linux kernel networking developer and
> creator of the Cilium project, Thomas Graf

---

The Linux kernel community recently announced [bpfilter], which will replace
the long-standing in-kernel implementation of iptables with high-performance
network filtering powered by Linux BPF, all while guaranteeing a non-disruptive
transition for Linux users.

From humble roots as the packet filtering capability underlying popular tools
like tcpdump and Wireshark, BPF has grown into a rich framework to extend the
capabilities of Linux in a highly flexible manner without sacrificing key
properties like performance and safety. This powerful combination has led
forward-leaning users of Linux kernel technology like [Google], [Facebook], and
[Netflix] to choose BPF for use cases ranging from network security and
load-balancing to performance monitoring and troubleshooting. Brendan Gregg of
Netflix first called BPF [Superpowers] for Linux. This post will cover how
these “superpowers” render long-standing kernel sub-systems like iptables
redundant while simultaneous enabling new in-kernel use cases that few would
have previously imagined were possible.

Having spent the past 15 years in the Linux kernel community authoring code to
many subsystems including the TCP/IP stack, iptables, and many more, allowed me
to observe BPF developments up close. I soon realized that BPF was not simply
yet another feature but instead represented a foundational technology shift
that in time would change nearly every aspect of networking and security within
Linux. I started contributing and became one of its biggest supporters along
side with Alexei Starovoitov and Daniel Borkmann which are now maintaining BPF
upstream. In this lens, the shift from iptables with bpfilter is just the next
logical step in a BPF’s journey to revitalize the Linux networking stack for
the modern area. To understand why this shift is so exciting, allow me to take
you on a bit of a tour through the history of iptables in the kernel.

##### iptables and the roots of sequential filtering

iptables has been the primary tool to implement firewalls and packet filters on
Linux for many years. iptables and its predecessor ipchains have been part of
my personal Linux journey from early on. Initially as a user, later as a kernel
developer. Over the years, iptables has been a blessing and a curse: a blessing
for its flexibility and quick fixes. A curse during times debugging a 5K rules
iptables setup in an environment where multiple system components are fighting
over who gets to install what iptables rules.

[Jerome Petazzoni](https://twitter.com/jpetazzo) once overheard a quote that
could not be more to the point:

<blockquote class="twitter-tweet" data-lang="en" align="center"> OH: &quot;In any team you need a tank, a healer, a damage dealer, someone with crowd control abilities, and another who knows iptables&quot; &mdash; Jérôme Petazzoni (@jpetazzo) June 27, 2015 </blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

When iptables started its life 20 years ago to replace its predecessor
ipchains, firewall functionality was written with a simple scope:

- Protect local applications from receiving unwanted network traffic (INPUT
 chain)
- Protect local applications sending undesired network traffic (OUTPUT chain)
- Filter network traffic forwarded/routed by a Linux system (FORWARD chain).

Back then, network speeds were slow. Remember the sound a modem made when
dialing in? That was the era when iptables was initially developed and
designed. The standard practice of implementing access control lists (ACLs) as
implemented by iptables was to use sequential list of rules, i.e. every packet
received or transmitted is matched against a list of rules, one by one.

However, linear processing has an obvious massive disadvantage, the cost of
filtering a packet can increase linearly with the number of rules added.

##### The intermediate workaround: ipset

Some time passed, network speeds increased and iptables setups had grown from a
dozen rules to sets of thousands of rules. Traversing sequential iptables lists
had become unbearable from a performance and latency perspective.

The community quickly identified the most common bottleneck: long lists of
rules either rejecting or allowing individual IP address and port combinations.
This lead to the introduction of [ipset]. ipset allows to compress list of
rules matching on IP addresses and/or port combinations into a hash table to
reduce the number of iptables rules overall. This has since been serving as a
workaround without addressing the core problem itself.

Unfortunately, ipset is not an answer to all problems. A prominent example is
`kube-proxy`, a component of [Kubernetes](https://kubernetes.io/) that uses
iptables and `-j DNAT` rules to provide load-balancing for services. It
installs multiple iptables rules for each backend a service is serving to. For
each service added to Kubernetes, the list of iptables rules to be traversed
grows exponentially.

A recent [KubeCon
talk](https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services)
examined kube-proxy performance in all details. The talk presents measurements
showing unpredictable latency and reduced performance as the number of services
grows. It also reveals another major weakness of iptables: lack of incremental
updates. The entire list of rules has to be replaced each time a new rule is
added. This leads to a total duration of 5 hours to install the 160K iptables
rules representing 20K Kubernetes services.

Using IP/port based mechanisms has many other obvious disadvantages in general,
in particular in the age of application containers. Containers are deployed and
torn down frequently. This can lead to a short lifetime of individual IP
addresses. An IP address may be used by a container for just seconds and then
gets reused by another container a couple of seconds later. This puts stress on
systems which rely on using IP addresses for security filtering purposes as all
nodes in a cluster must be constantly aware of the latest IP to container
mapping. While this is barely difficult within a cluster, it becomes incredibly
challenging across clusters. Diving into the details is outside of the scope of
this blog post so we will save it for a future post.

##### The rise of BPF

BPF has been evolving at an insane pace in recent years, unlocking what was
previously outside the scope of the kernel. This is made possible by the
incredible powerful and efficient programmability that BPF provides. Tasks that
previously required custom kernel development and kernel recompilations can now
be achieved with efficient BPF programs within the safe boundaries of the BPF
sandbox.

The following is a list of projects that show how BPF is being leveraged in
various projects and companies:

- [Cilium](http://github.com/cilium/cilium) is unleashing the powers of BPF to
the world of containers and provides powerful and efficient networking,
security and load-balancing at L3-L7. Read the [introduction to
Cilium](http://docs.cilium.io/en/stable/intro/) to get started on learning
about Cilium.

 <blockquote class="twitter-tweet" data-lang="en"> What @ciliumproject is doing with eBPF and XPD is the cleanest networking plugin I&#39;ve seen, mad props https://t.co/oOShMvT2iY pic.twitter.com/mPV864Aia1 &mdash; jessie frazelle (@jessfraz) August 16, 2017 </blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

- Facebook has presented exciting work on [BPF/XDP based load-balancing to
replace
IPVS](https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf)
that also includes DDoS mitigation logic. While IPVS is a tempting next step
compared to iptables, Facebook has already migrating away from IPVS to BPF
after seeing roughly a 10x improvement in performance.

 <blockquote class="twitter-tweet" data-lang="en"> Mind blown by eBPF performance! A simple ingress firewall I wrote using XDP processes 11 million packets/s. Time for more optimizations! &mdash; Diptanu Choudhury (@diptanu) August 21, 2017 </blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

- Netflix, in particular Brendan Gregg, have been utilizing the
 powers of BPF for performance profiling and tracing. The [bcc] project
 provides users access to the powers of BPF to, for example, generate amazing
 flamegraphs:

 For more details including many examples on how to leverage BPF for
 application tracing, checkout out Brendan Gregg’s [blog
 posts](http://www.brendangregg.com/blog/).

 <blockquote class="twitter-tweet" data-lang="en"> as I&#39;ve said to a number of people job hunting in systems engineering: BPF experience is hot and getting hotter https://t.co/bScbqE6m17 &mdash; Brendan Gregg (@brendangregg) May 20, 2017 </blockquote>
 <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

- Google has been working on bpfd which enables [Powerful Linux Tracing for
 Remote targets using
 eBPF](https://www.socallinuxexpo.org/sites/default/files/presentations/bcc-scale.pdf).
 Based on upstream engagement around BPF, they also seem to be looking at
 migrating various internal projects over to BPF.

- Cloudflare is using BPF to [mitigate DDoS attacks](https://www.netdevconf.org/2.1/slides/apr6/bertin_Netdev-XDP.pdf)
 and has put out several blog posts and presented multiple public talks on the
 topic.

- [Suricata](https://suricata-ids.org/) is an IDS that started using BPF and
 XDP to replace nfqueue which is an iptables based infrastructure to wiretap
 packets. More details can be found in this [Kernel Recipes
 talk](https://www.slideshare.net/ennael/kernel-recipes-2017-ebpf-and-xdp-eric-leblond).

- [Open vSwitch](https://www.openvswitch.org/) has been working on using an
 [eBPF powered datapath](http://www.openvswitch.org//support/ovscon2016/7/1120-tu.pdf).

There are too numerous examples to list them all in this blog post. We are
maintaining a more complete list of [projects using
BPF](http://cilium.readthedocs.io/en/stable/bpf/#projects-using-bpf) in the
[BPF reference guide](http://docs.cilium.io/en/stable/bpf/).

##### One BPF to rule them all

The most recent development in the evolution of BPF is an exciting proposal to
completely replace the kernel part of iptables with BPF in a way that is
completely transparent to the user, i.e. existing iptables client binaries and
libraries will continue to work.

You can find the upstream discussion on the [Kernel mailing
lists](https://www.mail-archive.com/netdev@vger.kernel.org/msg217095.html). The
proposal is authored by Daniel Borkmann (Covalent), the networking maintainer
David Miller (Red Hat), and Alexei Starovoitov (Facebook). The proposal was
covered by an [LWN article](https://lwn.net/Articles/747551/) that gives a
great summary of the initial discussion.

The following graph as presented by Quentin Monnet at FRnOG 30 shows some early
measurements of bpfilter in comparison with iptables and nftables. It shows
both the software only BPF implementation as well as a hardware offloaded test:

These early performance numbers are incredible promising and an indication of
the power of BPF. It must be noted that bpfilter and BPF itself will not solve
the performance problems caused by iptables's use of sequential lists. Doing so
will require native use of BPF such as done by the [Cilium](https://cilium.io/)
project.

##### How has the kernel community reacted?

Some of the Linux kernel mailing lists are famous for their flame wars. Have
flamewars burst out in this case? No, in fact, there have been immediate
proposals from core iptables maintainers all aiming into the direction of BPF.

- Florian Westphal has [proposed a
 framework](https://www.spinics.net/lists/netdev/msg486873.html) that will
 operate on top of bpfilter and will convert nftables to BPF. This allows to
 keep the domain specific nftables language but benefits from all the
 advantages of the BPF runtime with its JIT compiler, hardware offload, and
 tooling.

- Pablo Neira Ayuso seems to have been working on a [similar
 proposal](https://www.mail-archive.com/netdev@vger.kernel.org/msg217425.html)
 and has published a series which also translates nftables to BPF. The main
 difference in approach seemed to be that Pablo intended to perform the
 translation in the kernel. The community has since agreed that any injection of
 BPF programs must occur through user space and be passed through the BPF
 verifier to guarantee the safety behavior of BPF.

##### Summary

I consider BPF the most exciting Linux development since many years. We are
barely scratching the surface of its potential and it is still evolving. The
move to replace the kernel part of iptables with BPF is a logical first step.
The real transition will be to BPF native tooling as well as move away from
traditional IP address/port centric constructs.

Interested in learning more about BPF or Cilium? Talk to us on
[Slack](https://slack.cilium.io)

[ipset]: http://ipset.netfilter.org/
[bcc]: https://github.com/iovisor/bcc
[bpfilter]: https://lwn.net/Articles/747551/
[google]: https://www.socallinuxexpo.org/sites/default/files/presentations/bcc-scale.pdf
[facebook]: https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf
[netflix]: http://www.brendangregg.com/Slides/BSidesSF2017_BPF_security_monitoring.pdf
[superpowers]: http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.html

#### Cilium 1.0.0-rc9 - Feature Freeze for 1.0!

URL: https://cilium.io/blog/2018/04/02/cilium-1.0.0-rc9-code-freeze
Date: 2018-04-02
Categories: Release

We are excited to announce [Cilium
1.0.0-rc9](https://github.com/cilium/cilium/releases/tag/v1.0.0-rc9) with many,
many bugfixes and the delivery of the final feature we were waiting on prior for
1.0: Egress policy enforcement support. It is therefore only logical that we
announce full feature freeze with rc9. This means that we will only merge
critical bugfixes and release 1.0 as soon as we have resolved all release
blockers. More on this below. We are thrilled to have come this far and
appreciate all of the efforts by the wide range of contributors that have
helped to get us here.

##### Upgrade Instructions

No special upgrade instructions are required for this release. Please follow
out [simple upgrade guide](http://docs.cilium.io/en/stable/install/upgrade/)
for the generic instructions on how to upgrade.

##### Highlights

As usual, the full release notes are attached at the end of the blog but can be
found on the [1.0.0-rc9 release
page](https://github.com/cilium/cilium/releases/tag/v1.0.0-rc9). The vast
majority of the work in this release has been around bugfixes and testing. Here
is a list of some highlights:

##### Egress Policy Enforcement capability

Cilium uses an identity based policy enforcement mechanism as its standard
enforcement mechanism and only falls back to IP/CIDR based enforcement when
absolutely required. The identity based model implies that we encode the
identity of the sending endpoint with all packets and then enforce on the
receiving side whether that identity is allowed to communicate with the
respective peer. Cilium only falls back to an IP/CIDR based enforcement mode if
we are not in control of the sender.

With this release, we are now completing the egress policy enforcement by
adding labels and entities based enforcement on top of the existing IP/CIDR
egress enforcement that existed before.

###### A few simple egress examples

The following example is tailored for Kubernetes and shows how to enable
default deny at egress for all `role=frontend` pods and then explicitly
whitelist the connection to `role=backend` on port TCP/80:

```
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "Allow egress TCP/80 from frontend to backend"
metadata:
  name: "egress-rule"
spec:
  endpointSelector:
    matchLabels:
      role: frontend
  egress:
  - toEndpoints:
    - matchLabels:
      role: backend
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
```

This obviously also applies to L7 aware policies. Here is another example which
shows how to whitelist `POST /metric` on port TCP/8080 from pods with the
label `app=myService` to their respective local host.

```
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "Allow HTTP POST /metric from myService to local host"
metadata:
  name: "rule1"
spec:
  endpointSelector:
    matchLabels:
      app: myService
  egress:
  - toEntities:
    - host
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/metric$"
```

##### Configurable 403 HTTP access denied messages

The ability to specify the text as returned with 403 HTTP responses is
obviously a critical enterprise grade feature as explained in this separate
blog post:

- [Cilium Enterprise Edition 4.0: Repelling Attacks with Emojis, Rickrolling, and More!](/blog/2018/4/1/cilium-enterprise-edition)

No further explanation required.

##### Scale Improvements

We have done a series of scale and stress tests which lead to tweaking of
default limits and improvements that affect scalability:

- Several upper limits for BPF maps covering connection state have been
 increased. We will likely make this adjustable and improve defaults to be
 based on available system memory to take a good guess at expected network
 load.

- A new expedited garbage collector mode has been introduced which
 identifies connections that have never been established (no complete SYN-ACK
 handshake observed). Such incomplete connections are removed from state
 tables much more aggressively. This finds a good balance to keep long lived
 TCP connections in state tables for days without seeing any traffic while
 aggressively removing connections created by connection attempt floods or
 services such as Cassandra which perform retries _very_ aggressively.

- We have started enabling TCP keepalive for all proxied connections to gain
 a better understanding of the health of long lived connections with minimal
 traffic such as TCP connections used for health checking.

##### Known issues before 1.0

We have a couple of issues that are we tracking and fixing before releasing
1.0. If you are running into any issues, check the list of [1.0 blocker bugs]
(https://github.com/cilium/cilium/issues?q=is%3Aopen+is%3Aissue+label%3Apriority%2F1.0-blocker)
first.

##### Release Notes

###### Major Changes

- envoy: Make 403 message configurable. ([3430](https://github.com/cilium/cilium/pull/3430), @jrajahalme)
- Add support label-dependent L4 egress policy ([3372](https://github.com/cilium/cilium/pull/3372), @ianvernon)

###### Bugfixes Changes

- Fix entity dependent L4 enforcement ([3451](https://github.com/cilium/cilium/pull/3451), @tgraf)
- cli: Fix cilium bpf policy get ([3446](https://github.com/cilium/cilium/pull/3446), @tgraf)
- Fix CIDR ingress lookup ([3406](https://github.com/cilium/cilium/pull/3406), @joestringer)
- xds: Handle NACKs of initial versions of resources ([3405](https://github.com/cilium/cilium/pull/3405), @rlenglet)
- datapath: fix egress to world entity traffic, add e2e test ([3386](https://github.com/cilium/cilium/pull/3386), @ianvernon)
- bug: Fix panic in health server logs if /healthz didn't respond before checking status ([3378](https://github.com/cilium/cilium/pull/3378), @nebril)
- pkg/policy: remove fromEntities and toEntities from rule type ([3375](https://github.com/cilium/cilium/pull/3375), @ianvernon)
- Fix IPv4 CIDR lookup on older kernels ([3366](https://github.com/cilium/cilium/pull/3366), @joestringer)
- Fix egress CIDR policy enforcement ([3348](https://github.com/cilium/cilium/pull/3348), @tgraf)
- envoy: Fix concurrency issues in Cilium xDS server ([3341](https://github.com/cilium/cilium/pull/3341), @rlenglet)
- Fix bug where policies associated with stale identities remain in BPF policy maps, which could lead to "Argument list too long" errors while regenerating endpoints ([3321](https://github.com/cilium/cilium/pull/3321), @joestringer)
- Update CI and docs: kafka zookeeper connection timeout to 20 sec ([3308](https://github.com/cilium/cilium/pull/3308), @manalibhutiyani)
- Reject CiliumNetworkPolicy rules which do not have EndpointSelector field ([3275](https://github.com/cilium/cilium/pull/3275), @ianvernon)
- Envoy: delete proxymap on connection close ([3271](https://github.com/cilium/cilium/pull/3271), @jrajahalme)
- Fix nested cmdref links in documentation ([3265](https://github.com/cilium/cilium/pull/3265), @joestringer)
- completion: Fix race condition that can cause panic ([3256](https://github.com/cilium/cilium/pull/3256), @rlenglet)
- Additional NetworkPolicy tests and egress wildcard fix ([3246](https://github.com/cilium/cilium/pull/3246), @tgraf)
- Add timeout for getting etcd session ([3228](https://github.com/cilium/cilium/pull/3228), @nebril)
- conntrack: Cleanup egress entries and distinguish redirects per endpoint ([3221](https://github.com/cilium/cilium/pull/3221), @rlenglet)
- Silence warnings during endpoint restore ([3216](https://github.com/cilium/cilium/pull/3216), @tgraf)
- Fix MTU connectivity issue with external services ([3205](https://github.com/cilium/cilium/pull/3205), @joestringer)
- endpoint: Don't fail with fatal on l4 policy application ([3199](https://github.com/cilium/cilium/pull/3199), @tgraf)
- Add new Kafka Role to the docs ([3186](https://github.com/cilium/cilium/pull/3186), @manalibhutiyani)
- Fix log records for Kafka responses ([3127](https://github.com/cilium/cilium/pull/3127), @tgraf)

###### Other Changes

- Refactor /endpoint/{'{'}id}/config for API 1.0 stability ([3448](https://github.com/cilium/cilium/pull/3448), @tgraf)
- envoy: Add host identity ([nphds) gRPC client (3407](https://github.com/cilium/cilium/pull/nphds) gRPC client (3407), @jrajahalme)
- Increase capacity of BPF maps ([3391](https://github.com/cilium/cilium/pull/3391), @tgraf)
- daemon: Merge Envoy logs with cilium logs by default. ([3364](https://github.com/cilium/cilium/pull/3364), @jrajahalme)
- docs: Fix the Kafka policy to use the new role in the GSG ([3350](https://github.com/cilium/cilium/pull/3350), @manalibhutiyani)
- CI / GSG: make Kafka service headless ([3320](https://github.com/cilium/cilium/pull/3320), @manalibhutiyani)
- Use alpine as base image for Docs container ([3301](https://github.com/cilium/cilium/pull/3301), @iamShantanu101)
- Update kafka zookeeper session timeout to 20 sec in CI tests and docs ([3298](https://github.com/cilium/cilium/pull/3298), @manalibhutiyani)
- Support access log from sidecar and per-endpoint redirect stats ([3278](https://github.com/cilium/cilium/pull/3278), @rlenglet)
- Improve sanity checking in endpoint PATCH API ([3274](https://github.com/cilium/cilium/pull/3274), @joestringer)
- Update Kafka GSG policy and docs to use the new "roles" ([3269](https://github.com/cilium/cilium/pull/3269), @manalibhutiyani)
- maps: allow for migration when map properties change ([3267](https://github.com/cilium/cilium/pull/3267), @borkmann)
- bpf: Retire CT entries quickly for unreplied connections ([3238](https://github.com/cilium/cilium/pull/3238), @joestringer)
- CMD: Add json output on endpoint config ([3234](https://github.com/cilium/cilium/pull/3234), @eloycoto)
- Plumb the contents of the ip-identity cache to a BPF map for lookup in the datapath. ([3037](https://github.com/cilium/cilium/pull/3037), @ianvernon)

##### Release binaries

- [cilium-agent-x86_64](http://releases.cilium.io/v1.0.0-rc9/cilium-agent-x86_64) ([45085a62027fa2b30858](http://releases.cilium.io/v1.0.0-rc9/cilium-agent-x86_64.sha256sum))
- [cilium-bugtool-x86_64](http://releases.cilium.io/v1.0.0-rc9/cilium-bugtool-x86_64) ([f1d14a36f7c804d60c92](http://releases.cilium.io/v1.0.0-rc9/cilium-bugtool-x86_64.sha256sum))
- [cilium-health-x86_64](http://releases.cilium.io/v1.0.0-rc9/cilium-health-x86_64) ([f6f1b21fe7e45417109c](http://releases.cilium.io/v1.0.0-rc9/cilium-health-x86_64.sha256sum))
- [cilium-node-monitor-x86_64](http://releases.cilium.io/v1.0.0-rc9/cilium-node-monitor-x86_64) ([31cf402a44e32104ed4a](http://releases.cilium.io/v1.0.0-rc9/cilium-node-monitor-x86_64.sha256sum))
- [cilium-x86_64](http://releases.cilium.io/v1.0.0-rc9/cilium-x86_64) ([d9eb0c42c8b0a4a2ea17](http://releases.cilium.io/v1.0.0-rc9/cilium-x86_64.sha256sum))
- [v1.0.0-rc9.tar.gz](http://releases.cilium.io/v1.0.0-rc9/v1.0.0-rc9.tar.gz) ([30cebb959aa508d04814](http://releases.cilium.io/v1.0.0-rc9/v1.0.0-rc9.tar.gz.sha256sum))
- [v1.0.0-rc9.zip](http://releases.cilium.io/v1.0.0-rc9/v1.0.0-rc9.zip) ([8b5752cf20d6cb5cbc92](http://releases.cilium.io/v1.0.0-rc9/v1.0.0-rc9.zip.sha256sum))

As usual, let us know on [Slack](https://slack.cilium.io/) if you have any questions.

#### Cilium 1.0.0-rc4 released

URL: https://cilium.io/blog/2018/02/16/cilium-1.0.0-rc4
Date: 2018-02-16
Categories: Release

We are excited to have released Cilium 1.0.0-rc4. The release contains a lot of
bugfixes as usual plus a lot of CI work to ensure quality long term but there
are also some enhancements highlights and tooling worth mentioning.

As usual, the full release notes are attached at the end of the blog but can be
found on the [1.0.0-rc4 release
page](https://github.com/cilium/cilium/releases/tag/v1.0.0-rc4). Here is a list
of some highlights:

##### Envoy is the default HTTP/gRPC proxy \o/

We have finally ripped out the old custom HTTP proxy and made
[Envoy](https://github.com/envoyproxy/envoy) the default proxy for all L7
enforcement of HTTP and gRPC traffic. In the months up to this we have extended
Envoy in various ways to

- Introduction of listener filters to allow running filters per listener to
 retrieve per connection metadata. We use this to read metadata from BPF maps
 and make the L3/L4 forwarding context of Cilium available to Envoy.
 [See PR](https://github.com/envoyproxy/envoy/pull/2346)

- Addition of the original destination cluster type to allow configuring Envoy
 in a completely transparent manner so whenever a connection is redirected to
 Envoy, Envoy will always forward to whatever was the original destination of
 the redirected connection.
 [See PR](https://github.com/envoyproxy/envoy/pull/1246)

- Allow tying the HTTP version of an upstream connection to whatever HTTP
 version the downstream connection is using. This allows preserving full
 transparency.
 [See PR](https://github.com/envoyproxy/envoy/pull/2328)

- Allow HTTP filters to have read access to the downstream connections.
 [See PR](https://github.com/envoyproxy/envoy/pull/1300)

##### Simple health overview for connectivity and other errors

Cilium, like the majority of distributed systems software, is driven by events
and notifications. Cilium react to events such as addition of a new policy,
appearance of a new security identity in the cluster, removal of a container on
the local node, and so on. Unfortunately things can and will go wrong. The
kvstore can be become unreachable temporarily, the Kubernetes apiserver can
crash, cluster nodes can get rebooted,... Therefore, code that is associated
with such events will eventually fail. How should we notify you as a user? The
obvious answers are:

- Not at all, the code should be written in a resilient manner and retry on
 failure to eventually recover.

- The error messages indicating the failure are logged to a logfile.

While resilience is great and logfiles allow to reconstruct all actions
retrospectively, it makes it hard to know at a specific point in time, how well
the cluster is doing right now. For this purpose, we have introduced what
we call "controller status" to the `cilium status` output:

```
$ cilium status
[...]
Controller Status (0/2 failing)
  Name                               Last success   Last error   Count   Message
  sync-identity-to-k8s-pod (56326)   36s ago        never        0       no error
  sync-identity-to-k8s-pod (29898)   32s ago        never        0       no error
```

This will allow to give an immediate overview of what is failing right now, why
it is failing and how often it has been retried. Right now, the `cilium status`
command is available on each node. We will provide a cluster wide tool in one of
the next releases.

Another common source for overall cluster health issues are defects in the network
fabric itself which result in connectivity problems. To allow for simple and
effective monitoring, we have introduced `cilium-health`:

```
$ cilium-health status
Probe time:   2018-02-06T19:40:16Z
Nodes:
 k8s1 (localhost):
   Host connectivity to 192.168.36.11:
     ICMP:          OK, RTT=1.258166ms
     HTTP via L3:   OK, RTT=434.173µs
   Endpoint connectivity to 10.10.0.172:
     ICMP:          OK, RTT=1.266885ms
     HTTP via L3:   OK, RTT=554.219µs
 k8s2:
   Host connectivity to 192.168.36.12:
     ICMP:          OK, RTT=1.53503ms
     HTTP via L3:   OK, RTT=2.420321ms
   Endpoint connectivity to 10.10.1.172:
     ICMP:          OK, RTT=2.081433ms
     HTTP via L3:   OK, RTT=6.550839ms
```

A full blog post on this feature can be found [here](/blog/2018/2/6/cilium-troubleshooting-cluster-health-monitor)

##### Improved scalable kvstore interaction layer

The last big change is a heavily improved interaction layer with the kvstore.
We will provide a dedicated blog post on the exact details along with proper
documentation, the highlights are:

- All keys inserted by agents to manage the allocation of security identities
 for endpoints and pods are now protected by leases which means that if a
 node running an agent goes down and never comes up, the keys will eventually
 expire and the kvstore will not end up cluttered with unused keys.

- The process of allocating a security identity has become a lot more
 lightweight and requires less locking. With etcd 3.3 we hope to provide a
 completely lockless operation exclusively depending on conditional
 transactions to improve scalability even further.

- A new `cilium kvstore` command gives easy access to all kvstore keys and values.

##### Release Notes

###### Major Changes

- api: Introduce & expose endpoint controller statuses (#2720, @tgraf)
- More scalable kvstore interaction layer (#2708, @tgraf)
- Add agent notifications & access log records to monitor (#2667, @tgraf)
- Remove oxyproxy and make Envoy the default proxy (#2625, @jrajahalme)
- New controller pattern for async operations that can fail (#2597, @tgraf)
- Add cilium-health endpoints for datapath connectivity probing (#2315, @joestringer)

###### Bugfixes Changes

- Avoid concurrent access of rand.Rand (#2823, @tgraf)
- kafka: Use policy identity cache to lookup identity for L3 dependent rules (#2813, @manalibhutiyani)
- envoy: Set source identity correctly in access log. (#2807, @jrajahalme)
- replaced sysctl invocation with echo redirects (#2789, @aanm)
- Set up the k8s watchers based on the kube-apiserver version 2731 (##2735, @aanm)
- bpf: Use upper 16 bits of mark for identity (#2719, @tgraf)
- bpf: Generate BPF header in order after generating policy (#2718, @tgraf)
- Kubernetes NetworkPolicyPeer allows for PodSelector and NamespaceSelector fields to be optional. (#2699, @ianvernon)
 - Gracefully handle when these objects are nil when we are parsing NetworkPolicy.
- Enforce policy update immediately on ongoing connections 2569 #2408 (##2684, @aanm)
- envoy: fix rule regex matching by host (#2649, @aanm)
- Kafka: Correctly check msgSize in ReadResp before discarding. (#2637, @manalibhutiyani)
- Fix envoy deadlock after first crash (#2633, @aanm)
- kafka: Reject requests on empty rule set (#2619, @tgraf)
- CNP CRD schema versioning (#2614, @nebril)
- Fix race while updating L7 proxy redirect in L4PolicyMap (#2607, @joestringer)
- Don't allow API users to modify reserved labels for endpoints. (#2595, @joestringer)

##### Release binaries

- [cilium-agent-x86_64](http://releases.cilium.io/v1.0.0-rc4/cilium-agent-x86_64) ([c58a3a05d8531bd8f677](http://releases.cilium.io/v1.0.0-rc4/cilium-agent-x86_64.sha256sum))
- [cilium-bugtool-x86_64](http://releases.cilium.io/v1.0.0-rc4/cilium-bugtool-x86_64) ([5ba0547857d71a96d99c](http://releases.cilium.io/v1.0.0-rc4/cilium-bugtool-x86_64.sha256sum))
- [cilium-health-x86_64](http://releases.cilium.io/v1.0.0-rc4/cilium-health-x86_64) ([f0015f1345e9bb7eccec](http://releases.cilium.io/v1.0.0-rc4/cilium-health-x86_64.sha256sum))
- [cilium-node-monitor-x86_64](http://releases.cilium.io/v1.0.0-rc4/cilium-node-monitor-x86_64) ([81e189969dcf2a97aca3](http://releases.cilium.io/v1.0.0-rc4/cilium-node-monitor-x86_64.sha256sum))
- [cilium-x86_64](http://releases.cilium.io/v1.0.0-rc4/cilium-x86_64) ([2f63b204753aa7a96bb0](http://releases.cilium.io/v1.0.0-rc4/cilium-x86_64.sha256sum))
- [v1.0.0-rc4.tar.gz](http://releases.cilium.io/v1.0.0-rc4/v1.0.0-rc4.tar.gz) ([39ff5357ea5920af6bca](http://releases.cilium.io/v1.0.0-rc4/v1.0.0-rc4.tar.gz.sha256sum))
- [v1.0.0-rc4.zip](http://releases.cilium.io/v1.0.0-rc4/v1.0.0-rc4.zip) ([1c371d84ccad990c6915](http://releases.cilium.io/v1.0.0-rc4/v1.0.0-rc4.zip.sha256sum))

As usual, let us know on [Slack](https://slack.cilium.io/) if you have any questions.

#### Connectivity Troubleshooting with cilium-health

URL: https://cilium.io/blog/2018/2/6/cilium-troubleshooting-cluster-health-monitor
Date: 2018-02-07
Categories: How-To

As we approach the upcoming 1.0 release, the Cilium community has been putting a lot of effort towards monitoring and troubleshooting. This has led to the development of several new tools in the project which we'll explore in this blog series. In this first part, we will cover **cilium-health**, a tool for troubleshooting intra-cluster connectivity issues.

---

##### What’s cilium-health?

**cilium-health** is a new tool available in Cilium which provides visibility into the overall health of the cluster’s networking connectivity.

```
Agent for hosting and querying the Cilium health status API

Usage:
  cilium-health [flags]
  cilium-health [command]

Available Commands:
  get         Display local cilium agent status
  ping        Check whether the cilium-health API is up
  status      Display cilium connectivity to other nodes

Flags:
      --admin string     Expose resources over 'unix' socket, 'any' socket (default "any")
  -c, --cilium string    URI to Cilium server API
  -d, --daemon           Run as a daemon
  -D, --debug            Enable debug messages
  -H, --host string      URI to cilium-health server API
  -i, --interval int     Interval (in seconds) for periodic connectivity probes (default 60)
  -p, --passive          Only respond to HTTP health checks
      --pidfile string   Write the PID to the specified file

Use "cilium-health [command] --help" for more information about a command.
```

It aims to answer the following questions:

Is Cilium successfully deployed on all cluster nodes? Can my cluster nodes reach one another? Can endpoints/pods on each node reach each other? What is the network latency between nodes and endpoints? If there are connectivity issues, where do I look next to resolve them?

The following example shows usage of the **cilium-health** tool. It assumes that you've already deployed Cilium into your cluster, for which you can get more information here. **cilium-health** can be run from any node:

```
$ cilium-health status
Probe time:   2018-02-06T19:40:16Z
Nodes:
  k8s1 (localhost):
    Host connectivity to 192.168.36.11:
      ICMP:          OK, RTT=1.258166ms
      HTTP via L3:   OK, RTT=434.173µs
    Endpoint connectivity to 10.10.0.172:
      ICMP:          OK, RTT=1.266885ms
      HTTP via L3:   OK, RTT=554.219µs
  k8s2:
    Host connectivity to 192.168.36.12:
      ICMP:          OK, RTT=1.53503ms
      HTTP via L3:   OK, RTT=2.420321ms
    Endpoint connectivity to 10.10.1.172:
      ICMP:          OK, RTT=2.081433ms
      HTTP via L3:   OK, RTT=6.550839ms
```

The first line describes the time that the cluster connectivity was probed. By default, the connectivity is probed roughly once every sixty seconds. Running the command as above will return the status during the most recent probe. If you have any reason to suspect that a connectivity issue was introduced more recently, you can run `cilium-health status --probe` to actively probe the cluster connectivity at any point. In `--probe` mode, cilium-health will synchronously probe the connectivity and report it back when it gets the results. This may take a few seconds.

After this, the command prints a list of all known nodes in the cluster. This list is determined through the same mechanism that Cilium uses to connect all cluster nodes together. If the list provided by **cilium-health** does not include all of your cluster nodes, then this means that Cilium was not successfully deployed on all cluster nodes or it indicates a problem with the orchestration system (e.g. Kubernetes).

For each node, the output displays the name of the node, then multiple paths related to that node: Is the localhost able to reach the remote node via ICMP and HTTP? In this case, the tool reports "OK", and the Round Trip Time (RTT) of the last probe. Is the local node able to reach the special health-checking endpoint on the remote node over ICMP and HTTP? Again, we can see the status and a snapshot of the most recent RTT.

These paths for each node describe the nature of the connectivity to that node - can the node be reached at all? If so, can endpoints on that node be reached if there is no policy applied? Traditionally, ICMP (ping) is often used to probe connectivity and latency to a host, however in some deployments, operators may choose to drop ICMP traffic on the fabric. Even if ICMP is being dropped on the fabric, legitimate HTTP requests (over TCP) should still be allowed. Furthermore, the latency for the HTTP requests provide an indication of the expected response time for an application for which Cilium provides networking and security.

##### Hands-on: Troubleshooting an iptables issue

So what happens when things aren't working, and how can we interpret the output of **cilium-health**? We hit an issue with the following symptoms recently:

_We deployed Cilium and spawned up some endpoints but we were unable to connect between endpoints on different nodes._

In order to troubleshoot, we used the **cilium-health** tool to have a starting point on troubleshooting, with the following output:

```
$ cilium-health status
Probe time:   2018-02-06T23:57:45Z
Nodes:
  k8s1 (localhost):
    Host connectivity to 192.168.36.11:
      ICMP:          OK, RTT=424.945µs
      HTTP via L3:   OK, RTT=1.085883ms
    Endpoint connectivity to 10.10.0.172:
      HTTP via L3:   OK, RTT=2.100516ms
      ICMP:          OK, RTT=445.51µs
  k8s2:
    Host connectivity to 192.168.36.12:
      ICMP:          OK, RTT=573.414µs
      HTTP via L3:   OK, RTT=2.710255ms
    Endpoint connectivity to 10.10.1.172:
      ICMP:          Connection timed out
      HTTP via L3:   Connection timed out
```

For some reason, the nodes could connect to one another, but endpoints couldn't be reached on other nodes. The output was similar when observed from the other node, again with only the local endpoint appearing to be reachable. This tells us a few important pieces of information:

- The underlying connectivity between the nodes is working fine.
- The traffic gets dropped somewhere within the remote host.
- The health checking endpoint does not have connectivity which rules out a user error when configuring policy. The issue affects traffic before it reaches the policy evaluation layer.

Further investigation on the node revealed a netfilter misconfiguration. The FORWARD chain was set to the default policy DROP which resulted in all packets to be dropped when forwarded from the overlay network device to Cilium.

##### Under the hood

On each node where cilium is running, it spawns instances of the cilium-health daemon. One of these instances runs co-located with Cilium, with access to the Cilium API which feeds it the IP addresses of all cilium-health daemons operating in the cluster. This instance is responsible for periodically sending ICMP and HTTP requests to each of these IPs to check connectivity and measure the latency for the remote host to respond.

Another instance of the cilium-health daemon is launched on each node in a similar way to an endpoint. This one doesn’t communicate with the main Cilium daemon, but simply serves responses to incoming HTTP requests from the other cilium-health daemons. The probes which test this path are running over the same datapath logic as any other regular endpoint, so if it’s not working, then something is likely wrong for all endpoints on the node; whereas if connectivity to this node works, then it suggests that any other connectivity issues observed must occur with additional features above, for instance layer 7 policies or services. Finally, this endpoint has special “reserved” labels associated with it so that it can be easily identified:

```
root@k8s1:~# cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                      IPv6                 IPv4            STATUS
            ENFORCEMENT        ENFORCEMENT
29898      Disabled           Disabled          299        reserved:health                                 f00d::a0f:0:0:74ca   10.15.242.54    ready
...
```

##### Summary

Cilium-health provides an instant snapshot of the health of a Cilium cluster from a connectivity perspective.

Is Cilium running on all Cluster nodes? (All nodes are being listed) Can all my nodes reach each other? (No connectivity issues between nodes) Do any of the nodes have configuration issues which prevent the health-checking endpoints to be reached? (No connectivity issues to to health-checking endpoints) Is something on the network blocking ICMP or HTTP?

Stay tuned for upcoming blog posts that will dig into where to look next.

These insights can assist troubleshooting issues in the underlying network (such as routing or tunneling issues) or misconfigurations of node settings (such as firewall rules inadvertently interrupting the flow of traffic). They can help to point towards which tools will provide additional information, and speed up problem diagnosis in your clusters.

Happy troubleshooting!

#### Cilium 1.0.0-rc2 - gRPC, Kafka and much more

URL: https://cilium.io/blog/cilium1dot0rc
Date: 2017-12-06
Categories: Release

The Cilium community has been hard at work over the past weeks to get us closer to what we consider is required for a 1.0 release. We have made a ton of progress and are happy to announce the release of 1.0.0-rc2 at this point.

##### New functionality that was MERGED RECENTLY:

- Security policy enforcement at application protocol level for [Kafka](http://docs.cilium.io/en/v1.0.0-rc2/gettingstarted/kafka/), and [gRPC](/grpc).
- Initial roadmap of the [Istio integration](https://www.cilium.io/blog/istio) including the integration of the [Envoy](https://github.com/envoyproxy/envoy) proxy into the Cilium datapath.

- Lots of tooling around operating Cilium based clusters (cluster wide connectivity monitor, bug reporting tools, Prometheus metrics, security incident process,...)

- Lots and lots of documentation and guides.

Thanks goes out to all contributors to Cilium for this release. Let's keep up the great work.

##### When should a project announce 1.0?

There are many different philosophies around when exactly to release version 1.0 of an open source project. Some projects will never reach 1.0, other projects start off with a 1.0 release out of the gate.

The following is what the community has defined to be the requirements for the 1.0 release:

- Completeness of functionality to cover the initial problem definition scope. This includes:
 - Robust, flexible and scalable networking based on BPF technology in the Linux kernel for both IPv4 and IPv6.

 - Efficient and scalable BPF based L3-L4 load balancing between endpoints managed by Cilium.

 - Reliable, identity based policy enforcement between Cilium endpoints and CIDR based matching for external endpoints.

 - Policy enforcement on application protocol level (L7) for some of the most popular protocols: HTTP, gRPC and Kafka.

 - Integration with the most common orchestration frameworks including Kubernetes and Mesos.

 - Troubleshooting and monitoring utilities making use of the visibility advantages of BPF.

 - Metrics and structured logging to allow monitoring and operating at many Cilium nodes at scale

 - Good CI coverage and infrastructure including a nightly test infrastructure to allow for continued growth of the project

- Well-defined documentation covering project scope, installation, use, troubleshooting, contribution, and so on.

- A critical mass of initial users using functionality to cover the entire scope.

We are well on our way. Many foundational pieces including the networking and load-balancing layers have been proven to be very solid over the past 6 months. Other areas such as the layer 7 policy enforcement has been heavily worked on over the past couple of months.

The following is what we consider the missing pieces before declaring 1.0:

- Maturing of the recently merged integration with the [Envoy](https://github.com/envoyproxy/envoy) proxy providing enforcement for HTTP and gRPC going forward as well as the recently added Kafka protocol policy enforcement capability. While Envoy itself is already in heavy use by many users, the integration with Cilium is new.

- Completing the policy enforcement functionality on layer 7\. This includes the ability to integrate with services like Istio Auth for certificate management and the introduction of source dependent layer 7 rules to the Envoy proxy.

We looked at several options on what exactly to call this release and how to proceed. Just declaring 1.0 now did not make sense as the layer 7 functionality is at the core of what many users expect of Cilium. We are well past what is considered a beta. We have thus decided to call it a release candidate.

We have put all the usual processes in place that typically come with a 1.0 release:

- Stable back porting

- Security incident handling according to best practice

- Stability of all APIs and interfaces

This gives all users of Cilium the ability to use and leverage the already matured components of Cilium while we give the new layer 7 enforcement pieces the required time to mature.

##### What will come after 1.0?

There are several areas of focus that will define the direction after 1.0 has been released. Some of them are already in progress:

- Support for additional layer 7 protocols by either adding the relevant protocol parsers to the growing collection of supported proxies or by integrating with new proxies. What protocols are you interested in? Let us know!

- Integration of the in-kernel enforcement and proxying capability. The initial framework has already been merged into the upstream Linux kernel repositories. This effort will provide layer 7 functionality at …...

- End to end encryption and authentication

- Continued integration with the Istio project to make the Cilium datapath available to components such as Istio Mixer and to integrate with Istio Auth.

How does that sound? Let us know on [Slack](https://slack.cilium.io/) or by filling a [GitHub](https://github.com/cilium/cilium) issues.

#### Cilium Now Speaks gRPC: Fine-Grained Network Security for Modern Microservices

URL: https://cilium.io/blog/grpc
Date: 2017-12-06
Categories: Technology

The Cilium team is happy to announce tech preview support for gRPC-aware filtering!

While the majority of existing API-based services leverage HTTP REST as their primary protocol for inter-service communication, among teams designing new platforms from scratch, [gRPC](http://www.grpc.io) is quickly gaining steam.  gRPC is based on Google's popular protobuf project, which provides a more compact and efficiently serializable RPC payload.

Microservices written using gRPC typically include a large number of RPC "methods", all of which are exposed on a single TCP port belonging to the gRPC server.  As a result, a traditional network firewall would either open or close the port of the gRPC server, exposing either all or none of the gRPC methods for a service to each RPC client.  However, Cilium's API-aware filtering enables fine-grain security policies that selectively expose RPC methods to different remote callers, eliminating unnecessary attack surface.

We have created a Cilium + gRPC "Getting Started Guide" so you can try it out yourself: http://docs.cilium.io/en/stable/gettingstarted/grpc/.  Building on our tradition of Star Wars-themed demos, this guide explains how the lack of gRPC-aware network security helped the rebels escape from Cloud City during "The Empire Strikes Back".   Check out the video!

As always, we're very interested in your questions and feedback, so don't hesitate to reach out via Twitter ([@ciliumproject](https://twitter.com/ciliumproject)) or Slack ([https://slack.cilium.io/](https://slack.cilium.io/)).   And don't forget to check out the code and star us on [Cilium Github](http://github.com/cilium/cilium) .  Happy gRPC-ing!

#### What Cilium and BPF will bring to Istio

URL: https://cilium.io/blog/istio
Date: 2017-12-05
Categories: Technology

There is a lot of excitement around Istio this week at KubeCon. We are getting pinged multiple times a day with questions on how exactly Cilium and Istio relate to each other. This blog post serves to answer a couple of simple questions: How is Istio related to Cilium? Can I use both together? Will one benefit from the other?

Istio abstracts away a lot of networking specific complexity and provides visibility and control to application teams. We couldn't agree more with the move of networking to Layer 7 and the concept to provide the necessary instruments for efficient operation at the application protocol layer.

##### What is Istio?

If you are new to Istio, take a moment and read the [Istio Overview](https://istio.io/latest/docs/concepts/what-is-istio/) in the Istio documentation to make yourself familiar with the goals and non-goals of Istio.

##### What is Cilium?

Cilium comes in the form of a networking plugin and thus integrates at a lower level with the orchestration system. Cilium and Istio share a common goal though, both aim to move visibility and control to the application protocol level (HTTP, gRPC, Kafka, Mongo,...). Cilium uses a combination of components to provide this functionality:

- An agent written in golang that runs on all nodes to orchestrate everything. This agent is integrated with orchestration systems such as Kubernetes.
- A datapath component that utilizes the BPF (Berkley Packet Filter) functionality in the Linux kernel for very efficient networking, policy enforcement, and load balancing functionality.
- A set of userspace proxies, one of them is [Envoy](https://github.com/envoyproxy/envoy), to provide application protocol level filtering while we are completing the in-kernel version of this. More on this below.

Istio itself is using [Envoy](https://github.com/envoyproxy/envoy) as well for the implementation of its datapath. The deployment model of Envoy is slightly different though. Istio currently runs Envoy in a sidecar configuration inside of the application pod. Cilium runs Envoy outside of the application pod and configures separate listeners for individual pods. There is no right or wrong in this model, both have advantages and disadvantages on a variety of aspects including operational complexity, security, resource accounting, total footprint. Cilium will likely allow running either model in the future. In short, we had to start somewhere.

##### Can I run Cilium alongside Istio?

Absolutely. It is perfectly fine to run Cilium as a CNI plugin to provide networking, security, and loadbalancing and then deploy Istio on top.

If you want to give it a try right away, you can follow our getting started guide to get going:

- [Get Started Using Istio and Cilium](http://docs.cilium.io/en/stable/gettingstarted/istio/)

The following diagram shows how such an architecture will look like.

##### How will Istio benefit from Cilium?

We are very excited about BPF and how it is changing how security and networking are done with Linux. You can read up on a lot of details in our [Concepts](http://docs.cilium.io/en/stable/concepts/) section of the docs. The rest of this blog will focus on how exactly that applies to the Istio architecture.

There are multiple levels of integration between Cilium and Istio that make sense for both projects. We'll go into some details for all of them but focus on the key point first to explain why you may care.

The above diagram shows how the Cilium datapath can be leveraged by Istio and Kubernetes at the same time. Istio and Kubernetes can both be used as orchestration planes in a collaborative manner. Neat, what does this give me?

We talked about BPF and our in-kernel work before. We have already merged the initial bits of this work into the upstream Linux kernel and are at the point where we can perform initial measurements of the benefits of implementing layer 7 functionality in the kernel.

The following graph lists latency measurements in microseconds for various proxies that are known to be high performing. The latencies are ranked by percentile. The setup is very simple: Two containers running in a pod each are talking to each other via one of the listed proxies. No policy rules, no routing rules, no iptables.

You should take these measurements with a big tablespoon of salt. These are early measurements done on one particular system. We have done these measurements to verify the initial implementation and to validate whether it makes sense to continue. It is not our goal to make one proxy look better than the other. What we are demonstrating here is that it is worth rethinking how and where to perform the datapath operations because the difference is an order of magnitude. The numbers are **very** promising as you can see and have reassured us to continue down this path. While we are measuring latency here, this obviously translates to the use of compute spent handling individual requests as well.

##### Why is the In-kernel proxy faster than not running a proxy at all?

When changing how to approach a problem. Completely new solutions often present themselves. One of them is what we call socket redirect. The in-kernel proxy is capable of having two pods talk to each other directly from socket to socket without ever creating a single TCP packet. This is very similar to having two processes talk to each other using a UNIX domain socket. The difference is that the applications can remain unchanged while using standard TCP sockets.

The difference in the two lines between "No Proxy" and "Cilium In-Kernel" is thus the cost of the TCP/IP stack in the Linux kernel.

##### How else can Istio and Cilium benefit from each other?

While the difference in datapath performance and latency is the key element of what Cilium can bring to Istio. Istio can enrich Cilium in various aspects:

- Use of Istio Auth and the concept of identities to enforce the existing Cilium identity concept. This would allow enforcing existing NetworkPolicy with the automatically generated certificates as provided by Istio Auth.
- Ability to export telemetry from Cilium to Istio.
- Potential to offload Istio Mixer functionality in Cilium

Let us know what else is on your mind and what other questions we can answer. We are very excited about the fast evoution of this new style of networking with awareness of application protocols and application context.

As usual, feel free to drop by our Slack channel if you have questions or want to provide feedback.

See you all at KubeCon this week.

#### Cilium now supports Kafka!

URL: https://cilium.io/blog/2017/11/7/cilium-with-kafka
Date: 2017-11-30
Categories: Release

We have released Cilium v0.12 a couple of weeks back. One of the exciting feature additions is the introduction of Kafka protocol visibility and policy enforcement in form of a tech preview.

The following video will take you through a quick Kafka demo:

You can also check out the [Kafka Getting Started Guide](http://docs.cilium.io/en/stable/gettingstarted/#getting-started-using-kubernetes-with-kafka) to learn how to enforce Kafka-aware policies yourself.

##### Other Relevant CILIUM NEWS

- Iinitial implementation stages of <nobr data-preserve-html-node="true">XDP-based</nobr> filtering enable source IP filtering in the NIC driver, with incredible performance results for protecting a host that can be seen [here](https://www.slideshare.net/ThomasGraf5/cilium-network-security-for-microservices#slide7). Below are the highlights from Cilium
- Simplified policy model to express connectivity for special entities "world" (outside of the cluster) and "host" (system on which endpoint is running on)
- XDP policy enforcement for filtering out source IPs and allowing host and endpoint destination IPs
- Initial framework to support multiple user-space proxies brings the ability to parse many more application protocols
- Auto-population of IPv6 routes for all hosts in the cluster to minimize IPv6 control plane routing (applicable for non-overlay mode)
- Support for L3-dependent L4 policies on ingress, enabled by expanding _PolicyMap_ entry options in BPF
- Unified Cilium default policy behaviour: platform-agnostic enforcement behavior. Now, policy is defined as a whitelist on per-endpoint basis, globally (in-line with Kubernetes behaviour).
- Cluster-wide information on Cilium identities via CLI/API.
- Cilium support for Kubernetes 1.8
- Improved _DaemonSet_ file to automatically derive Kubernetes API access
- Support for _DaemonSet_ configuration, such as _etcd_ endpoints, via _ConfigMap_
- Support for ingress and egress rules with IP blocks, including blacklisting
- Prioritization of Kubernetes pod CIDR for node CIDR allocation.

We’ve also grown our documentation to include several specifics varying from policy enforcement and rules to BPF debugging datapath and developer documentation in the Contributor guide. Take a closer look at our [Docs](http://cilium.readthedocs.io/en/stable/) page.

As always, we’re here to help with any questions on [Cilium Slack](https://slack.cilium.io) or file any issues for the project on [GitHub](https://github.com/cilium/cilium/issues). You can tweet to us [@ciliumproject](https://twitter.com/ciliumproject) and follow us on Twitter for more updates.

~ The Cilium Team

#### BPF Updates 13

URL: https://cilium.io/blog/2017/11/20/bpf-updates-13
Date: 2017-11-20
Categories: Technology

This is issue 13 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

The v4.15 merge window is open and LWN.net already has a summary on [part 1](https://lwn.net/SubscriberLink/739341/f18f047b49653547/) out. Which contains a BPF section listing some of the new things:

> BPF
>
> The user-space bpftool utility can be used to examine and manipulate BPF programs and maps; see this man page for more information.
>
> Hooks have been added to allow security modules to control access to BPF objects; see this changelog for more information.
>
> A new BPF-based device controller has been added; it uses the version-2 control-group interface. Documentation for this feature is entirely absent, but one can look at the sample program added in this commit that uses it.

The highlights since last time

- New helper function `bpf_getsockops` to retrieve socket options. supports `TCP_CONGESTION` for now. The new `BPF_SOCK_OPS_BASE_RTT` feature significantly improves TCP-NV.
- It is now possible to attach multiple programs to tracepoint / kprobes / uprobes. The programs will run in sequence. With the change for trace points one application does not exclude others from attaching to the same call.

More interesting topics

- New helper function `bpf_override_function` under discussion to allow for error injection via kprobes.
- BPF runtime finally gets a [FAQ section](https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/Documentation/bpf/bpf_design_QA.txt) in the kernel's documentation directory.
- bpftool gets support for dumping JSON.

##### Presentations

###### [Cilium - Kernel Native Security & DDOS Mitigation for Microservices with BPF](https://dockercon.docker.com/watch/8RL2xBhXdhwz2NFCbVZzdF)

The slides of Cynthia's talk were already in the [last issue](https://www.cilium.io/blog/2017/10/24/bpf-updates-12). Docker has since published the recording as well, definitely worth watching the recording. Fun talk on Cilium, BPF, and Kafka.

###### [Linux Networking Development](http://vger.kernel.org/~davem/Seoul_NIPA_2017/Seoul_NIPA_2017_part1.pdf)

Focusing on development areas in the kernel. Also some advice in there for aspiring kernel developers.;-)

###### [XDP: The Future of Networks](http://vger.kernel.org/~davem/Seoul_NIPA_2017/Seoul_NIPA_2017_part2.pdf)

Great introduction to BPF and XDP. With some myth busting and potential improvements.

###### [A Gentle Introduction to [e]BPF - Michael Schubert, Kinvolk GmbH](https://schd.ws/hosted_files/osseu17/7e/a-gentle-introduction-to-ebpf.pdf)

Good introduction to BPF. Also nice that it shows the structures, links to some tools and verifier.

###### [LISA 17 - Fast and Safe Production Monitoring of JVM Applications with BPF Magic](https://www.dropbox.com/s/99594woy145d34n/JVM-BPF.pptx?dl=0)

Focusing on the tracing case with Java but the approaches could still be applied to other environments.

###### [LISA17 Container Performance Analysis](https://www.slideshare.net/brendangregg/lisa17-container-performance-analysis)

Goes through some of the tools used at Netflix and a lot of other smaller tools for tracing. The emphasis on identifying the bottlenecks sounds good.

###### [LISA17 Linux Performance Monitoring With BPF](https://www.dropbox.com/s/9syrh64qaxzz4pz/BPF-workshop.pptx?dl=0)

Lab session for tracing tools with BCC. This is useful for learning about tracing on Linux. It also answers basic question what is tracepoints, kprobes, uprobes, etc. and what are some of the limitations to dynamic tracing. Looks like a lot of fun.

###### [XDP – eXpress Data Path An in-kernel network fast-path A technology overview](http://people.netfilter.org/hawk/presentations/driving-IT2017/driving-IT-2017_XDP_eBPF_technology_Jesper_Brouer.pdf)

Great introduction to BPF and XDP. Also explains the problems and why it is needed.

##### In case you missed it

###### [Reports from Netconf and Netdev](https://lwn.net/Articles/738912/)

LWN.net coverage of the discussions from netconf and all the talks from netdev. All lot of interesting BPF topics in there. Check it out!

###### [security things in Linux v4.14](https://outflux.net/blog/archives/2017/11/14/security-things-in-linux-v4-14/)

The security summary contains a section eBPF JIT 32-bit ARM support and seccomp improvements.

###### [SystemTap 3.2 release](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1516567.html)

SystemTap now has an experimental eBPF backend.

###### [Another attempt to address the tracepoint ABI problem](https://lwn.net/SubscriberLink/737530/6321597a8a64352e/)

Steven Rostedt proposes different scheme where tracepoints are placed but no trace event. Then on userspace a kernel module have to be loaded and there would be no need to add this to the kernel ABI. Will moving the ABI to a module really solve this problem?

###### [Using eBPF and XDP in Suricata](https://lwn.net/SubscriberLink/737771/4862652df9170712/)

LWN.net coverage of Eric Leblond's talk from Kernel Recipes. The recording was already in the [last issue](https://www.cilium.io/blog/2017/10/24/bpf-updates-12).

##### Projects

###### [awesome-ebpf](https://github.com/zoidbergwill/awesome-ebpf)

> A curated list of awesome projects related to eBPF

###### [k8s-snowflake](https://github.com/jessfraz/k8s-snowflake)

> Configs and scripts for bootstrapping an opinionated Kubernetes cluster anywhere.

###### [libseccomp](https://github.com/seccomp/libseccomp)

> The libseccomp library provides an easy to use, platform independent, interface to the Linux Kernel's syscall filtering mechanism. The libseccomp API is designed to abstract away the underlying BPF based syscall filter language and present a more conventional function-call based filtering interface that should be familiar to, and easily adopted by, application developers.

###### [cbpf-rust](https://github.com/mmisono/rust-cbpf)

> Userspace cBPF interpreter and cBPF to eBPF converter

###### [vltrace](https://github.com/pmem/vltrace)

> vltrace is a syscall tracing tool which utilizes eBPF - an efficient tracing feature of the Linux kernel.

##### [Random cool note](https://twitter.com/bmatheny/status/924022314177409025)

> We blew way past 7Mpps with UDP+XDP. I’m sure you know that already though:)

##### Patches

Please note that netdev and llvm-commits receive a lot of patches and the list below is not meant to be comprehensive.

###### LLVM

- Alexei Starovoitov, [[llvm] r318615 - [bpf] remove unused variable](http://llvm.org/viewvc/llvm-project?rev=318615&view=rev)
- Alexei Starovoitov, [[llvm] r318614 - [bpf] allow direct and indirect calls](http://llvm.org/viewvc/llvm-project?rev=318614&view=rev)
- Yonghong Song, [[llvm] r318358 - bpf: enable llvm-objdump to print out symbolized jmp target](http://llvm.org/viewvc/llvm-project?rev=318358&view=rev)
- Yonghong Song, [[llvm] r318442 - bpf: print backward branch target properly](http://llvm.org/viewvc/llvm-project?rev=318442&view=rev)
- Yonghong Song, [[llvm] r316469 - bpf: fix a bug in bpf-isel trunc-op optimization](http://llvm.org/viewvc/llvm-project?rev=316469&view=rev)
- Yonghong Song, [[llvm] r316519 - bpf: fix an uninitialized variable issue](http://llvm.org/viewvc/llvm-project?rev=316519&view=rev)
- Yonghong Song, [[llvm] r316481 - bpf: fix a bug in trunc-op optimization](http://llvm.org/viewvc/llvm-project?rev=316481&view=rev)

###### netdev

- Jakub Kicinski, [[PATCH net 00/10] bpf: offload: check netdev pointer in the drivers and namespace trouble](https://patchwork.ozlabs.org/cover/839433/)
 - [[PATCH net 01/10] bpf: offload: add comment warning developers about double destroy](https://patchwork.ozlabs.org/patch/839423/)
 - [[PATCH net 02/10] bpf: offload: limit offload to `cls_bpf` and xdp programs only](https://patchwork.ozlabs.org/patch/839424/)
 - [[PATCH net 03/10] bpf: offload: rename the ifindex field](https://patchwork.ozlabs.org/patch/839432/)
 - [[PATCH net 04/10] bpf: offload: move offload device validation out to the drivers](https://patchwork.ozlabs.org/patch/839431/)
 - [[PATCH net 05/10] net: xdp: don't allow device-bound programs in driver mode](https://patchwork.ozlabs.org/patch/839425/)
 - [[PATCH net 06/10] bpf: turn `bpf_prog_get_type()` into a wrapper](https://patchwork.ozlabs.org/patch/839430/)
 - [[PATCH net 07/10] bpf: offload: ignore namespace moves](https://patchwork.ozlabs.org/patch/839429/)
 - [[PATCH net 08/10] bpftool: revert printing program device bound info](https://patchwork.ozlabs.org/patch/839428/)
 - [[PATCH net 09/10] bpf: revert report offload info to user space](https://patchwork.ozlabs.org/patch/839427/)
 - [[PATCH net 10/10] bpf: make `bpf_prog_offload_verifier_prep()` static inline](https://patchwork.ozlabs.org/patch/839426/)
- Song Liu, [[RFC v2 0/6] enable creating [k,u]probe with `perf_event_open`](https://patchwork.ozlabs.org/cover/837299/)
 - [[RFC v2 1/6] perf: Add new type `PERF_TYPE_PROBE`](https://patchwork.ozlabs.org/patch/837302/)
 - [[RFC v2 2/6] perf: copy new perf_event.h to tools/include/uapi](https://patchwork.ozlabs.org/patch/837306/)
 - [[RFC v2 3/6] perf: implement kprobe support to `PERF_TYPE_PROBE`](https://patchwork.ozlabs.org/patch/837301/)
 - [[RFC v2 4/6] perf: implement uprobe support to `PERF_TYPE_PROBE`](https://patchwork.ozlabs.org/patch/837307/)
 - [[RFC v2 5/6] bpf: add option for `bpf_load.c` to use `PERF_TYPE_PROBE`](https://patchwork.ozlabs.org/patch/837303/)
 - [[RFC v2 6/6] bpf: add new test `test_many_kprobe`](https://patchwork.ozlabs.org/patch/837304/)
 - [[RFC] bcc: Try use new API to create [k,u]probe with `perf_event_open`](https://patchwork.ozlabs.org/patch/837305/)
 - [[RFC] `perf_event_open.2`: add new type `PERF_TYPE_PROBE`](https://patchwork.ozlabs.org/patch/837300/)
- Yonghong Song, [[PATCH net-next 0/3 v3] bpf: improve verifier `ARG_CONST_SIZE_OR_ZERO` semantics](https://patchwork.ozlabs.org/cover/837288/)
 - [[PATCH net-next 1/3 v3] bpf: improve verifier `ARG_CONST_SIZE_OR_ZERO` semantics](https://patchwork.ozlabs.org/patch/837287/)
 - [[PATCH net-next 2/3 v3] bpf: change helper `bpf_probe_read` arg2 type to `ARG_CONST_SIZE_OR_ZERO`](https://patchwork.ozlabs.org/patch/837290/)
 - [[PATCH net-next 3/3 v3] bpf: fix and add test cases for `ARG_CONST_SIZE_OR_ZERO` semantics change](https://patchwork.ozlabs.org/patch/837289/)
- Lawrence Brakmo, [[PATCH net-next v2 0/6] bpf: Fix bugs in sock_ops samples](https://patchwork.ozlabs.org/cover/836997/)
 - [[PATCH net-next v2 1/6] bpf: Fix `tcp_synrto_kern.c` sample program](https://patchwork.ozlabs.org/patch/836992/)
 - [[PATCH net-next v2 2/6] bpf: Fix `tcp_rwnd_kern.c` sample program](https://patchwork.ozlabs.org/patch/836993/)
 - [[PATCH net-next v2 3/6] bpf: Fix `tcp_bufs_kern.c` sample program](https://patchwork.ozlabs.org/patch/836996/)
 - [[PATCH net-next v2 4/6] bpf: Fix `tcp_cong_kern.c` sample program](https://patchwork.ozlabs.org/patch/836991/)
 - [[PATCH net-next v2 5/6] bpf: Fix `tcp_iw_kern.c` sample program](https://patchwork.ozlabs.org/patch/836995/)
 - [[PATCH net-next v2 6/6] bpf: Fix `tcp_clamp_kern.c` sample program](https://patchwork.ozlabs.org/patch/836994/)
- Prashant Bhole, [[PATCH net-next V4 0/3] tools: bpftool: show filenames of pinned objects](https://patchwork.ozlabs.org/cover/835588/)
 - [[PATCH net-next V4 1/3] tools: bpftool: open pinned object without type check](https://patchwork.ozlabs.org/patch/835589/)
 - [[PATCH net-next V4 2/3] tools: bpftool: show filenames of pinned objects](https://patchwork.ozlabs.org/patch/835590/)
 - [[PATCH net-next V4 3/3] tools: bpftool: optionally show filenames of pinned objects](https://patchwork.ozlabs.org/patch/835591/)
- Jakub Kicinski, [[PATCH net-next v2 00/15] bpf: add offload as a first class citizen](https://patchwork.ozlabs.org/cover/834089/)
 - [[PATCH net-next v2 01/15] net: bpf: rename `ndo_xdp` to ndo_bpf](https://patchwork.ozlabs.org/patch/834103/)
 - [[PATCH net-next v2 02/15] bpf: offload: add infrastructure for loading programs for a specific netdev](https://patchwork.ozlabs.org/patch/834100/)
 - [[PATCH net-next v2 03/15] bpf: report offload info to user space](https://patchwork.ozlabs.org/patch/834104/)
 - [[PATCH net-next v2 04/15] bpftool: print program device bound info](https://patchwork.ozlabs.org/patch/834090/)
 - [[PATCH net-next v2 05/15] xdp: allow attaching programs loaded for specific device](https://patchwork.ozlabs.org/patch/834101/)
 - [[PATCH net-next v2 06/15] `cls_bpf`: allow attaching programs loaded for specific device](https://patchwork.ozlabs.org/patch/834102/)
 - [[PATCH net-next v2 07/15] nfp: bpf: drop support for `cls_bpf` with legacy actions](https://patchwork.ozlabs.org/patch/834095/)
 - [[PATCH net-next v2 08/15] nfp: bpf: remove the register renumbering leftovers](https://patchwork.ozlabs.org/patch/834096/)
 - [[PATCH net-next v2 09/15] nfp: bpf: remove unnecessary include of nfp_net.h](https://patchwork.ozlabs.org/patch/834098/)
 - [[PATCH net-next v2 10/15] nfp: bpf: refactor offload logic](https://patchwork.ozlabs.org/patch/834099/)
 - [[PATCH net-next v2 11/15] nfp: bpf: require seamless reload for program replace](https://patchwork.ozlabs.org/patch/834093/)
 - [[PATCH net-next v2 12/15] nfp: bpf: move program prepare and free into offload.c](https://patchwork.ozlabs.org/patch/834097/)
 - [[PATCH net-next v2 13/15] nfp: bpf: move translation prepare to offload.c](https://patchwork.ozlabs.org/patch/834091/)
 - [[PATCH net-next v2 14/15] nfp: bpf: move to new BPF program offload infrastructure](https://patchwork.ozlabs.org/patch/834094/)
 - [[PATCH net-next v2 15/15] bpf: remove old offload/analyzer](https://patchwork.ozlabs.org/patch/834092/)
- Christina Jacob, [[PATCH v4 0/1] XDP program for ip forward](https://patchwork.ozlabs.org/cover/834242/)
 - [[PATCH v4 1/1] xdp: Sample xdp program implementing ip forward](https://patchwork.ozlabs.org/patch/834241/)
- Dan Carpenter, [[PATCH net-next] xdp: sample: Missing curly braces in read_route()](https://patchwork.ozlabs.org/patch/837684/)
- Josef Bacik, [[PATCH 0/4] [v6] Add the ability to do BPF directed error injection](https://lkml.org/lkml/2017/11/17/446)
 - [[PATCH 1/4] add infrastructure for tagging functions as error injectable](https://lkml.org/lkml/2017/11/17/448)
 - [[PATCH 2/4] btrfs: make open_ctree error injectable](https://lkml.org/lkml/2017/11/17/449)
 - [[PATCH 3/4] bpf: add a `bpf_override_function` helper](https://lkml.org/lkml/2017/11/17/447)
 - [[PATCH 4/4] samples/bpf: add a test for `bpf_override_return`](https://lkml.org/lkml/2017/11/17/450)
- Lawrence Brakmo, [[PATCH net-next] bpf: Rename `tcp_bbf.readme` to `tcp_bpf.readme`](https://patchwork.ozlabs.org/patch/834432/)
- Sandipan Das, [[RFC PATCH] bpf: Add helpers to read useful `task_struct` members](https://patchwork.kernel.org/patch/10039583/)
- Roman Gushchin, [[PATCH v3 net-next 0/5] eBPF-based device cgroup controller](https://www.spinics.net/lists/netdev/msg465071.html)
 - [[PATCH v3 net-next 1/5] `device_cgroup`: add `DEVCG_` prefix to `ACC_*` and `DEV_*` constants](https://www.spinics.net/lists/netdev/msg465066.html)
 - [[PATCH v3 net-next 2/5] `device_cgroup`: prepare code for bpf-based device controller](https://www.spinics.net/lists/netdev/msg465070.html)
 - [[PATCH v3 net-next 3/5] bpf, cgroup: implement eBPF-based device controller for cgroup v2](https://www.spinics.net/lists/netdev/msg465072.html)
 - [[PATCH v3 net-next 4/5] bpf: move `cgroup_helpers` from samples/bpf/ to tools/testing/selftesting/bpf/](https://www.spinics.net/lists/netdev/msg465067.html)
 - [[PATCH v3 net-next 5/5] selftests/bpf: add a test for device cgroup controller](https://www.spinics.net/lists/netdev/msg465069.html)
- Jakub Kicinski, [[PATCH net-next] tools: bpftool: move `p_err()` and `p_info()` from main.h to common.c](https://www.mail-archive.com/netdev@vger.kernel.org/msg198226.html)
- Colin King, [[PATCH net-next] net: sched: `cls_bpf`: use bitwise & rather than logical && on `gen_flags`](https://patchwork.kernel.org/patch/10039257/)
- Craig Gallek, [[PATCH] [net-next v2] bpf: fix verifier NULL pointer dereference](https://patchwork.ozlabs.org/patch/833404/)
- Arnd Bergmann, [[PATCH 1/2] [net-next] bpf: fix link error without CONFIG_NET](https://patchwork.kernel.org/patch/10038423/)
 - [[PATCH 2/2] [net-next] bpf: fix out-of-bounds access warning in bpf_check](https://patchwork.kernel.org/patch/10038425/)
- Eric Dumazet, [[PATCH net] bpf: fix lockdep splat](https://www.spinics.net/lists/netdev/msg467241.html)
- Prashant Bhole, [tools: bpf: handle long path in jit disasm](https://patchwork.ozlabs.org/patch/833248/)
- Jakub Kicinski, [[PATCH net-next 0/8] nfp: TC block fixes, app fallback and dev_alloc()](https://patchwork.ozlabs.org/cover/833250/)
 - [[PATCH net-next 1/8] nfp: flower: app should use struct nfp_repr](https://patchwork.ozlabs.org/patch/833257/)
 - [[PATCH net-next 2/8] nfp: flower: vxlan - ensure no sleep in atomic context](https://patchwork.ozlabs.org/patch/833258/)
 - [[PATCH net-next 3/8] nfp: bpf: reject TC offload if XDP loaded](https://patchwork.ozlabs.org/patch/833255/)
 - [[PATCH net-next 4/8] nfp: reorganize the app table](https://patchwork.ozlabs.org/patch/833256/)
 - [[PATCH net-next 5/8] nfp: bpf: fall back to core NIC app if BPF not selected](https://patchwork.ozlabs.org/patch/833251/)
 - [[PATCH net-next 6/8] nfp: switch to `dev_alloc_page()`](https://patchwork.ozlabs.org/patch/833252/)
 - [[PATCH net-next 7/8] nfp: use a counter instead of log message for allocation failures](https://patchwork.ozlabs.org/patch/833253/)
 - [[PATCH net-next 8/8] nfp: improve defines for constants in ethtool](https://patchwork.ozlabs.org/patch/833254/)
- Daniel Borkmann, [[PATCH net-next 0/3] BPF range marking improvements for meta data](https://patchwork.ozlabs.org/cover/833134/)
 - [[PATCH net-next 1/3] bpf: minor cleanups after merge](https://patchwork.ozlabs.org/patch/833132/)
 - [[PATCH net-next 2/3] bpf: also improve pattern matches for meta access](https://patchwork.ozlabs.org/patch/833135/)
 - [[PATCH net-next 3/3] bpf: add test cases to bpf selftests to cover all meta tests](https://patchwork.ozlabs.org/patch/833133/)
- Jakub Kicinski, [[PATCH net-next] security: bpf: replace include of linux/bpf.h with forward declarations](https://patchwork.ozlabs.org/patch/833083/)
- Jakub Kicinski, [[PATCH net-next 0/2] nfp: bpf: rename `ALU_OP_NEG` and support `BPF_NEG`](https://patchwork.ozlabs.org/cover/833059/)
 - [[PATCH net-next 1/2] nfp: bpf: rename `ALU_OP_NEG` to `ALU_OP_NOT`](https://patchwork.ozlabs.org/patch/833058/)
 - [[PATCH net-next 2/2] nfp: bpf: support [`BPF_ALU` | `BPF_ALU64`] | `BPF_NEG`](https://patchwork.ozlabs.org/patch/833060/)
- Jesper Dangaard Brouer, [[net-next PATCH] bpf: cpumap micro-optimization in `cpu_map_enqueue`](https://www.spinics.net/lists/netdev/msg464218.html)
- Alexei Starovoitov, [[PATCH net-next] bpf: fix verifier memory leaks](https://patchwork.ozlabs.org/patch/832855/)
- John Fastabend, [[net PATCH] bpf: remove SK_REDIRECT from UAPI](https://www.spinics.net/lists/netdev/msg464135.html)
- Alexei Starovoitov, [[PATCH v2 net-next] bpf: reduce verifier memory consumption](https://patchwork.ozlabs.org/patch/832800/)
- Jakub Kicinski, [[RFC] net: dummy: add BPF offload callbacks for test purposes](https://patchwork.ozlabs.org/patch/832821/)
- Björn Töpel, [[RFC PATCH 00/14] Introducing AF_PACKET V4 support](https://www.mail-archive.com/netdev@vger.kernel.org/msg197187.html)
 - [[RFC PATCH 01/14] packet: introduce AF_PACKET V4 userspace API](https://www.mail-archive.com/netdev@vger.kernel.org/msg197191.html)
 - [[RFC PATCH 02/14] packet: implement PACKET_MEMREG setsockopt](https://www.mail-archive.com/netdev@vger.kernel.org/msg197188.html)
 - [[RFC PATCH 03/14] packet: enable AF_PACKET V4 rings](https://www.mail-archive.com/netdev@vger.kernel.org/msg197201.html)
 - [[RFC PATCH 04/14] packet: enable Rx for AF_PACKET V4](https://www.mail-archive.com/netdev@vger.kernel.org/msg197190.html)
 - [[RFC PATCH 05/14] packet: enable Tx support for AF_PACKET V4](https://www.mail-archive.com/netdev@vger.kernel.org/msg197189.html)
 - [[RFC PATCH 06/14] netdevice: add AF_PACKET V4 zerocopy ops](https://www.mail-archive.com/netdev@vger.kernel.org/msg197192.html)
 - [[RFC PATCH 07/14] packet: wire up zerocopy for AF_PACKET V4](https://www.mail-archive.com/netdev@vger.kernel.org/msg197197.html)
 - [[RFC PATCH 08/14] i40e: `AF_PACKET` V4 `ndo_tp4_zerocopy` Rx support](https://www.mail-archive.com/netdev@vger.kernel.org/msg197196.html)
 - [[RFC PATCH 09/14] i40e: `AF_PACKET` V4 `ndo_tp4_zerocopy` Tx support](https://www.mail-archive.com/netdev@vger.kernel.org/msg197194.html)
 - [[RFC PATCH 10/14] samples/tpacket4: added tpbench](https://www.mail-archive.com/netdev@vger.kernel.org/msg197195.html)
 - [[RFC PATCH 11/14] veth: added support for PACKET_ZEROCOPY](https://www.mail-archive.com/netdev@vger.kernel.org/msg197193.html)
 - [[RFC PATCH 12/14] samples/tpacket4: added veth support](https://www.mail-archive.com/netdev@vger.kernel.org/msg197199.html)
 - [[RFC PATCH 13/14] i40e: added XDP support for TP4 enabled queue pairs](https://www.mail-archive.com/netdev@vger.kernel.org/msg197198.html)
 - [[RFC PATCH 14/14] xdp: introducing `XDP_PASS_TO_KERNEL` for `PACKET_ZEROCOPY` use](https://www.mail-archive.com/netdev@vger.kernel.org/msg197200.html)
- Jason Wang, [[PATCH net-next V2 0/3] support changing steering policies in tuntap](https://patchwork.ozlabs.org/cover/832326/)
 - [[PATCH net-next V2 1/3] tun: abstract flow steering logic](https://patchwork.ozlabs.org/patch/832325/)
 - [[PATCH net-next V2 2/3] tun: introduce ioctls to set and get steering policies](https://patchwork.ozlabs.org/patch/832324/)
 - [[PATCH net-next V2 3/3] tun: add eBPF based queue selection method](https://patchwork.ozlabs.org/patch/832323/)
- Alexei Starovoitov, [[PATCH net-next] bpf: document answers to common questions about BPF](https://patchwork.ozlabs.org/patch/832218/)
- Alexei Starovoitov, [[PATCH net-next] bpf: reduce verifier memory consumption](https://www.spinics.net/lists/netdev/msg463754.html)
- Yonghong Song, [[PATCH net-next] bpf: avoid `rcu_dereference` inside `bpf_event_mutex` lock region](https://patchwork.ozlabs.org/patch/832161/)
- Alexei Starovoitov, [[PATCH net-next] selftests/bpf: remove useless `bpf_trace_printk`](https://patchwork.ozlabs.org/patch/831658/)
- Tushar Dave, [[PATCH net-next] samples/bpf: adjust rlimit `RLIMIT_MEMLOCK` for `xdp_redirect_map`](https://patchwork.ozlabs.org/patch/831562/)
- Tushar Dave, [[PATCH net-next] samples/bpf: adjust rlimit `RLIMIT_MEMLOCK` for xdp1](https://patchwork.ozlabs.org/patch/831544/)
- John Fastabend, [[net PATCH 0/2] sockmap fixes](https://www.mail-archive.com/netdev@vger.kernel.org/msg196615.html)
 - [[net PATCH 1/2] bpf: `bpf_compute_data` uses incorrect cb structure](https://www.mail-archive.com/netdev@vger.kernel.org/msg196614.html)
 - [[net PATCH 2/2] bpf: rename sk_actions to align with bpf infrastructure](https://www.mail-archive.com/netdev@vger.kernel.org/msg196616.html)
- Quentin Monnet, [[PATCH net-next] tools: bpftool: add bash completion for bpftool](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1522706.html)
- Gianluca Borello, [[PATCH net-next] bpf: remove `tail_call` and `get_stackid` helper declarations from bpf.h](https://www.spinics.net/lists/netdev/msg462765.html)
- Chenbo Feng, [[PATCH net-next v7 0/5] bpf: security: New file mode and LSM hooks for eBPF object permission control](https://patchwork.ozlabs.org/cover/827794/)
 - [[PATCH net-next v7 1/5] bpf: Add file mode configuration into bpf maps](https://patchwork.ozlabs.org/patch/827795/)
 - [[PATCH net-next v7 2/5] bpf: Add tests for eBPF file mode](https://patchwork.ozlabs.org/patch/827799/)
 - [[PATCH net-next v7 3/5] security: bpf: Add LSM hooks for bpf object related syscall](https://patchwork.ozlabs.org/patch/827796/)
 - [[PATCH net-next v7 4/5] selinux: bpf: Add selinux check for eBPF syscall operations](https://patchwork.ozlabs.org/patch/827797/)
 - [[PATCH net-next v7 5/5] selinux: bpf: Add addtional check for bpf object file receive](https://patchwork.ozlabs.org/patch/827798/)
- Jakub Kicinski, [[PATCH net-next 00/12] tools: bpftool: Add JSON output to bpftool](https://patchwork.ozlabs.org/cover/829394/)
 - [[PATCH net-next 01/12] tools: bpftool: copy JSON writer from iproute2 repository](https://patchwork.ozlabs.org/patch/829395/)
 - [[PATCH net-next 02/12] tools: bpftool: add option parsing to bpftool, --help and --version](https://patchwork.ozlabs.org/patch/829396/)
 - [[PATCH net-next 03/12] tools: bpftool: introduce --json and --pretty options](https://patchwork.ozlabs.org/patch/829407/)
 - [[PATCH net-next 04/12] tools: bpftool: add JSON output for `bpftool prog show *` command](https://patchwork.ozlabs.org/patch/829406/)
 - [[PATCH net-next 05/12] tools: bpftool: add JSON output for `bpftool prog dump jited *` command](https://patchwork.ozlabs.org/patch/829397/)
 - [[PATCH net-next 06/12] tools: bpftool: add JSON output for `bpftool prog dump xlated *` command](https://patchwork.ozlabs.org/patch/829401/)
 - [[PATCH net-next 07/12] tools: bpftool: add JSON output for `bpftool map *` commands](https://patchwork.ozlabs.org/patch/829402/)
 - [[PATCH net-next 08/12] tools: bpftool: add JSON output for `bpftool batch file FILE` command](https://patchwork.ozlabs.org/patch/829403/)
 - [[PATCH net-next 09/12] tools: bpftool: turn err() and info() macros into functions](https://patchwork.ozlabs.org/patch/829404/)
 - [[PATCH net-next 10/12] tools: bpftool: provide JSON output for all possible commands](https://patchwork.ozlabs.org/patch/829399/)
 - [[PATCH net-next 11/12] tools: bpftool: add cosmetic changes for the manual pages](https://patchwork.ozlabs.org/patch/829398/)
 - [[PATCH net-next 12/12] tools: bpftool: update documentation for --json and --pretty usage](https://patchwork.ozlabs.org/patch/829400/)
- Jakub Kicinski, [[PATCH net-next 0/8] tools: bpftool: add a "version" command, and fix several items](https://patchwork.ozlabs.org/cover/828402/)
 - [[PATCH net-next 1/8] tools: bpftool: add pointer to file argument to `print_hex()`](https://patchwork.ozlabs.org/patch/828394/)
 - [[PATCH net-next 2/8] tools: bpftool: fix return value when all eBPF programs have been shown](https://patchwork.ozlabs.org/patch/828395/)
 - [[PATCH net-next 3/8] tools: bpftool: use err() instead of info() if there are too many insns](https://patchwork.ozlabs.org/patch/828400/)
 - [[PATCH net-next 4/8] tools: bpftool: add `bpftool prog help` as real command i.r.t exit code](https://patchwork.ozlabs.org/patch/828401/)
 - [[PATCH net-next 5/8] tools: bpftool: print only one error message on byte parsing failure](https://patchwork.ozlabs.org/patch/828396/)
 - [[PATCH net-next 6/8] tools: bpftool: print all relevant byte opcodes for "load double word"](https://patchwork.ozlabs.org/patch/828397/)
 - [[PATCH net-next 7/8] tools: bpftool: show that `opcodes` or `file FILE` should be exclusive](https://patchwork.ozlabs.org/patch/828398/)
 - [[PATCH net-next 8/8] tools: bpftool: add a command to display bpftool version](https://patchwork.ozlabs.org/patch/828399/)
- Lawrence Brakmo, [[PATCH net-next 0/5] bpf: add support for `BASE_RTT`](https://patchwork.ozlabs.org/cover/828757/)
 - [[PATCH net-next 1/5] bpf: add support for `BPF_SOCK_OPS_BASE_RTT`](https://patchwork.ozlabs.org/patch/828760/)
 - [[PATCH net-next 2/5] bpf: Adding helper function `bpf_getsockops`](https://patchwork.ozlabs.org/patch/828759/)
 - [[PATCH net-next 3/5] bpf: Add `BPF_SOCKET_OPS_BASE_RTT` support to `tcp_nv`](https://patchwork.ozlabs.org/patch/828758/)
 - [[PATCH net-next 4/5] bpf: sample `BPF_SOCKET_OPS_BASE_RTT` program](https://patchwork.ozlabs.org/patch/828761/)
 - [[PATCH net-next 5/5] bpf: create samples/bpf/tcp_bpf.readme](https://patchwork.ozlabs.org/patch/828756/)
- John Fastabend, [[net PATCH 0/5] sockmap fixes for net](https://www.spinics.net/lists/netdev/msg461277.html)
 - [[net PATCH 1/5] bpf: enforce TCP only support for sockmap](https://www.spinics.net/lists/netdev/msg461278.html)
 - [[net PATCH 2/5] bpf: avoid preempt enable/disable in sockmap using `tcp_skb_cb` region](https://www.spinics.net/lists/netdev/msg461279.html)
 - [[net PATCH 3/5] bpf: remove mark access for SK_SKB program types](https://www.spinics.net/lists/netdev/msg461280.html)
 - [[net PATCH 4/5] bpf: require `CAP_NET_ADMIN` when using sockmap maps](https://www.spinics.net/lists/netdev/msg461281.html)
 - [[net PATCH 5/5] bpf: require `CAP_NET_ADMIN` when using devmap](https://www.spinics.net/lists/netdev/msg461282.html)
- Daniel Borkmann, [[PATCH net 0/3] Two BPF fixes for range marking](https://patchwork.ozlabs.org/cover/828901/)
 - [[PATCH net 1/3] bpf: fix off by one for range markings with L{'{'}T,E} patterns](https://patchwork.ozlabs.org/patch/828899/)
 - [[PATCH net 2/3] bpf: fix pattern matches for direct packet access](https://patchwork.ozlabs.org/patch/828902/)
 - [[PATCH net 3/3] bpf: add test cases to bpf selftests to cover all access tests](https://patchwork.ozlabs.org/patch/828900/)
- Jesper Dangaard Brouer, [[net-next PATCH] bpf: cpumap fix potential lost wake-up problem](https://patchwork.ozlabs.org/patch/829524/)
- Jakub Kicinski, [[PATCH net-next 0/9] nfp: bpf: stack support in offload](https://patchwork.ozlabs.org/cover/829570/)
 - [[PATCH net-next 1/9] nfp: bpf: add helper for emitting nops](https://patchwork.ozlabs.org/patch/829571/)
 - [[PATCH net-next 2/9] nfp: bpf: refactor `nfp_bpf_check_ptr()`](https://patchwork.ozlabs.org/patch/829594/)
 - [[PATCH net-next 3/9] nfp: bpf: add stack write support](https://patchwork.ozlabs.org/patch/829573/)
 - [[PATCH net-next 4/9] nfp: bpf: add stack read support](https://patchwork.ozlabs.org/patch/829590/)
 - [[PATCH net-next 5/9] nfp: bpf: optimize the RMW for stack accesses](https://patchwork.ozlabs.org/patch/829574/)
 - [[PATCH net-next 6/9] nfp: bpf: allow stack accesses via modified stack registers](https://patchwork.ozlabs.org/patch/829593/)
 - [[PATCH net-next 7/9] nfp: bpf: support accessing the stack beyond 64 bytes](https://patchwork.ozlabs.org/patch/829592/)
 - [[PATCH net-next 8/9] nfp: bpf: support stack accesses via non-constant pointers](https://patchwork.ozlabs.org/patch/829591/)
 - [[PATCH net-next 9/9] nfp: bpf: optimize mov64 a little](https://patchwork.ozlabs.org/patch/829572/)
- Yonghong Song, [[PATCH net-next v3 0/3] bpf: permit multiple bpf attachments for a single perf tracepoint event](https://patchwork.ozlabs.org/cover/829786/)
 - [[PATCH net-next v3 1/3] bpf: use the same condition in perf event set/free bpf handler](https://patchwork.ozlabs.org/patch/829788/)
 - [[PATCH net-next v3 2/3] bpf: permit multiple bpf attachments for a single perf event](https://patchwork.ozlabs.org/patch/829785/)
 - [[PATCH net-next v3 3/3] bpf: add a test case to test single tp multiple bpf attachment](https://patchwork.ozlabs.org/patch/829787/)
- Quentin Monnet, [[PATCH net-next] tools: bpftool: try to mount bpffs if required for pinning objects](https://patchwork.ozlabs.org/patch/830083/)
- John Fastabend, [[net PATCH] bpf: devmap fix arithmetic overflow in bitmap_size calculation](https://patchwork.ozlabs.org/patch/828187/)
- Alexei Starovoitov, [[PATCH v2 net-next] selftests/bpf: fix broken build of test_maps](https://patchwork.ozlabs.org/patch/829064/)

#### BPF Updates 12

URL: https://cilium.io/blog/2017/10/24/bpf-updates-12
Date: 2017-10-25
Categories: Technology

This is issue 12 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

The highlights since the last time

- Generic metadata transfer from XDP into skb via new helper function `bpf_xdp_adjust_meta`.
- `bpf_perf_event_read_value` helper function series got merged.
- Multiple programs can now be attached to a cgroup.
- A new map type `cpumap` for XDP got merged.

The addition of metadata transfer from XDP allows XDP programs to apply early filters to extract metadata from packets and make it available to BPF programs later in the forwarding chain such as programs attached to clsact qdiscs.

The new values available via the perf helper make it easier to normalize measurements. `cpumap` let's you redirect XDP frames to remote CPU's where these can handle the packet representation and fire up the networking stack.

More interesting topics

- More fields added to `bpf_prog_info`.
- Netronome bpftool upstreamed.
- LPM performance improvements.
- 32-bit eBPF encoding support.

The BPF tooling is evolving and introspection is getting better.

##### Presentations

###### Talks

###### [All Systems Go! 2017 - A gentle introduction to [e]BPF](https://media.ccc.de/v/ASG2017-92-a_gentle_introduction_to_e_bpf#video&t=63)

Beginner friendly overview of BPF.

###### [All Systems Go! 2017 - High-performance Linux monitoring with eBPF](https://media.ccc.de/v/ASG2017-139-high-performance_linux_monitoring_with_ebpf#video&t=64)

Intro to BPF and short part on how it's being leveraged in weaveworks.

###### [All Systems Go! 2017 - Using BPF in Kubernetes](https://media.ccc.de/v/ASG2017-134-using_bpf_in_kubernetes)

Nice talk focusing on some of the use cases in the cloud.

###### [Kernel Recipes 2017 - EBPF and XDP - Eric Leblond](https://kernel-recipes.org/en/2017/talks/ebpf-and-xdp/)

Overview of how Suricata uses BPF Nice to see the version number for the various features / work mentioned. The slides also has some code.

###### [Kernel Recipes 2017 - Performance Analysis with BPF - Brendan Gregg](https://kernel-recipes.org/en/2017/talks/performance-analysis-with-bpf/)

Introduction to BPF focusing on BCC and tracing.

###### Slides

###### [eBPF cgroup filters for data usage accounting on Android](https://www.linuxplumbersconf.org/2017/ocw//system/presentations/4791/original/eBPF%20cgroup%20filters%20for%20data%20usage%20accounting%20on%20Android.pdf)

Interesting work on how to use BPF for wifi data usage.

###### [Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF](https://www.slideshare.net/techcet/cilium-kernel-native-security-ddos-mitigation-for-microservices-with-bpf)

High level introduction to Cilium with some new ongoing work on Cilium. If you are using / interested in Kafka, worth checking out for the API filtering.

###### [eBPF Implementation for FreeBSD](https://docs.google.com/presentation/d/1jtqjq1x3XUOSmtBrEvZnHRgcJT6ldFmcFfdChaic2VA/edit#slide=id.p)

Interesting slides on eBPF port to FreeBSD.

##### In case you missed it

###### [Heap Allocation Flamegraphs](https://epickrram.blogspot.no/2017/09/heap-allocation-flamegraphs.html)

Post on using one of the grave tools for tracing JVM processes.

###### [Linux の BPF: (5) eBPF による Linux Kernel Tracing](http://mmi.hatenablog.com/entry/2017/09/04/231922)

Going through a tracing example using kernel 4.12.

###### [An intro to using eBPF to filter packets in the Linux kernel](https://opensource.com/article/17/9/intro-ebpf)

Beginner friendly post with a accompanying example for tracing.

###### [IP Accounting and Access Lists with systemd](http://0pointer.net/blog/ip-accounting-and-access-lists-with-systemd.html)

Simple cgroup/BPF packet counting and blacklist via systemd.

###### [XDP on Power](https://sthbrx.github.io/blog/2017/07/17/xdp-on-power/)

XDP on the Power architecture.

###### [Cilium v0.10 & v0.11 Released: Double the Fun - Two Updates in One!](https://www.cilium.io/blog/2017/9/29/cilium-v010-v011-released-double-the-fun-two-updates-in-one)

Great post on the recent progress made in the Cilium project.

##### Projects

A couple of new GitHub projects. Please star if you like the projects;)

###### [Cilium client API example](https://github.com/cilium/client-example)

> Simple example illustrating use of the Cilium API.

###### [ebpf_asm](https://github.com/solarflarecom/ebpf_asm)

> An assembler for eBPF programs written in an Intel-like assembly syntax.

###### [go-ebpf](https://github.com/andrewkroh/go-ebpf)

> eBPF programs without a libbcc dependency

###### [generic-ebpf](https://github.com/YutaroHayakawa/generic-ebpf)

> Generic eBPF VM. Currently support FreeBSD kernel, FreeBSD userspace, Linux kernel, Linux userspace and MacOSX userspace.

##### [Random cool note](https://twitter.com/netdev01/status/916396288253378561)

> Schedule is out! Dont miss this amazing event. Come to the land of Kimchi and Palaces. Hangout with awesome geeks. [https://www.netdevconf.org/2.2/schedule.html](https://www.netdevconf.org/2.2/schedule.html)

##### Patches

- Daniel Borkmann, [[PATCH net 0/3] Fix for BPF devmap percpu allocation splat](https://patchwork.ozlabs.org/cover/827067/)
 - [[PATCH net 1/3] mm, percpu: add support for `__GFP_NOWARN` flag](https://patchwork.ozlabs.org/patch/827066/)
 - [[PATCH net 2/3] bpf: fix splat for illegal devmap percpu allocation](https://patchwork.ozlabs.org/patch/827065/)
 - [[PATCH net 3/3] bpf: do not test for `PCPU_MIN_UNIT_SIZE` before percpu allocations](https://patchwork.ozlabs.org/patch/827064/)
- Richard Weinber, [[PATCH 1/3] bpf: Don't check for current being NULL](https://patchwork.kernel.org/patch/10009545/)
 - [[PATCH 2/3] bpf: Remove dead variable](https://patchwork.kernel.org/patch/10009543/)
 - [[PATCH 3/3] bpf: Make sure that ->comm does not change under us.](https://patchwork.kernel.org/patch/10009547/)
- Tariq Toukan, [[PATCH net-next 0/3] mlx4_en XDP TX improvements](https://www.spinics.net/lists/netdev/msg459623.html)
 - [[PATCH net-next 1/3] net/mlx4_en: Replace netdev parameter with priv in XDP xmit function](https://www.spinics.net/lists/netdev/msg459622.html)
 - [[PATCH net-next 2/3] `net/mlx4_en`: Obsolete call to generic `write_desc` in XDP xmit flow](https://www.spinics.net/lists/netdev/msg459628.html)
 - [[PATCH net-next 3/3] `net/mlx4_en`: `XDP_TX`, assign constant values of TX descs on ring creaion](https://www.spinics.net/lists/netdev/msg459629.html)
- Chenbo Feng, [[PATCH net-next v6 0/5] bpf: security: New file mode and LSM hooks for eBPF object permission control](https://www.spinics.net/lists/netdev/msg460707.html)
 - [[PATCH net-next v6 1/5] bpf: Add file mode configuration into bpf maps](https://www.spinics.net/lists/netdev/msg460712.html)
 - [[PATCH net-next v6 2/5] bpf: Add tests for eBPF file mode](https://www.spinics.net/lists/netdev/msg460708.html)
 - [[PATCH net-next v6 3/5] security: bpf: Add LSM hooks for bpf object related syscall](https://www.spinics.net/lists/netdev/msg460710.html)
 - [[PATCH net-next v6 4/5] selinux: bpf: Add selinux check for eBPF syscall operations](https://www.spinics.net/lists/netdev/msg460709.html)
 - [[PATCH net-next v6 5/5] selinux: bpf: Add addtional check for bpf object file receive](https://www.spinics.net/lists/netdev/msg460711.html)
- Jakub Kicinski, [[PATCH net-next 0/4] bpf: move context info out of the verifier](https://patchwork.ozlabs.org/cover/826552/)
 - [[PATCH net-next 1/4] bpf: split verifier and program ops](https://patchwork.ozlabs.org/patch/826553/)
 - [[PATCH net-next 2/4] bpf: remove the verifier ops from program structure](https://patchwork.ozlabs.org/patch/826554/)
 - [[PATCH net-next 3/4] bpf: move knowledge about post-translation offsets out of verifier](https://patchwork.ozlabs.org/patch/826556/)
 - [[PATCH net-next 4/4] bpf: allow access to skb->len from offloads](https://patchwork.ozlabs.org/patch/826555/)
- Jakub Kicinski, [[PATCH net v2] bpf: disallow arithmetic operations on context pointer](https://www.spinics.net/lists/netdev/msg460690.html)
- Jakub Kicinski, [[PATCH net-next] tools: bpftool: use more common tag format](https://www.mail-archive.com/netdev@vger.kernel.org/msg194010.html)
- Jakub Kicinski, [[PATCH net-next 00/12] nfp: bpf: support direct packet access](https://www.spinics.net/lists/netdev/msg460001.html)
 - [[PATCH net-next 01/12] bpf: verifier: set reg_type on context accesses in second pass](https://www.spinics.net/lists/netdev/msg460013.html)
 - [[PATCH net-next 02/12] nfp: bpf: reorder arguments to `emit_ld_field_any()`](https://www.spinics.net/lists/netdev/msg460010.html)
 - [[PATCH net-next 03/12] nfp: bpf: add missing return in jne_imm optimization](https://www.spinics.net/lists/netdev/msg460003.html)
 - [[PATCH net-next 04/12] nfp: bpf: fix compare instructions](https://www.spinics.net/lists/netdev/msg460011.html)
 - [[PATCH net-next 05/12] nfp: bpf: add mov helper](https://www.spinics.net/lists/netdev/msg460012.html)
 - [[PATCH net-next 06/12] nfp: bpf: implement byte swap instruction](https://www.spinics.net/lists/netdev/msg460004.html)
 - [[PATCH net-next 07/12] nfp: bpf: support BPF offload only on little endian](https://www.spinics.net/lists/netdev/msg460002.html)
 - [[PATCH net-next 08/12] nfp: bpf: fix context accesses](https://www.spinics.net/lists/netdev/msg460005.html)
 - [[PATCH net-next 09/12] nfp: bpf: separate I/O from checks for legacy data load](https://www.spinics.net/lists/netdev/msg460009.html)
 - [[PATCH net-next 10/12] nfp: bpf: add support for direct packet access - read](https://www.spinics.net/lists/netdev/msg460008.html)
 - [[PATCH net-next 11/12] nfp: bpf: direct packet access - write](https://www.spinics.net/lists/netdev/msg460007.html)
 - [[PATCH net-next 12/12] nfp: bpf: support direct packet access in TC](https://www.spinics.net/lists/netdev/msg460006.html)
- Jesper Dangaard Brouer, [[net-next V8 PATCH 0/5] New bpf cpumap type for XDP_REDIRECT](https://www.spinics.net/lists/netdev/msg460551.html)
 - [[net-next V8 PATCH 1/5] bpf: introduce new bpf cpu map type `BPF_MAP_TYPE_CPUMAP`](https://www.spinics.net/lists/netdev/msg460552.html)
 - [[net-next V8 PATCH 2/5] bpf: XDP_REDIRECT enable use of cpumap](https://www.spinics.net/lists/netdev/msg460555.html)
 - [[net-next V8 PATCH 3/5] bpf: cpumap xdp_buff to skb conversion and allocation](https://www.spinics.net/lists/netdev/msg460553.html)
 - [[net-next V8 PATCH 4/5] bpf: cpumap add tracepoints](https://www.spinics.net/lists/netdev/msg460554.html)
 - [[net-next V8 PATCH 5/5] samples/bpf: add cpumap sample program `xdp_redirect_cpu`](https://www.spinics.net/lists/netdev/msg460556.html)
- Jakub Kicinski, [[PATCH net-next v2 0/7] bpf: get rid of global verifier state and reuse instruction printer](https://patchwork.ozlabs.org/cover/823378/)
 - [[PATCH net-next v2 1/7] selftests/bpf: add a test for verifier logs](https://patchwork.ozlabs.org/patch/823379/)
 - [[PATCH net-next v2 2/7] bpf: encapsulate verifier log state into a structure](https://patchwork.ozlabs.org/patch/823385/)
 - [[PATCH net-next v2 3/7] bpf: move global verifier log into verifier environment](https://patchwork.ozlabs.org/patch/823383/)
 - [[PATCH net-next v2 4/7] bpf: move instruction printing into a separate file](https://patchwork.ozlabs.org/patch/823382/)
 - [[PATCH net-next v2 5/7] tools: bpftool: use the kernel's instruction printer](https://patchwork.ozlabs.org/patch/823380/)
 - [[PATCH net-next v2 6/7] bpf: don't rely on the verifier lock for metadata_dst allocation](https://patchwork.ozlabs.org/patch/823381/)
 - [[PATCH net-next v2 7/7] bpf: write back the verifier log buffer as it gets filled](https://patchwork.ozlabs.org/patch/823384/)
- Jiong Wang, [[PATCH, bpf-llvm] Fix bug on silently truncating 64-bit immediate](https://lists.iovisor.org/pipermail/iovisor-dev/2017-October/001123.html)
- Richard Weinberger, [[PATCH] bpf: devmap: Check attr->max_entries more carefully](https://patchwork.kernel.org/patch/10009601/)
- Colin King, [PATCH bpf-next: bpf: remove redundant variable old_flags](https://patchwork.kernel.org/patch/9999311/)
- Steven Rostedt, [[PATCH] tracing: bpf: Hide bpf trace events when they are not used](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1511785.html)
- Jakub Kicinski, [[PATCH net-next 00/15] nfp: bpf ABIv2 and multi port](https://patchwork.ozlabs.org/cover/823052/)
 - [[PATCH net-next 01/15] nfp: output control messages to `trace_devlink_hwmsg()`](https://patchwork.ozlabs.org/patch/823063/)
 - [[PATCH net-next 02/15] nfp: bpf: lift the single-port limitation](https://patchwork.ozlabs.org/patch/823065/)
 - [[PATCH net-next 03/15] nfp: bpf: use the power of sparse to check we encode registers right](https://patchwork.ozlabs.org/patch/823061/)
 - [[PATCH net-next 04/15] nfp: bpf: move software reg helpers and cmd table out of translator](https://patchwork.ozlabs.org/patch/823068/)
 - [[PATCH net-next 05/15] nfp: bpf: encode all 64bit shifts](https://patchwork.ozlabs.org/patch/823067/)
 - [[PATCH net-next 06/15] nfp: bpf: remove register rename](https://patchwork.ozlabs.org/patch/823066/)
 - [[PATCH net-next 07/15] nfp: bpf: remove packet marking support](https://patchwork.ozlabs.org/patch/823059/)
 - [[PATCH net-next 08/15] nfp: add more white space to the instruction defines](https://patchwork.ozlabs.org/patch/823057/)
 - [[PATCH net-next 09/15] nfp: bpf: encode LMEM accesses](https://patchwork.ozlabs.org/patch/823062/)
 - [[PATCH net-next 10/15] nfp: bpf: encode extended LM pointer operands](https://patchwork.ozlabs.org/patch/823053/)
 - [[PATCH net-next 11/15] nfp: bpf: move to datapath ABI version 2](https://patchwork.ozlabs.org/patch/823060/)
 - [[PATCH net-next 12/15] nfp: bpf: calculate code store ECC](https://patchwork.ozlabs.org/patch/823058/)
 - [[PATCH net-next 13/15] nfp: bpf: pad code with valid nops](https://patchwork.ozlabs.org/patch/823054/)
 - [[PATCH net-next 14/15] nfp: bpf: byte swap the instructions](https://patchwork.ozlabs.org/patch/823056/)
 - [[PATCH net-next 15/15] nfp: bpf: pass dst register to ld_field instruction](https://patchwork.ozlabs.org/patch/823055/)
- Yonghong Song, [[llvm] r315889 - bpf: fix bug on silently truncating 64-bit immediate](http://llvm.org/viewvc/llvm-project?rev=315889&view=rev)
- Shmulik Ladkani, [[PATCH v2] netfilter: `xt_bpf`: Fix `XT_BPF_MODE_FD_PINNED` mode of '`xt_bpf_info_v1`'](https://patchwork.ozlabs.org/patch/823229/)
- Christina Jacob, [[PATCH v2] XDP Program for Ip forward](https://lkml.org/lkml/2017/10/10/88)
 - [[PATCH 1/1] xdp: Sample xdp program implementing ip forward](https://lkml.org/lkml/2017/10/10/90)
- Martin KaFai Lau, [[PATCH net-next 0/3] bpf: Misc improvements and a new usage on bpf obj name](https://patchwork.ozlabs.org/cover/822200/)
 - [[PATCH net-next 1/3] bpf: Change `bpf_obj_name_cpy()` to better ensure map's name is init by 0](https://patchwork.ozlabs.org/patch/822203/)
 - [[PATCH net-next 2/3] bpf: Use char in prog and map name](https://patchwork.ozlabs.org/patch/822201/)
 - [[PATCH net-next 3/3] bpf: Append prog->aux->name in `bpf_get_prog_name()`](https://patchwork.ozlabs.org/patch/822202/)
- Yonghong Song, [[PATCH net-next v7 0/5] bpf: add two helpers to read perf event enabled/running time](https://patchwork.ozlabs.org/cover/821914/)
 - [[PATCH net-next v7 1/5] bpf: perf event change needed for subsequent bpf helpers](https://patchwork.ozlabs.org/patch/821919/)
 - [[PATCH net-next v7 2/5] bpf: add helper `bpf_perf_event_read_value` for perf event array map](https://patchwork.ozlabs.org/patch/821916/)
 - [[PATCH net-next v7 3/5] bpf: add a test case for helper `bpf_perf_event_read_value`](https://patchwork.ozlabs.org/patch/821917/)
 - [[PATCH net-next v7 4/5] bpf: add helper `bpf_perf_prog_read_value`](https://patchwork.ozlabs.org/patch/821918/)
 - [[PATCH net-next v7 5/5] bpf: add a test case for helper `bpf_perf_prog_read_value`](https://patchwork.ozlabs.org/patch/821915/)
- Craig Gallek, [[PATCH net-next v3 0/2] libbpf: support more map options](https://patchwork.ozlabs.org/cover/821847/)
 - [[PATCH net-next v3 1/2] libbpf: parse maps sections of varying size](https://patchwork.ozlabs.org/patch/821849/)
 - [[PATCH net-next v3 2/2] libbpf: use map_flags when creating maps](https://patchwork.ozlabs.org/patch/821848/)
- Jesper Dangaard Brouer, [[net-next PATCH 0/3] Improve xdp_monitor samples/bpf](https://www.spinics.net/lists/netdev/msg458700.html)
 - [[net-next PATCH 1/3] samples/bpf: xdp_monitor first 8 bytes are not accessible by bpf](https://www.spinics.net/lists/netdev/msg458701.html)
 - [[net-next PATCH 2/3] samples/bpf: `xdp_monitor` also record `xdp_exception` tracepoint](https://www.spinics.net/lists/netdev/msg458702.html)
 - [[net-next PATCH 3/3] samples/bpf: xdp_monitor increase memory rlimit](https://www.spinics.net/lists/netdev/msg458703.html)
- Alexei Starovoitov, [[PATCH net] bpf: fix liveness marking](https://patchwork.ozlabs.org/patch/822129/)
- Alexei Starovoitov, [[PATCH v2 net-next 0/8] bpf: muli prog support for cgroup-bpf](https://www.spinics.net/lists/netdev/msg457938.html)
 - [[PATCH v2 net-next 1/8] bpf: multi program support for cgroup+bpf](https://www.spinics.net/lists/netdev/msg457941.html)
 - [[PATCH v2 net-next 2/8] bpf: introduce `BPF_PROG_QUERY` command](https://www.spinics.net/lists/netdev/msg457940.html)
 - [[PATCH v2 net-next 3/8] bpf: enforce return code for cgroup-bpf programs](https://www.spinics.net/lists/netdev/msg457942.html)
 - [[PATCH v2 net-next 4/8] libbpf: introduce `bpf_prog_detach2()`](https://www.spinics.net/lists/netdev/msg457939.html)
 - [[PATCH v2 net-next 5/8] samples/bpf: add multi-prog cgroup test case](https://www.spinics.net/lists/netdev/msg457944.html)
 - [[PATCH v2 net-next 6/8] libbpf: sync bpf.h](https://www.spinics.net/lists/netdev/msg457945.html)
 - [[PATCH v2 net-next 7/8] libbpf: add support for `BPF_PROG_QUERY`](https://www.spinics.net/lists/netdev/msg457943.html)
 - [[PATCH v2 net-next 8/8] samples/bpf: use `bpf_prog_query()` interface](https://www.spinics.net/lists/netdev/msg457983.html)
- Yonghong Song, [[llvm] r314911 - bpf: fix an insn encoding issue for neg insn](http://llvm.org/viewvc/llvm-project?rev=314911&view=rev)
- Alexei Starovoitov, [[PATCH net] bpf: fix `bpf_tail_call()` x64 JIT](http://patchwork.ozlabs.org/patch/821051/)
- Jakub Kicinski, [[PATCH net-next v4 0/3] tools: add bpftool](https://www.spinics.net/lists/netdev/msg458469.html)
 - [[PATCH net-next v4 1/3] tools: rename tools/net directory to tools/bpf](https://www.spinics.net/lists/netdev/msg458470.html)
 - [[PATCH net-next v4 2/3] tools: bpf: add bpftool](https://www.spinics.net/lists/netdev/msg458472.html)
 - [[PATCH net-next v4 3/3] tools: bpftool: add documentation](https://www.spinics.net/lists/netdev/msg458471.html)
- Eric Dumazet, [[PATCH net] socket, bpf: fix possible use after free](https://marc.info/?l=linux-netdev&m=150697205813377&w=2)
- Stephen Hemminger, [[PATCH net-next] samples/bpf: fix warnings in `xdp_monitor_user`](https://www.spinics.net/lists/netdev/msg457697.html)
- Martin KaFai Lau, [[PATCH net-next] bpf: Fix compiler warning on info.map_ids for 32bit platform](https://patchwork.ozlabs.org/patch/820040/)
- Yonghong Song, [[llvm] r314469 - bpf: fix a bug for disassembling ld_pseudo inst](http://llvm.org/viewvc/llvm-project?rev=314469&view=rev)
- Paul Chaignon, [[PATCH RFC v3] bpf: allow map helpers access to map values directly](https://lists.iovisor.org/pipermail/iovisor-dev/2017-September/001105.html)
- Mark Rutland, [EBPF-triggered WARNING at mm/percpu.c:1361 in v4-14-rc2](https://patchwork.kernel.org/patch/9975851/)
- Yonghong Song, [[llvm] r314376 - bpf: add new insns for `bswap_to_le` and negation](http://llvm.org/viewvc/llvm-project?rev=314376&view=rev)
- Martin KaFai Lau, [[PATCH net-next 0/5] bpf: Extend `bpf_{prog,map}_info`](https://patchwork.ozlabs.org/cover/819319/)
 - [[PATCH net-next 1/5] bpf: Add name, `load_time`, uid and `map_ids` to `bpf_prog_info`](https://patchwork.ozlabs.org/patch/819321/)
 - [[PATCH net-next 2/5] bpf: Add `map_name` to `bpf_map_info`](https://patchwork.ozlabs.org/patch/819322/)
 - [[PATCH net-next 3/5] bpf: libbpf: Provide basic API support to specify BPF obj name](https://patchwork.ozlabs.org/patch/819323/)
 - [[PATCH net-next 4/5] bpf: Swap the order of checking `prog_info` and `map_info`](https://patchwork.ozlabs.org/patch/819318/)
 - [[PATCH net-next 5/5] bpf: Test new fields in `bpf_attr` and `bpf_{prog,map}_info`](https://patchwork.ozlabs.org/patch/819320/)
- Matt Redfearn, [[PATCH] MIPS: bpf: Fix uninitialised target compiler error](https://lkml.org/lkml/2017/9/27/109)
- Edward Cree, [[PATCH v2 net-next 0/2] bpf/verifier: disassembly improvements](https://www.spinics.net/lists/netdev/msg456853.html)
 - [[PATCH v2 net-next 1/2] bpf/verifier: improve disassembly of `BPF_END` instructions](https://www.spinics.net/lists/netdev/msg456848.html)
 - [[PATCH v2 net-next 2/2] bpf/verifier: improve disassembly of `BPF_NEG` instructions](https://www.spinics.net/lists/netdev/msg456851.html)
- Daniel Borkmann, [[PATCH net-next 0/6] BPF metadata for direct access](https://www.spinics.net/lists/netdev/msg456521.html)
 - [[PATCH net-next 1/6] bpf: rename `bpf_compute_data_end` into `bpf_compute_data_pointers`](https://www.spinics.net/lists/netdev/msg456522.html)
 - [[PATCH net-next 2/6] bpf: add meta pointer for direct access](https://www.spinics.net/lists/netdev/msg456525.html)
 - [[PATCH net-next 3/6] bpf: update bpf.h uapi header for tools](https://www.spinics.net/lists/netdev/msg456526.html)
 - [[PATCH net-next 4/6] bpf: improve selftests and add tests for meta pointer](https://www.spinics.net/lists/netdev/msg456524.html)
 - [[PATCH net-next 5/6] bpf, nfp: add meta data support](https://www.spinics.net/lists/netdev/msg456523.html)
 - [[PATCH net-next 6/6] bpf, ixgbe: add meta data support](https://www.spinics.net/lists/netdev/msg456527.html)
- Jason Wang, [[PATCH net-next] virtio-net: correctly set xdp_xmit for mergeable buffer](https://patchwork.kernel.org/patch/9965257/)
- Yonghong Song, [[PATCH] bpf: add support for neg insn and change format of bswap insn](https://www.mail-archive.com/netdev@vger.kernel.org/msg189757/0001-bpf-add-support-for-neg-insn-and-change-format-of-bs.patch)
- Edward Cree, [[PATCH net-next] bpf/verifier: improve disassembly of BPF_END instructions](https://www.spinics.net/lists/netdev/msg456041.html)
- Cong Wang, [[Patch net-next v2] net_sched: use idr to allocate bpf filter handles](https://www.spinics.net/lists/netdev/msg456677.html)
- Jiong Wang, [[llvm] r313958 - bpf: refactor inst patterns with more mnemonics](http://llvm.org/viewvc/llvm-project?rev=313958&view=rev)
- Jiong Wang, [[llvm] r313959 - bpf: refactor inst patterns with better inheritance](http://llvm.org/viewvc/llvm-project?rev=313959&view=rev)
- Jiong Wang, [[llvm] r313960 - bpf: add 32bit register set](http://llvm.org/viewvc/llvm-project?rev=313960&view=rev)
- Jiong Wang, [[llvm] r313961 - bpf: initial 32-bit ALU encoding support in assembler](http://llvm.org/viewvc/llvm-project?rev=313961&view=rev)
- Craig Gallek, [[PATCH net-next v2] bpf: Optimize lpm trie delete](https://patchwork.ozlabs.org/patch/817197/)
- Joel Fernandes, [[PATCH v4 0/4] Add cross-compilation support to eBPF samples](https://www.spinics.net/lists/netdev/msg455739.html)
 - [[PATCH v4 1/4] samples/bpf: Use getppid instead of getpgrp for array map stress](https://www.spinics.net/lists/netdev/msg455745.html)
 - [[PATCH v4 2/4] samples/bpf: Enable cross compiler support](https://www.spinics.net/lists/netdev/msg455744.html)
 - [[PATCH v4 3/4] samples/bpf: Fix pt_regs issues when cross-compiling](https://www.spinics.net/lists/netdev/msg455743.html)
 - [[PATCH v4 4/4] samples/bpf: Add documentation on cross compilation](https://www.spinics.net/lists/netdev/msg455742.html)
- Daniel Borkmann, [[PATCH iproute2 master 0/2] BPF/XDP json follow-up](https://www.spinics.net/lists/netdev/msg455965.html)
 - [[PATCH iproute2 master 1/2] json: move json printer to common library](https://www.spinics.net/lists/netdev/msg455967.html)
 - [[PATCH iproute2 master 2/2] bpf: properly output json for xdp](https://www.spinics.net/lists/netdev/msg455966.html)
- Edward Cree, [[PATCH net] net: change skb->`mac_header` when Generic XDP calls `adjust_head`](https://patchwork.ozlabs.org/patch/815745/)
- Jason Wang [[PATCH net-next 1/3] virtio-net: remove unnecessary parameter of `virtnet_xdp_xmit()`](https://www.spinics.net/lists/netdev/msg455385.html)
 - [[PATCH net-next 2/3] virtio-net: add packet len average only when needed during XDP](https://www.spinics.net/lists/netdev/msg455384.html)
 - [[PATCH net-next 3/3] virtio-net: support XDP_REDIRECT](https://www.spinics.net/lists/netdev/msg455383.html)
- Eric Dumazet, [[PATCH net] bpf: do not disable/enable BH in `bpf_map_free_id()`](https://patchwork.ozlabs.org/patch/815636/)
- Jiong Wang, [[PATCH RFC 0/4] Initial 32-bit eBPF encoding support](https://www.spinics.net/lists/xdp-newbies/msg00353.html)
 - [[PATCH RFC 1/4] Improve instruction encoding descriptions](https://www.spinics.net/lists/xdp-newbies/msg00354.html)
 - [[PATCH RFC 2/4] Improve class inheritance in instruction patterns](https://www.spinics.net/lists/xdp-newbies/msg00355.html)
 - [[PATCH RFC 4/4] Initial 32-bit ALU encoding support in assembler](https://www.spinics.net/lists/xdp-newbies/msg00356.html)
 - [[PATCH RFC 3/4] New 32-bit register set](https://www.spinics.net/lists/xdp-newbies/msg00357.html)
- Yonghong Song, [[PATCH net] bpf: one perf event close won't free bpf program attached by another perf event](https://www.spinics.net/lists/netdev/msg455274.html)
- Daniel Borkmann, [[PATCH net v2] bpf: fix ri->`map_owner` pointer on `bpf_prog_realloc`](https://www.spinics.net/lists/netdev/msg455557.html)
- Yonghong Song, [[llvm] r313593 - bpf: add inline-asm support](http://llvm.org/viewvc/llvm-project?rev=313593&view=rev)
- Craig Gallek, [[PATCH net-next 0/3] Implement delete for BPF LPM trie](https://www.spinics.net/lists/netdev/msg455193.html)
 - [[PATCH net-next 1/3] bpf: Implement `map_delete_elem` for `BPF_MAP_TYPE_LPM_TRIE`](https://www.spinics.net/lists/netdev/msg455194.html)
 - [[PATCH net-next 2/3] bpf: Add uniqueness invariant to trivial lpm test implementation](https://www.spinics.net/lists/netdev/msg455196.html)
 - [[PATCH net-next 3/3] bpf: Test deletion in `BPF_MAP_TYPE_LPM_TRIE`](https://www.spinics.net/lists/netdev/msg455195.html)
- Tobias Klauser, [[PATCH] bpf: devmap: pass on return value of `bpf_map_precharge_memlock`](https://patchwork.ozlabs.org/patch/814911/)
- Simon Dardis, [[llvm] r313281 - [bpf] Fix test to always use little endian.](http://llvm.org/viewvc/llvm-project?rev=313281&view=rev)
- Edward Cree, [[PATCH net] bpf/verifier: reject `BPF_ALU64|BPF_END`](https://patchwork.ozlabs.org/patch/814279/)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

#### Cilium v0.10 & v0.11 Released: Double the Fun - Two Updates in One!

URL: https://cilium.io/blog/2017/9/29/cilium-v010-v011-released-double-the-fun-two-updates-in-one
Date: 2017-09-29
Categories: Release

Over the last 2 months, we have released two new versions of Cilium:

- [v0.10](#cilium0.10) ([Release Notes](https://github.com/cilium/cilium/releases/tag/v0.10.0))

- [v0.11](#cilium0.11) ([Release Notes](https://github.com/cilium/cilium/releases/tag/v0.11))

This is a brief recap of some of the functionality that has been added that we believe is noteworthy. A lot of additional work has gone into these releases, in particular a heavily improved CI system and a lot of bug fixes. Thanks everyone for providing very valuable feedback and bug reports! For the full list of changes, please refer to the Release Notes, linked above.

##### Cilium Release v0.10

With the release of [v0.10](https://github.com/cilium/cilium/releases/tag/v0.10.0), we have expanded our Kubernetes integration as well as included several new features around network policy and simplicity for users.

###### Network Policy

- CIDR-based network policy for ingress & egress: this allows the lock down of containers with IP-based filtering. Controlling access to and from external services/endpoints limits the ability of a compromised container to exfiltrate data. Documentation and examples can be found here: [http://docs.cilium.io/en/stable/policy/#layer-3-ip-cidr-based](http://docs.cilium.io/en/stable/policy/#layer-3-ip-cidr-based).

- We’ve expanded our Kubernetes capabilities by allowing policies to be applied and enforced between ports/pods in different namespaces. The standard Kubernetes _NetworkPolicy_ resource allows either selected pods in the same namespace or an entire different namespace to consume the pods specified in the policy. _[CiliumNetworkPolicy](http://docs.cilium.io/en/stable/policy/)_ allows a more specific policy where pod X can be consumed from pod Z in namespace Y.

- The Kubernetes _CiliumNetworkPolicy_ resource have been updated to support multiple rules in a single import (relevant for Kubernetes < 1.7). This is useful if multiple rules need to be applied in a single transaction.

###### Simplicity

- We have introduced a simplified overlay mode that uses Kubernetes node resources to automatically build a mesh of encapsulation tunnels without any further configuration required. You can find more information here: [http://docs.cilium.io/en/stable/concepts/#overlay-network-mode](http://docs.cilium.io/en/stable/concepts/#overlay-network-mode).

- Automatic NAT rule when accessing external networks. This step was required to be performed manually before and is now done automatically. This behaviour can be disabled by running the `cilium-agent` with the optional field <nobr data-preserve-html-node="true">`--masquerade=false`</nobr>.

- Support for arbitrary cluster address prefix sizes. The cluster address block is the subnet from which all network endpoints in the cluster are allocated. Previously, Cilium required fixed /8 prefix to be configured.

##### Cilium Release v0.11

In Cilium release [v0.11](https://github.com/cilium/cilium/releases/tag/v0.11), we have included support for the latest features in Kubernetes 1.7, enhanced tools for tracing and statuses, and provided L7 policy examples in our Mesos Getting Started Guide ([www.cilium.io/try-mesos](http://www.cilium.io/try-mesos)).

###### Kubernetes

- CRD Support: with the deprecation of the _ThirdPartyResource_ (TPR) in Kubernetes 1.8 and the introduction of the _CustomResourceDefinition_ (CRD), only Kubernetes 1.7.x supports TPRs and CRDs independently. Starting in Cilium v0.11, the _CiliumNetworkPolicy_ supports CRDs. Please note: parallel usage of CRDs and TPRs leads to unexpected behaviour and is not supported ([https://github.com/kubernetes/kubernetes/issues/49424](https://github.com/kubernetes/kubernetes/issues/49424)). See <nobr data-preserve-html-node="true">[cilium.link/migrate-tpr](http://cilium.link/migrate-tpr)</nobr> for more details on migrating from TPR to CRD. In order to avoid confusion and to avoid accidentally using TPR and CRD in parallel, we have limited the use of TPR to resource version _cilium.io/v1_ and CRD to resource version _cilium.io/v2_. Upgrade your _CiliumNetworkPolicy_ resources to _cilium.io/v2_ in order to use CRD. Keep them at _cilium.io/v1_ to stay on TPR (more details here: [http://cilium.readthedocs.io/en/stable/install/#migrating-cilium-tpr-to-crd](http://cilium.readthedocs.io/en/stable/install/#migrating-cilium-tpr-to-crd)).

###### Simplicity

- Enhancements have been made to Cilium policy tracing to include traces based on security identities, endpoint IDs and Kubernetes YAML resources, such as pod names.

- Furthermore, the Kubernetes _CiliumNetworkPolicy_ resource has a policy enforcement status with relevant information per node that can be viewed via `kubectl get ciliumnetworkpolicies -o json`.

##### Mesos Integration and Getting Started Guide

In Cilium v0.10, we implemented the CNI 0.2.x specification to enable Mesos integration. In Cilium v0.11, we provided L7 policy examples in our Mesos Getting Started Guide.

If you’re using Mesos and want to try out Cilium L7 policy enforcement, try our Getting Started Guide ([www.cilium.io/try-mesos](http://www.cilium.io/try-mesos)) for a quick intro to our functionality in a self-contained Mesos environment. All you need is somewhere to install a Vagrant VM and we help you get set up with the rest!

As always, we’re here to help with any questions on [Cilium Slack](https://slack.cilium.io) or file any [issues](https://github.com/cilium/cilium/issues) for the project on [github](https://github.com/cilium/cilium). You can tweet to us [@ciliumproject](https://twitter.com/ciliumproject) and follow us on Twitter for more updates. Stay tuned for upcoming blogs on XDP and our Troubleshooting series!

~ The Cilium Team

#### BPF Updates 11

URL: https://cilium.io/blog/2017/9/18/bpf-updates-11
Date: 2017-09-18
Categories: Technology

This is issue 11 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

The highlights since last time are

- New helper functions `bpf_perf_read_counter_time` and `bpf_perf_prog_read_time`.
- Initial BPF assembly support in LLVM.
- LRU map lookup improvements.

Linux 4.13 was released last week and net-next closed around the same time. The last `[GIT] Networking` pull request includes a couple of BPF fixes and so do the two after the merge window opened up as well. See the dates for all the details

- [01 September 2017](https://www.spinics.net/lists/netdev/msg453325.html).
- [05 September 2017](https://www.spinics.net/lists/netdev/msg453873.html).
- [09 September 2017](https://marc.info/?l=linux-netdev&m=150493364601151&w=2).

LLVM [5.0.0](http://lists.llvm.org/pipermail/llvm-dev/2017-September/117136.html) was released. For BPF there is some fixes like improved code generation for certain instructions which can prevent your program from being rejected. Also the latest release of iproute2 [4.13](https://lkml.org/lkml/2017/9/5/547) is worth checking out for the map in map support and the improved error messages for tailcalls.

##### Presentations

###### Slides

###### [Making the Kernel’s Networking Data Path Programmable with BPF and XDP.](http://schd.ws/hosted_files/ossna2017/da/BPFandXDP.pdf)

Great slides covering the what, why and how on BPF and XDP. Also has an interesting graph on the number of contributions to the BPF subsystem from v4.1 - v4.13+.

###### [Cilium - Network security for microservices](https://www.slideshare.net/ThomasGraf5/cilium-network-security-for-microservices)

Updated Cilium slides presented at Open Source Summit North America. Includes new performance numbers around XDP for DDoS mitigation, kernel proxy (kproxy), and socket redirect.

###### [Our Experiences Deploying Kubernetes With IPv6](https://www.slideshare.net/AndrMartins102/our-experiences-deploying-kubernetes-with-i-pv6-79744014/1)

While primarily on Kubernetes and IPv6, this shows how Cilium fits into that picture.

###### [Performance Analysis Superpowers with Linux BPF](http://schd.ws/hosted_files/ossna2017/b8/OSS2017_BPF_superpowers.pdf)

Covering the use cases for BPF with a emphasis on tracing. Several programs shown that utilize BCC. The slide on possible improvements to BCC is interesting.

###### Videos

###### [Future:Net 2017 - Layer 7 is the New Layer 4: Cilium – Layer 7 Aware Networking & Security with BPF](https://www.youtube.com/watch?v=k0KQz6JrKXc)

Great talk on the potential of BPF for application level security and more.

###### [Future:Net 2017 - The Rise of Programmable Networks](https://www.youtube.com/watch?v=1o18dGtzWBM)

Nice introduction to BPF with a focus on some of the history of IO Visor and related things.

###### [Future:Net 2017 - The Role of Hardware and IO Processors in the Ongoing Network Transformation](https://www.youtube.com/watch?v=t8iFUng2ID8)

Panel from some of the NIC vendors talking about smartNICs. BPF is mentioned as a option for the dataplane and offloading.

##### In case you missed it

###### [Terraform recipes to test Cilium on Kubernetes](http://acalustra.com/terraform-recipes-to-test-cilium-on-kubernetes.html)

Mostly related to Cilium, but there is some introductory information on BPF and XDP.

###### [eBPF, Microservices, Docker, and Cilium: From Novice to Seasoned](http://www.adelzaalouk.me/2017/security-bpf-docker-cillium/)

Introduction to BPF focusing mostly on Cilium. The post also covers parts of the bpf(2) syscall with code from the v4.11 kernel.

###### [An update on gobpf - ELF loading, uprobes, more program types](https://kinvolk.io/blog/2017/09/an-update-on-gobpf---elf-loading-uprobes-more-program-types/)

A look at the progress made in gobpf and how far it's come. Also has some kprobe examples in there.

###### [Linux の BPF: (3) eBPF の基礎](http://mmi.hatenablog.com/entry/2017/09/01/173735)

Introductory post using code from the 4.7 kernel. While some of it is dated, most of it should still apply today, for example the sample code.

###### [Linux の BPF: (4) Clang による eBPF プログラムの作成と，BPF Compiler Collection (BCC)](http://mmi.hatenablog.com/entry/2017/09/03/213612)

Nice post going through some code from the kernel, LLVM, BCC programs and the code generation.

###### [[iovisor-dev] Accessing user memory and minor page faults](https://lists.iovisor.org/pipermail/iovisor-dev/2017-August/001018.html)

Interesting post on tracing experiences.

##### [Random cool note](https://twitter.com/diptanu/status/899424568422486016)

> Mind blown by eBPF performance! A simple ingress firewall I wrote using XDP processes 11 million packets/s. Time for more optimizations!

##### Patches

- Ingo Molnar, [[PATCH] tools/include: Sync kernel ABI headers with tooling headers](https://patchwork.kernel.org/patch/9950603/)
- Yonghong Song, [[llvm] r313055 - bpf: Add BPF AsmParser support in LLVM](http://llvm.org/viewvc/llvm-project?rev=313055&view=rev)
- Yonghong Song, [[llvm] r312978 - bpf: add " ll" in the `LD_IMM64` asmstring](http://llvm.org/viewvc/llvm-project?rev=312978&view=rev)
- Jesper Dangaard Brouer, [[V3 PATCH net] xdp: implement `xdp_redirect_map` for generic XDP](https://www.mail-archive.com/netdev@vger.kernel.org/msg187772.html)
- Rafael Buchbinder, [[PATCH] extensions: `libxt_bpf`: fix missing `__NR_bpf` declaration](https://www.spinics.net/lists/netfilter-devel/msg49552.html)
- Paul Chaignon, [[iovisor-dev] [PATCH RFC v2] bpf: allow map helpers access to map values directly](https://lists.iovisor.org/pipermail/iovisor-dev/2017-September/001049.html)
- John Fastabend, [[net PATCH 0/3] Fixes for XDP/BPF](https://www.mail-archive.com/netdev@vger.kernel.org/msg187693.html)
 - [[net PATCH 1/3] net: rcu lock and preempt disable missing around generic xdp](https://www.mail-archive.com/netdev@vger.kernel.org/msg187695.html)
 - [[net PATCH 2/3] bpf: add support for sockmap detach programs](https://www.mail-archive.com/netdev@vger.kernel.org/msg187694.html)
 - [[net PATCH 3/3] bpf: devmap, use `cond_resched` instead of `cpu_relax`](https://www.mail-archive.com/netdev@vger.kernel.org/msg187696.html)
- Eric Dumazet, [[PATCH net] tcp: fix a request socket leak](https://www.spinics.net/lists/netdev/msg454379.html)
- Yonghong Song, [[llvm] r312833 - bpf: proper print imm64 expression in inst printer](http://llvm.org/viewvc/llvm-project?view=revision&revision=312833)
- Daniel Borkmann, [[PATCH net] bpf: make error reporting in `bpf_warn_invalid_xdp_action` more clear](https://patchwork.ozlabs.org/patch/811887/)
- Yonghong Song, [[llvm] r312840 - bpf: fix test failures due to previous bpf change of assembly code syntax](http://llvm.org/viewvc/llvm-project?rev=312840&view=rev)
- Thomas Meyer, [[PATCH] selftests/bpf: Make `bpf_util` work on uniprocessor systems](https://patchwork.kernel.org/patch/9943741/)
- Jesper Dangaard Brouer, [[V2 PATCH net-next 0/2] Fixes for `XDP_REDIRECT` map](https://www.mail-archive.com/netdev@vger.kernel.org/msg187410.html)
 - [[V2 PATCH net-next 1/2] xdp: implement `xdp_redirect_map` for generic XDP](https://www.mail-archive.com/netdev@vger.kernel.org/msg187411.html)
 - [[V2 PATCH net-next 2/2] xdp: catch invalid `XDP_REDIRECT` API usage](https://www.mail-archive.com/netdev@vger.kernel.org/msg187412.html)
- Daniel Borkmann, [[PATCH net] bpf: don't select potentially stale ri->map from buggy xdp progs](https://www.spinics.net/lists/netdev/msg454187.html)
- Yonghong Song, [[PATCH net] perf/bpf: fix a clang compilation issue](https://patchwork.ozlabs.org/patch/811279/)
- Jiong Wang, [[PATCH RFC] Add BPF AsmParser support in LLVM](https://www.spinics.net/lists/xdp-newbies/msg00328.html)
- Naresh Kamboju, [[PATCH v2 1/2] selftests: bpf: `test_kmod.sh`: check if module is present in the path before insert](https://patchwork.kernel.org/patch/9941749/)
 - [[PATCH v2 2/2] selftests: bpf: `test_kmod.sh`: use modprobe on target device](https://patchwork.kernel.org/patch/9941747/)
- Jason Wang, [[PATCH net-next 1/2] tun: reserve extra headroom only when XDP is set](https://patchwork.ozlabs.org/patch/809504/)
 - [[PATCH net-next 2/2] tun: rename `generic_xdp` to `skb_xdp`](https://patchwork.ozlabs.org/patch/809503/)
- Eric Dumazet, [[PATCH net-next] bpf: fix numa_node validation](https://patchwork.ozlabs.org/patch/809934/)
- Daniel Borkmann, [[PATCH iproute2 master 0/2] Two minor BPF updates](https://www.spinics.net/lists/netdev/msg453706.html)
 - [[PATCH iproute2 master 1/2] bpf: minor cleanups for `bpf_trace_pipe`](https://www.spinics.net/lists/netdev/msg453707.html)
 - [[PATCH iproute2 master 2/2] bpf: consolidate dumps to use `bpf_dump_prog_info`](https://www.spinics.net/lists/netdev/msg453708.html)
- Joel Fernandes, [[PATCH RFC v3 0/4] Add cross-compilation support to eBPF samples](https://patchwork.ozlabs.org/cover/809345/)
 - [[PATCH RFC v3 1/4] samples/bpf: Use getppid instead of getpgrp for array map stress](https://patchwork.ozlabs.org/patch/809349/)
 - [[PATCH RFC v3 2/4] samples/bpf: Enable cross compiler support](https://patchwork.ozlabs.org/patch/809347/)
 - [[PATCH RFC v3 3/4] samples/bpf: Fix `pt_regs` issues when cross-compiling](https://patchwork.ozlabs.org/patch/809348/)
 - [[PATCH RFC v3 4/4] samples/bpf: Add documentation on cross compilation](https://patchwork.ozlabs.org/patch/809346/)
- Yonghong Song, [[PATCH v2 net-next 0/4] bpf: add two helpers to read perf event enabled/running time](https://www.spinics.net/lists/netdev/msg453393.html)
 - [[PATCH v2 net-next 1/4] bpf: add helper `bpf_perf_read_counter_time` for perf event array map](https://www.spinics.net/lists/netdev/msg453390.html)
 - [[PATCH v2 net-next 2/4] bpf: add a test case to read enabled/running time for perf array](https://www.spinics.net/lists/netdev/msg453392.html)
 - [[PATCH v2 net-next 3/4] bpf: add helper `bpf_perf_prog_read_time`](https://www.spinics.net/lists/netdev/msg453391.html)
 - [[PATCH v2 net-next 4/4] bpf: add a test case for helper `bpf_perf_prog_read_time`](https://www.spinics.net/lists/netdev/msg453394.html)
- John Fastabend, [[net-next PATCH] bpf: sockmap update/simplify memory accounting scheme](https://patchwork.ozlabs.org/patch/808940/)
- Sandipan Das, [[PATCH 1/1] bpf: take advantage of `stack_depth` tracking in powerpc JIT](https://patchwork.ozlabs.org/patch/808943/)
- William Tu, [[iovisor-dev] [PATCH RFC] bpf: add connection tracking helper functions](https://lists.iovisor.org/pipermail/iovisor-dev/2017-September/001023.html)
- David Ahern, [[PATCH net-next] bpf: Collapse offset checks in `sock_filter_is_valid_access`](https://www.spinics.net/lists/netdev/msg453252.html)
- Martin KaFai La, [[PATCH net-next 0/3] bpf: Improve LRU map lookup performance](https://patchwork.ozlabs.org/cover/808532/)
 - [[PATCH net-next 1/3] bpf: Add `lru_hash_lookup` performance test](https://patchwork.ozlabs.org/patch/808535/)
 - [[PATCH net-next 2/3] bpf: Inline LRU map lookup](https://patchwork.ozlabs.org/patch/808533/)
 - [[PATCH net-next 3/3] bpf: Only set node->ref = 1 if it has not been set](https://patchwork.ozlabs.org/patch/808534/)
- David Ahern, [[PATCH v3 net-next 0/7] bpf: Add option to set mark and priority in cgroup sock programs](https://patchwork.ozlabs.org/cover/808436/)
 - [[PATCH v3 net-next 1/7] bpf: Add mark and priority to sock options that can be set](https://patchwork.ozlabs.org/patch/808437/)
 - [[PATCH v3 net-next 2/7] bpf: Allow cgroup sock filters to use `get_current_uid_gid` helper](https://patchwork.ozlabs.org/patch/808441/)
 - [[PATCH v3 net-next 3/7] samples/bpf: Update sock test to allow setting mark and priority](https://patchwork.ozlabs.org/patch/808443/)
 - [[PATCH v3 net-next 4/7] samples/bpf: Add detach option to `test_cgrp2_sock`](https://patchwork.ozlabs.org/patch/808442/)
 - [[PATCH v3 net-next 5/7] samples/bpf: Add option to dump socket settings](https://patchwork.ozlabs.org/patch/808438/)
 - [[PATCH v3 net-next 6/7] samples/bpf: Update cgrp2 socket tests](https://patchwork.ozlabs.org/patch/808440/)
 - [[PATCH v3 net-next 7/7] samples/bpf: Update cgroup socket examples to use uid gid helper](https://patchwork.ozlabs.org/patch/808439/)
- Chenbo Feng, [[PATCH 0/3] Security: add lsm hooks for checking permissions on eBPF objects](https://www.spinics.net/lists/netdev/msg453071.html)
 - [[PATCH 1/3] security: bpf: Add eBPF LSM hooks to security module](https://www.spinics.net/lists/netdev/msg453072.html)
 - [[PATCH 2/3] security: bpf: Add eBPF LSM hooks and security field to eBPF map](https://www.spinics.net/lists/netdev/msg453073.html)
 - [[PATCH 3/3] selinux: bpf: Implement the selinux checks for eBPF object](https://www.spinics.net/lists/netdev/msg453074.html)
- Eric Dumazet, [[PATCH net-next] x86: `bpf_jit`: small optimization in `emit_bpf_tail_call()`](https://www.spinics.net/lists/netdev/msg452960.html)
- Tariq Toukan, [[PATCH net-next] samples/bpf: Fix compilation issue in redirect dummy program](https://patchwork.ozlabs.org/patch/808209/)
- Colin Ian King, [PATCH net-next V3 bpf: `test_maps`: fix typos, "conenct" and "listeen"](https://patchwork.kernel.org/patch/9930285/)
- Phil Sutter, [[iproute PATCH] lib/bpf: Fix bytecode-file parsing](https://patchwork.ozlabs.org/patch/807148/)
- Jesper Dangaard Brouer, [[PATCH net-next 0/7] XDP redirect tracepoints](https://patchwork.ozlabs.org/cover/807122/)
 - [[PATCH net-next 1/7] xdp: remove redundant argument to `trace_xdp_redirect`](https://patchwork.ozlabs.org/patch/807123/)
 - [[PATCH net-next 2/7] xdp: tracepoint `xdp_redirect` also need a map argument](https://patchwork.ozlabs.org/patch/807124/)
 - [[PATCH net-next 3/7] xdp: make xdp tracepoints report bpf prog id instead of `prog_tag`](https://patchwork.ozlabs.org/patch/807125/)
 - [[PATCH net-next 4/7] xdp: separate `xdp_redirect` tracepoint in error case](https://patchwork.ozlabs.org/patch/807126/)
 - [[PATCH net-next 5/7] xdp: separate `xdp_redirect` tracepoint in map case](https://patchwork.ozlabs.org/patch/807127/)
 - [[PATCH net-next 6/7] samples/bpf: `xdp_redirect` load XDP dummy prog on TX device](https://patchwork.ozlabs.org/patch/807128/)
 - [[PATCH net-next 7/7] samples/bpf: `xdp_monitor` tool based on tracepoints](https://patchwork.ozlabs.org/patch/807129/)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!;)

#### BPF updates 10

URL: https://cilium.io/blog/2017/8/29/bpf-updates-10
Date: 2017-08-29
Categories: Technology

This is issue 10 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

The highlights since last time are

- A new iteration of the Landlock unprivileged sandbox series.
- A new iteration of the socket redirect series.
- ARM eBPF JIT got finally [merged](https://www.spinics.net/lists/netdev/msg451025.html).
- Bug fixes and tests.

Now that there is 32bit eBPF JIT support for ARM, will more embedded devices start running eBPF? [Marvell routers](https://www.mail-archive.com/netdev@vger.kernel.org/msg169582.html), wifi devices soon?:) Also worth checking out the Landlock documentation, which is really nice, both rendered and the code comments.

Some interesting topics from the lists

- [XDP redirect measurements, gotchas and tracepoints](https://www.spinics.net/lists/xdp-newbies/msg00269.html)
- [Permissions for eBPF objects](https://www.mail-archive.com/netdev@vger.kernel.org/msg185150.html)
- [modifying packets in XDP](https://www.spinics.net/lists/xdp-newbies/msg00284.html)
- [What library to use?](https://www.spinics.net/lists/xdp-newbies/msg00266.html)

##### Presentations

###### Slides

###### [Past, Present And Future Of High Speed Packet Filtering On Linux](https://cdn.shopify.com/s/files/1/0177/9886/files/phv2017-gbertin.pdf)

Great slides on DDoS mitigation approaches with code examples for XDP.

###### Videos

###### [FRNOG 28 - Quentin Monnet (6Wind): Introduction à eBPF](http://www.dailymotion.com/video/x5x9pxh)

Short introductory talk to eBPF.

###### [USENIX ATC '17: Performance Superpowers with Enhanced BPF](https://www.youtube.com/watch?feature=em-subs_digest&v=oc9000dM9-k&app=desktop)

Entertaining talk introducing BPF and highlighting some of the challenges. One of the more interesting ideas mentioned is the need for a higher level language. This seems like great a opportunity for someone to create a new language that fits the mindset of the users better.

##### In case you missed it

###### [Cilium 0.10.1 release](https://github.com/cilium/cilium/releases/tag/0.10.1)

The 0.10.1 release had a couple nice updates like a more compact monitor output, policy revision number, CIDR based filter, and much more. See the [release notes](https://github.com/cilium/cilium/blob/0.10.1/NEWS.rst) for all the details.

###### [sched-time.py and generalizing workloads](https://josefbacik.github.io/kernel/scheduler/bcc/bpf/2017/08/03/sched-time.html)

Debugging post using BCC.

###### [Parse `BPF_ARRAY` macro in bcc](http://nanxiao.me/en/parse-bpf-array-macro-in-bcc/)

Closer look at one of the maps macros in BCC.

###### [Linux Load Averages: Solving the Mystery](http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)

Mostly about load averages, but there is one or two mentions of eBPF in there.

###### [Netdev 2.2 Registration](https://www.netdevconf.org/2.2/registration.html)

The early bird registration is still open til September, but why wait? There are already some interesting sessions scheduled like [XDP for the Rest of Us](https://www.netdevconf.org/2.2/session.html?gospodarek-xdp-workshop) and [XDP + Netem = XNetem](https://www.netdevconf.org/2.2/session.html?hemminger-XNetem-talk).

###### [XDP Newbies...](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

> Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### Projects

While grav itself is not strictly BPF related, the tools in the repository utilize BCC.

###### [grav](https://github.com/epickrram/grav)

> A collection of tools to help visualise process execution.

##### [Random cool note](https://twitter.com/sargun/status/885280696348037120)

> eBPF is just a gateway drug to writing kernel code.

##### Patches

- John Fastabend, [[net-next PATCH 0/9] sockmap UAPI updates and fixes](https://www.mail-archive.com/netdev@vger.kernel.org/msg185436.html)
 - [[net-next PATCH 1/9] bpf: convert sockmap field `attach_bpf_fd2` to type](https://www.mail-archive.com/netdev@vger.kernel.org/msg185437.html)
 - [[net-next PATCH 2/9] bpf: sockmap, remove STRPARSER `map_flags` and add multi-map support](https://www.mail-archive.com/netdev@vger.kernel.org/msg185438.html)
 - [[net-next PATCH 3/9] bpf: sockmap add missing `rcu_read_(un)lock` in `smap_data_ready`](https://www.mail-archive.com/netdev@vger.kernel.org/msg185439.html)
 - [[net-next PATCH 4/9] bpf: additional sockmap self tests](https://www.mail-archive.com/netdev@vger.kernel.org/msg185440.html)
 - [[net-next PATCH 5/9] bpf: more SK_SKB selftests](https://www.mail-archive.com/netdev@vger.kernel.org/msg185441.html)
 - [[net-next PATCH 6/9] bpf: harden sockmap program attach to ensure correct map type](https://www.mail-archive.com/netdev@vger.kernel.org/msg185442.html)
 - [[net-next PATCH 7/9] bpf: sockmap indicate sock events to listeners](https://www.mail-archive.com/netdev@vger.kernel.org/msg185443.html)
 - [[net-next PATCH 8/9] bpf: sockmap requires `STREAM_PARSER` add Kconfig entry](https://www.mail-archive.com/netdev@vger.kernel.org/msg185444.html)
 - [[net-next PATCH 9/9] bpf: test_maps add sockmap stress test](https://www.mail-archive.com/netdev@vger.kernel.org/msg185445.html)
- Eric Biggers, [strparser: initialize all callbacks](https://www.spinics.net/lists/netdev/msg451622.html)
- Mickaël Salaün, [[PATCH net-next v7 00/10] Landlock LSM: Toward unprivileged sandboxing](https://www.spinics.net/lists/linux-api/msg23437.html)
 - [[PATCH net-next v7 01/10] selftest: Enhance kselftest_harness.h with a step mechanism](https://www.spinics.net/lists/linux-api/msg23431.html)
 - [[PATCH net-next v7 02/10] bpf: Add eBPF program subtype and `is_valid_subtype()` verifier](https://www.spinics.net/lists/linux-api/msg23440.html)
 - [[PATCH net-next v7 03/10] bpf,landlock: Define an eBPF program type for a Landlock rule](https://www.spinics.net/lists/linux-api/msg23433.html)
 - [[PATCH net-next v7 04/10] bpf: Define `handle_fs` and add a new helper `bpf_handle_fs_get_mode()`](https://www.spinics.net/lists/linux-api/msg23435.html)
 - [[PATCH net-next v7 05/10] landlock: Add LSM hooks related to filesystem](https://www.spinics.net/lists/linux-api/msg23439.html)
 - [[PATCH net-next v7 06/10] seccomp,landlock: Handle Landlock events per process hierarchy](https://www.spinics.net/lists/linux-api/msg23441.html)
 - [[PATCH net-next v7 07/10] landlock: Add ptrace restrictions](https://www.spinics.net/lists/linux-api/msg23434.html)
 - [[PATCH net-next v7 08/10] bpf: Add a Landlock sandbox example](https://www.spinics.net/lists/linux-api/msg23432.html)
 - [[PATCH net-next v7 09/10] bpf,landlock: Add tests for Landlock](https://www.spinics.net/lists/linux-api/msg23438.html)
 - [[PATCH net-next v7 10/10] landlock: Add user and kernel documentation for Landlock](https://www.spinics.net/lists/linux-api/msg23436.html)
- William Tu, [[PATCH net-next 0/3] gre: add `collect_md` mode for ERSPAN tunnel](https://www.mail-archive.com/netdev@vger.kernel.org/msg185131.html)
 - [[PATCH net-next 1/3] gre: refactor the `gre_fb_xmit`](https://www.mail-archive.com/netdev@vger.kernel.org/msg185132.html)
 - [[PATCH net-next 2/3] gre: add `collect_md` mode to ERSPAN tunnel](https://www.mail-archive.com/netdev@vger.kernel.org/msg185134.html)
 - [[PATCH net-next 3/3] samples/bpf: extend `test_tunnel_bpf.sh` with ERSPAN](https://www.mail-archive.com/netdev@vger.kernel.org/msg185133.html)
- Jakub Kicinski, [[PATCH net-next] selftests/bpf: check the instruction dumps are populated](https://www.mail-archive.com/netdev@vger.kernel.org/msg185185.html)
- Dan Carpenter, [[PATCH net-next] bpf: fix oops on allocation failure](https://www.mail-archive.com/netdev@vger.kernel.org/msg185177.html)
- David Ahern, [[PATCH v2 net-next 0/8] bpf: Add option to set mark and priority in cgroup sock programs](https://www.spinics.net/lists/netdev/msg451856.html)
 - [[PATCH v2 net-next 1/8] bpf: Add support for recursively running cgroup sock filters](https://www.spinics.net/lists/netdev/msg451857.html)
 - [[PATCH v2 net-next 2/8] bpf: Add mark and priority to sock options that can be set](https://www.spinics.net/lists/netdev/msg451863.html)
 - [[PATCH v2 net-next 3/8] bpf: Allow cgroup sock filters to use `get_current_uid_gid` helper](https://www.spinics.net/lists/netdev/msg451864.html)
 - [[PATCH v2 net-next 4/8] samples/bpf: Update sock test to allow setting mark and priority](https://www.spinics.net/lists/netdev/msg451858.html)
 - [[PATCH v2 net-next 5/8] samples/bpf: Add detach option to `test_cgrp2_sock`](https://www.spinics.net/lists/netdev/msg451859.html)
 - [[PATCH v2 net-next 6/8] samples/bpf: Add option to dump socket settings](https://www.spinics.net/lists/netdev/msg451862.html)
 - [[PATCH v2 net-next 7/8] samples/bpf: Add test case for nested socket options](https://www.spinics.net/lists/netdev/msg451860.html)
 - [[PATCH v2 net-next 8/8] samples/bpf: Update cgroup socket examples to use uid gid helper](https://www.spinics.net/lists/netdev/msg451861.html)
- Edward Cree, [[PATCH v2 net-next 0/5] bpf: verifier fixes](https://www.spinics.net/lists/netdev/msg451320.html)
 - [[PATCH v2 net-next 1/5] selftests/bpf: add a test for a bug in liveness-based pruning](https://www.spinics.net/lists/netdev/msg451321.html)
 - [[PATCH v2 net-next 2/5] bpf/verifier: when pruning a branch, ignore its write marks](https://www.spinics.net/lists/netdev/msg451322.html)
 - [[PATCH v2 net-next 3/5] selftests/bpf: add a test for a pruning bug in the verifier](https://www.spinics.net/lists/netdev/msg451323.html)
 - [[PATCH v2 net-next 4/5] bpf/verifier: remove `varlen_map_value_access` flag](https://www.spinics.net/lists/netdev/msg451324.html)
 - [[PATCH v2 net-next 5/5] bpf/verifier: document liveness analysis](https://www.spinics.net/lists/netdev/msg451325.html)
- Sabrina Dubroca, [[PATCH net] tcp: fix refcnt leak with ebpf congestion control](https://patchwork.ozlabs.org/patch/805838/)
- Yuchung Cheng, [[PATCH net] bpf: fix bpf_setsockopts return value](https://www.spinics.net/lists/netdev/msg451634.html)
- Jesper Dangaard Brouer, [[V3 PATCH net-next 0/5] xdp: more work on xdp tracepoints](https://www.spinics.net/lists/netdev/msg451519.html)
 - [[V3 PATCH net-next 1/5] xdp: remove `bpf_warn_invalid_xdp_redirect`](https://www.spinics.net/lists/netdev/msg451520.html)
 - [[V3 PATCH net-next 2/5] xdp: make generic xdp redirect use tracepoint `trace_xdp_redirect`](https://www.spinics.net/lists/netdev/msg451521.html)
 - [[V3 PATCH net-next 3/5] ixgbe: use return codes from `ndo_xdp_xmit` that are distinguishable](https://www.spinics.net/lists/netdev/msg451522.html)
 - [[V3 PATCH net-next 4/5] xdp: remove `net_device` names from `xdp_redirect` tracepoint](https://www.spinics.net/lists/netdev/msg451523.html)
 - [[V3 PATCH net-next 5/5] xdp: get tracepoints `xdp_exception` and `xdp_redirect` in sync](https://www.spinics.net/lists/netdev/msg451518.html)
- Shubham Bansal, [[PATCH net-next] bpf, doc: Add arm32 as arch supporting eBPF JIT](https://www.spinics.net/lists/netdev/msg451357.html)
- Daniel Borkmann, [[PATCH net-next] bpf: netdev is never null in `__dev_map_flush`](https://www.spinics.net/lists/netdev/msg451399.html)
- Brenden Blanco, [[iovisor-dev] [RFC] pktgen: add bpf fill support](https://lists.iovisor.org/pipermail/iovisor-dev/2017-August/001001.html)
- Yonghong Song, [[llvm] r311567 - bpf: close the file descriptor after probe inside getHostCPUNameForBPF](http://llvm.org/viewvc/llvm-project?view=revision&revision=311567)
- Colin King, [PATCH net-next: MIPS, bpf: fix missing break in switch statement](https://www.spinics.net/lists/netdev/msg451169.html)
- Daniel Borkmann, [[PATCH net] bpf: fix map value attribute for hash of maps](https://www.spinics.net/lists/netdev/msg451161.html)
- Daniel Borkmann, [[PATCH net-next 0/2] Two minor BPF cleanups](https://www.spinics.net/lists/netdev/msg451175.html)
 - [[PATCH net-next 1/2] bpf: misc xdp redirect cleanups](https://www.spinics.net/lists/netdev/msg451173.html)
 - [[PATCH net-next 2/2] bpf: minor cleanups for `dev_map`](https://www.spinics.net/lists/netdev/msg451174.html)
- Yonghong Song, [http://llvm.org/viewvc/llvm-project?view=revision&revision=311522](http://llvm.org/viewvc/llvm-project?view=revision&revision=311522)
- Phil Sutter, [[iproute PATCH v2] lib/bpf: Don't leak fp in `bpf_find_mntpt()`](https://www.spinics.net/lists/netdev/msg450750.html)
- Daniel Borkmann, [[PATCH net-next] bpf: fix double free from `dev_map_notification()`](https://www.spinics.net/lists/netdev/msg450600.html)
- Daniel Borkmann, [[PATCH net] bpf, doc: also add s390x as arch to sysctl description](https://www.mail-archive.com/netdev@vger.kernel.org/msg183895.html)
- Eric Leblond, [[PATCH] tools lib bpf: improve warning](https://www.spinics.net/lists/netdev/msg450590.html)
- Daniel Borkmann [[PATCH net-next v2 0/2] BPF inline improvements](https://www.spinics.net/lists/netdev/msg450470.html)
 - [[PATCH net-next v2 1/2] bpf: make htab inlining more robust wrt assumptions](https://www.spinics.net/lists/netdev/msg450469.html)
 - [[PATCH net-next v2 2/2] bpf: inline map in map lookup functions for array and htab](https://www.spinics.net/lists/netdev/msg450471.html)
- Martin KaFai La, [[PATCH net-next 0/2] bpf: Allow selecting numa node during map creation](https://www.spinics.net/lists/netdev/msg450365.html)
 - [[PATCH net-next 1/2] bpf: Allow selecting numa node during map creation](https://www.spinics.net/lists/netdev/msg450364.html)
 - [[PATCH net-next 2/2] bpf: Allow numa selection in `INNER_LRU_HASH_PREALLOC` test of `map_perf_test`](https://www.spinics.net/lists/netdev/msg450363.html)
- Shubham Bansal, [[PATCH net-next v4] arm: eBPF JIT compiler](https://www.spinics.net/lists/arm-kernel/msg602416.html)
- David Daney, [[PATCH 0/3] MIPS,bpf: Improvements for MIPS eBPF JIT](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1473318.html)
 - [[PATCH 1/3] MIPS,bpf: Fix using `smp_processor_id()` in preemptible splat.](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1473319.html)
 - [[PATCH 2/3] MIPS,bpf: Implement JLT, JLE, JSLT and JSLE ops in the eBPF JIT.](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1473320.html)
 - [[PATCH 3/3] MIPS,bpf: Cache value of BPF_OP(insn->code) in eBPF JIT.](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1473317.html)
- Daniel Borkmann, [[PATCH net] bpf, doc: improve sysctl knob description](https://www.spinics.net/lists/netdev/msg450308.html)
- Dan Carpenter, [[PATCH net-next] bpf: fix a return in `sockmap_get_from_fd()`](https://patchwork.ozlabs.org/patch/803058/)
- Martin KaFai Lau, [[PATCH net-next] bpf: Fix map-in-map checking in the verifier](https://www.spinics.net/lists/netdev/msg450183.html)
- Daniel Borkmann, [[PATCH net-next 0/2] Two BPF smap related followups](https://www.spinics.net/lists/netdev/msg449999.htmlcwhttps://www.spinics.net/lists/netdev/msg449999.html)
 - [[PATCH net-next 1/2] bpf: don't enable preemption twice in `smap_do_verdict`](https://www.spinics.net/lists/netdev/msg449997.html)
 - [[PATCH net-next 2/2] bpf: reuse tc bpf prologue for sk skb progs](https://www.spinics.net/lists/netdev/msg449998.htmlhttps://www.spinics.net/lists/netdev/msg449998.html)
- Jesper Dangaard Brouer, [[PATCH 0/2] xdp: adjust xdp redirect tracepoint](https://www.mail-archive.com/netdev@vger.kernel.org/msg183305.html)
 - [[PATCH 1/2] ixgbe: change `ndo_xdp_xmit` return code on xmit errors](https://www.mail-archive.com/netdev@vger.kernel.org/msg183306.html)
 - [[PATCH 2/2] xdp: adjust xdp redirect tracepoint to include return error code](https://www.mail-archive.com/netdev@vger.kernel.org/msg183307.html)
- Daniel Borkmann, [[PATCH net-next] bpf: no need to nullify ri->map in `xdp_do_redirect`](https://www.spinics.net/lists/netdev/msg449962.html)
- Daniel Borkmann, [[PATCH net-next] bpf: fix liveness propagation to parent in stack slots](https://www.spinics.net/lists/netdev/msg449958.html)
- John Fastabend, [[net-next PATCH] net: rcu lock and preempt disable missing around generic xdp](https://www.spinics.net/lists/netdev/msg449746.html)
- John Fastabend, [[net-next PATCH 0/2] bpf: sockmap build fixes](https://www.spinics.net/lists/netdev/msg449792.html)
 - [[net-next PATCH 1/2] bpf: sockmap state change warning fix](https://www.spinics.net/lists/netdev/msg449793.html)
 - [[net-next PATCH 2/2] bpf: `sock_map` fixes for `!CONFIG_BPF_SYSCALL` and `!STREAM_PARSER`](https://www.spinics.net/lists/netdev/msg449794.html)
- John Fastabend, [[net-next PATCH 00/10] BPF: sockmap and sk redirect support](https://www.spinics.net/lists/netdev/msg449598.html)
 - [[net-next PATCH 01/10] net: early init support for strparser](https://www.spinics.net/lists/netdev/msg449599.html)
 - [[net-next PATCH 02/10] net: add `sendmsg_locked` and `sendpage_locked` to `af_inet6`](https://www.spinics.net/lists/netdev/msg449600.html)
 - [[net-next PATCH 03/10] net: fixes for `skb_send_sock`](https://www.spinics.net/lists/netdev/msg449601.html)
 - [[net-next PATCH 04/10] bpf: introduce new program type for skbs on sockets](https://www.spinics.net/lists/netdev/msg449602.html)
 - [[net-next PATCH 05/10] bpf: export `bpf_prog_inc_not_zero`](https://www.spinics.net/lists/netdev/msg449603.html)
 - [[net-next PATCH 06/10] bpf: sockmap with sk redirect support](https://www.spinics.net/lists/netdev/msg449605.html)
 - [[net-next PATCH 07/10] bpf: add access to sock fields and pkt data from `sk_skb` programs](https://www.spinics.net/lists/netdev/msg449606.html)
 - [[net-next PATCH 08/10] bpf: sockmap sample program](https://www.spinics.net/lists/netdev/msg449607.html)
 - [[net-next PATCH 09/10] bpf: selftests: add tests for new `__sk_buff` members](https://www.spinics.net/lists/netdev/msg449608.html)
 - [[net-next PATCH 10/10] bpf: selftests add sockmap tests](https://www.spinics.net/lists/netdev/msg449609.html)
- Jason Wang, [[PATCH] tun: thread safe `tun_build_skb()`](https://patchwork.kernel.org/patch/9903915/)
- Daniel Díaz, [[PATCH] tools lib bpf: Fix double file test in Makefile](https://patchwork.kernel.org/patch/9902185/)
- Edward Cree, [[PATCH v3 net-next] bpf/verifier: track liveness for pruning](https://patchwork.kernel.org/patch/9902429/)
- Daniel Borkmann, [[PATCH net] bpf: fix `bpf_trace_printk` on 32 bit archs](https://patchwork.ozlabs.org/patch/801802/)
- Michael Ellerman, [[PATCH v2] bpf: Update sysctl documentation to list all supported architectures](https://patchwork.ozlabs.org/patch/802502/)
- Daniel Borkmann, [[net-next PATCH] bpf: devmap: remove unnecessary value size check](https://patchwork.ozlabs.org/patch/801916/)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!;)

#### BPF updates 09

URL: https://cilium.io/blog/2017/8/15/bpf-updates-09
Date: 2017-08-16
Categories: Technology

This is issue 09 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

The highlights since the previous issue

- New comparison instructions for reducing register pressure, stack usage and potentially smaller programs.
- RFC patchset for BPF socket redirect with a awesome new helper function `bpf_sk_redirect_map`.
- Verifier fixes, more tests and alignment tracking work got [merged](https://www.spinics.net/lists/netdev/msg448342.html).
- The XDP redirect series got [merged](https://www.mail-archive.com/netdev@vger.kernel.org/msg177988.html).
- XDP support for tap got [merged](https://www.spinics.net/lists/kernel/msg2579871.html)

The latest couple of iteration of the alignment tracking are really interesting. It now also comes with some documentation describing the register value tracking and the verifier pruning. Checkout the patches section for all the links.

The Linux 4.13 merge window ended several weeks ago weeks ago and net-next opened up around the same time with a brand new [status page](http://vger.kernel.org/~davem/net-next.html). No need to ask for the status anymore for people not being attentive enough.:)

Some more interesting topics

- iproute improvements to error handling reports for tail calls and support for loading map in map.
- LLVM [5.0.0-rc1](http://lists.llvm.org/pipermail/llvm-dev/2017-July/115882.html) is ready for testing.
- MIPS eBPF JIT finally merged.
- Virtio optimizations for XDP.

While a MIPS patch did get applied in June. One file got lost in transmit, but that's now corrected.

##### Presentations

###### Videos

###### [Tech Talks @ Kinvolk: Introduction to eBPF Programming by Alban Crequy](https://www.youtube.com/watch?v=CmTtl_OtYvc)

Nice introduction to eBPF with demos, code and diagrams.

###### [Jesper Dangaard Brouer - XDP eXpress Data Path](https://www.youtube.com/watch?v=fQiM3PUJCL4&feature=youtu.be)

Great technology overview of XDP.

###### [eBPF and IO Visor Project](https://www.youtube.com/watch?v=p01BsNDX-lk)

Mostly high level introductory talk.

###### Slides

###### [Landlock: programmatic access control](https://landlock-lsm.github.io/talks/2017-06-21_landlock-linuxkit-sig.pdf)

A look at Landlock and how eBPF and some other things are used to reduce security threats.

##### In case you missed it

###### [Tracing a packet journey using Linux tracepoints, perf and eBPF](https://blog.yadutaf.fr/2017/07/28/tracing-a-packet-journey-using-linux-tracepoints-perf-ebpf/)

Cool tracing tutorial with some perf examples. Also checkout the companion github repository.

###### [The anatomy of “Hello World” python program in bcc](http://nanxiao.me/en/the-anatomy-of-hello-world-python-program-in-bcc/)

Detailed walkthrough of a BCC example.

###### [XDP Newbies...](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

> Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### Projects

A new batch of random projects from Github. Check them out and Remember to give a star, if you like the project;)

###### [Tracepkt](https://github.com/yadutaf/tracepkt)

> Trace a ping packet journey across network interfaces and namespace on recent Linux. Supports IPv4 and IPv6.

###### [ocaml-bpf](https://github.com/ygrek/ocaml-bpf)

> OCaml embedded eBPF assembler.

###### [bpftrace](https://github.com/ajor/bpftrace)

> BPFtrace is a DTrace-style dynamic tracing tool for linux, based on the extended BPF capabilities available in recent Linux kernels. BPFtrace uses LLVM as a backend to compile scripts to BPF-bytecode and makes use of BCC for interacting with the Linux BPF system.

###### [tcptracer-bpf](https://github.com/weaveworks/tcptracer-bpf)

> tcptracer-bpf is an eBPF program using kprobes to trace TCP events (connect, accept, close). The eBPF program is compiled to an ELF object file.

###### [FlameGraph](https://github.com/brendangregg/FlameGraph)

> Stack trace visualizer [http://www.brendangregg.com/flamegraphs.html](http://www.brendangregg.com/flamegraphs.html)

###### [BPF userspace tool](https://github.com/Netronome/bpf-tool)

> The tool allows listing programs and maps on the system as well as simple dumping and modification of the maps.

##### [Random cool note](https://twitter.com/netdev01/status/892567359285862400)

> Netdev 2.2 Call for papers is out! [https://www.netdevconf.org/2.2/submit-proposal.html](https://www.netdevconf.org/2.2/submit-proposal.html) … Dont wait for that last minute rush!

##### Patches

- Thomas Richter, [[PATCHv3] perf bpf: Fix endianness problem when loading parameters in prologue](https://patchwork.kernel.org/patch/9901463/)
- Edward Cree, [[PATCH net-next] bpf/verifier: track liveness for pruning](https://patchwork.kernel.org/patch/9899897/)
- Eric Dumazet, [[PATCH net] tcp: fix possible deadlock in TCP stack vs BPF filter](https://patchwork.ozlabs.org/patch/801382/)
- Jason Wang [[PATCH net-next V2 0/3] XDP support for tap](https://www.spinics.net/lists/kernel/msg2578715.html)
 - [PATCH net-next V2 1/3] tap: use build_skb() for small packet](https://www.spinics.net/lists/kernel/msg2578714.html)
 - [PATCH net-next V2 2/3] net: export some generic xdp helpers](https://www.spinics.net/lists/kernel/msg2578711.html)
 - [PATCH net-next V2 3/3] tap: XDP support](https://www.spinics.net/lists/kernel/msg2578713.html)
- Daniel Borkmann, [[PATCH net 0/2] Minor fix in `bpf_convert_ctx_access`](https://www.mail-archive.com/netdev@vger.kernel.org/msg182194.html)
 - [[PATCH net 1/2] net: fix compilation when busy poll is not enabled](https://www.mail-archive.com/netdev@vger.kernel.org/msg182196.html)
 - [[PATCH net 2/2] bpf: fix two missing `target_size` settings in `bpf_convert_ctx_access`](https://www.mail-archive.com/netdev@vger.kernel.org/msg182195.html)
- Daniel Borkmann, [[PATCH] bpf: fix `bpf_trace_printk` on 32 bit](https://www.mail-archive.com/netdev@vger.kernel.org/msg182199.html)
- William Tu, [[PATCHv2 net-next] selftests: bpf: add check for ip XDP redirect](https://www.spinics.net/lists/netdev/msg448881.html)
- Wang Nan, [[PATCH] perf test llvm: Fix f_mode endianness problem](https://www.spinics.net/lists/kernel/msg2578481.html)
- Daniel Borkmann, [[PATCH iproute2 master] bpf: unbreak libelf linkage for bpf obj loader](http://www.spinics.net/lists/netdev/msg448592.html)
- Daniel Borkmann, [[PATCH net-next v2 0/9] Add BPF_J{'{'}LT,LE,SLT,SLE} instructions](https://www.spinics.net/lists/netdev/msg448612.html)
 - [[PATCH net-next v2 1/9] bpf: add BPF_J{'{'}LT,LE,SLT,SLE} instructions](https://www.spinics.net/lists/netdev/msg448610.html)
 - [[PATCH net-next v2 2/9] bpf, x86: implement jiting of BPF_J{'{'}LT,LE,SLT,SLE}](https://www.spinics.net/lists/netdev/msg448607.html)
 - [[PATCH net-next v2 3/9] bpf, arm64: implement jiting of BPF_J{'{'}LT,LE,SLT,SLE}](https://www.spinics.net/lists/netdev/msg448614.html)
 - [[PATCH net-next v2 4/9] bpf, sparc64: implement jiting of BPF_J{'{'}LT, LE, SLT, SLE}](https://www.spinics.net/lists/netdev/msg448613.html)
 - [[PATCH net-next v2 5/9] bpf, s390x: implement jiting of BPF_J{'{'}LT,LE,SLT,SLE}](https://www.spinics.net/lists/netdev/msg448611.html)
 - [[PATCH net-next v2 6/9] bpf, ppc64: implement jiting of BPF_J{'{'}LT,LE,SLT,SLE}](https://www.spinics.net/lists/netdev/msg448605.html)
 - [[PATCH net-next v2 7/9] bpf, nfp: implement jiting of BPF_J{'{'}LT,LE}](https://www.spinics.net/lists/netdev/msg448608.html)
 - [[PATCH net-next v2 8/9] bpf: enable BPF_J{'{'}LT,LE,SLT,SLE} opcodes in verifier](https://www.spinics.net/lists/netdev/msg448606.html)
 - [[PATCH net-next v2 9/9] bpf: add test cases for new BPF_J{'{'}LT, LE, SLT, SLE} instructions](https://www.spinics.net/lists/netdev/msg448609.html)
- Tom Herbert, [[PATCH RFC 0/2] stap: Socket tap](https://www.mail-archive.com/netdev@vger.kernel.org/msg181428.html)
 - [[PATCH RFC 1/2] bpf: Add a BPF return code to disconnect a connection](https://www.mail-archive.com/netdev@vger.kernel.org/msg181430.html)
 - [[PATCH RFC 2/2] stap: Socket tap](https://www.mail-archive.com/netdev@vger.kernel.org/msg181429.html)
- Joel Fernandes, [[PATCH RFC v2 0/5] add arm64 cross compilation support to BPF samples](https://www.spinics.net/lists/kernel/msg2574591.html)
 - [[PATCH RFC v2 1/5] samples/bpf: Use getppid instead of getpgrp for array map stress](https://www.spinics.net/lists/kernel/msg2574592.html)
 - [[PATCH RFC v2 2/5] samples/bpf: Enable cross compiler support](https://www.spinics.net/lists/kernel/msg2574599.html)
 - [[PATCH RFC v2 3/5] samples/bpf: Fix inline asm issues building samples on arm64](https://www.spinics.net/lists/kernel/msg2574601.html)
 - [[PATCH RFC v2 4/5] samples/bpf: Fix pt_regs issues when cross-compiling](https://www.spinics.net/lists/kernel/msg2574596.html)
 - [[PATCH RFC v2 5/5] samples/bpf: Add documentation on cross compilation](https://www.spinics.net/lists/kernel/msg2574595.html)
- James Hogan, [[RFC PATCH 0/2] `bpf_trace_printk()` fixes](https://www.mail-archive.com/netdev@vger.kernel.org/msg181435.html)
 - [[RFC PATCH 1/2] bpf: Fix `bpf_trace_printk` on 32-bit architectures](https://www.mail-archive.com/netdev@vger.kernel.org/msg181436.html)
 - [[RFC PATCH 2/2] bpf: Initialise mod[] in `bpf_trace_printk`](https://www.mail-archive.com/netdev@vger.kernel.org/msg181434.html)
- Edward Cree, [[PATCH v5 net-next 00/12] bpf: rewrite value tracking in verifier](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463044.html)
 - [[PATCH v5 net-next 01/12] bpf/verifier: rework value tracking](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463048.html)
 - [[PATCH v5 net-next 02/12] bpf/verifier: track signed and unsigned min/max values](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463047.html)
 - [[PATCH v5 net-next 03/12] bpf/verifier: more concise register state logs for constant var_off](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463049.html)
 - [[PATCH v5 net-next 04/12] selftests/bpf: change test_verifier expectations](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463050.html)
 - [[PATCH v5 net-next 05/12] selftests/bpf: rewrite test_align](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463051.html)
 - [[PATCH v5 net-next 06/12] selftests/bpf: add a test to test_align](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463052.html)
 - [[PATCH v5 net-next 07/12] selftests/bpf: add test for bogus operations on pointers](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463053.html)
 - [[PATCH v5 net-next 08/12] selftests/bpf: don't try to access past `MAX_PACKET_OFF` in test_verifier](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463054.html)
 - [[PATCH v5 net-next 09/12] selftests/bpf: add tests for subtraction & negative numbers](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463055.html)
 - [[PATCH v5 net-next 10/12] selftests/bpf: variable offset negative tests](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463056.html)
 - [[PATCH v5 net-next 11/12] Documentation: describe the new eBPF verifier value tracking behaviour](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463057.html)
 - [[PATCH v5 net-next 12/12] bpf/verifier: increase complexity limit to 128k](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1463059.html)
- Yonghong Song, [[PATCH net-next v4 0/2] bpf: add support for `sys_{enter|exit}_*` tracepoints](https://www.spinics.net/lists/netdev/msg447768.html)
 - [[PATCH net-next v4 1/2] bpf: add support for `sys_enter__` and `sys_exit__` tracepoints](https://www.spinics.net/lists/netdev/msg447769.html)
 - [[PATCH net-next v4 2/2] bpf: add a test case for `syscalls/sys_{enter|exit}_*` tracepoints](https://www.spinics.net/lists/netdev/msg447770.html)
- Mickaël Salaün, [[PATCH net-next v2 1/2] bpf: Move `check_uarg_tail_zero()` upward](https://www.spinics.net/lists/netdev/msg448057.html)
- Mickaël Salaün, [PATCH net-next v2 2/2] bpf: Extend `check_uarg_tail_zero()` checks](https://www.spinics.net/lists/netdev/msg448056.html)
- William Tu, [[PATCH net-next] selftests: bpf: add a test for XDP redirect](https://www.spinics.net/lists/netdev/msg448116.html)
- John Fastabend, [[net-next PATCH v2] bpf: devmap fix mutex in rcu critical section](https://patchwork.ozlabs.org/patch/798146/)
- John Fastabend, [[RFC PATCH 0/6] BPF socket redirect](https://www.spinics.net/lists/netdev/msg447591.html)
 - [[RFC PATCH 1/6] net: early init support for strparser](https://www.spinics.net/lists/netdev/msg447592.html)
 - [[RFC PATCH 2/6] net: add `sendmsg_locked` and `sendpage_locked` to `af_inet6`](https://www.spinics.net/lists/netdev/msg447593.html)
 - [[RFC PATCH 3/6] net: fixes for `skb_send_sock`](https://www.spinics.net/lists/netdev/msg447594.html)
 - [[RFC PATCH 4/6] net: sockmap with sk redirect support](https://www.spinics.net/lists/netdev/msg447595.html)
 - [[RFC PATCH 5/6] net: bpf, add skb to sk lookup routines](https://www.spinics.net/lists/netdev/msg447596.html)
 - [[RFC PATCH 6/6] net: sockmap sample program](https://www.spinics.net/lists/netdev/msg447597.html)
- Daniel Borkmann, [[PATCH net] bpf: fix byte order test in test_verifier](https://www.mail-archive.com/netdev@vger.kernel.org/msg181028.html)
- Daniel Borkmann, [[PATCH net 0/2] Two BPF fixes for s390](https://www.spinics.net/lists/netdev/msg447642.html)
 - [[PATCH net 1/2] bpf, s390: fix jit branch offset related to ldimm64](https://www.spinics.net/lists/netdev/msg447641.html)
 - [[PATCH net 2/2] bpf, s390: fix build for libbpf and selftest suite](https://www.spinics.net/lists/netdev/msg447640.html)
- David Daney, [[PATCH] MIPS: Add missing file for eBPF JIT.](https://www.spinics.net/lists/kernel/msg2572599.html)
- John Fastabend, [[net-next PATCH v2] net: comment fixes against BPF devmap helper calls](https://patchwork.ozlabs.org/patch/797861/)
- Joel Fernandes, [[PATCH] samples/bpf: Fix cross compiler error with bpf sample](https://patchwork.ozlabs.org/patch/797628/)
- Thomas Richter, [[PATCHv2] bpf: fix `selftest/bpf/test_pkt_md_access` on s390x](https://www.spinics.net/lists/kernel/msg2572763.html)
- William Tu, [[PATCH net-next] bpf: fix the printing of ifindex](https://www.spinics.net/lists/netdev/msg447217.html)
- Phil Sutter, [[iproute PATCH] bpf: Make bytecode-file reading a little more robust](https://patchwork.ozlabs.org/patch/796617/)
- William Tu, [[PATCH net] samples/bpf: fix bpf tunnel cleanup](https://py3.patchwork.dja.id.au/patch/45517/)
- Daniel Borkmann, [[PATCH net] bpf: don't indicate success when `copy_from_user` fails](https://www.spinics.net/lists/netdev/msg446225.html)
- John Fastabend, [[PATCH] bpf: testing: fix devmap tests](https://patchwork.ozlabs.org/patch/794157/)
- Luiz Augusto von Dentz, [[PATCH BlueZ] monitor: Use BPF to filter packets by index](https://www.spinics.net/lists/linux-bluetooth/msg71245.html)
- Jakub Kicinski, [[PATCH net] bpf: don't zero out the info struct in `bpf_obj_get_info_by_fd()`](https://www.spinics.net/lists/netdev/msg445868.html)
- Jakub Kicinski, [[PATCH net-next] bpf: install libbpf headers on 'make install'](https://www.mail-archive.com/netdev@vger.kernel.org/msg178899.html)
- Jakub Kicinski, [[PATCH net-next] bpf: add helper capable of reading out instructions](https://www.spinics.net/lists/netdev/msg445628.html)
- Daniel Borkmann, [[PATCH iproute2 master v2 0/2] Minor BPF updates](https://www.spinics.net/lists/netdev/msg445415.html)
 - [[PATCH iproute2 master v2 1/2] bpf: improve error reporting around tail calls](https://www.spinics.net/lists/netdev/msg445416.html)
 - [[PATCH iproute2 master v2 2/2] bpf: fix mnt path when from env](https://www.spinics.net/lists/netdev/msg445417.html)
- Edward Cree, [[PATCH net 0/2] bpf: fix verifier min/max handling in BPF_SUB](https://www.spinics.net/lists/netdev/msg445323.html)
 - [[PATCH net 1/2] selftests/bpf: subtraction bounds test](https://www.spinics.net/lists/netdev/msg445324.html)
 - [[PATCH net 2/2] bpf/verifier: fix min/max handling in `BPF_SUB`](https://www.spinics.net/lists/netdev/msg445325.html)
- Dan Carpenter, [[PATCH net-next] bpf: `dev_map_alloc()` shouldn't return NULL](https://www.spinics.net/lists/netdev/msg445394.html)
- Douglas Caetano dos Santos, [[PATCH] bpf.2: ffix](https://www.spinics.net/lists/linux-man/msg11780.html)
- Daniel Borkmann, [[PATCH net 0/5] BPF map value adjust fix](https://www.spinics.net/lists/netdev/msg445226.html)
 - [[PATCH net 1/5] bpf: fix mixed signed/unsigned derived min/max value bounds](https://www.spinics.net/lists/netdev/msg445231.html)
 - [[PATCH net 2/5] bpf: allow to specify log level and reduce it for test_verifier](https://www.spinics.net/lists/netdev/msg445229.html)
 - [[PATCH net 3/5] bpf: fix up test cases with mixed signed/unsigned bounds](https://www.spinics.net/lists/netdev/msg445227.html)
 - [[PATCH net 4/5] bpf: add test for mixed signed and unsigned bounds checks](https://www.spinics.net/lists/netdev/msg445230.html)
 - [[PATCH net 5/5] bpf: more tests for mixed signed and unsigned bounds checks](https://www.spinics.net/lists/netdev/msg445228.html)
- Jason Wang [[PATCH net-next V2 0/5] Refine virtio-net XDP](https://www.mail-archive.com/netdev@vger.kernel.org/msg178282.html)
 - [[PATCH net-next V2 1/5] virtio_ring: allow to store zero as the ctx](https://www.mail-archive.com/netdev@vger.kernel.org/msg178281.html)
 - [[PATCH net-next V2 2/5] virtio-net: pack headroom into ctx for mergeable buffers](https://www.mail-archive.com/netdev@vger.kernel.org/msg178280.html)
 - [[PATCH net-next V2 3/5] virtio-net: switch to use new ctx API for small buffer](https://www.mail-archive.com/netdev@vger.kernel.org/msg178279.html)
 - [[PATCH net-next V2 4/5] virtio-net: do not reset during XDP set](https://www.mail-archive.com/netdev@vger.kernel.org/msg178278.html)
 - [[PATCH net-next V2 5/5] virtio-net: switch off offloads on demand if possible on XDP set](https://www.mail-archive.com/netdev@vger.kernel.org/msg178277.html)
- John Fastabend, [[net-next PATCH] net: fix build error in devmap helper calls](https://www.mail-archive.com/netdev@vger.kernel.org/msg178088.html)
- John Fastabend, [[net-next PATCH 00/12] Implement XDP bpf_redirect](https://www.mail-archive.com/netdev@vger.kernel.org/msg177974.html)
 - [[net-next PATCH 01/12] ixgbe: NULL xdp_tx rings on resource cleanup](https://www.mail-archive.com/netdev@vger.kernel.org/msg177975.html)
 - [[net-next PATCH 02/12] net: xdp: support xdp generic on virtual devices](https://www.mail-archive.com/netdev@vger.kernel.org/msg177976.html)
 - [[net-next PATCH 03/12] xdp: add bpf_redirect helper function](https://www.mail-archive.com/netdev@vger.kernel.org/msg177978.html)
 - [[net-next PATCH 04/12] xdp: sample program for new bpf_redirect helper](https://www.mail-archive.com/netdev@vger.kernel.org/msg177977.html)
 - [[net-next PATCH 05/12] net: implement XDP_REDIRECT for xdp generic](https://www.mail-archive.com/netdev@vger.kernel.org/msg177980.html)
 - [[net-next PATCH 06/12] ixgbe: add initial support for xdp redirect](https://www.mail-archive.com/netdev@vger.kernel.org/msg177981.html)
 - [[net-next PATCH 07/12] xdp: add trace event for xdp redirect](https://www.mail-archive.com/netdev@vger.kernel.org/msg177979.html)
 - [[net-next PATCH 08/12] bpf: add devmap, a map for storing net device references](https://www.mail-archive.com/netdev@vger.kernel.org/msg177982.html)
 - [[net-next PATCH 09/12] bpf: add `bpf_redirect_map` helper routine](https://www.mail-archive.com/netdev@vger.kernel.org/msg177983.html)
 - [[net-next PATCH 10/12] xdp: Add batching support to redirect map](https://www.mail-archive.com/netdev@vger.kernel.org/msg177984.html)
 - [[net-next PATCH 11/12] net: add notifier hooks for devmap bpf map](https://www.mail-archive.com/netdev@vger.kernel.org/msg177986.html)
 - [[net-next PATCH 12/12] xdp: bpf redirect with map sample program](https://www.mail-archive.com/netdev@vger.kernel.org/msg177985.html)
- Andy Gospodarek, [[PATCH net-next] samples/bpf: add option for native and skb mode for redirect apps](https://www.spinics.net/lists/netdev/msg444789.html)
- Cong Wang, [[Patch net] bpf: check NULL for `sk_to_full_sk()` return value](https://www.spinics.net/lists/netdev/msg444765.html)
- Daniel Borkmann, [[PATCH iproute2 -master 0/3] BPF updates](https://www.spinics.net/lists/netdev/msg444697.html)
 - [[PATCH iproute2 -master 1/3] bpf: remove obsolete samples](https://www.spinics.net/lists/netdev/msg444695.html)
 - [[PATCH iproute2 -master 2/3] bpf: support loading map in map from obj](https://www.spinics.net/lists/netdev/msg444698.html)
 - [[PATCH iproute2 -master 3/3] bpf: dump id/jited info for cls/act programs](https://www.spinics.net/lists/netdev/msg444696.html)
- Yonghong Song, [bpf: generate better lowering code for certain select/setcc instructions](http://llvm.org/viewvc/llvm-project?rev=308080&view=rev)
- Yonghong Song, [bpf: fix a compilation bug due to unused variable for release build](http://llvm.org/viewvc/llvm-project?rev=308083&view=rev)
- Kefeng Wang, [[PATCH] bpf: fix return in `bpf_skb_adjust_net`](https://www.spinics.net/lists/netdev/msg444333.html)
- Colin King, [[PATCH] [bpf-next] bpf: add missing break in for the `TCP_BPF_SNDCWND_CLAMP` case](https://patchwork.kernel.org/patch/9825277/)
- Lawrence Brakmo, [[PATCH net-next] bpf: fix return in `load_bpf_file`](https://www.spinics.net/lists/netdev/msg443487.html)
- Yonghong Song, [[PATCH net v2] samples/bpf: fix a build issue](https://www.spinics.net/lists/netdev/msg441745.html)
- Al Viro, [[RFC] `get_compat_bpf_fprog()`: don't copyin field-by-field](https://www.spinics.net/lists/netdev/msg443938.html)
- Rafael Espindola, [Fully fix the movw/movt addend.](http://llvm.org/viewvc/llvm-project?view=revision&revision=307730)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!;)

#### BPF updates 08

URL: https://cilium.io/blog/2017/7/10/bpf-updates-08
Date: 2017-07-10
Categories: Technology

This is issue 08 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

Linux 4.12 was released and net-next is closed. The Kernel Newbies release notes is still under construction but worth checking out for the BPF commits in [4.12](https://kernelnewbies.org/Linux_4.12).

Most of the new patches from the lists should show up in the next release candidate for 4.13\. Some highlights from the recent activity are

- i40e gets XDP support for drop, pass and tx actions.
- Iterations of the alignment tracking work. The main changes; dropped RFC tag and added more tests.
- NFP flag for XDP offload mode to offer more flexibility for programs that can be offloaded.
- The new `BPF_PROG_TYPE_SOCKET_OPS` series got [merged](https://www.spinics.net/lists/netdev/msg443208.html).

More interesting topics

- iproute gets support for `IFLA_XDP_PROG_ID`. Also `cls_bpf` and `act_bpf` start using the BPF program id.
- BPF program id available for i40e via `XDP_QUERY_PROG`.
- A new function helper `bpf_skb_adjust_room` for adjusting net headroom.

One issue reoccurring is the header asm issue. While BPF can mix and match headers from kernel and userspace, the asm headers seem to be causing pain. Will one more hack be added on top of BPF, or will we see a clean / nice solution emerge from the [disccusions](https://www.mail-archive.com/netdev@vger.kernel.org/msg174021.html)?

##### Presentations

###### Videos

###### [SmartNIC Architecture, Open Programming Models Overview - DXDD Europe](https://youtu.be/Bffoywnkytc)

Netronome has support for offloading XDP programs. The talk covers the options for programming their cards, and shows how BPF fits into their architecture.

###### Slides

###### [Velocity 2017 Performance analysis superpowers with Linux eBPF](https://www.slideshare.net/brendangregg/velocity-2017-performance-analysis-superpowers-with-linux-ebpf)

The slides are mostly focusing on the BCC tool chain. Very informative diagrams on the tracing options.

###### [The BSD Packet Filter A New Architecture for User-level Packet Capture](http://step.polymtl.ca/~suchakra/PWL-Jun28-MTL.pdf)

This is walk-through of the original BPF paper. The slides are really good. This serves as a great introduction or reviewing the concepts of the old and new BPF.

##### In case you missed it

###### [Notes on BPF & eBPF](https://jvns.ca/blog/2017/06/28/notes-on-bpf---ebpf/)

The post describes the basics. Also nice to see a short explanation on control flow graph. The links to all of the example programs / code are useful for beginners.

###### [[iovisor-dev] minutes: IO Visor TSC/Dev Call](https://lists.iovisor.org/pipermail/iovisor-dev/2017-June/000847.html)

Looks like there will be more XDP patches showing up soon:)

##### Projects

Below is a list of random projects on Github. Check them out and Remember to give a star, if you like the project;)

###### [tbpoc-bpf](https://github.com/qmonnet/tbpoc-bpf)

> Stateful packet processing: two-color token-bucket PoC in BPF

###### [VALE BPF Extention Module](https://github.com/YutaroHayakawa/vale-bpf)

> Vale-bpf module is an extention of VALE software switch.
>
> This module makes VALE possible to program with eBPF.

###### [Linux Tracing Workshops Materials](https://github.com/goldshtn/linux-tracing-workshop)

> This repository contains examples and hands-on labs for various Linux tracing workshops, focusing on modern tracing tools

###### [bpf-map](https://github.com/cilium/bpf-map)

> A small tool to generically introspect BPF maps without requiring to be aware of the specific data structures stored inside. Can print the metadata of the map or its contents in hexadecimal form.

###### [ebpf-disasm](https://github.com/badboy/ebpf-disasm)

> A simple eBPF disassembler, based on rbpf.
>
> It loads the compiled eBPF code from an ELF file and prints it out.

##### Patches

- Lawrence Brakmo, [[PATCH net-next] bpf: fix to bpf_setsockops](https://www.spinics.net/lists/netdev/msg443249.html)
- Daniel Borkmann, [[PATCH net-next 0/7] Misc BPF helper/verifier improvements](https://www.spinics.net/lists/netdev/msg443210.html)
 - [[PATCH net-next 1/7] bpf, net: add `skb_mac_header_len` helper](https://www.spinics.net/lists/netdev/msg443209.html)
 - [[PATCH net-next 2/7] bpf: add `bpf_skb_adjust_room` helper](https://www.spinics.net/lists/netdev/msg443214.html)
 - [[PATCH net-next 3/7] bpf: simplify narrower ctx access](https://www.spinics.net/lists/netdev/msg443213.html)
 - [[PATCH net-next 4/7] bpf: export whether tail call has jited owner](https://www.spinics.net/lists/netdev/msg443212.html)
 - [[PATCH net-next 5/7] bpf: extend `bpf_trace_printk` to support %i](https://www.spinics.net/lists/netdev/msg443211.html)
 - [[PATCH net-next 6/7] bpf, verifier: add additional patterns to `evaluate_reg_imm_alu`](https://www.spinics.net/lists/netdev/msg443216.html)
 - [[PATCH net-next 7/7] bpf: add various test cases for verifier selftest](https://www.spinics.net/lists/netdev/msg443215.html)
- Lawrence Brakmo, [[PATCH net-next v6 00/16] bpf: Adds support for sock_ops](https://www.spinics.net/lists/netdev/msg443170.html)
 - [[PATCH net-next v6 01/16] bpf: BPF support for sock_ops](https://www.spinics.net/lists/netdev/msg443173.html)
 - [[PATCH net-next v6 02/16] bpf: program to load and attach sock_ops BPF progs](https://www.spinics.net/lists/netdev/msg443168.html)
 - [[PATCH net-next v6 03/16] bpf: Support for per connection SYN/SYN-ACK RTOs](https://www.spinics.net/lists/netdev/msg443171.html)
 - [[PATCH net-next v6 04/16] bpf: Sample bpf program to set SYN/SYN-ACK RTOs](https://www.spinics.net/lists/netdev/msg443180.html)
 - [[PATCH net-next v6 05/16] bpf: Support for setting initial receive window](https://www.spinics.net/lists/netdev/msg443179.html)
 - [[PATCH net-next v6 06/16] bpf: Sample bpf program to set initial window](https://www.spinics.net/lists/netdev/msg443165.html)
 - [[PATCH net-next v6 07/16] bpf: Add setsockopt helper function to bpf](https://www.spinics.net/lists/netdev/msg443172.html)
 - [[PATCH net-next v6 08/16] bpf: Add TCP connection BPF callbacks](https://www.spinics.net/lists/netdev/msg443164.html)
 - [[PATCH net-next v6 09/16] bpf: Sample BPF program to set buffer sizes](https://www.spinics.net/lists/netdev/msg443166.html)
 - [[PATCH net-next v6 10/16] bpf: Add support for changing congestion control](https://www.spinics.net/lists/netdev/msg443175.html)
 - [[PATCH net-next v6 11/16] bpf: Sample BPF program to set congestion control](https://www.spinics.net/lists/netdev/msg443167.html)
 - [[PATCH net-next v6 12/16] bpf: Adds support for setting initial cwnd](https://www.spinics.net/lists/netdev/msg443169.html)
 - [[PATCH net-next v6 13/16] bpf: Sample BPF program to set initial cwnd](https://www.spinics.net/lists/netdev/msg443176.html)
 - [[PATCH net-next v6 14/16] bpf: Adds support for setting sndcwnd clamp](https://www.spinics.net/lists/netdev/msg443174.html)
 - [[PATCH net-next v6 15/16] bpf: Sample bpf program to set sndcwnd clamp](https://www.spinics.net/lists/netdev/msg443177.html)
 - [[PATCH net-next v6 16/16] bpf: update tools/include/uapi/linux/bpf.h](https://www.spinics.net/lists/netdev/msg443178.html)
- Edward Cree, [[TEST PATCH] bpf/verifier: roll back ptr&const handling, and fix signed bounds](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1435355.html)
- Yonghong Song, [[llvm] r306685 - bpf: remove unnecessary truncate operation](http://llvm.org/viewvc/llvm-project?view=revision&revision=306685)
- Daniel Borkmann, [[PATCH net] bpf: prevent leaking pointer via xadd on unpriviledged](https://www.spinics.net/lists/netdev/msg442767.html)
- Edward Cree, [[iovisor-dev] [PATCH v3 net-next 00/12] bpf: rewrite value tracking in verifier](https://www.spinics.net/lists/netdev/msg442458.html)
 - [[iovisor-dev] [PATCH v3 net-next 01/12] selftests/bpf: add test for mixed signed and unsigned bounds checks](https://www.spinics.net/lists/netdev/msg442459.html)
 - [[iovisor-dev] [PATCH v3 net-next 02/12] bpf/verifier: rework value tracking](https://www.spinics.net/lists/netdev/msg442460.html)
 - [[iovisor-dev] [PATCH v3 net-next 03/12] nfp: change bpf verifier hooks to match new verifier data structures](https://www.spinics.net/lists/netdev/msg442461.html)
 - [[iovisor-dev] [PATCH v3 net-next 04/12] bpf/verifier: track signed and unsigned min/max values](https://www.spinics.net/lists/netdev/msg442462.html)
 - [[iovisor-dev] [PATCH v3 net-next 05/12] bpf/verifier: more concise register state logs for constant var_off](https://www.spinics.net/lists/netdev/msg442470.html)
 - [[iovisor-dev] [PATCH v3 net-next 06/12] selftests/bpf: change test_verifier expectations](https://www.spinics.net/lists/netdev/msg442463.html)
 - [[iovisor-dev] [PATCH v3 net-next 07/12] selftests/bpf: rewrite test_align](https://www.spinics.net/lists/netdev/msg442464.html)
 - [[iovisor-dev] [PATCH v3 net-next 08/12] selftests/bpf: add a test to test_align](https://www.spinics.net/lists/netdev/msg442465.html)
 - [[iovisor-dev] [PATCH v3 net-next 09/12] selftests/bpf: add test for bogus operations on pointers](https://www.spinics.net/lists/netdev/msg442469.html)
 - [[iovisor-dev] [PATCH v3 net-next 10/12] selftests/bpf: don't try to access past `MAX_PACKET_OFF` in `test_verifier`](https://www.spinics.net/lists/netdev/msg442466.html)
 - [[iovisor-dev] [PATCH v3 net-next 11/12] selftests/bpf: add tests for subtraction & negative numbers](https://www.spinics.net/lists/netdev/msg442467.html)
 - [[iovisor-dev] [PATCH v3 net-next 12/12] selftests/bpf: variable offset negative tests](https://www.spinics.net/lists/netdev/msg442468.html)
- Martin KaFai Lau, [[PATCH net-next] bpf: Fix out-of-bound access on interpreters[]](https://patchwork.ozlabs.org/patch/781809/)
- Jason Wang, [[PATCH net] virtio-net: unbreak cusmed packet for small buffer XDP](https://patchwork.kernel.org/patch/9813373/)
- Martin KaiFai Lau, [[PATCH net-next 0/2] bpf: Add syscall lookup support for fd array and htab](https://www.spinics.net/lists/netdev/msg442604.html)
 - [[PATCH net-next 1/2] bpf: Add syscall lookup support for fd array and htab](https://www.spinics.net/lists/netdev/msg442605.html)
 - [[PATCH net-next 2/2] bpf: Add test for syscall on fd array/htab lookup](https://www.spinics.net/lists/netdev/msg442606.html)
- Daniel Borkmann, [[PATCH iproute2] bpf: indicate lderr when `bpf_apply_relo_data` fails](https://www.spinics.net/lists/netdev/msg442370.html)
- Jakub Kicinski, [[PATCH iproute2 0/3] ip-link: XDP flags and offload mode](https://www.spinics.net/lists/netdev/msg442363.html)
 - [[PATCH iproute2 1/3] bpf: print xdp offloaded mode](https://www.spinics.net/lists/netdev/msg442364.html)
 - [[PATCH iproute2 2/3] bpf: add xdpdrv for requesting XDP driver mode](https://www.spinics.net/lists/netdev/msg442366.html)
 - [[PATCH iproute2 3/3] bpf: allow requesting XDP HW offload](https://www.spinics.net/lists/netdev/msg442365.html)
- Yonghong Song, [[PATCH net-next] bpf: possibly avoid extra masking for narrower load in verifier](https://www.spinics.net/lists/netdev/msg441963.html)
- Shubham Bansal, [[PATCH] Added Support for `BPF_CALL` | `BPF_JMP`.](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1430003.html)
- David Daney, [[PATCH RFC 0/3] bpf/arm64/mips: Avoid inline asm in BPF](https://www.mail-archive.com/netdev@vger.kernel.org/msg173949.html)
 - [[PATCH RFC 1/3] arm64: Gate inclusion of asm/sysreg.h by **EMITTING_BPF**](https://www.mail-archive.com/netdev@vger.kernel.org/msg173951.html)
 - [[PATCH RFC 2/3] samples/bpf: Add define **EMITTING_BPF** when building BPF](https://www.mail-archive.com/netdev@vger.kernel.org/msg173950.html)
 - [[PATCH RFC 3/3] MIPS: Include file changes to enable building BPF code with llvm](https://www.mail-archive.com/netdev@vger.kernel.org/msg173948.html)
- Martin KaFai La, [[PATCH v3 net-next 0/9] bpf: xdp: Report `bpf_prog` ID in `IFLA_XDP`](https://www.spinics.net/lists/netdev/msg440723.html)
 - [[PATCH v3 net-next 1/9] net: Add `IFLA_XDP_PROG_ID`](https://www.spinics.net/lists/netdev/msg440721.html)
 - [[PATCH v3 net-next 2/9] bpf: mlx4: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440722.html)
 - [[PATCH v3 net-next 3/9] bpf: mlx5e: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440725.html)
 - [[PATCH v3 net-next 4/9] bpf: `virtio_net`: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440720.html)
 - [[PATCH v3 net-next 5/9] bpf: bnxt: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440719.html)
 - [[PATCH v3 net-next 6/9] bpf: thunderx: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440724.html)
 - [[PATCH v3 net-next 7/9] bpf: ixgbe: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440717.html)
 - [[PATCH v3 net-next 8/9] bpf: nfp: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440718.html)
 - [[PATCH v3 net-next 9/9] bpf: qede: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440726.html)
- Yonghong Song, [[llvm] r305559 - bpf: set missing types in insn tablegen file](http://llvm.org/viewvc/llvm-project?rev=305559&view=rev)
- Yonghong Song, [[llvm] r305560 - bpf: avoid load from read-only sections](http://llvm.org/viewvc/llvm-project?rev=305560&view=rev)
- Yonghong Song, [[llvm] r305608 - bpf: fix a strict-aliasing issue](http://llvm.org/viewvc/llvm-project?rev=305608&view=rev)
- Jakub Kicinski, [[PATCH net-next 0/8] xdp: offload mode](https://www.spinics.net/lists/netdev/msg441794.html)
 - [[PATCH net-next 1/8] xdp: pass XDP flags into install handlers](https://www.spinics.net/lists/netdev/msg441796.html)
 - [[PATCH net-next 2/8] xdp: add HW offload mode flag for installing programs](https://www.spinics.net/lists/netdev/msg441802.html)
 - [[PATCH net-next 3/8] nfp: xdp: move driver XDP setup into a separate function](https://www.spinics.net/lists/netdev/msg441801.html)
 - [[PATCH net-next 4/8] nfp: bpf: don't offload XDP programs in DRV_MODE](https://www.spinics.net/lists/netdev/msg441800.html)
 - [[PATCH net-next 5/8] nfp: bpf: release the reference on offloaded programs](https://www.spinics.net/lists/netdev/msg441795.html)
 - [[PATCH net-next 6/8] nfp: bpf: add support for `XDP_FLAGS_HW_MODE`](https://www.spinics.net/lists/netdev/msg441799.html)
 - [[PATCH net-next 7/8] xdp: add reporting of offload mode](https://www.spinics.net/lists/netdev/msg441797.html)
 - [[PATCH net-next 8/8] nfp: xdp: report if program is offloaded](https://www.spinics.net/lists/netdev/msg441798.html)
- Wang Nan, [[PATCH] perf test llvm: Avoid error when `PROFILE_ALL_BRANCHES` is set](https://patchwork.kernel.org/patch/9795735/)
- - Jeff Kirsher, [net-next 00/15 pull request\: 40GbE Intel Wired LAN Driver Updates 2017-06-20](https://www.spinics.net/lists/netdev/msg441505.html)
 - [[net-next 01/15] i40e: add XDP support for pass and drop actions](https://www.spinics.net/lists/netdev/msg441507.html)
 - [[net-next 02/15] i40e: add support for XDP_TX action](https://www.spinics.net/lists/netdev/msg441518.html)
 - [[net-next 03/15] i40evf: assign `num_active_queues` inside `i40evf_alloc_queues`](https://www.spinics.net/lists/netdev/msg441520.html)
 - [[net-next 04/15] i40e/i40evf: update WOL and `I40E_AQC_ADDR_VALID_MASK` flags](https://www.spinics.net/lists/netdev/msg441509.html)
 - [[net-next 05/15] i40e: use `dev_dbg` instead of `dev_info` when warning about missing routine](https://www.spinics.net/lists/netdev/msg441516.html)
 - [[net-next 06/15] i40e: comment that udp_port must be in host byte order](https://www.spinics.net/lists/netdev/msg441515.html)
 - [[net-next 07/15] i40e: Fix potential out of bound array access](https://www.spinics.net/lists/netdev/msg441513.html)
 - [[net-next 08/15] i40e: Support firmware CEE DCB UP to TC map re-definition](https://www.spinics.net/lists/netdev/msg441508.html)
 - [[net-next 09/15] i40e: Add message for unsupported MFP mode](https://www.spinics.net/lists/netdev/msg441511.html)
 - [[net-next 10/15] i40e: genericize the partition bandwidth control](https://www.spinics.net/lists/netdev/msg441519.html)
 - [[net-next 11/15] i40e: Add support for OEM firmware version](https://www.spinics.net/lists/netdev/msg441506.html)
 - [[net-next 12/15] i40e: fix disabling overflow promiscuous mode](https://www.spinics.net/lists/netdev/msg441517.html)
 - [[net-next 13/15] i40e: clear only cause_ena bit](https://www.spinics.net/lists/netdev/msg441512.html)
 - [[net-next 14/15] i40e: Handle PE_CRITERR properly with IWARP enabled](https://www.spinics.net/lists/netdev/msg441514.html)
 - [[net-next 15/15] i40e: don't hold RTNL lock for the entire reset](https://www.spinics.net/lists/netdev/msg441510.html)
- Daniel Borkmann, [[PATCH net-next] bpf, i40e: Report `bpf_prog` id during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg441686.html)
- Yonghong Song, [[PATCH] samples/bpf: fix a build problem](https://patchwork.ozlabs.org/patch/779110/)
- Daniel Borkmann, [[PATCH net-next] bpf: expose prog id for `cls_bpf` and `act_bpf`](https://patchwork.ozlabs.org/patch/779055/)
- Martin KaFai Lau, [[PATCH iproute2 net-next] bpf: Add support for `IFLA_XDP_PROG_ID`](https://patchwork.ozlabs.org/patch/779164/)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!

#### Tutorial: Applying HTTP security rules with Kubernetes

URL: https://cilium.io/blog/2017/7/7/57h5usdogcy8a0daqrgfu7ikxpgzmw
Date: 2017-07-07
Categories: How-To

This blog post focuses on Layer 7 (HTTP) policy rules and how to apply them for both outgoing and incoming connections in the context of a Kubernetes cluster using a [ThirdPartyResource](https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-third-party-resource/). This is a first step in integrating L7 policies into the Kubernetes world, next steps will involve integration with [Istio](https://istio.io/) and the [Envoy proxy](https://github.com/lyft/envoy). We will talk about our plans and the details how Cilium empowers both of them in one of the next blog posts.

The Cilium [0.9 release](https://www.cilium.io/blog/2017/5/31/cilium-v09-released-hello-kubernetes) ([Release Notes](https://github.com/cilium/cilium/releases/tag/v0.9.0)) was a big step towards awesome Kubernetes integration. One of the many things that we added is a new [ThirdPartyResource](https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-third-party-resource/) named `CiliumNetworkPolicy`. The purpose of `CiliumNetworkPolicy` is to extend the standardized [`NetworkPolicy`](https://kubernetes.io/docs/concepts/services-networking/network-policies/) resource and make all of the Cilium functionality available that is not yet accessible via the standard `NetworkPolicy`.

##### Step by Step Guide

This step by step guide shows how to apply HTTP security rules in three easy steps.

###### Step1: Deploy demo app

We start out with a standard Kubernetes cluster with three worker nodes:

```
$ kubectl get nodes
NAME      STATUS    AGE
worker0   Ready     115d
worker1   Ready     115d
worker2   Ready     115d
```

Cilium is [deployed as DaemonSet](http://docs.cilium.io/en/stable/gettingstarted/#getting-started-using-kubernetes):

```
$ kubectl -n kube-system get pods
NAME                                    READY     STATUS    RESTARTS   AGE
cilium-0srz0                            1/1       Running   0          10h
cilium-153hp                            1/1       Running   0          10h
cilium-5pk5c                            1/1       Running   2          10h
cilium-consul-0kf04                     1/1       Running   1          17h
```

We deploy a simple demo application in the form of Kubernetes deployments. This will create three deployments: app1, app2, and app3'\'. It will also make app1 available via a service app1-service.

```
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/master/examples/minikube/demo.yaml
service "app1-service" created
deployment "app1" created
deployment "app2" created
deployment "app3" created
```

We can now check the status of these deployments:

```
$ kubectl get pods
    NAME                       READY     STATUS              RESTARTS   AGE
    po/app1-2741898079-66lz0   0/1       ContainerCreating   0          40s
    po/app1-2741898079-jwfmk   1/1       Running             0          40s
    po/app2-2889674625-wxs08   0/1       ContainerCreating   0          40s
    po/app3-3000954754-fbqtz   0/1       ContainerCreating   0          40s
```

###### Step 2: Create L7/HTTP security policy

We want to define a Layer7 (HTTP) policy to protect app1'\'. app1 has two API endpoints which can be called: `GET /public` and `GET /private`. We want to continue allowing `GET /public` but prohibit all calls to `GET /private`. The following policy achieves this:

```yaml
apiVersion: 'cilium.io/v1'
kind: CiliumNetworkPolicy
description: 'L7 policy for getting started using Kubernetes guide'
metadata:
  name: 'rule1'
spec:
  endpointSelector:
    matchLabels:
      id: app1
  ingress:
    - fromEndpoints:
        - matchLabels:
            id: app2
    - toPorts:
        - ports:
            - port: '80'
              protocol: TCP
          rules:
            HTTP:
              - method: 'GET'
                path: '/public'
```

We can now import this Layer 7 (HTTP) policy using `kubectl`:

```
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/master/examples/minikube/l3_l4_l7_policy.yaml
```

###### Step 3: Test the policy

`app1` is now protected. While we can still access `app1/public` from `app2`...

```
$ kubectl exec $APP2_POD -- curl -s http://$\{SVC_IP\}/public
{ 'val': 'this is public' }
```

... and we can no longer access `app1/private`.

```
$ kubectl exec $APP2_POD -- curl -s http://${SVC_IP}/private
Access denied
```

###### Next Steps

This is just a first preview into our first step to integrate HTTP layer policies into Kubernetes. We will cover more of our upcoming next steps in follow-up blog posts:

- Adding L7/HTTP security rules definitions to the Kubernetes `NetworkPolicy` to no longer require a `ThirdPartyResource` or `CustomResourceDefinition`.
- Integration with [Envoy proxy](https://github.com/lyft/envoy) to enable protocols beyond HTTP (gRPC, MongoDB,...)
- The difference between a shared proxy vs a side car proxy model and how Cilium can provide to run a hybrid model where this decision can be made per pod.
- Tight cooperation with the Envoy proxy where Cilium can share the existing context information is has, e.g. source security identity for ingress rules, existing service loadbalancing/routing decision.
- Kernel-assisted acceleration of the Envoy proxy
- Adding support for `CustomResoureDefinition` as `ThirdPartyResource` will be deprecated with Kubernetes 1.8

Stay tuned for more blog posts but feel free to ask questions or provide feedback on our journey so far.

#### BPF updates 07

URL: https://cilium.io/blog/2017/6/15/bpf-updates-07
Date: 2017-06-15
Categories: Technology

This is issue 07 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

Linux 4.12-rc5 was recently released. No BPF changes where included, but a usual `[GIT] Networking` pull request was made afterwards. You can see the changes in [15th June](https://patchwork.ozlabs.org/patch/776103/).

There were also several other patches to netdev and related lists. Most of them are fixes. The highlights are

- New program type `BPF_PROG_TYPE_SOCKET_OPS` is in the works.
- Reporting XDP program ids via netlink.
- Improvements to the bpf tests.
- MIPS eBPF JIT got [applied](https://www.spinics.net/lists/netdev/msg440291.html).

##### In case you missed it

###### [eBPF and systems performance](https://www.oreilly.com/ideas/ebpf-and-systems-performance?cmp=tw-webops-confreg-article-vlca17_gregg_article_ac)

Short post explaining what eBPF is.

###### [Build issues for samples/bpf/ after rebase or git pull (solved)](https://www.spinics.net/lists/xdp-newbies/msg00208.html)

One of the not so obvious things when building bpf programs is the headers. If your project is reusing the kernel infrastructure you will need to keep your headers synchronized. This post tells you how.

###### [[iovisor-dev] minutes: IO Visor TSC/Dev Meeting](https://lists.iovisor.org/pipermail/iovisor-dev/2017-June/000808.html)

From the meeting notes, looks like there will some interesting upcoming BPF talks in LA.

###### [XDP Newbies...](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

> Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### [Random cool note](https://twitter.com/kinvolkio/status/872057905603911680)

> Merged! @weaveworks' #WeaveScope now uses #eBPF as default for connection info: [https://github.com/weaveworks/scope/pull/2535](https://github.com/weaveworks/scope/pull/2535) …
>
> Background: [https://www.weave.works/blog/improving-performance-reliability-weave-scope-ebpf/](https://www.weave.works/blog/improving-performance-reliability-weave-scope-ebpf/) …

##### Patches

- Daniel Borkmann, [[PATCH net] bpf, tests: fix endianness selection](https://patchwork.ozlabs.org/patch/773390/)
- Martin KaFai Lau, [[PATCH net-next 1/2] bpf: Fix `test_bpf_obj_id()` when the `bpf_jit_enable` sysctl is diabled](https://www.spinics.net/lists/netdev/msg439434.html)
- Martin KaFai Lau, [[PATCH net-next 2/2] bpf: Fix `test_obj_id.c` for llvm 5.0](https://www.spinics.net/lists/netdev/msg439433.html)
- Chenbo Feng, [[PATCH net-next] bpf: Remove duplicate tcp_filter hook in ipv6](https://patchwork.ozlabs.org/patch/774126/)
- Daniel Borkmann, [[PATCH net-next 0/8] Misc BPF updates](https://www.mail-archive.com/netdev@vger.kernel.org/msg172962.html)
 - [[PATCH net-next 1/8] bpf: avoid excessive stack usage for `perf_sample_data`](https://www.mail-archive.com/netdev@vger.kernel.org/msg172960.html)
 - [[PATCH net-next 2/8] bpf: don't check spilled reg state for non-STACK_SPILLed type slots](https://www.mail-archive.com/netdev@vger.kernel.org/msg172964.html)
 - [[PATCH net-next 3/8] bpf: reset id on CONST_IMM transition](https://www.mail-archive.com/netdev@vger.kernel.org/msg172967.html)
 - [[PATCH net-next 4/8] bpf: reset id on spilled regs in `clear_all_pkt_pointers`](https://www.mail-archive.com/netdev@vger.kernel.org/msg172965.html)
 - [[PATCH net-next 5/8] bpf, tests: add a test for htab lookup + update traversal](https://www.mail-archive.com/netdev@vger.kernel.org/msg172963.html)
 - [[PATCH net-next 6/8] bpf, tests: set rlimit also for test_align, so it doesn't fail](https://www.mail-archive.com/netdev@vger.kernel.org/msg172961.html)
 - [[PATCH net-next 7/8] bpf: remove `cg_skb_func_proto` and use `sk_filter_func_proto` directly](https://www.mail-archive.com/netdev@vger.kernel.org/msg172959.html)
 - [[PATCH net-next 8/8] bpf: add `bpf_set_hash` helper for tc progs](https://www.mail-archive.com/netdev@vger.kernel.org/msg172966.html)
- Daniel Borkmann, [[PATCH net-next] bpf, arm64: take advantage of stack_depth tracking](https://www.spinics.net/lists/netdev/msg439741.html)
- Jakub Kicinski, [[PATCH net-next] xdp: add reporting of offload mode](https://www.mail-archive.com/netdev@vger.kernel.org/msg173217.html)
- Jesper Dangaard Brouer, [[PATCH net-next] selftests/bpf: make correct use of exit codes in bpf selftests](https://patchwork.ozlabs.org/patch/775159/)
- Yonghong Song, [[llvm] r305301 - bpf: clang-format on BPFAsmPrinter.cpp](http://llvm.org/viewvc/llvm-project?rev=305301&view=rev)
- Lawrence Brakmo, [[RFC PATCH net-next 00/15] bpf: Add new SOCKET_OPS program type](https://www.spinics.net/lists/netdev/msg440117.html)
 - [[RFC PATCH net-next 01/15] net: BPF support for socket ops](https://www.spinics.net/lists/netdev/msg440120.html)
 - [[RFC PATCH net-next 02/15] bpf: program to load socketops BPF programs](https://www.spinics.net/lists/netdev/msg440123.html)
 - [[RFC PATCH net-next 03/15] bpf: Support for per connection SYN/SYN-ACK RTOs](https://www.spinics.net/lists/netdev/msg440124.html)
 - [[RFC PATCH net-next 04/15] bpf: Sample bpf program to set SYN/SYN-ACK RTOs](https://www.spinics.net/lists/netdev/msg440127.html)
 - [[RFC PATCH net-next 05/15] bpf: Support for setting initial receive window](https://www.spinics.net/lists/netdev/msg440119.html)
 - [[RFC PATCH net-next 06/15] bpf: Sample bpf program to set initial window](https://www.spinics.net/lists/netdev/msg440126.html)
 - [[RFC PATCH net-next 07/15] bpf: Add setsockopt helper function to bpf](https://www.spinics.net/lists/netdev/msg440121.html)
 - [[RFC PATCH net-next 08/15] bpf: Add TCP connection BPF callbacks](https://www.spinics.net/lists/netdev/msg440131.html)
 - [[RFC PATCH net-next 09/15] bpf: Sample BPF program to set buffer sizes](https://www.spinics.net/lists/netdev/msg440133.html)
 - [[RFC PATCH net-next 10/15] bpf: Add support for changing congestion control](https://www.spinics.net/lists/netdev/msg440134.html)
 - [[RFC PATCH net-next 11/15] bpf: Sample BPF program to set congestion control](https://www.spinics.net/lists/netdev/msg440132.html)
 - [[RFC PATCH net-next 12/15] bpf: Adds support for setting initial cwnd](https://www.spinics.net/lists/netdev/msg440122.html)
 - [[RFC PATCH net-next 13/15] bpf: Sample BPF program to set initial cwnd](https://www.spinics.net/lists/netdev/msg440125.html)
 - [[RFC PATCH net-next 14/15] bpf: Adds support for setting sndcwnd clamp](https://www.spinics.net/lists/netdev/msg440130.html)
 - [[RFC PATCH net-next 15/15] bpf: Sample bpf program to set sndcwnd clamp](https://www.spinics.net/lists/netdev/msg440129.html)
- Yonghong Song, [[PATCH 0/2] bpf: permit bpf program narrower loads for ctx fields](https://www.spinics.net/lists/netdev/msg440209.html)
 - [[PATCH 1/2] bpf: permits narrower load from bpf program context fields](https://www.spinics.net/lists/netdev/msg440210.html)
 - [[PATCH 2/2] selftests/bpf: Add test cases to test narrower ctx field loads](https://www.spinics.net/lists/netdev/msg440211.html)
- David Daney, [[PATCH v2 0/5] MIPS: Implement eBPF JIT.](https://www.spinics.net/lists/netdev/msg440206.html)
 - [[PATCH v2 1/5] MIPS: Optimize uasm insn lookup.](https://www.spinics.net/lists/netdev/msg440207.html)
 - [[PATCH v2 2/5] MIPS: Correctly define DBSHFL type instruction opcodes.](https://www.spinics.net/lists/netdev/msg440205.html)
 - [[PATCH v2 3/5] MIPS: Add some instructions to uasm.](https://www.spinics.net/lists/netdev/msg440202.html)
 - [[PATCH v2 4/5] MIPS: Add support for eBPF JIT.](https://www.spinics.net/lists/netdev/msg440204.html)
 - [[PATCH v2 5/5] MIPS: Give `__secure_computing()` access to syscall arguments.](https://www.spinics.net/lists/netdev/msg440203.html)
- David Daney, [[PATCH 0/4] bpf: Changes needed (or desired) for MIPS support](https://www.spinics.net/lists/netdev/msg440232.html)
 - [[PATCH 1/4] tools: `bpf_jit_disasm`: Handle large images.](https://www.spinics.net/lists/netdev/msg440233.html)
 - [[PATCH 2/4] test_bpf: Add test to make conditional jump cross a large number of insns.](https://www.spinics.net/lists/netdev/msg440231.html)
 - [[PATCH 3/4] bpf: Add MIPS support to samples/bpf.](https://www.spinics.net/lists/netdev/msg440229.html)
 - [[PATCH 4/4] samples/bpf: Fix tracex5 to work with MIPS syscalls.](https://www.spinics.net/lists/netdev/msg440230.html)
- Martin KaFai La, [[PATCH v2 net-next 0/9] bpf: xdp: Report `bpf_prog` ID in `IFLA_XDP`](https://www.spinics.net/lists/netdev/msg440175.html)
 - [[PATCH v2 net-next 1/9] net: Add `IFLA_XDP_PROG_ID`](https://www.spinics.net/lists/netdev/msg440174.html)
 - [[PATCH v2 net-next 2/9] bpf: mlx4: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440172.html)
 - [[PATCH v2 net-next 3/9] bpf: mlx5e: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440180.html)
 - [[PATCH v2 net-next 4/9] bpf: `virtio_net`: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440178.html)
 - [[PATCH v2 net-next 5/9] bpf: bnxt: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440183.html)
 - [[PATCH v2 net-next 6/9] bpf: thunderx: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440176.html)
 - [[PATCH v2 net-next 7/9] bpf: ixgbe: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440173.html)
 - [[PATCH v2 net-next 8/9] bpf: nfp: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440181.html)
 - [[PATCH v2 net-next 9/9] bpf: qede: Report `bpf_prog` ID during `XDP_QUERY_PROG`](https://www.spinics.net/lists/netdev/msg440182.html)
- Daniel Borkmann, [[PATCH iproute2 master] bpf: provide fallback defs for `__NR_bpf` when not avail](https://www.spinics.net/lists/netdev/msg440461.html)
- Tariq Toukan, [[PATCH net-next 00/10] mlx4 XDP performance improvements](https://www.mail-archive.com/netdev@vger.kernel.org/msg173775.html)
 - [[PATCH net-next 01/10] net/mlx4_en: Remove unused argument in TX datapath function](https://www.mail-archive.com/netdev@vger.kernel.org/msg173785.html)
 - [[PATCH net-next 02/10] net/mlx4_en: Optimized single ring steering](https://www.mail-archive.com/netdev@vger.kernel.org/msg173777.html)
 - [[PATCH net-next 03/10] net/mlx4_en: Improve receive data-path](https://www.mail-archive.com/netdev@vger.kernel.org/msg173780.html)
 - [[PATCH net-next 04/10] net/mlx4_en: Improve transmit CQ polling](https://www.mail-archive.com/netdev@vger.kernel.org/msg173776.html)
 - [[PATCH net-next 05/10] net/mlx4_en: Improve stack xmit function](https://www.mail-archive.com/netdev@vger.kernel.org/msg173782.html)
 - [[PATCH net-next 06/10] net/mlx4_en: Improve XDP xmit function](https://www.mail-archive.com/netdev@vger.kernel.org/msg173781.html)
 - [[PATCH net-next 07/10] net/mlx4_en: Poll XDP TX completion queue in RX NAPI](https://www.mail-archive.com/netdev@vger.kernel.org/msg173778.html)
 - [[PATCH net-next 08/10] net/mlx4_en: Increase default TX ring size](https://www.mail-archive.com/netdev@vger.kernel.org/msg173779.html)
 - [[PATCH net-next 09/10] `net/mlx4_en`: Replace `TXBB_SIZE` multiplications with shift operations](https://www.mail-archive.com/netdev@vger.kernel.org/msg173783.html)
 - [[PATCH net-next 10/10] `net/mlx4_en`: Refactor `mlx4_en_free_tx_desc`](https://www.mail-archive.com/netdev@vger.kernel.org/msg173784.html)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!

#### BPF updates 06

URL: https://cilium.io/blog/2017/6/7/bpf-updates-06
Date: 2017-06-08
Categories: Technology

This is issue 06 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

Linux 4.12-rc4 was released this week. No new BPF changes were in this release, but several patches were applied on netdev. The highlights are

- The BPF id patches which were ready last week but had to be re-spin because of merge conflicts.
- All perf events now have BPF support.

Other interesting topics

- VF XDP support for the qede driver.
- Better alignment tracking and improvements to the verifier.

See the patches section for all the links.

##### Videos

###### [Netdev 2.1 - XDP for the Rest of Us By Andy Gospodarek + Jesper Dangaard Brouer](https://youtu.be/iBkR4gvjxtE)

Extensive walk-through of the XDP programs in the [prototype-kernel](https://github.com/netoptimizer/prototype-kernel) repository. The talk is overall great and covers several BPF and XDP concepts from the programmer perspective. Also nice to hear tips, tricks and pitfalls being covered.

##### In case you missed it

###### [Cilium v0.9 Released: Hello Kubernetes!](https://www.cilium.io/blog/2017/5/31/cilium-v09-released-hello-kubernetes)

Cilium 0.9.0 was released last week.

###### [XDP Newbies...](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

> Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### [Random cool note](https://twitter.com/GianArb/status/871808740080615424)

> 55 pages about how to improve container security. @ciliumproject #BPF, best practices, @coreos clair, #apparmor [http://scaledocker.com](http://scaledocker.com)

##### Some patches

- Craig Topper, [[llvm] r304324 - [BPF] Correct the file name of the -gen-asm-matcher output file to not start with X86.](http://llvm.org/viewvc/llvm-project?rev=304324&view=rev)
- Chenbo Feng, [[PATCH net-next v2 1/2] bpf: Allow `CGROUP_SKB` eBPF program to access `sk_buff`](https://www.spinics.net/lists/netdev/msg437923.html)
- Chenbo Feng, [[PATCH net-next v2 2/2] bpf: Remove the capability check for cgroup skb eBPF program](https://www.spinics.net/lists/netdev/msg437922.html)
- Martin KaFai Lau, [[PATCH v3 net-next 0/8] Introduce bpf ID](https://www.spinics.net/lists/netdev/msg438573.html)
 - [[PATCH v3 net-next 1/8] bpf: Introduce bpf_prog ID](https://www.spinics.net/lists/netdev/msg438579.html)
 - [[PATCH v3 net-next 2/8] bpf: Introduce bpf_map ID](https://www.spinics.net/lists/netdev/msg438577.html)
 - [[PATCH v3 net-next 3/8] bpf: Add `BPF_(PROG|MAP)_GET_NEXT_ID` command](https://www.spinics.net/lists/netdev/msg438575.html)
 - [[PATCH v3 net-next 4/8] bpf: Add `BPF_PROG_GET_FD_BY_ID`](https://www.spinics.net/lists/netdev/msg438572.html)
 - [[PATCH v3 net-next 5/8] bpf: Add `BPF_MAP_GET_FD_BY_ID`](https://www.spinics.net/lists/netdev/msg438571.html)
 - [[PATCH v3 net-next 6/8] bpf: Add `jited_len` to struct `bpf_prog`](https://www.spinics.net/lists/netdev/msg438576.html)
 - [[PATCH v3 net-next 7/8] bpf: Add `BPF_OBJ_GET_INFO_BY_FD`](https://www.spinics.net/lists/netdev/msg438578.html)
 - [[PATCH v3 net-next 8/8] bpf: Test for bpf ID](https://www.spinics.net/lists/netdev/msg438574.html)
- David Miller, [[PATCH net-next] bpf: Take advantage of stack_depth tracking in sparc64 JIT](https://www.spinics.net/lists/netdev/msg437932.html)
- Alexei Starovoitov, [[PATCH v4 net-next 0/3] bpf: Add BPF support to all perf_event](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1411973.html)
 - [[PATCH v4 net-next 1/3] perf, bpf: Add BPF support to all perf_event types](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1411972.html)
 - [[PATCH v4 net-next 2/3] samples/bpf: add tests for more perf event types](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1411978.html)
 - [[PATCH v4 net-next 3/3] bpf: update perf event helper functions documentation](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1411979.html)
- Yuval Mintz, [[PATCH net-next 00/11] qed\*: Support VF XDP attachment](https://www.mail-archive.com/netdev@vger.kernel.org/msg139507.html)
 - [[PATCH net-next 01/11] qed: Add bitmaps for VF CIDs](https://www.mail-archive.com/netdev@vger.kernel.org/msg171602.html)
 - [[PATCH net-next 02/11] qed: Create L2 queue database](https://www.mail-archive.com/netdev@vger.kernel.org/msg171608.html)
 - [[PATCH net-next 03/11] qed\*: L2 interface to use the SB structures directly](https://www.mail-archive.com/netdev@vger.kernel.org/msg171609.html)
 - [[PATCH net-next 04/11] qed: Pass vf_params when creating a queue-cid](https://www.mail-archive.com/netdev@vger.kernel.org/msg171605.html)
 - [[PATCH net-next 05/11] qed: Assign a unique per-queue index to queue-cid](https://www.mail-archive.com/netdev@vger.kernel.org/msg171613.html)
 - [[PATCH net-next 06/11] qed: Make VF legacy a bitfield](https://www.mail-archive.com/netdev@vger.kernel.org/msg171607.html)
 - [[PATCH net-next 07/11] qed: IOV db support multiple queues per qzone](https://www.mail-archive.com/netdev@vger.kernel.org/msg171610.html)
 - [[PATCH net-next 08/11] qed: Multiple qzone queues for VFs](https://www.mail-archive.com/netdev@vger.kernel.org/msg171606.html)
 - [[PATCH net-next 09/11] qed: VFs to try utilizing the doorbell bar](https://www.mail-archive.com/netdev@vger.kernel.org/msg171612.html)
 - [[PATCH net-next 10/11] qed: VF XDP support](https://www.mail-archive.com/netdev@vger.kernel.org/msg171604.html)
 - [[PATCH net-next 11/11] qede: VF XDP support](https://www.mail-archive.com/netdev@vger.kernel.org/msg171611.html)
- Daniel Borkmann, [[PATCH net-next] bpf: cgroup skb progs cannot access ld_abs/ind](https://patchwork.ozlabs.org/patch/771946/)
- Daniel Borkmann, [[PATCH net] bpf, arm64: use separate register for state in stxr](https://patchwork.ozlabs.org/patch/772377/)
- Edward Cree, [[RFC PATCH net-next 0/5] bpf: rewrite value tracking in verifier](https://www.spinics.net/lists/kernel/msg2526933.html)
 - [[RFC PATCH net-next 1/5] selftests/bpf: add test for mixed signed and unsigned bounds checks](https://www.spinics.net/lists/kernel/msg2526942.html)
 - [[RFC PATCH net-next 2/5] bpf/verifier: rework value tracking](https://www.spinics.net/lists/kernel/msg2526951.html)
 - [[RFC PATCH net-next 3/5] bpf/verifier: feed pointer-to-unknown-scalar casts into scalar ALU path](https://www.spinics.net/lists/kernel/msg2526952.html)
 - [[RFC PATCH net-next 4/5] bpf/verifier: track signed and unsigned min/max values](https://www.spinics.net/lists/kernel/msg2526945.html)
 - [[RFC PATCH net-next 5/5] selftests/bpf: change test_verifier expectations](https://www.spinics.net/lists/kernel/msg2526950.html)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

#### BPF Weekly Updates 05

URL: https://cilium.io/blog/2017/5/31/bpf-updates-05
Date: 2017-06-02
Categories: Technology

This is issue 05 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

Linux 4.12-rc3 was released last week. One usual `[GIT] Networking` pull request with some BPF fixes made it in. You can read all the highlights in [26th May](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1406659.html). Also the [release email](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1407286.html) briefly mentions BPF

> Anyway, rc3 has a little bit of everything. The biggest single change is actually just a documentation update (the intel pstate docs were converted to rst format), so the diffstat actually looks a bit odd with a wuarter just being documentation. There's also some tooling updates (perf and some bpf selftest).

More interesting highlights

- ARM 32-bit eBPF JIT saw a v2.
- MIPS eBPF JIT is in the works.
- The stack depth tracking, one step closer BPF to BPF functions.

Checkout the patches section for all the links.

##### Videos

###### [Open-NFP - Transparent eBPF Offload: Playing Nice with the Linux Kernel](https://www.youtube.com/watch?v=W2v7zgUGp8A)

Introductory webinar on eBPF and XDP. It's mostly basic, so great talk for someone new to eBPF.

##### In case you missed it

###### [Further Reading](http://docs.cilium.io/en/stable/bpf/#further-reading)

The Cilium BPF and XDP reference guide has a growing collection of links to BPF content all around the web.

##### Patches

- Daniel Borkmann, [[PATCH net v2 0/5] Various BPF fixes](https://www.mail-archive.com/netdev@vger.kernel.org/msg170113.html)
 - [[PATCH net v2 1/5] bpf: fix incorrect pruning decision when alignment must be tracked](https://www.mail-archive.com/netdev@vger.kernel.org/msg170112.html)
 - [[PATCH net v2 2/5] bpf: properly reset caller saved regs after helper call and ld_abs/ind](https://www.mail-archive.com/netdev@vger.kernel.org/msg170115.html)
 - [[PATCH net v2 3/5] bpf: add `bpf_clone_redirect` to `bpf_helper_changes_pkt_data`](https://www.mail-archive.com/netdev@vger.kernel.org/msg170114.html)
 - [[PATCH net v2 4/5] bpf: fix wrong exposure of map_flags into fdinfo for lpm](https://www.mail-archive.com/netdev@vger.kernel.org/msg170116.html)
 - [[PATCH net v2 5/5] bpf: add various verifier test cases](https://www.mail-archive.com/netdev@vger.kernel.org/msg170111.html)
- David Daney, [[PATCH] `test_bpf`: Add a couple of tests for `BPF_JSGE`](https://www.spinics.net/lists/netdev/msg436901.html)
- David Daney, [[PATCH 0/5] MIPS: Implement eBPF JIT](https://www.spinics.net/lists/kernel/msg2517660.html)
 - [[PATCH 1/5] MIPS: Optimize uasm insn lookup.](https://www.spinics.net/lists/kernel/msg2517665.html)
 - [[PATCH 2/5] MIPS: Correctly define DBSHFL type instruction opcodes.](https://www.spinics.net/lists/kernel/msg2517661.html)
 - [[PATCH 3/5] MIPS: Add some instructions to uasm.](https://www.spinics.net/lists/kernel/msg2517662.html)
 - [[PATCH 4/5] MIPS: Sort uasm enum opcode elements.](https://www.spinics.net/lists/kernel/msg2517663.html)
 - [[PATCH 5/5] MIPS: Add support for eBPF JIT.](https://www.spinics.net/lists/kernel/msg2517664.html)
- Teng Qin, [[PATCH v2 net-next 0/3] bpf: Add BPF support to all perf_event](https://www.spinics.net/lists/netdev/msg437068.html)
 - [[PATCH v2 net-next 1/3] perf, bpf: Add BPF support to all perf_event types](https://www.spinics.net/lists/kernel/msg2517762.html)
 - [[PATCH v2 net-next 2/3] samples/bpf: add samples for more perf event types](https://www.spinics.net/lists/kernel/msg2517761.html)
 - [[PATCH v2 net-next 3/3] bpf: update perf event helper functions documentatio](https://www.spinics.net/lists/kernel/msg2517763.html)
- Yonghong Song, [[llvm] r304043 - [bpf] disallow global_addr+off folding](http://llvm.org/viewvc/llvm-project?view=revision&revision=304043)
- Jesper Dangaard Brouer, [[PATCH net] samples/bpf: `bpf_load.c` order of `prog_fd[]` should correspond with ELF order](https://patchwork.ozlabs.org/patch/768589/)
- Jesper Dangaard Brouer, [[PATCH RFC] bpf: handle XDP features for bpf tail calls](https://patchwork.ozlabs.org/patch/768529/)
- Shubham Bansal, [[PATCH v2] arm: eBPF JIT compiler](https://www.spinics.net/lists/arm-kernel/msg583771.html)
- Alexei Starovoitov, [[PATCH net-next 0/9] bpf: stack depth tracking](https://www.spinics.net/lists/netdev/msg437603.html)
 - [[PATCH net-next 1/9] bpf: free up `BPF_JMP` | `BPF_CALL` | `BPF_X` opcode](https://www.spinics.net/lists/netdev/msg437607.html)
 - [[PATCH net-next 2/9] bpf: split bpf core interpreter](https://www.spinics.net/lists/netdev/msg437602.html)
 - [[PATCH net-next 3/9] bpf: teach verifier to track stack depth](https://www.spinics.net/lists/netdev/msg437605.html)
 - [[PATCH net-next 4/9] bpf: reconcile `bpf_tail_call` and stack_depth](https://www.spinics.net/lists/netdev/msg437610.html)
 - [[PATCH net-next 5/9] bpf: track stack depth of classic bpf programs](https://www.spinics.net/lists/netdev/msg437611.html)
 - [[PATCH net-next 6/9] bpf: fix `stack_depth` usage by `test_bpf.ko`](https://www.spinics.net/lists/netdev/msg437612.html)
 - [[PATCH net-next 7/9] bpf: use different interpreter depending on required stack size](https://www.spinics.net/lists/netdev/msg437613.html)
 - [[PATCH net-next 8/9] bpf: change x86 JITed program stack layout](https://www.spinics.net/lists/netdev/msg437606.html)
 - [[PATCH net-next 9/9] bpf: take advantage of stack_depth tracking in x64 JIT](https://www.spinics.net/lists/netdev/msg437609.html)
- Martin KaFai Lau, [[PATCH net-next 0/8] Introduce bpf ID](https://www.mail-archive.com/netdev@vger.kernel.org/msg170871.html)
 - [[PATCH net-next 1/8] bpf: Introduce bpf_prog ID](https://www.mail-archive.com/netdev@vger.kernel.org/msg170872.html)
 - [[PATCH net-next 2/8] bpf: Introduce bpf_map ID](https://www.mail-archive.com/netdev@vger.kernel.org/msg170873.html)
 - [[PATCH net-next 3/8] bpf: Add `BPF_(PROG|MAP)_GET_NEXT_ID` command](https://www.mail-archive.com/netdev@vger.kernel.org/msg170874.html)
 - [[PATCH net-next 4/8] bpf: Add `BPF_PROG_GET_FD_BY_ID`](https://www.mail-archive.com/netdev@vger.kernel.org/msg170870.html)
 - [[PATCH net-next 5/8] bpf: Add `BPF_MAP_GET_FD_BY_ID`](https://www.mail-archive.com/netdev@vger.kernel.org/msg170876.html)
 - [[PATCH net-next 6/8] bpf: Add `jited_len` to struct `bpf_prog`](https://www.mail-archive.com/netdev@vger.kernel.org/msg170877.html)
 - [[PATCH net-next 7/8] bpf: Add `BPF_OBJ_GET_INFO_BY_FD`](https://www.mail-archive.com/netdev@vger.kernel.org/msg170869.html)
 - [[PATCH net-next 8/8] bpf: Test for bpf ID](https://www.mail-archive.com/netdev@vger.kernel.org/msg170875.html)
- Jakub Kicinski, [[PATCH net-next 0/9] nfp: move BPF offload code into app](https://www.mail-archive.com/netdev@vger.kernel.org/msg170996.html)
 - [[PATCH net-next 1/9] sched: add helper for updating statistics on all actions](https://www.mail-archive.com/netdev@vger.kernel.org/msg170995.html)
 - [[PATCH net-next 2/9] nfp: add missing fall through statements](https://www.mail-archive.com/netdev@vger.kernel.org/msg170992.html)
 - [[PATCH net-next 3/9] nfp: turn reading PCIe RTsym parameters into a helper](https://www.mail-archive.com/netdev@vger.kernel.org/msg170994.html)
 - [[PATCH net-next 4/9] nfp: move port init to apps](https://www.mail-archive.com/netdev@vger.kernel.org/msg170993.html)
 - [[PATCH net-next 5/9] nfp: report app name in ethtool -i](https://www.mail-archive.com/netdev@vger.kernel.org/msg170987.html)
 - [[PATCH net-next 6/9] nfp: move eBPF offload files to BPF app directory](https://www.mail-archive.com/netdev@vger.kernel.org/msg170989.html)
 - [[PATCH net-next 7/9] nfp: move bpf offload code to the BPF app](https://www.mail-archive.com/netdev@vger.kernel.org/msg170988.html)
 - [[PATCH net-next 8/9] nfp: move basic eBPF stats to app-specific code](https://www.mail-archive.com/netdev@vger.kernel.org/msg170990.html)
 - [[PATCH net-next 9/9] nfp: fix memory leak on FW load error](https://www.mail-archive.com/netdev@vger.kernel.org/msg170991.html)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!

#### Cilium v0.9 Released: Kubernetes DaemonSet, NetworkPolicy, and L7 HTTP Security

URL: https://cilium.io/blog/2017/5/31/cilium-v09-released-hello-kubernetes
Date: 2017-06-01
Categories: Release

The team is excited to announce the [v0.9 release](https://github.com/cilium/cilium/releases/tag/v0.9.0) of Cilium.

We've received a lot of great feedback since we released Cilium v0.8 at the end of March with support for L7 HTTP-aware network security.   By far the biggest requests have been:

- Making it easier to deploy and use Cilium in Kubernetes environments
- Testing and hardening Cilium to enable production deployments.

While there is still more work to do, the v0.9 release of Cilium is a major leap forward on both fronts.   Most interesting to new Cilium users will be our updated [Getting Started Guide](http://docs.cilium.io/en/stable/gettingstarted/), which now includes a dead simple workflow for using Minikube + Cilium on your laptop:

The Minikube setup leverages several of the key enhancements we've made in Cilium Kubernetes integration, which are also relevant for those of you looking to deploy Cilium + Kubernetes in a larger deployment.   These include:

- A Cilium DaemonSet for Kubernetes, enabling deployment of the cilium-agent to all nodes using a single kubectl command.
- Support for advanced network policies described directly in Kubernetes YAML files and applied using kubectl.  This leverages Kubernetes 3rd-party API resources for configuring advanced Cilium capabilities including L7-layer HTTP rules, egress filtering, etc.
- Support for the latest changes the the Kubernetes NetworkPolicy semantics with regards to when to enable isolation for pods. See full discussion on GitHub [here](https://github.com/kubernetes/kubernetes/pull/39164).
- Integration with Kubernetes role-based access control (RBAC), available in Kubernetes 1.6.
- Support for Cilium logging to FluentD, a CNCF project often used in conjunction with Kubernetes.

The v0.9 release also added support for L7 filtering of IPv6 traffic, improvements to the "cilium monitor" command, and numerous fixes and improvements from our ramped up testing efforts.   For all of the gory details, check out the [release notes](https://github.com/cilium/cilium/releases/tag/v0.9.0). These improvements of course also apply equally to users of Cilium + Docker.

In the coming weeks we'll use this blog to provide deep dives into some of these key aspects of v0.9 as well as to post a guide for a large scale Kubernetes + Cilium deployment.

The v0.9 release was also milestone for the Cilium community as it passed 2,000 total commits.   In just the past ~2 months the community has added 14 contributors to bring the total to 24 and reached 600 github stars (about double what we had just three months ago). Have you [starred us](https://github.com/cilium/cilium) yet?

We hope you have fun trying out the v0.9 release.   As we shift focus toward the next release, we'd love your input on bugs or new features by adding [issues on github](https://github.com/cilium/cilium/issues), pinging us on [Cilium Slack](https://slack.cilium.io) , or tweeting us [@ciliumproject](https://twitter.com/ciliumproject).

~ The Cilium Team

#### BPF updates 04

URL: https://cilium.io/blog/2017/5/24/bpf-updates-04
Date: 2017-05-24
Categories: Technology

This is issue 04 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

Linux 4.12-rc2 was released last week. One usual `[GIT] Networking` pull request made it in which includes two BPF fixes. See [18th May](http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1399669.html) for the details.

In [last week's issue](https://www.cilium.io/blog/2017/5/17/bpf-updates-03) ARM 32-bit was mentioned. As you can see from the patches section an eBPF JIT RFC just got posted recently. Checkout the discussion on the [linux-arm-kernel](https://www.spinics.net/lists/kernel/msg2514912.html) list.

Other interesting news 0day kernel testing bot for [BPF selftests](https://www.mail-archive.com/netdev@vger.kernel.org/msg168831.html). The patches do go through extensive review, but hopefully this catches regressions if any.

##### Videos

###### [Infrastructure 2017 - Alfonso Acosta - High-performance Linux monitoring with eBPF](https://www.youtube.com/watch?v=k4jqTLtdrxQ)

Nice talk on BPF and how Weaveworks is leveraging it in scope.

###### [PyCon 2017 - Alex Gartrell - Executing python functions in the linux kernel by transpiling to bpf](https://youtu.be/CpqMroMBGP4)

Good talk on BPF with a python centric perspective. The walk-through is great and easy to follow.

##### In case you missed it

###### [BPF Verifier Overview](https://www.spinics.net/lists/xdp-newbies/msg00185.html)

Interesting read on the verifier. It explains some of the checks performed and why.

###### [Alignment in BPF verifier](https://www.mail-archive.com/netdev@vger.kernel.org/msg169180.html)

Some of this stuff is over my head, but from the reading it looks like we have a new algorithm for alignment checking. The early implementation is in [python](https://gist.github.com/ecree-solarflare/0665d5b46c2d8d08de2377fbd527de8d). While initially on a different topic, the thread on [[PATCH v2 1/3] bpf: Use 1\<\<16 data-preserve-html-node="true" as ceiling for immediate alignment in verifier.](https://www.spinics.net/lists/netdev/msg435542.html) is worth reading to get the full context.

###### [An entertaining eBPF XDP adventure](https://suchakra.wordpress.com/2017/05/23/an-entertaining-ebpf-xdp-adventure/)

Fun read on the solving a challenge. Starts out with basics and goes into code examples.

###### [XDP Newbies...](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

> Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### [Random cool note](https://twitter.com/brendangregg/status/866078955530444800)

> as I've said to a number of people job hunting in systems engineering: BPF experience is hot and getting hotter

##### Patches

- Daniel Borkmann, [[PATCH net] bpf: adjust verifier heuristics](http://www.mail-archive.com/netdev@vger.kernel.org/msg168842.html)
- Daniel Borkmann, [[PATCH net 0/3] BPF pruning follow-up](https://www.mail-archive.com/netdev@vger.kernel.org/msg169875.html)
 - [[PATCH net 1/3] bpf: fix incorrect pruning decision when alignment must be tracked](https://www.mail-archive.com/netdev@vger.kernel.org/msg169874.html)
 - [[PATCH net 2/3] bpf: properly reset caller saved regs after helper call and ld_abs/ind](https://www.mail-archive.com/netdev@vger.kernel.org/msg169872.html)
 - [[PATCH net 3/3] bpf: add various verifier test cases](https://www.mail-archive.com/netdev@vger.kernel.org/msg169873.html)
- Yonghong Song, [[PATCH net v2] selftests/bpf: fix broken build due to types.h](https://www.spinics.net/lists/netdev/msg435609.html)
- Jesper Dangaard Brouer, [[RFC net-next PATCH 0/5] XDP driver feature API and handling change to xdp_buff](https://www.spinics.net/lists/netdev/msg435772.html)
 - [[RFC net-next PATCH 1/5] samples/bpf: `xdp_tx_iptunnel` make use of map_data[]](https://www.spinics.net/lists/netdev/msg435767.html)
 - [[RFC net-next PATCH 2/5] mlx5: fix bug reading `rss_hash_type` from CQE](https://www.spinics.net/lists/netdev/msg435770.html)
 - [[RFC net-next PATCH 3/5] net: introduce XDP driver features interface](https://www.spinics.net/lists/netdev/msg435769.html)
 - [[RFC net-next PATCH 4/5] net: new XDP feature for reading HW rxhash from drivers](https://www.spinics.net/lists/netdev/msg435768.html)
 - [[RFC net-next PATCH 5/5] mlx5: add XDP rxhash feature for driver mlx5](https://www.spinics.net/lists/netdev/msg435771.html)
- Teng Qin, [[PATCH net-next 0/2] perf, bpf: add support for HW_CACHE and RAW events](https://www.spinics.net/lists/netdev/msg436441.html)
 - [[PATCH net-next 1/2] perf, bpf: add support for HW_CACHE and RAW events](https://www.spinics.net/lists/netdev/msg436520.html)
 - [[PATCH net-next 2/2] samples/bpf: add samples for HW_CACHE / RAW events](https://www.spinics.net/lists/netdev/msg436442.html)
- Teng Qin, [[PATCH net-next] bpf: update perf event helper function signature and documentation](https://www.spinics.net/lists/netdev/msg436455.html)
- Shubham Bansal, [[PATCH] RFC: arm: eBPF JIT compiler](https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1403296.html)
- Gustavo A. R. Silva, [[PATCH] kernel: bpf: remove dead code](https://www.spinics.net/lists/netdev/msg436238.html)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!

#### Cilium @ GlueCon 2017 this Week!

URL: https://cilium.io/blog/2017/5/22/cilium-gluecon-2017-this-week
Date: 2017-05-22
Categories: Community

The Cilium team is excited to be at [GlueCon 2017](http://gluecon.com/) Wed + Thurs this week, just outside Denver, CO.   GlueCon is a great developer-focused conference focused on APIs, containers, microservices, serverless, etc. We've enjoyed attending as individuals in the past, and are excited to be sponsoring this year!

We will be presenting "Cilium + BPF: Least Privilege Security on API Call Level for Microservices" on **Wed from 2:50-3:20pm**.  **This talk is on Track 1**\.   Come on by!

And all conference we'll be available at our Cilium booth to answer questions about API-aware network security, give demos, and of course, give away our Cilium t-shirts.  See you there!

#### BPF updates 03

URL: https://cilium.io/blog/2017/5/17/bpf-updates-03
Date: 2017-05-18
Categories: Technology

This is issue 03 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

The Linux kernel merge window closed up last week and v4.12-rc1 was released. One usual `[GIT] Networking` pull request was sent before the rc and one the day after. The are some BPF related fixes, see dates for all the highlights

- [9th May](https://www.spinics.net/lists/netdev/msg434497.html)
- [15th May](https://www.spinics.net/lists/netdev/msg435111.html)

More interesting topics

- [BPF relocations](https://www.mail-archive.com/netdev@vger.kernel.org/msg167958.html)
- [ARM 32-bit JIT](https://www.spinics.net/lists/netdev/msg434852.html)

##### In case you missed it

A lot of good content showed up this week. Check them all out below

###### [bpf.h and you...](https://www.spinics.net/lists/xdp-newbies/msg00179.html)

Great walk-through on writing a eBPF program. Starts from the header includes to the main bits.

###### [eBPF, part 2: Syscall and Map Types](https://ferrisellis.com/posts/ebpf_syscall_and_maps/)

Overview of `bpf(2)` with sections on the commands and map types. Also nice to see links to the various upstream commits.

###### [Monitoring the Control Plane](http://firstclassfunc.com/2017/05/monitoring-the-control-plane/)

Interesting read on capturing routing information with BPF.

###### [Contextually speaking...](https://www.spinics.net/lists/xdp-newbies/msg00181.html)

Provides descriptions on the BPF program types and walkthrough some of them with a example at the end.

###### [Elf Hello World Tutorial](http://www.cirosantilli.com/elf-hello-world/)

Useful for understanding how loaders (iproute2, etc) parse BPF ELF file.

###### [XDP Newbies...](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

> Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### [Random cool note](https://twitter.com/qeole/status/862588507147169797)

> Debugging #eBPF: there is now an IDA processor available on GitHub.

##### Patches

- Daniel Borkmann, [[PATCH net] bpf, arm64: fix faulty emission of map access in tail calls](https://www.mail-archive.com/netdev@vger.kernel.org/msg167864.html)
- Daniel Borkmann, [[PATCH net v2 0/2] Two generic xdp related follow-ups](https://www.mail-archive.com/netdev@vger.kernel.org/msg167982.html)
 - [[PATCH net v2 1/2] xdp: add flag to enforce driver mode](https://www.mail-archive.com/netdev@vger.kernel.org/msg167984.html)
 - [[PATCH net v2 2/2] xdp: refine xdp api with regards to generic xdp](https://www.mail-archive.com/netdev@vger.kernel.org/msg167983.html)
- Daniel Borkmann, [[PATCH iproute2 -master 0/2] Two misc BPF updates](https://www.spinics.net/lists/netdev/msg434904.html)
 - [[PATCH iproute2 -master 1/2] bpf: update printing of generic xdp mode](https://www.spinics.net/lists/netdev/msg434905.html)
 - [[PATCH iproute2 -master 2/2] bpf: dump error to the user when retrieving pinned prog fails](https://www.spinics.net/lists/netdev/msg434903.html)
- David Miller, [[PATCH v2 0/7] bpf: Add alignment tracker to verifier.](https://www.spinics.net/lists/netdev/msg434707.html)
 - [[PATCH v2 1/7] bpf: Track alignment of register values in the verifier.](https://www.spinics.net/lists/netdev/msg434708.html)
 - [[PATCH v2 2/7] bpf: Do per-instruction state dumping in verifier when log_level > 1.](https://www.spinics.net/lists/netdev/msg434709.html)
 - [[PATCH v2 3/7] bpf: Add strict alignment flag for `BPF_PROG_LOAD`.](https://www.spinics.net/lists/netdev/msg434710.html)
 - [[PATCH v2 4/7] bpf: Add `bpf_verify_program()` to the library.](https://www.spinics.net/lists/netdev/msg434711.html)
 - [[PATCH v2 5/7] bpf: Add verifier test case for alignment.](https://www.spinics.net/lists/netdev/msg434715.html)
 - [[PATCH v2 6/7] bpf: Make use of alignment information in `check_val_ptr_alignment()`.](https://www.spinics.net/lists/netdev/msg434713.html)
 - [[PATCH v2 7/7] bpf: Adjust test_verifier for alignment changes.](https://www.spinics.net/lists/netdev/msg434714.html)
- David Miller, [[PATCH v2 0/3] bpf: Track MAP pointer alignment](https://www.spinics.net/lists/netdev/msg435073.html)
 - [[PATCH v2 1/3] bpf: Use 1\<\<16 data-preserve-html-node="true" as ceiling for immediate alignment in verifier.](https://www.spinics.net/lists/netdev/msg435074.html)
 - [[PATCH v2 2/3] bpf: Track alignment of MAP pointers in verifier.](https://www.spinics.net/lists/netdev/msg435071.html)
 - [[PATCH v2 3/3] bpf: Update MAP test_verifier.c tests wrt. alignment.](https://www.spinics.net/lists/netdev/msg435072.html)
- Andy Gospodarek, [[PATCH net] samples/bpf: run cleanup routines when receiving SIGTERM](https://www.spinics.net/lists/netdev/msg434761.html)
- Jakub Kicinski, [[PATCH net-next 0/9] nfp: LSO, checksum and XDP datapath updates](https://www.mail-archive.com/netdev@vger.kernel.org/msg168356.html)
 - [[PATCH net-next 1/9] nfp: don't enable TSO on the device when disabled](https://www.mail-archive.com/netdev@vger.kernel.org/msg168358.html)
 - [[PATCH net-next 2/9] nfp: rename `l4_offset` in struct `nfp_net_tx_desc` to `lso_hdrlen`](https://www.mail-archive.com/netdev@vger.kernel.org/msg168357.html)
 - [[PATCH net-next 3/9] nfp: support LSO2 capability](https://www.mail-archive.com/netdev@vger.kernel.org/msg168357.html)
 - [[PATCH net-next 4/9] nfp: don't assume RSS and IRQ moderation are always enabled](https://www.mail-archive.com/netdev@vger.kernel.org/msg168360.html)
 - [[PATCH net-next 5/9] nfp: version independent support for chained RSS metadata](https://www.mail-archive.com/netdev@vger.kernel.org/msg168355.html)
 - [[PATCH net-next 6/9] nfp: add CHECKSUM_COMPLETE support](https://www.mail-archive.com/netdev@vger.kernel.org/msg168354.html)
 - [[PATCH net-next 7/9] nfp: complete the XDP TX ring only when it's full](https://www.mail-archive.com/netdev@vger.kernel.org/msg168361.html)
 - [[PATCH net-next 8/9] nfp: add a helper for wrapping descriptor index](https://www.mail-archive.com/netdev@vger.kernel.org/msg168362.html)
 - [[PATCH net-next 9/9] nfp: eliminate an if statement in calculation of completed frames](https://www.mail-archive.com/netdev@vger.kernel.org/msg168353.html)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!

#### BPF updates 02

URL: https://cilium.io/blog/2017/5/10/bpf-updates-02
Date: 2017-05-10
Categories: Technology

This is issue 02 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.

---

The Linux kernel merge window opened up last week and some of the highlights can be found as always in the `[GIT] Networking` pull requests from [2nd May 2017](https://www.spinics.net/lists/netdev/msg433609.html) and [4th May 2017](https://www.spinics.net/lists/netdev/msg433995.html).

If you are interested in other subsystems [LWN.net](https://lwn.net/) has a summary on the current [merge window](https://lwn.net/Articles/721581/) status, and their section on BPF says

> The BPF virtual machine subsystem has seen a few improvements. Maps are now able to contain other maps, allowing them to be cascaded to multiple levels. There is a new in-kernel testing framework for BPF programs, controlled by the new `BPF_PROG_TEST_RUN` command to the bpf() system call. And there is now a just-in-time BPF compiler for the SPARC64 architecture.

There is ongoing work to add netlink extended ACK reporting in iproute2, see [ip: Initial support for extack errors](https://www.spinics.net/lists/netdev/msg433408.html) and [iproute: Add support for extended ack to rtnl_talk](https://www.spinics.net/lists/netdev/msg433823.html) for more details.

More interesting developments are:

- [b0e92279d3ec](https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=b0e92279d3ec3656152c4dfa1c8b28fa40ca66d7) (Merge branch 'thunderx-xdp', 2017-05-02)
- [85f68fe89832](https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=85f68fe89832) (bpf, arm64: implement jiting of BPF_XADD, 2017-05-01).
- [e3bf4c61da80](https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=e3bf4c61da80) (sparc64: Fix BPF JIT wrt. branches and ldimm64 instructions., 2017-05-01)
- [3a5795b83d57](https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=3a5795b83d57) (bpf: lru: Add map-in-map LRU example, 2017-04-14)

Not all of the changes and highlights hitting mainline now are new and might have been partially mentioned in last [week's issue](https://www.cilium.io/blog/2017/5/2/bpf-updates-01-2017-05-02).

##### Presentations

###### Videos

###### [Netdev 2.1 - Busy Polling By Eric Dumazet](https://www.youtube.com/watch?v=X0xBCoQGUvg)

The talk is mainly on Busy Polling, with one slide on future BPF work with regards to siloing through SO_REUSEPORT, which has recently been [addressed upstream](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b1d9fc41aab11f9520b2e0d57ae872e2ec5d6f32).

> linux-4.12 changes (cont)
>
> Ideally, we should add eBPF support so that `SO_REUSEPORT` enabled listeners can choose the appropriate silo (per RX queue listener) directly at SYN time, using an appropriate `SO_ATTACH_REUSEPORT_EBPF` program.
>
> Same eBPF filter would apply for UDP traffic.

###### Slides

###### [XDP - eXpress Data Path, Lund Linux Con](http://people.netfilter.org/hawk/presentations/LLC2017/XDP_DDoS_protecting_LLC2017.pdf)

Great beginner friendly slides on XDP. Also worth checking out the cool collection of sample programs in the [prototype-kernel](https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf) repository.

###### [Trace Aggregation and Collection with eBPF](http://step.polymtl.ca/~suchakra/eBPF-5May2017.pdf)

Good overview on tracing. The diagrams throughout the slides are also very informative and help get a overview of the programmer model in BPF land.

###### [What impact has CloudNative on kernel networking?](https://docs.google.com/presentation/d/1dwSKSBGpUHD3WO5xxzZWj8awV_-xL-oYhvqQMOBhhtk/edit#slide=id.g203aae413f_0_0)

Nice slides on HTTP aware networking security enforcement.

##### In case you missed it

If you are ready to do some BPF programming but not sure where to start? Some interesting projects to make that easier are:

###### [BPF Compiler Collection (BCC)](https://github.com/iovisor/bcc)

The IO Visor Project provides front ends so you can write [Python](https://github.com/iovisor/bcc/blob/master/docs/tutorial_bcc_python_developer.md), [Lua](https://github.com/iovisor/bcc/tree/master/src/lua) or [Go](https://github.com/iovisor/gobpf). They also have a nice collection of useful programs which can discover real bugs. Also worth checking out their [XDP intro](https://www.iovisor.org/technology/xdp).

###### [rbpf](https://github.com/qmonnet/rbpf)

Don't want to write Go, Lua, Python or restricted C code? There is a Rust alternative path in user space.

###### [p4c-xdp](https://github.com/vmware/p4c-xdp)

Translate p4 programs into XDP.

###### [USENIX/LISA 2016 Linux bcc/BPF Tools](http://www.brendangregg.com/blog/2017-04-29/usenix-lisa-2016-bcc-bpf-tools.html)

Good overview on some of the BCC tools and overall tracing.

###### [Demo: May the Fourth be with you](https://www.cilium.io/blog/2017/5/4/demo-may-the-force-be-with-you)

You probably have watched it, but if not this is an awesome demo.

###### [Linux Kernel Developers' Netconf 2017](http://vger.kernel.org/netconf2017.html)

The netconf page for [netdev 2.1](http://netdevconf.org/2.1/) got [recently updated](https://twitter.com/davem_dokebi/status/861713796913655810) with more links. Check it out!

###### [XDP Newbies Mailing List](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

> Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### [Random cool note](https://twitter.com/majek04/status/860066075140141056)

> Solarflare drivers v4.10.6.1002 "XDP: Preview of RX side XDP support; This allows for RX filtering to use the DROP" [https://t.co/fwwK2tYa1g](https://t.co/fwwK2tYa1g)

##### Patches

- Alexei Starovoitov, [[PATCH net-next] selftests/bpf: get rid of -D**x86_64**](https://www.mail-archive.com/netdev@vger.kernel.org/msg166839.html)
- David Miller, [[PATCH] selftests: bpf: Use `bpf_endian.h` in `test_xdp.c`](https://patchwork.ozlabs.org/patch/757650/)
- David Miller, [[PATCH 0/2] Fix some bpf program testing framework bugs](https://www.mail-archive.com/netdev@vger.kernel.org/msg166737.html)
 - [[PATCH 1/2] bpf: Do not dereference user pointer in `bpf_test_finish()`.](https://www.mail-archive.com/netdev@vger.kernel.org/msg166740.html)
 - [[PATCH 2/2] bpf: Align packet data properly in program testing framework.](https://www.mail-archive.com/netdev@vger.kernel.org/msg166741.html)
- Daniel Borkmann, [[PATCH] xdp: use common helper for netlink extended ack reporting](https://patchwork.ozlabs.org/patch/757822/)
- Daniel Borkmann, [[PATCH net] bpf, arm64: fix jit branch offset related to ldimm64](https://www.spinics.net/lists/arm-kernel/msg579070.html)
- Daniel Borkmann, [[PATCH net] bpf: don't let ldimm64 leak map addresses on unprivileged](https://patchwork.ozlabs.org/patch/759495/)
- Daniel Borkmann, [[PATCH net 0/2] Two generic xdp related follow-ups](https://www.mail-archive.com/netdev@vger.kernel.org/msg167734.html)
 - [[PATCH net 1/2] xdp: add flag to enforce driver mode](https://www.mail-archive.com/netdev@vger.kernel.org/msg167733.html)
 - [[PATCH net 2/2] xdp: disallow use of native and generic hook at once](https://www.mail-archive.com/netdev@vger.kernel.org/msg167732.html)
- Geert Uytterhoeven, [[PATCH] test_bpf: Use ULL suffix for 64-bit constants](https://www.mail-archive.com/netdev@vger.kernel.org/msg166904.html)
- Jesper Dangaard Brouer, [[net-next PATCH 0/4] Improve bpf ELF-loader under samples/bpf](https://www.mail-archive.com/netdev@vger.kernel.org/msg166673.html)
 - [[net-next PATCH 1/4] samples/bpf: adjust rlimit RLIMIT_MEMLOCK for traceex2, tracex3 and tracex4](https://www.mail-archive.com/netdev@vger.kernel.org/msg166671.html)
 - [[net-next PATCH 2/4] samples/bpf: make bpf_load.c code compatible with ELF maps section changes](https://www.mail-archive.com/netdev@vger.kernel.org/msg166670.html)
 - [[net-next PATCH 3/4] samples/bpf: load_bpf.c make callback fixup more flexible](https://www.mail-archive.com/netdev@vger.kernel.org/msg166672.html)
 - [[net-next PATCH 4/4] samples/bpf: export map_data[] for more info on maps](https://www.mail-archive.com/netdev@vger.kernel.org/msg166674.html)
- Sunil Goutham, [[PATCH 0/9] net: thunderx: Adds XDP support](https://www.spinics.net/lists/arm-kernel/msg579013.html)
 - [[PATCH 1/9] net: thunderx: Support for page recycling](https://www.spinics.net/lists/arm-kernel/msg579018.html)
 - [[PATCH 2/9] net: thunderx: Optimize RBDR descriptor handling](https://www.spinics.net/lists/arm-kernel/msg579014.html)
 - [[PATCH 3/9] net: thunderx: Optimize CQE_TX handling](https://www.spinics.net/lists/arm-kernel/msg579019.html)
 - [[PATCH 4/9] net: thunderx: Cleanup receive buffer allocation](https://www.spinics.net/lists/arm-kernel/msg579020.html)
 - [[PATCH 5/9] net: thunderx: Add basic XDP support](https://www.spinics.net/lists/arm-kernel/msg579015.html)
 - [[PATCH 6/9] net: thunderx: Add support for XDP_DROP](https://www.spinics.net/lists/arm-kernel/msg579016.html)
 - [[PATCH 7/9] net: thunderx: Add support for XDP_TX](https://www.spinics.net/lists/arm-kernel/msg579017.html)
 - [[PATCH 8/9] net: thunderx: Support for XDP header adjustment](https://www.spinics.net/lists/arm-kernel/msg579028.html)
 - [[PATCH 9/9] net: thunderx: Optimize page recycling for XDP](https://www.spinics.net/lists/arm-kernel/msg579021.html)
- Or Gerlitz, [[iproute] tc: Reflect HW offload status](https://patchwork.ozlabs.org/patch/758566/)
- Yuval Mintz, [[PATCH net 0/5] qed\*: General fixes](https://www.mail-archive.com/netdev@vger.kernel.org/msg167617.html)
 - [[PATCH net 1/5] qede: Fix XDP memory leak on unload](https://www.mail-archive.com/netdev@vger.kernel.org/msg167623.html)
 - [[PATCH net 2/5] qed: Fix VF removal sequence](https://www.mail-archive.com/netdev@vger.kernel.org/msg167621.html)
 - [[PATCH net 3/5] qed: Tell QM the number of tasks](https://www.mail-archive.com/netdev@vger.kernel.org/msg167622.html)
 - [[PATCH net 4/5] qed: Correct doorbell configuration for!4Kb pages](https://www.mail-archive.com/netdev@vger.kernel.org/msg167618.html)
 - [[PATCH net 5/5] qede: Split PF/VF ndos.](https://www.mail-archive.com/netdev@vger.kernel.org/msg167619.html)
- Yonghong Song, [[PATCH net-next] selftests/bpf: add a test case to check verifier pointer arithmetic](https://patchwork.ozlabs.org/patch/757842/).
- Yonghong Song, [[llvm] r302055 - [bpf] add relocation support](http://llvm.org/viewvc/llvm-project?rev=302055&view=rev)
- Yonghong Song, [[llvm] r302265 - [bpf] fix a bug which causes incorrect big endian reloc fixup](http://llvm.org/viewvc/llvm-project?view=revision&revision=302265)
- David Ahern, [[PATCH net-next iproute2 0/3] ip: Initial support for extack errors](https://www.spinics.net/lists/netdev/msg433408.html)
 - [[PATCH net-next iproute2 1/3] netlink: import netlink message parsing from kernel](https://www.spinics.net/lists/netdev/msg433409.html)
 - [[PATCH net-next iproute2 2/3] netlink: Add support for extended ack to rtnl_talk](https://www.spinics.net/lists/netdev/msg433412.html)
 - [[PATCH net-next iproute2 3/3] ip link: Add extack handling for setlink](https://www.spinics.net/lists/netdev/msg433410.html)
- Stephen Hemminger, [[RFC] iproute: Add support for extended ack to rtnl_talk](https://www.spinics.net/lists/netdev/msg433823.html)

Please note that netdev receives a lot of patches and the list above is not meant to be comprehensive.

Happy eBPF hacking!

#### Demo: May the Fourth be with you

URL: https://cilium.io/blog/2017/5/4/demo-may-the-force-be-with-you
Date: 2017-05-05
Categories: How-To

In celebration of today's date, May 4th, we are posting our Star Wars demo of Cilium that we showed at CNCF/KubeCon in Berlin or DockerCon in Austin. You can either watch the video linked below which starts directly at the time the demo begins or you can jump to the transcript of the demo embedded in the blog post. May the fourth be with you.

```
# A long time ago, in a container cluster far, far away....
#
# It is a period of civil war. The Empire has adopted
# microservices and continuous delivery, despite this,
# Rebel spaceships, striking from a hidden cluster, have
# won their first victory against the evil Galactic Empire.
#
# During the battle, Rebel spies managed to steal the
# swagger API specification to the Empire's ultimate weapon,
# the deathstar.
```

This first step creates a Docker network which the empire and rebel alliance can use to attach their containers to. It is of driver and IPAM type _cilium_ which implies that Cilium is in charge of both address management and providing networking whenever a container is attached to the network.

```
$ docker network create --ipv6 --subnet ::1/112 --driver cilium --ipam-driver cilium space
c712f2b0d915825bf16b45705f76d7bf5cb947159b9753edd1bfcf6c56749ca1
```

The empire starts constructing the deathstar by starting a container named _deathstar_ in the network _cilium_ with the label _id.empire.deathstar_. This label indicate that the container is a deathstar beloning to the empire.

```
$ docker run -dt --net=space --name deathstar -l id.empire.deathstar cilium/starwars
f55bacd444d0c552488650916183f045bdabcf9eb8d2771654f707bbbb986d60
```

The empire now wants to allow spaceship containers to launch and land at the deathstar. The spaceships have to use the deathstar's REST API to request landing permissions. This requires a network policy to allow for this communication. Cilium follows a whitelist policy model which means that if policy enforcement is enabled, all communication must be explicitly allowed, all other communication is prohibited.

```json
$ cat sw_policy_l4.json
{
  "name": "root",
  "rules": [{
    "coverage": ["id.empire.deathstar"],
    "allow": ["id.spaceship"]
  },{
    "coverage": ["id.empire.deathstar"],
    "l4": [{
      "in-ports": [{ "port": 80, "protocol": "tcp" }]
    }]
  }]
}
```

The policy above will allow spaceships to communicate to the deathstar by allowing:

- Any container with the label `id.spaceship` to communicate with any container with the label `id.empire.deathstar`. If a container does not have the label `id.spaceship` assigned, it will not be able to talk to `id.empire.deathstar`.
- Any container with the label `id.empire.deathtar` to only take incoming connections on port 80 using the TCP protocol. Use of any other port is prohibited.

The empire can import this policy...

```
$ cilium policy import sw_policy_l4.json
```

... and then test the connectivity by starting a spaceship container with the label `id.spaceship` and have it land on the deathstar after requesting landing permission:

```
$ docker run -dt --net=space --name ship1 -l id.spaceship tgraf/netperf
919cce790ef4344080aa0fd3ecf7435cb773c0a813b5dc6dce12c5e93f5c1102

$ docker exec -i ship1 curl -si -XPOST http://deathstar/v1/request-landing
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Fri, 28 Apr 2017 15:18:02 GMT
Content-Length: 12

Ship landed
```

---

In the meantime.... The rebel alliance has noticed that the empire has started to construct a deathstar. They send out an X-Wing to scout and explore the situation:

```
$ docker run -dt --net=space --name xwing -l id.spaceship tgraf/netperf
0a87e2677eae25118a10c24c903e3b7a0efca4a70157996f72d8f29b731cfc76
```

The X-wing spaceship flies up to the deathstar and starts probing it with his radar:

```
$ docker exec -i xwing ping -c 2 deathstar
PING deathstar (10.15.116.202): 56 data bytes
64 bytes from 10.15.116.202: seq=0 ttl=64 time=0.170 ms
64 bytes from 10.15.116.202: seq=1 ttl=64 time=0.087 ms

--- deathstar ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.087/0.128/0.170 ms
```

Why does the deathstar respond to the X-Wing? The X-Wing is a spaceship as well and thus has the label `id.spaceship` set which is covered by the network policy loaded by the empire.

Excited about the response, the X-Wing pilot triggers a REST API call `GET /` to the deathstar and gets a response:

```json
$ docker exec -i xwing curl -si -XGET http://deathstar/v1/
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Fri, 28 Apr 2017 15:21:47 GMT
Content-Length: 548

{
  "name": "Death Star",
  "model": "DS-1 Orbital Battle Station",
  "manufacturer": "Imperial Department of Military Research, Sienar Fleet Systems",
  "cost_in_credits": "1000000000000",
  "length": "120000",
  "crew": "342953",
  "passengers": "843342",
  "cargo_capacity": "1000000000000",
  "hyperdrive_rating": "4.0",
  "starship_class": "Deep Space Mobile Battlestation",
  "api": [
    "GET  /v1",
    "GET  /v1/healthz",
    "POST /v1/request-landing",
    "PUT  /v1/cargobay",
    "GET  /v1/hyper-matter-reactor/status",
    "PUT  /v1/exhaust-port"
  ]
}
```

Wow, that's a lot of information. The deathstar API responds with a list of properties including the entire API surface that is available.

And look at this... we can access `PUT /v1/exhaust-port`. The rebels immediately realize that they have found a way to gain access to the ultimate weakness of the deathstar, the reactor core.

---

The X-Wing fly by does not go unnoticed by the empire though and the empire SecOps teams realized the vulnerability of their API. They read up on [Cilium documentation](http://docs.cilium.io) to implement HTTP level security policies to protect their API.

```json
$ cat sw_policy_http.show.json
{
  "name": "root",
  "rules": [{
    "coverage": ["id.empire.deathstar"],
    "allow": ["id.spaceship", "reserved:host"]
  },{
    "coverage": ["id.spaceship"],
    "l4": [{
      "out-ports": [{
        "port": 80, "protocol": "tcp",
        "l7-parser": "http",
        "l7-rules": [
          { "expr": "Method(\"GET\") && Path(\"/v1/\")" },
          { "expr": "Method(\"POST\") && Path(\"/v1/request-landing\")" }
        ]
      }]
    }]
  }]
}
```

The policy the empire consists of two rules:

- The first rule allows `id.spaceship` and the local host where the enforcement proxy is running to talk to `id.empire.deathstar`. This is the same policy as before with the addition of the proxy used for the second rule.
- The second rule says that any container with the label `id.spaceship` can only initiate connections on port 80 with protocol TCP and that any connection needs to pass through an HTTP protocol parser which then filters all communication an only passes through the following two requests:
 - `GET /v1/`
 - `POST /v1/request-landing` All other REST API calls are prohibited by Cilium.

The empire loads this policy into Cilium:

 $ cilium policy import sw_policy_http.real.json

```

Unsuspecting about the efforts of the empire, the rebels execute their attack. Their ships can still probe the deathstar via radar:

    $ docker exec -i xwing ping -c 2 deathstar
    PING deathstar (10.15.116.202): 56 data bytes
    64 bytes from 10.15.116.202: seq=0 ttl=64 time=0.078 ms
    64 bytes from 10.15.116.202: seq=1 ttl=64 time=0.091 ms

    --- deathstar ping statistics ---
    2 packets transmitted, 2 packets received, 0% packet loss
    round-trip min/avg/max = 0.078/0.084/0.091 ms

But as the X-Wing approaches the deathstar and issues the REST API call to access the thermal exhaust port:

    $ docker exec -i xwing curl -si -XPUT http://deathstar/v1/exhaust-port
    HTTP/1.1 403 Forbidden
    Content-Type: text/plain; charset=utf-8
    X-Content-Type-Options: nosniff
    Date: Fri, 28 Apr 2017 15:37:47 GMT
    Content-Length: 14

    Access denied

The shields are up. End of demo...

Obviously we can't end May 4 like this and change the story. We don't want to be on the wrong side of history. So here is what you missed...

While the deathstar was being constructed and the empire SecOps team but the HTTP aware policy in place, the Jedi managed to infiltrate the deathstar and put a different policy in place.

So let's run `diff` on the loaded policy and the policy you saw earlier:

    $ diff -Nru sw_policy_http.show.json sw_policy_http.real.json
    --- sw_policy_http.show.json    2017-04-28 16:43:32.372615840 +0200
    +++ sw_policy_http.real.json    2017-04-28 16:43:32.372615840 +0200
    @@ -11,7 +11,8 @@
             "l7-parser": "http",
             "l7-rules": [
               { "expr": "Method(\"GET\") && Path(\"/v1/\")" },
    -          { "expr": "Method(\"POST\") && Path(\"/v1/request-landing\")" }
    +          { "expr": "Method(\"POST\") && Path(\"/v1/request-landing\")" },
    +          { "expr": "Method(\"PUT\") && Header(\"X-Has-Force\", \"true\") && Path(\"/v1/exhaust-port\")" }
             ]
           }]
         }]

So the Jedi managed to add an additional rule to the policy which says:

- You an do a `PUT /v1/exhaust-port` if the request has the header `X-Has-Force: true` set.

We all know what is going to happen now...

    $ docker run -dt --net=space --name xwing_luke -l id.spaceship tgraf/netperf
    7c9992bfd3866f54b1b9fc142bdc2e0169034b7c6bf21603b819fe571c712f56

    $ docker exec -i xwing_luke curl -si -H 'X-Has-Force: true' -XPUT http://deathstar/v1/exhaust-port
    HTTP/1.1 503 Service Unavailable
    Content-Length: 275
    Content-Type: text/plain
    Date: Fri, 28 Apr 2017 15:46:59 GMT

    Panic: deathstar exploded

    goroutine 1 [running]:
    main.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
            /code/src/github.com/empire/deathstar/
            temp/main.go:9 +0x64
    main.main()
            /code/src/github.com/empire/deathstar/
            temp/main.go:5 +0x85

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.11.0/styles/default.min.css">  <script>hljs.initHighlightingOnLoad();</script>
```

#### Why we love the packet.net cloud for CI/CD

URL: https://cilium.io/blog/2017/5/2/cicd-and-why-we-love-the-packetnet-cloud
Date: 2017-05-03
Categories: How-To

A while ago we started thinking about how to build a continous integration (CI) testing infrastructure that would help us maintain the reliability and stability of Cilium as we add more features and integrate with orchestration systems like Kubernetes and Docker. This post describes our process for building a CI environment capable of testing a container networking & security infrastructure. It's also a thank you to packet.net for supporting Cilium along with many other open source projects. For more, read on...

**tl;dr [packet.net](https://www.packet.net/) and their bare metal cloud made it easy for us to leverage our existing developer Vagrant setup to run complex CI tests that themselves need to create/destroy containers.**

The core function of Cilium is to provide visibility and control over network traffic coming in or out of a container. For example, when a new container is created, Cilium learns about the identity of the container from an orchestration framework like Docker/Kubernetes and then compiles a [BPF](http://docs.cilium.io/en/stable/bpf/) program to apply the appropriate L3/L4/L7 security policies associated with that container.

As a result, a major consideration for architecting our CI environment is that Cilium tests themselves need to create/destroy containers.

Common CI/CD environments such as Travis CI or CircleCI are fantastic and provided a great foundation to run build and unit tests but both leverage container technology themselves to sandbox builds. This makes starting containers from within such tests complicated.

We briefly looked at DinD (Docker in Docker) but quickly dismissed the option for [multiple reasons](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/). We also looked at Rkt and its capability to [run pods with KVM hypervisor as stage1](https://rocket.readthedocs.io/en/latest/Documentation/running-kvm-stage1/). The latter seemed interesting but at first glance seemed to have a lot of moving pieces and was a more significant departure from how developers were currently running tests on their own machines using [Vagrant](https://www.vagrantup.com/).

In parallel, the CNCF had just announced their [1000 node bare metal cluster](https://www.cncf.io/cluster/) to foster testing of community projects. Unfortunately, the cluster was intended for one time large scale tests, rather than fractional and continous use by multiple projects. Seeing our struggle, Zachary Smith, CEO of packet, reached out and kindly offered resources on the packet.net cloud.

For those not familiar with packet: packet is a bare metal cloud which makes provisioning of bare metal machines very simple and offers developer friendly APIs for automation. You can find more information on their [website](https://www.packet.net/features/).

Simple access to a bare metal resources was intriguing so we gave it a shot. Provisioning the machines and setting up as Jenkins on them was extremely simple and done in less than a morning. Only minutes after provisioning we had the first CI testrun ongoing using our existing [Vagrantfile](https://raw.githubusercontent.com/cilium/cilium/master/Vagrantfile) which was already capable of running the testsuite.

All it took was to add the simple `Jenkinsfile` below:

```
pipeline {
    agent {
        label 'vagrant'
    }
    options {
        timeout(time: 30, unit: 'MINUTES')
    }
    stages {
        stage('Build') {
            environment {
                MEMORY = '4096'
                RUN_TEST_SUITE = '1'
            }
            steps {
                sh './contrib/vagrant/start.sh'
            }
        }
    }
    post {
        always {
            sh 'vagrant destroy -f'
        }
    }
}
```

For us, using a bare metal cloud had two keep benefits:

- Performance benchmark tests execute quickly and reliably.
- The Vagrant environment for CI testing matches exactly what developers already have in place on their own machines, making it trivial to reproduce test failures locally.

We couldn't be happier with our CI environment and we are continuously extending coverage.

Huge shoutout to [packet.net](https://www.packet.net/), we are very grateful for their support of Cilium!

#### BPF Updates 01: SPARC eBPF JIT, Generic XDP, and ixgbe Support

URL: https://cilium.io/blog/2017/5/2/bpf-updates-01-2017-05-02
Date: 2017-05-02
Categories: Technology

This is the start of a regular newsletter around BPF written by Alexander Alemayhu. It will summarize ongoing development, presentations, videos and other information related to BPF and XDP. It will be released roughly once a week.

---

Recently some interesting developments in the BPF space occurred and the highlights are

- SPARC gained support for eBPF in [7a12b5031c6b (sparc64: Add eBPF JIT., 2017-04-17)](https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=7a12b5031c6b).
- A generic XDP implementation landed in [b5cdae3291f7 (net: Generic XDP, 2017-04-18)](https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=b5cdae3291f7).
- BPF support for binutils is [in the works](https://www.spinics.net/lists/netdev/msg433108.html).
- XDP support for ixgbe [got added](https://www.spinics.net/lists/netdev/msg433035.html).

The ixgbe patches were ready earlier but some lost in transit cleanup and build errors required a v2\. Hopefully the binutils effort will lead to BPF support in GCC. While LLVM and clang is available on most distributions, the opportunity to use the familiar GCC will make it even easier for newbies to get started.

Being a XDP newbie is getting more pleasant by the day. Thanks to all of the nice investments being made by kernel developers. The infrastructure and tooling around BPF is improving. You can test your programs with the new `BPF_PROG_TEST_RUN` command and with the generic implementation of XDP you no longer need a special network card. The excuses for not trying out XDP are getting shorter;->

##### Videos

###### [Cilium: Network and Application Security with BPF and XDP](https://www.youtube.com/watch?v=ilKlmTDdFgk)

Your friends have probably heard about BPF, Cilium and XDP by now. If not they can watch this great talk on how containers can leverage the true superpowers of BPF.

###### [Netdev 2.1 - Keynote By David S. Miller](https://www.youtube.com/watch?v=8Cxg7mpVIWw&feature=youtu.be)

Without spoiling the talk the last parallels drawn at the end is really important to understand where XDP effort is headed next.

###### [Netdev 2.1 - Droplet: DDoS countermeasures powered by BPF + XDP](https://www.youtube.com/watch?v=YEU2ClcGqts&feature=youtu.be)

Nice talk on how Facebook is leveraging BPF and XDP in production.

###### [Netdev 2.1 - XDP in practice: integrating XDP in our DDoS mitigation pipeline](https://www.youtube.com/watch?v=7OuOukmuivg)

This talk shows some of CloudFlare current DDoS mitigation strategies / tools. The whole talk is good, but the sections where what is preventing adoption and other disadvantages of XDP is interesting.

##### In case you missed it

###### [eBPF, part 1: Past, Present, and Future](https://ferrisellis.com/posts/ebpf_past_present_future/)

Good walk through on the differences between BPF and eBPF. The historical context makes this a nice read for newbies.

###### [Cilium - BPF and XDP Reference Guide](http://docs.cilium.io/en/stable/bpf/)

This documentation section is targeted at developers and users who want to understand BPF and XDP in great technical depth.

###### [XDP Newbies Mailing List](https://www.mail-archive.com/netdev@vger.kernel.org/msg162375.html)

Which is a place where people can talk about getting up to speed with setting up an XDP build environment and writing XDP programs.

You can subscribe by sending a email to `majordomo@vger.kernel.org`, with a message body containing `subscribe xdp-newbies`. No subject is needed, but you can of course add one if you like.

##### [Random cool note](https://x.com/davem_dokebi/status/855595121424859138)

> The JSET cBPF instruction is for parsing DECNET packet headers. See Acknowledgements section of: [http://www.tcpdump.org/papers/bpf-usenix93.pdf](http://www.tcpdump.org/papers/bpf-usenix93.pdf)

##### Patches

- Alexei Starovoitov, [[net-next] bpf: `map_get_next_key` to return first key on NULL](https://www.spinics.net/lists/netdev/msg431959.html)
- Alexei Starovoitov, [[PATCH net-next] bpf: enhance verifier to understand stack pointer arithmetic](https://patchwork.ozlabs.org/patch/756817/)
- Daniel Borkmann, [[PATCH net-next] bpf: make `bpf_xdp_adjust_head` support mandatory](https://www.spinics.net/lists/netdev/msg431901.html)
- Daniel Borkmann, [[PATCH net-next 0/5] Misc BPF updates](https://www.spinics.net/lists/netdev/msg432526.html)
 - [[PATCH net-next 1/5] bpf, x86_64/arm64: remove old ldimm64 artifacts from jits](https://www.spinics.net/lists/netdev/msg432522.html)
 - [[PATCH net-next 2/5] bpf: add various test cases to verifier selftests](https://www.spinics.net/lists/netdev/msg432527.html)
 - [[PATCH net-next 3/5] bpf: bpf_lock on kallsysms doesn't need to be irqsave](https://www.spinics.net/lists/netdev/msg432528.html)
 - [[PATCH net-next 4/5] bpf: fix `_htons` occurences in `test_progs`](https://www.spinics.net/lists/netdev/msg432523.html)
 - [[PATCH net-next 5/5] bpf: provide a generic macro for percpu values for selftests](https://www.spinics.net/lists/netdev/msg432524.html)
- Daniel Borkmann, [[PATCH iproute2 net-next v2] bpf: add support for generic xdp](https://www.mail-archive.com/netdev@vger.kernel.org/msg166054.html)
- Daniel Borkmann, [[PATCH net-next] bpf, arm64: implement jiting of BPF_XADD](https://patchwork.ozlabs.org/patch/756977/)
- David Miller, [[PATCH net-next v2] sparc64: Improve 64-bit constant loading in eBPF JIT.](https://www.spinics.net/lists/netdev/msg431967.html)
- David Miller, [[PATCH] sparc64: Fix BPF JIT wrt. branches and ldimm64 instructions.](https://patchwork.ozlabs.org/patch/757381/)
- David Ahern, [[PATCH net-next] samples/bpf: Add support for `SKB_MODE` to xdp1 and `xdp_tx_iptunnel`](https://www.spinics.net/lists/netdev/msg432665.html)
- Eric Dumazet, [[PATCH net-next] net: move xdp_prog field in RX cache lines](https://www.spinics.net/lists/netdev/msg432237.html)
- Eric Dumazet, [[PATCH net-next] bpf: restore skb->sk before pskb_trim() call](https://www.mail-archive.com/netdev@vger.kernel.org/msg165642.html)
- John Fastabend, [[PATCH 0/2] ixgbe updates](https://www.spinics.net/lists/netdev/msg431699.html)
 - [[PATCH 1/2] ixgbe: add XDP support for pass and drop actions](https://www.spinics.net/lists/netdev/msg431700.html)
 - [[PATCH 2/2] ixgbe: add support for XDP_TX action](https://www.spinics.net/lists/netdev/msg431701.html)
- Martin KaFai Lau, [[RFC net-next 0/2] Introduce bpf_prog ID and iteration](https://www.spinics.net/lists/netdev/msg432550.html)
 - [[RFC net-next 1/2] bpf: Introduce bpf_prog ID](https://www.spinics.net/lists/netdev/msg432552.html)
 - [[RFC net-next 2/2] bpf: Test for `bpf_prog` ID and `BPF_PROG_GET_NEXT_ID`](https://www.spinics.net/lists/netdev/msg432551.html)
- Jakub Kicinski, [[PATCH net-next 0/6] nfp: optimize XDP TX and small fixes](https://www.mail-archive.com/netdev@vger.kernel.org/msg165967.html)
 - [[PATCH net-next 1/6] nfp: replace -ENOTSUPP with -EOPNOTSUPP](https://www.mail-archive.com/netdev@vger.kernel.org/msg165971.html)
 - [[PATCH net-next 2/6] nfp: drop rx_ring param from buffer allocation](https://www.mail-archive.com/netdev@vger.kernel.org/msg165972.html)
 - [[PATCH net-next 3/6] nfp: do simple XDP TX buffer recycling](https://www.mail-archive.com/netdev@vger.kernel.org/msg165968.html)
 - [[PATCH net-next 4/6] nfp: avoid reading TX queue indexes from the device](https://www.mail-archive.com/netdev@vger.kernel.org/msg165973.html)
 - [[PATCH net-next 5/6] nfp: don't completely refuse to work with old flashes](https://www.mail-archive.com/netdev@vger.kernel.org/msg165970.html)
 - [[PATCH net-next 6/6] nfp: provide 256 bytes of XDP headroom in all configurations](https://www.mail-archive.com/netdev@vger.kernel.org/msg165969.html)
- Jakub Kicinski, [[PATCH net-next 0/4] xdp: use netlink extended ACK reporting](https://www.mail-archive.com/netdev@vger.kernel.org/msg166365.html)
 - [[PATCH net-next 1/4] netlink: add NULL-friendly helper for setting extended ACK message](https://www.mail-archive.com/netdev@vger.kernel.org/msg166364.html)
 - [[PATCH net-next 2/4] xdp: propagate extended ack to XDP setup](https://www.mail-archive.com/netdev@vger.kernel.org/msg166367.html)
 - [[PATCH net-next 3/4] nfp: make use of extended ack message reporting](https://www.mail-archive.com/netdev@vger.kernel.org/msg166368.html)
 - [[PATCH net-next 4/4] virtio_net: make use of extended ack message reporting](https://www.mail-archive.com/netdev@vger.kernel.org/msg166366.html)
- Jesper Dangaard Brouer, [[net-next PATCH V1] samples/bpf: bpf_load.c detect and abort if ELF maps section size is wrong](https://www.spinics.net/lists/netdev/msg432861.html)
- Jesper Dangaard Brouer, [[net-next PATCH 0/2] samples/bpf: two bug fixes to `XDP_FLAGS_SKB_MODE` attaching](https://www.mail-archive.com/netdev@vger.kernel.org/msg166370.html)
 - [[net-next PATCH 1/2] samples/bpf: fix SKB_MODE flag to be a 32-bit unsigned int](https://www.mail-archive.com/netdev@vger.kernel.org/msg166371.html)
 - [[net-next PATCH 2/2] samples/bpf: fix `XDP_FLAGS_SKB_MODE` detach for `xdp_tx_iptunnel`](https://www.mail-archive.com/netdev@vger.kernel.org/msg166372.html)

Please note that netdev receives a lot of patches and the list above is not meant to be complete.

#### Launch of BPF & XDP Documentation

URL: https://cilium.io/blog/2017/4/24/launch-of-bpf-xdp-documentation
Date: 2017-04-24
Categories: Technology

We are excited to announce the "**BPF & XDP Reference Guide**" as part of the Cilium project documentation. We have received various requests on further technical information about BPF and XDP with the desire to learn more about the technology that is driving the Cilium project.

Daniel Borkmann, kernel developer, cilium contributor and one of the BPF subsystem maintainers, has started to put together a comprehensive and detailed document which describes BPF and XDP in great depth:

**[BPF and XDP Reference Guide](http://docs.cilium.io/en/stable/bpf/)**

It is still being actively worked on but we wanted to share this with other developers interested in BPF and XDP as early as possible.

##### BPF and XDP Reference Guide

BPF is a highly flexible and efficient “virtual machine”-like construct in the Linux kernel allowing to execute bytecode at various hook points in a safe manner. It is used in a number of Linux kernel subsystems, most prominently networking, tracing and security (f.e. sandboxing).

While BPF has existed since 1992, this document covers the extended Berkley Paket Filter (eBPF) version which has first appeared in Kernel 3.18 and obsoletes the original version which is being referred to as “classic” BPF (cBPF) these days. cBPF is known to many as being the packet filter language use by tcpdump. Nowadays, the Linux kernel runs eBPF only and loaded cBPF bytecode is transparently translated into an eBPF representation in the kernel before program execution. This documentation will generally refer to the term BPF unless explicit differences between eBPF and cBPF are being pointed out.

Even though the name Berkley Packet Filter hints at a packet filtering specific purpose, the instruction set is generic and flexible enough these days that there are many use cases for BPF apart from networking. See [BPF Users](http://docs.cilium.io/en/stable/bpf/#projects-using-bpf) for a list of projects which use BPF.

Cilium uses BPF heavily in its data path, see [Architecture Guide](http://docs.cilium.io/en/stable/architecture/) for further information. The goal of this chapter is to provide an BPF reference guide in oder to gain understanding of BPF its networking specific use including loading BPF programs with tc (traffic control) and XDP (eXpress Data Path), and to aide developing Cilium’s BPF templates.

###### BPF Architecture

BPF does not define itself by only providing its instruction set, but also by offering further infrastructure around it such as maps that act as efficient key / value stores, helper functions to interact with and leverage kernel functionality, tail calls for calling into other BPF programs, security hardening primitives, a pseudo file system for pinning objects (maps, programs), and infrastructure for allowing BPF to be offloaded, for example, to a network card.

LLVM provides an BPF back end, such that tools like clang can be used to compile C into an BPF object file, which can then be loaded into the kernel.  BPF is deeply tied into the Linux kernel and allows for full programmability without sacrificing native kernel performance.

Last but not least, also the kernel subsystems making use of BPF are part of BPF’s infrastructure. The two main subsystems discussed throughout this document are tc and XDP where BPF programs can be attached to. XDP BPF programs are attached at the earliest networking driver stage and trigger a run of the BPF program upon packet reception. By definition, this achieves the best possible packet processing performance since packets cannot get processed at an even earlier point in software. Driver support is necessary in order to use XDP BPF programs, though. However, tc BPF programs don’t need any driver support and can be attached to receive and transmit paths of any networking device, including virtual ones such as veth devices since they hook later in the kernel stack compared to XDP. Apart from tc and XDP programs, there are various other kernel subsystems as well that use BPF such as tracing (kprobes, uprobes, tracepoints, etc).

The following subsections provide further details on individual aspects of the BPF architecture.

Read the full guide here: **[BPF and XDP Reference Guide](http://docs.cilium.io/en/stable/bpf/)**

#### Learn More About Cilium at KubeCon in Berlin!

URL: https://cilium.io/blog/2017/3/29/learn-more-about-cilium-at-kubecon-in-berlin
Date: 2017-03-29
Categories: Community

This week the Cilium team is excited to be in the beautiful city of Berlin at KubeCon / CloudNativeCon EU!

Come by our booth to learn more about how Cilium provides HTTP-aware network security for microservices applications running on Kubernetes!  Our booth is located just to the left when you enter the exhibit area (booth S5).

At the booth, you'll see a super-cool "Star Wars" themed demo of Cilium or to pick up one of our fresh off the presses Cilium t-shirts.  Let's help Linux love Microservices!

#### Cilium: Networking and security for containers with BPF and XDP

URL: https://cilium.io/blog/2017/3/16/cilium-networking-and-security-for-containers-with-bpf-and-xdp
Date: 2017-03-16
Categories: Technology

_This is a guest post by Daniel Borkmann who was recently recognized through the Google [Open Source Peer Bonus](https://opensource.googleblog.com/2016/09/google-open-source-peer-bonus-program.html) program for his work on the Cilium project. We invited Daniel to share his project on our blog._

Our open source project, called [Cilium](https://github.com/cilium/cilium), started as an experiment for Linux container networking tackling four requirements:

##### Scale

How can we scale in terms of addressing and with regards to network policy?

##### Extensibility

Can we be as extensible as user space networking in the Linux kernel itself?

##### Simplicity

What is an appropriate abstraction away from traditional networking?

##### Performance

Do we sacrifice performance in the process of implementing the aforementioned aspects?

We realize these goals in Cilium with the help of [eBPF](https://en.wikipedia.org/wiki/Berkeley_Packet_Filter#Extensions_and_optimizations). eBPF is an efficient and generic in-kernel bytecode engine, that allows for full programmability. There are many subsystems in the Linux kernel that utilize eBPF, mainly in the areas of networking, tracing and security.

eBPF can be attached to key ingress and egress points of the kernel's networking data path for every network device. As input, eBPF operates on the kernel's network packet representation and can thus access and mangle various kinds of data, redirect the packet to other devices, perform encapsulations, etc.

This is a typical workflow: eBPF is programmed in a subset of C, compiled with [LLVM](https://en.wikipedia.org/wiki/LLVM)which contains an eBPF back-end. LLVM then generates an ELF file containing program code, specification for maps and related relocation data. In eBPF, maps are efficient key/value stores in the kernel that can be shared between various eBPF programs, but also between user space. Given the ELF file, tools like [tc](<https://en.wikipedia.org/wiki/Tc_(Linux)>) (traffic control) can parse its content and load the program into the kernel. Before the program is executed, the kernel verifies the eBPF bytecode in order to make sure that it cannot affect the kernel's stability (e.g. crash the kernel and out of bounds access) and always terminates, which requires programs to be free of loops. Once it passed verification, the program is [JIT (just-in-time) compiled](https://en.wikipedia.org/wiki/Just-in-time_compilation).

Today, architectures such as x86_64, arm64, ppc64 and s390 have the ability to compile a native opcode image out of an eBPF program, so that instead of an execution through an in-kernel eBPF interpreter, the resulting image can run natively like any other kernel code. tc then installs the program into the kernel's networking data path, and with a capable [NIC](https://en.wikipedia.org/wiki/Network_interface_controller), the program can also be offloaded entirely into the hardware.

Cilium acts as a middle layer, plugs into container runtimes and orchestrators such as [Kubernetes](http://kubernetes.io/), [Docker](<https://en.wikipedia.org/wiki/Docker_(software)>) or [CNI](https://github.com/containernetworking/cni), and can generate and atomically update eBPF programs on the fly without requiring a container to restart. Thus, unlike connection proxies, an update of the datapath does not cause connections to be dropped. These programs are specifically tailored and optimized for each container, for example, a feature that a particular container does not need can just be compiled out and the majority of configuration becomes constant, allowing LLVM for further optimizations.

We have many implemented building blocks in Cilium using eBPF, such as [NAT64](https://en.wikipedia.org/wiki/NAT64), [L3/L4](https://en.wikipedia.org/wiki/OSI_model) load balancing with direct server return, a connection tracker, port mapping, access control, NDisc and [ARP](https://en.wikipedia.org/wiki/Address_Resolution_Protocol) responder and integration with various encapsulations like [VXLAN](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN), [Geneve](https://cto.vmware.com/geneve-vxlan-network-virtualization-encapsulations/) and [GRE](https://en.wikipedia.org/wiki/Generic_Routing_Encapsulation), just to name a few. Since all these building blocks run in the Linux kernel and have a stable API, there is of course no need to cross kernel/user space boundary, which makes eBPF a perfectly suited and flexible technology for container networking.

One step further in that direction is [XDP](https://github.com/iovisor/bpf-docs/blob/master/Express_Data_Path.pdf), which was recently merged into the Linux kernel and allows for [DPDK-like](https://en.wikipedia.org/wiki/Data_Plane_Development_Kit) performance for the kernel itself. The basic idea is that XDP is tightly coupled with eBPF and hooks into a very early ingress path at the driver layer, where it operates with direct access to the packet's [DMA](https://en.wikipedia.org/wiki/Direct_memory_access) buffer.

This is effectively as low-level as it can get to reach near-optimal performance, which mainly allows for tailoring high-performance load balancers or routers with commodity hardware. One advantage that comes with XDP is also that it reuses the kernel's security model for accessing the device as opposed to user space based mechanisms. It doesn't require any third party modules and works in concert with the Linux kernel. Both XDP and tc with eBPF are complementary to each other, and constitute a bigger piece of the puzzle for Cilium itself.

If you’re curious, check out the Cilium code or demos on [GitHub](https://github.com/cilium/cilium).

_By Daniel Borkmann, Cilium contributor_

#### External posts index

521 posts on the cilium.io blog link out to articles hosted elsewhere (community sites, adopter engineering blogs, conference recaps). Their full text is not part of cilium.io, so they are indexed here by title only, newest first.

- 2026-07-31: [Cilium 1.20, Network Policy enhancements, Multi Pool IPAM goes stable, IPv6 progress, and more!](https://isovalent.com/blog/post/cilium-1-20/?utm_medium=referral&utm_campaign=cilium-blog)
- 2026-05-06: [Tetragon 1.7: Precision filtering, richer context, and better performance](https://isovalent.com/blog/post/tetragon-v1.7-release)
- 2026-02-24: [Cilium 1.19, Network Policy enhancements, Multi Pool IPAM goes stable, IPv6 progress, and more!](https://isovalent.com/blog/post/cilium-119-ztunnel-transparent-encryption-multi-pool-ipam-goes-stable-ipv6-progress-and-more?&utm_medium=referral&utm_campaign=cilium-blog)
- 2026-02-10: [Cilium Networking & eBPF-Powered Security Policies on RKE2](https://diabelmehdi.de/cilium-networking-ebpf-powered-security-policies-on-rke2/)
- 2026-01-21: [2026 Kubernetes and Cilium Networking Predictions](https://vmblog.com/archive/2026/01/08/2026-kubernetes-and-cilium-networking-predictions.aspx)
- 2026-01-16: [Day 2 with Cilium: Small configurations that keep large clusters boring](https://www.datadoghq.com/blog/cilium-operations-at-scale/)
- 2025-12-18: [Cilium Annual Report 2025: A Decade of Cilium](https://www.cncf.io/blog/2025/12/18/cilium-releases-2025-annual-report-a-decade-of-cloud-native-networking/)
- 2025-12-03: [Navigating the Ingress-nginx Archival: Why Now Is the Time to Move to Cilium](https://isovalent.com/blog/post/navigating-the-ingress-nginx-archival-why-now-is-the-time-to-move-to-cilium/)
- 2025-11-03: [SuperSkyRay: Running Ray AI Apps Across K8s Clusters with Cilium](https://www.elotl.co/blog/superskyray-part-1-running-ray-ai-apps-across-k8s-clusters-for-resource-and-time-efficiency/)
- 2025-10-31: [Connecting distributed Kubernetes with Cilium and SD-WAN: Building an intelligent network fabric](https://www.cncf.io/blog/2025/10/25/connecting-distributed-kubernetes-with-cilium-and-sd-wan-building-an-intelligent-network-fabric/)
- 2025-10-07: [Cilium Network Policies, from first principles to production](https://veducate.co.uk/cilium-network-policies-from-first-principles-to-production/)
- 2025-08-07: [Cilium 1.18 – Expanded IPv6 Support, Encrypted Overlay, Ingress Bandwidth Controls, Policy Performance Improvements, and More!](https://isovalent.com/blog/post/cilium-1-18//?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2025-07-27: [Uncovering Pod-to-Pod Traffic in Kubernetes Using Cilium and Hubble](https://medium.com/@0.all_existence.0/uncovering-pod-to-pod-traffic-in-kubernetes-using-cilium-and-hubble-0e59a096cce9)
- 2025-07-21: [Help Us Map The State of Kubernetes Networking](https://isovalent.com/blog/post/state-of-kubernetes-networking/)
- 2025-07-03: [Automated Kubernetes Threat Detection with Tetragon and Azure Sentinel](https://medium.com/@saraswathilakshman/automated-kubernetes-threat-detection-with-tetragon-and-azure-sentinel-28c32ee152bc)
- 2025-06-25: [Simplifying Network Management with Cilium’s BGP Auto-Discovery](https://medium.com/@nachyu4/simplifying-network-management-with-ciliums-bgp-auto-discovery-c89e55096078)
- 2025-06-20: [Shift Left with eBPF: Using Tetragon in Your CI/CD Pipeline](https://medium.com/careerbytecode/building-a-professional-grade-devsecops-pipeline-with-tetragon-ebpf-security-monitoring-on-azure-6a36b863ee37)
- 2025-05-25: [CiliumCon North America 2025](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/co-located-events/cfp-colocated-events/#submit-your-talk)
- 2025-01-20: [Why Cilium Is Crushing the Competition as the Go-To CNI for Kubernetes](https://isovalent.com/blog/post/tetragon-ec2-image-builder/)
- 2025-01-14: [Streamlining Tetragon Deployment with Amazon EC2 Image Builder](https://isovalent.com/blog/post/tetragon-ec2-image-builder/)
- 2025-01-06: [Tetragon Configuration Gotchas](https://www.reddit.com/r/RedditEng/comments/1hv3sc7/tetragon_configuration_gotchas/?rdt=35485)
- 2024-12-12: [Telemetry to Tactics: Tetragon Through the Lens of the MITRE ATT&CK Framework](https://isovalent.com/blog/post/mitre-attack-tetragon/)
- 2024-12-05: [First Cilium Pull Request](https://pchaigno.github.io/cilium/2024/12/05/first-cilium-pull-request.html)
- 2024-10-30: [Network Policies: microsegmentação em clusters Kubernetes](https://pedroscloudsecurity.com/microssegmentacao-com-cilium)
- 2024-10-28: [Multi-Cluster with Cilium on Civo cloud](https://everythingdevops.dev/multi-cluster-with-cilium-on-civo-cloud/)
- 2024-10-23: [Spin up a local Kubernetes environment with Kind and Cilium without kube-proxy](https://www.linkedin.com/pulse/spin-up-local-kubernetes-environment-kind-cilium-without-malke-lidrc/?trackingId=ZBrTkm0KTKC8m%2Bko2QhAgw%3D%3D)
- 2024-10-22: [Cilium Cluster Mesh dans AKS](https://blog.cellenza.com/cloud-2/cilium-cluster-mesh-dans-aks/)
- 2024-10-17: [Créer 511 clusters Kubernetes interconnectés avec Cilium Cluster Mesh (Partie 5)](https://medium.com/@littel.jo/cr%C3%A9er-511-clusters-kubernetes-interconnect%C3%A9s-avec-cilium-cluster-mesh-partie-5-881321510704)
- 2024-10-17: [Cilium: Advanced Network Policies and Observability in Kubernetes](https://medium.com/@simardeep.oberoi/cilium-advanced-network-policies-and-observability-in-kubernetes-fbb4fdd747ba)
- 2024-10-16: [Azure Kubernetes Service + Cilium](https://pedroscloudsecurity.com/aks-cilium)
- 2024-10-16: [Debug Cilium toFQDN network policies](https://mcvidanagama.medium.com/debug-cilium-tofqdn-network-policies-b5c4837e3fc4)
- 2024-10-15: [Local Kubernetes LoadBalancer Service Using Cilium BGP](https://kamrul.dev/kubernetes-loadbalancer-cilium-bgp/)
- 2024-10-15: [Interview with Cilium Certified Associate Certification (CCA) creator](https://www.cncf.io/blog/2024/10/15/interview-with-cilium-certified-associate-certification-cca-creator/)
- 2024-10-10: [EKS, Bottlerocket, and Cilium with Pulumi](https://blog.scottlowe.org/2024/10/10/eks-bottlerocket-cilium-with-pulumi/)
- 2024-10-10: [Cilium vs. Calico Comparison](https://www.linkedin.com/pulse/cilium-vs-calico-comparison-tahmid-ul-muntakim-tpqcc/?trackingId=UrFhHvRpSTm4GGVWvuHuiA%3D%3D)
- 2024-10-10: [KubeCon + CloudNativeCon North America 2024 co-located event deep dive: Cilium + eBPF Day](https://www.cncf.io/blog/2024/10/11/kubecon-cloudnativecon-north-america-2024-co-located-event-deep-dive-cilium-ebpf-day/)
- 2024-10-10: [How to Pass CCA (Cilium Certified Associate) Exam](https://medium.com/@nvsajeeva/how-to-pass-cca-cilium-certified-associate-exam-5a1b4f6ca63b)
- 2024-10-05: [Sveltos: Optimising Day-2 Operations with Cilium and Tetragon](https://blog.grosdouli.dev/blog/sveltos-cilium-tetragon-day2-operations)
- 2024-10-04: [Unlocking the Power of eBPF: How Cilium enhances BMC Helix Innovation Suite](https://community.bmc.com/s/news/aA3Kj000000TV4JKAW/unlocking-the-power-of-ebpf-how-cilium-enhances-bmc-helix-innovation-suite)
- 2024-10-04: [Cilium: A Comprehensive Guide to Networking, Security, and Observability in Kubernetes](https://medium.com/@simardeep.oberoi/cilium-a-comprehensive-guide-to-networking-security-and-observability-in-kubernetes-41e11fa69d15)
- 2024-10-02: [Optimizing Enterprise Networks: Addressing Overlapping CIDR with Cilium](https://isovalent.com/blog/post/overlapping-pod-cidr-cilium-cluster-mesh/)
- 2024-10-01: [Now let’s talk about Cilium and how it leverages eBPF](https://medium.com/@addeybob/now-lets-talk-about-cilium-and-how-it-leverages-ebpf-e36bfd98ff53)
- 2024-09-30: [Cilium: IPv6 on EKS](https://medium.com/@amitmavgupta/cilium-ipv6-in-eks-e1fe7735feff)
- 2024-09-27: [Installation et configuration avancée de K3s avec Cilium](https://vecteur.it/installation-et-configuration-avancee-de-k3s-avec-cilium/)
- 2024-09-24: [Apply a Cilium eBGP policy and Redistribute it into an XRd ISIS topology](https://www.linkedin.com/pulse/cilium-michael-buenrostro-o1v6e/)
- 2024-09-21: [Networking in Kubernetes: Part-2 (with Cilium)](https://medium.com/@shreysms31/networking-in-kubernetes-part-2-with-cilium-bc85be2cb3bc)
- 2024-09-19: [FQDN Filtering in AKS: Enhancing Security with Advanced Container Networking Services (ACNS)](https://pixelrobots.co.uk/2024/09/fqdn-filtering-in-aks-enhancing-security-with-advanced-container-networking-services-acns/)
- 2024-09-19: [eBPF - Cilium on FHIR® - A Star Wars Story](https://community.intersystems.com/post/ebpf-cilium-fhir%C2%AE-star-wars-story)
- 2024-09-19: [EP10 (Deep Dive) - How To Migrate From Flannel CNI to Cilium CNI](https://kubecounty.substack.com/p/ep10-deep-dive-how-to-migrate-from)
- 2024-09-17: [70% Faster Deployments and High-Performance Private Network](https://www.koyeb.com/blog/70-faster-deployments-and-high-performance-private-network)
- 2024-09-15: [Real-Time Security Observability with Tetragon in Kubernetes](https://blog.adyog.com/2024/09/15/real-time-security-observability-with-tetragon-in-kubernetes/)
- 2024-09-08: [Understanding eBPF and Its Application in Modern Cloud Environments](https://dev.to/hkhelil/understanding-ebpf-and-its-application-in-modern-cloud-environments-3f99)
- 2024-08-30: [Cluster Mesh with Cilium](https://overcast.blog/cluster-mesh-with-cilium-4f1f7d44c3be)
- 2024-08-30: [Enhancing Security in the Cloud-Edge-IoT Continuum with ICOS](https://www.icos-project.eu/blog/security-with-cilium-tetragon)
- 2024-08-21: [OpenTofu: RKE2 Cluster with Cilium on Azure](https://blog.grosdouli.dev/blog/opentofu-rke2-cilium-azure)
- 2024-08-16: [Proactive Incident Management: Achieving SLA and SLO Success with Tetragon in Kubernetes](https://dyota.substack.com/p/proactive-incident-management-achieving)
- 2024-08-15: [eBPF Security Observability: Top Tetragon Use Cases (Part 2)](https://isovalent.com/blog/post/top-tetragon-use-cases-part-2/)
- 2024-08-10: [Sveltos Templating: Cilium Cluster Mesh in One Run](https://blog.grosdouli.dev/blog/sveltos-templating-cilium-cluster-mesh)
- 2024-08-08: [Demystifying the CNI by Writing One From Scratch](https://isovalent.com/blog/post/demystifying-cni/)
- 2024-08-07: [Talos Kubernetes on Proxmox using OpenTofu](https://blog.stonegarden.dev/articles/2024/08/talos-proxmox-tofu/)
- 2024-08-04: [Setting Up Cilium Networking on EKS Without Default Add-Ons](https://www.codecentric.de/wissens-hub/blog/integrating-dapr-with-cilium-a-sidecar-less-service-mesh-approach-combined-with-a-powerful-distributed-application-runtime)
- 2024-08-02: [Enhancing OKE Security with Cilium Network Policy](https://umashankar-s.medium.com/enhancing-oke-security-with-cilium-network-policy-9535bca0bcc0)
- 2024-08-01: [Integrating Dapr with Cilium: A Sidecar-Less Service Mesh Approach combined with a powerful distributed application runtime](https://www.codecentric.de/wissens-hub/blog/integrating-dapr-with-cilium-a-sidecar-less-service-mesh-approach-combined-with-a-powerful-distributed-application-runtime)
- 2024-08-01: [Setting up cilium cni plugin on a 2-node cluster on x86 using kind](https://medium.com/@pavankalyanmeda5779/setting-up-cilium-cni-plugin-on-a-2-node-cluster-on-x86-using-kind-8a434099e2a7)
- 2024-07-31: [eBPF Security Observability: Top Tetragon Use Cases (Part 1)](https://isovalent.com/blog/post/top-tetragon-use-cases)
- 2024-07-30: [About Cilium native authentication feature](https://blog.teknews.cloud/aks/security/network/2024/07/30/About_Cilium_native_authentication_feature.html)
- 2024-07-30: [Achieving PCI-DSS Compliance With Isovalent, Cilium, and Zero Trust](https://isovalent.com/blog/post/achieving-pci-dss-compliance-with-isovalent-cilium-and-zero-trust/)
- 2024-07-28: [Can Cilium Be a Control Plane Beyond Kubernetes?](https://thenewstack.io/can-cilium-be-a-control-plane-beyond-kubernetes/)
- 2024-07-26: [Architecting Cloud-Native Platforms: The Role of Domain-Driven Design and Cell-Based Architecture](https://cloudnativenow.com/social-facebook/architecting-cloud-native-platforms-the-role-of-domain-driven-design-and-cell-based-architecture/)
- 2024-07-25: [Cilium for AI/ML Workload](https://kubestory.substack.com/p/cilium-for-aiml-workload)
- 2024-07-25: [Cilium 1.16 – High-Performance Networking With Netkit, Gateway API Gamma Support, BGPV2 and More!](https://isovalent.com/blog/post/cilium-1-16/)
- 2024-07-24: [Making Damn Vulnerable Web Application (DVWA) almost unhackable with Cilium and Tetragon](https://holdmybeersecurity.com/2024/07/24/making-damn-vulnerable-web-application-dvwa-almost-unhackable-with-cilium-and-tetragon/)
- 2024-07-17: [Getting Started with Cilium Service Mesh on Amazon EKS](https://aws.amazon.com/blogs/opensource/getting-started-with-cilium-service-mesh-on-amazon-eks/)
- 2024-07-16: [Enabling Enterprise features for Cilium in Elastic Kubernetes Service (EKS)](https://isovalent.com/blog/post/isovalent-enterprise-aws-marketplace/)
- 2024-07-16: [How Open Source Project Tetragon Is Evolving Security via eBPF](https://dzone.com/articles/open-source-project-tetragon-is-evolving-security)
- 2024-07-15: [Cilium on EKS with Sveltos](https://blog.grosdouli.dev/blog/cilium-eks-sveltos)
- 2024-07-14: [Exploring Cilium Network Integration with AWS EKS](https://alpharm.hashnode.dev/exploring-cilium-network-integration-with-aws-eks)
- 2024-07-11: [Cilium netkit: The Final Frontier in Container Networking Performance](https://isovalent.com/blog/post/cilium-netkit-a-new-container-networking-paradigm-for-the-ai-era/)
- 2024-07-11: [Installer Cilium sur WSL2](https://formulenuagique.com/blog/2024/07/11/installer-cilium-sur-wsl2/)
- 2024-07-09: [Cilium BGP Lab, locally!](https://arielantigua.com/weblog/2024/07/cilium-bgp-lab-locally/)
- 2024-07-07: [Cilium no EKS [Lab Session]](https://dev.to/aws-builders/cilium-no-eks-lab-session-1dom')
- 2024-07-07: [Automating Talos Installation on Proxmox with Packer and Terraform, Integrating Cilium and Longhorn](https://surajremanan.com/posts/automating-talos-installation-on-proxmox-with-packer-and-terraform/)
- 2024-07-02: [Cilium: Installing Cilium in GKE with no Kube-Proxy](https://medium.com/@amitmavgupta/cilium-installing-cilium-in-gke-with-no-kube-proxy-826e84f971b4)
- 2024-07-02: [Cilium Gateway API: Bridging Networks Beyond Envoy Proxies](https://codechill.hashnode.dev/cilium-gateway-api-bridging-networks-beyond-envoy-proxies)
- 2024-06-30: [Run your Kubernetes Cluster on “Bare Metal” with Cilium CNI — Part. 1](https://medium.com/@aristeidis/run-your-kubernetes-cluster-on-bare-metal-with-cilium-cni-part-1-e88028800d90)
- 2024-06-29: [Cilium, Gateway API, Cert-Manager and Let’s Encrypt — Updates](https://itnext.io/cilium-gateway-api-cert-manager-and-lets-encrypt-updates-cc730818cb17)
- 2024-06-24: [Cilium: Open-source eBPF-based networking, security, observability](https://www.helpnetsecurity.com/2024/06/21/cilium-open-source-ebpf-based-networking-security-observability/)
- 2024-06-24: [Enabling Multicast securely with IPsec in the cloud native landscape with Cilium](https://isovalent.com/blog/post/cilium-multicast-cloud/)
- 2024-06-17: [Cloud Native-nätverk med Cilium](https://www.ip-solutions.se/blogg/cloud-native-natverk-med-cilium/)
- 2024-06-16: [Magical ability to peek inside running Kubernetes Cluster](https://www.linkedin.com/pulse/magical-ability-peek-inside-running-kubernetes-cluster-devidas-jadhav-t4eyf/)
- 2024-06-10: [K8s on a Shoestring: Building a Production-Ready Cluster from a Rusty Laptop](https://blog.devgenius.io/k8s-on-a-shoestring-building-a-production-ready-cluster-from-a-rusty-laptop-1ae8b963d557)
- 2024-06-08: [EKS Anywhere, jiving with Cilium OSS and BGP Load Balancer](https://ambar-thecloudgarage.medium.com/eks-anywhere-jiving-with-cilium-oss-and-bgp-load-balancer-12af1d10099c)
- 2024-06-06: [Créer 511 clusters Kubernetes interconnectés avec Cilium Cluster Mesh (Partie 1)](https://medium.com/@littel.jo/cr%C3%A9er-511-clusters-kubernetes-en-cilium-cluster-mesh-partie-1-9d1868a102e7)
- 2024-06-06: [AKS & Cilium, une histoire d’amour?](https://blog.cellenza.com/cloud-2/aks-cilium-une-histoire-damour/)
- 2024-06-06: [How to Deploy Cilium and Egress Gateway in Azure Kubernetes Service (AKS)](https://isovalent.com/blog/post/cilium-egress-gateway-aks/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-06-04: [KubeEdge meets Cilium](https://kubeedge.io/blog/enable-cilium/)
- 2024-06-03: [Cloud-Native Security and Networking with Liz Rice](https://www.theiacpodcast.com/episode/cloud-native-security-and-networking-with-liz-rice)
- 2024-05-29: [Kubernetes Networking for Network Engineers - C2C034](https://www.cables2clouds.com/2129055/15152709-kubernetes-networking-for-network-engineers-c2c034)
- 2024-05-28: [Understanding Cilium, eBPF, and Integrating with Prometheus/Grafana](https://alpharm.hashnode.dev/understanding-cilium-ebpf-and-integrating-with-prometheusgrafana?ref=twitter-share)
- 2024-05-28: [Cluster mesh with Cilium in AKS](https://blog.teknews.cloud/aks/network/2024/05/28/Testing_Cilium_Cluster_Mesh_in_AKS.html)
- 2024-05-23: [Istio to Cilium: a grand yak-shave](https://blog.goorzhel.com/istio-to-cilium-a-grand-yak-shave/)
- 2024-05-23: [Tetragon Process Lifecycle Observation: Tetragon Agent Part](https://yuki-nakamura.com/2024/05/23/tetragon-process-lifecycle-observation-tetragon-agent-part/)
- 2024-05-21: [Considérations réseaux pour Azure Kubernetes Services (partie 2)](https://blog.cellenza.com/cloud-2/considerations-reseaux-pour-azure-kubernetes-services-partie-2/)
- 2024-05-21: [Networks Are Under AI Pressure: Can Cilium Provide Relief?](https://isovalent.com/blog/post/cilium-the-network-and-security-platform-for-the-cloud-native-ai-era/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-05-16: [CILIUM - SERVICE MESH](https://www.linkedin.com/pulse/cilium-service-mesh-hudson-coutinho-wjhvf/)
- 2024-05-16: [Introducing the Isovalent Lab Champion Program](https://isovalent.com/blog/post/cilium-lab-champion/)
- 2024-05-16: [Multi cluster networking with Cilium Cluster Mesh](https://nomadxd.github.io/blog/multi-cluster-networking-with-cilium-cluster-mesh)
- 2024-05-09: [Cilium - Kubernetes CNI ที่ดีที่สุดที่เป็นมากกว่า CNI](https://nopnithi.com/posts/cilium-the-best-kubernetes-cni-and-beyond/)
- 2024-05-08: [Service Meshes Decoded: a performance comparison of Istio vs Linkerd vs Cilium](https://livewyer.io/blog/2024/05/08/comparison-of-service-meshes/)
- 2024-05-07: [Cilium: the future of cloud native network solutions?](https://devops.cisel.ch/cilium-the-future-of-cloud-native-network-solutions)
- 2024-05-06: [cert-manager: All-in-One Kubernetes TLS Certificate Manager](https://developer-friendly.blog/2024/05/06/cert-manager-all-in-one-kubernetes-tls-certificate-manager/)
- 2024-05-03: [Cilium’s Past Points to Its Future](https://thenewstack.io/ciliums-past-points-to-its-future/)
- 2024-05-03: [Context-aware security incident response with Dynatrace Automations and Tetragon](https://www.dynatrace.com/news/blog/context-aware-security-incident-response/)
- 2024-05-03: [Isovalent Strengthens Cloud Native Security via Tetragon Enterprise Update](https://cloudnativenow.com/news/isovalent-strengthens-cloud-native-security-via-tetragon-enterprise-update/)
- 2024-05-03: [Mastering Kubernetes: Deploy Cilium on Azure Kubernetes Service with a Star Wars Demo](https://www.iamachs.com/p/kubernetes-networking/part-4-aks-cilium-star-wars-demo/)
- 2024-05-03: [Supercharge Your Kubernetes Networking with Cilium and eBPF](https://www.iamachs.com/p/kubernetes-networking/part-3-supercharge-with-cilium-ebpf/)
- 2024-05-02: [CILIUM - Redes e Políticas](https://www.linkedin.com/pulse/cilium-redes-e-pol%25C3%25ADticas-hudson-coutinho-ckhlf/)
- 2024-05-01: [Cilium Cheat Sheet – Master the Cilium CLI to manage and configure your Kubernetes Network](https://isovalent.com/blog/post/cilium-cheat-sheet/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-05-01: [Cilium: Network Policies](https://jamali.hashnode.dev/cilium-network-policies)
- 2024-05-01: [Kubernetes Gateway API with Cilium](https://kubito.dev/posts/kubernetes-gateway-api-cilium/)
- 2024-04-30: [How is Kubernetes Network Security Evolving?](https://www.cloudsecuritypodcast.tv/videos/how-is-kubernetes-network-security-evolving)
- 2024-04-30: [Live Migrating Production Clusters From Calico to Cilium](https://samsungads.ca/engineering-blog/live-migrating-production-clusters-from-calico-to-cilium/)
- 2024-04-26: [VMblog Expert Interview: Nico Vibert of Isovalent Discusses New eBook: Kubernetes Networking and Cilium for the Network Engineer](https://vmblog.com/archive/2024/04/26/vmblog-expert-interview-nico-vibert-of-isovalent-discusses-new-ebook-kubernetes-networking-and-cilium-for-the-network-engineer.aspx)
- 2024-04-24: [Benchmark results of Kubernetes network plugins (CNI) over 40Gbit/s network [2024]](https://itnext.io/benchmark-results-of-kubernetes-network-plugins-cni-over-40gbit-s-network-2024-156f085a5e4e)
- 2024-04-24: [Detect XZ Utils CVE 2024-3094 with Tetragon](https://www.dbi-services.com/blog/detect-xz-utils-cve-2024-3094-with-tetragon/)
- 2024-04-16: [Dual Stack on AKS with Cilium](https://isovalent.com/blog/post/cilium-dual-stack-aks/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-04-16: [Unveiling the Simplicity of Cluster Mesh for Kubernetes Deployments](https://dev.to/camptocamp-ops/unveiling-the-simplicity-of-cluster-mesh-for-kubernetes-deployments-1bfc)
- 2024-04-12: [Cisco Completes Acquisition of Isovalent to Define the Future of Multicloud Networking and Security](https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2024/m04/cisco-completes-acquisition-of-isovalent-to-define-the-future-of-multicloud-networking-and-security.html)
- 2024-04-09: [Use Cilium to provide networking services to containers inside Oracle Container Engine for Kubernetes](https://www.iwanhoogendoorn.nl/index.php/Use_Cilium_to_provide_networking_services_to_containers_inside_Oracle_Container_Engine_for_Kubernetes_-_OKE_-)
- 2024-04-09: [Cilium Hubble CLI – Using a local configuration file](https://veducate.co.uk/cilium-hubble-cli-using-a-local-configuration-file/)
- 2024-04-09: [Improve your EKS cluster with Istio and Cilium](https://medium.com/@seifeddinerajhi/improve-your-eks-cluster-with-istio-and-cilium-d7771c009d66)
- 2024-04-09: [Sortie de la version 0.2 du provider terraform pour cilium](https://medium.com/@littel.jo/sortie-de-la-version-0-2-du-provider-terraform-pour-cilium-702a15a7c5f6)
- 2024-04-04: [eBPF & Tetragon: Tools for detecting XZ Utils CVE 2024-3094 Exploit](https://isovalent.com/blog/post/ebpf-tetragon-xz-utils-cve-policy/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-04-03: [Exploring Network Policies with Cilium](https://www.civo.com/learn/network-policies-with-cilium)
- 2024-03-31: [Extending a service using Private Link from Azure and securing it with Cilium’s Network Policy](https://medium.com/@amitmavgupta/extending-a-service-using-private-link-from-azure-and-securing-it-with-ciliums-network-policy-ae3248281bbd)
- 2024-03-29: [AKS and Cilium 101](https://blog.teknews.cloud/aks/network/2024/03/29/AKS_and_Cilium_101.html)
- 2024-03-29: [KubeCon EU Highlights: CloudEvents & Falco Graduate, Beta Tetragon, Linkerd Meshes Legacy Systems](https://www.infoq.com/news/2024/03/kubecon-cncf-graduated-projects/)
- 2024-03-27: [KubeCon Europe 2024 Wrap-Up](https://isovalent.com/blog/post/kubecon-europe-2024-wrap-up/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-03-25: [La mutualisation de services sur plusieurs clusters avec Cilium et Microsoft AKS](https://www.linkedin.com/pulse/la-mutualisation-de-services-sur-plusieurs-clusters-avec-leclerc-v9qfe/)
- 2024-03-22: [How to Install Lightweight Kubernetes on Ubuntu 22.04](https://developer-friendly.blog/2024/03/22/how-to-install-lightweight-kubernetes-on-ubuntu-2204/)
- 2024-03-21: [Cilium: The De Facto Kubernetes Networking Layer and Its Exciting Future](https://www.insightsfromanalytics.com/post/cilium-the-de-facto-kubernetes-networking-layer-and-its-exciting-future?postId=65f8d070e3a3816db10b3d71)
- 2024-03-21: [Cilium: Installing Cilium in EKS with no Kube-Proxy](https://medium.com/@amitmavgupta/cilium-installing-cilium-in-eks-with-no-kube-proxy-86f54a56c360)
- 2024-03-21: [Tutorial: Using The Network Policy Editor (Part 3)](https://isovalent.com/blog/post/tutorial-network-policy-editor/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-enterprise)
- 2024-03-20: [Cilium at the Edge](https://isovalent.com/blog/post/cilium-at-the-edge/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-enterprise)
- 2024-03-19: [Kubernetes Network Policy by using Cilium – Beginner Level](https://www.dbi-services.com/blog/kubernetes-network-policy-by-using-cilium-beginner-level/)
- 2024-03-18: [Cilium: The De Facto Kubernetes Networking Layer and Its Exciting Future](https://dzone.com/articles/cilium-the-de-facto-kubernetes-networking-layer-an)
- 2024-03-15: [Tutorial: Cilium Network Policy in Practice (Part 2)](https://isovalent.com/blog/post/tutorial-cilium-network-policy/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-enterprise)
- 2024-03-14: [What’s New With Cilium and What’s Next?](https://cloudnativenow.com/features/whats-new-with-cilium-and-whats-next/)
- 2024-03-13: [Solo.io's Gloo Network for Cilium is generally available](https://www.solo.io/blog/accelerating-cilium-adoption-with-gloo-network-for-cilium/)
- 2024-03-13: [File Monitoring with eBPF and Tetragon (Part 1)](https://isovalent.com/blog/post/file-monitoring-with-ebpf-and-tetragon-part-1/)
- 2024-03-13: [Solo.io Announces General Availability of Gloo Network for Cilium](https://www.apmdigest.com/soloio-announces-general-availability-of-gloo-network-for-cilium)
- 2024-03-13: [Solo.io Delivers on Cilium Support Promise for Gloo Networks](https://cloudnativenow.com/features/solo-io-delivers-on-cilium-support-promise-for-gloo-networks/)
- 2024-03-13: [Solo.io launches Gloo Network for Cilium to help customers improve network management](https://www.itopstimes.com/itops/solo-io-launches-gloo-network-for-cilium-to-help-customers-improve-network-management/)
- 2024-03-13: [Solo.io Announces General Availability of Gloo Network for Cilium to Streamline Integration Across Hybrid Cloud Environments](https://vmblog.com/archive/2024/03/13/solo-io-announces-general-availability-of-gloo-network-for-cilium-to-streamline-integration-across-hybrid-cloud-environments.aspx)
- 2024-03-11: [Introduction to Cilium Network Policies](https://isovalent.com/blog/post/intro-to-cilium-network-policies/)
- 2024-03-11: [Kubernetes Network Security with Cilium](https://www.sheddy.xyz/blog/intro-to-kubernetes-networking-with-cilium)
- 2024-03-06: [Cilium Certified Associate (CCA) Launches](https://isovalent.com/blog/post/intro-to-cilium-network-policies/)
- 2024-03-06: [“ServiceLB” with cilium on Talos Linux](https://blog.devgenius.io/servicelb-with-cilium-on-talos-linux-8a290d524cb7)
- 2024-03-06: [Cilium, Azure, Rancher & Terraform: let’s call it CART](https://medium.com/@amitmavgupta/cilium-azure-rke2-terraform-lets-call-it-cart-f4c0b8dadf4e)
- 2024-03-05: [Installing Cilium with ArgoCD on GKE](https://www.solo.io/blog/installing-cilium-argocd-gke/)
- 2024-03-05: [Kubernetes Networking by Using Cilium – Advanced Level – eBPF Routing](https://www.dbi-services.com/blog/kubernetes-networking-by-using-cilium-advanced-level-ebpf-routing/)
- 2024-02-29: [Bootstrapping K3s with Cilium](https://blog.stonegarden.dev/articles/2024/02/bootstrapping-k3s-with-cilium/)
- 2024-02-29: [Play with Cilium native routing in Kind cluster](https://medium.com/@nahelou.j/play-with-cilium-native-routing-in-kind-cluster-5a9e586a81ca)
- 2024-02-28: [S&P Global eliminates networking and developer team silos with Cilium](https://isovalent.com/blog/post/eliminnate-team-silos-with-cilium/)
- 2024-02-26: [Install Cilium and Customize](https://www.linkedin.com/pulse/install-cilium-customize-rong-li-etlce/)
- 2024-02-26: [OpenShift Deployment on AWS with Cilium: A Practical Guide](https://medium.com/@eleni.grosdouli/openshift-deployment-on-aws-with-cilium-a-practical-guide-5dd97566e1df)
- 2024-02-26: [Building a MultiCluster Environment with Cilium on BareMetal Kubernetes Cluster: A Comprehensive Guide](https://cloudopskube.com/blog/post/building-a-multicluster-environment-with-cilium-on-baremetal-kubernetes-cluster-a-comprehensive-guide)
- 2024-02-22: [Cilium on a Private AKS cluster](https://isovalent.com/blog/post/cilium-private-aks-cluster/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-02-21: [Isovalent Enterprise for Tetragon: Deeper Host Network Observability with eBPF](https://isovalent.com/blog/post/tetragon-network-observability-dashboards/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-02-20: [Choosing the Right Routing in Cilium](https://www.solo.io/blog/choosing-right-routing-cilium/)
- 2024-02-20: [Kubernetes Networking by Using Cilium – Intermediate Level – Part-1](https://www.dbi-services.com/blog/kubernetes-networking-by-using-cilium-intermediate-level-part-1/)
- 2024-02-14: [Cilium: Evaluating pod identities on an AKS cluster running Azure CNI powered by Cilium](https://medium.com/@amitmavgupta/cilium-evaluating-workload-identities-on-an-aks-cluster-running-azure-cni-powered-by-cilium-14b0d3ab65a1)
- 2024-02-14: [Kubernetes Networking by Using Cilium – Beginner Level](https://www.dbi-services.com/blog/kubernetes-networking-by-using-cilium-beginner-level/)
- 2024-02-11: [Security and observability with Cilium on my 5G network](https://luislogs.com/posts/security-and-observability-with-cilium-on-my-5g-network/)
- 2024-02-06: [Enabling Cluster Mesh on an AKS cluster from Azure Marketplace](https://isovalent.com/blog/post/cilium-cluster-mesh-in-aks/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-02-06: [Migration from Calico CNI to Cilium CNI in BareMetal Kubernetes Cluster and Monitoring traffic using Hubble UI](https://dev.to/gittest20202/migration-from-calico-to-cilium-in-baremetal-kubernetes-cluster-and-monitoring-traffic-using-hubble-ui-40ga)
- 2024-02-06: [Taming Tetragon With Cribl.Cloud](https://cribl.io/blog/taming-tetragon-with-cribl-cloud/)
- 2024-02-03: [ArgoCD Deployment on RKE2 with Cilium Gateway API](https://medium.com/@eleni.grosdouli/argocd-deployment-on-rke2-with-cilium-gateway-api-ab1769cc28a3)
- 2024-02-02: [Installing Cilium](https://observability-360.com/docs/ViewDocument?id=cilium-aks-getting-started)
- 2024-02-01: [Cilium 1.15 – Gateway API 1.0 Support, Cluster Mesh Scale Increase, Security Optimizations and more!](https://isovalent.com/blog/post/cilium-1-15/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-02-01: [Cilium Hubble now included with DigitalOcean Kubernetes at no cost](https://www.digitalocean.com/blog/cillium-hubble-on-digitalocean-kubernetes)
- 2024-01-26: [Tutorial: Redirect, Rewrite and Mirror HTTP with Cilium Gateway API](https://isovalent.com/blog/post/tutorial-redirect-rewrite-and-mirror-http-requests-with-cilium-gateway-api/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-01-25: [Google Cloud Logging and Cloud Monitoring example with Tetragon](https://medium.com/google-cloud/google-cloud-logging-and-cloud-monitoring-example-with-tetragon-5eb2012066d4)
- 2024-01-24: [Isovalent Enterprise for Cilium and Azure Arc: solving the multi-cloud cluster manageability conundrum](https://isovalent.com/blog/post/cilium-and-azure-arc/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-01-24: [Unlocking Kubernetes Simplicity with RKE2 and Empowering Network Security with Cilium](https://www.linkedin.com/pulse/unlocking-kubernetes-simplicity-rke2-empowering-network-lopez-wjqnc/)
- 2024-01-24: [Fixing Cilium on Kind](https://www.robert-jensen.dk/posts/2024-fixing-cilium-with-kind/)
- 2024-01-24: [A closer look at Ingress, Gateway API and TLS certificates with Cilium and cert-manager](https://www.viktorious.nl/2024/01/25/a-closer-look-at-ingress-gateway-api-and-tls-certificates-with-cilium-and-cert-manager/)
- 2024-01-22: [Pure Cilium: A Guide for Local Load Balancing and BGP](https://blog.kubesimplify.com/pure-cilium-a-guide-for-local-load-balancing-and-bgp)
- 2024-01-19: [Multi-Kubernetes cluster connectivity with OKE and Cilium for stateful workloads on Oracle Cloud](https://medium.com/oracledevs/multi-kubernetes-cluster-connectivity-with-oke-and-cilium-for-stateful-workloads-on-oracle-cloud-763da3139843)
- 2024-01-18: [The value of Cilium backports](https://isovalent.com/blog/post/value-of-cilium-backports/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-01-18: [VMblog Expert Interview with Shedrack Akintayo, Cilium and eBPF Community Manager at Isovalent](https://vmblog.com/archive/2024/01/18/vmblog-expert-interview-with-shedrack-akintayo-cilium-and-ebpf-community-manager-at-isovalent.aspx)
- 2024-01-17: [Cisco buy highlights container networking, security](https://www.networkworld.com/article/1291149/cisco-buy-highlights-container-networking-security.html)
- 2024-01-15: [Cilium Week Recap: Preparing for Cilium Certification](https://www.solo.io/blog/preparing-cilium-certification/)
- 2024-01-14: [Cilium: Next generation Networking & Security of CNCF with Golang, eBPF & Hubble](https://codechill.hashnode.dev/cilium-next-generation-networking-security-of-cncf-with-golang-ebpf-hubble)
- 2024-01-14: [Kubernetes on FreeBSD with Linux worker nodes and Cilium](https://medium.com/@norlin.t/kubernetes-on-freebsd-with-linux-worker-nodes-and-cilium-a87c50daef03)
- 2024-01-12: [Demystifying Cilium: Learn How to Build an eBPF CNI Plugin from Scratch](https://www.solo.io/blog/cilium-build-ebpf-cni-plugin/)
- 2024-01-12: [Learned Cilium Network Policies](https://mischavandenburg.com/zet/cilium-network-policies/)
- 2024-01-11: [Scaling Cilium to New Heights With xDS](https://www.solo.io/blog/scaling-cilium-to-new-heights-with-xds/)
- 2024-01-10: [Networking and eBPF Predictions for 2024 and Beyond](https://isovalent.com/blog/post/networking-and-ebpf-predictions-for-2024/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2024-01-10: [Expose Loadbalanced Kubernetes Services With Cilium](https://sue.eu/blogs/expose-loadbalanced-kubernetes-services-with-bgp-cilium/)
- 2024-01-05: [Setup basic L4 Load Balancing with Cilium CNI and Ubuiqiti Edge Router](https://www.viktorious.nl/2024/01/05/setup-basic-l4-load-balancing-with-cilium-cni-and-ubuiqiti-edge-router/)
- 2024-01-04: [Exclusive interview: Santa switches to Cilium](https://qmonnet.github.io/whirl-offload/2024/01/02/santa-switches-to-cilium/)
- 2024-01-04: [How to migrate from Red Hat OpenShiftSDN/OVN-Kubernetes to Cilium](https://veducate.co.uk/migrate-red-hat-openshiftsdn-ovn-kubernetes-cilium/)
- 2024-01-03: [Cilium dual-stack k3s pi-setup](https://www.redpill-linpro.com/techblog/2024/01/03/cilium-dual-stack-k3s-pi-setup.html)
- 2024-01-02: [Gateway API with Cilium and Cert-manager](https://blog.stonegarden.dev/articles/2023/12/cilium-gateway-api/)
- 2024-01-02: [Top Technology Trends to Watch Out for in 2024 for Platform Engineering and SRE ](https://medium.com/@seifeddinerajhi/top-technology-trends-to-watch-out-for-in-2024-for-platform-engineering-and-sre-e267130c7e22)
- 2024-01-01: [As Cisco swoops on Cilium, community efforts look critical](https://www.thestack.technology/cisco-cilium-isovalent/)
- 2024-01-01: [Container Network Interface (CNI) in Kubernetes: Ciliums Rolle und die Übernahme von Isovalent durch Cisco](https://www.ayedo.de/posts/cni-in-kubernetes-isovalents-rolle-und-die-uebernahme-von-cisco/)
- 2023-12-27: [Cilium: Pod Sandboxing in AKS and Azure CNI powered by Cilium](https://medium.com/@amitmavgupta/cilium-pod-sandboxing-in-aks-and-azure-cni-powered-by-cilium-b6586e831c64)
- 2023-12-26: [Three Cloudnative Projects to Watch in 2024](https://dbodky.me/blog/three-cloudnative-projects-2024/)
- 2023-12-22: [Cisco to Acquire Isovalent to Secure Cloud-Native Networking](https://www.channele2e.com/news/cisco-to-acquire-isovalent-to-secure-cloud-native-networking)
- 2023-12-22: [Cisco Bets Big on Multicloud Security With Isovalent Deal](https://www.darkreading.com/cloud-security/cisco-bets-big-on-multicloud-security-with-isovalent-deal)
- 2023-12-22: [Cisco signals open source intentions with Isovalent acquisition](https://www.itpro.com/software/open-source/cisco-signals-open-source-intentions-with-isovalent-acquisition)
- 2023-12-22: [Cisco snaps up open-source networking and security start-up](https://www.siliconrepublic.com/business/cisco-isovalent-acquisition-open-source-security)
- 2023-12-22: [Cisco buys Isovalent for multi-cloud](https://www.silverliningsinfo.com/multi-cloud/cisco-buy-isovalent-multi-cloud)
- 2023-12-22: [Cisco goes Christmas shopping, buys Cilium project originator Isovalent](https://www.theregister.com/2023/12/22/cisco_acquires_isovalent/)
- 2023-12-22: [Why Cisco is Paying a High Price to Acquire a Cloud Networking Startup](https://www.theinformation.com/articles/why-cisco-is-paying-a-high-price-to-acquire-a-cloud-networking-startup)
- 2023-12-21: [Cisco Acquires Isovalent to Advance Cloud-Native Networking Based on eBPF](https://cloudnativenow.com/features/cisco-acquires-isovalent-to-advance-cloud-native-networking-based-on-ebpf/)
- 2023-12-21: [Cisco to Acquire Cloud Native Networking & Security Leader Isovalent](https://isovalent.com/blog/post/cisco-acquires-isovalent/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2023-12-21: [Cisco to Acquire Isovalent to Define the Future of Multicloud Networking and Security](https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2023/m12/cisco-to-acquire-isovalent-to-define-the-future-of-multicloud-networking-and-security.html)
- 2023-12-21: [Defining the Future of Multicloud Networking and Security: Cisco Announces Intent to Acquire Isovalent](https://blogs.cisco.com/news/defining-the-future-of-multicloud-networking-and-security-cisco-announces-intent-to-acquire-isovalent)
- 2023-12-21: [Cisco Buys Cilium Project Creator Isovalent To Boost Security Cloud](https://www.crn.com/news/networking/cisco-buys-cilium-project-creator-isovalent-to-boost-cisco-security-cloud)
- 2023-12-21: [Cisco to acquire Isovalent for open source cloud native networking and security](https://www.sdxcentral.com/articles/news/cisco-to-acquire-isovalent-for-open-source-cloud-native-networking-and-security/2023/12/)
- 2023-12-21: [Cisco acquires eBPF networking startup Isovalent](https://siliconangle.com/2023/12/21/cisco-acquires-ebpf-networking-startup-isovalent/)
- 2023-12-21: [Cisco to acquire cloud native networking and security startup Isovalent](https://techcrunch.com/2023/12/21/cisco-to-acquire-cloud-native-networking-and-security-startup-isovalent/)
- 2023-12-21: [Cisco Security Cloud adds Isovalent for multi-cloud networks](https://www.techtarget.com/searchitoperations/news/366564394/Cisco-Security-Cloud-adds-Isovalent-for-multi-cloud-networks)
- 2023-12-21: [Revolutionizing Kubernetes: Unleashing the Power of Cilium and eBPF for Enhanced Observability, Security, and Cross-Cluster Communication](https://www.linkedin.com/pulse/revolutionizing-kubernetes-unleashing-power-cilium-ebpf-tony-lagumen-st62c%3F/)
- 2023-12-20: [Tetragon Cloud Native Security](https://rodrigtech.com/tetragon-cloud-native-security/)
- 2023-12-18: [A Quick Glance at Cilium CNI](https://blog.andreasm.io/2023/12/18/a-quick-glance-at-cilium-cni/)
- 2023-12-18: [From Reactive to Proactive: Transforming Security with Tetragon](https://blog.shellnetsecurity.com/2023/12/698/kubernetes/from-reactive-to-proactive-transforming-security-with-tetragon/)
- 2023-12-07: [Cilium, eBPF and Beyond](https://openatintel.podbean.com/e/cilium-ebpf-and-beyond/)
- 2023-12-03: [Cilium Cluster Mesh + CoreDNS](https://medium.com/@mattiaforc/cilium-cluster-mesh-coredns-17e0c4f108b4)
- 2023-12-01: [The Big FIFO in the Cloud](https://blog.tohojo.dk/2023/12/the-big-fifo-in-the-cloud.html)
- 2023-12-01: [Migrating from MetalLB to Cilium](https://blog.stonegarden.dev/articles/2023/12/migrating-from-metallb-to-cilium/)
- 2023-11-30: [Auditing Tetragon logs for better eBPF log analytics in Kubernetes clusters](https://medium.com/@seifeddinerajhi/auditing-tetragon-logs-for-better-ebpf-log-analytics-in-kubernetes-clusters-08bf3cd06224)
- 2023-11-29: [A Season Of Cilium: One Surprise Every Day](https://isovalent.com/blog/post/a-season-of-cilium-one-surprise-per-day/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2023-11-28: [Isovalent Enterprise for Cilium on EKS & EKS-A in AWS Marketplace](https://isovalent.com/blog/post/isovalent-aws-marketplace/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2023-11-25: [Cilium: Fixed IP allocation vs. Prefix delegation in AKS](https://medium.com/@amitmavgupta/cilium-fixed-ip-allocation-vs-prefix-delegation-in-aks-873a6700a4ba)
- 2023-11-25: [Transform Your Raspberry Pi: Lightweight Power with k3s and Cilium!](https://medium.com/@juanfbl9307/transform-your-raspberry-pi-lightweight-power-with-k3s-and-cilium-a43c691346ef)
- 2023-11-19: [Connecting your Kubernetes island to your network with Cilium BGP](https://isovalent.com/blog/post/connecting-your-kubernetes-island-to-your-network-with-cilium-bgp/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2023-11-17: [Cilium: The IPAM conundrum-AKS](https://medium.com/@amitmavgupta/cilium-where-are-my-ip-addresses-in-aks-on-azpc-b8f567a2ce68)
- 2023-11-16: [KubeCon North America 2023 Wrap-Up](https://isovalent.com/blog/post/kubecon-north-america-2023-wrap-up/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2023-11-15: [Isovalent, Azure Linux, and Azure Kubernetes Service come together](https://isovalent.com/blog/post/isovalent-azure-linux/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2023-11-15: [Tetragon 1.0 Promises a New Era of Kubernetes Security and Observability](https://thenewstack.io/tetragon-1-0-promises-a-new-era-of-kubernetes-security-and-observability/)
- 2023-11-13: [Cilium Chronicles: Personally Navigating the Labyrinth of Kubernetes Networking](https://www.linkedin.com/pulse/cilium-chronicles-personally-navigating-labyrinth-tony-lagumen-q1llc/?trackingId=fZUNtiMrTnumWOnkmsAlqA%3D%3D)
- 2023-11-13: [Cilium: Multi-Cluster mesh Setup Explained Part 2](https://cloudwithjk.hashnode.dev/cilium-multi-cluster-mesh-setup-explained-part-2)
- 2023-11-10: [Kubernetes security remains a big challenge for enterprise developers](https://siliconangle.com/2023/11/10/kubernetes-security-remains-big-challenge-enterprise-developers/)
- 2023-11-10: [It is time to let go on your service mesh dream](https://kubestory.substack.com/p/it-is-time-to-let-go-on-your-service)
- 2023-11-10: [Tetragon adds visibility to Kubernetes with open-source runtime security platform](https://www.sdxcentral.com/articles/analysis/tetragon-adds-visibility-to-kubernetes-with-open-source-runtime-security-platform/2023/11/)
- 2023-11-06: [COMING SOON! Cilium Certified Associate (CCA)](https://training.linuxfoundation.org/blog/coming-soon-cilium-certified-associate-cca/)
- 2023-11-06: [All About The New CNCF Cilium Certified Associate (CCA) Certification](https://isovalent.com/blog/post/cilium-certified-associate-cca/)
- 2023-11-06: [Isovalent Extends Cilium’s Enterprise Networking and Security Features at KubeCon](https://cioinfluence.com/edge-computing/isovalent-extends-ciliums-enterprise-networking-and-security-features-at-kubecon/)
- 2023-11-03: [Securing Cilium policies with Kyverno](https://kubestory.substack.com/p/securing-cilium-policies-with-kyverno)
- 2023-11-03: [KubeCon points to the future of enterprise IT](https://www.infoworld.com/article/3709930/kubecon-points-to-the-future-of-enterprise-it.html)
- 2023-11-02: [Catch the Cilium Wave at KubeCon+CloudNativeCon Chicago](https://www.solo.io/blog/cilium-kubecon-2023/)
- 2023-11-02: [The Big Interview: Thomas Graf, CTO, Isovalent, on eBPF, cloud-native networking and why Cilium is so hot right now](https://www.thestack.technology/thomas-graf-cto-isovalent-ebpf-cilium/)
- 2023-11-01: [Combining Powerful Cilium Features Together In Kubernetes](https://www.dbi-services.com/blog/combining-powerful-cilium-features-together-in-kubernetes/)
- 2023-11-01: [Isovalent, Creators of eBPF and Cilium, Launch Tetragon: High-Performance Kernel-based Kubernetes Security](https://www.prnewswire.com/news-releases/isovalent-creators-of-ebpf-and-cilium-launch-tetragon-high-performance-kernel-based-kubernetes-security-301973670.html9)
- 2023-11-01: [Testing Cilium Cluster Mesh installation/configuration using Helm Charts (GitOps approach) — part 1](https://medium.com/@vadimzharov/testing-cilium-cluster-mesh-installation-configuration-using-helm-charts-gitops-approach-part-1-4fef93012631)
- 2023-11-01: [Testing Cilium Cluster Mesh installation/configuration using Helm Charts (GitOps approach) — part 2](https://medium.com/@vadimzharov/testing-cilium-cluster-mesh-installation-configuration-using-helm-charts-gitops-approach-part-2-3ad3c7428469)
- 2023-10-31: [How We Implemented Zero Trust in Choreo](https://wso2.com/library/blogs/how-we-implemented-zero-trust-in-choreo/)
- 2023-10-31: [Showing Blocked Requests in Notebooks with Cilium Hubble](https://www.singlestore.com/blog/showing-blocked-requests-in-notebooks-with-cilium-hubble/)
- 2023-10-31: [Tetragon 1.0: Kubernetes Security Observability & Runtime Enforcement with eBPF](https://isovalent.com/blog/post/tetragon-release-10/)
- 2023-10-25: [Migrating Cilium from Legacy iptables Routing to Native eBPF Routing in Production](https://deezer.io/migrating-cilium-from-legacy-iptables-routing-to-native-ebpf-routing-in-production-84a035af1cd6)
- 2023-10-24: [Kubernetes Community Days UK: Keynote Cilium and eBPF](https://percona.community/blog/2023/10/24/kcduk-cilium-ebpf/)
- 2023-10-23: [Isovalent Enterprise for Cilium 1.14: Introducing Cilium Multi-Network](https://isovalent.com/blog/post/isovalent-enterprise-for-cilium-1-14-multi-network/)
- 2023-10-21: [Kubernetes multi-cluster implementation in under 10 minutes](https://akyriako.medium.com/kubernetes-multi-cluster-implementation-in-under-10-minutes-2927952fb84c )
- 2023-10-20: [Migration du routage de cilium de iptables vers eBPF... à chaud](https://blog.zwindler.fr/2023/10/20/migration-routage-cilium-iptables-ebpf/)
- 2023-10-19: [CNCF firms up Cilium cell structure](https://www.computerweekly.com/blog/Open-Source-Insider/CNCF-firms-up-Cilium-cell-structure)
- 2023-10-19: [Linuxカーネル内部をフックするeBPFを用いてコンテナ間通信を実現する「Cilium」、十分成熟したソフトウェアに到達したとして、CNCFの卒業プロジェクトに](https://www.publickey1.jp/blog/23/linuxebpfciliumcncf.html)
- 2023-10-19: [The Future of Kubernetes: Rancher RKE2 and Cilium CNI](https://levelup.gitconnected.com/the-future-of-kubernetes-rancher-rke2-and-cilium-cni-1b9072eadea1)
- 2023-10-18: [Get started with eBPF log analytics in your Kubernetes cluster](https://www.parseable.io/blog/ebpf-log-analytics)
- 2023-10-13: [Platform Engineering Trends in Cloud Native: Q&A With Thomas Graf](https://dzone.com/articles/platform-engineering-trends-in-cloud-native-qampa/)
- 2023-10-11: [All Azure Network plugins lead to Cilium](https://isovalent.com/blog/post/upgrade-cilium-in-azure/)
- 2023-10-11: [Cloud Native Computing Foundation Announces Cilium Graduation](https://www.cncf.io/announcements/2023/10/11/cloud-native-computing-foundation-announces-cilium-graduation/)
- 2023-10-11: [CNCF Graduates Cilium Networking Software Project](https://cloudnativenow.com/features/cncf-graduates-cilium-networking-software-project/)
- 2023-10-11: [Cilium Graduates at the CNCF](hhttps://isovalent.com/blog/post/cilium-graduates-cncf/)
- 2023-10-11: [Cloud Native Computing Foundation Announces Cilium Graduation](https://www.itopstimes.com/itops/cncf-announces-graduation-of-cilium-project/)
- 2023-10-11: [Container networking engine Cilium graduates from CNCF incubation](https://siliconangle.com/2023/10/11/container-networking-engine-cilium-graduates-cncf-incubation/)
- 2023-10-11: [Cilium Graduates From CNCF Incubator](https://tfir.io/cilium-graduates-from-cncf-incubator)
- 2023-10-11: [Cilium leaves incubator, gets the nod from Cloud Native Computing Foundation](https://www.theregister.com/2023/10/12/cilium_graduated/)
- 2023-10-04: [Cilium vs Calico](https://www.linkedin.com/pulse/cilium-vs-calico-ghassan-malke-/)
- 2023-10-02: [Cilium: Installing Cilium on AKS- Mariner in BYOCNI mode](https://medium.com/@amitmavgupta/cilium-installing-cilium-on-aks-mariner-in-byocni-mode-f94cc1dcd94f)
- 2023-10-02: [Kube-Proxy and CNI: The Hidden Components of Kubernetes Networking](https://medium.com/@seifeddinerajhi/kube-proxy-and-cni-the-hidden-components-of-kubernetes-networking-eb30000bf87a)
- 2023-10-02: [Tutorial: Setting Up a Cybersecurity Honeypot with Tetragon to Trigger Canary Tokens](https://isovalent.com/blog/post/tetragon-canary-tokens-tutorial/)
- 2023-10-02: [Cilium CNCF Graduation Could Mean Better Observability, Security with eBPF](https://thenewstack.io/cilium-cncf-graduation-could-mean-better-observability-security-with-ebpf/)
- 2023-09-29: [Cilium – A Fascinating Comprehensive Guide](https://dotcommagazine.com/2023/09/cilium-a-fascinating-comprehensive-guide/)
- 2023-09-26: [Architecting for Resilience: Crafting Opinionated EKS Clusters with Karpenter & Cilium Cluster Mesh — Part 1](https://aws.plainenglish.io/architecting-for-resilience-crafting-opinionated-eks-clusters-with-karpenter-cilium-cluster-mesh-c87cee1df934)
- 2023-09-25: [Learn Cilium the Easy Way with the Cilium Learning Paths](https://isovalent.com/blog/post/cilium-learning-path/)
- 2023-09-25: [DataDog Third Party Projects: Cilium](https://opensource.datadoghq.com/projects/cilium/)
- 2023-09-18: [Deep Dive — Inspect Deployment Network Traffic in Kubernetes](https://jysk.tech/inspect-deployment-network-traffic-in-kubernetes-8a348110df4d)
- 2023-09-17: [Gateway API: Can I replace my Ingress Controller with Cilium?](https://blog.ogenki.io/post/cilium-gateway-api/)
- 2023-09-14: [eBPF Summit 2023 CTF Writeup](https://hemslo.io/ebpf-summit-2023-ctf-writeup/)
- 2023-09-14: [Unlock Cilium in EKS-Anywhere](https://isovalent.com/blog/post/unlock-cilium-in-eks-anywhere/)
- 2023-09-13: [Unleashing the Power of k3s for Edge Computing: Deploying 3000+ in-store Kubernetes Clusters — Part 1](https://jysk.tech/unleashing-the-power-of-k3s-for-edge-computing-deploying-3000-in-store-kubernetes-clusters-part-77ecc5378d31)
- 2023-09-12: [Tutorial: How to Use Cilium Hubble for Observability in CNI Chaining Mode (Part 1)](https://isovalent.com/blog/post/cilium-hubble-observability-cni-chaining-mode/)
- 2023-09-12: [What is next for observability?](https://www.infoworld.com/article/3706048/what-s-next-for-observability.html)
- 2023-09-05: [Mastering Kubernetes with Cilium: Empowering L7 Traffic Control](https://nws.netways.de/tutorials/author/jlamp/)
- 2023-09-05: [Can I Use Tetragon without Cilium? Yes!](https://isovalent.com/blog/post/can-i-use-tetragon-without-cilium-yes/)
- 2023-09-05: [Переход с Calico на Cilium. Причины и итоги](https://habr.com/ru/companies/dbraincloud/articles/759244/)
- 2023-09-02: [Cilium: Support for ENI Prefix Delegation in an EKS cluster](https://medium.com/@amitmavgupta/cilium-support-for-eni-prefix-delegation-in-an-eks-cluster-feddf894160b)
- 2023-09-01: [174 Cilium by using eBPF with Raymond De Jong from Isovalent (ENG)](https://ittalks.libsyn.com/174-cilium-by-using-ebpf-with-raymond-de-jong-from-isovalent-eng)
- 2023-08-26: [Cilium: Installing Azure CNI powered by Cilium using Terraform (Part 2)](https://medium.com/@amitmavgupta/cilium-installing-azure-cni-powered-by-cilium-using-terraform-part-2-6f20f70fd394)
- 2023-08-22: [Top 20 Cilium Use Cases](https://isovalent.com/blog/post/top-20-cilium-use-cases/)
- 2023-08-19: [Exploring Cilium: Revolutionizing Network Security and Observability](https://medium.com/@gurpreet.singh_89/exploring-cilium-revolutionizing-network-security-and-observability-7c328b9397b1)
- 2023-08-19: [Cilium: Installing Azure CNI powered by Cilium using Terraform (Part 1)](https://medium.com/@amitmavgupta/cilium-installing-azure-cni-powered-by-cilium-using-terraform-part-1-eed719c884a4)
- 2023-08-16: [Auditing of eBPF Programs with Tetragon](https://cloudyuga.guru/hands_on_lab/ebpf-logs-with-tetragon)
- 2023-08-13: [Installer k3s et déployer cilium sous Debian 12](https://j.hommet.net/installer-k3s-et-cilium-debian/)
- 2023-08-04: [Cilium Mutual Auth DIY](https://xxradar.medium.com/cilium-mutual-auth-diy-5d5036a82cf9)
- 2023-08-03: [Kubernetes - Migrating from Flannel and Friends to Cilium](https://www.linkedin.com/pulse/kubernetes-migrating-from-flannel-friends-cilium-greg-abrams/)
- 2023-08-03: [Sortie de Cilium 1.14](https://medium.com/@littel.jo/sortie-de-cilium-1-14-ce22d2bfd8d9)
- 2023-08-03: [A Quick Tour of Cilium 1.14 with Istio](https://www.solo.io/blog/cilium-1-14-istio/)
- 2023-08-02: [Cilium: Decoding the packet path with Security Groups for Pods in EKS](https://medium.com/@amitmavgupta/security-groups-for-pods-in-eks-cilium-and-networking-f809cf72fc31)
- 2023-08-02: [Prevent Overlayfs Privilege Escalation on Ubuntu Kernels with Yaml (bpf)](https://djalal.opendz.org/post/prevent-kernel-overlayfs-ubuntu-cves-with-yaml/)
- 2023-08-02: [Installing Cilium in Azure Kubernetes Service (BYOCNI) with no Kube-Proxy](https://medium.com/@amitmavgupta/installing-cilium-in-azure-kubernetes-service-byocni-with-no-kube-proxy-825b9007b24b)
- 2023-07-28: [VMblog Expert Interview: Isovalent on the Launch of Cilium 1.14](https://vmblog.com/archive/2023/07/28/vmblog-expert-interview-isovalent-on-the-launch-of-cilium-1-14.aspx)
- 2023-07-27: [Kubernetes LoadBalance service using Cilium BGP control plane](https://medium.com/@valentin.hristev/kubernetes-loadbalance-service-using-cilium-bgp-control-plane-8a5ad416546a)
- 2023-07-27: [Cilium 1.14 expands networking beyond Kubernetes, offers higher speeds](https://www.sdxcentral.com/articles/news/cilium-1-14-expands-networking-beyond-kubernetes-offers-higher-speeds/2023/07/)
- 2023-07-26: [Unleashing the Power of Cilium CNI to Propel Trendyol’s Performance Up to 40%](https://medium.com/trendyol-tech/trendyols-cilium-transformation-ff064ff79fce)
- 2023-07-26: [Install Kubernetes 1.27 with Cilium on Ubuntu](https://akyriako.medium.com/install-kubernetes-1-27-with-cilium-on-ubuntu-16193c7c2ac6)
- 2023-07-25: [Cilium 1.14 – Effortless Mutual Authentication, Service Mesh, Networking Beyond Kubernetes, High-Scale Multi-Cluster, and Much More](https://isovalent.com/blog/post/cilium-release-114/)
- 2023-07-24: [Hubble Series (Part 2): Cilium Hubble for the Enterprise](https://isovalent.com/blog/post/cilium-hubble-enterprise-part-2/)
- 2023-07-17: [Comparing Networking Solutions for Kubernetes: Cilium vs. Calico vs. Flannel](https://www.civo.com/blog/calico-vs-flannel-vs-cilium?utm_content=buffer405e0)
- 2023-07-13: [How to monitor Kubernetes network and security events with Hubble and Grafana](https://grafana.com/blog/2023/07/13/how-to-monitor-kubernetes-network-and-security-events-with-hubble-and-grafana/)
- 2023-07-12: [Deploying Isovalent Enterprise in AKS (Part 2): Azure ARM Templates and Azure CLI](https://isovalent.com/blog/post/tutorial-deploying-isovalent-using-azure-specific-resources/)
- 2023-07-11: [Cilium: Empowering Kubernetes Networking and Security](https://medium.com/cloud-native-daily/cilium-empowering-kubernetes-networking-and-security-9d25750e8f44)
- 2023-07-11: [Mon été avec Cilium et EKS (Partie 2)](https://medium.com/@littel.jo/mon-%C3%A9t%C3%A9-avec-cilium-et-eks-partie-2-ea8ba7a9dcae)
- 2023-07-04: [Cilium CNI: A Comprehensive Deep Dive Guide for Networking and Security Enthusiasts!](https://itnext.io/cilium-cni-a-comprehensive-deep-dive-guide-for-networking-and-security-enthusiasts-588afbf72d5c)
- 2023-07-03: [Ask me anything with Bill Mulligan from Isovalent](https://www.youtube.com/watch?v=T8vAyrnbqsQ)
- 2023-07-03: [Cilium: Empowering Kubernetes Networking and Security](https://blog.slycreator.com/cilium-empowering-kubernetes-networking-and-security)
- 2023-07-01: [K8s & Cilium Study Guide](https://ulttransformer.com/posts/cilium/k8s-cilium-study-guide/)
- 2023-06-28: [Cilium for Kubernetes](https://www.solo.io/topics/cilium/cilium-kubernetes/)
- 2023-06-26: [Lost in transit: debugging dropped packets from negative header lengths](https://blog.cloudflare.com/lost-in-transit-debugging-dropped-packets-from-negative-header-lengths/)
- 2023-06-22: [Azure CNI with Cilium: Most scalable and performant container networking in the Cloud](https://azure.microsoft.com/en-us/blog/azure-cni-with-cilium-most-scalable-and-performant-container-networking-in-the-cloud/)
- 2023-06-22: [Tutorial: How to Migrate to Cilium (Part 1)](https://isovalent.com/blog/post/tutorial-migrating-to-cilium-part-1/)
- 2023-06-19: [Cilium Network Policy in AKS](https://blog.cloudtrooper.net/2023/06/16/cilium-network-policy-in-aks/)
- 2023-06-19: [Choosing a CNI for Kubernetes](https://deploy.equinix.com/developers/guides/choosing-a-cni-for-kubernetes)
- 2023-06-14: [Securing Your Kubernetes Cluster: Cilium and Network Policies](https://www.learncloudnative.com/blog/2023-06-14-cilium-network-policy)
- 2023-06-05: [Hubble Series (Part 1): Re-introducing Hubble](https://isovalent.com/blog/post/hubble-series-re-introducing-hubble/)
- 2023-06-04: [Installing Cilium Service Mesh with the Kubernetes Control Plane externally (illumos)](https://medium.com/@norlin.t/installing-cilium-service-mesh-with-external-kubernetes-control-plane-illumos-e5517253e011)
- 2023-06-01: [Saying Goodbye to Ingress: Embracing the Future of Kubernetes Traffic Management with Gateway API and Cilium](https://itnext.io/saying-goodbye-to-ingress-embracing-the-future-of-kubernetes-traffic-management-with-gateway-api-6584b7b8f913)
- 2023-06-01: [Supercharging Kubernetes Fury Distribution`s Networking OSS Module with Cilium](https://blog.sighup.io/supercharging-kubernetes-fury-distributions-networking-oss-module-with-cilium/)
- 2023-05-30: [Mastering Kubernetes Networking with Cilium](https://www.civo.com/blog/cilium-introduction)
- 2023-05-30: [General Availability: Isovalent Cilium Enterprise through Azure Marketplace](https://azure.microsoft.com/en-us/updates/general-availability-isovalent-cilium-enterprise-through-azure-marketplace/)
- 2023-05-30: [Better policies, less tool sprawl: Tietoevry uses Cilium and Hubble](https://isovalent.com/blog/post/tietoevry-with-isovalent-cilium-enterprise/)
- 2023-05-27: [Race condition between Kube-proxy and Cilium](https://blog.zhouhaibing.com/posts/race-condition-between-kube-proxy-and-cilium/ )
- 2023-05-26: [Cilium Mesh: A new way to extend Kubernetes benefits across on-premises and cloud networking](https://www.techrepublic.com/article/cilium-mesh/)
- 2023-05-26: [Setup Kubernetes + eBPF Cilium + Hubble with Performance Tuning](https://luthfi-dev.medium.com/setup-kubernetes-ebpf-cilium-hubble-with-performance-tuning-38a76669c352)
- 2023-05-24: [An inside look at the new Intro to Cilium course with course creator, Jef Spaleta!](https://www.cncf.io/blog/2023/05/24/an-inside-look-at-the-new-intro-to-cilium-course-with-course-creator-jef-spaleta/)
- 2023-05-23: [Cilium Installation Tips](https://mallakimahdi.medium.com/cilium-installation-tips-17a870fdc4f2)
- 2023-05-23: [Cosmonic Builds the Ideal Container Toolset For Wasm at the Edge; Open Sources Netreap](https://cosmonic.com/blog/company/cosmonic-open-sources-netreap)
- 2023-05-23: [Tutorial: Deploying Isovalent Enterprise for Cilium from Azure Marketplace](https://isovalent.com/blog/post/tutorial-deploying-isovalent-enterprise-for-cilium-from-azure-marketplace)
- 2023-05-11: [Zero Trust Security Journey with Cilium and eBPF](https://isovalent.com/blog/post/your-zero-stress-path-to-kubernetes-security-cilium-and-ebpf)
- 2023-05-10: [Isovalent Enterprise for Cilium 1.13: SRv6 L3VPN, Overlapping CIDR Support in Meshed Clusters, FromFQDN Ingress in Network Policy, Hubble Plugin for Grafana and more!](https://isovalent.com/blog/post/isovalent-enterprise-1-13/)
- 2023-05-05: [Tutorial: Cross-Namespace Routing with Cilium Gateway API](https://isovalent.com/blog/post/tutorial-cross-namespace-routing-with-cilium-gateway-api/)
- 2023-05-05: [Isovalent’s Cilium Mesh bridges gap between Kubernetes and legacy workloads](https://siliconangle.com/2023/05/05/isovalents-cilium-mesh-bridges-gap-kubernetes-legacy-workloads-kubecon/)
- 2023-05-04: [Netreap: A Practical Guide to Running Cilium in Nomad](https://cosmonic.com/blog/engineering/netreap-a-practical-guide-to-running-cilium-in-nomad/)
- 2023-05-03: [Cilium CNI - Usages Avancés en Français](https://blog.wescale.fr/cilium-cni-deep-dive-usages-avanc%C3%A9s)
- 2023-04-29: [Getting Started with Cilium eBPF](https://medium.com/@luishrsoares/getting-started-with-cilium-ebpf-778d00c113aa/)
- 2023-04-27: [Cilium NetworkPolicy with AWS Security Group Rules](https://medium.com/codex/cilium-networkpolicy-with-aws-security-group-rules-fc91d25712f4)
- 2023-04-27: [Cilium CNI - Politiques Réseaux](https://blog.wescale.fr/cilium-cni-deep-dive-politiques-r%C3%A9seaux)
- 2023-04-27: [Cilium and Hubble for Kubernetes](https://gmontalvoy.com/2023/04/27/cilium-and-hubble-for-kubernetes/)
- 2023-04-25: [Cilium CNI - Introduction à Cilium](https://blog.wescale.fr/introduction_a_cilium)
- 2023-04-20: [How Ikea, The New York Times and Bloomberg use open source for cloud native networking and security](https://www.sdxcentral.com/articles/analysis/how-ikea-the-new-york-times-and-bloomberg-use-open-source-for-cloud-native-networking-and-security/2023/04/)
- 2023-04-19: [Using Cilium for Kubernetes Networking and Observability](https://www.blueshoe.io/blog/cilium-kubernetes-networking-observability/)
- 2023-04-18: [Cilium Mesh – One Mesh to Connect Them All](https://isovalent.com/blog/post/introducing-cilium-mesh/)
- 2023-04-12: [Isovalent Cilium Enterprise now Available on Microsoft Azure Marketplace](https://isovalent.com/blog/post/isovalent-cilium-enterprise-microsoft-azure-marketplace/)
- 2023-04-12: [Netreap: Bringing Cilium to the World Outside Kubernetes](https://cosmonic.com/blog/engineering/netreap-bringing-cilium-to-the-world-outside-kubernetes)
- 2023-04-11: [Container Connectivity, and Networking in the Cloud-Native Era with Isovalent Cilium](https://gestaltit.com/exclusive/sulagna/container-connectivity-and-networking-in-the-cloud-native-era-with-isovalent-cilium/)
- 2023-04-07: [The power of eBPF Security, Tetragon #Part2](https://luthfi.dev/posts/the-power-of-ebpf-security-tetragon-part-2/)
- 2023-03-20: [CiliumCon Europe 2023 Schedule](https://isovalent.com/blog/post/ciliumcon-eu-2023-schedule/)
- 2023-03-15: [Tutorial: Getting Started with the Cilium Gateway API](https://isovalent.com/blog/post/tutorial-getting-started-with-the-cilium-gateway-api/)
- 2023-03-15: [A Deep Dive into Cilium Gateway API: The Future of Ingress Traffic Routing](https://isovalent.com/blog/post/cilium-gateway-api/)
- 2023-03-07: [Cilium for Kubernetes networking: Why we use it and why we love it](https://blog.palark.com/why-cilium-for-kubernetes-networking/)
- 2023-03-02: [How Ascend leverages Cilium as a networking layer](https://www.cncf.io/case-studies/ascend/)
- 2023-02-27: [Kubernetes Network Policy Debugger​ eBPF / Kernel Space](https://www.linkedin.com/pulse/kubernetes-network-policy-debugger-ebpf-kernel-space-zak-hassan/)
- 2023-02-23: [Isovalent Update Advances Cilium Networking](https://containerjournal.com/features/isovalent-update-advances-cilium-networking/)
- 2023-02-23: [K8S/Cilium eBPF 기반 서비스 메쉬 분석 (Per-Node Proxy)](https://medium.com/dataplatform-lab/k8s-cilium-ebpf-%EA%B8%B0%EB%B0%98-%EC%84%9C%EB%B9%84%EC%8A%A4-%EB%A9%94%EC%89%AC-%EB%B6%84%EC%84%9D-per-node-proxy-3a810761f649)
- 2023-02-23: [MTU in Cilium ClusterMesh with EKS](https://rocky-chen.medium.com/mtu-in-cilium-clustermesh-with-eks-c05b2aca2f4f)
- 2023-02-17: [Cilium and Its Features (open source)](https://medium.com/@shahneel2409/cilium-and-its-features-open-source-852c70914802)
- 2023-02-16: [Netzwerkverkehr: Cilium 1.13 leitet Übergang von der Ingress zur Gateway API ein](https://www.heise.de/news/Netzwerkverkehr-Cilium-1-13-leitet-Uebergang-von-der-Ingress-zur-Gateway-API-ein-7518077.html)
- 2023-02-15: [BIG Performances with BIG TCP on Cilium](https://isovalent.com/blog/post/big-tcp-on-cilium/)
- 2023-02-15: [Cilium Wars: Scaling Up Your Network with Jedi-like Cilium Handling Techniques](https://itsyash.hashnode.dev/cilium-wars-scaling-up-your-network-with-jedi-like-cilium-handling-techniques)
- 2023-02-15: [Cilium 1.13 – Gateway API, mTLS datapath, Service Mesh, BIG TCP, SBOM, SNI NetworkPolicy](https://isovalent.com/blog/post/cilium-release-113/)
- 2023-02-14: [Packet, packet, who’s got the packet?](https://blog.oddbit.com/post/2023-02-14-whos-got-the-packet/)
- 2023-02-13: [A well-secured project: Cilium Security Audits 2022](https://www.cncf.io/blog/2023/02/13/a-well-secured-project-cilium-security-audits-2022-published/)
- 2023-02-12: [Cloud Native Networking Using eBPF](https://dev.to/deepeshaburse/cloud-native-networking-using-ebpf-4p9h)
- 2023-02-10: [Azure CNI Powered by Cilium](https://rodrigtech.com/azure-cni-powered-by-cilium/)
- 2023-02-08: [Eine Einführung in API-Gateways und die Cilium Implementation der Kubernetes Gateway-API](https://b-nova.com/home/content/an-introduction-to-api-gateways-and-ciliums-implementation-of-kubernetes-gateway-api)
- 2023-02-06: [Cilium Registers Massive Growth In Contribution And Adoption In 2022](https://www.tfir.io/cilium-registers-massive-growth-in-contribution-and-adoption-in-2022/)
- 2023-02-06: [WTF is Cilium?](https://blog.container-solutions.com/wtf-is-cilium)
- 2023-02-05: [Establish Cilium ClusterMesh with Helm Chart](https://medium.com/codex/establish-cilium-clustermesh-whelm-chart-11b08b0c995c)
- 2023-02-02: [Four Cilium trends to watch in 2023](https://www.solo.io/blog/cilium-trends-2023/)
- 2023-02-01: [Debugging Cilium Envoy Upstream Connection Failures](https://superorbital.io/journal/debugging-cilium-envoy-connection-failures/)
- 2023-01-27: [How I moved from MetalLB to Cilium](https://dariomader.io/post/how_i_moved_from_metallb_to_cilium/)
- 2023-01-26: [Announcing the Cilium annual report](https://www.cncf.io/blog/2023/01/26/announcing-the-cilium-annual-report/)
- 2023-01-16: [Tutorial: Tips and Tricks to install Cilium](https://isovalent.com/blog/post/tutorial-tips-and-tricks-to-install-cilium/)
- 2023-01-12: [Multi-cluster Networking with Cilium and Istio](https://blog.thingsgreener.com/multi-cluster-networking-with-cilium-and-istio/)
- 2023-01-10: [EKS Series Part 3. All That eBPF: Exploring Cilium in the Real World](https://svodwood.github.io/devops-pastebin/eks-part-three-cilium-webstore/)
- 2023-01-09: [Simplifying kubernetes network policies with cilium](https://punksecurity.co.uk/blog/kubernetes_network_policies/)
- 2023-01-09: [Configuração de rede Cilium para Kubernetes](https://andrebassi.com.br/configuracao-de-rede-cilium-para-kubernetes)
- 2023-01-06: [C2C 2Chat: Switzerland and Austria - Introduction to Cilium - Key Takeaways](https://community.c2cglobal.com/switzerland-and-austria-132/c2c-2chat-switzerland-and-austria-introduction-to-cilium-key-takeaways-4223)
- 2022-12-28: [WSL2+Cilium: The rise of eBPF](https://wsl.dev/wslcilium/)
- 2022-12-22: [EKS Series Part 2. All That eBPF: Operationalizing EKS with Cilium, Karpenter and Flux](https://svodwood.github.io/devops-pastebin/eks-part-two-cilium-karpenter-flux/)
- 2022-12-20: [Badges for Cilium Labs: Catch ‘em over the holidays!](https://isovalent.com/blog/post/badges-for-cilium-labs-catch-em-over-the-holidays/)
- 2022-12-19: [Pod Traffic Rate Limiting with Cilium Bandwidth Manager](https://isovalent.com/resources/video-pod-traffic-rate-limiting-with-cilium-bandwidth-manager/)
- 2022-12-16: [eBPF and its capabilities](https://medium.com/exness-blog/ebpf-and-its-capabilities-9a3a1dce3802)
- 2022-12-14: [KubeLift getting started (Part 3) - Observing ingress and pod-to-pod traffic with Cilium Hubble](https://polverio.com/blog/kubelift-getting-started-part-3-observing-ingress-and-pod-to-pod-traffic-with-cilium-hubble/)
- 2022-12-13: [How the Hive Came To Bee – a story of eBPF and Cilium so far](https://isovalent.com/blog/post/how-the-hive-came-to-bee-a-story-of-ebpf-and-cilium-so-far/)
- 2022-12-05: [Microsoft and Isovalent partner to bring next generation eBPF dataplane for cloud-native applications in Azure](https://azure.microsoft.com/en-us/blog/microsoft-and-isovalent-partner-to-bring-next-generation-ebpf-dataplane-for-cloudnative-applications-in-azure/)
- 2022-12-05: [Microsoft and Isovalent bring eBPF-based Networking and Security to Azure](https://isovalent.com/blog/post/microsoft-and-isovalent-bring-ebpf-based-networking-to-azure/)
- 2022-12-04: [Kubernetes on Illumos — exposing ClusterIP](https://medium.com/@norlin.t/kubernetes-on-illumos-exposing-clusterip-4da89a3abd8d)
- 2022-12-01: [Cilium - service mesh without sidecar explained!](https://medium.com/@ghasmir/cilium-service-mesh-without-side-car-explained-b572533f39de)
- 2022-11-27: [Migrate to Cilium from Amazon VPC CNI with Zero Downtime](https://medium.com/codex/migrate-to-cilium-from-amazon-vpc-cni-with-zero-downtime-493827c6b45e)
- 2022-11-25: [Transparent encryption of node to node traffic on Amazon EKS using WireGuard and Cilium](https://aws.amazon.com/de/blogs/containers/transparent-encryption-of-node-to-node-traffic-on-amazon-eks-using-wireguard-and-cilium/)
- 2022-11-22: [Tutorial: Transparent Encryption with IPsec and WireGuard](https://isovalent.com/blog/post/tutorial-transparent-encryption-with-ipsec-and-wireguard/)
- 2022-11-17: [Introducing the Cilium Enterprise integration in Grafana Cloud for Kubernetes network monitoring](https://grafana.com/blog/2022/11/17/introducing-the-cilium-enterprise-integration-in-grafana-cloud-for-kubernetes-network-monitoring/)
- 2022-11-15: [Tutorial: Azure CNI Powered by Cilium](https://isovalent.com/blog/post/tutorial-azure-cni-powered-by-cilium/)
- 2022-11-11: [Kubernetes: Cloudbees troubleshooting with Cilium](https://www.dbi-services.com/blog/kubernetes-cloudbees-troubleshooting-with-cilium/)
- 2022-11-10: [What are Cilium & Hubble - With Thomas Graf](https://www.youtube.com/watch?v=H5RqSAX-eo4)
- 2022-11-03: [Cilium on AKS Too?!? And the MVP CNI of the Year is…](https://www.solo.io/blog/cilium-azure-kubernetes-service/)
- 2022-11-02: [Cilium Network Configuration For Kubernetes (enter eBPF)](https://dev.to/thenjdevopsguy/cilium-network-configuration-for-kubernetes-enter-ebpf-4363)
- 2022-10-30: [Initial impressions of the Cilium Grafana Observability Demo](https://medium.com/@norlin.t/initial-impressions-of-the-cilium-grafana-observability-demo-dd81d73df96e)
- 2022-10-26: [Announcing Azure CNI Powered by Cilium](https://isovalent.com/blog/post/azure-cni-cilium/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2022-10-26: [Azure CNI Powered by Cilium for Azure Kubernetes Service (AKS)](https://techcommunity.microsoft.com/t5/azure-networking-blog/azure-cni-powered-by-cilium-for-azure-kubernetes-service-aks/ba-p/3662341)
- 2022-10-26: [Strengthen your system with Tetragon's eBPF-based Security Observability and Runtime Enforcement capabilities](https://b-nova.com/en/home/content/strengthen-your-system-with-tetragons-ebpf-based-security-observability-and-runtime-enforcement-capabilities)
- 2022-10-24: [Grafana and Cilium: Deep eBPF-powered observability for Kubernetes and cloud native infrastructure](https://grafana.com/blog/2022/10/24/grafana-and-cilium-deep-ebpf-powered-observability-for-kubernetes-and-cloud-native-infrastructure/)
- 2022-10-22: [Topology Aware Routing and Service Mesh across Clusters with Cluster Mesh](https://isovalent.com/blog/post/topology-aware-routing-and-service-mesh-across-clusters-with-cluster-mesh/)
- 2022-10-19: [Cilium with Ingress, OpenTelemetry and L7 policies](https://medium.com/@norlin.t/cilium-with-ingress-opentelemetry-and-l7-policies-44522e349abe)
- 2022-10-13: [Accelerate network performance with Cilium BBR](https://isovalent.com/blog/post/accelerate-network-performance-with-cilium-bbr/)
- 2022-10-06: [Tutorial: Run and Observe IPv6 on Kubernetes with Cilium and Hubble](https://isovalent.com/blog/post/tutorial-run-and-observe-ipv6-on-kubernetes-with-cilium/)
- 2022-10-05: [Kubernetes Network Policy — Cilium](https://jansutris10.medium.com/kubernetes-network-policy-cilium-bfab4c1b893e)
- 2022-10-04: [Cilium CNI on EKS using secondary CIDR and prefix delegation](https://medium.com/@benoit.mouquet/cilium-cni-on-eks-using-secondary-cidr-and-prefix-delegation-55e57ffd2537)
- 2022-09-23: [Cilium installation in Minikube using a multi-cluster environment](https://www.dbi-services.com/blog/cilium-installation-in-minikube-using-a-multi-cluster-environment/)
- 2022-09-14: [Raspberry Pi Kubernetes Cluster with Cilium CNI](https://charmingwebdesign.com/raspberry-pi-kubernetes-cluster-with-cilium-cni/)
- 2022-09-09: [Community Roundup: Kubernetes, Operators, Cilium](https://austince.github.io/blog/community-roundup/)
- 2022-09-09: [An exciting journey of adding Cilium support to Deckhouse](https://blog.deckhouse.io/cilium-support-in-deckhouse-kubernetes-342fcf3da56)
- 2022-09-07: [The Rise of Kubernetes and the End of Networking & Security as You Know It. What is Next?](https://isovalent.com/blog/post/isovalent-series-b/)
- 2022-09-07: [This new connectivity layer will define the next decade of cloud infrastructure](https://www.techrepublic.com/article/new-layer-cloud-infrastructure/)
- 2022-09-07: [Kubernetes Day 2 challenges — Isovalent brings secure connectivity, nabs funding](https://medium.com/@charled.breteche/kubernetes-security-control-pod-to-pod-communications-with-cilium-network-policies-d7275b2ed378)
- 2022-09-05: [Ciliumを試す -サービスメッシュにサイドカーが必須だと思っていたがそんなことはなかったぜ-](https://caddi.tech/archives/3864)
- 2022-09-05: [How to Deploy a Tanzu Kubernetes Grid cluster using the Cilium CNI](https://veducate.co.uk/tanzu-kubernetes-grid-cilium/)
- 2022-08-31: [Cilium Projectから公開！ eBPFを用いてセキュリティの可観測性をもたらすTetragon](https://gihyo.jp/article/2022/08/kubernetes-cloudnative-topics-01)
- 2022-08-30: [Addressing Bandwidth Exhaustion with Cilium Bandwidth Manager](https://isovalent.com/blog/post/addressing-bandwidth-exhaustion-with-cilium-bandwidth-manager/)
- 2022-08-24: [Insights on How eBPF & Cilium Changes How We Leverage the Linux Kernel](https://open.spotify.com/episode/6sr7Ic8cavV8XBoTTfp9gX?si=LKG_I3LKTY2mSHkCDpi22g&nd=1)
- 2022-08-24: [eBPF, Cilium and #GKE Dataplane V2 | Google Developers North America](https://www.youtube.com/watch?v=W2NF8T_lVSQ)
- 2022-08-24: [Day Two Cloud 160: Going Deep Into Cilium Service Mesh With eBPF](https://daytwocloud.io/podcast/day-two-cloud-160-going-deep-into-cilium-service-mesh-with-ebpf/)
- 2022-08-21: [Learning eBPF exploitation](https://stdnoerr.github.io/writeup/2022/08/21/eBPF-exploitation-(ft.-D-3CTF-d3bpf).html)
- 2022-08-18: [KubeCon EU開幕 前日に行われたプレカンファレンスからeBPFとTetragonを紹介](https://thinkit.co.jp/article/19777)
- 2022-08-13: [How to monitor Cilium on your Civo k3s cluster with Prometheus](https://blog.ediri.io/how-to-monitor-cilium-on-your-civo-k3s-cluster-with-prometheus)
- 2022-08-11: [Build a managed Kubernetes cluster from scratch — part 4](https://medium.com/@norlin.t/build-a-managed-kubernetes-cluster-from-scratch-part-4-3856f0756a03)
- 2022-08-10: [Ep. #30, Cilium and eBPF with Thomas Graf of Isovalent](https://www.heavybit.com/library/podcasts/the-kubelist-podcast/ep-30-cilium-and-ebpf-with-thomas-graf-of-isovalent?)
- 2022-08-04: [Why sidecar-less Cilium Service Mesh is a game-changer](https://www.eficode.com/devops-podcast/sidecar-less-cilium-mesh)
- 2022-08-04: [Commandline Cloaking 2 - Tetragon and Nim](https://blog.tofile.dev/2022/08/04/tetragon.html)
- 2022-08-04: [How to enable the Cilium Hubble UI in a Civo k3s cluster](https://blog.ediri.io/how-to-enable-the-cilium-hubble-ui-in-a-civo-k3s-cluster)
- 2022-07-31: [Redefining service mesh with Cilium](https://medium.com/nerd-for-tech/redefining-service-mesh-with-cilium-4738cdfb9899)
- 2022-07-28: [Cilium lance un service mesh Kubernetes alimenté par eBPF](https://www.lemondeinformatique.fr/actualites/lire-cilium-lance-un-service-mesh-kubernetes-alimente-par-ebpf-87457.html)
- 2022-07-27: [Cilium: eBPF powered CNI, a NOS Solution for Modern Clouds](https://dev.to/otomato_io/cilium-ebpf-powered-cni-a-nos-solution-for-modern-clouds-1hl1)
- 2022-07-26: [Kubernetes, Networking, and Finding the VMware of Cloud Native](https://future.com/kubernetes-networking-and-finding-the-vmware-of-cloud-native/)
- 2022-07-25: [Monitor Cilium-managed infrastructure with Datadog](https://www.datadoghq.com/blog/monitor-cilium-cni-with-datadog/)
- 2022-07-25: [Key metrics for monitoring Cilium](https://www.datadoghq.com/blog/cilium-metrics-and-architecture/)
- 2022-07-25: [Monitor Cilium and Kubernetes performance with Hubble](https://www.datadoghq.com/blog/monitor-cilium-and-kubernetes-performance-with-hubble/)
- 2022-07-25: [Nexxiot User Story](https://www.cncf.io/case-studies/nexxiot/)
- 2022-07-22: [My first impressions of Cilium](https://medium.com/codefactory/my-first-impressions-of-cilium-b205a31efd61)
- 2022-07-22: [Containerisierung: Cilium 1.12 bietet Service-Mesh-Alternative zu Istio](https://www.heise.de/news/Containerisierung-Cilium-1-12-bietet-Service-Mesh-Alternative-zu-Istio-7186886.html)
- 2022-07-22: [Cilium Service Mesh Extends eBPF for Cloud Deployments](https://www.itprotoday.com/cloud-computing-and-edge-computing/cilium-service-mesh-extends-ebpf-cloud-deployments)
- 2022-07-21: [오픈소스 네트워크 소프트웨어 실리움, 서비스 메시 기능 추가](https://www.itworld.co.kr/tags/178360/246130/246130)
- 2022-07-21: [Kubernetes Networking with Cilium CNI and OKE on Oracle Cloud](https://lmukadam.medium.com/cni-adventures-with-kubernetes-on-oracle-cloud-cilium-5c6f011746d5)
- 2022-07-21: [Cilium speeds sidecar-optioned cloud-native networking](https://www.computerweekly.com/blog/Open-Source-Insider/Cilium-speeds-sidecar-optioned-cloud-native-networking)
- 2022-07-21: [Cilium 1.12 Adds Cilium Service Mesh And Other New Features For Enterprise Kubernetes](https://www.tfir.io/cilium-1-12-adds-cilium-service-mesh-and-other-new-features-for-enterprise-kubernetes/)
- 2022-07-20: [Cilium 1.12 – Ingress, Multi-Cluster, Service Mesh, External Workloads, and much more](https://isovalent.com/blog/post/cilium-release-112/)
- 2022-07-20: [Cilium Service Mesh – Everything You Need to Know](https://isovalent.com/blog/post/cilium-service-mesh/)
- 2022-07-20: [Cilium 1.12 GA: Cilium Service Mesh and other major new features for enterprise Kubernetes](https://www.cncf.io/blog/2022/07/20/cilium-1-12-ga-cilium-service-mesh-and-other-major-new-features-for-enterprise-kubernetes/)
- 2022-07-20: [Cilium launches eBPF-powered Kubernetes service mesh](https://www.infoworld.com/article/3667493/cilium-launches-ebpf-powered-kubernetes-service-mesh.html)
- 2022-07-20: [Cilium Service Mesh: A new bridge back to the kernel for cloud-native infrastructure](https://www.techrepublic.com/article/cilium-service-mesh/)
- 2022-07-20: [VMblog Expert Interview: Isovalent on the Launch of Cilium Service Mesh](https://vmblog.com/archive/2022/07/20/vmblog-expert-interview-isovalent-on-the-launch-of-cilium-service-mesh.aspx#.YtgbN-xByrN)
- 2022-06-16: [Isovalent Open Sources Tetragon, eBPF-Based Observability Platform](https://thenewstack.io/isovalent-open-sources-tetragon-ebpf-based-observability-platform/)
- 2022-06-16: [Cilium CNI Integration in Kubermatic Kubernetes Platform](https://www.kubermatic.com/blog/cilium-cni-integration-in-kubermatic-kubernetes-platform/)
- 2022-06-10: [How to migrate from the Amazon VPC CNI to Cilium in K8s](https://www.youtube.com/watch?v=6Sks_Th99t0)
- 2022-06-01: [Dual-stack home K8s cluster with Cilium](https://yolops.net/k8s-dualstack-cilium.html)
- 2022-05-21: [The innovation around eBPF and Cilium with Dan Wendlandt](https://soundcloud.com/cloudunfiltered/ep135-the-innovation-around-ebpf-and-cilium-with-dan-wendlandt?)
- 2022-05-18: [Build a managed Kubernetes cluster from scratch — part 3](https://medium.com/@norlin.t/build-a-managed-kubernetes-cluster-from-scratch-part-3-10dec988757)
- 2022-05-16: [Tetragon - eBPF-based Security Observability & Runtime Enforcement](https://isovalent.com/blog/post/2022-05-16-tetragon)
- 2022-05-15: [CNCF Paris - 5 May - Introduction to Cilium and first feedback on Crossplane](https://www.youtube.com/watch?v=TsmlSMsLXAA)
- 2022-05-11: [How Meltwater migrated from the Amazon VPC CNI plugin to Cilium in K8s](https://www.youtube.com/watch?v=w6S6baRHHu8)
- 2022-04-28: [Hunting Down an Intermittent Failure in Cilium](https://superorbital.io/journal/hunting-intermittent-cilium-error/)
- 2022-04-23: [Cilium Kubernetes CNI Provider: Part 4, IP Routing Modes (Direct and Encapsulated)](https://www.youtube.com/watch?v=j2aox7K-7wU)
- 2022-04-13: [It is always DNS except when it is not](https://www.datadoghq.com/blog/engineering/grpc-dns-and-load-balancing-incident/)
- 2022-04-05: [Bring your own Container Network Interface (CNI) plugin with Azure Kubernetes Service (AKS)](https://pixelrobots.co.uk/2022/04/bring-your-own-container-network-interface-cni-plugin-with-azure-kubernetes-service-aks-preview/)
- 2022-04-02: [Cilium Kubernetes CNI Provider Deep Dive: Part 3, Cluster Mesh](https://www.youtube.com/watch?v=gkrPt0ZcCfo)
- 2022-03-23: [Cilium now integrates with Mirantis Kubernetes Engine](https://www.mirantis.com/blog/cilium-now-integrates-with-mirantis-kubernetes-engine/)
- 2022-03-15: [Kubernetes Security — Explore Cilium host firewall and host policies](https://medium.com/@charled.breteche/kubernetes-security-explore-cilium-host-firewall-and-host-policies-de93ea9da38c)
- 2022-03-14: [Kubernetes Network Policies with Cilium](https://medium.com/rahasak/kubernetes-network-policies-with-cilium-17be223fe67b)
- 2022-03-06: [Cilium Kubernetes CNI Provider Deep Dive: Part 2](https://www.youtube.com/watch?v=5EcVrm01rAU)
- 2022-03-05: [Kubernetes Security — Control pod to pod communications with Cilium network policies](https://medium.com/@charled.breteche/kubernetes-security-control-pod-to-pod-communications-with-cilium-network-policies-d7275b2ed378)
- 2022-02-23: [Cilium Kubernetes CNI Provider Deep Dive: Part 1](https://www.youtube.com/watch?v=aLq3O3l2LF4&t=0s)
- 2022-02-22: [Getting Linux based eBPF programs to run with eBPF for Windows](https://cloudblogs.microsoft.com/opensource/2022/02/22/getting-linux-based-ebpf-programs-to-run-with-ebpf-for-windows/)
- 2022-02-21: [Exploring eBPF – Part 3: Getting Started with Hubble](https://nicovibert.com/2022/02/21/exploring-ebpf-part-3-hubble/)
- 2022-02-16: [How to use Cluster Mesh for Multi-Region Kubernetes Pod Communication](https://www.cockroachlabs.com/blog/cockroachdb-kubernetes-cilium/)
- 2022-02-07: [Increase Kubernetes Network Security with Cilium](https://itnext.io/increase-kubernetes-network-security-with-cilium-ba6af15c8f5f)
- 2022-02-02: [Exploring eBPF – Part 2: Getting Started with Cilium](https://nicovibert.com/2022/02/02/exploring-ebpf-part-2-cilium/)
- 2022-01-31: [Liz Rice on Programming the Linux Kernel with eBPF, Cilium and Service Meshes](https://www.infoq.com/podcasts/liz-rice-ebpf/)
- 2022-01-28: [Kind, Cilium, MetalLB, and still no kube-proxy](https://medium.com/@charled.breteche/kind-cilium-metallb-and-no-kube-proxy-a9fe66ddfad6)
- 2022-01-24: [Kind cluster with Cilium and no kube-proxy](https://medium.com/@charled.breteche/kind-cluster-with-cilium-and-no-kube-proxy-c6f4d84b5a9d)
- 2021-12-19: [Trip.com: First Step towards Cloud Native Security](https://arthurchiao.art/blog/trip-first-step-towards-cloud-native-security/)
- 2021-12-16: [Cilium Host Policies](https://www.puzzle.ch/de/blog/articles/2021/12/16/cilium-host-policies)
- 2021-12-14: [Full Stack Journey 061: Linux Networking And Observability With eBPF And Cilium](https://packetpushers.net/podcast/full-stack-journey-061-linux-networking-and-observability-with-ebpf-and-cilium/)
- 2021-12-09: [What’s new in Cilium 1.11? Service Mesh Beta, Topology Aware Routing, OpenTelemetry](https://isovalent.com/blog/post/2021-12-release-111)
- 2021-10-25: [How eBPF Streamlines the Service Mesh](https://thenewstack.io/how-ebpf-streamlines-the-service-mesh/)
- 2021-09-09: [AWS picks Cilium for Networking & Security on EKS Anywhere](https://isovalent.com/blog/post/2021-09-aws-eks-anywhere-chooses-cilium/?utm_source=website-cilium&utm_medium=referral&utm_campaign=cilium-blog)
- 2021-07-26: [Ep 16.tech - Supercharge your Kubernetes clusters with Cilium](https://techpodcast.form3.tech/episodes/ep-16-tech-supercharge-your-kubernetes-clusters-with-cilium)
- 2021-07-19: [Visualize Network Traffic: A Simple Way to Enable Cilium on Kubernetes](https://kubesphere.io/blogs/cilium-as-cni/)
- 2021-02-08: [Episode 153 Kubernetes Networking and Security, and Building Business on Open Source with Isovalent Founder, Thomas Graf](https://discopossepodcast.com/ep-153-kubernetes-networking-and-security-and-building-business-on-open-source-with-isovalent-founder-thomas-graf/)
- 2021-02-02: [Cilium: Programmable Linux Networking with Dan Wendlant and Thomas Graf](https://softwareengineeringdaily.com/2021/02/02/cilium-programmable-linux-networking-with-dan-wendlant-and-thomas-graf/)
- 2021-01-12: [Cilium with Thomas Graf on Google Kubernetes Podcast](https://kubernetespodcast.com/episode/133-cilium/)
- 2020-11-18: [eBPF & Cilium Cloud-native Networking](https://www.thecloudcast.net/2020/11/ebpf-cloud-native-networking.html)
- 2020-11-04: [Contributor Podcast - Cilium with Thomas Graf](https://www.contributor.fyi/cilium)
- 2020-10-21: [Episode 40 CNI and EBPF in Kubernetes with Thomas Graf from Cilium](https://www.youtube.com/watch?v=XezvY056PJQ)
- 2020-08-19: [New GKE Dataplane V2 increases security and visibility for containers](https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine)
- 2019-09-02: [Thomas Graf on Cilium, the 1.6 Release, eBPF Security, & the Road ahead](https://www.infoq.com/podcasts/open-source-cilium-security/)
- 2019-01-22: [Why MobiLabs switched to Cilium](https://mobilabsolutions.com/2019/01/why-we-switched-to-cilium/)

### Get help

URL: https://cilium.io/get-help

Where to get help.

### Get involved

URL: https://cilium.io/get-involved

How to get involved.

## Optional

### Artificial Intelligence

URL: https://cilium.io/industries/ai

The AI revolution is here, and with it comes unprecedented demands on network infrastructure. From training massive language models to serving real-time inference, AI workloads require high-performance networking, robust security, and deep observability. Kubernetes has evolved from just a platform for running workloads like web services and microservices to the ideal platform for supporting the end-to-end lifecycle of large artificial intelligence (AI) and machine learning (ML) workloads. Cilium is the cloud native solution for providing, securing, and observing network connectivity between workloads, empowering AI workloads to thrive in cloud native environments. Organizations ranging from artificial intelligence research institutions that build sophisticated training models to financial institutions and startups rely on Cilium to support their distributed AI/ML workloads on Kubernetes.

#### Fulfilling the Networking Demands of AI/ML Workloads

AI workloads typically require massive data transfers, ultra-low latency, high throughput, and high bandwidth networking. Traditional networking solutions struggle to keep up, leading to bottlenecks and inefficiencies. Cilium leverages eBPF to deliver kernel-level networking performance, eliminating the overhead of traditional Linux networking. With features like eXpress Data Path (XDP) and BIG TCP, Cilium ensures high-throughput, low-latency networking ideal for needs of AI/ML workloads. Cilium provides a comprehensive networking toolset for deploying AI/ML models with load balancers, ingress controllers, network policies, egress gateway, service mesh, and more. These features facilitate the seamless deployment of AI/ML workloads and their integration into services and applications.

#### Robust Security for AI/ML Models and Data

AI models are the result of significant investment in research and infrastructure. Protecting these models and the sensitive data they process is non-negotiable for enterprises. Traditional security solutions often lack the granularity and scalability for dynamic, cloud native environments. Cilium provides robust security features that enhance Kubernetes security. These features include zero-trust security with identity-aware security policies, mutual authentication, and advanced network policies. Cilium supports native HTTP and DNS protocol enforcement, ensuring only authorized services can access endpoints. Cilium’s Transparent Encryption (using IPsec or WireGuard) effortlessly encrypts data in transit, safeguarding intellectual property and easing compliance.

#### Deep Observability for AI/ML Infrastructure and Workloads

AI/ML workflows can be incredibly complex, with data flowing across multiple services and clusters. Monitoring performance, debugging issues, or optimizing resource usage without deep observability becomes a hassle. Cilium’s Hubble observability platform provides granular insights into network traffic, API calls, and service dependencies. You can monitor DNS performance, HTTP latency, and error rates with real-time metrics, ensuring AI/ML workloads run smoothly.

#### Scaling AI/ML Infrastructure Efficiently

AI workloads are resource intensive and often experience fluctuating demands. Scaling infrastructure to meet these demands without overspending or inefficiently allocating resources is a significant challenge for operators. The ability to scale up and down depending on the resource demand is one of the most significant advantages Kubernetes brings to AI/ML. Cilium further enhances this advantage, empowering you to scale AI workloads efficiently. Cilium’s advanced load balancing and traffic management capabilities ensure your AI applications can scale dynamically without disruption. By optimizing resource allocation and reducing overhead, Cilium helps you maximize the ROI of your AI/ML investments.

#### Seamless Management of Multi-Cluster and Hybrid Cloud Environments

AI workloads often span multiple clusters, clouds, and on-premises environments. Managing networking, observability, and security across these heterogeneous environments can be an operational nightmare. Cilium integrates seamlessly across environments, providing a unified networking, observability, and security layer. Abstracting away the underlying infrastructure provides a consistent and reliable experience for your entire AI infrastructure. Cilium Cluster Mesh effectively allows multiple clusters to be joined into a large unified network, regardless of the Kubernetes distribution or location where each is running. Cilium host firewall extends Kubernetes declarative, policy-driven security model to the nodes hosting your workloads, delivering seamless, consistent protection across your entire environment.

_Adopters & resources referenced on this page:_ Transforming Networking with Cilium at Ecco; OpenAI Uses Cilium for a Common Networking Foundation for AI Infrastructure; Building the core fabric of accelerated hybrid AI clusters using Cilium; High-Scale Networking for ML Workloads With Cilium; Meltwater's Live Migration to Cilium for Richer Features; High Performance Data Backend for Your AI/ML with MinIO and Cilium Cluster Mesh.

### Cloud Providers

URL: https://cilium.io/industries/cloud-providers

Most cloud providers have their own custom CNIs, resulting in operational complexity for customers operating in multi-cloud environments. For example, the configuration and capabilities of the AWS VPC CNI and OpenShift SDN are significantly different. This fragmentation increases overhead in installing, operating, and troubleshooting network interfaces across cloud environments. Cilium addresses these issues by presenting a unified networking solution tailored for Kubernetes that is compatible across various clouds, providing a consistent workflow that simplifies operations. Using eBPF, Cilium extends beyond Kubernetes basic networking capabilities, offering enhanced observability, security, and reduced latency. Major cloud providers, including Google Cloud, AWS, Azure, and Alibaba, have all standardized on Cilium for Kubernetes networking and security as a testament to its scalability, performance, and rich feature set.

#### Empowering Azure’s AKS Clusters with Cilium

Cilium's open source data plane is natively integrated with Azure CNI in Azure Kubernetes Service (AKS). This integration allows users to tap into Cilium's feature-rich ecosystem, right from the Azure Marketplace.

#### Hetzner Cloud Adopts Cilium for a Cloud Native Ingress Architecture

Hetzner Cloud, a leading German public cloud provider, recently overhauled its ingress architecture to pursue modern and scalable cloud services. Hetzner was scouting for a Kubernetes-native solution with ECMP, firewall, IPv6, and DSR capabilities. Cilium emerged as the ideal choice. Before adopting Cilium, Hetzner ran tests deploying different Cilium configurations in a Kubernetes cluster to benchmark RPS and throughput and then checking the CPU usage as a metric to compare the tests. These tests showed Cilium provided a massive increase in both RPS and throughput while still reducing CPU usage. With Cilium's Kubernetes-native support, scalability, enhanced security, IPv6 compatibility, and efficiency with DSR, Cilium is a compelling choice for cloud providers. The results of Hetzner Cloud's performance tests highlight the value of adopting Cilium to elevate performance, security, and scalability in the cloud infrastructure landscape.

_Adopters & resources referenced on this page:_ Cilium on Azure: Scale and Performance; From Managed Kubernetes to App Platform: Cilium Usage at DigitalOcean; Cilium, eBPF, and EKS Anywhere; AWS picks Cilium for Networking & Security on EKS Anywhere; Google chooses Cilium in Datapath V2 for GKE and Anthos; Azure CNI Powered by Cilium; Alibaba Cloud Turns to Cilium for Improved Network Performance and Scalability; DigitalOcean Picks Cilium as the Preferred Networking Solution for DOKS; SysEleven Picks Cilium as the Preferred Networking Solution for their Platform; QingCloud boosted performance and networking with Cilium; OVHcloud uses Cilium as the default CNI for their Managed Kubernetes Service.

### Financial Services

URL: https://cilium.io/industries/financial-services

Regulatory oversight is a hallmark of the financial industry. Cilium's detailed traffic monitoring and logging mechanisms support institutions in establishing transparent audit trails, complying with stringent regulations, and conducting forensic investigations. The world of fintech revolves around APIs – from mobile banking apps to trading platforms. Cilium's API-aware network security ensures that these critical APIs can be secured, empowering teams with the toolsets needed to understand and guard against malicious patterns in API calls. Cilium supports L7 policies (e.g., allow HTTP GET /foo) for fine-grained access control to shared API services running common cloud native protocols like HTTP, gRPC, Kafka, etc. Cilium also supports deny-based, cluster-wide, and host-layer firewall network policies. With Cilium, companies in the financial services industry can achieve improved observability, maintain security controls, and weave compliance and security governance for Kubernetes environments directly into the DevOps process.

#### Weave Compliance and Security Throughout the DevOps Process

By moving from IP to identity, Cilium empowers you to free your security and operations teams from the need to manually review and audit every policy change. Cilium's native integration with the Kubernetes ecosystem enables it to seamlessly weaves compliance and security governance into the DevOps process. For example, operators can write network policies based on namespaces or labels rather than hardcoding IPs.

#### Enforce Zero-Trust Security with Ease. No Disparate Tools, No Performance Hits

Adopting a Zero Trust-based approach to network security architecture is essential for companies in the financial service industry to maintain compliance with stringent regulations and keep customer data secure. Cilium provides a unified tool that implements a range of features to enforce Zero Trust networking security principles. These features include identity-aware service to service communication and observability, advanced network policies with native HTTP and DNS protocol support, transparent encryption, and using in-kernel IPsec or WireGuard. Enforcement of TLS via Network Policy allows operators to restrict the allowed TLS SNIs in their network, and provide a more secure environment. Tetragon provides powerful security observability and real time enforcement guardrails. Cilium offers broad cluster-wide network policies that can be mixed with Tetragon tracing policies for more specific protocol controls like system calls, TCP/IP, file access, and namespace privilege capabilities.

#### “Always On” Workload Analysis for Continuous Compliance

In a highly regulated industry such as financial services, ensuring that software and systems comply with these regulations is a constant challenge for development and operations teams. Cilium empowers teams with tools to ensure continuous compliance with standards like FIPS, PCI DSS, Open Banking, and SOC 2 by offering real-time monitoring and analysis of workload communication to identify non-compliant connections. It emphasizes encryption, traffic analysis, and full-stack monitoring without imposing performance constraints, ensuring confidentiality, integrity, and availability in regulated environments.

_Adopters & resources referenced on this page:_ x12k; 1000s; Transparent Encryption; Network Policy; Egress Gateway; eBPF, a road to invisible network: S&P Global's Network Transformation Journey; Cilium in Practice: Building Data Sandboxes at Bloomberg; High Scale Networking for ML workloads with Cilium; Post Finance picks Cilium for Cloud Native Networking; How Capital One used eBPF and Cilium to build a secure, maintainable PaaS; More Churn No Problem: Lessons Learned Running Cilium in Production; Strengthening Security Across Distributed Kubernetes Clusters; Building a Resilient Payments Platform with Cilium; Self-service, Zero Trust Network Security; High Availability for Cilium Egress Gateway; High-Scale Networking for ML Workloads With Cilium.

### Media and Entertainment

URL: https://cilium.io/industries/media-entertainment

The Media and Entertainment industry is navigating a digital transformation era with scalability, security, and latency challenges, traditional tooling struggles to handle rising consumer demands for high-quality, on-demand media. Security vulnerabilities can threaten valuable content and customer data, while latency issues undermine user experience. Rising operational costs and increasingly stringent regulatory environments compound these challenges. Media companies must deliver high-quality content to a vast and global audience. This requires infrastructure that can handle spikes in demand, especially during live events and popular content releases. Cilium offers intelligent load balancing, ensuring that the high volume of user requests typical for OTT platforms are distributed efficiently. This results in consistent content and smooth media experiences for end users. Load balancing across multi-cloud and hybrid cloud clusters is easier to configure and operate with Cilium, allowing the design of more resilient and fault-tolerant infrastructure. Leverage the Cilium advantage to deliver seamless experiences to your users, reduce operational complexity for your teams, and secure your environments.

#### Global Content Delivery at Scale with Cilium's Load Balancing

Companies in the media and entertainment industry with strict latency and failover requirements can benefit from Cilium's scalable load balancing with XDP, an eBPF-based high-performance data path. Production users found it doubled throughput while reducing CPU usage by 72x. Cilium's standalone Layer 4 load balancer comes at a fraction of the cost of traditional hardware load balancers while delivering higher performance.

#### Robust Security, Scalability, and Flexibility at a Reduced Operational Overhead

Many companies in the media and entertainment industries maintain legacy infrastructure for varying reasons. Cilium is flexible enough to integrate with both traditional and cloud native infrastructures. Companies like Cosmonic have extended Cilium beyond Kubernetes to platforms like Nomad. In scenarios where it is impossible to use Cilium as the Default CNI, Cilium's CNI chaining mode lets you utilize Cilium's features like Hubble for observability and Tetragon for security. Companies running workloads on non-k8s environments can leverage Cilium's Layer 4 standalone load balancer for efficient and scalable load balancing. This flexibility ensures your organization can benefit from Cilium regardless of your infrastructure choices.

_Adopters & resources referenced on this page:_ TikTok's IPv6 Journey To Cilium; Software L4 Load Balancing for Kubernetes Services at Yahoo; Designing and Securing a Multi-Tenant Runtime Environment at the New York Times; Zero Trust Networking at Scale (20k+ VCPUs, 100+ Dev Teams); Scaling Network Performance and Cutting Costs with Cilium; How Zynga Scaled Past Conntrack Limits with Cilium; Load Balancing; Bandwidth and Latency Optimization; Service Map.

### E-commerce

URL: https://cilium.io/industries/e-commerce

E-commerce platforms have a unique set of networking needs, in most cases, a highly dynamic microservices architecture, exponential increases in user traffic, diverse geographic distribution, and stringent data security requirements. Cilium leverages eBPF, a new powerful Linux kernel technology, to address these unique needs, offering high-performance cloud native-aware networking, observability, and security. E-commerce teams can scale globally and deliver better user experiences through a low-latency network path from Cilium's eBPF-optimized data plane. Cilium also provides a wide range of security and observability features that support E-commerce teams in implementing compliance frameworks such as FIPS, PCI DSS, GDPR, Fedramp, and SOC 2. Join the next wave of high-performance e-commerce platforms leveraging the advanced capabilities of eBPF with Cilium to build scalable and secure e-commerce experiences. Gain the edge in speed, security, and observability.

#### Build Globally Distributed Services. Gain The Edge in Speed, Security, and Observability

E-commerce platforms with offerings spanning a wide geographic distribution often need to deploy their platforms close to their users to reduce latency. Cilium provides a wide range of features enabling e-commerce platforms to build infrastructure across multiple clouds, regions, availability zones, and geographic locations without sacrificing performance. Cilium Cluster Mesh effectively allows joining multiple clusters into a large unified network, regardless of the Kubernetes distribution or location each of them is running. Cilium also offers scalable and efficient load balancing and ingress, enabling maglev-supported load balancing on L3/L4 for N/S traffic and E/W, including DSR and transparent k8s ingress controllers. With a Cilium-enabled infrastructure, e-commerce platforms can confidently expand their reach and offer fast, reliable, and globally accessible e-commerce experiences to their customers.

_Adopters & resources referenced on this page:_ How Flipkart Scales Kubernetes Networking to Millions of RPS with Cilium; Unleashing the Power of Cilium CNI to Propel Trendyol’s Performance Up to 40%!; IKEA Private Cloud: Connecting 390+ Stores and 4.3 Billion Website Visitors; How Trip.com Group switched to Cilium For Scalable and Cloud Native Networking; Building a Kubernetes Platform with Cilium; High Performance Data Backend for Your AI/ML with MinIO and Cilium Cluster Mesh; Bandwidth and Latency Management; Service Mesh; High Performance Networking (CNI); What Does Cilium Offer the E-commerce Industry?; Microservices Architectures; Multi-Cluster and Hybrid-Cloud Environments; Large-Scale Kubernetes Deployments; Securing Customer Data.

### Telcos and Data Center Operators

URL: https://cilium.io/industries/telcos-datacenters

Telcos grapple with the challenges of maintaining legacy technologies, complying with regulatory constraints, and the complexity of managing and delivering diversified service offerings. In an increasingly evolving landscape, advancements with technologies like 5G further amplify the demand for flexible and efficient network solutions. Cilium provides a comprehensive suite of capabilities that empower telcos and data center providers to transition seamlessly from traditional infrastructure to cloud native environments. As operators transition from IPv4 to IPv6, Cilium's NAT46/64 support provides a smooth pathway to ease this transition. Cilium's IPv6 support with BGP lets users advertise their IPv6 Pod CIDRs. Cilium's SRv6 L3VPN offers a scalable and flexible solution for interconnecting multiple sites while maintaining end-to-end network slicing and service isolation. Notably, Cilium supports SCTP, a protocol typically used by service providers and mobile operators. Simplify your Telco and data center networks, comply with regulatory requirements, and ease your cloud native journey with Cilium.

#### Simplify Integration Between Kubernetes Environments and Existing Network Infrastructure With CIlium’s BGP

Telco and data center operators often have a blend of cloud native and legacy systems that require seamless communication. BGP, a common data center connectivity protocol is not natively integrated in Kubernetes. With Cilium's native BGP support, operators can interconnect their legacy environment with a Cilium-managed environment, integrate seamlessly with Top of Rack devices, and automatically advertise pod CIDRs to BGP neighbors. Operators can also significantly reduce CAPEX by leveraging Cilium's standalone load balancer to replace expensive legacy hardware load balancers.

#### Future-proof Operations with Cilium’s Dual Stack IPv4/IPv6 and NAT46/64 Support

Cilium's dual-stack IPv4/IPv6 networking allows each pod to be allocated IPv4 and IPv6 addresses, ensuring seamless communication with modern IPv6 systems and legacy IPv4 applications and services. For businesses in the process of transitioning from IPv4 to IPv6, Cilium's NAT46/64 support provides a pathway to ease this transition. Cilium's IPv6 support with BGP lets users advertise their IPv6 Pod CIDRs. Together, these features offer scalability and prepare businesses for a future where IPv6 will be the norm rather than the exception.

#### Connect Multiple Sites While Maintaining End-to-End Network Slicing and Service Isolation with Cilium’s SRv6 L3VPN

Cilium supports SRv6 L3VPN, enabling users to cross-connect Kubernetes worker nodes to other services and Kubernetes clusters using Segment Routing over IPv6 (SRv6). This feature allows users to create virtual private networks that span multiple sites, providing secure and isolated connectivity between Kubernetes clusters, data centers, and public clouds.

_Adopters & resources referenced on this page:_ Leveraging Cilium and SRv6 for Telco Networking; eCHO Episode 59: Dual Stack with Cilium; BGP with Cilium; Implementing Telco Services with the Cloud; High Performance Cloud Native Networking (CNI); Bandwidth and Latency Optimization.

### Edge Computing

URL: https://cilium.io/industries/edge-computing

Edge computing is transforming how businesses deploy and manage applications across distributed environments. As workloads move closer to data sources and end-users, Cilium emerges as the ideal networking solution to address the unique challenges of edge infrastructures. In the race to bring computing power closer to data sources and users, edge deployments face security, observability, and network efficiency challenges. By leveraging the power of eBPF, Cilium tackles these hurdles head-on, providing a unified networking solution that secures edge infrastructures, delivers deep visibility, and optimizes performance, even in resource-constrained environments.

#### Deep Visibility for Edge Operations

Cilium revolutionizes observability in complex edge environments where traditional monitoring tools fall short. By leveraging Hubble for real-time network flow analysis and Tetragon for security event monitoring, Cilium provides deep security and visibility across distributed edge topologies. Cilium's seamless integration with SIEM and monitoring platforms allows organizations to maintain consistency in their security and observability practices from cloud to edge. With Cilium, operators gain deep, actionable insights into their edge infrastructure, empowering them to quickly identify, troubleshoot, and resolve issues across their entire network, thereby maintaining optimal performance and security in even the most challenging edge computing scenarios.

#### Optimized for Resource-Constrained Environments

Edge computing often means working with limited resources. Cilium is designed to be lightweight and efficient, leveraging cutting-edge eBPF technology to deliver unparalleled performance and efficiency in resource-constrained edge environments. Cilium is optimized for both x86 and ARM-based architectures. By implementing load balancing directly in XDP, Cilium achieves remarkable resource efficiency. With Cilium, businesses can maximize the potential of their edge infrastructure, handling more traffic with fewer resources. This efficiency translates to reduced hardware costs, lower energy consumption, and the ability to deploy more powerful edge applications on existing hardware.

#### Seamless Connectivity for Edge Infrastructures

Cilium's ClusterMesh feature revolutionizes edge computing by seamlessly extending Cilium's powerful networking and security capabilities across multiple clusters. It enables secure, high-performance connectivity between edge locations, data centers, and cloud resources without complex VPN setups. ClusterMesh facilitates global service discovery, unified security policies, and intelligent cross-cluster load balancing, ensuring consistent protection and optimal performance across your entire edge infrastructure. This unified approach significantly reduces operational complexity while enhancing the resilience and efficiency of distributed edge applications.

_Adopters & resources referenced on this page:_ Which Edge Are You on? Service Affinity with Cilium Cluster Mesh; Connecting Clusters on the Edge with Deep Dive into Cilium Cluster Mesh; Meshing It up Securely: How Roche Manages Network Connectivity for 1000+ Edge Clusters.

### Security

URL: https://cilium.io/industries/security

For businesses at the forefront of cybersecurity and those for whom security is integral to their operations, the rise of cloud native applications presents immense opportunities and accompanying challenges. The dynamism of cloud native architectures, from microservices to containerization and distributed computing, demands a paradigm shift in securing these environments. Traditional security approaches struggle to adapt to the ephemeral and dynamic nature of cloud native applications. Cilium delivers robust cloud native security with features like transparent encryption, mutual authentication, security observability, advanced network polices, egress gateway, and runtime enforcement. Leveraging eBPF, Cilium offers efficient observability across the entire application stack, integrates seamlessly with SIEM systems, and ensures compliance with standards like FIPS, FedRAMP, and SOC. Cilium aligns security practices with the realities of building, maintaining, and scaling cloud native applications.

#### Better Understand Security Breaches and Recover Quickly with Cilium’s Forensics Capabilities

Critical workloads that run in a Kubernetes environment require cloud native-aware tooling to perform efficient incident investigations and monitor key compliance requirements. Cilium's forensic capabilities enable SecOps teams and App owners to conduct security analysis with a native understanding of cloud native identities. Cilium provides a deep understanding of network flows from L3/L4 up to L7 and runtime visibility from OS to code executions in the pod.

#### Secure Modern Applications with Cilium's Advanced Network Policies. Scale Policies with Identities, Not IPs

Modern distributed applications rely on containers to facilitate agility in deploying new versions of their application and to scale out on demand. Typical firewalls secure workloads by filtering source IP addresses and ports, but in Kubernetes and other cloud native platforms, IP addresses are ephemeral. Traditional firewalls are not cloud native aware and can't be programmed on the fly as applications scale out or new versions are deployed. Updating the firewall constantly to adapt to the constant changes becomes impossible at scale. Cilium features network policies that operate at layers 3, 4, and 7, providing more flexibility in managing ingress and egress traffic. By leveraging eBPF, Cilium can insert security rules based on service/pods/container identity rather than an IP address for identification as in the traditional systems. Cilium makes applying security policies in a dynamic container environment scalable by decoupling security from IP addressing, providing stronger security isolation, and adding the following functionality to the Kubernetes cluster.

#### Ease Integration with Traditional Firewall Systems Using Cilium's Static Egress Gateway

With Cilium's Static Egress Gateway, cloud native workloads can be presented from stable IP addresses, easing integration with traditional firewall systems. This approach ensures that firewall requirements remain consistent even as workloads scale, fostering a bridge between cloud native and conventional environments.

#### Bolster Security, Streamline Infrastructure, Reduce Complexity Without Sacrificing Performance Using Cilium's Mutual Authentication

Organizations are increasingly looking to bolster their security posture in cloud native environments. Cilium's sidecar-free service mesh uses mutual authentication to optimize security and performance, ensuring that services authenticate each other's identities before communication occurs. Cilium features network policies that operate at layers 3, 4, and 7, providing more flexibility in managing ingress and egress traffic. By leveraging eBPF, Cilium can insert security rules based on service/pods/container identity rather than an IP address for identification as in the traditional systems. Cilium makes applying security policies in a dynamic container environment scalable by decoupling security from IP addressing, providing stronger security isolation, and adding security functionality directly to the Kubernetes cluster.

_Adopters & resources referenced on this page:_ Tetragon: eBPF-based Security Observability and Runtime Enforcement; Monitor Process Execution; Runtime Security Policies; Real Time Enforcement; The Next Log4jshell? Preparing for CVEs with eBPF!; Tutorial: Getting Familiar with Security Observability Using eBPF &Cilium Tetragon; Mutual Authentication with Cilium; Migrating to Cilium for Better Networking, Visibility and Security; Transparent Encryption; Network Policy; Runtime Security.

### SaaS, Software, and DBaaS

URL: https://cilium.io/industries/software

In today's fast-evolving software landscape, teams behind SaaS, standalone software, and DBaaS offerings grapple with networking, security, and scalability challenges. Traditional networking solutions often fail to support distributed systems' complex demands spanning multiple clusters and clouds. Furthermore, the rise of microservices intensifies the need for fine-grained security measures and deep visibility into application communications. Cilium is purpose-built to secure and accelerate the delivery of cloud native applications. It leverages eBPF to provide API-aware networking and security with unparalleled granularity without compromising performance. With Cilium, SaaS, software, and DBaaS teams can establish infrastructure capable of scaling their products globally to reach customers wherever they may be. Cilium enables the creation of secure Kubernetes multi-tenant environments, empowering these companies to meet the challenges of modern software deployment head-on.

#### Align with DevOps and GitOps Principles

One of the core tenets of DevOps is automation. Cilium's architecture ensures that all its components can be deployed automatically, integrating into existing workflows and augmenting automation capabilities. Platform engineering teams can seamlessly integrate Cilium into their CI/CD pipelines, ensuring continuous delivery and integration without manual intervention. Cilium is designed to be configured using an API. This aligns with the GitOps principles, where the desired state of infrastructure is declared in code and stored in Git repositories. The API-driven model of Cilium enables organizations to manage, scale, and modify their infrastructure using code, ensuring consistency and repeatability.

#### Streamline Onboarding New Applications and Teams and Accelerate Debugging with Hubble's Service Map

Cilium's Hubble Service Map allows platform teams to furnish application developers with a portal to view their own workloads, dependencies, and network flows. This self-service model makes it easier to identify service connections, calls, and even DNS operations, all crucial for diagnosing application networking problems. The service map helps streamline the onboarding of new applications and teams and accelerates the debugging process. By mitigating disputes between application and infrastructure operation teams, Cilium accelerates the investigation of application-layer issues, enabling frictionless, self-service root cause analysis for developers.

#### Build Secure Kubernetes Multi-tenant Environments With Cilium

For enterprises building muti-tenant architectures, Cilium enforces isolation between tenants and ensures different teams can safely run on the same platform. With Cilium's multi-tenant observability features, teams can easily access application metrics without compromising security. Traditional logging systems often fall short when segregating data by tenants. Cilium provides multi-tenant, self-service access using the OpenID Connect (OIDC) standard. Teams can have RBAC-based access to relevant data and connectivity metrics, such as network policy drops, DNS lookup failures, and more.

_Adopters & resources referenced on this page:_ Tales from an eBPF Program’s Murder Mystery; Running Cilium with Nomad; eBPF at Adobe, Brandon Cook, Adobe; How Ascend Leverages Cilium as a Networking Layer; How ClickHouse is Using Cilium to Implement Efficient Network Policies; Implementing Zero Trust Security with Cilium.

### Consulting

URL: https://cilium.io/industries/consulting

Consultancy companies build and maintain software and infrastructure for diverse customers varying in size, industry, and technology stack. Customers typically have different requirements for fault tolerance, scalability, financial expenses, security, and more. As infrastructure stacks continue to evolve, consultancy firms need solutions that meet client expectations and also prepare their platforms for the future. World-leading consulting companies are now turning to Cilium, leveraging its capabilities to deliver secure, high-performance, and observable cloud native solutions to clients ranging from startups, nationwide banks, and large enterprises. By adopting Cilium, consultancies ensure streamlined operations across varied client environments, positioning themselves at the forefront of cloud native innovation.

#### Easily Integrate With Customer's Infrastructure.

Consulting companies cater to various clients with different choices of infrastructures and cloud providers. Operating across different environments often results in additional operational and resource overhead. Cilium provides a consistent experience for operators across different environments. With Cilium's cluster mesh, multiple clusters can be joined into a large unified network regardless of the Kubernetes distribution or location each is running. Cilium also reduces the number of tools operators have to deal with; for example, Cilium's load balancing can replace traditional hardware load balancers. Cilium's Layer 2 announcement can eliminate the need for extra tooling like MetalLB. Together, these features ensure that consulting companies can offer uniform, high-quality services regardless of their client's chosen infrastructure.

#### Visibility Into Customers Infrastructure Across Clouds, Clusters, and Premises, Independent of The Underlying Platform

Cilium's Hubble delivers comprehensive visibility across various platforms, whether on-premises, cloud, or clusters. This unified data plane approach allows for consistent monitoring across deployments, aiding in workload migration between clusters and simplifying the onboarding of new infrastructure providers. Observability data from Cilium is made available using modern standards such as Prometheus and can be visualized using powerful tools such as Grafana. Cilium also supports traditional standards such as sFlow and NetFlow.

_Adopters & resources referenced on this page:_ Cilium for Kubernetes networking: Why we use it and why we love it; Reducing Kubernetes tool sprawl: Tietoevry uses Cilium and Hubble; Fostering a cloud native approach with Cilium.