11. Cilium Service Mesh
With release 1.12 Cilium enabled direct ingress support and service mesh features like layer 7 loadbalancing
Task 11.1: Installation
helm upgrade -i cilium cilium/cilium --version 1.12.10 \
--namespace kube-system \
--reuse-values \
--set ingressController.enabled=true \
--wait
For Kubernetes Ingress to work kubeProxyReplacement needs to be set to strict or partial. This is why we stay on the kubeless cluster.
Wait until cilium is ready (check with cilium status). For Ingress to work it is necessary to restart the agent and the operator.
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/cilium
Task 11.2: Create Ingress
Cilium Service Mesh can handle ingress traffic with its Envoy proxy.
We will use this feature to allow traffic to our simple app from outside the cluster. Create a file named ingress.yaml with the text below inside:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: backend
spec:
ingressClassName: cilium
rules:
- http:
paths:
- backend:
service:
name: backend
port:
number: 8080
path: /
pathType: PrefixApply it with:
kubectl apply -f ingress.yaml
Check the ingress and the service:
kubectl describe ingress backend
kubectl get svc cilium-ingress-backend
We see that Cilium created a Service with type Loadbalancer for our Ingress. Unfortunately, Minikube has no loadbalancer deployed, in our setup the external IP will stay pending.
As a workaround, we can test the service from inside Kubernetes.
SERVICE_IP=$(kubectl get svc cilium-ingress-backend -ojsonpath={.spec.clusterIP})
kubectl run --rm=true -it --restart=Never --image=curlimages/curl -- curl --connect-timeout 5 http://${SERVICE_IP}/public
You should get the following output:
[
{
"id": 1,
"body": "public information"
}
]pod "curl" deleted
Task 11.3: Layer 7 Loadbalancing
Ingress alone is not really a Service Mesh feature. Let us test a traffic control example by loadbalancing a service inside the proxy.
Start by creating the second service. Create a file named backend2.yaml and put in the text below:
---
apiVersion: v1
data:
default.json: |
{
"private": [
{ "id": 1, "body": "another secret information from a different backend" }
],
"public": [
{ "id": 1, "body": "another public information from a different backend" }
]
}
kind: ConfigMap
metadata:
name: default-json
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-2
labels:
app: backend-2
spec:
replicas: 1
selector:
matchLabels:
app: backend-2
template:
metadata:
labels:
app: backend-2
spec:
volumes:
- name: default-json
configMap:
name: default-json
containers:
- name: backend-container
env:
- name: PORT
value: "8080"
ports:
- containerPort: 8080
image: docker.io/cilium/json-mock:1.2
imagePullPolicy: IfNotPresent
volumeMounts:
- name: default-json
mountPath: /default.json
subPath: default.json
---
apiVersion: v1
kind: Service
metadata:
name: backend-2
labels:
app: backend-2
spec:
type: ClusterIP
selector:
app: backend-2
ports:
- name: http
port: 8080Apply it:
kubectl apply -f backend2.yaml
Call it:
kubectl run --rm=true -it --restart=Never --image=curlimages/curl -- curl --connect-timeout 3 http://backend-2:8080/public
We see output very similiar to our simple application backend, but with a changed text.
As layer 7 loadbalancing requires traffic to be routed through the proxy, we will enable this for our backend Pods using a CiliumNetworkPolicy with HTTP rules. We will block access to /public and allow requests to /private:
Create a file cnp-l7-sm.yaml with the following content:
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "rule1"
spec:
description: "enable L7 without blocking"
endpointSelector:
matchLabels:
app: backend
ingress:
- fromEntities:
- "all"
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: "GET"
path: "/private"
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "rule2"
spec:
description: "enable L7 without blocking"
endpointSelector:
matchLabels:
app: backend-2
ingress:
- fromEntities:
- "all"
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: "GET"
path: "/private"And apply the CiliumNetworkPolicy with:
kubectl apply -f cnp-l7-sm.yaml
Until now only the backend service is replying to Ingress traffic. Now we configure Envoy to loadbalance the traffic 50/50 between backend and backend-2 with retries.
We are using a CustomResource called CiliumEnvoyConfig for this. Create a file envoyconfig.yaml with the following content:
apiVersion: cilium.io/v2
kind: CiliumEnvoyConfig
metadata:
name: envoy-lb-listener
spec:
services:
- name: backend
namespace: default
- name: backend-2
namespace: default
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
name: envoy-lb-listener
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: envoy-lb-listener
rds:
route_config_name: lb_route
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
- "@type": type.googleapis.com/envoy.config.route.v3.RouteConfiguration
name: lb_route
virtual_hosts:
- name: "lb_route"
domains: ["*"]
routes:
- match:
prefix: "/private"
route:
weighted_clusters:
clusters:
- name: "default/backend"
weight: 50
- name: "default/backend-2"
weight: 50
retry_policy:
retry_on: 5xx
num_retries: 3
per_try_timeout: 1s
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
name: "default/backend"
connect_timeout: 5s
lb_policy: ROUND_ROBIN
type: EDS
outlier_detection:
split_external_local_origin_errors: true
consecutive_local_origin_failure: 2
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
name: "default/backend-2"
connect_timeout: 3s
lb_policy: ROUND_ROBIN
type: EDS
outlier_detection:
split_external_local_origin_errors: true
consecutive_local_origin_failure: 2Note
If you want to read more about Envoy configuration Envoy Architectural Overview is a good place to start.Apply the CiliumEnvoyConfig with:
kubectl apply -f envoyconfig.yaml
Test it by running curl a few times – different backends should respond:
for i in {1..10}; do
kubectl run --rm=true -it --image=curlimages/curl --restart=Never curl -- curl --connect-timeout 5 http://backend:8080/private
done
We see both backends replying. If you call it many times the distribution would be equal.
[
{
"id": 1,
"body": "another secret information from a different backend"
}
]pod "curl" deleted
[
{
"id": 1,
"body": "secret information"
}
]pod "curl" deleted
This basic traffic control example shows only one function of Cilium Service Mesh, other features include i.e. TLS termination, support for tracing and canary-rollouts.
Task 11.4: Cleanup
We don’t need this cluster anymore and therefore you can delete the cluster with:
minikube delete --profile kubeless