k8s运维:记 istio ingressgateway 关于TCP路由的一个小坑

因为之前都是用 istio ingressgateway 做 L7 负载,这次收到一个集群 kafka broker 暴露需求,在设置的时候习惯了 HTTP 的思路,所以才了坑。

首先来说下 istio ingressgateway 做 L7 层负载工作原理。

首先我们会先设置一个 gateway 绑定作用的 ingressgateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: external-test-gateway
spec:
selector:
istio: ingressgateway-external
servers:
- hosts:
- '*.rcmd.testing.mpengine'
port:
name: http
number: 16354
protocol: HTTP

gateway 资源会被 istiod 监控,当一个新的 gateway 创建的时候, istiod 会根据 gateway 的 selector 选择符合条件的 istio proxyv2 的 pod (也就是istio ingressgateway的运行实例),我们可以看 istiod 日志:

$ kubectl logs -f istiod-xxxxx -nistio-system
2020-12-30T09:40:57.178237Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:17
2020-12-30T09:40:57.178307Z info ads EDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:16 empty:0 cached:16/16
2020-12-30T09:40:57.178581Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:4
2020-12-30T09:40:57.178896Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:2
2020-12-30T09:41:13.584044Z info ads Push debounce stable[5] 1: 100.121839ms since last change, 100.121637ms since last push, full=true
2020-12-30T09:41:13.584239Z info ads XDS: Pushing:2020-12-30T09:41:13Z/1 Services:63 ConnectedEndpoints:2
2020-12-30T09:41:13.584569Z info ads CDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:1
2020-12-30T09:41:13.584615Z info ads LDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:0
2020-12-30T09:41:13.585106Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:17
2020-12-30T09:41:13.585248Z warn buildGatewayListeners: found 0 services on port 16354: map[]
2020-12-30T09:41:13.585466Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:5
2020-12-30T09:41:13.585674Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:2
2020-12-30T09:41:13.595165Z warn constructed http route config for route http.16354 on port 16354 with no vhosts; Setting up a default 404 vhost
2020-12-30T09:41:13.595493Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-vrmwq.istio-system resources:3

对于 HTTP 协议的 gateway 会直接在 istio-ingressgateway 上开启监听端口:

$ kubectl exec -it -nistio-system istio-ingressgateway-external-xxxxxx -- bash
$ netstat -anp | grep 16354
tcp 0 0 0.0.0.0:16354 0.0.0.0:* LISTEN 19/envoy

这时候再设置 VirtualService 让 envoy 匹配转发到对应服务,如下:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: test-backend
spec:
gateways:
- external-test-gateway
hosts:
- test.rcmd.testing.mpengine
http:
- route:
- destination:
host: prometheus.cluster-monitor.svc.cluster.local
port:
number: 80

这是 HTTP 协议转发实现,但istio ingressgateway 在 TCP 协议是不同的,对于 TCP 类型的 gateway 只有同时修改 gateway 和 virtualService才能开启端口,同样做个测试:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: external-test-gateway
namespace: rcmd
spec:
selector:
istio: ingressgateway-external
servers:
- hosts:
- '*.rcmd.testing.mpengine'
port:
name: tcp
number: 16354
protocol: TCP # 改成TCP协议接口

到 ingressgateway 的 pod 里面可以发现 16354 端口并没有启动:

$ kubectl exec -it -nistio-system istio-ingressgateway-external-xxxxxx -- bash
$ netstat -anp | grep 16354 # 为空

看 istiod 日志:

020-12-30T10:14:30.345439Z     info    ads     Push debounce stable[23] 1: 100.094272ms since last change, 100.094082ms since last push, full=true
2020-12-30T10:14:30.345683Z info ads XDS: Pushing:2020-12-30T10:14:30Z/11 Services:63 ConnectedEndpoints:2
2020-12-30T10:14:30.346030Z info ads CDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:1
2020-12-30T10:14:30.346065Z info ads LDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:0
2020-12-30T10:14:30.346784Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:17
2020-12-30T10:14:30.346816Z warn buildGatewayListeners: found 0 services on port 16354: map[]
2020-12-30T10:14:30.346823Z warn no virtual service bound to gateway: rcmd/external-test-gateway
2020-12-30T10:14:30.346941Z info gateway omitting listener "0.0.0.0_16354" due to: must have more than 0 chains in listener "0.0.0.0_16354"
2020-12-30T10:14:30.347122Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:4
2020-12-30T10:14:30.347295Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:2

添加一个 vs:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: external-test-router
namespace: rcmd
spec:
gateways:
- external-test-gateway
hosts:
- "kafka-broker0.rcmd.testing.mpengine"
tcp:
- match:
- port: 16354
route:
- destination:
host: kafka-broker0.rcmd.svc.cluster.local
port:
number: 9092

提交后,查看 istiod 日志,可以看到端口被成功创建 :

2020-12-30T10:21:11.541307Z     info    ads     XDS: Pushing:2020-12-30T10:21:11Z/14 Services:63 ConnectedEndpoints:2
2020-12-30T10:21:11.541573Z info ads CDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:1
2020-12-30T10:21:11.541621Z info ads LDS: PUSH for node:istio-ingressgateway-85b5cf599b-qcj85.istio-system resources:0
2020-12-30T10:21:11.542446Z info ads CDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:17
2020-12-30T10:21:11.542602Z warn buildGatewayListeners: found 0 services on port 16354: map[]
2020-12-30T10:21:11.542767Z info ads LDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:5
2020-12-30T10:21:11.543005Z info ads RDS: PUSH for node:istio-ingressgateway-external-5b4765b84-v6bxd.istio-system resources:2

shikanon wechat
欢迎您扫一扫,订阅我滴↑↑↑的微信公众号!