istio代理集群外部的gitlab

背景

合作伙伴的私有云平台只有一个LB资源和一个公网IP，DNS解析记录*.aa.bb.cc.cn已通配至公网IP。LB资源已绑定到k8s的istio网关上。

ingress gw上已经配置了tls实现，gw上的tls和后端gitlab上的tls是同一个自签名证书

需要实现：通过gitlab.aa.bb.cc.cn来访问集群外的gitlab服务

思路

通过istio的CR ServiceEntry手动将网格外部服务添加到网格内部的服务注册表registry中，再使用vs将ingressgateway处的流量路由到serviceentry

实现

这里有两种方法：

1. 后端gitlab使用http协议访问
2. 后端gitlab使用https协议访问

原因是：外部client使用https请求到istio的gw后进行tls终止/卸载，使用http请求后端同样是只有http协议的gitlab服务，gitlab根据external_url返回重定向到http链接再返回给外部client导致外部client访问从https变成http

实现

方法1

创建serviceEntry

root@k8s-master01:/data/istio/gitlab-service-entry/host# cat gitlab-serviceentry-host.yaml 
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: gitlab-external
  namespace: istio-system
spec:
  hosts:
  - gitlab.aa.bb.cc.cn
  addresses:
  - 10.17.3.95/32
  location: MESH_EXTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: STATIC
  endpoints:
  - address: 10.17.3.95

创建vs资源版本1可以实现http代理，

root@k8s-master01:/data/istio/gitlab-service-entry/host# cat gitlab-vs-host.yaml 
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: gitlab-vs
  namespace: istio-system
spec:
  hosts:
  - gitlab.aa.bb.cc.cn
  gateways:
  - istio-system/ingressgateway
  http:
  - name: gitlab
    match:
    - uri:
        prefix: /
    rewrite:
      uri: /
    route:
    - destination:
        host: gitlab.aa.bb.cc.cn
        port:
          number: 80

版本2，可以实现部分url自动跳转到https，但一些url还需要手动添加https协议，影响体验。

后端gitlab会基于client请求的协议生成重定向 URL。

在vs里面添加headers告诉gitlab客户端是来自https的请求，服务端gitlab在收到istio的请求后，重定向至https上，

root@k8s-master01:/data/istio/gitlab-service-entry/host# cat gitlab-vs-host-https.yaml 
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: gitlab-vs
  namespace: istio-system
spec:
  hosts:
  - gitlab.aa.bb.cc.cn
  gateways:
  - istio-system/ingressgateway
  http:
  - name: gitlab
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: gitlab.aa.bb.cc.cn
        port:
          number: 80
    headers:
      request:
        set:
          X-Forwarded-Proto: "https"

方法2

gitlab配置https

将https证书和key放在/etc/gitlab/ssl文件夹中
/etc/gitlab/gitlab.rb中更改两个地方

mkdir -pv /etc/gitlab/ssl
chmod 755 /etc/gitlab/ssl
# 上传证书文件

gitlab.rb配置

external_url 'https://gitlab.aa.bb.cc.cn'
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/aa.bb.cn.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/aa.bb.cn.key"

应用配置

gitlab-ctl reconfigure

验证80&443端口是否监听

ss -tanlp |egrep "80|443"

istio配置

使用gitlab的ca.crt创建secrets，使得istio从k8s的资源secrets中动态加载sds配置

kubectl create secret generic gitlab-ca-secret --from-file=ca.crt=./aa.bb.cn.crt -n istio-system

验证secrets中字段是否是ca.crt，否则会报错upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: Secret is not supplied by SDS

root@k8s-master01:/data/istio/gitlab-service-entry/host-https# kubectl describe secrets gitlab-ca-secret
Name:         gitlab-ca-secret
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
ca.crt:  1233 bytes

serviceEntry资源配置

root@k8s-master01:/data/istio/gitlab-service-entry/host-https# cat gitlab-serviceentry-host.yaml 
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: gitlab-external
  namespace: istio-system
spec:
  hosts:
  - gitlab.aa.bb.cc.cn
  addresses:
  - 10.17.3.95/32
  location: MESH_EXTERNAL # 指定服务的位置是在服务网格外部
  ports:
  - number: 443
    name: https
    protocol: TLS
  resolution: STATIC # 静态IP地址解析模式
  endpoints: # 后端服务gitlab的地址
  - address: 10.17.3.95

vs资源配置

root@k8s-master01:/data/istio/gitlab-service-entry/host-https# cat gitlab-vs-host-https.yaml 
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: gitlab-vs
  namespace: istio-system
spec:
  hosts:
  - gitlab.aa.bb.cc.cn
  gateways:
  - istio-system/ingressgateway
  http:
  - name: gitlab
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: gitlab.aa.bb.cc.cn
        port:
          number: 443

DS资源配置

root@k8s-master01:/data/istio/gitlab-service-entry/host-https# cat gitlab-ds-https.yaml 
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: gitlab-destinationrule
  namespace: istio-system
spec:
  host: gitlab.aa.bb.cc.cn
  trafficPolicy:
    tls:
      mode: SIMPLE 
      credentialName: gitlab-ca-secret 

验证

使用https访问会自动跳转至https协议，gitlab-runner也注册正常。

reference

gitlab ssl配置
https://docs.gitlab.com/omnibus/settings/ssl/
https://istio.io/latest/docs/reference/config/networking/service-entry/