nginx - Websocket在客户端上返回500，在服务器上返回101

Question

我们尝试使用nginx-ingress Controller 在Kubernetes集群上实现WebSocket。

ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/proxy-read-timeout: "7200"
    ingress.kubernetes.io/proxy-send-timeout: "7200"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
  generation: 2
  labels:
    app: websocket
    app.kubernetes.io/managed-by: Helm
    chart: websocket-0.2.2693
    release: websocket
  name: websocket
  namespace: %NAME_SPACE%
spec:
  rules:
  - host: %HOST_NAME%
    http:
      paths:
      - backend:
          serviceName: websocket
          servicePort: 443
        path: /
      - backend:
          serviceName: websocket
          servicePort: 443
        path: /socket.io
status:
  loadBalancer:
    ingress:
    - ip: X.X.X.X
    - ip: Y.Y.Y.Y

service.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: websocket
    meta.helm.sh/release-namespace: %NAME_SPACE%
  creationTimestamp: "2020-04-27T20:58:28Z"
  labels:
    app: websocket
    app.kubernetes.io/managed-by: Helm
    chart: websocket-0.2.2723
    release: websocket
  name: websocket
  namespace: %NAME_SPACE%
  resourceVersion: "2916073"
  selfLink: /api/v1/namespaces/%NAME_SPACE%/services/websocket
  uid: e4c08a00-6824-4e16-a3fa-cace0c9be519
spec:
  clusterIP: 10.0.3.45
  ports:
  - name: websocket
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: websocket
    release: websocket
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "16"
    meta.helm.sh/release-name: websocket
  labels:
    app: websocket
    release: websocket
  name: websocket
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: websocket
      release: websocket
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    spec:
      containers:
        image: %NAME_SPACE%.azurecr.io/websocket:2723
        imagePullPolicy: Always
        name: websocket
        ports:
        - containerPort: 443
          name: websocket
          protocol: TCP
        resources:
          limits:
            cpu: 1500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 64Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

当从nginx-ingress pod获取日志时，我们注意到状态码是101，这意味着它正在工作。

kubectl logs %POD_NAME% -n nginx-ingress --since 1m | grep websocket

输出:

[11/May/2020:12:47:29 +0000] "GET /socket.io/?EIO=3&transport=websocket HTTP/1.1" 101 91

但是，客户端返回500:

Error during WebSocket handshake: Unexpected response code: 500

并且连接自动关闭。

我熟悉以下线程，但没有一个适合我:

https://gist.github.com/jsdevtom/7045c03c021ce46b08cb3f41db0d76da#file-ingress-service-yaml

https://github.com/kubernetes/ingress-nginx/issues/3746

任何帮助都感激不尽。

Answer 1

看起来问题出在Cloudflare方面，我们在不同的域上测试了当前解决方案，并且运行良好。
很少有文章和答案可能会有所帮助:
cloudflare-and-socket-io
Socket.io + nginx + cloudflare problems
Socket.io over https with cloudflare