kubernetes - Kubernetes 中带有 Ingress 的基于 url 的 websocket 连接负载均衡

Question

我想使用 nginx.ingress.kubernetes.io/upstream-hash-by 进行多个客户端的 Websocket 连接，其中相关客户端(基于 URL)应坚持同一服务器。

当副本数量发生变化时，Ingress-nginx 似乎会重新平衡流量(Pod 下降并会自动替换为新副本，或者数量按运行时规模增加)。

重新平衡的问题是它不会终止现有连接。因此，已经存在的 websocket 连接(对于已经散列的 URL)保留在 pod A 上，而到同一 URL 的新连接突然被分发到新生成的 pod B。

这是我的入口定义:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: websocket-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/upstream-hash-by: "$1"
spec:
  rules:
  - http:
      paths:
      - path: /socket-service/?clients/(.*)/.*
        backend:
          serviceName: websocket-test
          servicePort: 80

是否有配置可以通过关闭“重新平衡”或自动终止现有连接以某种方式控制此行为？

Answer 1

我认为在这种情况下您可以查看 Envoy和 Istio方向。

您可能对 LoadBalancerSettings.ConsistentHashLB 感兴趣标志:

Consistent Hash-based load balancing can be used to provide soft session affinity based on HTTP headers, cookies or other properties. This load balancing policy is applicable only for HTTP connections. The affinity to a particular destination host will be lost when one or more hosts are added/removed from the destination service.

来自Envoy Supported load balancers - ring-hash文档

The ring/modulo hash load balancer implements consistent hashing to upstream hosts. Each host is mapped onto a circle (the “ring”) by hashing its address; each request is then routed to a host by hashing some property of the request, and finding the nearest corresponding host clockwise around the ring. This technique is also commonly known as “Ketama” hashing, and like all hash-based load balancers, it is only effective when protocol routing is used that specifies a value to hash on.

Each host is hashed and placed on the ring some number of times proportional to its weight. For example, if host A has a weight of 1 and host B has a weight of 2, then there might be three entries on the ring: one for host A and two for host B. This doesn’t actually provide the desired 2:1 partitioning of the circle, however, since the computed hashes could be coincidentally very close to one another; so it is necessary to multiply the number of hashes per host—for example inserting 100 entries on the ring for host A and 200 entries for host B—to better approximate the desired distribution. Best practice is to explicitly set minimum_ring_size and maximum_ring_size, and monitor the min_hashes_per_host and max_hashes_per_host gauges to ensure good distribution. With the ring partitioned appropriately, the addition or removal of one host from a set of N hosts will affect only 1/N requests.

When priority based load balancing is in use, the priority level is also chosen by hash, so the endpoint selected will still be consistent when the set of backends is stable.