sockets - Kubernetes是否支持蓝绿色部署？

Question

我想问一下在kubernetes中停止 pod 的机制。

在问问题之前，我读过https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods。

大概我们有一个具有正常关机支持的应用程序
(例如，我们在Go https://play.golang.org/p/5tmkPPMiSSt上使用简单的http服务器)。

服务器有两个端点:

/fast ，始终发送200个http状态代码。

/slow ，等待10秒钟并发送200个http状态代码。

具有该配置的部署/服务资源:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app/name: test
  template:
    metadata:
      labels:
        app/name: test
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: service
          image: host.org/images/grace:v0.1
          livenessProbe:
            httpGet:
              path: /health
              port: 10002
            failureThreshold: 1
            initialDelaySeconds: 1
          readinessProbe:
            httpGet:
              path: /health
              port: 10002
            failureThreshold: 1
            initialDelaySeconds: 1
---
apiVersion: v1
kind: Service
metadata:
  name: test
spec:
  type: NodePort
  ports:
    - name: http
      port: 10002
      targetPort: 10002
  selector:
    app/name: test

为了确保Pod正常删除，我进行了两个测试。

第一种选择(端点缓慢)流:

创建副本值等于1的部署。

等待容器就绪。

在/slow端点(curl http://ip-of-some-node:nodePort/slow)上发送请求并删除pod(同时，不同步1秒)。

预期的:

在http服务器完成我的请求之前， pod 一定不能结束。

得到了:

是的，http服务器将在10秒内完成处理，并为我返回响应。
(如果我们将--grace-period = 1选项传递给kubectl，则curl将写入-curl:(52)来自服务器的空回复)

一切都按预期进行。

第二种选择(快速端点)流程:

创建副本值等于10的部署。

等待容器就绪。

使用“连接:关闭” header 启动 wrk 。

随机删除一个或两个Pod(kubectl delete pod/xxx)。

预期的:

没有套接字错误。

得到了:

$ wrk -d 2m --header "Connection: Close" http://ip-of-some-node:nodePort/fast
Running 2m test @ http://ip-of-some-node:nodePort/fast
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   122.35ms  177.30ms   1.98s    91.33%
    Req/Sec    66.98     33.93   160.00     65.83%
  15890 requests in 2.00m, 1.83MB read
  Socket errors: connect 0, read 15, write 0, timeout 0
Requests/sec:    132.34
Transfer/sec:     15.64KB

读取时出现15个套接字错误，即某些Pod在处理所有请求之前已与服务断开连接(也许)。

当应用新的部署版本，按比例缩小和撤消撤消操作时，将出现问题。

问题:

该行为的原因是什么？

如何解决？

Kubernetes版本:v1.16.2

编辑1。

错误数量每次都会更改，但是在两分钟内删除2-5个Pod时，错误数量仍在10-20之间。

P.S.如果我们不删除 Pane ，则不会出现错误。

Answer 1

Does Kubernetes support green-blue deployment?

是的，它确实。您可以在Zero-downtime Deployment in Kubernetes with Jenkins上阅读有关内容，

A blue/green deployment is a change management strategy for releasing software code. Blue/green deployments, which may also be referred to as A/B deployments require two identical hardware environments that are configured exactly the same way. While one environment is active and serving end users, the other environment remains idle.

容器技术提供了运行所需服务的独立环境，这使得创建蓝/绿部署所需的相同环境变得异常容易。松散耦合的服务-副本集以及Kubernetes中基于标签/选择器的服务路由使在不同后端环境之间轻松切换成为可能。

我也建议阅读Kubernetes Infrastructure Blue/Green deployments。

Here是一个存储库，其中包含来自codefresh.io的有关蓝绿色部署的示例。

This repository holds a bash script that allows you to perform blue/green deployments on a Kubernetes cluster. See also the respective blog post

Prerequisites

As a convention the script expects

1. The name of your deployment to be $APP_NAME-$VERSION
2. Your deployment should have a label that shows it version
3. Your service should point to the deployment by using a version selector, pointing to the corresponding label in the deployment

Notice that the new color deployment created by the script will follow the same conventions. This way each subsequent pipeline you run will work in the same manner.

You can see examples of the tags with the sample application:

• service
• deployment

您可能也对Canary deployment感兴趣:

Another deployment strategy is using Canaries (a.k.a. incremental rollouts). With canaries, the new version of the application is gradually deployed to the Kubernetes cluster while getting a very small amount of live traffic (i.e. a subset of live users are connecting to the new version while the rest are still using the previous version). ...

The small subset of live traffic to the new version acts as an early warning for potential problems that might be present in the new code. As our confidence increases, more canaries are created and more users are now connecting to the updated version. In the end, all live traffic goes to canaries, and thus the canary version becomes the new “production version”.

编辑

Questions:

1. What's reason of that behavior?

在应用新的部署时，将删除旧的Pod，并计划新的Pod。
这是由控制计划完成的

For example, when you use the Kubernetes API to create a Deployment, you provide a new desired state for the system. The Kubernetes Control Plane records that object creation, and carries out your instructions by starting the required applications and scheduling them to cluster nodes–thus making the cluster’s actual state match the desired state.

您仅设置了一个 readinessProbe ，它告诉您的service是否应将流量发送到广告连播。这不是一个很好的解决方案，就像您在示例中看到的那样，如果您有10个 pods 并卸下一个或两个 pods ，则会出现间隙并收到套接字错误。

2. How to fix it?

您必须了解它没有损坏，因此不需要修复。

可以通过在应用程序中实现检查以确保其将请求发送到工作地址或利用负载均衡(例如ingress)之类的其他功能来缓解这种情况。

另外，在更新部署时，您可以在删除Pod之前进行检查，以检查它是否确实有任何传入/传出的流量，并将更新仅滚动到未使用的Pod。