kubernetes - 运算符(operator)获取部署配置的最佳实践

Question

我在做operator-sdk，在controller中，我们经常需要创建一个Deployment对象，而Deployment资源有很多配置项，比如环境变量或者端口定义等等。我想知道获取这些值的最佳方法是什么，我不想对它们进行硬编码，例如 variable_a 或 variable_b。

或许，您可以将它们作为规范放在 CRD 中，然后将它们传递给 Operator Controller；或者你可以将它们放在 configmap 中，然后将 configmap 名称传递给 Operator Controller，Operator Controller 可以访问 configmap 来获取它们；或者您可以放入模板文件，然后在运算符(operator) Controller 中， Controller 必须读取该模板文件。

处理这种情况的最佳方法或最佳实践是什么？感谢您分享您的想法或观点。

    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
            Labels:    ls,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: ls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: ls,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image: "....",
                        Name: m.Name,
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: port_a,
                            Name:          "tcpport",
                        }},
                        Env: []corev1.EnvVar{
                            {
                                Name:  "aaaa",
                                Value: variable_a,
                            },
                            {
                                Name:  "bbbb",
                                Value: variable_b,
                            },

Answer 1

使用环境变量

您的应用可以方便地将您的数据作为环境变量获取。

来自 ConfigMap 的环境变量

对于非敏感数据，您可以将变量存储在 ConfigMap 中然后使用 ConfigMap 定义容器环境变量数据。

Example from Kubernetes docs :

创建 ConfigMap第一的。文件 configmaps.yaml :

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: default
data:
  special.how: very
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: env-config
  namespace: default
data:
  log_level: INFO

创建 ConfigMap:

kubectl create -f ./configmaps.yaml

然后在Pod中定义环境变量规范，pod-multiple-configmap-env-variable.yaml :

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      env:
        - name: SPECIAL_LEVEL_KEY
          valueFrom:
            configMapKeyRef:
              name: special-config
              key: special.how
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: env-config
              key: log_level
  restartPolicy: Never

创建 Pod :

kubectl create -f ./pod-multiple-configmap-env-variable.yaml

现在在您的 Controller 中您可以读取这些环境变量 SPECIAL_LEVEL_KEY (这会给你 special.how 来自 special-config ConfigMap 的值)和 LOG_LEVEL (这将为您提供 log_level 来自 env-config ConfigMap 的值):

例如:

specialLevelKey := os.Getenv("SPECIAL_LEVEL_KEY")
logLevel := os.Getenv("LOG_LEVEL")

fmt.Println("SPECIAL_LEVEL_KEY:", specialLevelKey)
fmt.Println("LOG_LEVEL:", logLevel)

来自 Secret 的环境变量

如果您的数据是敏感的，您可以将其存储在 Secret 中然后使用 Secret作为环境变量。

To create a Secret manually :

您首先需要使用 base64 对您的字符串进行编码.

# encode username
$ echo -n 'admin' | base64
YWRtaW4=

# encode password
$ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm

然后创建一个Secret有了以上数据:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

创建一个 Secret与 kubectl apply :

$ kubectl apply -f ./secret.yaml

请注意，还有其他创建 secret 的方法，请选择最适合您的方法:

• Creating a Secret using kubectl
• Creating a Secret from a generator
• Creating a Secret from files
• Creating a Secret from string literals

现在您可以 use this created Secret for environment variables .

To use a secret in an environment variable in a Pod:

1. Create a secret or use an existing one. Multiple Pods can reference the same secret.
2. Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret's name and key in env[].valueFrom.secretKeyRef.
3. Modify your image and/or command line so that the program looks for values in the specified environment variables.

这是一个Pod来自 Kubernetes 文档的示例展示了如何使用 Secret对于环境变量:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: password
  restartPolicy: Never

最后，如文档中所述:

Inside a container that consumes a secret in an environment variables, the secret keys appear as normal environment variables containing the base64 decoded values of the secret data.

现在在您的 Controller 中您可以读取这些环境变量 SECRET_USERNAME (这会给你 username 来自 mysecret Secret 的值)和 SECRET_PASSWORD (这将为您提供 password 来自 mysecret Secret 的值):

例如:

username := os.Getenv("SECRET_USERNAME")
password := os.Getenv("SECRET_PASSWORD")

使用卷

您也可以同时挂载 ConfigMap和 Secret作为一个卷给你的 pod 。

Populate a Volume with data stored in a ConfigMap :

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "ls /etc/config/" ]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        # Provide the name of the ConfigMap containing the files you want
        # to add to the container
        name: special-config
  restartPolicy: Never

Using Secrets as files from a Pod :

To consume a Secret in a volume in a Pod:

1. Create a secret or use an existing one. Multiple Pods can reference the same secret.
2. Modify your Pod definition to add a volume under .spec.volumes[]. Name the volume anything, and have a .spec.volumes[].secret.secretName field equal to the name of the Secret object.
3. Add a .spec.containers[].volumeMounts[] to each container that needs the secret. Specify .spec.containers[].volumeMounts[].readOnly = true and .spec.containers[].volumeMounts[].mountPath to an unused directory name where you would like the secrets to appear. Modify your image or command line so that the program looks for files in that directory. Each key in the secret data map becomes the filename under mountPath.

Pod 的示例安装一个 Secret在一卷中:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret