kubernetes - 我很难在 gke 上设置 kafka 并且想知道设置它的最佳方式？

Question

我试图使用 statefulset 在 gke 的集群中部署 zookeeper 和 Kafka 服务器，但 Kafka 和 zookeeper 之间的通信失败，日志中出现错误消息。我想知道在 kubernetes 中设置 Kafka 的最简单方法是什么。

我尝试了以下配置，发现 Kafka 无法与 zookeeper 通信，但不知道为什么？我知道我可能需要 headless 服务，因为通信由 Kafka 和 zookeeper 自己处理。

动物园管理员

apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  type: LoadBalancer
  selector:
    app: zookeeper
  ports:
  - port: 2181
    targetPort: client
    name: zk-port
  - port: 2888
    targetPort: leader
    name: zk-leader
  - port: 3888
    targetPort: election
    name: zk-election
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
spec:
  replicas: 3
  selector:
    matchLabels:
      app: zookeeper
  serviceName: zookeeper
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - name: zk-pod
          image: zookeeper:latest
          imagePullPolicy: Always
          ports:
            - name: client
              containerPort: 2181
            - name: leader
              containerPort: 2888
            - name: election
              containerPort: 3888
          env:
            - name: ZOO_MY_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ZOO_TICK_TIME
              value: "2000"
            - name: ZOO_INIT_LIMIT
              value: "5"
            - name: ZOO_SYNC_LIMIT
              value: "2"
            - name: ZOO_SERVERS
              value: zookeeper:2888:3888

卡夫卡

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  serviceName: kafka-svc
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:latest
        ports:
          - containerPort: 9092
            name: client
        env:
          - name: KAFKA_ZOOKEEPER_CONNECT
            value: zookeeper:2181
          - name: KAFKA_ADVERTISED_LISTENERS
            value: kafka.default.svc.cluster.local:9092
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-svc
spec:
  type: LoadBalancer
  selector:
    app: kafka
  ports:
  - port: 9092
    targetPort: client
    name: kfk-port
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
spec:
  selector:
    matchLabels:
      app: kafka
  minAvailable: 2

我希望能够向某个主题发送消息并能够读回它们。我一直在使用kafkacat测试连接。

Answer 1

这是 Official Kubernetes Documentation about StateFulsets 中指定的限制之一。 ， 那

• StatefulSets currently require a Headless Service to be responsible for the network identity of the Pods. You are responsible for creating this Service.

所以，正如你提到的，你需要 Headless Service并且您可以轻松地将 headless 服务 yaml 添加到您的配置顶部，类似于下面的两个 StatefulSet:

apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  labels:
    app: zookeeper
spec:
  ports:
  - port: 2181 
    name: someport
  clusterIP: None
  selector:
    app: zookeeper

希望能帮助到你!