docker - kube-dns 无法解析 'kubernetes.default.svc.cluster.local'

Question

使用kargo部署kubernetes集群后，发现kubedns pod运行不正常:

$ kcsys get pods -o wide

NAME          READY STATUS           RESTARTS AGE  IP           NODE
dnsmasq-alv8k 1/1   Running          2        1d   10.233.86.2  kubemaster
dnsmasq-c9y52 1/1   Running          2        1d   10.233.82.2  kubeminion1
dnsmasq-sjouh 1/1   Running          2        1d   10.233.76.6  kubeminion2
kubedns-hxaj7 2/3   CrashLoopBackOff 339      22h  10.233.76.3  kubeminion2

PS : kcsys 是的别名 kubectl --namespace=kube-system

每个容器(kubedns、dnsmasq)的日志似乎都正常，除了 healthz 容器如下:

2017/03/01 07:24:32 Healthz probe error: Result of last exec: nslookup: can't resolve 'kubernetes.default.svc.cluster.local' error exit status 1

更新

kubedns rc 描述

apiVersion: v1
kind: ReplicationController
metadata:
  creationTimestamp: 2017-02-28T08:31:57Z
  generation: 1
  labels:
    k8s-app: kubedns
    kubernetes.io/cluster-service: "true"
    version: v19
  name: kubedns
  namespace: kube-system
  resourceVersion: "130982"
  selfLink: /api/v1/namespaces/kube-system/replicationcontrollers/kubedns
  uid: 5dc9f9f2-fd90-11e6-850d-005056a020b4
spec:
  replicas: 1
  selector:
    k8s-app: kubedns
    version: v19
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kubedns
        kubernetes.io/cluster-service: "true"
        version: v19
    spec:
      containers:
      - args:
        - --domain=cluster.local.
        - --dns-port=10053
        - --v=2
        image: gcr.io/google_containers/kubedns-amd64:1.9
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: kubedns
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP          
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 100m
            memory: 170Mi
          requests:
            cpu: 70m
            memory: 70Mi
        terminationMessagePath: /dev/termination-log
      - args:
        - --log-facility=-
        - --cache-size=1000
        - --no-resolv
        - --server=127.0.0.1#10053
        image: gcr.io/google_containers/kube-dnsmasq-amd64:1.3
        imagePullPolicy: IfNotPresent
        name: dnsmasq
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 170Mi
          requests:
            cpu: 70m
            memory: 70Mi
        terminationMessagePath: /dev/termination-log
      - args:
        - -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
          && nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null
        - -port=8080
        - -quiet
        image: gcr.io/google_containers/exechealthz-amd64:1.1
        imagePullPolicy: IfNotPresent
        name: healthz
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: 10m
            memory: 50Mi
          requests:
            cpu: 10m
            memory: 50Mi
        terminationMessagePath: /dev/termination-log
      dnsPolicy: Default
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  fullyLabeledReplicas: 1
  observedGeneration: 1
  replicas: 1`

kubedns 服务描述:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2017-02-28T08:31:58Z
  labels:
    k8s-app: kubedns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: kubedns
  name: kubedns
  namespace: kube-system
  resourceVersion: "10736"
  selfLink: /api/v1/namespaces/kube-system/services/kubedns
  uid: 5ed4dd78-fd90-11e6-850d-005056a020b4
spec:
  clusterIP: 10.233.0.3
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  selector:
    k8s-app: kubedns
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

我在 kubedns 容器中发现了一些错误:

1 reflector.go:199] pkg/dns/dns.go:145: Failed to list *api.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: i/o timeout
1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: i/o timeout

更新 2

1. 在使用 3 个 pod 创建主机名服务时由 kube-proxy 创建的 iptables 规则:

1. controller-manager pod 的标志:

2. pod 状态

Answer 1

根据您发布的错误，kubedns 无法与 API 服务器通信:

dial tcp 10.233.0.1:443: i/o timeout

这可能意味着三件事:

---

您的容器网络结构配置不正确

• 在您正在使用的网络解决方案的日志中查找错误
• 确保每个 Docker 守护进程都使用自己的 IP 范围
• 验证容器网络不与宿主网络重叠

---

您的 kube-proxy 有问题，当使用 kubernetes 内部服务 (10.233.0.1) 时，网络流量没有转发到 API 服务器)

• 检查您节点 (kubeminion{1,2}) 上的 kube-proxy 日志，并根据您可能发现的任何错误更新您的问题

---

如果您还看到身份验证错误:

kube-controller-manager 不生成有效的服务帐户 token

• 检查 --service-account-private-key-file 和 --root-ca-file 标志 kube-controller -manager 设置为有效的 key /证书并重新启动服务

• 删除 kube-system 命名空间中的 default-token-xxxx secret 并重新创建 kube-dns Deployment