重启kubernetes后mongodb容器的数据变成 "read-only"，用glusterfs存储？

Question

我的 mongo 在 kubernetes 上作为 docker 容器运行，glusterfs 提供持久卷。我重启kuberntes后(机器断电重启)，所有的mongo pod都回不来了，他们的日志:

chown: changing ownership of `/data/db/user_management.ns': Read-only file system
chown: changing ownership of `/data/db/storage.bson': Read-only file system
chown: changing ownership of `/data/db/local.ns': Read-only file system
chown: changing ownership of `/data/db/mongod.lock': Read-only file system

这里的/data/db/是挂载的gluster卷，我可以确定它是rw模式!

# kubectl get pod mongoxxx -o yaml
apiVersion: v1
kind: Pod
spec:
  containers:
  - image: mongo:3.0.5
    imagePullPolicy: IfNotPresent
    name: mongo
    ports:
    - containerPort: 27017
      protocol: TCP
    volumeMounts:
    - mountPath: /data/db
      name: mongo-storage
  volumes:
  - name: mongo-storage
    persistentVolumeClaim:
      claimName: auth-mongo-data

# kubectl describe pod mongoxxx
...
    Volume Mounts:
      /data/db from mongo-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wdrfp (ro)
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  mongo-storage:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  auth-mongo-data
    ReadOnly:   false
...

# kubect get pv xxx
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  name: auth-mongo-data
  resourceVersion: "215201"
  selfLink: /api/v1/persistentvolumes/auth-mongo-data
  uid: fb74a4b9-e0a3-11e6-b0d1-5254003b48ea
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 4Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: auth-mongo-data
    namespace: default
  glusterfs:
    endpoints: glusterfs-cluster
    path: infra-auth-mongo
  persistentVolumeReclaimPolicy: Retain
status:
  phase: Bound

当我在 kubernetes 节点上 ls 时:

# ls -ls /var/lib/kubelet/pods/fc6c9ef3-e0a3-11e6-b0d1-5254003b48ea/volumes/kubernetes.io~glusterfs/auth-mongo-data/
total 163849
    4 drwxr-xr-x. 2 mongo input     4096 1月  22 21:18 journal
65536 -rw-------. 1 mongo input 67108864 1月  22 21:16 local.0
16384 -rw-------. 1 mongo root  16777216 1月  23 17:15 local.ns
    1 -rwxr-xr-x. 1 mongo root         2 1月  23 17:15 mongod.lock
    1 -rw-r--r--. 1 mongo root        69 1月  23 17:15 storage.bson
    4 drwxr-xr-x. 2 mongo input     4096 1月  22 21:18 _tmp
65536 -rw-------. 1 mongo input 67108864 1月  22 21:18 user_management.0
16384 -rw-------. 1 mongo root  16777216 1月  23 17:15 user_management.ns

虽然卷安装为 rw，但我无法chown。

我的主机是CentOs 7.3: Linux c4v160 3.10.0-514.el7.x86_64 #1 SMP 11 月 22 日星期二 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux。

Answer 1

我猜是因为我提供的glusterfs卷不干净。 glusterfs 卷 infra-auth-mongo 可能包含脏目录。一种解决方案是删除此卷并创建另一个。

另一种解决方案是破解 mongodb dockerfile，强制它在启动 mongodb 进程之前更改 /data/db 的所有权。像这样:https://github.com/harryge00/mongo/commit/143bfc317e431692010f09b5c0d1f28395d2055b