python - 使用 kubernetes python 客户端重试 watch 时如何避免 "resource too old"？

Question

以下代码将在第二个 watch.stream() 上抛出 ApiException 410 资源太旧:

# python3 -m venv venv
# source venv/bin/activate
# pip install 'kubernetes==23.3.0'

from kubernetes import client,config,watch

config.load_kube_config(context='my-eks-context')

v1 = client.CoreV1Api()
watcher = watch.Watch()


namespace = 'kube-system'
last_resource_version=0

# this watch will timeout in 5s to have a fast way to simulate a watch that need to be retried 
for i in  watcher.stream(v1.list_namespaced_pod, namespace, resource_version=last_resource_version, timeout_seconds=5):
   print(i['object'].metadata.resource_version)
   last_resource_version = i['object'].metadata.resource_version


# we retry the watch starting from the last resource version known
# but this will raise a kubernetes.client.exceptions.ApiException: (410)
# Reason: Expired: too old resource version: 379140622 (380367990)
for i in  watcher.stream(v1.list_namespaced_pod, namespace, resource_version=last_resource_version, timeout_seconds=5):
    print('second loop', i['object'].metadata.resource_version)
    last_resource_version = i['object'].metadata.resource_version

kubernetes documentation指出:

If a client watch is disconnected then that client can start a new watch from the last returned resourceVersion

这就是我在上面的代码中的意图，它总是给出以下异常:

Traceback (most recent call last):
  File "main.py", line 24, in <module>

  File "/Users/rubelagu/git/python-kubernetes-client/venv/lib/python3.8/site-packages/kubernetes/watch/watch.py", line 182, in stream
    raise client.rest.ApiException(
kubernetes.client.exceptions.ApiException: (410)
Reason: Expired: too old resource version: 379164133 (380432814)

我做错了什么？

Answer 1

似乎在对 watch 的初始响应(来自 EKS 集群 1.21)中，事件可以按任何顺序返回。

我随后进行了两次间隔两秒的观察，它们包含相同的 30 个事件，但顺序完全不同。

因此，不能保证您看到的最后一个资源版本实际上是最后一个，也不能保证您可以从该 resourceVersion/resource_version 恢复。此外，您也不允许按 resourceVersion 对这些事件进行排序/整理，因为 kubernetes documentation for Resource Version Semantics明确地说:

Resource versions must be treated as opaque [...] You must not assume resource versions are numeric or collatable.

您必须通过捕获资源太旧的异常并在不指定资源版本的情况下重试来解决此问题，请参阅下面的示例:

from kubernetes import client,config,watch
from kubernetes.client.exceptions import ApiException

config.load_kube_config(context='eks-prod')

v1 = client.CoreV1Api()
# v1 = config.new_client_from_config(context="eks-prod").CoreV1Api()
watcher = watch.Watch()


namespace = 'my-namespace'


def list_pods(resource_version=None):
   print('start watch from resource version: ', str(resource_version))
   try: 
      for i in  watcher.stream(v1.list_namespaced_pod, namespace, resource_version=resource_version, timeout_seconds=2):
         print(i['object'].metadata.resource_version)
         last_resource_version = i['object'].metadata.resource_version
   except ApiException as e: 
      if e.status == 410: # Resource too old
         return list_pods(resource_version=None)
      else:
         raise

   return last_resource_version

last_resource_version = list_pods()

print('last_resource_version', last_resource_version)

list_pods(last_resource_version)