我正在尝试确定各种工作状态。 Bigquery 提供了我所知道的三种状态:DONE, PENDING, and RUNNING 。但是,我试图根据以下内容获取状态:
- 完成
- 待处理
- 成功
- 错误
- 已取消
- 正在运行
我如何以一种不太“昂贵”的方式做到这一点,因为我以一种“长轮询”的方式迭代大约 100 个结果,大约每十秒一次。目前我正在做类似的事情:
jobs = [job for job in self.bq_client.list_jobs(project=PROJECT_ID]
if state is not None:
jobs = [job for job in jobs if job.state == state]
如果状态是 DONE、RUNNING 或 PENDING 之一,则上述方法有效。但我如何覆盖其他州呢?
最佳答案
状态跟踪作业进度,如果您需要成功/失败信息,您需要查看 response 的 errorResult
。对于成功的作业,这将为无
,对于取消的作业,您将得到{u'reason': u'stopped', u'message': u'作业执行已取消:用户请求取消'}
。我用来测试这个的代码:
from google.cloud import bigquery
client = bigquery.Client()
project = "[PROJECT-ID]"
states = ["RUNNING", "PENDING", "SUCCESSFUL", "CANCELLED", "FAILED"]
def returnState(job):
if job.state == "DONE":
if job.error_result is None:
return "SUCCESSFUL"
elif job.error_result['reason'] == u'stopped':
return "CANCELLED"
else:
return "FAILED"
else:
return job.state
jobs = [job for job in client.list_jobs(project=project, max_results=10)]
for state in states:
matching_jobs = [job for job in jobs if returnState(job) == state]
for job in matching_jobs:
print "Job ID: {0}, State: {1}, Error Result: {2}".format(job.job_id, state, job.error_result)
这将打印出类似的内容:
$ python bq-status.py
Job ID: bquijob_..., State: SUCCESSFUL, Error Result: None
Job ID: bquijob_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: scheduled_query_..., State: SUCCESSFUL, Error Result: None
Job ID: bquijob_..., State: SUCCESSFUL, Error Result: None
Job ID: bquijob_..., State: CANCELLED, Error Result: {u'reason': u'stopped', u'message': u'Job execution was cancelled: User requested cancellation'}
Job ID: bquijob_..., State: FAILED, Error Result: {u'reason': u'invalidQuery', u'message': u'Syntax error: Illegal input character "\\\\" at [2:18]', u'location': u'query'}
请记住,加载作业可能会成功,但允许一些 maxBadRecords
,以便 errorResult
不会为空,等等。
关于python - 如何在特定的 "DONE"状态下获取工作,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/54702811/