python - 如何列出 Google Big Query 中所有数据集中所有表的大小

Question

我正在尝试弄清楚如何列出 Google Big Query 中所有项目中所有表的所有大小。也许它将是多个表的 SQL 联合。虽然，我在这里查看了很多表格，所以我想要某种自动化解决方案。我可以使用 R 代码来执行此任务。或者我冷甚至使用 Python 来做它。如果这里有人有解决方案列出一些指标，主要是每个对象(表)的大小，以及其他相关指标，请在这里分享。非常感谢!

Answer 1

这个 Python 示例列出了所有项目中的所有表及其大小(以字节为单位)。您可以以此为例来构建适合您的用例的脚本:

from google.cloud import bigquery
from google.cloud.bigquery import Dataset
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials

# credentials to list project
credentials = GoogleCredentials.get_application_default()
service = discovery.build('cloudresourcemanager', 'v1', credentials=credentials)

# list project
request = service.projects().list()
response = request.execute()

# Main loop for project
for project in response.get('projects', []):
    client = bigquery.Client(project['projectId']) # Start the client in the right project

    # list dataset
    datasets = list(client.list_datasets())
    if datasets: # If there is some BQ dataset
        print('Datasets in project {}:'.format(project['name']))
        # Second loop to list the tables in the dataset
        for dataset in datasets: 
            print(' - {}'.format(dataset.dataset_id))
            get_size = client.query("select table_id, size_bytes as size from "+dataset.dataset_id+".__TABLES__") # This query retrieve all the tables in the dataset and the size in bytes. It can be modified to get more fields.
            tables = get_size.result() # Get the result
            # Third loop to list the tables and print the result
            for table in tables:
                print('\t{} size: {}'.format(table.table_id,table.size))

引用:

列出项目:
https://cloud.google.com/resource-manager/reference/rest/v1/projects/list#embedded-explorer

要列出数据集:
https://cloud.google.com/bigquery/docs/datasets#bigquery-list-datasets-python