google-bigquery - 将 Firebase 导出到 BigQuery 的数据展平为 1 行 = 1 个事件的表(嵌套数据中的嵌套数据)

Question

我以为我可以通过引用一个更简单的数据示例提出一个更简单的问题来获得所需的信息 here ，但我仍然需要一些帮助。

我对在 BigQuery 中查询 json 样式数据非常陌生，并且在 Firebase 为我转储到 BigQuery 的分析(事件)数据方面遇到了问题。 1 行数据的格式如下(修剪掉一些绒毛)。

{
  "user_dim": {
    "user_id": "some_identifier_here",
    "user_properties": [
      {
        "key": "special_key1",
        "val": {
          "val": {
            "str_val": "894",
            "int_val": null
          }
        }
      },
      {
        "key": "special_key2",
        "val": {
          "val": {
            "str_val": "1",
            "int_val": null
          }
        }
      },
      {
        "key": "special_key3",
        "val": {
          "val": {
            "str_val": "23",
            "int_val": null
          }
        }
      }
    ],
    "device_info": {
      "device_category": "mobile",
      "mobile_brand_name": "Samsung",
      "mobile_model_name": "model_phone"
    },
    "dt_a": "1470625311138000",
    "dt_b": "1470620345566000"
  },
  "event_dim": [
    {
      "name": "user_engagement",
      "params": [
        {
          "key": "firebase_event_origin",
          "value": {
            "string_value": "auto",
            "int_value": null,
            "float_value": null,
            "double_value": null
          }
        },
        {
          "key": "engagement_time_msec",
          "value": {
            "string_value": null,
            "int_value": "30006",
            "float_value": null,
            "double_value": null
          }
        }
      ],
      "timestamp_micros": "1470675614434000",
      "previous_timestamp_micros": "1470675551092000"
    },
    {
      "name": "new_game",
      "params": [
        {
          "key": "total_time",
          "value": {
            "string_value": "496048",
            "int_value": null,
            "float_value": null,
            "double_value": null
          }
        },
        {
          "key": "armor",
          "value": {
            "string_value": "2",
            "int_value": null,
            "float_value": null,
            "double_value": null
          }
        },
        {
          "key": "reason",
          "value": {
            "string_value": "power_up",
            "int_value": null,
            "float_value": null,
            "double_value": null
          }
        }
      ],
      "timestamp_micros": "1470675825988001",
      "previous_timestamp_micros": "1470675282500001"
    },
    {
      "name": "user_engagement",
      "params": [
        {
          "key": "firebase_event_origin",
          "value": {
            "string_value": "auto",
            "int_value": null,
            "float_value": null,
            "double_value": null
          }
        },
        {
          "key": "engagement_time_msec",
          "value": {
            "string_value": null,
            "int_value": "318030",
            "float_value": null,
            "double_value": null
          }
        }
      ],
      "timestamp_micros": "1470675972778002",
      "previous_timestamp_micros": "1470675614434002"
    },
    {
      "name": "won_game",
      "params": [
        {
          "key": "total_time",
          "value": {
            "string_value": "497857",
            "int_value": null,
            "float_value": null,
            "double_value": null
          }
        },
        {
          "key": "level",
          "value": {
            "string_value": null,
            "int_value": "207",
            "float_value": null,
            "double_value": null
          }
        },
        {
          "key": "sword",
          "value": {
            "string_value": "iron",
            "int_value": null,
            "float_value": null,
            "double_value": null
          }
        }
      ],
      "timestamp_micros": "1470677171374007",
      "previous_timestamp_micros": "1470671343784007"
    }
  ]
}

根据对我原来问题的回答，我已经能够很好地处理对象的第一部分 user_dim .然而，每当我尝试类似的方法时 event_dim字段(取消嵌套)查询失败并显示消息“错误:标量子查询产生多个元素”。我怀疑这是因为 event_dim本身就是一个数组，并且包含其中也有数组的结构。

如果它有帮助，这里是给我错误的基本查询，尽管应该注意的是，我完全无法在 BQ 中处理此类数据，并且可能会完全偏离方向:

SELECT
  (SELECT name FROM UNNEST(event_dim) WHERE name = 'user_engagement') AS event_name
FROM
  my_table;

我要的最终结果 是一个查询，它可以将包含许多此类对象的表转换为一个表，该表为每个对象中的每个事件输出 1 行 event_dim大批。即对于上面的示例对象，我希望它输出 4 行，其中第一组列是相同的，并且只是来自 user_dim 的元数据.然后我想要可以根据我知道的每个可能事件存在的内容明确定义的列，例如 event_name, firebase_event_origin, engagement_time_msec, total_time, armor, reason, level, sword然后用该事件参数的值填充，如果不存在则用 NULL 填充。

Answer 1

基于 Mikhail 的回答，但基于实际的 Firebase 数据集:

SELECT 
  user_dim.app_info.app_instance_id,
  timestamp_micros,
  (SELECT value.int_value FROM UNNEST(dim.params) WHERE key = "level") AS level,
  (SELECT value.int_value FROM UNNEST(dim.params) WHERE key = "coins") AS coins,
  (SELECT value.int_value FROM UNNEST(dim.params) WHERE key = "powerups") AS powerups
FROM `dataset.table`, UNNEST(event_dim) AS dim
WHERE timestamp_micros=1464718937589000 

(将其保存在此处以供将来引用，并且更易于复制粘贴)