Python 3.5 Pandas 和 MongoDB -json_normalize : raise TypeError ("data argument can' t be an iterator")

Question

我正在尝试在 python3.5 上使用 pandas 进行数据转换。 使用 MongoClient() 和 json_normalize 从 MongoDB 获取数据。 但是，当我执行下面的代码时，它会抛出错误，因为数据参数不能是迭代器。任何指示都会有帮助。

示例数据:

{'bank_code': 'CID005', 'status': 'Init', 'cpgmid': '7847', 'blaze_transId': 'ZI4YQFFOTGG96ZRUQWZS121111632121509-9173782788741', 'currency': 'INR', 'amount': 7800, 'merchant_trans_id': '121111632121509-9173782788741', 'date_time': datetime.datetime(2016, 11, 11, 14, 1, 14, 44000), 'consumer_mobile': 9999999999.0, 'consumer_email': 'test@test.com', '_id': ObjectId('5825cf2a11eae123023730a9')}
{'bank_code': 'CID001', 'status': 'Init', 'cpgmid': '228', 'blaze_transId': '1rjfeklmg2281610111931334hjlm4j8xwl', 'currency': 'INR', 'amount': 651.4, 'merchant_trans_id': '161111569056', 'date_time': datetime.datetime(2016, 11, 11, 14, 1, 14, 333000), 'consumer_mobile': 9999992399.0, 'consumer_email': 'test@air.com', '_id': ObjectId('5825cf2a11eae123023730af')}
{'bank_code': 'CID001', '_id': ObjectId('5825cf2a097752b55d0f17ac'), 'custom_params': {'suppress_trans': 1}, 'currency': 'INR', 'merchant_trans_id': 'BX819215014788728725757', 'date_time': datetime.datetime(2016, 11, 11, 14, 1, 14, 421000), 'consumer_mobile': 0, 'status': 'Init', 'cpgmid': '1656', 'blaze_transId': '1bygejlxl16561610111931423bkgfe1uxx', 'amount': 577, 'consumer_email': 'p.25@gmail.com'}

代码:

start_datetime1 = (datetime.now() - timedelta(days=1)).replace(hour=18, minute=30, second=00, microsecond=0)
start_datetime2 = (datetime.now() - timedelta(days=0)).replace(hour=18, minute=29, second=59, microsecond=0)
client = MongoClient(host_val, int(port_val))
db = client.cit
transactions_collection = db.transactions

cursor = json_normalize(transactions_collection.find({'date_time': {'$lt': start_datetime2, '$gte': start_datetime1}},
                                               {'_id': 1, 'blaze_transId': 1, 'status': 1, 'merchant_trans_id': 1,
                                                'date_time': 1, 'amount': 1, 'status': 1, 'cpgmid': 1, 'currency': 1,
                                                'status_msg': 1, 'bank_code': 1, 'custom_params.suppress_trans': 1,
                                                'consumer_email': 1,'consumer_mobile': 1}))

df_txn = pd.DataFrame(cursor)

错误:

ERROR:root:Exception in fetch
Traceback (most recent call last):
File "/opt/Analytics-services/ETLservices/transformationService/Blazenet_Txns_Fact.py", line 174, in fetchBlazenetTxnsFromDB
'consumer_email': 1,'consumer_mobile': 1}))
File "/usr/local/lib/python3.5/site-packages/pandas/io/json.py", line 717, in json_normalize
    return DataFrame(data)
File "/usr/local/lib/python3.5/site-packages/pandas/core/frame.py", line 283, in __init__
raise TypeError("data argument can't be an iterator")

• 类型错误:数据参数不能是迭代器

Answer 1

您需要将光标转换为列表，然后再将其传递给 json_normalize。

cursor = transactions_collection.find({'date_time': {'$lt': start_datetime2, '$gte': start_datetime1}},
                                               {'_id': 1, 'blaze_transId': 1, 'status': 1, 'merchant_trans_id': 1,
                                                'date_time': 1, 'amount': 1, 'status': 1, 'cpgmid': 1, 'currency': 1,
                                                'status_msg': 1, 'bank_code': 1, 'custom_params.suppress_trans': 1,
                                                'consumer_email': 1,'consumer_mobile': 1})

df_txn = pd.DataFrame(json_normalize(list(cursor)))

您可能还想查看monary如果您想避免将大量数据转换为列表。