我正在使用 tweepy 的 Streamlistener 来收集 Twitter 数据,我使用的代码会生成一个包含一堆元数据的 JSONL 文件。 现在我想将该文件转换为 CSV,我为此找到了一个代码。不幸的是我遇到了错误阅读:
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 7833)
我已经阅读了其他线程,我认为这与 json.loads
无法处理 json 文件中的多个数据部分有关(这当然是我的情况) json 列表文件)。
我如何在代码中规避这个问题?或者我是否必须使用完全不同的方法来转换文件? (我使用的是 python 3.6,我流式传输的推文大部分是阿拉伯语)。
__author__ = 'seandolinar'
import json
import csv
import io
'''
creates a .csv file using a Twitter .json file
the fields have to be set manually
'''
data_json = io.open('stream_____.jsonl', mode='r', encoding='utf-8').read() #reads in the JSON file
data_python = json.loads(data_json)
csv_out = io.open('tweets_out_utf8.csv', mode='w', encoding='utf-8') #opens csv file
fields = u'created_at,text,screen_name,followers,friends,rt,fav' #field names
csv_out.write(fields)
csv_out.write(u'\n')
for line in data_python:
#writes a row and gets the fields from the json object
#screen_name and followers/friends are found on the second level hence two get methods
row = [line.get('created_at'),
'"' + line.get('text').replace('"','""') + '"', #creates double quotes
line.get('user').get('screen_name'),
unicode(line.get('user').get('followers_count')),
unicode(line.get('user').get('friends_count')),
unicode(line.get('retweet_count')),
unicode(line.get('favorite_count'))]
row_joined = u','.join(row)
csv_out.write(row_joined)
csv_out.write(u'\n')
csv_out.close()
最佳答案
如果数据文件由多行组成,每行都是一个 json 对象,您可以使用生成器一次解码一行。
def extract_json(fileobj):
# Using "with" ensures that fileobj is closed when we finish reading it.
with fileobj:
for line in fileobj:
yield json.loads(line)
对代码的唯一更改是未显式读取 data_json
文件,并且 data_python
是调用 extract_json
而不是的结果json.loads
。修改后的代码如下:
import json
import csv
import io
'''
creates a .csv file using a Twitter .json file
the fields have to be set manually
'''
def extract_json(fileobj):
"""
Iterates over an open JSONL file and yields
decoded lines. Closes the file once it has been
read completely.
"""
with fileobj:
for line in fileobj:
yield json.loads(line)
data_json = io.open('stream_____.jsonl', mode='r', encoding='utf-8') # Opens in the JSONL file
data_python = extract_json(data_json)
csv_out = io.open('tweets_out_utf8.csv', mode='w', encoding='utf-8') #opens csv file
fields = u'created_at,text,screen_name,followers,friends,rt,fav' #field names
csv_out.write(fields)
csv_out.write(u'\n')
for line in data_python:
#writes a row and gets the fields from the json object
#screen_name and followers/friends are found on the second level hence two get methods
row = [line.get('created_at'),
'"' + line.get('text').replace('"','""') + '"', #creates double quotes
line.get('user').get('screen_name'),
unicode(line.get('user').get('followers_count')),
unicode(line.get('user').get('friends_count')),
unicode(line.get('retweet_count')),
unicode(line.get('favorite_count'))]
row_joined = u','.join(row)
csv_out.write(row_joined)
csv_out.write(u'\n')
csv_out.close()
关于python - 将 JSONL 文件转换为 CSV - "JSONDecodeError: Extra data",我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/46648387/