Python Twitter 工具和 gzip 错误 "IOError: CRC check failed"

Question

我正在使用Python Twitter tools下载大量用户的最新 200 条推文。我收到仅间歇性发生的 gzip 错误。在看似随机的时间间隔内，循环将崩溃并显示下面的错误堆栈。如果我立即重新启动循环并发送同一用户，则下载它永远不会有问题。我查看了崩溃时推文的标题，似乎与不会引起问题的标题没有任何不同。我已经确认，我毫无问题地返回的大量结果也经过 gzip 压缩并且未压缩。

有人以前见过这个问题和/或可以建议修复/解决方法吗？

这是错误堆栈，它的值(value)是:

File "/Users/martinlbarron/Dropbox/Learning Python/downloadTimeline.py", line 33, in <module>
    result=utility.downloadTimeline(kwargs,t)
  File "/Users/martinlbarron/Dropbox/Learning Python/utility.py", line 73, in downloadTimeline
    response=t.statuses.user_timeline(**kargs)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 173, in __call__
    return self._handle_response(req, uri, arg_data)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 184, in _handle_response
    data = f.read()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 245, in read
    self._read(readsize)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 299, in _read
    self._read_eof()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.py", line 338, in _read_eof
    hex(self.crc)))
IOError: CRC check failed 0xf4196259 != 0x34967f68L
logout

添加我的代码(请温柔一点，我是 Python 新手)

我有一个 Twitter 名称列表。我在下面的代码中循环遍历它们，调用我的 Twitter 下载函数 (downloadTimeline)。

t = Twitter(
    auth=OAuth("XXX", "XXX",
               "XXX", "XXX"))
for i in range(startRange,endRange):
    #Get the id string for user
    row=newlist[i]
    sc=row[3]
    kwargs = dict(count=200, include_rts=False, include_entities=False, trim_user=True, screen_name=sc)
    result=utility.downloadTimeline(kwargs,t)

在downloadTimeline中，我得到twitter响应(response)，然后将其解析到字典中

def downloadTimeline(kargs, t):

    #Get timeline
    mylist = list()
    counter=1000
    try:
        response=t.statuses.user_timeline(**kargs)
        counter=response.rate_limit_remaining
        #parse the file out
        if len(response)>0:
            for tweet in response:
                user=tweet['user']
                dict =  {
                    'id_str': cleanLines(tweet['id_str']), 
                    #ommitting the whole list of all the variables I save
                }
                mylist.append(dict)

    except twitter.TwitterError as e:
            print("Fail: %i" % e.e.code)

    return  (mylist, counter)

最后，虽然这显然不是我的代码，但在 Python Twitter 工具框架中，这是一段似乎令人窒息的代码(特别是在 f = gzip.GzipFile(fileobj=buf) 处)

   def _handle_response(self, req, uri, arg_data):
        try:
            handle = urllib_request.urlopen(req)
            if handle.headers['Content-Type'] in ['image/jpeg', 'image/png']:
                return handle
            elif handle.info().get('Content-Encoding') == 'gzip':
                # Handle gzip decompression
                buf = StringIO(handle.read())
                f = gzip.GzipFile(fileobj=buf)
                data = f.read()
            else:
                data = handle.read()

            if "json" == self.format:
                res = json.loads(data.decode('utf8'))
                return wrap_response(res, handle.headers)
            else:
                return wrap_response(
                    data.decode('utf8'), handle.headers)
        except urllib_error.HTTPError as e:
            if (e.code == 304):
                return []
            else:
                raise TwitterHTTPError(e, uri, self.format, arg_data)

---

事实证明，在 Python Twitter 工具中关闭接受 gzip header 非常容易。但是当我这样做时，我收到以下错误。我想知道响应是否以某种方式被截断:

  File "/Users/martinlbarron/Dropbox/Learning Python/downloadTimeline.py", line 33, in <module>
    result=utility.downloadTimeline(kwargs,t)
  File "/Users/martinlbarron/Dropbox/Learning Python/utility.py", line 73, in downloadTimeline
    response=t.statuses.user_timeline(**kargs)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 175, in __call__
    return self._handle_response(req, uri, arg_data)
  File "/Library/Python/2.7/site-packages/twitter-1.9.0-py2.7.egg/twitter/api.py", line 193, in _handle_response
    res = json.loads(handle.read().decode('utf8'))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 13699 (char 13699)
logout

Answer 1

而不是:

buf = StringIO(handle.read())
f = gzip.GzipFile(fileobj=buf)
data = f.read()

试试这个:

decomp = zlib.decompressobj(16+zlib.MAX_WBITS)
data = decomp.decompress(handle.read())

不要忘记导入 zlib