我有以下代码,它在线程内部运行(完整代码在这里 - https://github.com/eWizardII/homobabel/blob/master/lovebird.py)
for null in range(0,1):
while True:
try:
with open('C:/Twitter/tweets/user_0_' + str(self.id) + '.json', mode='w') as f:
f.write('[')
threadLock.acquire()
for i, seed in enumerate(Cursor(api.user_timeline,screen_name=self.ip).items(200)):
if i>0:
f.write(", ")
f.write("%s" % (json.dumps(dict(sc=seed.author.statuses_count))))
j = j + 1
threadLock.release()
f.write("]")
except tweepy.TweepError, e:
with open('C:/Twitter/tweets/user_0_' + str(self.id) + '.json', mode='a') as f:
f.write("]")
print "ERROR on " + str(self.ip) + " Reason: ", e
with open('C:/Twitter/errors_0.txt', mode='a') as a_file:
new_ii = "ERROR on " + str(self.ip) + " Reason: " + str(e) + "\n"
a_file.write(new_ii)
break
现在没有线程锁我会生成以下错误:
Exception in thread Thread-117: Traceback (most recent call last): File "C:\Python27\lib\threading.py", line 530, in __bootstrap_inner
self.run() File "C:/Twitter/homobabel/lovebird.py", line 62, in run
for i, seed in enumerate(Cursor(api.user_timeline,screen_name=self.ip).items(200)): File "build\bdist.win-amd64\egg\tweepy\cursor.py", line 110, in next
self.current_page = self.page_iterator.next() File "build\bdist.win-amd64\egg\tweepy\cursor.py", line 85, in next
items = self.method(page=self.current_page,
*self.args, **self.kargs) File "build\bdist.win-amd64\egg\tweepy\binder.py", line 196, in _call
return method.execute() File "build\bdist.win-amd64\egg\tweepy\binder.py", line 182, in execute
result = self.api.parser.parse(self, resp.read()) File "build\bdist.win-amd64\egg\tweepy\parsers.py", line 75, in parse
result = model.parse_list(method.api, json) File "build\bdist.win-amd64\egg\tweepy\models.py", line 38, in parse_list
results.append(cls.parse(api, obj)) File "build\bdist.win-amd64\egg\tweepy\models.py", line 49, in parse
user = User.parse(api, v) File "build\bdist.win-amd64\egg\tweepy\models.py", line 86, in parse
setattr(user, k, parse_datetime(v)) File "build\bdist.win-amd64\egg\tweepy\utils.py", line 17, in parse_datetime
date = datetime(*(time.strptime(string, '%a %b %d %H:%M:%S +0000 %Y')[0:6])) File "C:\Python27\lib\_strptime.py", line 454, in _strptime_time
return _strptime(data_string, format)[0] File "C:\Python27\lib\_strptime.py", line 300, in _strptime
_TimeRE_cache = TimeRE() File "C:\Python27\lib\_strptime.py", line 188, in __init__
self.locale_time = LocaleTime() File "C:\Python27\lib\_strptime.py", line 77, in __init__
raise ValueError("locale changed during initialization") ValueError: locale changed during initialization
问题在于线程锁定,每个线程基本上都是串行运行的,并且每个循环都需要很长时间才能运行,因为不再有线程有任何优势。因此,如果没有办法摆脱线程锁,有没有办法让它更快地运行 try 语句内的 for 循环?
最佳答案
根据之前的Answer在 StackOverflow 上,time.strptime
不是线程安全的。不幸的是,该问题中引用的错误与您遇到的错误不同。
他们的解决方案是在初始化任何线程之前调用 time.strptime
,然后在各个线程中对 time.strptime
的后续调用将起作用。
在查看了 _strptime
和 locale
标准库模块后,我认为相同的解决方案可能适用于您的情况。由于我无法在本地测试您的代码,因此我不确定它是否有效,但我想我会为您提供一个潜在的解决方案。
让我知道这是否有效。
编辑:
我做了更多研究,发现 Python 标准库在 locale.h
C 头文件中调用了 setlocale
。根据setlocale documentation ,这不是线程安全的,并且像我之前提到的那样,应该在初始化线程之前调用 setlocale
。
不幸的是,每次调用 time.strptime
时都会调用 setlocale
。所以,我建议如下:
- 测试之前提出的解决方案,尝试在初始化线程和移除锁之前调用
time.strptime
。 - 如果#1 不起作用,您可能需要运行自己的
time.strptime
函数,该函数是线程安全的,如 locale 的 Python 文档中所述。模块。
关于python - 修复了带有线程锁的 strptime 异常,但会减慢程序速度,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/4625149/