python - Bottle 导致 python 程序崩溃？使用线程、队列和 fork 的简单实现

Question

我已尝试尽可能简化此操作，但仍然收到错误。我有一个简单的http服务器(bottle)，它在收到post请求后执行一个应该快速 fork 自身的函数。父进程只是返回一个作业 ID 并关闭，而子进程继续处理相同的数据(这是一个 URL 列表)。我已经删除了所有输入和输出函数并对数据进行了硬编码，但我的程序仍然崩溃。有趣的是，当我将程序更改为直接在命令行运行而不是启动 http 服务器并等待 Bottle 执行它时，一切正常!

#!/usr/bin/python
#This is a comment
import sys, time, bottle, os
from threading import Thread
from Queue import Queue
from bottle import route, run, request, abort

num_fetch_threads = 2
url_queue = Queue()

def fetchURLContent(i, q):
  while True:
    #print '%s: Looking for URLs in queue' % i
    url = q.get()
    #print 'URL found: %s' % url[0]
    q.task_done()
    time.sleep(1)

@route('/', method='POST') # or @route('/login', method='POST')
def main():

  urls = ['http://www.yahoo.com', 'http://www.google.com']

  newpid = os.fork()
  if newpid == 0:

    for i in range(num_fetch_threads):
      worker = Thread(target=fetchURLContent, args=(i, url_queue))
      worker.setDaemon(True)
      worker.start()
    print 'Queuing: ', url
    for url in urls:
      url_queue.put(url)
    time.sleep(2)
    print 'main thread waiting...'
    url_queue.join()
    print 'Done'

  else:
    print "Your job id is 5"
    return
def webServer():
  run(host='33.33.33.10', port=8080)

if __name__ == "__main__":
  print 'Listening on 8080...'
  webServer()

我收到的错误消息如下:

Listening on 8080...
Bottle v0.11.3 server starting up (using WSGIRefServer())...
Listening on http://33.33.33.10:8080/
Hit Ctrl-C to quit.

33.33.33.1 - - [19/Oct/2012 21:21:24] "POST / HTTP/1.1" 200 0
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.finish_content()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 246, in finish_content
    self.send_headers()
  9 url_queue = Queue()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 268, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 189, in send_preamble
    self._write('HTTP/%s %s\r\n' % (self.http_version,self.status))
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 389, in _write
    self.stdout.write(data)
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('33.33.33.1', 57615)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
    self.wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------

有什么想法吗？

Answer 1

您的 main() 函数立即终止而不返回任何内容。 Bottle 向套接字写入一个空的 HTTP 响应，然后 Web 服务器关闭连接。

您的 fork 进程在 main() 中停留的时间稍长，但随后也终止并导致 Bottle 向已关闭的套接字写入另一个空响应。这就是您得到的错误(管道损坏)。

此时 fork 无法工作。 HTTP 不允许每个请求有多个响应。您可以阻塞直到所有工作完成然后发送响应，或者立即发送响应并在不同的线程中完成工作。