我正在使用Python wrapper for Weka这是基于 python-javabridge 。我有一项很长的任务要执行,因此,我正在使用 Celery这样做。问题是我明白
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007fff91a3c16f, pid=11698, tid=3587
JRE version: (8.0_31-b13) (build )
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode bsd-amd64 compressed oops)
Problematic frame:
C [libdispatch.dylib+0x616f] _dispatch_async_f_slow+0x18b
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
If you would like to submit a bug report, please visit:
http://bugreport.java.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.
在线程内启动 JVM 时。为此使用了这两行代码(来自 weka.core.jvm ):
javabridge.start_vm(run_headless=True)
javabridge.attach()
据我所知,这可能是由于 JVM 未附加到 Celery 线程所致。但是,javabridge.attach()
确实在其中运行。
我错过了什么?
编辑:我确定了导致问题的代码。它与 NLTK 有关。分词器。以下代码(根据 Vebjorn's answer )将重现该错误:
# hello.py
from nltk.tokenize import RegexpTokenizer
import javabridge
from celery import Celery
app = Celery('hello', broker='amqp://guest@localhost//', backend='amqp')
started = False
@app.task
def hello():
global started
if not started:
print 'Starting the VM'
javabridge.start_vm(run_headless=True)
started = True
sentence = "This is a sentence with some numbers like 1, 2 or and some weird symbols like @, $ or ! :)"
tokenizer = RegexpTokenizer(r'\w+')
tokenized_sentence = tokenizer.tokenize(sentence.lower())
print "Tokens:", tokenized_sentence
return javabridge.run_script('java.lang.String.format("Hello, %s!", greetee);',
dict(greetee='world'))
在不启动JVM的情况下,代码可以正常运行。当不作为 Celery 任务运行时它也可以工作。我不明白为什么它会崩溃。
编辑2:它实际上可以在干净的Ubuntu环境(Dockerized)中运行,但不能在Mac OS X Yosemite(v10.3)上运行。
编辑3:正如评论中提到的,如果from nltk.tokenize import RegexpTokenizer
在任务包装器内完成,即在hello内,它就可以工作()
函数。
最佳答案
默认情况下,Celery 启动四个独立的工作进程。 (请参阅 celery worker
的 -c
命令行选项。)您需要确保在所有这些中启动 JVM。这个例子对我有用:
# hello.py
import os
import threading
from celery import Celery
import javabridge
app = Celery('hello', broker='amqp://guest@localhost//', backend='amqp')
started = False
@app.task
def hello():
global started
if not started:
print 'Starting the VM'
javabridge.start_vm(run_headless=True)
started = True
return javabridge.run_script('java.lang.String.format("Hello, %s!", greetee);',
dict(greetee='world'))
和
# client.py
from hello import hello
r = hello.delay()
print r.get(timeout=1)
在原始 Ubuntu 14.04 机器上安装:
$ sudo apt-get update -y $ sudo apt-get install -y openjdk-7-jdk python-pip python-numpy python-dev rabbitmq-server $ sudo pip install celery javabridge $ sudo /etc/init.d/rabbitmq-server start
启动工作线程:
$ celery -A hello worker ... -------------- celery@a7cc1bedc40d v3.1.17 (Cipater) ---- **** ----- --- * *** * -- Linux-3.16.7-tinycore64-x86_64-with-Ubuntu-14.04-trusty -- * - **** --- - ** ---------- [config] - ** ---------- .> app: hello:0x7f5464766b50 - ** ---------- .> transport: amqp://guest:**@localhost:5672// - ** ---------- .> results: amqp - *** --- * --- .> concurrency: 4 (prefork) -- ******* ---- --- ***** ----- [queues] -------------- .> celery exchange=celery(direct) key=celery [2015-04-21 10:04:31,262: WARNING/MainProcess] celery@a7cc1bedc40d ready.
在另一个窗口中,运行客户端五次:
$ python client.py Hello, world! $ python client.py Hello, world! $ python client.py Hello, world! $ python client.py Hello, world! $ python client.py Hello, world!
在工作窗口中观察到 JVM 在来自客户端的前四次调用(进入四个不同的进程)时启动,但在第五次调用时未启动:
[2015-04-21 10:05:53,491: WARNING/Worker-1] Starting the VM [2015-04-21 10:05:55,028: WARNING/Worker-2] Starting the VM [2015-04-21 10:05:56,411: WARNING/Worker-3] Starting the VM [2015-04-21 10:05:57,318: WARNING/Worker-4] Starting the VM
关于python - 在 Mac OS X 上使用 NLTK 在 Celery 线程中使用 python-javabridge JVM 时出现 fatal error ,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/29767845/