java - 斯坦福 NER 重新训练的内存要求

Question

我正在使用自己的训练数据重新训练斯坦福 NER 模型以提取组织。但是，无论我使用 4GB RAM 机器还是 8GB RAM 机器，我都会遇到相同的 Java 堆空间错误。

有人能告诉我们可以在不出现这些内存问题的情况下重新训练模型的机器的一般配置吗？

我使用了以下命令:

 java -mx4g -cp stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -prop newdata_retrain.prop

我正在处理训练数据(多个文件 - 每个文件大约有 15000 行，格式如下) - 每行一个单词及其类别

谢欧 是 O 工作O 在O 微软组织

我们还能做些什么来使这些模型可靠运行吗？我确实尝试减少训练数据中的类别数量。但这会影响提取的准确性。例如，某些位置或其他实体被分类为组织名称。我们可以在不影响准确性的情况下减少具体的类别数量吗？

我使用的一个数据是 Alan Ritter twitter nlp 数据:https://github.com/aritter/twitter_nlp/tree/master/data/annotated/ner.txt

属性文件如下所示:

#location of the training file
trainFile = ner.txt
#location where you would like to save (serialize to) your
#classifier; adding .gz at the end automatically gzips the file,
#making it faster and smaller
serializeTo = ner-model-twitter.ser.gz

#structure of your training file; this tells the classifier
#that the word is in column 0 and the correct answer is in
#column 1
map = word=0,answer=1

#these are the features we'd like to train with
#some are discussed below, the rest can be
#understood by looking at NERFeatureFactory
useClassFeature=true
useWord=true
useNGrams=true
#no ngrams will be included that do not contain either the
#beginning or end of the word
noMidNGrams=true
useDisjunctive=true
maxNGramLeng=6
usePrev=true
useNext=true
useSequences=true
usePrevSequences=true
maxLeft=1
#the next 4 deal with word shape features
useTypeSeqs=true
useTypeSeqs2=true
useTypeySequences=true
wordShape=chris2useLC
saveFeatureIndexToDisk = true

我收到的错误:堆栈跟踪是这样的:

CRFClassifier invoked on Mon Dec 01 02:55:22 UTC 2014 with arguments:
   -prop twitter_retrain.prop
usePrevSequences=true
useClassFeature=true
useTypeSeqs2=true
useSequences=true
wordShape=chris2useLC
saveFeatureIndexToDisk=true
useTypeySequences=true
useDisjunctive=true
noMidNGrams=true
serializeTo=ner-model-twitter.ser.gz
maxNGramLeng=6
useNGrams=true
usePrev=true
useNext=true
maxLeft=1
trainFile=ner.txt
map=word=0,answer=1
useWord=true
useTypeSeqs=true
[1000][2000]numFeatures = 215032
setting nodeFeatureIndicesMap, size=149877
setting edgeFeatureIndicesMap, size=65155
Time to convert docs to feature indices: 4.4 seconds
numClasses: 21 [0=O,1=B-facility,2=I-facility,3=B-other,4=I-other,5=B-company,6=B-person,7=B-tvshow,8=B-product,9=B-sportsteam,10=I-person,11=B-geo-loc,12=B-movie,13=I-movie,14=I-tvshow,15=I-company,16=B-musicartist,17=I-musicartist,18=I-geo-loc,19=I-product,20=I-sportsteam]
numDocuments: 2394
numDatums: 46469
numFeatures: 215032
Time to convert docs to data/labels: 2.5 seconds
Writing feature index to temporary file.
numWeights: 31880772
QNMinimizer called on double function of 31880772 variables, using M = 25.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:923)
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:885)
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:879)
        at edu.stanford.nlp.optimization.QNMinimizer.minimize(QNMinimizer.java:91)
        at edu.stanford.nlp.ie.crf.CRFClassifier.trainWeights(CRFClassifier.java:1911)
        at edu.stanford.nlp.ie.crf.CRFClassifier.train(CRFClassifier.java:1718)
        at edu.stanford.nlp.ie.AbstractSequenceClassifier.train(AbstractSequenceClassifier.java:759)
        at edu.stanford.nlp.ie.AbstractSequenceClassifier.train(AbstractSequenceClassifier.java:747)
        at edu.stanford.nlp.ie.crf.CRFClassifier.main(CRFClassifier.java:2937)

Answer 1

尝试减少类数量的一种方法是不使用 B-I 表示法。例如，将 B-设施和 I-设施合并到设施中。当然，另一种方法是使用更大内存的机器。