以下 word2ngrams
函数从一个单词中提取字符 3grams:
>>> x = 'foobar'
>>> n = 3
>>> [x[i:i+n] for i in range(len(x)-n+1)]
['foo', 'oob', 'oba', 'bar']
这篇文章展示了单个单词 Quick implementation of character n-grams using python 的字符 ngrams 提取。
但是如果我有句子并且我想提取字符 ngrams 怎么办,有没有比迭代调用 word2ngram()
更快的方法?
实现相同 word2ngram
和 sent2ngram
输出的正则表达式版本是什么?会更快吗?
我试过:
import string, random, time
from itertools import chain
def word2ngrams(text, n=3):
""" Convert word into character ngrams. """
return [text[i:i+n] for i in range(len(text)-n+1)]
def sent2ngrams(text, n=3):
return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))
def sent2ngrams_simple(text, n=3):
text = text.lower()
return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]
# Generate 10000 random strings of length 100.
sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]
start = time.time()
x = [sent2ngrams(i) for i in sents]
print time.time() - start
start = time.time()
y = [sent2ngrams_simple(i) for i in sents]
print time.time() - start
print x==y
[输出]:
0.0205280780792
0.0271739959717
True
已编辑
正则表达式方法看起来很优雅,但它的执行速度比迭代调用 word2ngram()
慢:
import string, random, time, re
from itertools import chain
def word2ngrams(text, n=3):
""" Convert word into character ngrams. """
return [text[i:i+n] for i in range(len(text)-n+1)]
def sent2ngrams(text, n=3):
return list(chain(*[word2ngrams(i,n) for i in text.lower().split()]))
def sent2ngrams_simple(text, n=3):
text = text.lower()
return [text[i:i+n] for i in range(len(text)-n+1) if not " " in text[i:i+n]]
def sent2ngrams_regex(text, n=3):
rgx = '(?=('+'\S'*n+'))'
return re.findall(rgx,text)
# Generate 10000 random strings of length 100.
sents = [" ".join([''.join(random.choice(string.ascii_uppercase) for j in range(10)) for i in range(100)]) for k in range(100)]
start = time.time()
x = [sent2ngrams(i) for i in sents]
print time.time() - start
start = time.time()
y = [sent2ngrams_simple(i) for i in sents]
print time.time() - start
start = time.time()
z = [sent2ngrams_regex(i) for i in sents]
print time.time() - start
print x==y==z
[输出]:
0.0211708545685
0.0284190177917
0.0303599834442
True
最佳答案
为什么不只是 (?=(...))
编辑 同样的东西,但不是空格 (?=(\S\S\S))
edit2 您也可以随意使用。前任。仅使用字母数字 (?=([^\W_]{3}))
使用前瞻来捕获 3 个字符。然后引擎每次将位置向上颠簸1次
匹配。然后捕获下一个 3。
foobar
的结果是
富
oob
奥巴马
酒吧
# Compressed regex
# (?=(...))
# Expanded regex
(?= # Start Lookahead assertion
( # Capture group 1 start
. # dot - metachar, matches any character except newline
. # dot - metachar
. # dot - metachar
) # Capture group 1 end
) # End Lookahead assertion
关于python - 如何从句子中提取字符ngram? - Python,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/22428020/