python - 计算标记化网址中的单词数

Question

对Python非常陌生，希望你们能给我一些帮助。

我有一本关于第一次世界大战的书，想计算一个国家在书中出现的次数。到目前为止我有这个:

>>> from __future__ import division 
>>> import nltk, re, pprint
>>> from urllib import urlopen
>>> url = "http://www.gutenberg.org/files/29270/29270.txt"
>>> raw = urlopen(url).read() 
>>> type(raw)
<type 'str'>
>>> len(raw)
1067008
>>> raw[:75]
'The Project Gutenberg EBook of The Story of the Great War, Volume II (of\r\nV'
>>>

标记化。 将字符串分解为单词和标点符号。

>>> tokens = nltk.word_tokenize(raw)
>>> type(tokens)
<type 'list'>
>>> len(tokens)
189743
>>> tokens[:10] //vind de eerste 10 tokens
['The', 'Project', 'Gutenberg', 'EBook', 'of', 'The', 'Story', 'of', 'the', 'Great']
>>>

更正书的开头和结尾

    >>> raw.find("PART I")
    >>> 2629
    >>> raw.rfind("End of the Project Gutenberg")
    >>> 1047663
    >>> raw = raw[2629:1047663]
    >>> raw.find("PART I")
    >>> 0

不幸的是，我不知道如何将这本书实现到字数统计中。我理想的结果是这样的:

Germany 2000
United Kingdom 1500
USA 1000
Holland 50
Belgium 150

等等

请帮忙!

Answer 1

Python 有一个内置方法来计算字符串中的子字符串。

from urllib import urlopen

url = "http://www.gutenberg.org/files/29270/29270.txt"
raw = urlopen(url).read()
raw = raw[raw.find("PART I"):raw.rfind("End of the Project Gutenberg")]

countries = ['Germany', 'United Kingdom', 'USA', 'Holland', 'Belgium']
for c in countries:
    print c, raw.count(c)

产生

Germany 117
United Kingdom 0
USA 0
Holland 10
Belgium 63

编辑:eumiro 是对的，如果你想计算确切的单词，这是行不通的。如果您想搜索确切的单词，请使用此:

import re
from urllib import urlopen

url = "http://www.gutenberg.org/files/29270/29270.txt"
raw = urlopen(url).read()
raw = raw[raw.find("PART I"):raw.rfind("End of the Project Gutenberg")]

for key, value in {c:len(re.findall(c + '[^A-Za-z]', raw)) for c in countries}.items():
    print key, value

编辑:如果您想要行号:

from urllib import urlopen
import re
from collections import defaultdict

url = "http://www.gutenberg.org/files/29270/29270.txt"
raw = urlopen(url).readlines()

count = defaultdict(list)
countries = ['Germany', 'United Kingdom', 'USA', 'Holland', 'Belgium']
for c in countries:
    for nr, line in enumerate(raw):
        if re.search(c + r'[^A-Za-z]', line):
            count[c].append(nr + 1) #nr + 1 so the first line is 1 instead of 0
    print c, len(count[c]), 'lines:', count[c]