javascript - 使用 python 或 javascript 从文本中提取困难的英语单词以构建词汇

Question

我想从在线英语文本中获取困难的单词，例如使用 python 或 javascript 来构建词汇表的 gutenberg。我不习惯使用简单的单词，但使用独特的词汇，如“帝王”、“伪经”等。

如何确保在分割文本时我只得到独特的词汇而不是简单的单词。

Answer 1

我将“非常见单词”定义为未出现在前 10000 个最常见英语单词中的单词。

这 10 k 个最常见的单词是任意边界，但如 the github repo 中所述。 :

According to analysis of the Oxford English Corpus, the 7,000 most common English lemmas account for approximately 90% of usage, so a 10,000 word training corpus is more than sufficient for practical training applications.

import requests

english_most_common_10k = 'https://raw.githubusercontent.com/first20hours/google-10000-english/master/google-10000-english-usa-no-swears.txt'

# Get the file of 10 k most common words from TXT file in a github repo
response = requests.get(english_most_common_10k)
data = response.text

set_of_common_words = {x for x in data.split('\n')}

# Once we have the set of common words, we can just check.
# The check is in average case O(1) operation,
# but you can use for example some sort of search three with O(log(n)) complexity
while True:
    word = input()
    if word in set_of_common_words:
        print(f'The word "{word}" is common')
    else:
        print(f'The word "{word}" is difficult')