python - 如何在 NLTK 中对二元语言模型进行单词级别的 Kneser-Ney 平滑？

Question

从 nltk 包中，我发现我们可以仅使用三元组来实现 Kneser-Ney 平滑，但当我尝试在 bigrams 上使用相同的函数时，它会抛出错误。有没有办法可以对二元组实现平滑？

## Working code for trigrams 
tokens = "What a piece of work is man! how noble in reason! how infinite in faculty! in \
    form and moving how express and admirable! in action how like an angel! in apprehension how like a god! \
    the beauty of the world, the paragon of animals!".split()
gut_ngrams = nltk.ngrams(tokens,3)
freq_dist = nltk.FreqDist(gut_ngrams)
kneser_ney = nltk.KneserNeyProbDist(freq_dist)

Answer 1

首先，让我们看一下代码和实现。

当我们使用二元组时:

import nltk

tokens = "What a piece of work is man! how noble in reason! how infinite in faculty! in \
    form and moving how express and admirable! in action how like an angel! in apprehension how like a god! \
    the beauty of the world, the paragon of animals!".split()
gut_ngrams = nltk.ngrams(tokens,2)
freq_dist = nltk.FreqDist(gut_ngrams)
kneser_ney = nltk.KneserNeyProbDist(freq_dist)

代码抛出错误:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-1ce73b806bb8> in <module>
      4 gut_ngrams = nltk.ngrams(tokens,2)
      5 freq_dist = nltk.FreqDist(gut_ngrams)
----> 6 kneser_ney = nltk.KneserNeyProbDist(freq_dist)

~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/nltk/probability.py in __init__(self, freqdist, bins, discount)
   1737         self._trigrams_contain = defaultdict(float)
   1738         self._wordtypes_before = defaultdict(float)
-> 1739         for w0, w1, w2 in freqdist:
   1740             self._bigrams[(w0, w1)] += freqdist[(w0, w1, w2)]
   1741             self._wordtypes_after[(w0, w1)] += 1

ValueError: not enough values to unpack (expected 3, got 2)

如果我们看一下实现，https://github.com/nltk/nltk/blob/develop/nltk/probability.py#L1700

class KneserNeyProbDist(ProbDistI):
    def __init__(self, freqdist, bins=None, discount=0.75):
        if not bins:
            self._bins = freqdist.B()
        else:
            self._bins = bins
        self._D = discount

        # cache for probability calculation
        self._cache = {}

        # internal bigram and trigram frequency distributions
        self._bigrams = defaultdict(int)
        self._trigrams = freqdist

        # helper dictionaries used to calculate probabilities
        self._wordtypes_after = defaultdict(float)
        self._trigrams_contain = defaultdict(float)
        self._wordtypes_before = defaultdict(float)
        for w0, w1, w2 in freqdist:
            self._bigrams[(w0, w1)] += freqdist[(w0, w1, w2)]
            self._wordtypes_after[(w0, w1)] += 1
            self._trigrams_contain[w1] += 1
            self._wordtypes_before[(w1, w2)] += 1

我们看到在初始化中计算当前单词之前和之后的 n-gram 时做出了一些假设:

 for w0, w1, w2 in freqdist:
        self._bigrams[(w0, w1)] += freqdist[(w0, w1, w2)]
        self._wordtypes_after[(w0, w1)] += 1
        self._trigrams_contain[w1] += 1
        self._wordtypes_before[(w1, w2)] += 1

在这种情况下，只有三元组适用于 KneserNeyProbDist 对象的 KN 平滑!!

让我们尝试一下四元组:

tokens = "What a piece of work is man! how noble in reason! how infinite in faculty! in \
    form and moving how express and admirable! in action how like an angel! in apprehension how like a god! \
    the beauty of the world, the paragon of animals!".split()
gut_ngrams = nltk.ngrams(tokens,4)
freq_dist = nltk.FreqDist(gut_ngrams)
kneser_ney = nltk.KneserNeyProbDist(freq_dist)

[输出]:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-60a48ed2ffce> in <module>
      4 gut_ngrams = nltk.ngrams(tokens,4)
      5 freq_dist = nltk.FreqDist(gut_ngrams)
----> 6 kneser_ney = nltk.KneserNeyProbDist(freq_dist)

~/.pyenv/versions/3.8.0/lib/python3.8/site-packages/nltk/probability.py in __init__(self, freqdist, bins, discount)
   1737         self._trigrams_contain = defaultdict(float)
   1738         self._wordtypes_before = defaultdict(float)
-> 1739         for w0, w1, w2 in freqdist:
   1740             self._bigrams[(w0, w1)] += freqdist[(w0, w1, w2)]
   1741             self._wordtypes_after[(w0, w1)] += 1

ValueError: too many values to unpack (expected 3)

瞧!它也不起作用!!!

问:那么这是否意味着 KN 平滑无法在 NLTK 中用于语言建模？

答:这并不完全正确。 NLTK nltk.lm 中有一个合适的语言模型模块，这里有一个使用它的教程示例 https://www.kaggle.com/alvations/n-gram-language-model-with-nltk/notebook#Training-an-N-gram-Model

但这只是显示了普通 MLE 模型的用法。我想要一个具有 Kneser-Ney 平滑功能的 LM

然后你只需要定义正确的语言模型对象就正确=)

TL;DR

from nltk.lm import KneserNeyInterpolated
from nltk.lm.preprocessing import padded_everygram_pipeline

tokens = "What a piece of work is man! how noble in reason! how infinite in faculty! in \
    form and moving how express and admirable! in action how like an angel! in apprehension how like a god! \
    the beauty of the world, the paragon of animals!".split()

n = 4 # Order of ngram
train_data, padded_sents = padded_everygram_pipeline(n, tokens)

model = KneserNeyInterpolated(n) 
model.fit(train_data, padded_sents)