用于高可用性的 Hadoop 2.0 名称节点、辅助节点和检查点节点

Question

读完Apache Hadoop documentation , 在理解secondary node & check point node的职责上有一个小困惑

我清楚Namenode的角色和职责:

• The NameNode stores modifications to the file system as a log appended to a native file system file, edits. When a NameNode starts up, it reads HDFS state from an image file, fsimage, and then applies edits from the edits log file. It then writes new HDFS state to the fsimage and starts normal operation with an empty edits file. Since NameNode merges fsimage and edits files only during start up, the edits log file could get very large over time on a busy cluster. Another side effect of a larger edits file is that next restart of NameNode takes longer.

但是我在理解辅助名称节点和检查点名称节点的职责时有点困惑。

辅助 NameNode:

• The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode.

检查点节点:

• The Checkpoint node periodically creates checkpoints of the namespace. It downloads fsimage and edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. The Checkpoint node usually runs on a different machine than the NameNode since its memory requirements are on the same order as the NameNode. The Checkpoint node is started by bin/hdfs namenode -checkpoint on the node specified in the configuration file.

Secondary namenode 和 Checkpoint 节点之间的职责似乎并不明确。两者都在进行编辑。那么最终谁来修改呢？

另一方面，我在 jira 中创建了两个错误，以消除理解这些概念时的歧义。

issues.apache.org/jira/browse/HDFS-8913 
issues.apache.org/jira/browse/HDFS-8914 

Answer 1

NameNode(主节点)

NameNode 存储HDFS 的元数据。 HDFS 的状态存储在一个名为 fsimage 的文件中，它是元数据的基础。在运行时修改只是写入一个名为 edits 的日志文件。在 NameNode 的下一次启动时，状态从 fsimage 读取，编辑的更改应用于它，新状态写回 fsimage。在清除此编辑并包含现在准备好新的日志条目之后。

检查点节点

为了解决NameNode的弊端，引入了Checkpoint Node。更改仅写入编辑，不会在运行时合并到 fsimage。如果 NameNode 运行一段时间，编辑会变得很大，下一次启动将花费更长的时间，因为必须对状态应用更多更改以确定元数据的最后状态。

Checkpoint Node周期性地从NameNode中获取fsimage和edits并合并它们。结果状态称为检查点。之后将结果上传到NameNode。

还有一种类似类型的节点称为“辅助节点”，但它没有“上传到 NameNode”功能。所以 NameNode 需要从 Secondary NameNode 获取状态。这也让人感到困惑，因为顾名思义，如果 NameNode 失败，Secondary NameNode 会接受请求，但事实并非如此。

备份节点

备份节点提供与检查点节点相同的功能，但与名称节点同步。它不需要定期获取更改，因为它会收到一系列文件系统编辑。来自 NameNode。它在内存中保存当前状态，只需将其保存到图像文件中即可创建新的检查点。