elasticsearch - 在[…]上超出磁盘高水位[90％]，分片将被移离此节点

标签 elasticsearch kubernetes search-guard

我在kubernetes集群(v1.18-kind)上运行Elasicsearch(floragunncom/sg-elasticsearch:7.0.1-oss-35.0.0)。
我已经为每个Elasticsearch数据节点(即pod)分配了PVC的200Mi作为存储。但是，当elasticseach群集(3个主节点，3个数据，2个接收)启动时，它显示以下警告消息:

{"type": "server", "timestamp": "2020-07-20T06:27:14,354+0000", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "topology-es", "node.name": "master-topology-es-1", "cluster.uuid": "U0Yf6tiJSHC57R_QHa2qiw", "node.id": "tDet2llLQaydoxfYT4TcpQ", "message": "high disk watermark [90%] exceeded on [C-TUE5_cR1ihqrhThpWnsg][data-topology-es-1][/usr/share/elasticsearch/data/nodes/0] free: 5.5gb[7%], shards will be relocated away from this node" }


  • 这个5.5gb[7%]来自哪里？

    • 我也检查了碎片的状态:
    $ curl -XGET "https://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason" -u "admin:XXXXX" --insecure
searchguard 0 p STARTED    
searchguard 0 r UNASSIGNED REPLICA_ADDED
searchguard 0 r UNASSIGNED REPLICA_ADDED
  • 如果200Mi不够用，那么如何在data-node-0中成功分配主分片，但是副本分别不在data-node-1和data-node-2中呢？

    • 更新:
    $ curl -XGET "https://localhost:9200/_nodes/stats/fs?pretty" -u "admin:XXXX" --insecure
{
  "_nodes" : {
    "total" : 8,
    "successful" : 8,
    "failed" : 0
  },
  "cluster_name" : "topology-es",
  "nodes" : {
    "jYkduRCmQ2GEHMn2DeBrxg" : {
      "timestamp" : 1595231421976,
      "name" : "data-topology-es-0",
      "transport_address" : "10.244.0.59:9300",
      "host" : "10.244.0.59",
      "ip" : "10.244.0.59:9300",
      "roles" : [
        "data"
      ],
      "fs" : {
        "timestamp" : 1595231421989,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77306,
              "read_operations" : 7745,
              "write_operations" : 69561,
              "read_kilobytes" : 230532,
              "write_kilobytes" : 500284
            }
          ],
          "total" : {
            "operations" : 77306,
            "read_operations" : 7745,
            "write_operations" : 69561,
            "read_kilobytes" : 230532,
            "write_kilobytes" : 500284
          }
        }
      }
    },
    "N7ql_AneQoa6w8yvgBhYZg" : {
      "timestamp" : 1595231421977,
      "name" : "client-topology-es-0",
      "transport_address" : "10.244.0.49:9300",
      "host" : "10.244.0.49",
      "ip" : "10.244.0.49:9300",
      "roles" : [
        "ingest"
      ],
      "fs" : {
        "timestamp" : 1595231421981,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 84052,
              "read_operations" : 7853,
              "write_operations" : 76199,
              "read_kilobytes" : 231804,
              "write_kilobytes" : 565340
            }
          ],
          "total" : {
            "operations" : 84052,
            "read_operations" : 7853,
            "write_operations" : 76199,
            "read_kilobytes" : 231804,
            "write_kilobytes" : 565340
          }
        }
      }
    },
    "gse01W_XR7qKvZPHPq7n7g" : {
      "timestamp" : 1595231421984,
      "name" : "master-topology-es-2",
      "transport_address" : "10.244.0.57:9300",
      "host" : "10.244.0.57",
      "ip" : "10.244.0.57:9300",
      "roles" : [
        "master"
      ],
      "fs" : {
        "timestamp" : 1595231421993,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77627,
              "read_operations" : 7747,
              "write_operations" : 69880,
              "read_kilobytes" : 230560,
              "write_kilobytes" : 504608
            }
          ],
          "total" : {
            "operations" : 77627,
            "read_operations" : 7747,
            "write_operations" : 69880,
            "read_kilobytes" : 230560,
            "write_kilobytes" : 504608
          }
        }
      }
    },
    "bBNEuz2pSkWeOUmDIowEDQ" : {
      "timestamp" : 1595231421991,
      "name" : "client-topology-es-1",
      "transport_address" : "10.244.0.51:9300",
      "host" : "10.244.0.51",
      "ip" : "10.244.0.51:9300",
      "roles" : [
        "ingest"
      ],
      "fs" : {
        "timestamp" : 1595231421995,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 82173,
              "read_operations" : 7788,
              "write_operations" : 74385,
              "read_kilobytes" : 231144,
              "write_kilobytes" : 546544
            }
          ],
          "total" : {
            "operations" : 82173,
            "read_operations" : 7788,
            "write_operations" : 74385,
            "read_kilobytes" : 231144,
            "write_kilobytes" : 546544
          }
        }
      }
    },
    "-hwsv3rqQeSVdfOsKqzrwA" : {
      "timestamp" : 1595231422013,
      "name" : "master-topology-es-0",
      "transport_address" : "10.244.0.53:9300",
      "host" : "10.244.0.53",
      "ip" : "10.244.0.53:9300",
      "roles" : [
        "master"
      ],
      "fs" : {
        "timestamp" : 1595231422080,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 79542,
              "read_operations" : 7762,
              "write_operations" : 71780,
              "read_kilobytes" : 230792,
              "write_kilobytes" : 521656
            }
          ],
          "total" : {
            "operations" : 79542,
            "read_operations" : 7762,
            "write_operations" : 71780,
            "read_kilobytes" : 230792,
            "write_kilobytes" : 521656
          }
        }
      }
    },
    "C-TUE5_cR1ihqrhThpWnsg" : {
      "timestamp" : 1595231421984,
      "name" : "data-topology-es-1",
      "transport_address" : "10.244.0.61:9300",
      "host" : "10.244.0.61",
      "ip" : "10.244.0.61:9300",
      "roles" : [
        "data"
      ],
      "fs" : {
        "timestamp" : 1595231422003,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77169,
              "read_operations" : 7736,
              "write_operations" : 69433,
              "read_kilobytes" : 230396,
              "write_kilobytes" : 498544
            }
          ],
          "total" : {
            "operations" : 77169,
            "read_operations" : 7736,
            "write_operations" : 69433,
            "read_kilobytes" : 230396,
            "write_kilobytes" : 498544
          }
        }
      }
    },
    "zlL7F5kRTLWWogzffZXomA" : {
      "timestamp" : 1595231422000,
      "name" : "data-topology-es-2",
      "transport_address" : "10.244.0.63:9300",
      "host" : "10.244.0.63",
      "ip" : "10.244.0.63:9300",
      "roles" : [
        "data"
      ],
      "fs" : {
        "timestamp" : 1595231422001,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77048,
              "read_operations" : 7736,
              "write_operations" : 69312,
              "read_kilobytes" : 230396,
              "write_kilobytes" : 496864
            }
          ],
          "total" : {
            "operations" : 77048,
            "read_operations" : 7736,
            "write_operations" : 69312,
            "read_kilobytes" : 230396,
            "write_kilobytes" : 496864
          }
        }
      }
    },
    "tDet2llLQaydoxfYT4TcpQ" : {
      "timestamp" : 1595231421993,
      "name" : "master-topology-es-1",
      "transport_address" : "10.244.0.55:9300",
      "host" : "10.244.0.55",
      "ip" : "10.244.0.55:9300",
      "roles" : [
        "master"
      ],
      "fs" : {
        "timestamp" : 1595231422020,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77766,
              "read_operations" : 7748,
              "write_operations" : 70018,
              "read_kilobytes" : 230576,
              "write_kilobytes" : 505892
            }
          ],
          "total" : {
            "operations" : 77766,
            "read_operations" : 7748,
            "write_operations" : 70018,
            "read_kilobytes" : 230576,
            "write_kilobytes" : 505892
          }
        }
      }
    }
  }
}

    最佳答案

    这是从C-TUE5_cR1ihqrhThpWnsg输出中获取的主节点_nodes/stats的文件系统统计信息:

        "total" : {
      "total_in_bytes" : 84054204416,
      "free_in_bytes" : 10231291904,
      "available_in_bytes" : 5917450240
    },
    如您所见，5917450240/84054204416 =〜7％，因此该节点上的存储空间已满93％，因此为什么收到该警告。

    关于elasticsearch - 在[…]上超出磁盘高水位[90％]，分片将被移离此节点，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/62990765/

    上一篇：powershell - 获取解决方案中包含特定 NuGet 包的所有项目

    下一篇：excel - 将哈希表导出为 CSV，并将键作为列标题

    相关文章：

    elasticsearch - sgadmin.sh 失败 elasticsearch && searchguard

    docker - 如何从Pod访问本地计算机

    elasticsearch - Elasticsearch:无法使用script_score排序结果

    Elasticsearch 查询要求数组中的所有值都存在

    php - 流明中的 Elasticsearch 在结果数据中给出空项目

    Ubuntu Kubernetes 上的 Nginx Ingress IP 待定

    kubernetes - 向现有 Helm chart 添加额外注释

    elasticsearch - 安装搜索 guard 失败

    regex - 在dls(文档级安全性)查询中使用正则表达式进行Elasticsearch过滤器访问

    elasticsearch - 由于所谓的磁盘可用空间不足，ElasticSearch错误地暂停了索引编制

    ©2024 IT工具网  联系我们