elasticsearch - 使用Elasticsearch的Logstash:我正在使用Logstash将数据提取到ES索引中。但是现在我希望logstash运行24/7

标签 elasticsearch service logstash

    #file:db.conf

input { 
    jdbc { 
           jdbc_driver_library => ""
           jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver" 
           jdbc_connection_string => "jdbc:oracle:thin:@abcd.klm.uvw:1521/qtp1"
           jdbc_user =>"user_wew"
           jdbc_password => "password_wew"

           
           
                  statement => "select col1, col2, col3, col4, col5, col6, countid,max(version) as mv from master_object_table where version >:sql_last_value group by countid"

            schedule => "* * * * *"
           last_run_metadata_path => "C:/ES1/ELK_stack_7.4.2/logstash-7.4.2/logstash-7.4.2/Master_refresh_a.txt"

           use_column_value => true

            tracking_column => "version"

         } 
       }
       
        filter {
      mutate {
        convert => {
          "countid" => "string"
          }
          }
          }
       
output {
  elasticsearch { 
           hosts => "localhost:9200" 
           index =>"refresh_index_a"
           document_id =>"%{countid}"
           #document_type="_doc"
           
           }
 file {
         path => "C:\\ES1\\ELK_stack_7.4.2\\logstash-7.4.2\\logstash-7.4.2\\bin\\logstashESRecordsIngestionDetails_refresh_a.txt"
         
         codec => rubydebug
       }
           
  stdout {   codec => rubydebug }
}
上面是我的logstash配置文件设置。我要运行24/7的logstash,并且还要在运行该logstash的机器关闭的情况下运行,那么如何管理该Logstash,因为此Logstash正在将实时数据提取到ES索引中。请提出建议。 如果一台服务器关闭另一节点上的logstash,是否有任何方法可以继续工作。

最佳答案

按照documentation

Logstash is horizontally scalable and can form groups of nodes running the same pipeline. Logstash’s adaptive buffering capabilities will facilitate smooth streaming even through variable throughput loads. If the Logstash layer becomes an ingestion bottleneck, simply add more nodes to scale out. Here are a few general recommendations:

  1. Beats should load balance across a group of Logstash nodes.
  2. A minimum of two Logstash nodes are recommended for high availability.
  3. It’s common to deploy just one Beats input per Logstash node, but multiple Beats inputs can also be deployed per Logstash node to expose independent endpoints for different data sources.

关于elasticsearch - 使用Elasticsearch的Logstash:我正在使用Logstash将数据提取到ES索引中。但是现在我希望logstash运行24/7,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/62715212/

相关文章:

api - 超媒体API链接遍历与实用性

xml - 将XML文件中的Logstash读取到Elasticsearch

elasticsearch - Logstash - 使用当前日期作为时间戳日期

elasticsearch - Kibana用另一个术语选择所有真实值

android - onBind() 永远不会在服务中被调用

Elasticsearch 最佳实践 : use directly from the frontend or from the backend

windows - 如何获取多线程Windows Service应用程序中正在运行的线程数

elasticsearch - 在现有图表elasticsearch kibana中添加数据的平均值

elasticsearch - 上载时Elasticsearch批量插入异常

java - ElasticSearch 5.3 SearchHit hit.field