systemctl restart logstash.service && tail -f /var/log/logstash/logstash-plain.log [2020-07-01T09:02:20,251][INFO ][logstash.runner ] Logstash shut down. [2020-07-01T09:02:36,912][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 25.252-b09 on 1.8.0_252-8u252-b09-1~deb9u1-b09 +indy +jit [linux-x86_64]"} [2020-07-01T09:02:37,832][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]} [2020-07-01T09:02:38,272][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-07-01T09:02:40,270][WARN ][logstash.runner ] SIGTERM received. Shutting down. [2020-07-01T09:02:43,149][INFO ][logstash.runner ] Logstash shut down. [2020-07-01T09:02:59,563][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 25.252-b09 on 1.8.0_252-8u252-b09-1~deb9u1-b09 +indy +jit [linux-x86_64]"} [2020-07-01T09:03:00,510][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]} [2020-07-01T09:03:00,900][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-07-01T09:03:05,836][INFO ][logstash.runner ] Logstash shut down.
Sample Logstash configuration for creating a simple# Beats -> Logstash -> Elasticsearch pipeline.input { beats { port => 5044 }}output { elasticsearch { hosts => ["http://localhost:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" #user => "elastic" #password => "changeme" }}
systemctl restart logstash.service && tail -f /var/log/logstash/logstash-plain.log[2020-07-02T02:55:50,177][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}[2020-07-02T02:55:50,201][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}[2020-07-02T02:55:50,287][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}[2020-07-02T02:55:50,361][INFO ][org.logstash.beats.Server][main][d66a9f29bcec26d5c4e24a65ddb036a49584858a82822da2dcbf41d1c6f957c8] Starting server on port: 5044[2020-07-02T02:55:50,643][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}[2020-07-02T02:56:01,036][WARN ][logstash.runner ] SIGTERM received. Shutting down.[2020-07-02T02:56:06,222][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>30, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.9-java/lib/logstash/inputs/beats.rb:197:in `run'"}, {"thread_id"=>26, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>27, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>28, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>29, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}]}}[2020-07-02T02:56:06,231][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.[2020-07-02T02:56:07,487][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>"main"}[2020-07-02T02:56:08,336][INFO ][logstash.runner ] Logstash shut down.
input { udp { port => 5044 type => mikrot }}output { if [type] == "mikrot" { elasticsearch { hosts => ["http://localhost:9200"] index => "logstash-mikrot-%{+YYYY.MM.dd}" } # stdout { codec => rubydebug } }}
systemctl restart logstash.service && tail -f /var/log/logstash/logstash-plain.log[2020-07-02T04:53:19,611][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/mikrot.conf"], :thread=>"#<Thread:0x5b4228de run>"}[2020-07-02T04:53:19,637][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}[2020-07-02T04:53:20,440][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}[2020-07-02T04:53:20,579][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}[2020-07-02T04:53:20,595][INFO ][logstash.inputs.udp ][main][2823f63909bd4903d2eaad80db3dc85bcffdd851ea64819cae1c6979d10a466f] Starting UDP listener {:address=>"0.0.0.0:5044"}[2020-07-02T04:53:20,756][INFO ][logstash.inputs.udp ][main][2823f63909bd4903d2eaad80db3dc85bcffdd851ea64819cae1c6979d10a466f] UDP listener started {:address=>"0.0.0.0:5044", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}[2020-07-02T04:53:20,958][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}[2020-07-02T04:55:51,032][WARN ][logstash.runner ] SIGTERM received. Shutting down.[2020-07-02T04:55:51,621][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>"main"}[2020-07-02T04:55:52,523][INFO ][logstash.runner ] Logstash shut down.[2020-07-02T04:56:09,562][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 25.252-b09 on 1.8.0_252-8u252-b09-1~deb9u1-b09 +indy +jit [linux-x86_64]"}[2020-07-02T04:56:11,692][INFO ][org.reflections.Reflections] Reflections took 43 ms to scan 1 urls, producing 21 keys and 41 values[2020-07-02T04:56:12,524][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}[2020-07-02T04:56:12,718][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}[2020-07-02T04:56:12,771][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}[2020-07-02T04:56:12,777][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}[2020-07-02T04:56:12,849][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}[2020-07-02T04:56:12,958][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template[2020-07-02T04:56:12,973][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/mikrot.conf"], :thread=>"#<Thread:0x76f6d179 run>"}[2020-07-02T04:56:13,083][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}[2020-07-02T04:56:13,787][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}[2020-07-02T04:56:13,907][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}[2020-07-02T04:56:13,927][INFO ][logstash.inputs.udp ][main][2823f63909bd4903d2eaad80db3dc85bcffdd851ea64819cae1c6979d10a466f] Starting UDP listener {:address=>"0.0.0.0:5044"}[2020-07-02T04:56:14,045][INFO ][logstash.inputs.udp ][main][2823f63909bd4903d2eaad80db3dc85bcffdd851ea64819cae1c6979d10a466f] UDP listener started {:address=>"0.0.0.0:5044", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}[2020-07-02T04:56:14,257][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
