Tuesday, March 10, 2020
Inserting custom logs types to Logstash (ELK) using Grok
You can parse your custom logs and collect them to ELK.
For parsing logs you use Grok (you have to know some RegEx). You can find Grok Constructor on this link: https://grokconstructor.appspot.com/do/construction
Sample of LOG:
[2020-03-09 10:38:23,483] [ INFO] (xxxxn.java: [si.AA.AA.ejb.YYYYYYYY:ZZZZZZZ:747]) [20DEDFEC-AF17-1F4D-A5B2-B95EF33623EB] AA
[2020-03-09 10:38:23,484] [ INFO] (xxxxn.java: [si.AA.AA.ejb.YYYYYYYY:ZZZZZZZ:213]) [20DEDFEC-AF17-1F4D-A5B2-B95EF33623EB] AA
[2020-03-09 10:38:48,667] [ERROR] (xxxxn.java: [si.AA.AA.ejb.YYYYYYYY:ZZZZZZZ:218]) ZZZZZZZZZ
si.AA.utils.exceptions.Exception:ZZZZZZ.
at si.AA.utils.AA.AA(AA.java:179) ~[YY-3.3.jar:?]
at si.AA.AA.ja.utils.Utils.AA(AA.java:28) ~[XX-3.7.0-SNAPSHOT.jar:?]
[2020-03-09 10:38:48,683] [ERROR] (xxxxn.java: [si.AA.xxxxn.rs.xxxxn:handleException:54]) AA.
[2020-03-09 10:40:10,074] [ INFO] (xxxxn.java: [si.v.xxxxn.ejb.xxxxn:xxxxn:213]) [CC36B034-5646-A7E2-1BDB-C87A2AE81F20] ZZZZZZZZZZZZZZZZ
Grok parse string:
^\[(?<timestamp>(.*))\]\s+(\[(?<loglevel>(.*))\]\s+)(\((?<eventprog>(.*))\))\s(?<message>(?m:.*))?
input {
file {
type => "applog"
path => "/log destination/*/*.log"
codec => multiline {
pattern => "^(?!\[)"
negate => "false"
what => "previous"
}
}
}
For parsing logs you use Grok (you have to know some RegEx). You can find Grok Constructor on this link: https://grokconstructor.appspot.com/do/construction
Sample of LOG:
[2020-03-09 10:38:23,483] [ INFO] (xxxxn.java: [si.AA.AA.ejb.YYYYYYYY:ZZZZZZZ:747]) [20DEDFEC-AF17-1F4D-A5B2-B95EF33623EB] AA
[2020-03-09 10:38:23,484] [ INFO] (xxxxn.java: [si.AA.AA.ejb.YYYYYYYY:ZZZZZZZ:213]) [20DEDFEC-AF17-1F4D-A5B2-B95EF33623EB] AA
[2020-03-09 10:38:48,667] [ERROR] (xxxxn.java: [si.AA.AA.ejb.YYYYYYYY:ZZZZZZZ:218]) ZZZZZZZZZ
si.AA.utils.exceptions.Exception:ZZZZZZ.
at si.AA.utils.AA.AA(AA.java:179) ~[YY-3.3.jar:?]
at si.AA.AA.ja.utils.Utils.AA(AA.java:28) ~[XX-3.7.0-SNAPSHOT.jar:?]
[2020-03-09 10:38:48,683] [ERROR] (xxxxn.java: [si.AA.xxxxn.rs.xxxxn:handleException:54]) AA.
[2020-03-09 10:40:10,074] [ INFO] (xxxxn.java: [si.v.xxxxn.ejb.xxxxn:xxxxn:213]) [CC36B034-5646-A7E2-1BDB-C87A2AE81F20] ZZZZZZZZZZZZZZZZ
Grok parse string:
^\[(?<timestamp>(.*))\]\s+(\[(?<loglevel>(.*))\]\s+)(\((?<eventprog>(.*))\))\s(?<message>(?m:.*))?
- \[(?<timestamp>(.*))\]\s - Parses timestamp in [] with space at the end and content is parsed into element timestamp
- (\[(?<loglevel>(.*))\]\s+) - Parses level in [] with space at the end and content is parsed into element loglevel
- ...
Because the messsage can be in multiline you have to set multilinefilter = ^(?!\[) . That means the parser will put message where char [ is present.
In configuratin file for Logstash (was.conf ) the source looks like:
input {
file {
type => "applog"
path => "/log destination/*/*.log"
codec => multiline {
pattern => "^(?!\[)"
negate => "false"
what => "previous"
}
}
}
filter {
if [type] == "applog" {
grok {
match => { "message" => "^(\[(?<loglevel>(.*))\]\s+)(?<timestamp>%{DATE_EU} %{TIME})\s+(\((?<eventprog>(.*))\))\s(?<message>(?m:.*))?" }
overwrite => [ "message" ]
}
if "_grokparsefailure" in [tags] {
grok {
match => { "message" => "^\[(?<timestamp>(.*))\]\s+(\[(?<loglevel>(.*))\]\s+)(\((?<eventprog>(.*))\))\s(?<message>(?m:.*))?" }
overwrite => [ "message" ]
remove_tag => ["_grokparsefailure"]
}
}
}
}
In sample above you have two diferent filters (parsing rules) and if one fails another is used.
