This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Using Rsyslog sending to Loggly and scrub mongodb log data

I am trying to send information to loggly via rsyslog with data from mongodb 4.4.2. However I cannot get the data in a way that I can manipulate it and scrub out certain information. When I follow the guide on the Loggly site it works for non mongodb information. If I leave %$!msg% as %msg% I get the mongodb data but I am not able to manipulate it. Loggly gets a log back but the raw message is empty and all that is passed is the other fields inside of the $template

config file for reading mongo logs


$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-bundle.crt

# Input for FILE1
input(type="imfile" tag="mongo_lou_qa" ruleset="filelog" file="/var/log/mongodb/mongod.log") #wildcard is allowed at file level only

$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [token@41058  tag=\"tag1\" tag=\"tag2\" ] %$!msg%" 

set $!msg = $msg;

if re_match($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])')    
  set $!ext = re_extract($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])',0,1,"");
  set $!msg= replace($!msg, $!ext, "xxxxxxxxx");
action(type="omfwd" protocol="tcp" target="" port="6514" template="LogglyFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*")

Mongodb sample log

{"t":{"$date":"2021-01-01T00:00:00.000-00:00"},"s":"I",  "c":"ACCESS",   "id":20000,   "ctx":"conn79","msg":"Successful authentication from 000000000","attr":{"mechanism":"ABC","principalName":"__system","authenticationDatabase":"local","client":""}}