Parsing Audiocodes Logs with Fluentd Parser

Question

Hi I am trying to parse logs from a docker container log file. The log file has looks like this:

{"log":"[S=26853604] [BID=fb0e19:3]  (N 24959591) HandleARecordQuery - Host:sip3.pstnhub.microsoft.com is not in cache, setting timer [Time:18-09@08:30:55.728]\n","stream":"stdout","time":"2020-09-18T07:30:57.683119437Z"} {"log":"[S=26853605] [BID=fb0e19:3]  (N 24959592) DNSResolver::HandleTimerExpiredOnWaitForARecord: host:sip3.pstnhub.microsoft.com [Time:18-09@08:30:55.788]\n","stream":"stdout","time":"2020-09-18T07:30:57.683123737Z"} {"log":"[S=26853606] [BID=fb0e19:3]  (N 24959593) DNSResolver::HandleTimerExpOnWaitARecord - Host:sip3.pstnhub.microsoft.com resolved in external table [Time:18-09@08:30:55.788]\n","stream":"stdout","time":"2020-09-18T07:30:57.683127537Z"} {"log":"[S=26853607] [BID=fb0e19:3]  (N 24959594) SIPServersIPList::AddResolvedProxiesIPToList (ProxySet 1) - sip3.pstnhub.microsoft.com was resolved by DNS to 52.114.7.24 [Time:18-09@08:30:55.788]\n","stream":"stdout","time":"2020-09-18T07:30:57.683207738Z"} {"log":"[S=26853608] [BID=fb0e19:3]  (N 24959595) SIPServersIPList::UpdateList (ProxySet 1) - Update process finished [Time:18-09@08:30:55.788]\n","stream":"stdout","time":"2020-09-18T07:30:57.683236038Z"}

Here is my fluentd parser config:

<source>
  @id fluentd-containers.log
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/containers.log.pos
  tag raw.kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>
<filter kubernetes.var.log.containers.**count-log**.log>
  @id filter_rsyslog
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type regexp
    expression /^.*?\[S=(?<skey>.[0-9]+)\].*?(\[SID=(?<sid>.*?)\] | \s)(?<message>.*?)\[Time:(?<timedata>.*?)\]$/
  </parse>
</filter>

The problem is that fluentd combines multiple log messages into one entry and the resulting json is as follows:

{
"log": "[S=26852255] [SID=fb0e19:3:768876]  (N 24958326) (#498)gwSession[Allocated]. Handle:00007FD174227408; Global session ID: 899236dede666591 [Time:18-09@08:26:32.000]\n",
"stream": "stdout",
"skey": "26852255",
"sid": "fb0e19:3:768876",
"message": " (N 24958326) (#498)gwSession[Allocated]. Handle:00007FD174227408; Global session ID: 899236dede666591  (N 24958335) ---- Incoming SIP Message from 52.114.148.0:5061 to SIPInterface #1 (Teams_SIPInterface) TLS TO(#1520) SocketID(362205) ----  SIP/2.0 200 OK #012FROM: <sip:11.1.0.5>;tag=1c1113388569 #012TO: <sip:11.1.0.5> #012CSEQ: 1 OPTIONS #012CALL-ID: 494156082189202082632@sbc.connecttoteams.com #012VIA: SIP/2.0/TLS sbc.connecttoteams.com:5061;branch=z9hG4bKac2067450505 #012CONTENT-LENGTH: 0 #012ALLOW: INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY #012SERVER: Microsoft.PSTNHub.SIPProxy v.2020.9.5.3 i.USWE2.4 #012 #012  (N 24958336) SIPLadder::FillVQMData not generated  (N 24958337) AcSIPDialog(#10663): Handling 200 OK in state DialogInitiated  (N 24958338) States: (#10663)AcSIPDialog[DialogInitiated->DialogConnected]  (N 24958339) SIPServersMngr::UpdateSetWithOnlineServer - Server 52.114.148.0 already on working servers list  (N 24958340) AcSIPDialog(#10663): Handling DIALOG_DISCONNECT_REQ in state DialogConnected  (N 24958341) States: (#10663)AcSIPDialog[DialogConnected->DialogDisconnected]  (N 24958342) SIPAppMngr::FreeDialogAPI - (#1557)  (N 24958343) States: (#10663)AcSIPDialog[Deallocated] #012 (#10663)AcSIPDialog[DialogDisconnected->DialogIdle]  (N 24958344) (#498)gwSession[Deallocated]  (N 24958345) (#518)gwSession[Allocated]. Handle:00007FD1742270E8; Global session ID: 012e10c242bc7c0b  (N 24958346) Condition Table matched on condition Index 0  (N 24958347) Classification Succeeded - Source IP Group #1 (Teams_IPGroup)  (N 24958348) States: (#1170)SBCRoutesIterator[InitialRouting->AlternativeRouting]  (N 24958349) SBC_ADMIT_DIALOGS_EV: (#1170)SBCRoutesIterator -> (#0)SBCAdmissionControlMngr  (N 24958350) CAC: Add SBC Incoming Other, IPG 1 (Teams_IPGroup): 1, SRD 0 (DefaultSRD): 1, SipIF 1 (Teams_SIPInterface): 1  (N 24958351) CAC: Add SBC Outgoing Other, IPG 1 (Teams_IPGroup): 1, SRD 0 (DefaultSRD): 1, SipIF 1 (Teams_SIPInterface): 1  (N 24958352) (#1170)Route found (0), Route by Address, IP Group 1 -> 1 (Teams_IPGroup -> Teams_IPGroup), Url:internal:0;" 
}

How do i modify the regex or the fluentd config to make sure that message only contains content for the single log entry?

Parsing Audiocodes Logs with Fluentd Parser

0 Answers0