Parsing Logs With JSON and Non-JSON Payloads

[kieronlanning]

Hi,

I've rather a complex log that I can't change the format of that I'm hoping to parse for some analysis.

Each record looks a little like this:

$>{***DATETIME***} {***LEVEL***}: {***CLASS***}: {***LINE***}  {***Method***}
{***PROPERTY-NAME***}: {***GUID/STRING/INT***}
{***COLLECTIONAME***}:
  {***PROPERTY-NAME***}: {***GUID/STRING/INT***}
  {***PROPERTY-NAME***}: {***GUID/STRING/INT***}
  {***PROPERTY-NAME***}: {***GUID/STRING/INT***}
{*** MULTILINE OR SINGLELINE COMPLEX JSON ***}

The start of the record is always marked by a $>, but the combination of properties, collections and JSON payloads are all optional.

It's all working apart from the JSON part, which I'm only really interested in as a string value, rather than doing anything with it at this stage.

My issue seems to be getting it to stop parsing - it will continue parsing into the next record, instead of correctly detecting a new record.

In my tests if I remove the JSON payload from the logs, it will process all combinations of records without an issue.

Here's an example JSON payload (with bearer tokens and GUIDs replaced):

{
	"@odata.type": "#microsoft.graph.commsNotifications",
	"value": [{
		"@odata.type": "#microsoft.graph.commsNotification",
		"changeType": "deleted",
		"resourceUrl": "/communications/calls/guid",
		"resource": "/app/calls/guid",
		"resourceData": {
			"@odata.type": "#microsoft.graph.call",
			"state": "terminated",
			"resultInfo": {
				"@odata.type": "#microsoft.graph.resultInfo",
				"code": 200,
				"subcode": 0,
				"message": "This conversation has ended as only one participant was remaining in the conversation. DiagCode: 0#5010"
			},
			"meetingInfo": {
				"@odata.type": "#microsoft.graph.tokenMeetingInfo",
				"token": "bearer-token"
			},
			"callChainId": "guid",
			"id": "guid"
		}
	}]
}

[johngit22]

Thanks

…

On Fri, Aug 21, 2020, 9:04 AM Kieron Lanning ***@***.***> wrote: Hi, I've rather a complex log that I can't change the format of that I'm hoping to parse for some analysis. Each record looks a little like this: $>{***DATETIME***} {***LEVEL***}: {***CLASS***}: {***LINE***} {***Method***} {***PROPERTY-NAME***}: {***GUID/STRING/INT***} {***COLLECTIONAME***}: {***PROPERTY-NAME***}: {***GUID/STRING/INT***} {***PROPERTY-NAME***}: {***GUID/STRING/INT***} {***PROPERTY-NAME***}: {***GUID/STRING/INT***} {*** MULTILINE OR SINGLELINE COMPLEX JSON ***} The start of the record is always marked by a $>, but the combination of properties, collections and JSON payloads are all optional. It's all working apart from the JSON part, which I'm only really interested in as a string value, rather than doing anything with it at this stage. My issue seems to be getting it to stop parsing - it will continue parsing into the next record, instead of correctly detecting a new record. In my tests if I remove the JSON payload from the logs, it will process all combinations of records without an issue. Here's an example JSON payload (with bearer tokens and GUIDs replaced): { ***@***.***": "#microsoft.graph.commsNotifications", "value": [{ ***@***.***": "#microsoft.graph.commsNotification", "changeType": "deleted", "resourceUrl": "/communications/calls/guid", "resource": "/app/calls/guid", "resourceData": { ***@***.***": "#microsoft.graph.call", "state": "terminated", "resultInfo": { ***@***.***": "#microsoft.graph.resultInfo", "code": 200, "subcode": 0, "message": "This conversation has ended as only one participant was remaining in the conversation. DiagCode: 0#5010" }, "meetingInfo": { ***@***.***": "#microsoft.graph.tokenMeetingInfo", "token": "bearer-token" }, "callChainId": "guid", "id": "guid" } }] } — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#157>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYXHWKJTDYLYSQPPHYF4TTSBZWEPANCNFSM4QHIONCQ> .