Data Connection Rule / Endpoint and Log Analytics DCR-driven table for structured logging

[Ryan-Palmer]

I have just set up structured logging in Log Analytics using Serilog's most recent plugin.

As part of that, you have to follow a guide showing you how to use the Log Ingestion API.

The steps are roughly

1. Create a Data Collection Rule (DCR) driven table in your log analytics workspace.
2. Create and assign it a DCR, which itself references to a Data Collection Endpoint (DCE).
3. Both the table and the DCR need an example of the JSON blob which will be sent, and the DCR needs to specify which column is the timestamp.
4. The DCR needs to assign the Monitoring Metrics Publisher role to the identity used to push logs. The guide shows using an EntraId app but it actually supports System Identity now.
5. The app needs to load the DCE endpoint and DCR immutable id / stream name into config for the Serilog plugin to use.

I had to do all of these steps in the portal as I couldn't find Farmer support for them, although I may have missed it.

There is a guide to deploying with ARM.

I think we would need the abilities to

• Create a DCR, which can automatically create its DCE if using ARM to deploy.
• Add a DCR driven table in the LogAnalytics builder, referencing the DCR resource.
• Supply the JSON template strings to the DCR Data Source and Log Analytics table schema, and the Timestamp field name to the DCR for the Transformation mapping
• Assign the MonitoringMetricPublisher role to the web app's SystemIdentity in the DCR builder
• Pluck out the DCR / DCE app settings we need and load them into the web app.

This is another one I am happy to pick up and have a go at if it sounds like I am on the right track? :)

[ninjarobot]

There is not Farmer support, and it would definitely be appreciated. I tried to figure out the changes to get a VM to send syslog using DCR a while back and wasn't able to get it working end to end through ARM. If you have more to share on what you got working or could submit a draft PR to get things started, please mention me and I'll be happy to help.

[Ryan-Palmer]

I managed to get a few things working with the JSON escape hatch.

• DCR table added to Log Analytics
• DCR added targeting the table

let loggingName = "log-analytics"

let logging = logAnalytics {
    name loggingName
    retention_period 30<Days>
    daily_cap 1<Gb> // Need alert if we hit limit, see below
}

let serilogDcr // Hard coded Serilog columns
    serilogTableName
    dcrName
    location
    workspaceResourceId
    logWorkspaceName
    streamName =
    $"""{{
        "type": "microsoft.insights/datacollectionrules",
        "apiVersion": "2023-03-11",
        "dependsOn": [
            "[resourceId('Microsoft.OperationalInsights/workspaces/tables', '{logWorkspaceName}', '{serilogTableName}')]"
        ],
        "name": "{dcrName}",
        "location": "{location}",
        "tags": {{}},
        "properties": {{
            "streamDeclarations": {{
                "{streamName}": {{
                    "columns": [
                        {{
                            "name": "TimeGenerated",
                            "type": "datetime"
                        }},
                        {{
                            "name": "Event",
                            "type": "dynamic"
                        }}
                    ]
                }}
            }},
            "dataSources": {{}},
            "destinations": {{
                "logAnalytics": [
                    {{
                        "workspaceResourceId": "{workspaceResourceId}",
                        "name": "{logWorkspaceName}"
                    }}
                ]
            }},
            "dataFlows": [
                {{
                    "streams": [
                        "{streamName}"
                    ],
                    "destinations": [
                        "{logWorkspaceName}"
                    ],
                    "transformKql": "source",
                    "outputStream": "{streamName}"
                }}
            ]
        }}
    }}"""
    |> Resource.ofJson

let serilogTable // Hard coded Serilog columns / Analytics plan
    serilogTableName
    logWorkspaceName
    retentionInDays =
    $"""{{
        "type": "Microsoft.OperationalInsights/workspaces/tables",
        "apiVersion": "2023-09-01",
        "dependsOn": [
            "[resourceId('Microsoft.OperationalInsights/workspaces', '{logWorkspaceName}')]"
        ],
        "name": "{logWorkspaceName}/{serilogTableName}",
        "properties": {{
            "plan": "Analytics",
            "retentionInDays": "{retentionInDays}",
            "schema": {{
                "columns": [
                    {{
                        "name": "TimeGenerated",
                        "type": "datetime"
                    }},
                    {{
                        "name": "Event",
                        "type": "dynamic"
                    }}
                ],
                "name": "{serilogTableName}"
            }},
            "totalRetentionInDays": "{retentionInDays}"
        }}
    }}"""
    |> Resource.ofJson

let deployLocation = Location.UKSouth
let workspaceResourceId = $"/subscriptions/{subId}/resourceGroups/{resGroupName}/providers/Microsoft.OperationalInsights/workspaces/{loggingName}"

let serilogTableName = "Serilog_CL"

let logTable = serilogTable serilogTableName loggingName 30

let dcr =
    serilogDcr
        serilogTableName
        "SerilogDCR"
        deployLocation.ArmValue
        workspaceResourceId
        loggingName
        $"Custom-{serilogTableName}"

[Ryan-Palmer]

I managed to get the Serilog ingestion working by clicking it together in the portal before I started, and the only thing I have to add to the Farmer deploy now to finish the recreation is the MonitoringMetricsPublisher role for my AppService's SystemIdentity with scope of the DCR.

That will be something like this but I am still trying to get it to deploy

let metricsPublishingRole =
    {  Name = createRoleName app.Name.ResourceName.Value app.SystemIdentity.PrincipalId Roles.MonitoringMetricsPublisher
       RoleDefinitionId = Roles.MonitoringMetricsPublisher
       PrincipalId = app.SystemIdentity.PrincipalId
       PrincipalType = Arm.RoleAssignment.PrincipalType.ServicePrincipal
       Scope = Arm.RoleAssignment.AssignmentScope.SpecificResource dcr.ResourceId
       Dependencies = Set.ofList [ dcr.ResourceId; app.ResourceId ] } :> IArmResource

[Ryan-Palmer]

Got it - scoping to the resource seems bugged, returning e.g.

"Deployment template validation failed: 'The template reference '9ad14d86-68ad-4f28-8395-5c33873955f4' is not valid: could not find template resource or resource copy with this name.

If I just scope to the resource group it deploys though.

    let metricsPublishingRole =
        {  Name = createRoleName app.Name.ResourceName.Value app.SystemIdentity.PrincipalId Roles.MonitoringMetricsPublisher
           RoleDefinitionId = Roles.MonitoringMetricsPublisher
           PrincipalId = app.SystemIdentity.PrincipalId
           PrincipalType = Arm.RoleAssignment.PrincipalType.ServicePrincipal
           Scope = Arm.RoleAssignment.AssignmentScope.ResourceGroup // Tried to scope to DCR but failed with unknown resource id
           Dependencies = Set.empty }

I will test if the Serilog and OTel data is flowing through and confirm.

[Ryan-Palmer]

Role assignment worked as I have metrics coming through. I will need to create a DCE even though they are optional for ARM-created DCRs, because the Serilog plugin needs the URL.

Once I have this all working I'll share the complete sample.

[Ryan-Palmer]

@ninjarobot Here's a complete working sample.

It has Serilog structured logging, OTel metric and Traces plus the search query alert for data limits.

I will hopefully find time to make a proper Farmer PR rather than using the JSON, I just need to re-familiarise myself with the code, it's been a while!

https://github.com/Ryan-Palmer/Safe-Serilog-Farmer-Demo