DirectMethod UploadModuleLogs causes to many AppendBlock transactions

[steffenbeermann]

Expected Behavior

When calling the UploadModuleLogs for one Module the logs should be uploaded to a log file in the specified container with a reasonable amount of transactions

Current Behavior

It seems like the EdgeAgent produces a lot of transactions when uploading the logs to the blob container. When I called the UploadModuleLogs method to upload 50.000 lines of logs (file size is 5.7mb) it produced around 90.000 transactions (reported by the metrics of the container with no other transactions happening expect the upload of the logs)

For us this results in around 3.5 Million! transactions each day to upload our edge logs.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. deploy a module that has a lot of logs
2. use UploadModuleLogs to upload the logs
3. the the sum of transactions in storage account metric

[bishal41]

@steffenbeermann is this still an active issue for you? How did you find about this? What's the impact of this issue? Cc: @huguesBouvier

[steffenbeermann]

@bishal41 yes, this is still an active issue for us. We figured this out because the storage account that we used to store the logs from our edge device caused very high costs for the amount of data it has. We are talking about one edge device with 2 modules we get the logs every hour (but with filter since, so no data is written multiple times). This is causing Storage Account costs of about 200$ per month, only because of the high amount of transactions.

When investigating we found out that the main cost driver is the huge sum of transactions that are executed against this storage account. Here is a screenshot of the sum of transaction of the storage account when I called the UploadModuleLogs for one module and with filter since = 1 hour. I used this call:

{
    "schemaVersion": "1.0",
    "sasUrl": "Sas URL",
    "items": [
        {
            "id": "Module Name",
            "filter": {
                "since": "1h"
            }
        }
    ],
    "encoding": "none",
    "contentType": "text"
}

It uploaded one file with 6.2 mb!

As you can see, this method call caused over 55.000 transactions.

You can easily try this (as I described in the steps to reproduce).

[bishal41]

@steffenbeermann would you be able to open a support ticket for this issue via Azure Portal?

[bishal41]

@steffenbeermann were you able to create a support ticket for this issue?

[steffenbeermann]

@bishal41 sorry, I was allocated to other task and could not check back. I just created the support ticket on the IoT Hub Service