Rogue Parent ID generate in Azure Container App #28870

dickiebowuk · 2023-11-02T18:17:33Z

Component(s)

No response

Describe the issue you're reporting

I have written a simple API based on the scaffolded weather app API.

Scenario 1: I have the API running on my laptop, using Swagger Docs to call the end points that calls an Azure SQL Database to get the weather forecasts and exporting the telemetry data to an OTel Collector Contrib running in an Azure Container App with the ingress set to accept traffic from anywhere. The span collected does not have a Parent ID for the InstrumentationScope Microsoft.AspNetCore section.

Scenario 2: I have the same API running in an Azure Container App within a secure VNet. I use an APIM to call the API endpoints and it uses the same Azure SQL Database to get the weather reports. It exports the Otel data to a collector in an Azure Container App running in the same VNet. The span collected in this environment has a rogue Parent ID in the InstrumentationScope Microsoft.AspNetCore section that is not included in the Otel data.

What could be causing Otel libraries to think there is a parent span that it is not including in the data?

OTel Console output for Scenario 1:

2023-11-02T15:56:55.649302952Z 2023-11-02T15:56:55.649Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 2}
2023-11-02T15:56:55.650107441Z 2023-11-02T15:56:55.649Z	info	ResourceSpans #0
2023-11-02T15:56:55.650140021Z Resource SchemaURL: 
2023-11-02T15:56:55.650149810Z Resource attributes:
2023-11-02T15:56:55.650158075Z      -> telemetry.sdk.name: Str(opentelemetry)
2023-11-02T15:56:55.650166351Z      -> telemetry.sdk.language: Str(dotnet)
2023-11-02T15:56:55.650174285Z      -> telemetry.sdk.version: Str(1.6.0)
2023-11-02T15:56:55.650182391Z      -> environment: Str(Development)
2023-11-02T15:56:55.650190346Z      -> service.name: Str(Otelapiexample.API.Development)
2023-11-02T15:56:55.650210353Z      -> service.instance.id: Str(a2dfbcf1-4342-4202-aa86-d858a331fd80)
2023-11-02T15:56:55.650218919Z ScopeSpans #0
2023-11-02T15:56:55.650227275Z ScopeSpans SchemaURL: 
2023-11-02T15:56:55.650235820Z InstrumentationScope OpenTelemetry.Instrumentation.SqlClient 1.0.0.0
2023-11-02T15:56:55.650244396Z Span #0
2023-11-02T15:56:55.650252772Z     Trace ID       : e79bfa6236b05ace3a85656f5abe71c2
2023-11-02T15:56:55.650261508Z     Parent ID      : 1dcf7d47cef0ed68
2023-11-02T15:56:55.650270145Z     ID             : dc607b3c2ca2ced4
2023-11-02T15:56:55.650278680Z     Name           : WeatherForecast
2023-11-02T15:56:55.650287337Z     Kind           : Client
2023-11-02T15:56:55.650295332Z     Start time     : 2023-11-02 15:56:41.944259 +0000 UTC
2023-11-02T15:56:55.650303938Z     End time       : 2023-11-02 15:56:41.9706715 +0000 UTC
2023-11-02T15:56:55.650312123Z     Status code    : Unset
2023-11-02T15:56:55.650320328Z     Status message : 
2023-11-02T15:56:55.650341909Z Attributes:
2023-11-02T15:56:55.650356626Z      -> db.system: Str(mssql)
2023-11-02T15:56:55.650365212Z      -> db.name: Str(WeatherForecast)
2023-11-02T15:56:55.650373328Z      -> net.peer.name: Str(appdrabtest.database.windows.net)
2023-11-02T15:56:55.650382505Z      -> db.statement_type: Str(StoredProcedure)
2023-11-02T15:56:55.650391401Z      -> db.statement: Str(prWeatherSummaryList)
2023-11-02T15:56:55.650400839Z      -> sqlCommand.commandTimeOut: Str(30)
2023-11-02T15:56:55.650410286Z      -> peer.service: Str(appdrabtest.database.windows.net)
2023-11-02T15:56:55.650429633Z      -> collector.hostname: Str(collector--6eit1ja-544d484b97-8pnxn)
2023-11-02T15:56:55.650492510Z      -> azure.container_app.revision: Str(collector--6eit1ja)
2023-11-02T15:56:55.650584001Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:56:55.650616963Z      -> source.blog: Str(true)
2023-11-02T15:56:55.650645125Z ScopeSpans #1
2023-11-02T15:56:55.650978658Z ScopeSpans SchemaURL: 
2023-11-02T15:56:55.650999678Z InstrumentationScope Microsoft.AspNetCore 
2023-11-02T15:56:55.651022490Z Span #0
2023-11-02T15:56:55.651202797Z     Trace ID       : e79bfa6236b05ace3a85656f5abe71c2
2023-11-02T15:56:55.651221091Z     Parent ID      : 
2023-11-02T15:56:55.651261667Z     ID             : 1dcf7d47cef0ed68
2023-11-02T15:56:55.651282877Z     Name           : api/WeatherForecast/list
2023-11-02T15:56:55.651292695Z     Kind           : Server
2023-11-02T15:56:55.651301622Z     Start time     : 2023-11-02 15:56:41.9231418 +0000 UTC
2023-11-02T15:56:55.651310288Z     End time       : 2023-11-02 15:56:41.9843922 +0000 UTC
2023-11-02T15:56:55.651318944Z     Status code    : Unset
2023-11-02T15:56:55.651327780Z     Status message : 
2023-11-02T15:56:55.651335916Z Attributes:
2023-11-02T15:56:55.651344322Z      -> net.host.name: Str(localhost)
2023-11-02T15:56:55.651353128Z      -> net.host.port: Int(7200)
2023-11-02T15:56:55.651358989Z      -> http.method: Str(GET)
2023-11-02T15:56:55.651364640Z      -> http.scheme: Str(https)
2023-11-02T15:56:55.651370310Z      -> http.target: Str(/api/WeatherForecast/list)
2023-11-02T15:56:55.651376031Z      -> http.url: Str(https://localhost:7200/api/WeatherForecast/list)
2023-11-02T15:56:55.651381711Z      -> http.flavor: Str(2.0)
2023-11-02T15:56:55.651388614Z      -> http.user_agent: Str(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36)
2023-11-02T15:56:55.651397811Z      -> requestProtocol: Str(HTTP/2)
2023-11-02T15:56:55.651406778Z      -> http.route: Str(api/WeatherForecast/list)
2023-11-02T15:56:55.651414853Z      -> http.status_code: Int(200)
2023-11-02T15:56:55.651423129Z      -> collector.hostname: Str(collector--)
2023-11-02T15:56:55.651431364Z      -> azure.container_app.revision: Str(collector--)
2023-11-02T15:56:55.651437125Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:56:55.651442705Z      -> source.blog: Str(true)
2023-11-02T15:56:55.651448346Z 	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2023-11-02T15:56:55.651469055Z 2023-11-02T15:56:55.649Z	debug	[email protected]/otlp.go:127	Preparing to make HTTP request	{"kind": "exporter", "data_type": "traces", "name": "otlphttp/withauth", "url": "https://url/data/v1/trace"}

Otel log for Scenario 2:

2023-11-02T15:57:49.923527097Z 2023-11-02T15:57:49.923Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 2}
2023-11-02T15:57:49.924143138Z 2023-11-02T15:57:49.923Z	info	ResourceSpans #0
2023-11-02T15:57:49.924161152Z Resource SchemaURL: 
2023-11-02T15:57:49.924170489Z Resource attributes:
2023-11-02T15:57:49.924189435Z      -> telemetry.sdk.name: Str(opentelemetry)
2023-11-02T15:57:49.924197370Z      -> telemetry.sdk.language: Str(dotnet)
2023-11-02T15:57:49.924600348Z      -> telemetry.sdk.version: Str(1.6.0)
2023-11-02T15:57:49.924758661Z      -> environment: Str(Development)
2023-11-02T15:57:49.924807142Z      -> service.name: Str(Otelapiexample.API.Development.Azure)
2023-11-02T15:57:49.924818754Z      -> service.instance.id: Str(819c8322-2d73-448e-ae9c-4adebdc878f2)
2023-11-02T15:57:49.924827450Z ScopeSpans #0
2023-11-02T15:57:49.924835886Z ScopeSpans SchemaURL: 
2023-11-02T15:57:49.924844352Z InstrumentationScope OpenTelemetry.Instrumentation.SqlClient 1.0.0.0
2023-11-02T15:57:49.924856725Z Span #0
2023-11-02T15:57:49.924866213Z     Trace ID       : ea9b45ad8197256c0792e276e3e7cd4f
2023-11-02T15:57:49.924886331Z     Parent ID      : 6fe8f6d99948db15
2023-11-02T15:57:49.924894937Z     ID             : d4cecc62bc5a5684
2023-11-02T15:57:49.924903603Z     Name           : WeatherForecast
2023-11-02T15:57:49.924914604Z     Kind           : Client
2023-11-02T15:57:49.924922338Z     Start time     : 2023-11-02 15:57:40.6872252 +0000 UTC
2023-11-02T15:57:49.924954539Z     End time       : 2023-11-02 15:57:40.8726132 +0000 UTC
2023-11-02T15:57:49.924964487Z     Status code    : Unset
2023-11-02T15:57:49.924972903Z     Status message : 
2023-11-02T15:57:49.924984234Z Attributes:
2023-11-02T15:57:49.925398904Z      -> db.system: Str(mssql)
2023-11-02T15:57:49.925428810Z      -> db.name: Str(WeatherForecast)
2023-11-02T15:57:49.925437597Z      -> net.peer.name: Str(test.database.windows.net)
2023-11-02T15:57:49.925445582Z      -> db.statement_type: Str(StoredProcedure)
2023-11-02T15:57:49.925453196Z      -> db.statement: Str(prWeatherSummaryList)
2023-11-02T15:57:49.925460660Z      -> sqlCommand.commandTimeOut: Str(30)
2023-11-02T15:57:49.925468364Z      -> peer.service: Str(appdrabtest.database.windows.net)
2023-11-02T15:57:49.925478443Z      -> collector.hostname: Str(collector--)
2023-11-02T15:57:49.925486929Z      -> azure.container_app.revision: Str(collector--)
2023-11-02T15:57:49.925495005Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:57:49.925502699Z      -> source.blog: Str(true)
2023-11-02T15:57:49.925510614Z ScopeSpans #1
2023-11-02T15:57:49.925518238Z ScopeSpans SchemaURL: 
2023-11-02T15:57:49.925526193Z InstrumentationScope Microsoft.AspNetCore 
2023-11-02T15:57:49.925534108Z Span #0
2023-11-02T15:57:49.925542363Z     Trace ID       : ea9b45ad8197256c0792e276e3e7cd4f
2023-11-02T15:57:49.925550138Z     Parent ID      : 5f7bb59693824b8e
2023-11-02T15:57:49.925558544Z     ID             : 6fe8f6d99948db15
2023-11-02T15:57:49.925566228Z     Name           : api/WeatherForecast/list
2023-11-02T15:57:49.925574724Z     Kind           : Server
2023-11-02T15:57:49.925583160Z     Start time     : 2023-11-02 15:57:38.985265 +0000 UTC
2023-11-02T15:57:49.925590593Z     End time       : 2023-11-02 15:57:41.0932854 +0000 UTC
2023-11-02T15:57:49.925598869Z     Status code    : Unset
2023-11-02T15:57:49.925606684Z     Status message : 
2023-11-02T15:57:49.925614398Z Attributes:
2023-11-02T15:57:49.925622273Z      -> net.host.name: Str(otelapiexample.azurecontainerapps.io)
2023-11-02T15:57:49.925630308Z      -> http.method: Str(GET)
2023-11-02T15:57:49.925637792Z      -> http.scheme: Str(http)
2023-11-02T15:57:49.925656327Z      -> http.target: Str(/api/WeatherForecast/list)
2023-11-02T15:57:49.925664993Z      -> http.url: Str(http://otelapiexample.azurecontainerapps.io/api/WeatherForecast/list)
2023-11-02T15:57:49.925672417Z      -> http.flavor: Str(1.1)
2023-11-02T15:57:49.925680452Z      -> requestProtocol: Str(HTTP/1.1)
2023-11-02T15:57:49.925688497Z      -> http.route: Str(api/WeatherForecast/list)
2023-11-02T15:57:49.925696131Z      -> http.status_code: Int(200)
2023-11-02T15:57:49.925704116Z      -> collector.hostname: Str(collector--j38bkcz-6f84c5bb7d-sdkhp)
2023-11-02T15:57:49.925711771Z      -> azure.container_app.revision: Str(collector--j38bkcz)
2023-11-02T15:57:49.925719856Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:57:49.925728021Z      -> source.blog: Str(true)
2023-11-02T15:57:49.925735806Z 	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2023-11-02T15:57:49.925756164Z 2023-11-02T15:57:49.923Z	debug	[email protected]/otlp.go:127	Preparing to make HTTP request	{"kind": "exporter", "data_type": "traces", "name": "otlphttp/withauth", "url": "https://url/data/v1/trace"}

The text was updated successfully, but these errors were encountered:

crobert-1 · 2023-11-02T19:23:34Z

Hello @dickiebowuk, can you share the configuration of the collector that you're running?

dickiebowuk · 2023-11-02T19:30:10Z

Here is the config. It is set to send data to both New Relic and AppDynamics CNAO platform.

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlphttp/withauth:
    auth:
      authenticator: oauth2client
    logs_endpoint: $CNAO_LOGS_ENDPOINT
    metrics_endpoint: $CNAO_METRICS_ENDPOINT
    traces_endpoint: $CNAO_TRACES_ENDPOINT
  otlp:
    endpoint: $NEW_RELIC_END_POINT
    headers:
      api-key: $NEW_RELIC_API_KEY

  logging:
    loglevel: debug

processors:
  batch:
    send_batch_size: 8192
    timeout: 10s
  attributes/collector_info:
    actions:
      - key: collector.hostname
        value: $HOSTNAME
        action: insert
      - key: azure.container_app.revision
        value: $CONTAINER_APP_REVISION
        action: insert
      - key: azure.container_app.name
        value: $CONTAINER_APP_NAME
        action: insert
      - key: source.blog
        value: "true"
        action: insert
  filter/healthcheck:
    spans:
      exclude:
        match_type: strict
        attributes:
          - Key: http.target
            Value: /health

extensions:
  oauth2client:
    client_id: $CNAO_CLIENT_ID
    client_secret: $CNAO_CLIENT_SECRET
    token_url: $CNAO_TOKEN_URL

service:
  extensions: [oauth2client]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch,filter/healthcheck,attributes/collector_info]
      exporters: [logging, otlphttp/withauth, otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/withauth, otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/withauth, otlp]
  telemetry:
    logs:
      level: "debug"

github-actions · 2024-01-02T03:29:18Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

github-actions · 2024-03-04T03:41:44Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

github-actions · 2024-07-08T03:31:53Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

atoulme · 2024-09-03T23:56:05Z

I don't understand how to help you. I am not sure where the collector plays a role here. I suggest you provide a bug report with reproducible steps we can follow to help and assign the issue. I am closing the issue, please reopen if more work is needed.

dickiebowuk added the needs triage New item requiring triage label Nov 2, 2023

crobert-1 added bug Something isn't working question Further information is requested labels Nov 2, 2023

This was referenced Nov 7, 2023

Weekly Report: 2023-10-31 - 2023-11-07 #29000

Closed

Weekly Report: 2023-11-07 - 2023-11-14 #29245

Closed

This was referenced Nov 21, 2023

Weekly Report: 2023-11-14 - 2023-11-21 #29422

Closed

Weekly Report: 2023-11-21 - 2023-11-28 #29517

Closed

This was referenced Dec 5, 2023

Weekly Report: 2023-11-28 - 2023-12-05 #29650

Closed

Weekly Report: 2023-12-05 - 2023-12-12 #29753

Closed

This was referenced Dec 19, 2023

Weekly Report: 2023-12-12 - 2023-12-19 #30067

Closed

Weekly Report: 2023-12-19 - 2023-12-26 #30206

Closed

github-actions bot mentioned this issue Jan 2, 2024

Weekly Report: 2023-12-26 - 2024-01-02 #30242

Closed

88 tasks

github-actions bot added the Stale label Jan 2, 2024

crobert-1 removed the Stale label Jan 2, 2024

github-actions bot mentioned this issue Jan 9, 2024

Weekly Report: 2024-01-02 - 2024-01-09 #30334

Closed

github-actions bot mentioned this issue Jan 16, 2024

Weekly Report: 2024-01-09 - 2024-01-16 #30565

Closed

This was referenced Jan 23, 2024

Weekly Report: 2024-01-16 - 2024-01-23 #30711

Closed

Weekly Report: 2024-01-23 - 2024-01-30 #30848

Closed

github-actions bot mentioned this issue Feb 6, 2024

Weekly Report: 2024-01-30 - 2024-02-06 #31055

Closed

This was referenced Feb 13, 2024

Weekly Report: 2024-02-06 - 2024-02-13 #31192

Closed

Weekly Report: 2024-02-13 - 2024-02-20 #31323

Closed

This was referenced Feb 20, 2024

Weekly Report: 2024-02-13 - 2024-02-20 asuresh4/opentelemetry-collector-contrib#11541

Open

Weekly Report: 2024-02-20 - 2024-02-27 #31422

Closed

Weekly Report: 2024-02-20 - 2024-02-27 asuresh4/opentelemetry-collector-contrib#11542

Open

github-actions bot added the Stale label Mar 4, 2024

crobert-1 removed the Stale label Mar 4, 2024

github-actions bot mentioned this issue Jul 2, 2024

Weekly Report: 2024-06-25 - 2024-07-02 #33839

Closed

github-actions bot added the Stale label Jul 8, 2024

github-actions bot mentioned this issue Jul 9, 2024

Weekly Report: 2024-07-02 - 2024-07-09 #33962

Closed

This was referenced Jul 16, 2024

Weekly Report: 2024-07-09 - 2024-07-16 #34087

Closed

Weekly Report: 2024-07-16 - 2024-07-23 #34202

Closed

This was referenced Jul 30, 2024

Weekly Report: 2024-07-23 - 2024-07-30 #34301

Closed

Weekly Report: 2024-07-30 - 2024-08-06 #34410

Closed

This was referenced Aug 13, 2024

Weekly Report: 2024-08-06 - 2024-08-13 #34626

Closed

Weekly Report: 2024-08-13 - 2024-08-20 #34743

Closed

This was referenced Aug 27, 2024

Weekly Report: 2024-08-20 - 2024-08-27 #34856

Closed

Weekly Report: 2024-08-27 - 2024-09-03 #34966

Closed

atoulme closed this as not planned Won't fix, can't repro, duplicate, stale Sep 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rogue Parent ID generate in Azure Container App #28870

Rogue Parent ID generate in Azure Container App #28870

dickiebowuk commented Nov 2, 2023 •

edited

Loading

crobert-1 commented Nov 2, 2023

dickiebowuk commented Nov 2, 2023 •

edited

Loading

github-actions bot commented Jan 2, 2024

github-actions bot commented Mar 4, 2024

github-actions bot commented Jul 8, 2024

atoulme commented Sep 3, 2024

Rogue Parent ID generate in Azure Container App #28870

Rogue Parent ID generate in Azure Container App #28870

Comments

dickiebowuk commented Nov 2, 2023 • edited Loading

Component(s)

Describe the issue you're reporting

crobert-1 commented Nov 2, 2023

dickiebowuk commented Nov 2, 2023 • edited Loading

github-actions bot commented Jan 2, 2024

github-actions bot commented Mar 4, 2024

github-actions bot commented Jul 8, 2024

atoulme commented Sep 3, 2024

dickiebowuk commented Nov 2, 2023 •

edited

Loading

dickiebowuk commented Nov 2, 2023 •

edited

Loading