Enhance opentelemetry-demo to demonstrate OTel correlation and contextualization

[morningspace]

Feature Request

Enhance the opentelemetry-demo to demonstrate Instana correlation and contextualization capabilities with OTel data.

Basically, this item includes a tutorial (a doc) to setup Instana and opentelemetry-demo, then go through the correlation and contextualization features along with tips and best practices. During the time, figure out the gaps within opentelemetry-demo and fill the gaps by contributing back to the upstream repo.

Some demonstration areas including:

Correlation:

• Correlation via service.name: This seems to be OOB available in opentelemetry-demo, but there's one issue: it looks some service such as otel-col itself in opentelemetry-demo does not have service.name.
• Correlation via peer.service: It does have some service supporting this attribute, e.g.: checkout service. So, it appears to be just a documentation work for this item.
• Correlation specific in k8 env, via service.name, service.instance.id, process.pid, container.id, host.id, k8s.pod.uid: May need to test and verify.
• The use of spanmetrics: This seems already supported OOB in opentelemetry-demo.

Contextualization:

• Contextualization via tags for different supported technologies, e.g.:
  • HTTP: many service communication in opentelemetry-demo is done by HTTP
  • RPC: many service communication in opentelemetry-demo is done by RPC
  • Databases: there's no database service in opentelemetry-demo yet, some discussion ongoing in the community.
  • Messaging systems: opentelemetry-demo already has kafka service.

Advanced:

• Define scenarios to demonstrate OpenTelemetry monitoring and issue detection/remediation by Instana.
• Combine with Instana agent, e.g.: for IBM Middleware: Probably we can replace kafka w/ IBM MQ?

[morningspace]

We can define some specific scenarios for demo purpose by following: https://opentelemetry.io/docs/demo/feature-flags/. Here is an example: https://opentelemetry.io/docs/demo/scenarios/recommendation-cache/.