Corrupt breadcrumbs in captureException

[dmudro]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

I have a concerning case in our cloud @sentry/nextjs instance as I am seeing incorrect information attached in the breadcrumbs.

Our fairly standard Sentry implementation hosted in Vercel logs issues via captureException; a serverless function attaches extra data in captureException(think of custom transaction id and data) so that the extra data is available in Sentry UI via breadcrumbs.

In one particular issue instance there is completely unrelated data in breadcrumbs. The issue itself was expected but down in the breadcrumbs I see data matching another lambda execution running 5 minutes earlier (that execution did not log any issue Sentry). I literally see someone else's transaction id in the breadcrumb running 5 minutes later.

It almost looks like an issue with AWS Lambda global scope inside Sentry's Node package which would raise concerns.

Apologies for a rather abstract description though am not able to replicate (having the corrupt issue archived). Am opening this ticket to see if there is or has been a known similar issue before. Or maybe someone from the Sentry team might be able to relate to another bug or a sub-optimality in the code base. Found a GH ticket about global Sentry.setUser() but that's probably a long shot.

The app was running @sentry/nextjs v7.75.1 at the time.

Expected Result

Breadcumb trace should belong to the current execution

Actual Result

Breadcumb trace logs data from earlier Lambda execution (in memory)

Product Area

Issues

Link

No response

DSN

https://[email protected]/5680289

Version

7.75.1

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-issues for triage ⏲️

[lobsterkatie]

Moving this to the SDK repo, since it's about what data is gathered rather than about displaying that data.

[lforst]

Hi, can you share a link to an event that has these corrupt breadcrumbs? Thanks!

[dmudro]

thanks everyone.

the breadcrumbs in event link contain sensitive PII so am unable to share. I was able to reproduce I would have shared such replicated event.

I understand this might be too anecdotal and not easy to investigate so feel free to close. that's unless someone can decipher the issue by locating the code area with potentially leaky global scope in Lambda.

[lforst]

Data from the Sentry SDK may be shared between invocations. The current architecture of the Sentry SDK requires some data to reside in the global scope. What should not happen is that you will get leakage from another AWS customer or vice versa. If that is the case we (and most likely AWS) have a bigger issue on our hands.

Breadcrumbs are a bit buggy at the moment and they may leak right now. We are planning to fix this in the next major.

[dmudro]

thanks for the feedback earlier:

The current architecture of the Sentry SDK requires some data to reside in the global scope.

this sounds suspicious and could account for unrelated breadcrumbs data across different sentry events

What should not happen is that you will get leakage from another AWS customer or vice versa.

didn't have that yet. that would be quite an issue indeed.

Breadcrumbs are a bit buggy at the moment and they may leak right now. We are planning to fix this in the next major.

great to hear that. we just had a case where a completely unrelated endpoint was logged as a breadcrumb entry. I am literally looking at our health check (private devops) endpoint logged in breadcrumbs for a regular Next.js page render which are two completely unrelated part of the system. vaguely similar as the original issue in this thread and it all points to a common low level root issue in breadcrumbs.

not sure what can be done right now and how actionable is this particular issue; feel free to close as I won't be able to provide more specific data. sounds like the team knows the root cause and hopefully to be addressed in the next major version.

[Lms24]

We're working on the new major at the moment. The breadcrumbs situation should improve with our new scope model.

[lforst]

We released a new major with improved breadcrumbing isolation. Feel free to let us know if the situation improved for you after upgrading.