RemixJS - Cloudflare Server Adapter - Sentry Wrapper

[jcpaulco]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which package are you using?

@sentry/remix

SDK Version

7.11.0

Framework Version

Remix-Run: 1.6.7 & React 17.0.2

Link to Sentry event

No response

Steps to Reproduce

1. Opted to use Cloudflare workers as opposed to Remix's built in app server
2. Followed Sentry integration instructions
3. Sentry.captureException(new Error("Error) is only being reported to Sentry if in primary tsx component, Not in the loader / action functions.

It looks like if Express is used as the server adapter as opposed to cloudflare-workers then there exists the wrapExpressCreateRequestHandler to handle it but nothing exists for the cloudflare-workers.

See current server.js below utilizing cloudflare-workers:

import { createEventHandler } from "@remix-run/cloudflare-workers";
import * as build from "@remix-run/dev/server-build";

addEventListener(
  "fetch",
  createEventHandler({ build, mode: process.env.NODE_ENV })
);

Current Sentry.init in app/entry.server.tsx:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  tracesSampleRate: 0.2,
});

Expected Result

Sentry capture exceptions should be reported in the server code as well.

Actual Result

No sentry event is reported from the server, only client.

[AbhiPrasad]

Hey, There is currently no support for cloudflare workers since the Sentry SDK does not officially support the cloudflare worker runtime. Backlogging for the team, but PRs welcome in the meantime!

[AbhiPrasad]

Related: #2484

[huw]

Spent the morning on a workaround until we have proper WinterCG support. It’s a first pass, largely hacky, and relying on what we have in Toucan today.

First, copy-paste this into server/instrumentBuild.ts (create it next to server/index.ts):

import { type EntryContext, type HandleDocumentRequestFunction, type ServerBuild } from "@remix-run/cloudflare";
import { extractData, type AppData } from "@remix-run/react/dist/data";
import { isResponse } from "@remix-run/server-runtime/dist/responses";
import { type ActionFunction, type LoaderFunction } from "@remix-run/server-runtime/dist/routeModules";
import { type Exception, type WrappedFunction } from "@sentry/types";
import { fill, normalize } from "@sentry/utils";
import * as cookie from "cookie";
import type Toucan from "toucan-js";
import { type Options } from "toucan-js";

const extractResponseError = async (response: Response): Promise<unknown> => {
  const responseData = (await extractData(response)) as unknown;

  if (typeof responseData === "string") {
    return responseData;
  }

  if (response.statusText) {
    return response.statusText;
  }

  return responseData;
};

const captureRemixServerException = async (
  sentry: Toucan,
  error: unknown,
  name: string,
  request: Request
): Promise<void> => {
  // Skip capturing if the thrown error is not a 5xx response
  // https://remix.run/docs/en/v1/api/conventions#throwing-responses-in-loaders
  if (isResponse(error) && error.status < 500) {
    return;
  }

  const exception = isResponse(error) ? await extractResponseError(error) : error;

  sentry.withScope((scope) => {
    // Transformed & handled later in `beforeSend`
    // Attempts to match https://github.com/getsentry/sentry-javascript/blob/b49082e9aa0f0c1d7ffff5116fdf0412269e0204/packages/remix/src/utils/instrumentServer.ts#L107
    scope.setExtra("request", request);
    scope.setExtra("exceptionMechanism", {
      type: "instrument",
      handled: true,
      data: {
        function: name,
      },
    });

    sentry.captureException(exception);
  });
};

const makeWrappedDocumentRequestFunction =
  (sentry: Toucan) =>
  (origDocumentRequestFunction: HandleDocumentRequestFunction): HandleDocumentRequestFunction => {
    return async function (
      this: unknown,
      request: Request,
      responseStatusCode: number,
      responseHeaders: Headers,
      context: EntryContext
    ): Promise<Response> {
      let response: Response;

      try {
        // eslint-disable-next-line @babel/no-invalid-this -- We need to be able to pass `this` to the original function to wrap it correctly.
        response = await origDocumentRequestFunction.call(this, request, responseStatusCode, responseHeaders, context);
      } catch (error) {
        await captureRemixServerException(sentry, error, "documentRequest", request);
        throw error;
      }

      return response;
    };
  };

interface MakeWrappedDataFunction {
  (sentry: Toucan, dataFunctionType: "action", origFunction: ActionFunction): ActionFunction;
  (sentry: Toucan, dataFunctionType: "loader", origFunction: LoaderFunction): LoaderFunction;
}

const makeWrappedDataFunction: MakeWrappedDataFunction = (sentry: Toucan, dataFunctionType, origFunction) => {
  return async function (this: unknown, args: Parameters<typeof origFunction>[0]): Promise<Response | AppData> {
    let response: unknown;

    try {
      // eslint-disable-next-line @babel/no-invalid-this -- We need to be able to pass `this` to the original function to wrap it correctly.
      response = (await origFunction.call(this, args)) as unknown;
    } catch (error) {
      await captureRemixServerException(sentry, error, dataFunctionType, args.request);
      throw error;
    }

    return response;
  };
};

const makeWrappedAction =
  (sentry: Toucan) =>
  (origAction: ActionFunction): ActionFunction => {
    return makeWrappedDataFunction(sentry, "action", origAction);
  };

const makeWrappedLoader =
  (sentry: Toucan) =>
  (origLoader: LoaderFunction): LoaderFunction => {
    return makeWrappedDataFunction(sentry, "loader", origLoader);
  };

export const beforeSend: Options["beforeSend"] = (event) => {
  // Add data from the `request` extra to the event request object.
  // Mostly adapted from https://github.com/getsentry/sentry-javascript/blob/b49082e9aa0f0c1d7ffff5116fdf0412269e0204/packages/node/src/requestdata.ts#L140
  if (event.extra?.request) {
    const request = event.extra.request as Request;
    const headers = Object.fromEntries(request.headers.entries());
    const cookieHeader = headers.cookie;
    delete headers.cookie;

    event.request = {
      ...event.request,
      url: request.url,
      method: request.method,
      data: normalize(request.body) as unknown,
      query_string: new URL(request.url).search.replace("?", ""),
      cookies: (cookieHeader && cookie.parse(cookieHeader)) || {},
      // Add more to the allowlist later if you'd like.
      headers: { user_agent: headers["user-agent"] },
    };

    delete event.extra.request;
  }

  // Add the exception mechanism from extra to the event object.
  // Adapted from https://github.com/getsentry/sentry-javascript/blob/b49082e9aa0f0c1d7ffff5116fdf0412269e0204/packages/utils/src/misc.ts#L93
  if (event.extra?.exceptionMechanism) {
    const firstException = event.exception?.values?.[0];
    const newMechanism = event.extra?.exceptionMechanism as Exception["mechanism"];

    if (firstException) {
      const defaultMechanism = { type: "generic", handled: true };
      const currentMechanism = firstException.mechanism;
      firstException.mechanism = { ...defaultMechanism, ...currentMechanism, ...newMechanism };

      if (newMechanism && "data" in newMechanism) {
        const mergedData = { ...currentMechanism?.data, ...newMechanism?.data };
        firstException.mechanism.data = mergedData;
      }
    }

    delete event.extra.exceptionMechanism;
  }

  return event;
};

/**
 * Instruments `remix` ServerBuild for performance tracing and error tracking.
 */
export const instrumentBuild = (sentry: Toucan, build: ServerBuild): ServerBuild => {
  const routes: ServerBuild["routes"] = {};

  const wrappedEntry = { ...build.entry, module: { ...build.entry.module } };

  // Not keeping boolean flags like it's done for `requestHandler` functions,
  // Because the build can change between build and runtime.
  // So if there is a new `loader` or`action` or `documentRequest` after build.
  // We should be able to wrap them, as they may not be wrapped before.
  if (!(wrappedEntry.module.default as WrappedFunction).__sentry_original__) {
    fill(wrappedEntry.module, "default", makeWrappedDocumentRequestFunction(sentry));
  }

  for (const [id, route] of Object.entries(build.routes)) {
    const wrappedRoute = { ...route, module: { ...route.module } };

    if (wrappedRoute.module.action && !(wrappedRoute.module.action as WrappedFunction).__sentry_original__) {
      fill(wrappedRoute.module, "action", makeWrappedAction(sentry));
    }

    if (wrappedRoute.module.loader && !(wrappedRoute.module.loader as WrappedFunction).__sentry_original__) {
      fill(wrappedRoute.module, "loader", makeWrappedLoader(sentry));
    }

    routes[id] = wrappedRoute;
  }

  return { ...build, routes, entry: wrappedEntry };
};

export default instrumentBuild;

If you’re keeping score at home, this modifies instrumentServer.ts to use Toucan instead of @sentry/node. It drops anything that Toucan doesn’t natively support, which means you’ll lose some metadata about what action/loader was run as well as all tracing support.

Then modify server/index.ts:

import { createEventHandler } from "@remix-run/cloudflare-workers";
import * as build from "@remix-run/dev/server-build";
import Toucan from "toucan-js";
import instrumentBuild, { beforeSend } from "./instrumentBuild";

addEventListener("fetch", (event: FetchEvent) => {
  const sentry = new Toucan({
    dsn: SENTRY_DSN,
    context: event,
    beforeSend,
    // `allowedHeaders` and `allowedSearchParams` are ignored, see `beforeSend` in `instrumentBuild.ts`
    // allowedHeaders: ["user-agent"],
    // allowedSearchParams: /(.*)/u,
  });

  createEventHandler({
    build: instrumentBuild(sentry, build),
    mode: process.env.NODE_ENV,
  })(event);
});

Hope this helps anyone stuck on this one ^_^ I’ll try and patch it up a bit further to get metadata & tracing in.

[huw]

I finally got around to updating the above thanks to Toucan v3. Everything’s in this gist, but essentially you need to:

1. Rewrite instrumentBuild to accept a passed-through Hub instance, rather than using getCurrentHub() (including, perniciously, in helper functions such as hasTracingEnabled())
2. In server/index.ts, create a root transaction from the current route’s name and wrap the loaders and actions in spans via instrumentBuild
3. In root.tsx, pass through your tracing & baggage into the meta function
4. In root.tsx, include the current domain in tracePropagationTargets (because Remix fetchers will fetch from the entire URL rather than / root, which will confuse Sentry)

One major caveat is that server-side timing will be extremely inaccurate. Cloudflare freeze Date.now() on the server to avoid side-channel timing attacks. This means that within Workers, the time will never increment during a single request, and in Durable Objects, it will increment with the time of the last I/O. Relative times of multiple worker invocations during a single trace will be accurate, and so will anything measured on the client-side.

If anyone stumbles across this in the future and is having trouble with the code gimme a yell. (And if Sentry are interested I’ll happily modify the existing code to support Workers, but it’ll be a decent refactor because of getCurrentHub()).

[huw]

I’ve updated my Gist for Toucan v3.2.0 and Remix v2. It’s a lot more complicated & hacky now thanks to Remix v2, but still very possible. Please reply there if you have any questions :)

[huw]

@AbhiPrasad (or others) I’d like to start work on contributing back some stuff that should make non-Node runtime support a bit easier, but I wanted to run it by the team first in case you have better ideas than me :) (I know you’re working on Deno next—unsure how much of that would be reusable here)

The main change I’d like to start with would be (1) exporting the utility functions such as instrumentBuild and (2) refactoring the utility functions to accept getCurrentHub as an argument. This would allow Cloudflare Workers & Toucan users to run ex.:

const hub = new Toucan({ … });
const getCurrentHub = () => hub;
const newBuild = instrumentBuild(getCurrentHub, build);

This would go 90% of the way to preventing me from having to vendor & maintain your code in my repo while remaining fully backwards-compatible with the existing implementation. Would you be open to this?

[AbhiPrasad]

We have #9225 - feel free to take a look at that.

(2) refactoring the utility functions to accept getCurrentHub as an argument

What might be the easiest is just to change the behaviour of getCurrentHub by defining your own async context strategy. Something that looks like so https://github.com/getsentry/sentry-javascript/blob/develop/packages/vercel-edge/src/async.ts

[Lms24]

To confirm I understood things correctly:

1. First, the app source code is built with Vite. Source maps are generated and our vite plugin uploads these source maps to sentry
2. In a second step, wrangler take sthe build output from 1 and "re-builds" it for CF.

If so, is there a way you can add one of our plugins to that wrangler build? I know next to nothing about wrangler though 😬
If that doesn't work, you probably need to use Sentry CLI to inject debugids and upload source maps after the wrangler build.

If I understand things correctly, the source maps emitted by the wrangler build don't point back to the original source code but to the initial Vite build, correct? In that case you might need to use a tool like sorcery to "flatten" the source map chain. I guess this needs to happen post build but pre sentry CLI.

[charlie-bud]

@aaronadamsCA thanks for the feedback. We'll look at this next week as some of us are out of the office currently.

Hey @andreiborza - wondering whether your team made any progress on investigating this issue? Our team are still majorly hindered by vague "Unexpected Server Errors" being reported when unhandled server errors are thrown. Would be great to understand why this is a little better and if there's anything we can do in the meantime to mitigate 🙏

[AbhiPrasad]

@charlie-bud are you not able to set up and use handleError function? You should be able to call Sentry.captureException from there.

[charlie-bud]

@AbhiPrasad In entry.server.ts we have tried using export const handleError = Sentry.sentryHandleError; and a custom handleError that uses captureException and neither solution works when the app is built. It just logs an "Unexpected Server Error" with no further information or stack trace. We followed suggestions from this and this doc for reference but have had no luck. Reporting does work as expected in dev mode though 🤔

[charlie-bud]

@AbhiPrasad We have a basic reproduction of the issue here. If you build and start this app, you'll see the issue.

[geemanjs]

Just to add some meat onto @charlie-bud's great example from some first hand experience and following this issue for a some time 👀 .. I wonder if this should be split into two issues as there are two different problems. The second one is a bit of a wild ride so do strap in!!

Problem 1 - Remix deployed onto a Cloudflare worker results in incorrect sourcemap mapping
As described by @Lms24 above

• Source maps are uploaded to Sentry from the vite output
• The server bundle output from vite is loaded into a different js file which is what gets deployed onto the worker See the [[path]].ts file wrangler builds/runs the contents of the functions dir and you can see the import from the vite output in there

import * as build from "../build/server";

I wonder if this might help when it becomes more widely supported..
https://developers.cloudflare.com/workers/observability/source-maps/

 

Problem 2 - Making Remix handleError work with the @sentry/remix package in cloudflare
@charlie-bud's project demonstrates this well. I've raised a PR onto that with what I believe to be a "working version".

I focussed on the loader specifically and here there are two things happening in @charlie-bud's example:

1. The init of the sentry/cloudflare instance is called in middleware.ts imported into [[path]].ts
2. The server side handleError is using the sentry/remix instance which has not had init called upon it
3. The error occurs in the loader
4. As the sentry/remix instance hasn't been initd the error caught in handleError is dropped as the Sentry instance doesn't know where to send it
5. After passing through handleError the error is sanitised by remix and continues up to the client to be handled by an ErrorBoundary
6. There is no error boundary so the default one is used
7. When the error arrives at the client theres a sentry/remix version configured (in entry.client.js which mops up the sanitised error ("Unexpected Server Error") and fires this to sentry.

^^ Hopefully the wording makes sense but heres a picture that might help if not..

I raised a PR to show the changes made
ollie-bud/sentry-cloudflare-remix#1

From the image above that PR does the following to the image:

You will always get 2 errors (unless you add a custom error boundary)

1. From the server (worker) reporting the error - what @charlie-bud is after
2. From the client (browser) reporting the error after sanitisation as it wasn't handled by an error boundary

Something of note is if you try and call the @sentry/remix Sentry.init in a worker it will fail (so that options is out for now)

They're both pretty complex issues that are funnily not Cloudflare's, Remix's or Sentry's fault - just a side effect of using them all together 🙏

Unless the original @sentry/remix package can be made to work on a worker I think a @sentry/remix-cloudflare package might be needed as the implementation is quite specific to the combination - but hopefully the Sentry team has a better idea than me!!

[AbhiPrasad]

@geemanjs amazing writeup! We probably will go ahead an add workers export condition to the remix sdk to make this work.

@onurtemizkan I know you touched upon this before, now that we have @sentry/cloudflare how complex do you think it'll be to move forward with this?

[charlie-bud]

We probably will go ahead an add workers export condition to the remix sdk to make this work.

@AbhiPrasad Would this also cover fixing the sourcemaps, mentioned in @geemanjs's write up?

[AbhiPrasad]

Would this also cover fixing the sourcemaps

Nope, but that requires help from cloudflare, there's not much we can do alone here.