Almost every hosted identity provider ships a backdoor for your tests

I looked at 284 public repositories that use a hosted identity provider and have end-to-end tests. Exactly one has a signup test that reads a real verification code out of a real inbox. One. And it is a vendor's own repository - Stytch's - paying a third-party inbox service to do it.

Thirty-five build a way around the problem instead: a forged session cookie, a mock-jwt-token written straight into localStorage, an environment variable that switches auth off for the test run. Three write a comment saying they tried and could not.

That distribution is not laziness. It is what the documentation asks for.

The backdoor is documented, and it has a name

If you use a hosted identity provider and have ever written an end-to-end test that has to log in, you have probably already met it. It is not a hack anyone discovered. It has a name, a documentation page, and usually a fixed code the vendor tells you to type.

Clerk: "Any email with the +clerk_test subaddress is a test email address." The verification code is 424242, and the docs are explicit about what does not happen: "When testing email verification codes, no email with the verification code will be sent." Turning that on for a production instance is possible, and Clerk's own comment on it is "However, this is highly discouraged."

Stytch publishes sandbox values: OTP 000000, phone +10000000000, and this description of the result: "If your API credentials and the request format are correct you will receive a 200 status response, but no email will actually be sent."

Firebase, for phone auth: "When you provide the fictional phone number and send the verification code, no actual SMS is sent."

Descope describes the mechanism it recommends like this: "Utilizing test users, you can generate OTP codes and Magic/Enchanted link tokens for test users directly using the Descope API and SDK, without sending actual communications to the test account." And on the static OTP variant of that same test-user feature, Descope prints its own warning callout: "This is an insecure method and is only recommended when generated OTP codes are not viable for testing."

That warning is worth placing precisely, because I got it wrong in my own notes first. It sits on static OTP codes, not on the generate-OTP mechanism Descope recommends. "Vendor labels its own backdoor insecure" is the better story and the false one.

Auth0 is the most explicit of the ten. Its own writeup on Cypress testing states the principle - "The best practice is never to visit or test third-party sites over which you have no control." - and then argues its way around it: "Keep in mind that you must not use this grant on your public clients. This is an exception to this rule because it is an end-to-end test that won't be used by real users."

Out of ten providers, eight give you an official way to skip the real message; seven name the feature outright. Supabase gets there differently: in local development, auth.email.enable_confirmations defaults to false and an inbucket mail catcher is already in the stack, so the backdoor is the default rather than a feature you switch on. Two have nothing. SuperTokens' testing documentation covers API testing with Postman, debug logs and troubleshooting, with no test mode and no fixed OTP. And Kinde says the quiet part out loud - "Kinde requires OTP email verification when signing up for a new user." - then points at somebody else's product: "Test email services like Mailosaur or Mailtrap provide API access to test inboxes, making it easy to retrieve OTP codes programmatically."

What the backdoor buys, and what it takes

The trade is worth stating fairly, because for most teams it is a good trade. A documented backdoor exercises your code: route guards, session handling, post-login redirects, role checks, all running against the real provider, deterministically, at CI speed. That is most of what most people want from an auth test.

What it does not touch is the message. Whether the template renders. Whether it arrives. Whether the link is clickable, points at the right host, and still works when a human takes eleven seconds to read the email first. The backdoor's whole point is that the message never gets made.

The vendors do not hide this seam. Stytch marks where it is: "The sandbox values below are only available when calling the Stytch API directly. They will not work when used with a frontend or mobile Stytch SDK."

Clerk's own Playwright example repo says it in two comment lines that belong together:

// Unique email per run so concurrent runs don't collide.
// Uses +clerk_test so 424242 works as the verification code.

clerk/clerk-playwright-nextjs, e2e/app.spec.ts:35-36.

A repository in my corpus, woody34/rescope, says it plainer still, on the line after it reads a code out of an emulator's escape-hatch API: "In real apps this would come from the email inbox."

The part I measured

A word on sample selection, since it decides everything downstream. The ten providers are the nine whose SDKs the corpus search actually turned up, plus Dynamic, which is in the documentation survey only. The 284 repositories are what survives a rule set fixed before the data was seen: a package.json holding both an identity SDK and an E2E runner, then not a fork, not archived, pushed within eighteen months, an E2E config actually present. That ran against a frame which censused every qualifying repository for the five smaller providers and took a seeded random 250 per provider for the four largest. No query anywhere in the pipeline sorted by stars or by "best match", and vendor-owned repositories are counted in their own tier rather than mixed into the denominator.

Then the method, including where it broke. I re-downloaded all 284 repositories over a different transport than the original pass used - the tarball API instead of git clone - recording each HEAD SHA, then applied detection signals written from scratch. All 284 still existed. Then I audited my own download, because I had piped curl into tar and swallowed stderr, which means a truncated download fails silently and every count after it is quietly short. Two repositories had in fact been truncated, one at 36 of 272 files and one at 14 of 865. I refetched both. Neither had a hit on either signal, so no number moved. It could easily have gone the other way.

The one repository that consumes a real code is stytchauth/stytch-browser, Stytch's own, and it does it with Mailosaur, a paid service:

const email = `${emailName}+${timestamp.getTime()}@${MAILOSAUR_SERVER_ID}.mailosaur.net`;

then cy.mailosaurGetMessage(, then const tokenLink = email.text.links[0].href;, then cy.visit(tokenLink);. The dependency is pinned as "cypress-mailosaur": "5.0.0", and the key comes from CI as cypress_mailosaur_api_key: ${{ secrets.CYPRESS_MAILOSAUR_API_KEY }}. The entire E2E suite is three spec files.

That "1" survived three unrelated ways of asking. The first pass scanned test bodies for vendor-specific SDK verbs. The second asked something else: can this repository talk to an inbox at all? It scanned dependency manifests, docker-compose files, CI workflows and test code for test-inbox SDKs, IMAP clients, local mail catchers, and Gmail/Graph APIs. 43 repositories had some inbox capability somewhere; 6 had it inside a test file near identity code; opening all six by hand left one, the same one. The other five: a Mailinator mention inside a skipped test's comment; ethereal.email in a non-identity flow; IMAP as a product feature; a getMailhogEmails helper never called; and test@testmail.com, a hardcoded login my regex mistook for the testmail.app service.

The third was rebuilding that capability scan from scratch and running it again, because the second pass had left prose and no code. Its first stage is much tighter - 15 repositories with any inbox capability, not 43 - and it still lands on the same six candidates and the same single confirmed case. Two nets of very different mesh, dragged through the same 284 repositories, brought up the same six fish. That is the strongest thing I can say about this number, and I could not have said it without re-deriving it.

Thirty-five repositories out of 284 build their own backdoor. They fall into two shapes. Forged credentials, injected straight into the browser:

• Luigi-Faldetta/fit-log: win.localStorage.setItem('clerk-db-jwt', 'mock-jwt-token');
• SDG-AI-Lab/Digital_Technologies_Radar, which annotates itself: // Set logged in state above window.localStorage.setItem('drr-current-user-id', 'admin');
• tensr-xyz/tensr-platform-web: a cookie name: 'stytch_session_token', / value: 'e2e-playwright-session', plus the same key in localStorage - and the product code reads that exact key: const SESSION_TOKEN_KEY = 'stytch_session_token';

And switches, mostly set in the harness's own webServer.command - command: 'cross-env VITE_E2E_SKIP_AUTH=true vite --port 5174 --strictPort', in vandean25/auto-core-platform, VITE_E2E_BYPASS_AUTH in gumacahin/mis-capstone, NEXT_PUBLIC_BYPASS_AUTH in a committed .env. Those prefixes are part of the names: a bare BYPASS_AUTH or SKIP_AUTH appears in none of the 284, only in my own earlier notes, where I had shortened them. The rest are in the published manifest.

That number moved three times, and the moves are the most useful thing I can show you about it.

The first pass published 31/284 and hand-checked it. A verification pass found 34 - the same 31 plus three it had missed, because its keyword list required an injected storage key to contain one of a fixed set of words. SDG-AI-Lab/Digital_Technologies_Radar (cypress/e2e/create-disaster.cy.ts:2-4) and ubcdiscovery/ubc-discovery, which sets ubc-discovery-test-google-user at web/e2e/identity-convergence.spec.ts:104-110 and reads that key back in product code at web/app/lib/firebase.ts:22, both use keys containing user, which was not on the list. kil-dev/kil.dev has a whole module for the job at src/lib/admin-test-bypass.ts:1-2 and was missed only because the spec imports the cookie constant instead of writing the string.

Then, writing this, I noticed that the verification pass had produced prose and no code, so its 34 could not be re-run by anyone, including me. I rebuilt it from scratch and ran it again against a freshly downloaded corpus. It came back 35.

The thirty-fifth is sefi-uzan/yanshuf-ai. It keeps a hardcoded next-auth.session-token JWT in tests/e2e/fixtures/config.ts:8-13 and injects it with addCookies(userCookies) at tests/e2e/pages/website.ts:13. Two earlier passes missed it for the same structural reason kil.dev was missed: the argument is a variable, so there is no string literal beside the call for a regex to catch. It exists, it is a forged session, and it was found only because the number was re-derived rather than quoted.

That is what the re-run bought, and it cuts both ways. The published figure was too low, twice. The set is a strict superset each time - nothing found earlier failed to reappear - so the error has consistently been undercounting. And I still have not opened all 35 by hand: the shared 31 rest on the first pass's hand check, which self-reported 17 of 20 correct on a 20-repository sample.

One thing I removed rather than kept. That re-run first returned 36, including a repository whose only hit was AUTH_META = { name: 'mockAuth' } inside a file of route-interception mocks - a mocked API response, not a switch that turns authentication off. It matched because my rewritten pattern included MOCK|FAKE, which appears in neither the original regex nor the written definition of this category. I deleted the token and re-ran everything rather than excluding the repository, because deleting a token that was never in the definition is checkable and could have dropped others too. It dropped exactly one. Both counts are published.

Three repositories say in a comment that they gave up. The most complete is drifter089/orgOS, tests/auth-unauthenticated.spec.ts:117-127:

  test.skip("should complete sign-in flow with valid credentials", async ({
    page,
  }) => {
    // NOTE: This test is skipped because WorkOS requires email verification (OTP)
    // which cannot be automated in this test environment without access to the email inbox.
    //
    // To enable this test, you would need:
    // 1. Configure WorkOS to disable email verification for test environment
    // 2. Use WorkOS test mode API if available
    // 3. Integrate with an email testing service (e.g., Mailinator, Mailtrap)
    // 4. Use WorkOS impersonation feature if available in your plan

All four items are there on purpose. Cutting the list after item 2 makes it read as a dead end, and it is not one: the developer who gave up knew commercial inboxes exist and named two of them. The second is intelogroup/ugent - "NOTE: Full OTP login requires a real email. Tests that need auth are marked with [auth-required] and skip if UGENT_TEST_OTP is not set." The third, amirrudd/flyerboard, does not belong in an email argument and I will not pretend otherwise: its blocker is Descope SMS OTP.

Three out of 284, and that three is itself a correction: an earlier summary table of mine said five. The machine output of that same pass listed three, and rescanning all 284 confirmed three.

Where the backdoor stops: running in parallel

The backdoor gets you logged in. It does not get you logged in twice at once.

Clerk addresses this directly in its Playwright guide, and the answer is to give up the parallelism: "Setup must be run serially, this is necessary if Playwright is configured to run fully parallel", with the sample calling setup.describe.configure({ mode: 'serial' }).

Playwright's own documentation recommends a pool of accounts for exactly this case: "This is the recommended approach for tests that modify server-side state. ... We will need multiple testing accounts, one per each parallel worker." (Elided: three sentences explaining that each worker authenticates once and reuses that state.) The sample code then calls const account = await acquireAccount(id); - twice, in two separate examples - and never defines acquireAccount anywhere on the page. The doc gives the call site, not the implementation, which is a reasonable place to stop; Playwright cannot know your provider. The only guidance is in the sample's comments: acquire a unique account, or keep a list of precreated ones, and make sure they are unique so teammates running tests at the same time do not collide. The pool is your problem.

yravan/cashlens did the assignment. It has per-account storage state and a two-account pool in its global setup - apps/web/e2e/global.setup.ts:16-17, keyed a and b, each with its own email and its own storageState file. It follows the official pattern. And its Playwright config still reads:

  workers: 1, // parallel clerk.signIn is flaky (clerk/javascript#7891)

That issue body describes it precisely: "When using @clerk/testing with Playwright and --workers=2 (or more), all tests fail with TimeoutError: page.waitForFunction: Timeout 15000ms exceeded at clerk.signIn(). With --workers=1, authentication works 100% reliably."

The issue is closed - 2026-04-03, state_reason: completed. It has exactly two comments, both from the same user, whose author_association is NONE. No comment on it from the vendor.

The fix people take away from that thread is usually stated as "give each worker its own test account". The actual text is more modest, and getting it right matters: it is the second suggestion in that comment, introduced as "One practical workaround while waiting for a proper fix: pre-authenticate each worker with a dedicated test user and save the storage state to separate JSON files, then load each worker's state from its own file rather than calling signIn() concurrently." The primary suggestion in the same comment was a separate browser profile directory per worker plus a file lock. One community member, two ideas, one of them explicitly labelled a stopgap. That is the documented state of the art for this wall.

Most of you do not need what I am building

I started on this because of my own product, giaoanai. It is mine, which means it represents nobody but me and proves nothing about anybody else's stack, and that is why I am publishing zero numbers from it here. It sends real email in real signup flows, and I could not test that end to end without a real inbox. Everything above is what I found when I went looking for how other people had solved it.

If you control your own sending stack, you do not need a product for this. Run mailpit next to your dev server, point SMTP at it, read the message back over its HTTP API. Single binary, web UI, JSON API. The glue is a few dozen lines and you own all of it. Supabase already bundles an equivalent for local development.

If your mail goes out through a third party and you want support behind the inbox, buy one. Mailosaur is what Stytch recommends in its own docs - "we generally recommend using a platform like Mailosaur to set up a programmatically accessible email or SMS inbox" - and what Stytch's own suite uses, as quoted above. Kinde points at Mailosaur or Mailtrap. These products work, they predate me, and if one fits, use it.

What is left over is narrow: several workers at once, each needing its own identity and its own real message, against a provider whose test mode is per message rather than per worker. That is the case AuthStunt is for. It hands each worker its own address and its own real message, and that is all I am going to claim in a post whose whole argument is that claims need receipts.

Check my work

Every quotation above carries a repository, a path, a line number and the SHA it was read at; vendor quotes carry the URL, read 2026-08-16. All of it ships with the corpus, the detection signals and the case-by-case verdicts, so you can disagree with a specific judgment rather than with a summary. Public repositories change: if a line number no longer matches, the SHA still pins what I read.

Dataset and per-quote provenance: github.com/ivermin1123/authstunt/tree/main/research/post-1-repro-kit

If I filed a repository wrong, tell me which one and why. The number that would embarrass me most is the "1 out of 284", and the fastest way to move it is somebody pointing at repository number two.