Auto-balancing Playwright test shards #13384

Hey @aslushnikov , thanks for the message! Flakiness.io sounds like a great tool, especially for a real-world Playwright setup like ours.

I've pinged the folks on our team who handle our E2E tests (@daniellicnerski1 and @Victor-w-Madeira) to check this out. We'd love to see how it can help with our shard balancing. Thanks again for offering this to the OSS community!

Super beautiful website, btw!

Hey, @aslushnikov

Thanks for reaching out!

I'm Daniel, QA engineer at OrionTech, I currently own the E2E regression test suite for Langflow. The suite actually lives in a separate repo from Langflow itself: it's an independent Playwright + TypeScript project that runs against any Langflow
instance via URL, covering core flows, components, MCP, auth, playground, and so on, with nightly and weekly CI schedules.

Happy to try Flakiness.io on it. It's a real-world Playwright workload with LLM-backed flows (so we do see real flakiness), and we've been collecting historical run data in JSONL logs already, should be a useful target. Let me know what setup steps
you'd like me to take and I'll get it wired up.

Hey @daniellicnerski1!

The suite actually lives in a separate repo from Langflow itself: it's an independent Playwright + TypeScript project that runs against any Langflow instance via URL

With this poly-repo setup, Flakiness.io test analytics & regression analysis is pretty useless today: we attribute test results with the source code revision, thinking that commit id fully defines both the tests and the system-under-test. Maybe we can come up with some workarounds, depending on how exactly you set it up. Is the testing repository private? I can't find it, somehow.

On the plus side, shard auto-balancing should work just fine: it only uses historical timing data. For it to work, you just need to create an org + a project on Flakiness.io, and install the @flakiness/playwright reporter to upload the testing data. In a few weeks, once we release the experimental sharding, I'll come back to send a PR and enable the experiment for you. By this time, the project should already accumulate timing data, so we should be able to see how well it works.

Hi @aslushnikov
The repository is currently private, but I'm looking into the possibility of making it public, and I'll send it your way once I do.

Either way, I'm going to test it out in the repo to use some of the features. Definitely keep me updated on new features; I found it really interesting! Also, I'll talk to the team in charge of some of Langflow's current automations and tell them about the tool.

Let's stay in touch, thanks!

@daniellicnerski1 hey Daniel,

FYI: we've released the shard balancing as part of the flakiness/playwright reporter. The opt in should be straight-forward: just use npx flakiness-playwright-shard instead of the npx playwright test in your CI - it accepts all the same arguments:

-npx playwright test --shard 1/2 --project foo
+npx flakiness-playwright-shard --shard 1/2 --project foo

The documentation lives here: https://docs.flakiness.io/shard-balancing/

Let me know if I can be of any help with integration; we'd be really curious to see how it works for you.

@daniellicnerski1 hey Daniel!

I noticed you uploaded your first run to the https://flakiness.io/Orion/langflow-e2e. Congrats! Let us know how it goes!

I also noticed it took 1GB of space since there are also a lot of artifacts. Since it was almost the full quota of the free plan we provide, I promoted your org to a "Dedicated Open Source" plan - which gives you 100GB space for free. Let's see how long it'll last :)

We also recommend setting some reasonable artifact data retention, like 7 or 14 days, so that they don't eat through your storage quota.

❤️ Flakiness.io Team

Hi @aslushnikov — thank you so much for the Dedicated Open Source plan, we really appreciate the support! We will keep an eye on our artifact usage and manage what we upload to stay within a reasonable footprint.

Also, our E2E project is now public — feel free to take a look if you are curious: https://github.com/oriontech-me/langflow-e2e. Thanks again!