Inside the WooCommerce Release Process

James:
Hello. Welcome to this episode of Inside Woo, where I think this is my first episode of Inside Woo officially, which will be interesting. Previously, I was doing the product talk. I am James Kemp from WooCommerce. I’m the core product manager there, and today I am joined by Julia Amva, and we’re going to talk about the release process for WooCommerce and how that’s kind of gone since Julia was last on, which was—we were just discussing—five years ago now. What’s kind of changed since then, what the process looks like now, and how people can help. Julia, do you want to introduce yourself?

Julia:
Sure, thanks, James. Thanks for having me here today. I’m excited to be back on this podcast after five years. Like you said, last time I checked, it was March 2020 that I was here. So yes, I’m Julia. I’m currently leading the WooCommerce quality team. We are responsible for WooCommerce, in particular, the stability and performance of releases. Five years ago, I was a member of the team that worked on releases, and I had another quality engineer. I was an individual contributor. I worked with another quality engineer on that team at the time. Since then, I have built my own team. I hired a team of quality engineers. I had two teams split from my team that sort of went out and built their own functions within Woo. But I remained leading the effort in ensuring the quality, stability, and performance of WooCommerce releases. This is what we do day in and day out as a team, and I’m here to talk about this today.

James:
That’s awesome. How big is your team now, then?

Julia:
We’re currently a team of eight engineers, including myself, located around the world—Europe, Asia, the United States. Pretty exciting.

James:
Yeah. Amazing. So during those five years, at what point did you switch to leading the team?

Julia:
So this October, we celebrated four years—our four-year birthday as a team. So it’s been four years that I’ve been leading the team.

James:
Amazing. So not long after the last podcast.

Julia:
Right? Exactly.

James:
And for reference, I guess the last podcast was discussing pretty similar things that we’re going to discuss today, right?

Julia:
Yep. We’ll talk about the release process, all the things that we have done to improve, and we’ll talk a little bit about the challenges that we have today, how the community can help with releases, and also what’s coming up in 2025. I’ll share a bit about the plans that we have.

James:
Amazing. Yeah. I know you faced some release issues at the beginning of 2024. Did you want to dive into that a little bit?

Julia:
Sure, yes. I think it’s important to mention. So 2024 was a really challenging year for us, and we did start with a couple of not-so-great releases—8.5 and 8.6 in particular. We took it, of course, very seriously. We recognize the importance of—well, we always knew the importance of our work—but we recognize the importance of continuing with that stability and good-quality releases for merchants and builders alike in the community. We took it to heart, and we worked the entire year on making sure we got back on track to where we were before. But not only that, we also worked to improve the releases.

James:
Yeah, it’s a hard challenge, right? Because WooCommerce is rolled out into so many different environments and so many different configurations that having a solid way to test every single possible configuration is a challenge in itself.

Julia:
It’s definitely a challenge. It is definitely a challenge. And also, the product grows with time. Five years ago, for example, we had just one team that we would call a core team that developed the core experience, and then that team would release the software. Today, fast forward five years, we have on average 10 engineering teams contributing to core daily, plus the community, of course. So the product has scaled significantly. It’s not only the variety of things we have to test for and account for during testing, but also how to organize the process within the company to ensure everybody follows the same way of working and ensures the quality of the releases. That’s also a big challenge, but we’re here for it, we’re working on it, and that’s an exciting thing to be a part of.

James:
Yeah, obviously, it’s something you enjoy doing because you’ve been doing it for so long.

Julia:
Yes.

James:
Which is great. I think it’s the ideal scenario when the people doing the work actually enjoy the challenge of that kind of thing. Yeah, really cool. So you mentioned there were some issues with 8.5 and 8.6. What did you end up implementing, or what kind of came out of that that has now made that process better?

Julia:
Sure, sure, sure. Yeah, that’s a great question. Great starter. So 8.5 and 8.6 were the releases soon after we merged WooCommerce Blocks into core, which happened in December 2023. These first two releases followed that and also introduced a couple of bigger features, like order attribution to core. This ties back to what I was saying about multiple teams contributing to core. The product started to grow more and more, and basically, what we found is that the processes we used in the past, when we had just one or fewer teams contributing, in terms of how we test and develop software, no longer worked for us. We had to keep up with the pace of the product growing and scaling and make adjustments.

We made a bunch of different changes, but I’ll focus on some of the ones that I think made the biggest difference. When you think about product quality as a whole, I think where it really originates is in the development process—how engineers contribute code and what happens at the pull request level. We’ve adopted the mindset that every pull request we merge must be production-ready. I’m not saying we’re there yet, but this is something we are working towards, and we started to make changes to make that a reality.

We looked into how engineers introduce new features—what goes in, what the processes are—and we worked with them on ensuring they create some kind of a test plan and think in advance about how they’re going to test the new feature, how it’s going to interact with everything else already in core, and what the different milestones are along the way for a feature being ready to be included in core. Instead of, “Here is my feature, now can somebody please test it?” we try to put some responsibility for code quality on the engineers themselves.

We also looked into the way we write testing instructions in pull requests. We ask engineers to improve how they document testing steps. It might seem like a simple thing, but when you write very clear testing instructions on a pull request, first of all, you’ve likely tested it yourself. Then it’s very clear to the reviewer what to test. And we ask for something more complex, not just a basic “happy path” test, but also: What are the integrations this feature must work with? What are the different ways it could break?

So we worked quite a bit with engineers on that to ensure that when a pull request is merged into core (into trunk), it’s as close to production-ready as possible, as if it were being released in a continuous delivery model. WooCommerce doesn’t work that way just yet, but we try to think of it in those terms. Again, we’re not there yet, but it’s something we’re paying a lot of attention to.

James:
You mentioned testing—Is that primarily automated testing, or is there a manual testing process that’s required?

Julia:
It’s a mix of both. If we’re talking about just pull requests and not releases yet, we have lots of automated testing in place. For example, we have a test suite of end-to-end tests, unit tests, API tests, and lots of other CI checks. But in particular, our unit, API, and end-to-end tests cover the majority of core’s critical flows. We run an evergreen list of critical functionality that must always work in WooCommerce, so it’s easier for us to check whether those flows are working or not. This runs on every pull request.

Then, of course, there’s manual testing. We ask pull request reviewers to manually test every PR. We try to make it easier for them—for example, they can quickly spin up a test site. There’s a comment that gets published on the pull request, and they can easily click on it and go to a test environment to check the PR.

James:
That’s the Playground, right?

Julia:
Exactly, which I think simplifies the process a lot. In terms of automated testing, we’ve also worked quite a bit on improving the flakiness of our end-to-end tests, which are notoriously very flaky. We’ve spent a good amount of time improving stability in that area. I’d say the test suite is now pretty stable—it’s mostly green every day. Of course, today, for example, I checked and it was red, so my team is looking into those failing tests right now.

James:
And that’s on the trunk branch, is it?

Julia:
Yes.

James:
Some of the other things you mentioned earlier—back to the improvements you’ve made in the process—was expanding the environments where releases are tested. Can you talk more about that?

Julia:
Yes, we significantly expanded the list of environments where we test releases. We now cover our top 10 hosting environments, top-used plugins, top-used themes, and various combinations of all those factors. We’ll talk more about that when I go over how we actually test releases today.

But at a high level, on the development process side, I’d say the two biggest improvements were:

1. Every pull request must be production-ready before merging.
2. Major improvements in testing—both automated and manual.

Those two changes have made a big difference.

James:
Yeah, that makes sense. So alongside continuous testing, you’re rigorously testing any code that goes into core before it actually gets merged.

Julia:
Right, exactly.

James:
You touched on that as part of the process before releasing WooCommerce itself as a new version. Regarding the version numbers WooCommerce gets, has that changed? As far as I’m aware, do we use SemVer?

Julia:
Yes, we use Semantic Versioning (SemVer). That part hasn’t changed—we’ve continued with that pattern.

James:
Yeah, I like that pattern. I think it helps identify what kind of release it is.

Julia:
Agreed.

James:
As long as you’re aware of what those numbers mean. I’m glad we stuck with that. In terms of actually releasing WooCommerce, what’s the process like now?

Julia:
Sure. I’ll talk about some of the improvements we’ve made in the last five years, particularly in the last year.

For most of the last five years, we released WooCommerce on a rigid cadence—every second Tuesday of the month. That meant 12 major releases per year, roughly every four to five weeks, without accounting for holidays or major events like Black Friday. The idea was to keep it predictable—so teams could plan their features, and the community would know what to expect.

However, this schedule had challenges. If we needed to delay a release due to a critical issue, we wouldn’t necessarily adjust the follow-up schedule. That sometimes meant working on two releases at once, which was incredibly challenging. We also found that the beta testing period was too short. Given the growing complexity of WooCommerce, we realized we didn’t have enough time to properly test releases. Plus, the community was providing feedback that the releases were happening too frequently.

So towards the end of last year, we lengthened the release process. Now, instead of every four weeks, we release every five weeks. We also moved releases from Tuesdays to Mondays.

James:
What was the reasoning behind moving releases to Monday?

Julia:
Releasing on Monday gives us a bigger window to follow up with a fix release if needed. If we released on a Tuesday and found a critical issue, we might not be able to release a fix on Wednesday. That would push us to Thursday, which is too close to the weekend. We’re hesitant to release close to weekends because fewer team members are around to react to any potential issues. Moving to Monday gives us a safer window.

The five-week release cycle also gives us a three-week beta period, which is a huge improvement. We also started adjusting for major holidays. For example, last year, we skipped an October release because it didn’t align well with upcoming holidays worldwide.

Internally, this change has given us more breathing room. I’m curious to know if it’s made a difference to the community as well. We always welcome feedback, and anyone can share their thoughts on the WooCommerce Community Slack or the developer blog.

James:
Yeah, I remember seeing a lot of feedback when it was a four-week release cycle—people saying it was too much. Since changing to five weeks, I haven’t seen as much feedback like that.

Julia:
That’s great to hear. But honestly, our end goal is for people not to even notice the release happened. Ideally, updates should just work, with no issues—no disruptions.

James:
Yeah, exactly. If it was causing problems, we’d definitely hear about it. If things are going unnoticed, that’s probably a good sign that it’s working smoothly.

Julia:
Definitely.

James:
Cool. So you changed the cadence of releases—was there anything else about the actual deployment of those releases that changed?

Julia:
Yes! But before I get into deployment, I’d like to talk about how we test a release before deployment.

We talked earlier about what happens at the pull request level, but once all those pull requests are merged into trunk, we cut what we call a beta package. That’s when we enter a code freeze—nothing new can be added to that release. This helps testing significantly because we know exactly what’s in scope.

We cut the beta on the same day as the previous release. For example, WooCommerce 9.7 will be released on February 24th, and on the same day, we’ll cut the beta for WooCommerce 9.8.

James:
So the code freeze is in place for 9.8 immediately after 9.7 is released?

Julia:
Exactly.

James:
But there are code freeze exceptions, right? If necessary, there’s a way to submit new code?

Julia:
Yes, and that’s actually one of the challenges we’re trying to overcome.

BobWP:
Weglot is the easiest way to translate your WordPress site or WooCommerce shop. As an official partner of Woo, you will find them on the Woo Marketplace. Their extension is easy to install, built for maximum compatibility, and optimized for SEO. Once it’s in place, it will help your clients increase their shop’s visibility, reduce bounce rates, enhance the user experience, localize media assets, and boost their content with AI. Just visit WooCommerce.com and find them in their marketplace, or go to Weglot.com.

Julia:
We do allow code freeze exceptions, but that’s actually one of the challenges we are trying to overcome. At this beta stage, the beta period starts, which lasts three weeks. At the same time, we also announce to the community that WooCommerce 9.8 is coming through the WooCommerce Developer Blog. For listeners who don’t follow it, that’s where we share information about what’s coming, including new features, changes, and any API or database modifications.

We hope the community will start testing alongside us. Our own testing process is now very thorough and includes both automated and manual testing, with a large group of people involved in testing the release.

My team of quality engineers is just eight people, which isn’t enough to fully test WooCommerce releases, especially given their complexity. So, we involve engineering teams in testing as well. We believe it helps with ownership—when engineers know their code will be tested by others, they tend to be more careful. So, releases are tested by everyone, including engineers.

The release branch goes through multiple automated tests, including extended tests covering critical core flows. We run end-to-end tests, API tests, and, of course, unit tests. Additionally, we’ve worked on introducing different critical environments where we run these tests—not just on a vanilla CI container but on actual live sites hosted on various platforms.

Currently, this process is fully available for WP Cloud (Automattic’s hosting platform) and Pressable (another Automattic hosting solution), but we have plans to expand it to other hosting providers. One challenge we face is ensuring that our test suite works seamlessly across different hosting environments. For example, certain flows—like onboarding—can vary significantly depending on the host. We are actively working on making the testing framework adaptable to these differences.

James:
That makes sense. It’s interesting that you’re testing not just in controlled environments but also across different real-world hosting setups.

Julia:
Yes, absolutely. In addition to all that, we also conduct performance testing on releases. We use a tool called K6, which is a popular performance testing framework. We run our tests on large test sites with thousands of orders and products, simulating different functionalities and measuring how the site performs. This helps us spot potential regressions.

Another thing we introduced in 2024 is AI-driven security testing. Everyone wants to leverage AI these days, and we’ve found a great way to use it for security scanning. We use AI to scan the code and flag potential vulnerabilities, and it has proven to be quite valuable. It has caught security issues that we then addressed before release.

James:
That’s cool! Is that an in-house solution, or are you using an external service?

Julia:
It’s a mix—we rely on OpenAI’s technology, but the implementation itself is an in-house solution built specifically for WooCommerce.

James:
That’s great! And I assume it’s tailored specifically to WooCommerce’s needs?

Julia:
Yes, exactly.

James:
So, we’ve talked about testing WooCommerce core, but as you mentioned earlier, no one is running just core. Every store has multiple plugins installed. How do you handle compatibility testing?

Julia:
Yes, great question. That’s actually a huge challenge. We know that the average WooCommerce site runs 50 to 60 plugins, and each combination of extensions introduces potential compatibility risks.

To tackle this, we built a tool called the Quality Insights Toolkit (QIT). It allows us to test WooCommerce core with various extensions activated at the same time to check for compatibility issues.

Right now, this tool is available for extensions sold through WooCommerce Marketplace (woocommerce.com). We have over 800 extensions in the Marketplace, and every time a new plugin is submitted or an existing plugin gets updated, the tool runs a series of automated tests on it.

We also use this tool during WooCommerce releases. We can run a mass test where we activate all 800 extensions and run end-to-end tests to detect compatibility issues. This helps us catch a lot of potential conflicts before they reach merchants.

One of the coolest features we introduced last year is custom tests. This allows extension developers to write their own test cases that are then included in our automated testing process. For example, Stripe has custom tests that set up payment methods, create orders, and check compatibility within WooCommerce core. That way, we’re not just testing core, but also ensuring third-party extensions work properly with new releases.

James:
That’s really impressive—so, tons of testing.

Julia:
Yes, tons of testing. Like I said, my team literally does this every day—it’s what we breathe and live. It’s challenging but also exciting.

James:
You mentioned earlier that WooCommerce releases first roll out on WP Cloud sites. How does that work?

Julia:
Yes, this is what we call our “bake period”, and it has been a game-changer for us.

After we complete all our internal testing, we release WooCommerce on WordPress.org—but we do not update the stable tag immediately. That means users don’t get an update notification yet, and sites set to auto-update won’t receive the new version either.

Instead, we roll out the release only to WP Cloud-powered sites (around 300,000 sites). This allows us to monitor the release in a controlled environment. Since WP Cloud is Automattic’s hosting platform, we have real-time access to logs, so we can immediately see any errors that occur.

We monitor the release for four hours, tracking logs and looking for any spikes in fatal errors. If we detect a serious issue, we can roll back the release on WP Cloud within 30 minutes, preventing the problem from reaching the broader WooCommerce user base.

James:
That’s really smart. So, in 2024, how often did this process catch issues before they reached the full WooCommerce user base?

Julia:
We had 11 major releases last year, and our canary testing process caught critical issues in five of them. These were major bugs that could have caused widespread problems.

Because of this system, we were able to prevent those issues from rolling out to millions of WooCommerce stores. Instead, we reverted the release on WP Cloud, issued a fix, and then re-released a stable version.

James:
That’s a great safeguard. Do you think other hosting providers could implement a similar process?

Julia:
Yes, absolutely. And we’d love to collaborate with other hosting companies to implement similar staged rollouts. If more hosting providers adopted this approach—rolling out to a subset of users first, monitoring for errors, and rolling back if necessary—the quality of releases across the WooCommerce ecosystem would improve dramatically.

James:
That makes a lot of sense. It’s similar to how mobile OS updates are rolled out gradually.

Julia:
Exactly. The more controlled we can make the process, the smoother updates will be for merchants.

James:
Yeah, I think I touched on that at the beginning. There are so many different combinations—not only of plugins but also different server configurations and settings—that affect how an update works. I do think it would be very beneficial for other hosts to adopt this kind of rollout process. And in theory, they could, right? If they monitored the non-stable release, they could choose to update a subset of sites first.

Julia:
Indeed. And we could also prolong the bake period if needed. Right now, we monitor for four hours, but that could be adjusted depending on the hosting provider. Since WP Cloud is an internal platform, we can act quickly, but if another host wanted to implement this, we could work with them to set up a process that fits their infrastructure.

James:
So, you mentioned four hours. You do the initial release—let’s call it the “pre-stable release”—and monitor it for four hours. Assuming everything goes well, you then tag it as stable, correct?

Julia:
Yes, exactly. If everything looks good, we mark it as stable, and the release rolls out to the broader WooCommerce community.

James:
And if something goes wrong during that bake period?

Julia:
If we detect a serious issue, we roll back the release on WP Cloud immediately and do not mark it as stable. Instead, we analyze the issue, work on a fix, and then re-release as a minor version update.

For example, let’s say we’re releasing WooCommerce 9.7. If we find a critical bug during the bake period, we don’t mark 9.7 as stable. Instead, we fix the issue and release 9.7.1 as the stable version.

We also notify the community via the WooCommerce Developer Blog. You might have seen some of these posts where we say, “Our canary testing caught a pre-release issue, and we will follow up with a fix in the next version.”

This process has been really effective. For example, in our last two releases—9.5 and 9.6—the bake period helped us catch an issue in 9.5, which led to a fix before it reached the broader WooCommerce audience. In 9.6, no issues were detected, so it rolled out smoothly without any follow-up releases.

James:
That’s awesome. It’s great that we have that safeguard in place. When I ran Iconic (a WooCommerce plugin company), I used a similar strategy. I released updates gradually to a percentage of users before doing a full rollout. But what we’re doing with WooCommerce is even more beneficial because we can roll back if needed.

Julia:
Yes, exactly. And we use a similar staged rollout approach for new WooCommerce features as well.

For example, when we released WooCommerce Brands in 9.6, it was actually available in 9.5 but hidden behind a feature flag. That meant developers could enable it early, test it, and provide feedback before we made it fully available in 9.6.

For some features, we also do incremental rollouts—starting with 10% of users, then gradually increasing the percentage before making it fully available. This helps us catch potential issues in real-world environments without affecting everyone at once.

James:
That makes a lot of sense. It also lets us capture different environment combinations that wouldn’t be possible to test internally.

Julia:
Exactly. It’s a critical part of our process.

James:
So, was there anything else about the release process that you wanted to touch on?

Julia:
I think we’ve covered most of it! But I would like to emphasize how much the community can help. One of the biggest challenges we still face is ensuring we test as many real-world scenarios as possible.

That’s why community beta testing is so important. The more people test beta releases and provide feedback, the better our releases will be.

James:
Right! And we have a beta testing plugin for WooCommerce, right?

Julia:
Yes, we do! The WooCommerce Beta Tester plugin lets merchants and developers install beta versions of WooCommerce and test upcoming releases before they go live.

We also publish Developer Blog posts announcing new betas, listing the changes, and asking for feedback. Right now, we rely on the community voluntarily testing and reporting issues, but we’d like to make this process more structured.

James:
What do you mean by making it more structured?

Julia:
We used to have a formal community beta testing program, but we sunsetted it a while back. Now, we’re looking at bringing it back.

The challenge in the past was getting consistent feedback. Because we release every five weeks, people might test one release but then be too busy to test the next. We want to create a program that makes it easier for people to participate regularly.

Another idea we’re exploring is a “WooCommerce Compatibility Checker” plugin. This would be a plugin that merchants can install, and it would automatically test their site for compatibility with upcoming WooCommerce updates.

James:
Oh, that sounds cool! So, merchants install the plugin while running the current version of WooCommerce, and it checks if their setup is compatible with the next release?

Julia:
Exactly. The idea is to proactively detect potential issues before they upgrade.

Right now, we ask people to manually test beta versions, but not everyone has time to do that. If we could automate the process, we’d likely get more participation and better coverage.

It’s still early days, though—we’re just discussing feasibility at this point. But I think it could be a game-changer for WooCommerce stability.

James:
That would be great. But I imagine it would also add complexity on our end because we’d need to write tests that validate code before it even exists on live sites.

Julia:
Yes, that’s definitely one of the challenges. But we already have a structured process for writing tests for new features. For example, when we introduced WooCommerce Brands in 9.5, the team responsible for that feature wrote its own test suite.

So, if we extend that approach and package those tests in a compatibility-checking plugin, it could work. The real challenge would be ensuring the tests run reliably across thousands of unique store setups.

James:
That makes sense. Okay, so looking ahead—what are the main goals for WooCommerce releases in 2025?

Julia:
Our top priority is ensuring we don’t need unplanned fix releases.

We started 2024 with a few rough releases (8.5 and 8.6), but by the end of the year, we stabilized. In fact, WooCommerce 9.6 didn’t require any fix releases at all—that’s what we want to continue doing in 2025.

Another key goal is reducing the time it takes to issue a fix release if one is needed. Right now, our average time to release a fix is 2.5 days—we want to improve that.

James:
And who handles the fix release process? Does your team do that, or does it go back to engineering?

Julia:
It goes back to engineering, but my team facilitates the process. We determine:

• What the root cause is
• How many sites are affected
• What testing needs to be done before release

James:
That makes sense. It sounds like we’re already on the right track!

Julia:
Yes, we’re prepared and ready for 2025!

James:
I’m looking forward to seeing more hosts adopt this strategy and work with us to make WooCommerce releases as reliable as possible.

Julia:
Same!

James:
Cool. I think we’ve covered a lot today. Where can people find you if they want to connect?

Julia:
I don’t use Twitter (or X) much, but I’m active in the WooCommerce Community Slack. My username is just my first and last name, Julia Amva, and I check it regularly. Feel free to reach out!

James:
Awesome. Thanks for coming on the show—maybe we’ll chat again in five years!

Julia:
Thank you so much, James! Yes, looking forward to the next one. Thanks for having me.

James:
Thanks, Julia.

Julia:
Thanks. Cheers. Bye.