How WordPress Core Performance Drives Better Plugin and Site Speed

Zach: Welcome to another, Do the Woo dev chat. I’m Zach Stepek and I have with me, Carl and Till. How are you two doing?

Till: Pretty good, man.

Carl: Doing great.

Zach: So today we have a guest from the WordPress core performance team, Adam Silverstein.

Adam: Hey.

Zach: And Adam, we brought you in because we want to talk to you about all this topic. We keep coming back to you on these dev chats performance, and you are really heavily involved in performance in WordPress core. And we wanted to talk to you about that and how it ties into performance for large plugins, like WooCommerce.

Adam: Great, excited to be here.

Zach: So why don’t you tell us a little bit about yourself before we really get into the meet here?

Adam: Sure. Just a brief history of, I guess, of my developer journey, long time building websites in HTML and CSS, and then found WordPress at some point. Really started contributing about 10 years ago, 3.6. I helped build the revisions rewrite. That was my first involvement in WordPress core. And then I’ve been pretty much involved in every release since then, and wound up switching over to working at an agency 10up for about six years before joining Google, the developer relations team, working with some other WordPress core people, trying to move the web forward, improve the web as a whole at scale, which WordPress is a big part of that. So, that’s me in a nutshell, I guess.

Zach: Now, there was a project at 10up that led to a few people going over to Google. What was that project that started that whole migration?

Adam: That actually happened at the same time, but those were not related. So the project is Sitekit. It’s a plugin from Google that connects your WordPress site to Google services. So, analytics and search console and AdSense data, like all brought right into the WordPress dashboard. And that was a 10up project that they’re still working on for Google, or with Google, I guess. But actually right around the same time that Google came out with that plugin, there were a number of core committers and contributors that were hired, about five people. I think all at once that were hired, it was a momentous moment to me in the WordPress community. I remember going to WordCamp in, I forget where it was St. Louis, or no, it was even before that, Nashville maybe and there was a Google booth in the vendor area.

I was like, “Wow, what is Google doing here at the WordCamp?” To me, that seemed like a great thing to be happening. And I was very excited about it. That was before all this happened, but right around that time, you saw some people from the WordPress community get hired by developer relations team at Google. And also they released this first-party plugin. They’ve made other efforts in the past to have plugins for WordPress, but this was really the first time they did something, I think really seriously and successfully. It’s a very successful plugin, has over two million installs now and five star rating on the support form, which is not an easy thing to do.

Zach: Right and there were some folks from XWP that joined it about the same time as well.

Adam: Yep, exactly.

Zach: So, yeah, it was a big thing at the time, all the people that were leaving and joining Google and how Google was seeming to start to take WordPress very seriously.

Till: And why is that? Why is Google taking WordPress seriously and what are they getting out of it? I’m a little bit suspicious.

Adam: And people were asking that question at the time. I remember the question at State of the Word was when is Google going to buy WordPress? So, it’s like, it’s that even possible? But yeah, I mean, so the way I like to think of it and I’m speaking here from my own personal perspective, I’m not speaking for Google, I guess, is that Google grew up and lives on the web. The web is the most important thing for Google. It’s what we all go use Google to search for, is things on the web and most content created on the web was created in WordPress. So, there’s a natural synergy there, where it is in Google’s interest to see WordPress be as successful product. And for example, in the case of performance, and why we’d be focused on performance if sites are horrible experiences for users, the web is not going to be a place people want to go looking for information. They’re going to go to apps that have a better user experience.

So, there’s a need for the web to actually be a good place to read content. It’s not only about how easy it is to create content, but what is the quality of that content quality-wise, but also how does it perform? Are users having a good experience with it? So, there’s definitely a synergy there. I would say externally, from WordPress, it seemed like Google is paying a lot of attention to WordPress, suddenly hiring five people, but internally being a Googler, I have the opposite perception. I feel like Google should pay way more attention to WordPress than we do.

There’s all kinds of products at Google. And I mentioned, we have Sitekit, which covers like four products or maybe five products, but there’s probably 20 other products that Google has that have really significant usage on the web APIs. And in all of the cases, those APIs are probably most used by WordPress sites. So, it would pay for Google to pay more attention to how are developers using our APIs. I think, from the WordPress perspective, it seemed like suddenly Google was paying a lot of attention, but from my perspective, I wish we paid more attention and put more resources towards making WordPress better. So, that’s what I’m advocating for internally, as best I can.

Carl: So a good start, but needs more.

Adam: Yeah, it’s a good start. I think this year in particular, we’ve really refocused on WordPress. My team is focused on CMSs in general. So, content management systems, when you look around the world, there are hundreds of them, maybe even a thousand and in some markets, WordPress is not a big thing. There’s definitely markets where there’s other CMSs that are wildly popular and WordPress is like second or third. And so when you are a company at the scale of Google, you really do have to look at all the things. WordPress is obviously the biggest.

Carl: Do you have an example?

Adam: Yeah, ECQ is one in Japan. There’s another one called Silverstripe that I think is in Japan, that’s very popular. WordPress is also very popular in Japan, so I don’t know if it’s the top one, but it’s markets like that. It’ll be a very specific market and there’ll be some CMS that’s more native to that market. I also know, there are some languages that WordPress is not localized in. I think in India, the primary language, which I’m probably going to get this wrong, is Hindi and we don’t have a localized version. So, other CMSs are way more popular.

Till: Really?

Adam: But it’s just to say that my team, we’re focused on a lot of different CMSs and it’s very interesting to see the place that WordPress fits in that ecosystem, in the world of CMSs, from a perspective being inside Google, it’s very different. I think working inside WordPress, it seems like it’s the whole world, I guess that every CMS is probably like that. Drupal’s definitely like that. I recently attended DrupalCon and it seemed like every everything in the world must be developed with Drupal.

Carl: But yeah, the language thing is interesting. I do consulting for Global Voices and they’re in like 60 different languages and some of them, there’s no font support. We have to have a custom font just to display the language, so it’s definitely … I think it’s always something to strive for, but there are a lot more languages than people think of in the world. So, even if WordPress supports a lot of them, there’s an insane amount. And if you consider sub dialects and things like that as well, so it’s wild. So, I can definitely believe that there are regions where WordPress doesn’t penetrate just for language reasons.

Zach: Well, then there are other areas where people are language specific, so they don’t want to use WordPress because it’s PHP and their shop uses .net for everything. So they end up on something…

Adam: Yeah. It is fascinating to like, I’m particularly interested in publishing, the publishing industry. Here, at least in the US, everyone’s switched over. There’s still print publications, but a lot of people have moved online or they’ve figured out, or they’re trying to figure out how to monetize. A lot of places in the world, they’re still way more in the print era and they haven’t switched to online and WordPress could play a big role in making that a better experience or having, a good approach to monetization, for example, which is a real challenge moving into the online world for publishers.

Yeah, but back to the whole performance thing, that’s what I’m here to talk about. I think, I mean, our team again, is super focused on making WordPress core more performant, which there is a lot of room for improvement and things that core can do, but a big realization of mine in the last couple years is that it’s not just core, it’s the whole ecosystem that needs to evolve around performance. And so certainly, WooCommerce would fall into this category of what is the performance characteristic of sites that have WooCommerce running and can that be improved?

Till: Do you mean that its plugins and themes taking performance more seriously as opposed to just here’s the experience?

Adam: Exactly.

Till: Yeah, that it’s core’s already pretty.

Adam: Yeah. It’s not even just taking it more seriously. I feel like we have the opposite incentives in WordPress right now. As a plugin developer, your incentive is to add as many features as possible to your plugin to make it look better than the other plugins to compete with the other plugins that offer similar features. And in general, that’s the opposite of performance. Adding more and more stuff is going to generally degrade your performance. So, I think the incentives right now, performance is not necessarily something that we would put on a feature list for your plugin. I hope that changes to where, plugins are touting their performance.

Carl: Yeah. Till definitely does that. Everything is about performance on his marketing.

Till: Yeah, it’s definitely my life. Someone was recently bringing this up somewhere, I don’t know if this was a ticket or it was the performance meeting, but someone said to have some kind of suite and you know when you have a plugin boiler plate where you start up with a class and some readmes and all of those things, to also have some basic tests in there as the plugin node and then the next thing is that there is something automated,I think it was Blackfire and they have some kind of sponsorship with the performance team to automatically run your plug-in through.

How does this perform on the node concurrent? How does it work with a persistent object cache, without an object cash? How many queries are being fired? And you can keep an eye on how much extra load does this add on top of regular WordPress requests, where you only have WordPress as a baseline with, let’s say, 30 queries, I think it is, to load WordPress. I’m not sure, don’t quote me on that. But then how much does the plugin add and just have interesting out of the books test and benchmarks.

Of course, it would only apply to new plugins unfortunately, if they use even this boilerplate. But I thought it was an interesting idea to make it a bit more approachable and easier for developers.to easily run a test and see how bad their plugin performs, because that’s usually the case.

Adam: Yeah. I think there is a group trying to build like a whole testing environment that you can run as part of like your continuous integration. So, you’d run it on every commit or every time you do a merge and the idea would be to have some sort of standard environment that developers could use. They could add it to their workflows while they’re building their plugins. And especially to be able to catch, if you’ve done something that’s really degrading the performance, it is super challenging, like what you talked about, like how many queries and how quickly does the plugin load? That’s one component, that is important to measure. Then there’s all the stuff that you might be doing on the front end. That’s a whole nother thing, to try to figure out how to measure that.

The Gutenberg Project has that, which is pretty cool. They do a performance analysis of the editor itself and how quickly it loads. And they test a very large post and then they test the typing speed. Like how many milliseconds does it take to respond and they do that on merges. And so it’s not actually on every commit, but at least they’re able to go back if they do see a degradation performance and figure out which code change actually introduced that problem. So, more stuff like that in the community would be great if we can build out that tooling and make it relatively straightforward for people to add it to their projects.

Till: And maybe even, I’m just thinking out loud here, how to make the ecosystem better, because I guess that’s what we’re talking about here. But if someone like WooCommerce would enforce this as a, you have to pass certain tests or like five KPIs, I’m going to say. If you want to be listed in the WooCommerce plugin repository or be associated, or have like a little check mark. I don’t know when it was like early 2000 of like HTML4 validated and CSS, no errors, these little badges, but actually have this again on plugins, but this is performance tested and this is front and back end, this is a good experience for all the users. The end user, but also the actual site maintainers, that we need to push it through these channels and be approved. I’m doing air quotes here because otherwise, who’s going to do it? So much extra effort.

Zach: I think in order for us to even get to that point though, where we enforce restrictions on the marketplace, the core plugin for WooCommerce itself needs to be at a point where it’s performant enough to warrant that. And for most sites, it is, for the vast majority, but for those who are on the top end of the marketplace, those in the top 5% of stores by volume and total sales, we’re looking at some issues that continuously are run into. Probably the largest one of which is inserting orders.

And we’ve talked about this before, but the reason why inserting orders is so difficult is because there’s this minimum of 50 pieces of post meta per order that get inserted and you can’t bulk insert post meta. You can only insert post meta one call at a time. So, you it’s add post meta rather than add post metas. We could consolidate it down to a single database call for the entire order, if there was a way to bulk add post meta. But instead we’re adding 50 pieces of post meta as 50 separate queries on a site that is potentially doing 50 orders, just to keep the math simple here, 50 orders a minute. So, that’s 50 orders a minute at 50 pieces of post meta, that’s what? 2,500 post meta inserts.

Adam: And you don’t even have transactions, right?

Zach: Right.

Adam: You don’t have database transactions either, right?

Till: It sounds like something for Johnny Harris to tackle. I don’t know if he’s listening to this.

Adam: Is that literally 50 separate rights?

Zach: Correct.

Adam: Wow. Yeah, that’s news to me. Is there an open track ticket to add this that’s like 10 years old?

Zach: Probably. I know that it’s been brought up many times in the past. This is why they’re moving to custom tables rather than just trying to get core to fix this.

Adam: Oh, WooCommerce is?

Zach: They’re going to move to a custom orders table and circumvent posts, just because of this one issue that is very easily fixed in core.

Adam: Maybe. I would never use the word easy when describing changes.

Zach: Well, true, true, but-

Adam: True. It’s just because sometimes, you think something on the face of it is easy, but then when you get into it, you realize, “Oh, this is going to break this other thing.” Or there’s a lot of unintended consequences that can easily occur in core.

Zach: Let me rephrase, it may not be easy to fix in core, but it should be fixed.

Adam: It should be, it should be prioritized

Till: If it’s just an addition.

Adam: Yeah, like you said, Johnny would be perfect for this, so as soon as he gets back, we’ll test him.

Zach: But that’s the biggest bottleneck in large stores is inserting these orders. Because when balloon to 2,500 inserts a minute on a database that’s not meant to have that level of traffic, your only option at that point is either throw more hardware on it if you’re in traditional hosting environments or to use a cloud-based, serverless hosting environment where you’re not worried about that.

Carl: I mean, as the serverless guy, you need the database. I mean now, Aurora-2 is out, so that’s okay but before that it was you couldn’t actually scale the database side that fast. So, even if you need a huge write capacity on demand, scaling a database on demand was not something that was really feasible. So, you would just crater your database or if you’re with AWS run out of IOPS, basically, and then your database craters too. And there’s a lot of things that can go wrong. So usually, doing that many writes is just not a good solution, but it’s always been the WordPress way. It’s like, “Oh, there’s a problem. Just throw more hardware. Can’t go wrong with more hardware.”

Till: I guess this is why the performance team now exists and Google is hiring people because you can make things more efficient. We don’t have to just throw more hardware at it, whether this is a more powerful iPhone 97 with more compute, that can render 12 megabytes of JavaScript and CSS, just making things a bit, I think slimmer, and more efficient.

Zach: Well, and the thing is we should never get to a point where a company needs to throw $24,000 a month at hosting just to make WooCommerce work well. There may be some actual real world reference there, but I can’t tell you who.

Carl: But there’s also a limit. This is something me and Till are much more understanding of, but WordPress doesn’t really scale horizontally that well, so there’s a limit. As far as I know, we haven’t invented like 500 core CPUs yet, so there’s only so much hardware you can actually throw at it and then you’ve more or less reached how far you can go.

And that’s also one of the challenges with performance, that I think the performance team is a bit more closer to achieving that one, than helping WordPress scale horizontally. But that’s also a challenge because that’s another way to throw hardware at the problem and it’s not even one that you can do with WordPress that well right now.

Thanks to our Pod Friends PeachPay and Yoast SEO

Till: And it’s also a little bit tricky. So Adam, I always see him every week in the weekly meeting, and if I understand that’s correct and I don’t pay too close to attention to the image component, it’s not my life, but do we optimize images and image delivery and formats for 99% of the word sites or 99.9%? Or do we optimize how WordPress works internally,and so on for the 0.01% of sites that are actually high traffic?

I think this is always a good trade off and definitely so far, the performance team, all of us have just been focusing on how can we make the biggest impact to the most sites or the most end users visiting WordPress websites? Whereas if someone has $24,000 a month to make WooCommerce work, too bad. I hope to make enough revenue to offset that, it sounds like it, because reaching the masses and making WordPress a better experience for everyone, it usually seems to take the preference or has a higher priority.

Carl: I think that’s probably the right strategy.

Adam: Yeah, I think there’s a lot of add-on benefits, like all the work that we’re doing. You’re exactly right and the WordPress philosophy pushes us in this direction to consider the changes are going to have the benefits for everyone. For the vast majority of our users, that’s really what WordPress core is focused on. And so, yeah, something like enabling WebP images by default is a no brainer. It’s like, suddenly your images are 30% smaller and you don’t have to do anything and everyone gets a faster website.

I think the other changes, like the small caching improvements or fixing queries that are cutting milliseconds off, 10 milliseconds off, they still are really significant for the wide ecosystem and they help the enterprise level eCommerce sites that have all the resources and problems like the enterprise people bring up, like the fact that you can’t write multiple meta fields at once, maybe that’s not a common use case for small sites, but at the same time, if we fixed that issue, it would also benefit those smaller sites because maybe they’re writing three metas at the same time, but they’re still doing it. They’re still just doing it on a smaller scale.

So, I always felt like that at 10up, we really pushed WordPress to the limits, building these sites that could handle millions of visitors a day. I felt like the work, that pushing WordPress limits, we brought a lot of questions and problems back to core that eventually hopefully will make core better. It’s not always an immediate thing, but I do feel like both tiers benefit from each other.

Carl: Yeah, the indexing one’s a good one.

Adam: Which one?

Carl: Well, it’s stuck now, but the adding indexes to the table, I feel like that’s a good candidate for what you were talking about because it’s good for 99% of people and it’s especially good for that 0.1% and so I’m hoping to see it unstuck.

Till: It’s fascinating, that one.

Carl: That’s a good example of that. That’s a good example of the tickets you were talking about, where you can actually target things that affect both sides.

Till: Carl is talking about a guy called Ollie Jones, I think. I think that’s his name. And he’s made a plugin called super longs slug, WordPress performance Index Adder or something like that. And you can add indices or indexes to your MySQL tables to make the lookups of, “Hey, give me the post meta for this post,” much faster. The trade of this, you have to use a bit more RAM to hold these indices in memory, but you can retrieve data a lot faster, which saves electricity, it’s good for the planet. It’s good for impatient people, want faster load times. It’s a good thing overall.

Till: But the issue right now is that we’re facing, is if you already have an existing table that has a lot of data, adding this in index might take five seconds, 20 seconds, 30 seconds, maybe 60 seconds. And because we’re in this PHP environment, we don’t have long running processes. If you’re on something like Emu, you have your 15 second limit sometimes even, or now it’s a bit higher, I know.

Carl: Yeah, it’s higher now.

Till: But we are limited by time. And we can’t just say, “Hey, run the process for 30 seconds or for two minutes,” to make everything faster. And these interesting constraints is like, how could we make WordPress faster for everybody? Cool, we have a solution, but we don’t quite know how to roll this out. And maybe we need to talk to hosting companies. It’s fascinating to solve this at scale and being so constrained that just by resources or just be using PHP itself.

Adam: I was going to guess exactly that as the sticking point, and I will say we are working on, as part of the imagery generation work, having long-running background process, API and WordPress, so that you could run some process that exceeds the PHP limits and it would run it in batches. So, it’s possible that in the future we’ll have a better way of doing this.

I remember another ticket where we were trying to increase the size of the field that WordPress uses to store the password. WordPress passwords, the field was like 24 characters. Literally the length of the data, maybe it’s 50 characters, but we wanted to make it like 255 or some much larger length. And the big challenge was we can’t just change this database field because people have a certain percentage of people.

Their upgrades will fail when we try to do this because their database is too large and PHP will time out and then their upgrade will die. And we don’t have a solution for that really. We have a really hard time changing the database schema. So, that’s a known issue, I guess and we need to solve it by having some sort of approach where there’s a way to have something that’s going to take minutes or maybe hours to complete.

Till: Yeah and maybe that is talking to hosting companies and getting WPCLI a bit more adopted because often those processes have 15 minutes, 30 minute time out, or sometimes unlimited but then there would be a hosting provider implementation. I’m sure a lot of them are already supported, but even having a button in Bluehost that says, “Okay, apply the indices, indexes, for the whole site.”

Carl: Yeah, that’s what I was thinking. Maybe put in a health check, something in Health Check. If you can’t run it like for existing site, you put in a health check and then that could be a way to alert somebody, because that’s the point of help check, right?

Till: Yeah, that’s what we’re doing at the moment.

Carl: It’s to point out issues that you can resolve yourself or choose to resolve. So, that was always an angle I thought could be used for things like that, where you’re completely right, just applying that, blindly to everyone, you’re going to hit these edge cases that are just going to break. So, that’s a way to keep it safe. You make it default for the new databases and then add a warning or something.

Zach: It would be nice though.

Carl: But I agree, for the long-running processes, we need something, there’s all these background jobs, like Action Scheduler and things like that.

Adam: There’s all kinds of plugins, yeah.

Carl: Yeah, I feel like that’s something that makes sense in core more and more.

Zach: It would be really nice to see something like Action Scheduler in core that makes that really easy to do, that isn’t proprietary or owned by anybody, that belongs to core.

Adam: I’m not sure it’ll help us with database updates now that I’m thinking about it, but I still think we need to have it. It just points out the challenges of making changes in core, like I was saying before, it’s never easy because there’s always use cases that are outside the box.

I remember when I rewrote the revisions thing, the revisions UI has this slider where you can see, you can drag to go through all the different revisions of a post. We spent a lot of time making sure all the tick marks matched up with the number of revisions and that the slider stopped in exactly … all this complicated math and different browsers that do math differently. Then I remember right after we tested it with hundreds of revisions, like, “Oh, this will work. This is great.”

And then I remember right after released that version of WordPress, someone opened and took it, and it was like, basically the person was writing a novel in one post and they had tens of thousands of revisions, they had literally thousands of thousands of revisions and they were like, “The revision screen won’t load for me.” And I was like, “Okay, everything is possible in WordPress.”

Someone will use it in a way that you just did not predict. And so when you change things, it’s likely to break. Anytime you change behavior, unless it’s an additive thing where you’re adding some new thing, but it’s very difficult. It makes it really challenging to move forward, you know? But also fun, I guess, to try to change things and not break the internet.

Zach: Yeah, it’s interesting because if you look back, most of the changes that have happened in WordPress, they stem from somebody using WordPress in a way that we didn’t expect. And then core changes to support that use case and goes from blogging to more content management because that’s the direction the industry was going and people were using WordPress in a way that wasn’t exactly what was intended initially. I think that’s really cool, that as a platform, we continue to grow and adapt to these new emerging use cases.

Till: Where do you think it’s going to go? I notice this is slightly off topic, but if we move from our blogger sphere to content management and we’ve been kind of in that for the last decade, I’d say maybe, I don’t know, where do you think is going to go next Zach, VR?

Zach: I think the next step is full scale applications. We’ve already seen people do it, but where we get more into supporting these very custom workflows, custom applications built on top of WordPress core, and you look at some of the companies that have done this and actually built crazy JavaScript, single page applications on top of WordPress, using WordPress as the API and really more of a data store than anything. I think that’s a use case we’re going to see emerging more. And then headless, I think is a huge thing moving forward.

Till: I was thinking, you’re going to say crypto, but okay.

Carl: I remember when Matt was like, “I want WordPress to be the OS for the web.” When the rest API came out, I think that was like, you’d use the rest API. And then WordPress would be your data layer, which is what it is with headless.

Zach: Yeah. I think that’s the future as of right now, but yeah, we’re going to see more growth in what WordPress core can do and now with Gutenberg, we’ve got this layer for building modular front end. So what can happen from there? I’m not sure yet, but I think we’re just going to see growth in multiple directions. I think headless and single page applications built on top of WordPress are the future though.

Till: Hey Adam, so the WebP image format is like in the works, it’s been couple of months, it’s looking pretty good from the outside. Where do you think is the next area that you see over and over? What would be another interesting area to explore at making WordPress?

Adam: For performance?

Till: Yeah. Or just end user experience.

Adam: The other big one for me, other than images is script loading. When you do a lighthouse report on your webpage and you see, “Oh, what’s slow?” And often there’s a big image that’s the lowest thing. So, the images pop up as like, “Oh, we can make these smaller,” but then when you start to dig into it, you realize that part of the reason, the big image takes so long to load is there’s all this JavaScript that’s blocking, that prevents it from even starting to load. So, there’s the size of the image, but then there’s also the JavaScript. And then there’s also what’s the time to first bite. How quickly is the server responding? Those all play a part in that calculation.

But I do think the JavaScript is an issue for core right now. You can add the defer attribute to script tags, but not through our primary, like WP enqueue script API. So, I want to for sure add that there. And my thinking currently is to use similar to what next JS did, where instead of there is a ticket open, that’s been open for a decade or whatever, to add this, but instead of allowing developers to control attributes directly, literally control the attributes, which is sort of what you can do. Now you can add attributes to the tag. Instead, we would give them like a strategy.

So, you would say that the strategy that I want to use is defer and then core would actually add the tag, based on for example, if all the scripts that depended on you are also deferred because once you start deferring scripts, which to just to be clear, deferring means that when you add the defer tag, it allows the browser to execute the script later and therefore it becomes non-blocking.

So, this is a really great strategy. Right now in core, when we enqueue scripts, we have two options, header or footer, and generally header are things that we want to load immediately so that they load before the page is loaded. And footer are things that we don’t care about. Defer is a little more refined because it lets the browser essentially decide when it has the resources to spend loading that JavaScript. And in most cases, a lot of stuff can be deferred. A lot of, even stuff that we’re currently putting in the head, could probably be deferred. So, I want to build that API indirectly. So that also, not only is it directly part of our API, but the developers are faced with a choice. They have to come up with a strategy. What is my strategy for this script?

Another one that’s really interesting as a strategy is this thing called, I love the name, Partytown, but it’s a Java script library that takes typically for third party Java, like analytics, for example, and puts it in the worker, puts it in the worker part of the browser so that the script can run without interfering with the main thread. So, JavaScript is single threaded. And when I’m saying blocking, it’s all this factor that basically the browser has to load the JavaScript and execute it before it can do the next thing. And by putting JavaScript into the worker or by deferring it, we can allow the page to render more quickly, the browser to like display the page to the user and then the JavaScript can, can still load, but not in a blocking way.

So, that’s the biggest one for me right now, is figuring out how to land that and also use it in core, there’s a few scripts that we can actually use it for already in core that I know, like the commenting script is a good example. I think the embed script’s another one, where it could easily just be deferred, but that’s another one where we’ll add the API. That doesn’t change much. We still have to get developers to start to use this new API, but putting it in into core and having an API, direct API for it, will at least tell developers they need to think about it.

Carl: Yeah, for sure and not leave it to the realm of the optimization plugins, like the speed plugins to do it.

Adam: Right, right, where they’ll add it for you. It’s very difficult to automate too, there’ve been a lot of attempts, even by browser the browser itself, to try to automatically defer scripts. But because of the way … for example, you can have a script that you enqueue and then right below it, you can have an inline script that depends on something that was in that enqueued script. And if you defer the enqueued script, then the inline thing won’t work. So, it’s hard to automate in most cases. So, developers need to consciously make a choice and then make sure-

Carl: Yeah, you can’t just blindly put defer on everything.

Adam: Yeah, or like everything that’s in the footer should be deferred. That sounds great, but it’s not quite that simple. And especially when you get into script dependencies, which we have, we do have a system for that already in core. That’s why I want to add it as a strategy and not as something that developers have to pick out, because it can even depend if another script enqueues later, and it depends on your script, that you’ve enqueued, let’s say you make a popular library and then you enqueue it, if something else depends on it, then if it’s not deferred, then you also can’t defer. So, there’s these rules, they’re very clear rules you can follow as a platform to get the correct behavior, but by giving developers a strategy option. And it’s also very future proof because then if new strategies emerge, or new approaches emerge, we can add them in there.

I’ve seen some stuff where people are adding both async and defer to the same script tag, which is not a thing, async and defer are one or the other and in general, async is not really recommended at this point. But I think it’s just hard as a developer, to keep up with what am I supposed to put here? What is the best thing to put here? So, I’d rather have it be something that core helps developers with. But I think that one could be huge.

Again, landing in core doesn’t make it happen immediately. But if we can get the community to start thinking about it and get more and more scripts deferred, I think that’s JavaScript enqueuing in WordPress is a huge problem. Some of it’s just people enqueuing, jQuery to handle one click on a form, and they’re, and they’re queuing all of jQuery in the head, but some of it is more subtle stuff, just scripts that are too large, or could be parts of them could be deferred. So, there’s definitely a lot of room for improvement there as well.

Till: Maybe we need badges for themes as well, where anything that’s in the repository, we do lighthouse automated testing, or whatever it is. And you just get a performance go for your theme. And then people will find way to hack it and cheat.

Adam: That was part of the Tide project, right?

Till: Oh, I’m not familiar with that. What’s that?

Zach: Yeah, definitely. That was plugins mainly I believe, but …

Adam: Right, and it was static analysis. Yeah and I think it’s still a thing, but I don’t think they ever got any badging on the plugin direct. There’s a lot of controversy over that. And if you don’t have the right metric, it can penalize people, or people will start to try to game the system by writing to the metrics, so they’ll write their plugin so that it performs really well when you run the automated test.

Carl: I mean, that’s happening already. I’m not going to name any plugins, but there’s definitely performance plugins that basically rewrite the entire HTML of JS and everything just to game the web performance index.

Adam: Even the image formats, that happens with. There’s all these modern image formats like WebP, AVIF, JB, Excel. And a lot of them, they’re still developing them, particularly AVIF and JB Excel. And they’re all competing with each other to be the best, but the way that you test whether an image compression algorithm is good is using another algorithm that tests image quality. And there’s half a dozen of these, like DDSim, and there’s all these different tools that you use that will say, “Oh, is this image that I compressed, how different is it from the original image?”

But what happens is, over time, is the actual compression algorithms, the tools, they’re writing to be better at the tests, at the automated tests. So, they’re not gaming the system, but they’re writing to the test essentially, which is different than how actual people perceive the images.

Carl: It’s like college all over again. You do the exam for the teacher, not for actually learning anything.

Adam: Right, it is a little bit like that.

Till: But you couldn’t even test us. If you write an image format, you have to test it against something and you have to test it against repeatable algorithm. You can’t test it against human perception.

Adam: Well, you can. The human perception thing is you do like the Mechanical Turk where you hire a bunch of people and you sit them down with a bunch of images and you just have them sit there, comparing images and you can get a human score, which is different than what you get from an algorithm. The algorithms are all designed to approximate human perception, and there’s whole scientific papers about them.

But there’s different ones and different image formats will perform differently on different ones and certain image formats, it’s very technical, but they don’t do as well with certain types of color spaces or with darker images. But it’s very interesting and we’re the beneficiaries of it, because everyone’s just trying to make the images smaller and smaller. And ultimately we’re wind up with a format that looks great and is smaller than it was before.

Carl: And to be clear, compression is a really hard problem too. It’s like PhD, almost post … I forget what it’s called when year done your PhD. But it’s like there’s another level after that and-

Adam: Post-doctorate.

Carl: Post-doctorate, yeah. Postdoc, it’s like a postdoc level work, in computer science compression. It’s very, very hard work.

Till: Yeah, have a little story with this. We’ve been tinkering with relay around and Zstandard compression, which is a Facebook algorithm. I’m not sure what they actually use it for, I think it’s decompress is faster than it compresses, but both it’s stupid fast compared to GZ or all the other, alt tar, all the formats, not tar GZ, GZ and so on. So, Zstandard, LZ4, LZF, they’re all just quite fast at compressing and decompressing and they shifted towards, one end of spectrum. And what we saw is that with ZSTD, ZStandard, you can actually create a dictionary. So, what we started doing is we toggle a little flag and now we watch all the data that goes through our PHP API, everything that gets sent through and we build a dictionary of, hey, this is what we think normal data looks like.

And after a couple of minutes we say save and we have a tiny little file that instructs and alters the algorithm to be as efficient as possible for this data that goes through this particular site, which is like these tiny little micro optimizations. But then we saw is we get like 40% performance out of this for an individual site, which is … it sounds negligible, but if you have a million requests a minute or an hour, it doesn’t really matter. All of this is suddenly 40% more efficient as one part. I find it fascinating. To use the algorithms, I am clueless with it, but just being able to customize the data compression algorithm for this one site’s data structures and if they type Hindi, you said, or if they type German into the text field of English, all of these would handle it differently. Algorithms are interesting.

Adam: That’s so interesting. Yeah, I’ve never heard of that before that. So essentially, it’s building custom tokens based on your particular data set to do the compression.

Till: Yeah, I’m not super familiar with it yet. We just test it and we see that we see really good performance results and we’re running into the same issue. Now, we do have to run a long-running process or can we avoid this? We solve this at the PHP extension level, which just fairly easy, but just for me, oh,how do I get a snapshot of the data? And then again, doing all of this at scale, that it works for every single site without crashing it and making sure that the dictionaries are lowered efficiency. I like the pain of maybe trying to make things faster. When you talk about your JavaScript optimizations for core, I’m like, “I’m so sorry already the pain that you will go through, of trying to make this work for everybody.” Including the lady who writes a novel in a WordPress post.

Yeah, I think there’s a lot more performance stuff on the table for us, for just people who have a bit of a sadistic side and like that. So, maybe like Johnny too, who is trying to reduce the core queries and just digging into these messy, old parts of core to make them faster. I think there’s a lot more on the table for us to improve without having to add new things and building new features. It’s more, how can we get another 20%, 30% here and there and reducing it because all of this is good. It’s less resources being used. You can either serve more traffic, have smaller servers, less electricity. It’s a win-win and then the faster end result. I’m all for it, it’s my kink.

Carl: Can’t say I have that one. I’m just trying to make it run on … I have my own set of masochistic ways of trying to get WordPress to run on stuff that it wasn’t designed to run on. So, that’s my problem.

Adam: Do you have it running on your refrigerator?

Carl: No, I just had it running on Lambda, on the hub functions. So, that’s interesting problem too, but me and Till discover a lot of scaling issues. There’s just a lot, I mean, it’s a legacy application, you talked about how it’s hard to make changes, but that’s kind of like the classic legacy application. It’s 20 years old, making changes is not something you can just blatantly do, you have to be a bit more targeted? A bit like a scalpel, like a surgeon. Surgical, that’s what I was trying to say.

Till: Yeah, the issues that I see Carl run into is the same thing where WordPress core has been doing things a certain way. You can just write to the WP content directory, for example, and then you have a theme, generating their own CSS to game the light speed or the page test, or page test results. But if you’re on serverless or any kind of ephemeral file system, like Heroku, Platform.sh. You can’t really write to this. And if you do, it’s just discarded after a couple of minutes, or sometimes you can’t even do it. Just having to change the path of where themes write on, there’s so much legacy force behind, like you have one server, PHP and WordPress [inaudible 00:48:41] it and trying to make this work in any kind of other more modern environment. It’s a little bit of a pain, or a lot.

Carl: It’s definitely something. There’s been times where I’m like, what am I doing here? Clearly, you’re fighting a bit against the grain, but I think it’s important. It highlights problems, like I said, there’s only so much you can do with one server eventually. You know, even if you’re paying them $24,000 a month, like Zach said. It’s just, Google doesn’t run on one server, Shopify doesn’t run on one server. WordPress should be able to run on multiple server. I know it does, but it’s very obvious if you spend any sort of time on it, that there is very big structural issues, architectural issues to doing that, and just whittling away at it. True performance is a good way to do it and it benefits the entire community.

Zach: Yeah. There are other areas though, where you can very efficiently run WordPress across a cluster of web heads, but the problem is clustering the data layer.

Adam: That’s what I was going to ask about. It seems like when you were talking about the writing, like the MySQL is still the big limitation, especially if you’re writing.

Carl: Or object caching.

Adam: Reading is easier, right.

Carl: Even with Till, I ran into a problem, as if WordPress works only if the object cache is local. Because the minute it’s not local, you add basically one or two milliseconds to your readus query, which sounds trivial. The problem is that because WordPress, this goes back to the initial discussion where we were talking about developers not necessarily be super efficient. I ran into site making 2,000 object cache queries in one request.

Till: Per page load.

Carl: Per page load. So at one, two MS per readus request, you’re still adding two seconds, which is insane.

Zach: Yeah, would be.

Till: Yeah, it sounds like you need to keep that data locally in PHP memory, if only there was a product for that, or any kind of solution.

Carl: Yeah, if there’s only a product that did that, if there was only a PHP extension that was developed to do that.

Zach: Well, and so clustering web heads, easy load balancing across web heads, we’ve done this for years now. This is not a difficult solution. Now, in recent years, we’ve had new options on the database front, particularly MariaDB and MariaDB’s Galera clusters. That eliminates one of the issues that I’ve had in the past with trying to cluster the database side of WordPress, which is you basically data collisions, read and write collisions, where let’s say you’re on a WooCommerce site and the customer checks out. And then in mid checkout, you change what database server they’re connected to and suddenly their order doesn’t exist anymore.

So, they get to the confirmation page and they get a, “We’re sorry, this order doesn’t exist error.” So a Galera Cluster isn’t a master/slave type environment. It isn’t a read replica type environment. It is an environment where once one server gets that data as new data, all of them lock until the transaction completes across the cluster and that eliminates that problem completely. So, you’re able to cluster the database, have a lot more transactions happening because you have more hardware, but you’re still running into eventually you’re going to hit that limit of how much bare metal I can throw at this.

There are solutions that are starting to help make this easier, but it’s still not perfect and so, as we really dig in and we focus more and more on performance and on making WordPress core more performant, making WooCommerce more performant with some of these things like custom tables, as that develops, who knows how long it’s going to be until we see that and who knows how long it’s going to be really, before that’s the default, rather than an opt-in?

But as we start to see these things, we’ll start to see more performant WordPress sites. I really appreciate the work that the performance team is doing to try and push the platform forward as a whole. So, we’re at a point where we need to start to wrap up here, but I want to ask Adam, where can people find you? Where can people find the performance team?

Adam: Yeah. So performance team, we have our own blog now on the Make Core WordPress blog. We have our weekly meeting in the performance. I think it’s core performance channel in Slack. I’m on Twitter @roundearth. R-O-U-N-D-E-A-R-T-H, my DMs are open. So yeah, and also we generally post, the performance team generally post, minutes of our meetings to the main WordPress core blog, so you can follow along there.

Till: Sorry. Why is your username Round Earth? Are you a recovering earther or space is fake or how did that come to be?

Adam: It’s the name of my farm? So, my other job as a farmer when I’m not a web developer. So, it’s the name of the farm and the name, yeah, it is a little bit of a play on the flat Earth thing. Of course the Earth isn’t really around its spherical. Round is actually not the right dimensions. But yeah, my farm used to be called Earthbound, which was also the name of my web consulting business.

Carl: Also a great RPG for Super Nintendo.

Adam: Yeah, I know. I got a lot of traffic from that actually, but there’s also a big farm out in California called Round Earth and they really wanted us to change our name, so we agreed to do that. They’re Round Earth Farm. I still have roundearth.com, but they were Earthbound, so I was Earthbound. I had to change it. And now Round Earth was what we came up with. Seemed like a good handle at the time. Obviously, I guess my name wasn’t available when I signed up for Twitter

Carl: Earthbound?

Adam: No, Adam. Adam Silverstein.

Carl: Oh, Okay. I thought, I mean, Earthbound seemed also like a unlikely thing to be available when-

Adam: Probably unlikely, but I have earthbound.com, so there you go.

Carl: My goodness, that is actually worth probably a decent amount of change, because Earthbound is-

Zach: Oh, I can imagine.

Carl: Earthbound’s a really … if you’re not an old geezer like us, but if you play … it’s a very well known RPG for Super Nintendo and it’s got a huge fan following.

Adam: Yeah, no, I get traffic from that. Especially anytime they have a new release or something about it, people will come to my website looking for it. I can see the searches that people wind up on my site. Although less so these days, because I haven’t put anything new on the site in years. So, it’s probably deprecated on Google search at this point. Yeah, there’s also a clothing company called Earthbound, so I get those too. It’s funny when you have a name like that and suddenly your website … I used to get phone calls from people who were trying to return their clothes to them.

Zach: That’s hilarious.

Till: Thanks so much for coming on, Adam.

Carl: Yeah. Thank you so much for coming.

Adam: Yeah, it was really fun.