WooCommerce Custom Order Tables, Databases and Performance

Zach: Hey everybody. This is Zach. I’m here with Carl and Till for our first Woo DevChat.

Carl: Ooh.

Zach: I know. It’s crazy. This just started with a little conversation about, well, performance that we did together as a podcast episode. And Bob liked it so much that he decided he wanted us to do a whole series. And here we are.

Carl: Here we are.

Zach: Right? So, well, I guess let’s just jump in. There’s been some interesting announcements for WooCommerce lately, specifically around the topic of custom tables.

Carl: Yes. A topic you know intimately, Zach.

Zach: I know nothing about it. I didn’t help build the first plugin for custom tables for orders. Yeah. I’ve been doing custom tables in WooCommerce for what? It’s five years now since 2017.

Carl: Yeah. So, I mean, it’s a long time coming. So, I think, a lot of developers or people working on large WooCommerce sites are probably sighing a bit in relief at the prospect of this happening, because that was always like we were talking a bit before the show started, me and Till. The database side has always been one of the rough edges of WooCommerce. And the lack of custom tables especially is one.

I think we should probably just explain a bit what it is for people that aren’t necessarily diving into commerce all the time. You want to give a bit of rundowns, Zach?

Zach: Yeah. So, back in 2017, WooCommerce 3.0 came out. And a lot of people will remember that release because a lot of things broke. It’s probably the most memorable part of that release, but the reason they broke is because they introduced these CRUD classes. CRUD stands for create, read, update and delete. And these classes were created to abstract away database access. That was the whole point of putting them in place, so that people could change the data store for the various object types in WooCommerce.

So, the point of that was to make it, so that we could set up dedicated tables. And back then, a company called Liquid Web had reached out to me and wanted to explore WooCommerce performance for a dedicated WooCommerce posting offering. And that’s how I started my former agency, was with a project to do that based on these new CRUD classes. So, the capability has been there for a long time. There have been a couple of problems with getting there. One of them was that even WooCommerce itself had not implemented the CRUD classes everywhere.

Carl: Yeah. So, they gave you the tools, but they didn’t use their own tools.

Zach: Right. So, there were some things like reporting, that when you put things in a custom table, just completely broke.

Till: Like broke or just didn’t show up in the reporting?

Zach: No, they broke. They started throwing errors and we got it to the point where they just didn’t show up, which was preferable, but it was a big deal.

Carl: Because, WooCommerce without reporting is like flying in the dark.

Till: Are they adding all the orders or the products? What are they putting extra acting into dedicated tables?

Zach: So, the first thing they’re going to put in a dedicated table is orders.

Carl: Yeah. It’s orders.

Zach: And that’s the big thing. Orders are one of the two points that really needed custom tables. That’s why that’s where we started back in 2017. The other problem I mentioned that WooCommerce wasn’t using these CRUD classes. The other problem is that a number of plugin developers don’t use the CRUD classes. And so, if they’re reaching directly into post meta, rather than using these CRUD classes, the second you move all that stuff, everything breaks.

Carl: I mean, that’s the problem when you transition architecture like that, it takes a while for third party to catch up, especially when you don’t have any sort of dependency. They can’t just say, I mean, even with dependencies, I mean, it still works, I guess. That’s always been the hard part with transitioning.

Till: Have they been deprecating these, let’s say, old… I guess if people directly access the post meta tables or object’s old functions?

Carl: Yeah. That’s a problem. As long as everything’s in post meta, you can just use classic WordPress.

Zach: Well, and that’s the problem they’d have to deprecate parts of WordPress in order to stop people from using it. Right?

Till: Or hook into specific functions, check if its WooCommerce related key. Okay, that’s a big challenge.

Carl: Well, That’s going to force a lot of these plugins to migrate either their code or just start using those tables. I mean, hopefully their code and not just start doing database queries directly on those tables. But I think, that forces things, because now you can’t rely on Vanilla WordPress code to do things. So, it puts a bit of fire under those developers to migrate things. But yeah, overall I think it’s a really good change, because I think performance is always a topic. It’s probably going to be a recurring topic with WooCommerce in general, but I think a lot of struggles and Zach can maybe also corroborate. But, one of the problems with using WordPress tables with WooCommerce is scaling. When I did consulting work on WooCommerce sites, my first stop was always the database.

Till: And do what? Adding indices?

Carl: Well, yeah. So, indices is a big one and I find that really hard to believe in 2022 that you have to use a plugin. What was the name of the plugin?

Till: Oh, yeah. So, last night our shared friend Patrick Leblanc, Index WP MySQL For Speed.

Carl: Yeah. Okay. That’s a mouthful.

Till: Oliver Jones and Rick James. Oh, you know what I want to say after Rick James, if you know me.

Carl: But that plugin, if you’re struggling with, or you’re not sure if your database is up to snuff for your WooCommerce install, I highly recommend, it’s a really lightweight plugin. There is stuff that it does, even if you keep it installed. So, you might want to evaluate that, but at worst you can just install it. It’ll tell you what the index is you should create for your database and then you can remove it.

Till: And it seems to only have a 1000-ish, less than 2000 installs. That’s my guess here, which I’m quite surprised by.

Carl: Well, I mean, if you don’t have to keep it installed, that could explain. But also, I find that’s one of the things and that’s why the custom order table is such a big deal. Because, before that, you really had to rely on those indexes. If you had to do any performance adjustment at the database level, I install Query Monitor. And then, I look at what are the slow queries. And then I build indexes, but that only solves part of the problem, because the larger problem which custom tables solves is that before that, you would have all sorts of different information that WooCommerce had to piece together. And they were always all in the post meta table.

Till: Yeah. So, he was talking about database optimization, which maybe to Hypernodes or Geeks is something very familiar. And we do it all the time. But, for the average WordPress maintainer or a store owner, that’s… I would say, way far outside their comfort zone. Even for me too, if you would add indices on the production table and you don’t know what happens. When you’ve managed databases like DigitalOcean, Heroku, I’m not sure what AWS does, but there’s several providers who will provide these managed databases that your WordPress app and connect so. They already show you and keep monitoring slow queries which is quite nice, easy to have an insight effect. Where can you improve? I don’t know, auto product search in your WordPress admin, if it’s slow, you know where to add some indices.

Zach: Yeah. Well, and I use New Relic for most of my application performance monitoring.

Till: But again, it’s such an advanced tool. And I wonder if a part of the WordPress performance team, the last two months or something like that, I’m not sure when we start, maybe November, December, and making our all of this a little bit more accessible or a lot more accessible and easier to at least know what’s going on by some health checks, or maybe, I don’t know if we can have good suggestions of like, “Hey, put an index on these two composite keys or something.”

Carl: I mean, I feel like it should be at least covered by health check. Again, in 2022, it makes sense that WordPress should be trying at least to add these indexes when it creates the database.

Till: Yeah. Or recommend adding that after the fact, because we have all these existing science.

Carl: Yes, exactly. True help check.

Till: Maybe even keeping track of particular slow queries, if it repeats and pointing it out. Do something about it. Talk to your hosting company about it.

Carl: Yeah. Yeah. Because, that’s a lot of what I did, at least for WooCommerce. It was 90% of the time, it was the database. And then, yeah. And like you said, Till, it’s so out of scope, you have to hire people like Zach or me that deal with this because it’s so out of scope of not just business owners, but also like most developers. And I say this, as somebody that would never dare call himself a DBA, because a database administrator, that’s what DBA stands for, is like, there’s a lot to know about databases.

And you can make your entire career on databases. And I’m definitely not at that level at all, but I think, without being at that level, there’s still a gap between people that know where generally you should be looking for database performance issues at a high level and people that are completely out of their debt.

And that’s a lot of repress developers because honestly you just deal with WordPress and you assume that WordPress is doing the right things for you. And it’s just not something that you consider. And as somebody that writes software that’s trying to scale websites, it’s hard to know necessarily where it’s going to break necessarily. So, I mean, it’s not necessarily the developer’s fault either, but it’s just like those things are just not well understood. And it’s a good change, that’s why a lot of more intense WooCommerce developers are really stoked about this custom order table, because it’s the beginning of a big change for the performance of these WooCommerce sites.

Because before that, like Zach said, you basically had to rely on consultants or hosting providers that paid consultants to write these custom plugins for them and write their platform to take advantage of that. So, it’s like, that’s why a lot of people are on Nexus. That’s what Liquid Web, WooCommerce host thing is now, but that’s why a lot of people are Nexcess because Nexcess did a lot of the work.

And now, it’s really good, because now it’s bringing back some of the work. Obviously, it sucks a bit for Zach, because they’re not really taking his implementation, but the overall is good. The overall is good because it’s bringing that back to everyone. And that’s going to be a huge boom to a lot of shop owners and a lot of developers that have to maintain those shops.

Zach: And what Nexcess and Liquid Web are using today is not what we built in 2017. It’s built off of it. But, I can’t say that I wrote the code. I didn’t. But, I was involved in the project, but what this gives us, there are two things. One, you mentioned indices. I know this is something that WooCommerce, the core team had implemented.

Carl: You go. You brought it up, you do it.

Till: Wow. How do you even start? Maybe you should just Google it. If anybody’s listening wants to know what it is, just Google it.

Carl: No, I’ll give a sample. I’ll save you. I’ll save you. I’ll give. So, the idea with indexes is, so with databases, you’re storing data obviously in your database and you want to retrieve that data. So, you’re like, you’re doing search queries and things like that. But, for the database to be able to find things fast, it uses indexes. And what the index does is you can say, “Okay, I’m going to be often searching for a combination of…” For example, you could do an index on post, the published date. So, you know that you’re going to search often for how often posts are published. So, you can say to the database, “I want to create an index on post date.”

And what that tells the database is that it creates, and this is where it gets really complicated and completely out of scope, but the different types of indexes, how they’re built, I took a college course, I took actually two college courses just on that topic alone. So, that’s really out of scope, but basically, the database is going to create an index. And what that allows it to do is search for that information much drastically faster. And the problem with indexes is that the database can’t just say, “Okay, I’m going to index whatever you want.” One, indexes take space. Two, a database is a dumb storage device. It’s not really that dumb, but for intensive purposes, it’s dumb. You tell it to store things, how you want to store things, and then it does it for you, but it doesn’t really know how that data is related to each other. That’s the job of you as a developer.

To instruct a database, things that WordPress doesn’t use, but are common with database, for example, foreign keys or things like that. You can structure the data a bit. And that helps the database understand how the data that you’ve stored in it is related to one another. And the indexes really just helps you query that data faster, but it doesn’t know how that data is structured. So, it’s really your job as a developer to create those indexes. So, that’s the short 101 on indexes, but that’s why they’re important. And that’s why it falls a bit on the WooCommerce slash WordPress team to design them, because they know more than anyone how that data is structured, how it’s used. Yeah. And how often it’s query… What types of queries are run often, which ones are tend to be slow and could use an index. Because, like I said, the indexes takes space. And doesn’t really matter when you have a small database, but when you’re dealing with gigabytes or terabytes of data, your index can be quite big.

So, you can’t just index everything. So, that’s why it falls a bit on them to do that because they have the insight. Especially for WordPress, WordPress.com is the largest multisite. I think it’s still a multisite. So, it’s like the largest multi-site install. So, they have insight. They know what’s slow. They probably have indexes already. And that’s why it’s like, it would be a good thing for the larger community as a whole to have these indexes as well, and not rely necessarily on the good work that… I forgot who wrote the plugin. But, it’s not well documented. And if anything, if I knew better what to do, I would probably write a post about it, because I think it’s a really important topic. And it’s just an important thing you would reference very often. That’s why it falls more on the WordPress slash WooCommerce team to work with these indexes. Okay. I’m done.

Till: Hey, I’m going to derail this for a second here, because a few days ago, or two, three days ago, yesterday, I don’t know when. Probably also doesn’t matter. But recently, WordPress.com, if you go to it and the business and e-commerce plans now allow SFTP for direct file access and database access. So, I don’t know what they’re doing, but it seems like they’re moving away from their insane multisite installation. That would be my guess. I don’t have any insights. I don’t even know anybody there.

Carl: I mean, I haven’t talked to an Automattician in a few years because of COVID. So, I don’t have the insight, but I think it’s like, if you’re VIP or the higher paid tiers, you’re not on a multisite. I think, it’s the free.com stuff that’s on. I’m sure I’m going to get corrected online once this episode publishes. But, if I remember correctly, it’s the free blogs that are on a multisite.

Zach: I think it’s everything up until business is a multisite

Carl: Yeah. So, it’s not exactly like every.com site, but they still have a large, large, large multi-site installation. But yeah, if you have access to the database, I would go ahead and say that it’s probably, they were working on some docker setup. I don’t know if that’s what it is now, but yeah, I assume it’s a standalone isolated database, but they might have indexes. So, it might be interesting for you to connect to the database and look at what indexes they have on the tables. Because I don’t have a paid.com site. So, I can’t tell you.

Zach: Well, and so, core WooCommerce added indices in 2019 to products and orders if I remember correctly. Some of the work that was done to do that is in a repository, separate from WooCommerce in the WooCommerce organization called WooCommerce custom indexes, which is Mike Jolley and Rodrigo Primo doing that work. So, that was where a lot of their exploration came and where they wanted to implement indexes for everybody. But in some cases, those indexes at scale will actually slow the site down. So, moving to a custom table structure will help to eliminate some of these bottlenecks.

And I think it’s important to talk about how an order is created in WooCommerce. When an order is created, when somebody places an order on the site, it creates the order itself. And then, it has to add order meta to the meta table. And every order has a minimum of around 50 pieces of order meta. There is no way to bulk add meta in WordPress.

Till: Wait, so we have one order and each single order has 50 entries and a different table of all the meta?

Zach: Correct. And so, that’s what 51 minimum database calls create the order and then create all the order meta by calling the admeta function every time, 50 times. So, that’s 51 database calls to create one order, right? Minimum. And then, you do that over 20 simultaneous orders. You have a significant number over 1000 database calls happening, because it’s creating all of that meta. So, by flattening the table structure with their proposed table structure that they have on the blog post for this, which we will include in the show notes by flattening that as much as possible. So, there’s a wp_wc_order’s table. And that table will have most of the current core fields and all of the important meta keys. Then there’s going to be a order addresses table. And that will store the addresses associated with that order, both shipping and billing.

And then, there’s an order’s meta table similar to post meta, that will allow extensions to store data related to orders. So, by flattening it that much though, they make it, so that that create order call can mostly happen in three SQL queries. It looks like instead of 51.

Till: That’s pretty good.

Zach: Yeah, I’d say so. And we’ve run into this where on some of the larger sites that I’ve been involved with where the creation of orders simply due to the amount of traffic and the amount of simultaneous orders that are being processed has been the bottleneck that’s taken down a site, because that’s a lot of database queries to queue up. Right? And that’s primarily why the work on the custom order table plugin was so important, because it allowed those sites that had that particular scaling issue to circumvent it. And in those cases, the things that broke were worth keeping the site up. Right?

So, it’s nice that we’re going to have at some point in the future, all of this data in custom tables. I’m not sure what they’re targeting as of right now. But, they do have a GitHub project board. It looks like they are currently in some research spikes and they’re creating the scaffolding for the project. And then, they’ll be working on each of the various to-dos that are in the to-do column. It’s all in a GitHub project if you want to track it. We’ll link that as well. But, it’s linked from the blog post and the spike right now. Yeah. Finding a performance and a reliable approach to migrating data from posts and post meta, includes testing the migration approach.

Carl: Yeah. That’s definitely going to be one of the hard problems.

Zach: Yeah. That was one of the problems that we had when we built the custom order tables plugin for Liquid Web back in the day.

Thanks to our Pod Friends Yoast and WP Activity Log

Till: How is this going to work? Let’s say, you install WooCommerce. What is it? Six, five or seven, whatever the version is that has custom audit tables. How is this going to work with migrating all your own data and then testing if this works really all your plugins? Because there seems to be two steps. And then if it doesn’t work with the custom orders table with the new structure, if you migrate all the data over and, let’s say, you test it and you have three custom subscription plugins for WooCommerce from ThemeForest or something like that, and it doesn’t work. Can you then go back and say, “Hey, I got to run with the old post meta approach anymore.” Or is it like one way migration?

Zach: I don’t think it’s going to be an option.

Carl: I think that’s going to be part of the pro design struggles. Yeah.

Zach: If they make it optional again, like they did in 2017, if they make it optional, then people will continue to build things that break it.

Carl: Maybe they could go like, new installs, get it by default and then an optional migration for existing installations. But then, they have to maintain two things. Although, to be fair, that’s what their CRUD abstraction was meant to solve. So, that actually allows them that possibility. Because, I agree that’s a big one, because how you handle. Because, some people might already have moved away. Somebody might be using already a custom table plugin. Right? No, but that’s why the CRUD abstraction they did was so,

I think, important in retrospect is because if you rely on that, that probably will force a lot of migrations again from plugin authors because, now you might have to support three, let’s say, somebody already has a custom table plugin, somebody migrated to the WooCommerce custom tables or somebody hasn’t migrated and is still using post meta and that abstraction lets developers not care about it and simplifies their life. It’ll probably be much more interesting for them to use that than to use WordPress functions because then the WordPress functions only work in one of those cases and not the other two.

Zach: I have a feeling that what we’ll see is a number of releases where the migration is optional. And as they get feedback from people testing the optional migration, eventually they’ll move to including it in core as a schema update that happens for everyone during an upgrade. And they’ll use action scheduler, just like they do to schedule the database update in the background and have that happen. And migrate everything.

Carl: I mean that’s what they do for order creation. Do they use the action scheduler for that?

Zach: I don’t know if they do currently. I think it’s still just a whole bunch of add post meta calls.

Till: Do you just basically DDoS your database if you have a sale?

Zach: Yeah, pretty much. That is a big problem.

Till: Because you couldn’t queue it for concurrency, if you have five, I don’t know.

Carl: I mean, that’s like with YMIR there is a lot of tuning is around the database. Even though I can scale websites to handle thousands of people coming in, there’s nothing that exists on the database side to handle that. So, tuning the database becomes the more critical aspect. Like we discussed, this is the whole point of this episode is, WooCommerce database stuff is hard, especially once you start having big stores. Big stores with lots of SKUs cause issues even if they don’t have a lot of traffic because that’s what the index is and all the custom table work also helps eventually. Right now, it’s orders, but eventually they’ll have products and that’ll help those people.

Because right now, you have the same problem as orders with products, because let’s say you have a shop with 5,000 SKUs in it, all the metadata for those products is all stored as post meta. So, let’s say you want to request a product, you might need to do one crazy query or dozens of queries to get the information out.

And versus if that was all consolidated in a table or a set of tables really, because I assume there’ll always be a product meta table, at least two. But, that would simplify things. And in fact, most of the consulting work I did was more focused on the products than the shops with the large SKUs than the actual orders, because that affects a lot of a shop owner’s daily life. Because, if you’re looking for reporting, you want to go edit a product, you name it. That’s what somebody does much more lot on a daily basis, or if people are navigating your site to browse products, that’s what they’re looking at. The order happens once, but the products happen constantly.

So, that’s why I assume what if they’re successful with that, that’ll probably be the next thing they tackle, because in my experience that’s been the utter part that has really suffered in large e-commerce sites.

Zach: Yeah. And the interesting thing when you look at it is that these CRUD data stores that the data store classes enable, they touch every WooCommerce object type. So, products, orders, those are the big ones, right? But, there’s also customers and coupons and those also use CRUD glasses. So, every object type that WooCommerce handles. So orders, order line items, products, coupons, customers, customer downloads, payment tokens, and shipping zones all have CRUD objects.

Carl: I assume some of the popular extensions do as well, like Subscription.

Zach: Yeah. I would imagine so.

Carl: Because yeah, I remember I did a decent amount of work around subscriptions for customers because that one also buckles under the weight of that architecture.

Till: That’s how actually we started Object Cache Pro, because I had one customer using the free plugin many years ago and they had 20,000 subscriptions that would renew every Saturday morning as a food delivery service and renewing 20,000 subscriptions without using an object cache, it wouldn’t work, just everything would collapse. And then, they don’t know which were renewed, which weren’t renewed. And they think they had a… I won’t say 80, maybe it was an 80 gigabyte Redis cluster to handle that up to catching just to renew these amount of subscriptions. Of course, we’re talking about a massive, 20,000 subscriptions is a lot, that’s not your 50 or a hundred renewing every month.

Carl: Yeah. And a lot of these subscriptions have all this metadata with them. So, it’s not just like, oh, it’s 20,000 lines, but it’s like each subscription probably has, I forgot, 50 rows. Yeah. Yeah, exactly. It’s probably 50 rows total across multiple tables. And they’re querying, they’re doing four or five joints on the data on the same table and it’s always the same table.

Zach: And every subscription renewal creates an order, which we already talked about.

Carl: Yeah, exactly. So yeah, you have this cascading effect, so. Which is why you want to offload, that’s why plugins like Till’s Object Cache Pro is really critical, because at least for reading information, you don’t have to touch the database. So, that reduces the load on it. And on top of being faster in the first place, but there’s something to be said about just making your database work less for that stuff. Because, if it’s doing other things at the same time, like Till said, with 20,000 renewals at the same time, it’s a spike. So, anything that you can do to alleviate the spike is helpful. But, that’s why it’s so hard to plan for these things. If you’re a plugin developer, it’s so hard to plan for these things because I’m sure people that design the word commerce subscriptions didn’t think that, “Oh wait, obviously somebody’s going to do 20,000 renewals on a Saturday morning, every week or every month.” Or something like that.

Till: Yeah. And it’s very hard to test. to have a plugin developer use a position Object Cache Pro to developing it, to see if it works, flush it. Does it work with transient and persistent cache data or just data. But then saying, “Hey, what happens if you have a 1000 orders in five minutes to test that you, Carl are doing.” How do you simulate those? You need to write benchmark tools to test your plugin, that’s a big ask for little developer who wants to start making a plugin or just publishes something open source. Like I did or still do. Like, “Hey, I want to share this with the world, but how does it work at scale?” And someone will use it at scale and it’s fascinating and it’s complex.

Carl: Yeah. It’s complex. That’s why it’s easy to throw the developers under the bus. But, it’s really hard to think about those edge cases. And sometimes it’s the simplest way. It’s like a balance, because as a developer, you always hear about premature optimization. Do not premature optimize. And then you’re like, “Well, what if they’re doing 10,000 subscriptions at once? What do I do?” On one hand, it’s like, “Well, you should be able to handle that.” On the other hand you’re like, “Well, until somebody hits that problem, you should probably not care about it.”

Till: Also having the experience to prematurely optimize the right way. Sometimes, it’s a good way. If you think of some, let’s say WooCommerce itself, they have to do that, they have to think about these cases, and maybe also have to team and the experience to do that. But again, it’s too high of an ask.

For your average free open source, WordPress plugin developer, which is the core of it all, this is what we’re all here.

Carl: Yeah. Or it’s somebody managing a store or somebody that built a business around a store.

I’m thinking of our friend, Jillian. And he’s super qualified. And he knows probably a lot more about optimizing a WooCommerce site, but it took so much time, sweat, pain, experience to figure that out. There’s no environment that’s going to teach that to someone outside of experiencing that yourself. He’ll make tickets or things to improve things because he had to fix them. But yeah, I mean, you always run into issues so that it’s just like, that’s the problem. But, there’s always a few good things. And again, that’s why this episodes on the custom orders table is, because those are things that are good.

They’re good. They’re universally good. And they don’t require fundamental knowledge of these extreme scenarios to know that they’re good because they’re good fundamental design practices to have custom tables like this. It’s just because of how WooCommerce came to be. And that’s a problem. I think this is a good segue too, although we’re talking about WooCommerce, this isn’t a WooCommerce specific problem. A lot of large plugins that do plugin that build applications on top of WordPress have suffered this problem and have either already done this transition or will do it in the future. I can think of, I don’t know LearnDash, but if LearnDash does it already, but LearnDash is a big plugin that would, if they don’t do it already, we’ll do it.

But, I think something that’s really old that not everybody touches, but BuddyPress. BuddyPress had cut a lot of custom tables right out of the gate. And that was really useful because if BuddyPress had been built with just using WordPress table, they would’ve had to do this transition eventually because every major plugin that builds an application on top of WordPress hits that problem, because there’s a reason why this is how a lot of applications are designed outside WordPress. If you were building something like LearnDash or WooCommerce. I mean, e-commerce is a bit more tricky, there’s different… There’s entity attribute value which is what WordPress does. But anyways, that’s like a design paradigm for databases and Magento uses it. And stuff like that, something like post with post meta is something that is a known way to solve a specific problem, but has scaling issues.

And it’s used a lot, but it always runs into the same problem and it always ends up that you need to create these custom tables. If you build an e-commerce platform or a learning platform or a social network, you’re always going to do more custom tables than something like WordPress. Because in general, that’s what scales better, and that’s what makes the database perform better. So, that’s why it’s a huge milestone for WooCommerce to do it. But it’s not something that’s specific to WooCommerce. It’s really something that all the large, which I call application on top of WordPress plugin hit this problem.

Till: Maybe, what you’re saying is that premature optimization is actually very much appropriate when it comes to open source or workers development. Because if you have your own app, you don’t have to prematurely optimize and over architect things, you can’t just switch it. Hey, let’s take this week. Let’s refactor this, move it. But if you have a product or a plugin or piece of software that is used by many things, premature optimization could actually be quite beneficial.

Carl: I mean, it’s a pipe dream of mine, but I really wish that WordPress made it easier for developers to create custom tables. And then, we maybe would not end up in this scenario as often, because I talked about premature optimization, but I feel like custom tables doesn’t quite fall into premature optimization. One thing that you should really think about if you’re designing applications, that’s a lot of what I do for consulting now, is think about how… That’s why I did that whole nice little speech on indexing. When you’re building an application early on, it’s really worth your time to think about it. Obviously, you can’t think about every edge case and things like that, but structuring your data is really important, because as we can see with the custom orders change, it is very painful to come back later and re-architect things.

Not that you can’t, but it’s much more painful than if you had just done a bit of work ahead of time and been like, maybe I should use a custom table here. There’s a lot of stuff I shouldn’t be doing. Here’s a simple witness test. If you have to add, let’s say I’m going to be, Zach and Till can decide. But if you have to add five post metas to whatever you’re doing, whatever information you have to store for your plugin, for every entry, you need a custom table. Well, you should probably have a custom table because I think when you’re starting to add a code smell, would be like, I have to add a lot of custom post meta to just track all this information.

Then maybe, I need a custom table. Maybe, this is what I should be doing. I should invest in that now and not later when it’s not five post meta, but it’s like 50 post meta that you’re adding each time. And then, people are adding custom stuff to it. And those are like, that is not premature optimization. It’s really hard. It’s always hard to know what is premature optimization and what isn’t. I fall into that trap.

Every developer falls into that trap. But I think, for data structure, it’s not something that’s taught a lot. I had courses in college on data structures, how to design data structure. It’s not something you hear about a lot now in web developments. It’s like, yeah, whatever, use a database, which database are you going to use? Are you going to do MongoDB or this or that or Postscript? But never about how to structure data. That is not taught or talked a lot about.

Till: How many WordPress developers theme the whole ecosystem? How many do you think have a comp science degree or went to university?

Zach: Very few

Till: Percentage?

Carl: I wouldn’t know. I like to say, I don’t know a lot. So, for some question like that, I don’t know. I think, even whether you went to computer, I didn’t even do computer science. I did computer engineering.

Till: Right? Because if you have a computer science degree and you look at a WordPress, you just probably walk away and go somewhere else. That would be my guess.

Carl: Oh, well, I mean, in computer science, you take courses that make no sense. You take compiler classes like, who here is going to design and compile? I love my compiler class, but in practice, not super useful. But the database course, I would say I didn’t keep a lot of textbooks, but one of the courses that I did keep my textbook of was the database course, because I actually do think that it’s something that’s been a bit lost.Not lost, but lost in importance, I would say. How do data structure designing or thinking about? Not necessarily design, but I’ll just think about how you’re structuring your data and how you’re going to query and things like that. And just having a bit of foreplanning about it or just looking up how other people have done it is really beneficial and will save you a lot, a lot of headaches in the long run, because data migrations is a hard problem.

And there’s a reason people pay for a plugin like or WP Migrate Pro, but there’s also a reason why they gave up on migration bot. I think it was called the merge bot. Because merging data is really, really hard. I get that question a lot with YMIR. I have customers that are like, “Yeah, I want to move data between two environments. Can you do this for me?” I’m like, “Companies have died trying to fix this problem.”That’s why Dropbox was such a huge hit when it came out too, because it solved that problem for file sharing, for file syncing and things like that. There are hard problems.

Zach: As developers, when we’re working in a Git Repository, how many times are we resolving merge conflicts when we’re creating poll requests?

Carl: Not as often as when I use SVN. Oh my God. I remember when I switched from Git from SVN, I was like, before that I would dread, absolutely dread pulling any change. It was like, I would lose an hour, easy.

Till: I think this is the same thing. Switching from SVN to Git, both are software version control tools. Whatever the term is. It is the same with WooCommerce subscriptions. All these plugins, what at least I see as a trend and maybe all industries, but certainly with WordPress, there is a high and high expectation. Now, we in 2022 of making things easy, accessible, and fast that people have short attention spans. With social media, everything is getting more convenient. You turn on Netflix. There’s a billion shows that you could watch, maybe not quite, but a lot. And making software as developers who as technical people or a developer, making these things easier to access, like the full side editing. We said, we’re not going to talk about it as a joke there, making it easier for everybody who is maybe technically illiterate or blind or whatever is making these things more accessible at scale. And I think WordPress does a good job in that sense, but it’s a slow moving beast. At least that’s what I see.

Carl: That’s why that’s legacy software.

Till: Yeah. It’s our responsibility as developers to not read a compiler book, but for us to make sure that these low level tools that we do that have good interfaces for not so technical people to use and not expect them to be also developers and make it exclusive. If you’re not technical, you can’t use it. That I think is a big tension zone. At least what I always aspire to completely bridge or try to bridge as much as possible to make just a little click and drag and drop interface. And it works suddenly.

Zach: Yep. And I think the key thing to remember here is that the reason why we’re talking about custom tables in WooCommerce today is because WooCommerce is growing. And we have gotten to a point now where there are enough stores that are running into these scaling issues at the top end, that it’s now a priority. And it wasn’t a priority for a long time, because the fringe was less than a half of a percent of stores. It was less than a 10th of a percent of stores that were running into these issues. But now, as the platform is growing and the stores that are on the platform are growing, we’re starting to see these issues more frequently. And now is the appropriate time for the core team to address these bottlenecks.

Carl: Yeah. And it’s also WooCommerce aspiration as well. I think, it’s a very popular e-commerce platform

Till: And maybe also, that’s why there’s a WordPress performance team though.

Zach: There have been more large WordPress installations than ever before in the last five years. And so, seeing that trend toward really large projects, I mean, it just makes sense that the WordPress performance team exists, the WooCommerce team is focusing on these things. And, I just, I cannot wait to see what their implementation is and how it ends up becoming part of the core product. So, yeah.

Carl: I think for us as hosts and things, it’s just education making people aware, because like I said, it’s not for everyone to know this, but they’re just not things that developers think a lot about, especially the database developers love to not have to think about their database.

Zach: That’s the reason why object relational mapping is so popular, but we are coming up on our hour here. So, I’m going to end with just a couple of thoughts. One, this is the first Woo Dev chat. So, if you liked this episode, if you like hearing us be gigantic geeks about WordPress and WooCommerce performance, because we’re really good at that. We’ll be here again in a month and we’ll be doing this every month from this point forward until Carl and Till gets sick of me.

Carl: Yeah. And if you have topic ideas, or things you’d like to learn about, I think this is a good place also to discuss. Well, I mean, we can’t teach, it’s not a course, but if there are topics you’d like to know more about, I think, this is a good place for that as well. Because like we discussed a bit, not all of this is accessible to all developers, not all of this is known or discussed or written about. And some of it comes from experience from our experience. Some of it comes from just our educational background. Like I said, I did computer engineering. So, I took database class. I also took useless things that I never use, like networking classes. I know I learned a lot about networking protocols and things like that. I don’t use that, but database, I use. So, this a good place. So, if you’re interested, let us know.

Zach: So, yeah. Where can we find you? If somebody wants to find you.

Till: Oh me, I’m on Twitter @tillkruss. T-I-L-L-K-R-U-S-S. I respond, but I didn’t tweet very much, except when I think of a very funny day. And I make WordPress plugins, you can check up my Object Cache Pro. If you Google my name, you’ll probably find it. But yeah, Twitter’s a good place to find me.

Carl: Well, you can find me, I’m active on Twitter. So, I’m active on Twitter, trolling Tom McFarland most of the time, or posting memes or other things. I’m @twigpress. T-W-I-G-P-R-E-S-S. I’ve written a lot on, and I speak a lot, I’ve actually applied to WordCamp Europe and a bunch of other conferences, but it’s carlalexander.ca. And if you’re interested in serverless, I’m working on a serverless WordPress platform ymirapp.com

Zach: And as always, you can find me at Z-S-T-E-P-E-K, zstepek, pretty much everywhere. Thank you guys for joining me on this today. And we’ll see you guys in a month.