Scaling WooCommerce with Multi‑Tenant Architectures

Zach: Well, welcome to another episode of the Do the Woo Dev Chats. I’m Zach Stepek, I’m here with Carl. Carl Alexander. How are you doing today, Carl?

Carl: I’m doing good. How about you? We got a lot of snow here.

Zach: Yeah, we’ve got a little here. We got snowed on pretty hard yesterday, but today we’re going to be talking a bit about multi-tenant WordPress and how that applies to WooCommerce stores. So we brought in an expert, yeah, an expert on multi-tenant WordPress, somebody who’s built a business around it. So, Wijnand van Leeuwen, we who would like to introduce you to the Do the Woo community. Why don’t you tell us a little bit about yourself?

Wijnand: Yeah, so I’m Wijnand van Leeuwen, I’m CEO and co-founder of wpcs.io and we have a multi-tenant WordPress platform that we have had for, I think it’s about two years now. And then we just keep on expanding, keep on growing and we try to get the word out of the bliss that is multi-tenant purpose.

Zach: We can definitely start out just by talking about some of the perils of using multi-site in an environment where you have multiple WooCommerce stores. We know WooCommerce is very data heavy to begin with, so adding that layer of multi-site on top of it and having all of these various stores worth of data living in a single database is kind of my nightmare scenario.

Wijnand: Yeah, and I think rightfully so. In that sense, it sort of depends on how deep you want to go into the whole infrastructural layer of it all. I think that’s the biggest point, at least with the whole multi-site and why multi-tenancy could actually be a nice thing. If you don’t actually want to learn a lot about what makes a server go quick, then multi-site might really not be for you, especially with the data heavy application as WooCommerce. Because before you know it, you’re going to have to do something with the database that might be called charting or it might be something else, and now instead of working with orders and working with post meta and working with PHP, you’re actually doing Linux administration. You’ve got to want that, you’ve really got to want that.

Zach: Yeah, you do. You really do. And everybody on the call here I think is very familiar with sitting in a terminal and dealing with command line.

Carl: I have a customer that’s multi-site WooCommerce and the database side is fun. The export has a thousand tables.

It’s fun. It’s fun. I learned a lot. Actually I had to learn, really quickly, a lot of DBA things for just doing better backups. Just doing a MySQL dump of that and then re-importing it in the cloud just kind of blew up the database server in the cloud. It was just like…

Wijnand: I can imagine.

Carl: So there’s a lot to learn about that. The database side is definitely not on the fun side of things.

Wijnand: I mean, I would even say it’s on the expert side of things. If you really want to do a thousand table, and these are not small tables probably for your client, if you actually want to do an export and an import for that in production, I can imagine that it would be really helpful to be a DBA with years of experience.

Carl: Yeah, that’s why I said I had to do a crash course on the backup side just to basically do MySQL backup, the defaults just didn’t apply to that kind of thing. And I’m just like, I would never call myself a DBA because I know better. I’m at the stage of the Dunning Kroger effect where I wouldn’t call myself a security expert because I’d know enough about security to not call myself a security expert. And then the same thing I feel about DBA is, I’d know enough about database administration to not call myself one.

Wijnand: I feel the same way about myself. Yeah, definitely, definitely. I know enough to avoid these certain things if I don’t want to run into any problems there.

Carl: DBAs, to me, are real magicians how they know the internal guts of… I’ve been saying this, that’s probably a good tangent to go into, but I think we talked about this, Zack, too, but it was like, I really feel Work Press and WooCommerce even separately should both have a DBA on staff just so that they can run code things through them and be like, “Does this make sense?” And they’re like, “No, you’re crazy. Please don’t do that. This is misery.”

Zach: The problem with that is that the DBA would say that the WordPress way is wrong from day one, just because the structure is so against everything they’re taught. And it works for us, it works incredibly well in most cases. The whole entity value relationship and the one-to-one mapping of everything. But the way that we store post meta is a DBA’s nightmare, especially in a WooCommerce site where there’s so much of it. So most DBAs that I’ve worked with like having well structured tables that relate to entities and not just to everything.

Wijnand: How odd. Why would I do that? Yeah,

Zach: It doesn’t make any sense. Well, we do it the right way.

Carl: I mean, it’s a tricky one. I think we talked a bit about it. Entity attribute value is a very valid data structure to use. That’s what Magento uses as well for, again, internally it is just as you said, it’s kind of a DBA’s like worst nightmare because you basically let them do whatever they want with the database and they store anything and there’s no structure to the data that you can easily figure out and query on.

Zach: And that’s really the biggest problem is now we have plug-in authors who have built anything you can think of on top of this entity attribute value structure, and one of those things was an entire e-commerce platform and nobody-

Wijnand: A popular one, nowadays.

Zach: It’s incredibly popular. And I don’t think at the beginning, way back in the days of Jiggo Shop, that anybody working on that team thought, oh, this is going to power a hundred million dollar websites in the future. I think it just was this is a cool way to do e-commerce on WordPress. Let’s build it. And now we’re at the point where there are these eight, nine figure stores sitting on WooCommerce and now the database structure doesn’t look as good as it once did, especially for orders. And so the one thing that did happen while we were on break here is high performance order storage came out. So we have the flag now, the feature flag for being able to enable high performance order storage, but the default still stores everything into post meta 2, which means it’s actually increasing the load on the database to maintain a backward compatibility layer until we can turn that backward compatibility off.

Wijnand: Yeah, and one of the big effects that has had is that a lot of that strain comes down to the infrastructure that you’re actually hosting this stuff on. So if you want to actually run, let alone a multi-site, if you want to run this million dollar store and you’re just going to have to have big hardware, simple as that. And then with performance gains like what came out, that will definitely help, and that will also maybe make it possible to actually lower your infrastructure maybe a bit, maybe make it a bit cheaper. But all in all, I think that’s really what’s happening there in terms of the top layer of WordPress, for example, can remain a bit more simplistic, a bit easier to grasp and a bit easier, a bit more flexible also. But then something else has to carry that complexity, and in, I think, most cases that just comes down to having a wicked CPU and a bunch of ram.

Zach: I’d agree with that. And I’ve run some really large WooCommerce sites and those sites, they required a very scalable infrastructure with the ability to scale with demand and nearly instantaneously. And they required things like elastic beanstalk, and just the ability, if they were going to have a marketing event, to pre-scale before the marketing event, right? So that, because even elastic beanstalk couldn’t spit up servers fast enough to meet demand sometimes, which is what happens when you have 2 million people on an email marketing list. These types of things, these types of scenarios aren’t the norm for WooCommerce. But at the top end of the market where people are doing this, these things become very important. And I remember years ago having conversations with the WooCommerce team about this when they first released the data object classes, the CRUD classes for all of the object types in WooCommerce in 2017.

And that was WooCommerce 2.7 that all of this started in, and it’s been six years, and now, just now, the WooCommerce team has released something that uses that to move data out of the posts and post meta tables. And of course it was orders, because orders have been a bottleneck. We’ve talked about that on this show before, how creating 50 pieces of post meta per order is a gigantic bottleneck when WordPress only allows you to add one post meta record per call. So that’s 50 separate database calls with every order. So of course they tackled high performance order storage first because that bottleneck stops you from making money.

Wijnand: Simple as that. Yeah, exactly.

Zach: Yeah, it’s the one that made the most sense to take care of first. So I don’t want to stay on high performance order storage the whole time.

Carl: I feel like that’s a recurring topic whenever I speak to anyone that does any sort of serious WooCommerce hosting.

Wijnand: No, it definitely is. And I think in that sense, even if you’re not running million dollar sites, if you’re running 50 relatively popular sites, you run into exactly the same thing. And when we’re talking about million dollar sites, that’s a big exception, right? That’s an outlier in that sense. But having 50 ish performing sites, that’s pretty easy to come by, it seems. Having these sites that actually will also ruin your database on a day-to-day basis simply by having a couple of hundred customers come in and doing 50 requests to the database because everybody actually orders something because three or four of those stores have a marketing thing going on. So I think in that sense, making that outlier of a million dollar store more sensible for more, if not most people, just try to handle it with 50 sites in a multi-site, for example, especially in a multi-site.

And I think that’s probably one of the first points where multi-tenancy comes in, because having the ability to not use the same database for each and every one of your sites and actually having those sites, be it logically isolated, but maybe even having them on completely different servers. That is already a major win where more success, because you’re really good at the WordPress thing that you’re doing, because you’re really good at e-commerce, does not necessarily equate requiring to become good with Linux or even becoming a bit of a DBA.

Zach: Absolutely. And I think that a big thing there is talking about the differentiation between what multi-site is and what multi-tenancy is, and we’ll get to that in just a moment. But before we leave the HPOS topic completely-

Carl: Yeah, it’s kind of funny that we’re talking about this because I was doing load tests yesterday with someone and just to give an idea for people. So we were doing a pretty extreme load test. We were trying to basically hammer 1500 people checking out for 15 minutes basically, hammering it for 15 minutes. So on RDS, the number of inserts per second was 1500. Oh, please. So just to give you an idea of how many inserts happening on the database when you have this kind like Zach’s talking about it and what Wijnand, I need to practice saying it. I didn’t say it yet, so that’s why I’m messing it up. But what he was talking about is that, because when you shard… Let’s say that amount of orders is coming through different stores, then you can distribute that across databases. Here we were testing one store, but it’s just to give people a bit of an idea of what we’re talking about when we’re saying why they need to change that is because when you’re doing a lot of post metas like that, it creates a lot of inserts. The high order table, you would insert data in one row and before you would need maybe 20 inserts to do that. So I’m just throwing numbers out, but basically you’re dropping it by a factor of-

Zach: Yeah, it’s actually 50 per order.

Carl: Yeah, exactly. So it’s an insane amount. So just here where I’m talking, I’m doing 1500 a second for that because that wasn’t using high order tables, so I was just trying to brute force an installation. But if we use that, if we use a 50, so you would be dividing that by 50. So it’s a huge, huge, huge difference in terms of how much strain your database server can handle. That’s why it comes up all the time because for a lot of people don’t really… For most people, you’re running a store and it’s a small store, it’s not getting necessarily a ton of orders all the time. So it doesn’t really, “Okay, I’m doing 50 inserts, yet that’s not a big deal. I’m getting three, four orders.”

But for those larger stores, you are going to a realm where that’s why the DBA is basically crying in a corner, in a fetal position because he is like, “Why? Why are you doing this to me? What did I do to you to deserve this from you?” And that’s why there’s so much pressure on this change because this change drops it by a factor of 50, and that’s massive.

Zach: Close to, yes.

Carl: Yeah, we’re just throwing numbers, but it’s a big number. Even if it was 20, like I said initially, dropping something by a factor of 20 is a 95% reduction, so it’s still insane. By 50, you’re like 98% reduction. It’s a mind-boggling amount of performance change at the database level.

Zach: But the key is if you want to take advantage of that, if you turn on high performance orders storage, now it defaults to having another feature enabled. And that’s the backward compatibility feature where it’s still saving everything to post meta. So that’s actually going to hurt performance for a little while.

Carl: Yeah. Can you disable that? I don’t know enough about the guts. Can you disable…?

Zach: You can turn the compatibility off. The caveat is if you turn the compatibility layer off, there’s no way to go back ever. So know that you’re making this change that is a line in the sand and you can’t go back afterwards. So back up your site, make sure all your data is in an offline location that you control so that you have the before just in case you ever need to go and look at it and come back. I don’t see that being a problem. They’ve done some great work on making sure this works. There will be migrations moving forward if they have to change data structure at any point. Just to be safe, make sure you have a good copy of all of your data before you make that change, because this is an irreversible change without the backward compatibility enabled. So yeah, just a caveat, keep that in mind.

Wijnand: It’s a small little detail.

Zach: If you’re starting a new store though, there is absolutely no reason not to start from day one with high performance order storage enabled, just absolutely no reason.

Carl: Yeah, you’re saving yourself and your host a lot of trouble, basically. Yeah, grief, if we want to be more explicative about it.

Zach: That’s all I really have to say about high performance order storage right now. I think we’ve beaten this to death on this podcast over the time that it was in development, and it’s just nice to see it here. But I did mention that we were going to talk about the differences between multi-site and multi-tenancy. What’s the difference with multi-tenancy, what’s the differentiator there?

Wijnand: So I would say the biggest differentiator is the fact that with a multi-site, you make one multi-site out of one installation. Exactly what you said, as if you’re one server, if you’re one database and you can do with that what you can also do with a normal WordPress installation. Multi-tenancy would be, at least how we also envision it, would be the idea of having a lot of single sites that share their functionality, but have different databases in the sense of having different tenants. That idea allows you to, for example, put different infrastructure under different databases, under different servers running PHP. Basically having the idea of multi-site that you install a plugin and it’s available and maybe even activated everywhere but having that within a single site context, because I think that’s also sort of where that came from for us. We had an agency before this, and we build worker’s website.

Carl: Side note. Sorry, I have to say there’s so many hosting companies that started off as agencies.

Wijnand: We all become very salty.

Carl: Yeah, it’s just something I like to point out to people. Oftentimes they’re like, “Oh, how did you start a hosting company?” I’m like, “Well, half the time they were agencies before.”

Wijnand: I think that’s the perfect place to start to see, “No, but this can be done differently and better,” and I want to make sure that that actually happens. That’s also, why I wouldn’t necessarily call us a hosting company, but more like a cloud platform, because the hosting is definitely there, but it’s a part of it, right?

Carl: Yeah, I agree. Sorry, but you know what I mean, just hosting websites.

Wijnand: Totally, totally. Yeah.

Carl: But like you said, because also you get an idea of the pain from just doing it for your clients as well, so that makes an easy transition as well. So you probably started working on this kind of multi-tenant setup through the agency, right?

Wijnand: I mean, basically. Yeah, basically, because the big annoyance for me was sort of twofold. It was on the one hand, at some point you have a bit of a preferential stack. Somebody asked for a new site and you just know exactly what you’re going to use because you’re like, “I know these tools, I know these plugins, and that’s just what’s going to happen. I’ll make sure it’s configured properly for you.” I was getting sick of having to set that up all the time, and obviously there’s a bunch of templating things out there. But the other thing that I was getting a bit sick of was the idea that after four or five clients, I would definitely make a change in my stack. I would definitely make sure that this plugin maybe is replaced by a better plugin, by a newer plugin. Obviously a bunch of updates happen, and that would always be in the last site I built, and that would never actually go back to one of the first sites I built, because I can do that, but that’s free work, and that’s really not how an agency works.

Carl: But also your clients, I mean, from my days in agency, the clients didn’t want to pay for that either. They didn’t want to pay for updates, they didn’t want to pay to make sure the site worked or the plugin, till they got hacked or something. Then you’d scare them about getting hacked and they’d be like, “Oh no, okay, please.”

Wijnand: And that’s the type of client that wouldn’t buy a lock for their door if they didn’t know why. Yeah, yeah,

Carl: Exactly. I completely relate to that agency story because yeah, the newer projects have the newer stuff and then everything else slowly falls behind because you never updated.

Wijnand: And also, after you’ve revamped your whole stack and you’re really happy with it and you’re really happy to work on that website again, you’ll always get a request from one of your previous clients, and you have to go back to your old stack that you’re kind of disappointed with now because you’ve got this new shiny toy that’s running in the newest website. But you still have to work with the old kind of muddy toy that really still makes money.

Carl: Old and busted.

Wijnand: The old and busted one. Yeah. All of this was a thing that I thought, well, a lot of this is actually solved by multi-site, but I did have a bit of experience at that point with infrastructure, and I also knew that I didn’t want to fall into the headache of having my one very popular client slowing down all of my other clients. Because those other clients are going to ask, why is my website slow? And if I want to make sure that doesn’t happen, I just have to buy really expensive hardware, and that was a rat race I didn’t want to fall into. So that leads to multi-tenancy, making sure that you can have the same code base, you can have the same plug-in, beanstalk, whatever, but having a lot of single site websites actually share that.

And they have their own databases, they have their own uploads in something like EFS or wherever, and suddenly you can actually work with this stuff, with stuff like containers, and you can make Kubernetes do things with multi-tenancy. And this whole world opens to making a lot of websites manageable with very little people because suddenly you get to work on this, well SaaS or VaaS, kind of depending on what it does. But you get to actually work with that with maybe one or two people if you really want to up to production and if you really want to make sure that some things happen.

But basically one person can actually know exactly what the effect of some new code will be, all of those ten hundred, thousand websites because you know what the websites look like. That’s maybe also a point I have to dig in a bit deeper, because that was actually one of the other things I really hated about the way that I worked in that agency style, that I would have my preferential stack, but there’s always these exceptions.

This one client asked for a certain thing that maybe they wanted a carousel or something on their website, so I had to add a whole new plugin for that, and this other client wanted this, and maybe there’s a weird mail plugin that’s happening in this other website. So now suddenly, if I would actually be able to update my core stack, I need to test all of these websites because I don’t actually know what the code will do to any new code, to any new update and having everything being uniform. So basically starting thinking about it as a SaaS as a WAS, that makes it so much easier, so much cognitively less of a load to actually think and reason about this product that you’ve built.

Because just what the code looks like, the only factor that you have to think about is what could be in the database and how can I handle that? How can I catch my exceptions for that in a proper way, et cetera, et cetera. But you’ll never have to guess, could there be maybe a plugin that ruins everything and I’ll get four sites that go down simply because there’s this carousel thing that’s actually not compatible. And I think that was maybe actually a 10 explanation in short, in a nutshell about the difference, I think.

Carl: No, I mean it’s a good explanation because I think, I’ve been in your shoes, so I completely get it. There’s also under edge cases, but I think you covered it pretty well with the idea of, okay, we have no way of testing things basically because let’s say you have an older site that uses plugin that was using plugin A, but in an older version, and then something breaks with how you integrated that plugin A into older site versus the newer site. Updating that plugin can still cause issues, for example, if you don’t not testing all the sites, which is another huge problem in WordPress, is just testing is really painful.

Wijnand: It really is. It really is. And after testing, if it doesn’t actually work out, actually rolling back to some previous version of-

Carl: Well I think your platform must be pretty straightforward. I mean, it must be somewhat straightforward to do. You can just basically restore either the older version or just reinstall the older plugin.

Wijnand: So basically what we do, we have a versioning system that you’ll make a version of your product, of your SaaS, so that’s a certain plug-in stack, et cetera, et cetera. And what you can do is you can create a new version based on that old version. So you can sort of see that as a branch, as a Git branch. You can make changes in that branch, you can add plugins, update plugins, whatever you will. And what you can then do is you can move tenants from the first version to the later version, and that will actually make sure that those tenants then use the newer code base, the updated plugins, et cetera.

If that goes wrong and you actually kept your older version, you can just move your tenant back and they’ll run on the old code base again. So that’s the simplest way that we could think of to have this versioning system within a multi-tenant context and have a safe deployment system in there.

I mean, all of this can be dealt with an API. So there’s also the possibility of having a CICD pipeline that does some form of automated testing, which should be possible because you could fairly well know what your plug-in stack will actually produce in terms of HTML, and you could fairly well know what you can actually test in there. So then you can move that one tenant to the new version, run some end-to-end tests on that tenant, and if they fail, you just move it back and then you flag that one tenant as we probably should check this out because something’s off here, and then you go to the next tenant and you go to the next, and you go to the next.

Thanks to our Pod Friends OSTraining and Nexcess

Zach: Is there any automated testing involved, anything with unit testing or even visual regression testing to try and test those sites when you change versions?

Wijnand: So not from our side, because in that sense, we give you the cloud platform to run multi-tenancy, right? But we can’t be sure how your side is actually testable. The moment that you have a video there, our visual tester might not take that into account and if we do take that into account, we might not take other stuff into account. So we’ll never really be able to have a generic way of testing all of this stuff. But our customers know their own products really, really well, and they should be able to do everything perfectly,

Carl: Yeah, and they should be able to want to use the tools they want to use as well. I think it’s good. Well, from my perspective too, because I’m kind of developer focused, there’s nothing wrong having a blueprint or something like, “Oh, you don’t know how to start visual regression testing. Well, you could use this tool and then here’s a small guide of how to do it.” But tying the entire platform to a specific way of doing things probably would anger more developers than necessarily helpful.

Wijnand: And I mean, we’re already on the edge there because we offer multi-tenant WordPress, which is already kind of an opinionated thing.

Carl: It’s true. But I feel like all hosting is a bit opinionated and it’s a bit also the same. You were talking earlier, you just have a comfortable way, and the way I usually describe it is just like most WordPress hosting now is all vanilla, but it’s like what flavor of vanilla do you like? Do you like French vanilla? Do you want just the cheesy fakes tasting vanilla that you get with soft serve ice cream? What type of vanilla flavor do you want? But it’s still vanilla.

Zach: Madagascar vanilla, it’s a requirement. Yeah.

Carl: See we have fancy Zach here with his Madagascar vanilla?

Zach: But I think the interesting thing about all of this is that, yeah, we’re decoupling the management of these stacks from what necessarily the infrastructure is, which is a good thing. Now the infrastructure can be managed separately from the stack of website code, and I think that’s huge, being able to bring your own tools and do your own thing. Most of the hosts are using opinionated solutions for things like visual regression testing. Now, there’s every host it seems at this point has their own tool for VRT and for automatic updates of plugins, and automatic updates can be dangerous. They can be very risky. And now, it depends on the site, but in the context of WooCommerce, which we’re talking about here, automatic updates on a plugin could stop your checkout flow from running completely, and a VRT tool isn’t going to catch that. So chances are, visual regression testing is going to see a checkout that looks the same. It’s not going to see a checkout that suddenly doesn’t run when you hit place order.

So it’s really important to have a really defined method for handling these updates and being able to test them across a stack of stores. And if you were doing this without multi-tenancy, you’d have to make those changes and test against every single store individually. Whereas with multi-tenancy, you’re going to want to test them all. But if you’ve tested one, chances are the rest are probably going to be okay. And you can do just smoke testing on the rest of them, run a single order through or whatever, whatever your smoke testing protocol is at that point. But you don’t have to do the full regression testing on every site once you know that the stack itself is working because of that consistency.

And I think that’s a really important point. If you’re running multiple WooCommerce sites as an agency, this is probably your ideal situation where you have a stack where you can run the same things. You’re running the same payment processors or a group of payment processors that are consistent and you turn them on or off based on the client’s needs. You have a number of plugins that are used for marketing and other features like shipping, tax calculation, these types of things that are going to remain relatively consistent across all of the sites you’re working on.

Wijnand: And in that sense, let’s also not forget the new client asks for something new. You implement that feature into your multi-tenant product and you can then contact your old clients and say like, “Oh, would you maybe also a carousel on your homepage,” or a bigger feature that’s a bit more meaningful for that client, obviously. But you actually get to provide more value and you can keep providing value for your customers, even if they’ve basically long ago only bought that website. But you can still say, “Okay, so we actually have this new feature, would you like us to do anything with that?” So you can keep that account management going in that way also. And then where you can go after that is also, “Okay, we’ve got this nice bigger SaaS class and we can start automatically selling that to people in a self surface way. Because eventually, I think, and I think that’s also the beauty of, for example, is that you basically have the ability to know a lot about a niche, you know something about WordPress, and you can make something in WordPress exactly for that niche.

You can make the perfect website, the perfect e-commerce site for a very specific group of people, and that will work so much better than any Shopify or Squarespace ever can, simply because you have that knowledge, because you have that will to actually make this for those people. If you can actually automatically sell that then and work with that as a SaaS, as a WAS, then you as a person can learn a lot about WordPress and you can learn a lot about DBA and you can learn a lot about maybe even the underlying infrastructure or third party services. And you can have that value multiplied by the amount of actual customers that you have very, very easily without having to go into the whole infrastructural layer and also having to learn how an AWS data center works or how an AWS VPC works.

Carl: There’s a lot to learn on a AWS.

Wijnand: I can very much relate to that.

Carl: I mean, my product’s also on top of AWS and part of the price is just not having to deal with that. Part of the value is just not having to deal with that.

Wijnand: And that’s for a good reason because it’s hard and you can do it wrong and you can do it really wrong. And if you do it really wrong, then your e-commerce site is going to go down. That’s going to cost you a lot of money besides the servers that are still up and running and costing you money via AWS. If you can actually get somebody else to do that stuff for you, that’s brilliant. Yeah.

Carl: I see it all the time. Mean my consulting is cloud audits and stuff like that. I’d go through people’s like AWS bills and one client had $500 a month hard drives that they had provisioned but not attached to anything. So they were just sitting there. They were just sitting there, not used, costing them $500 a month. It was just like, “No, get rid of it.”

Wijnand: Get rid of that, please.

Carl: Yeah. I mean, that’s half the thing with AWS is it’s really easy sometimes if you’re not careful to shoot yourself in the foot and leave stuff running and then you end up paying for it and you can’t find it because the console’s straight out of… I mean, I started as a Windows sysadmin, so I get a lot of PTSD from my Windows sysadmin base from the AWS console. It’s very large, everything’s all over the place. It’s hard to find things.

Wijnand: So many context menus.

Zach: They make it incredibly easy to spend money and hard to stop spending it. I think that might be a feature.

Carl: I have a lot of discussions around it, this is a bit of a tangent, but I feel like they should have some sort of safety valve thing. So the larger thing is I think ethically and morally they should have a thing because I’m just waiting for the one day that somebody is going to do some sort of… I’m sure it’s some AI thing because they’re going to just provision some GPU, leave it running, forget about it, and end up with a $20,000 bill. And then they’ll freak out and they’ll be like, “My life’s ruined.” And do something really drastic. The thing is, if you know AWS and you’re in the kind of ecosystem that if you make a mistake like that, you can contact support.

Wijnand: This is one story about a guy who made the Lambda.

Carl: Yeah, you see a story every year, year and a half, it’s often through Reddit. But basically I’m just kind of waiting for the one day where somebody is going to have that reaction, but not talk to anyone and just think their life’s over. Or they didn’t know, and then it’ll be on AWS’s side. So that’s my position on the whole thing, but they make it really, really easy for you to, and it’s a hundred percent a feature because for the larger enterprise customers. When I do some sales calls, I talk with people who work with companies, they get a $200 million discount.

Wijnand: Wow.

Carl: From the AWS, discount, not actually that’s how much they pay. That’s the discount they get on how much they’re spending on it. So for them, it sounds like it’s a rounding error, even that $50,000, but for that person, it’s like-

Wijnand: That can be life ending. Yeah.

Carl: Exactly. So I’m very, very displeased with that. It’s a feature, I get why they do it, but they need something for the individual that just spin up an account and then prevent them from shooting themselves in the foot.

Wijnand: Totally. I mean guess it’s also a bit of a question of how much, because you go to AWS to have this whole data center at your fingertips and have all of that power, but at the same time, that also usually means all of that responsibility. I agree that there needs to be a way to go into AWS where your hand is held much better and much more and you can’t make $70,000 mistakes. But then again, you do still want to leave people open to the complexity and the power of it.

Carl: Exactly. Because I have to deal with that too because people connect with Ymir, people connect their AWS account. So I don’t want it to be so restrained that I’m like, “Okay, you need to sign up, but now you need to make these 15 support tickets to get your quotas changed so that you can actually use the product.” So I get it, but I don’t know what the solution is. It just feels like it’s something waiting to happen. That scenario that I’m talking about, everybody that’s in that space knows that it’s going to happen someday and it’s going to be terrible, and they’re going to get tons of bad PR out of it. The only way something’s going to change is until that happen. And it’s tragic that everybody sees it and nothing gets done basically.

Zach: Well, and it may be helpful for them to stop having an administrative dashboard full of anti-patterns. That might help a little?

Carl: I feel like they just hired the old designers from Windows server and then they were just make us a UI, make us a console. And then they’re like… But I don’t know for you, but I deal with the APIs a lot and the APIs are all different. They’re all different. I’m basically, half my product is just standardizing some of the APIs so that I can use them in a more cohesive way. But I’m doing tagging right now and every service tags differently, it’s maddening. They all have tags, but it’s all handled differently, and I’m just like, “This is maddening.”

Zach: There is no standard across their APIs, by any means.

Carl: No, exactly. But they’re stable, so I’ll give them that. I would still rather be in business with AWS than anybody else because the APIs are stable, but it’s maddening that something like tags is so different, basically. You’d think that, “Oh, this will be the easiest thing to do.” And it’s like, “Oh no, this is actually maddeningly annoying that it’s so different.” Some of them, you can update them directly from the resources. Some of them you have to tag them. Some of them use associative arrays, some of them it’s key value. You have to do key ICS, value that and it’s like, “Oh my god.”

Zach: Be careful what you ask for though, because you may end up with ATS, the Amazon tagging service in the near future.

Wijnand: Oh, you’ll probably end up with that service and they’ll call it something like Amazon Dart.

Zach: Yeah, all it does is tag things.

Carl: Yeah, some automating tag thing would be great.

Zach: But yeah, I mean that seems to be the thing when they need to standardize something, they decided, “Hey, we need to standardize access.” What did we get for that? We got a standardized access layer. We got IAM, right? They needed to standardize other things like how to run small snippets of code. So we got Lambda. A lot of the things that are AWS now have come out of the needs of customers and become their own services. I still remember when it was just EC2 and S3 and that was pretty much it, and now there’s like 700 services.

Carl: So many services.

Wijnand: And I guess that’s also a part of not having to put it in one big package. I guess that’s sort of the blessing, but also the curse that AWS has, that they have all of these different services that you could actually use in isolation, and they never actually have to think about how to get all of those services to work together. So you get APIs that work in a completely different way because obviously those teams are going to make different decisions than these teams. I mean, imagine if you have to talk to maybe 5,000 different engineers if you want to debate how to actually tag something, so you’ll get this sort of difference that becomes bigger and bigger and bigger. I can imagine that’s actually really, really hard to-

Carl: But sometimes they just come out with weird products. Like the joke, especially for you since you use containers a lot, is how many container services AWS has. It’s like, “Yeah.” There’s so many container services, they could just have one, but instead they have 13, instead I have 13 of them. So it’s like…

Zach: They all serve different purposes, supposedly, and I think this whole space is really interesting, the infrastructure side, but it’s really cool that with WPCS you’ve been able to kind of standardize the multi-tenancy piece a bit. Carl, with Ymir.

Carl: Do the Mad Scientisty things.

Zach: You create some stability to running auto-scaling WordPress on top of AWS.

Carl: Yeah, that’s the main thing is the scaling also I’m kind of like Wijnand, Got it?

Wijnand: Yeah. Perfect. Love it.

Carl: Second try, I got it. I’ve been a sysadmin since 16, I don’t want to do it. I don’t want to be on call anymore.

Zach: Yeah, you’re building tools that do the work for you basically.

Carl: Well, I’m building tools that try to just keep the site up regardless of what you blast at it, which was what I was testing yesterday, basically. I was just like, “Okay, what happens if 1500 people are just checking out and buying things right away?” And it’s like, “What’s going to break? How is it going to behave?” And it was just like, “Oh, this is okay. This holds.” So I’m like, “This is great.”

Wijnand: Perfect.

Carl: Yeah, but because it’s the same thing, right? Because that’s a bit what WCS is solving as well, is you don’t want to have to constantly be like, “Okay, I have to think about upscaling. I have to downscale. If I don’t downscale…” I do sales calls around that. Even if you don’t have to worry about the infrastructure, you have to worry about scaling it up and remembering to scale it down. And then if somebody doesn’t warn you and the whole thing is just… It’s still stress, it’s still a responsibility that you would just rather not have. And I talk a lot with people from the JavaScript ecosystem and they get it right away, because they’re like, “did they plug them or sell them?” They’re like, “Doesn’t matter if I get 1500 people, it works.” It works. We don’t want to have to think about this.

Wijnand: And I think it’s also that exact reason is why something like WPCS can exist and can actually serve people, but that’s exactly the same reason why something like a WAS can exist and actually serve people.

Carl: Yeah, exactly. I mean I’m going to talk about that too. I work Camp Asia, but yeah, you’re right. Because before if you wanted to do a WAS you’d need an entire ops team and now you can just get WPCS or me and then basically, oh, I don’t actually need a cloud engineer or an army of cloud engineers to run my business. I can focus on the code, optimizing the code. That doesn’t mean that you can just be stupid and just be like, “I want to do a hundred inserts per order and then it’s not going to blow up.” But that’s a code problem.

Wijnand: Exactly. You get to focus on that layer, right?

Carl: Yeah, exactly. You can focus on the code and how it affects the infrastructure, but not so much on running and maintaining that infrastructure.

Wijnand: And in return for that, our customers get to focus on their, presumably business because they get to just enjoy a WordPress website that just sort of works for their specific use case because the builder just knew that use case, that niche really well. So you get the sort of delegation of responsibilities that-

Carl: Or specialization. That’s the way I see it, it’s more like specializations. This is how I feel, and we’re kind of in the same boat, you and me, but we’re the people talking in the early two thousands about the cloud. They’re like, “The cloud, you have not heard about the cloud.” And it’s like, “No, you’re crazy. You run your own data centers and you have an army of sysadmins to run them.” Or you could not have an army of sysadmins, pay AWS a premium and let their experts, they’ll have way better experts than you’ll ever have, deal with that, and then you focus. It’s the same thing. You’re not AWS, but you’re making all these multi-tenant pieces work together really well and you focus on that and they don’t have to focus on making WordPress perform super, super well. They just have to make sure that either their site or their WordPress product, the WAS scales and does things properly.

We were talking about database inserts, but that’s what they can specialize in. That’s what they have the expertise in. They’re plugin developers, they’re WordPress developers. They’re not sysadmins. I’m not a DBA, and I’m a pretty good developer and I still don’t call myself a DBA. So you let those people do what they’re good at and then you just hire that specialization.

Wijnand: And then you, as the consumer of that service can apply that in scale. Because I think that’s the biggest thing there, whenever you apply something in scale that will take, not necessarily all of your time, close to all of your time. And the people that are actually experts at making sure that there’s hard drive failover and that all of these servers keep running and that the actual metal will still work and not catch fire. I’ve never really been in a data center physically, but I think I get to get the gist there.

Carl: Yeah, if you’re not stuck, if you were on OVH, OVH when their data center caught fire last year, and then you were like, “Oh, I’m on OVH.” And it’s like, do you have backups? No. But that’s it, those are things you’re responsible for when you’re the one that has to manage all that. You have to be like, “I have backup.”

Wijnand: Yeah, and that works on scale on all of those levels. We, as WPCS, need to make sure that there’s backups of a plethora of different things that are much bigger than one client, basically. But our clients, our customers have to make sure that they have backups of very specific things of their customers, of the websites, their tenants in that sense. And that is also a completely different way of looking at those two things. We can’t actually restore a backup that restores every database in our cloud system because, well, I mean this person may need a restore, but the person on the other side of the system definitely did not. And we can restore both of those things, but we can actually get our customers to to think about that in the context of their product much easier because they don’t have to worry about the whole storage bit. They don’t have to worry about how to actually do a database dump. They just have to think when is it backup actually necessary for this one product that I’m building.

Carl: Especially if they’re not aware that there’s sharding and all that stuff going on behind the scenes as well.

Wijnand: Exactly. You get to focus on the actual WordPress part of it, and you get to specialize in PHP, you get to specialize in WordPress and WooCommerce and you get to specialize in selling that product that you build, and you don’t accidentally have to go knee-deep into EC2.

Zach: Yeah, I think that’s a very important point, and I think if you manage WooCommerce stores that you should be looking at some of these things that the possibility of multi-tenancy at things like auto-scaling and having to not worry about your infrastructure as much. Because all of those things are going to take you as an agency owner that works on WooCommerce sites or as an individual contributor who works on WooCommerce sites and let you focus on the needs of store owners rather than just on how can I make PHP do what I want it to do today. And how can I make these servers stop catching fire every time that there’s a traffic event? So I think this has been a very valuable conversation. We’re at the hour here, we’re actually a little beyond the hour, so I want to make sure we have a chance to wrap up here and give people a chance to follow you, Wijnand.

Wijnand: The best place to find us is on our website wpcs.io, but we are also on LinkedIn, we’re also on Facebook. If you look for WPCS, you’ll find it, and wpcs.io is where you can actually sign up and you can get started right away.

Zach: So thank you Wijnand, for joining us today. It’s been a great conversation. Fascinating stuff.

Wijnand: That was an absolute pleasure. Thank you so much for having me.