Under the Hood: Deep Dive into EDD 3 Architecture & Challenges

Zach: Well, we’re back again with another episode of the Woo Dev chat. I’m here with Carl and Till, and we have Jay Trip joining us, John James Jacoby.

JJJ: Hello?

Zach: From East Troy? Not far from me.

JJJ: Not far from you, right? You’re still in Rockfords. So yeah, an easy hour.

Zach: Yeah, not bad at all. I make that trip pretty much every time I go to The Rave.

JJJ: Oh yeah. What are you going for next? I’m going to jump right into this and ask you a question. What are you going to The Rave for next?

Zach: Ah, I’m not a hundred percent sure, but probably Mudvayne and Starset. Awesome.

JJJ: Gosh.

Carl: Okay. For those of us that don’t live an hour apart, what is The Rave?

Zach: The Rave is a legendary multistage music venue in Milwaukee, Wisconsin.

Carl: Okay, cool.

JJJ: Yeah. And on top of it is the Eagles Ballroom, the top floor. Which is where I saw Run DMC and Kraftwerk, which were both amazing, amazing shows.

Zach: Nice.

JJJ: It’s cool. It’s just one of those venues that there’s not much like it. And so, when there’s a cool artist or somebody performing there-

Zach: It’s one of a kind.

JJJ: It’s really, really cool. It’s like an old ballroom. So the floor is legitimately a ballroom, and it’s all ornate and very cool and decorated neat. And there’s a bunch of floors and it’s super open, and the acoustics are fun.

Zach: Yeah, cool.

Till: You know, speaking of concerts, Rage Against The Machine is touring again this year, in North America. And I’m 35 now and none of my friends want to go with me. I’m pretty bummed out. I’m contemplating just going by myself. Because I’ve never seen them, I was too young.

JJJ: Funny story about that is, there’s not just The Rave, but in East Troy, where I live, excuse me, is the second largest outdoor amphitheater in the United States, called Alpine Valley. And so Alpine Valley Music Theater is going to be hosting Rage Against The Machine. So they will be two seconds away from there. It’s like a half a block away from our house.

Till: And are you going to go?

Carl: Oh, so you don’t even need a ticket. You’ll just hear it.

JJJ: Oh, of course. No. Yeah, we’re just going to hear it. And so because it’s so close, we have friends, family friends, that work for the venue or Clear Channel, or that volunteer and help out. And so it’s guaranteed to be an amazing time. Excited about Rage Against The Machine.

Zach: Yeah. Alpine valley didn’t have shows for a while, but it’s nice to see them coming back and having shows. I know they were trying to restart right before the large gap in performances happened. But it’s nice to see them coming back. Alpine Valley was famous for a number of very large tours, not the least of which being Ozzfest, but also when Lollapalooza was still a traveling show, Lollapalooza was there. So very, very legendary outdoor.

Till: I think we’re alienating anybody under 30 right now.

JJJ: Maybe a little bit, maybe a little bit.

Carl: But yeah. So we have you here not to discuss music, but programming, as much as we’d like. So we were discussing before the episode started, you used to work on EDD right?

JJJ: That is right? Yeah.

Carl: Yeah. What did you find the most challenging about working on it?

JJJ: So when I started at Sandhills, then 2017, it was with a very explicit intent of moving all of easy digital downloads as data, out into custom database tables. Which sort of transitions nicely from your last episode to this one where ..

Carl: Yeah. That last episode. Exactly.

JJJ: And so it started off as let’s just try and find a way to move that data out of the posts and post types and taxonomies, and put them all into custom database tables. That’s not something that is particularly challenging, in and of itself, I feel like. Because it’s something that I’ve worked a lot with, with BuddyPress, and worked on just the WPDB class itself, and migrating bbPress over the opposite direction. And so, moving data around for me is something that like I enjoy doing, and think is a lot of fun.

Carl: You’re probably one of the handful of people in the entire ecosystem that has that expertise though.

JJJ: Pretty unique experiences, I guess. Yeah.

Carl: Because bbPress, and bbPress, for those who don’t know, is basically a Bolton, it’s a forum built on top of WordPress. And then bbPress is basically Facebook on top of WordPress. BuddyPress, sorry, BuddyPress is. But they’re both data, they all contain a ton of data.

JJJ: And so it was one of those things where like when Pippin said, do you want to come on and do this? It was like, of course I do. That would be a ton of fun, and it would help EDD, and it would help you. And Pippin and I have been friends for a long time, and I really enjoy just jamming with my friends, right? We can sit down and work on something, if it’s music or code or anything else. Of course I’m down to help. So the hardest part about working on EDD was because it has, similar to Woo, where it has a pretty enormous ecosystem of third party plugins and things, that just maintaining backwards compatibility, trying to write as many tests as possible, to make sure that everything that was working before, is going to continue to work.

And that includes areas, sort of darker areas of WordPress core, where you might not have a hook to intercept something, and go grab it from the new table. Or there are places where it just, you can’t, if somebody decides to do their own WP query, you can do some guesswork for metakeys and you can kind of translate that over to some custom SQL, or a custom function call. And you can be clever about it, but you can’t always do everything. So we really spent a lot of time. And afterwards, even when I left the EDD team, the existing team, Ashley and Chris and everyone else, spent a lot of time just reviewing and deeply scrutinizing every add-on, to make sure that all the backwards compatibility stuff worked. So it wasn’t even the database stuff. The database stuff, we spent some time back and forth talking about, and architecting out, and trying to future proof a lot of stuff.

And did a lot of research, not just with WooCommerce, but other open source pieces of software that let you take a look at what the data scheme is. OS Commerce is a super old one that everyone has poked at a long time ago. But still worthwhile platform and still open source and out there killing it. So we looked at a lot of platforms to figure out, what can we do with EDD to make it? WordPress is a blogging platform and that’s great. They can do so much more. And so I think the vision with EDD is yeah, people know it for being a piece of software to help download digital items and things. But there’s just not much difference between that, and moving on to something like Woo. And so, if you could invent a brand new data scheme, to store your online store data, what would you do, and how would you do it?

And we took a lot of inspiration, I think, from the folks at GiveWP, because that was a, it started off as a fork of EDD. And so they have a great development team where they forked it, and they did a bunch of really awesome improvements to it. So we looked at what they had done and tried to incorporate some of the stuff back in. And so there was just a lot of planning, and architecture, and research that went into the early days of it. And then it was like wow, this is not the three month project that everyone starts off thinking that putting custom database tables in.

Carl: No.

JJJ: Right?

Carl: It’s never, it’s never, it’s never a three month project. It’s never. Planning this stuff is always a challenge. Even if you just do the magic two Xing, or just bumping it out.

Till: Three X, I three X everything.

Carl: Well, it’s the unknown unknowns, that get you.

JJJ: And one of the things we did, is we ended up having to shim a bunch of objects. Like the EDD had a bunch of objects for customers, and products, and all prices, and all these things had their own PHP objects in them. And in order to make sure that we could maintain some bit of backwards compatibility with all of those things, we have new classes in the middle, and we made them magic, so that if somebody tries to interact with some piece of data that wasn’t there before, that should be that we can go out and get it just in time. And all the database stuff that we did, all comes with query caching and object caching for free. Which EDD never had before, was some of that old stuff. It’s just custom queries all the time.

And so we ended up doing like all these things that really made EDD three, an amazing piece of software. But I think the place that it, and so the project went a little bit off the rails. I mean, for all good reasons. But once we saw how awesome that it was working, and saw the test passing, and saw that this was really, truly alive and working, and working really well.

Carl: Yeah. You were getting there.

JJJ: Yeah, we were. All of a sudden it was like, well if we’re going to do all of these things, what else can we do? And then, it’s sat in beta for a really long time, because the sky was all of a sudden the limit. We could do really good reporting. We could do really good, really complex partial refunds. And transaction things that we couldn’t do with the old version of BDD, because we weren’t storing data in a way that allowed for us to do those things. Custom database tables can’t be the only feature in EDD 3. We should make sure that there’s a reason that we’re doing this. And so we redid reporting. We added some things for partial refunds, which meant we needed new UI, which meant we needed new UI elements, which meant we sort of looked at what everyone else was doing for UI and decided what we wanted to sample, and what we wanted to invent.

And all these things sort of just, they go downhill a little bit, real fast. We’re on a small team that can’t move quickly and can’t crank stuff out, because there’s just so much opportunity to improve it all. Yeah, it sort of sat. And the successful things, I think Berlin DB was one of those pieces of software came out of. We could take the logic that EDD 3 needed for custom tables, and knowing that WordPress doesn’t have any of that was sort of the birth of Berlin DB. Which is an open source project that just powers custom database tables for WordPress, in a very WordPress kind of way. And so we use it for Sugar Calendar. We use it in Restrict Content Pro. So it didn’t have a name. I was up on stage at Word Camp Europe in 2019 and was showing it off, and didn’t have a name for it. And, then it was suggested that I name it after Berlin, where I showed it off. And so yeah, I name ended up naming it Berlin.

Carl: I love Berlin. So I’m okay with that. I’m okay with that.

Till: Did someone just yell it in the audience?

Carl: No, sort of my closer at the end of my talk was, we have all of this cool tech, but we don’t have a name for it. And so we will open source it, it will be available for free. But if you have any ideas, then find me afterwards and let me know. And then it was Peter Wilson, I think. Pete Wilson, who was at Human Made for a while, and was at Awesome Motive and is at 10Up now. I think he was ultimately the one that suggested to name it Berlin. And like on GitHub, Berlin, just the name Berlin, is squatted by somebody and is just an empty username, or repo. And so I tried contacting them, to get it. But they ghosted me. So Berlin DB is what it ended up being.

Till: I find myself in a similar situation with developing big features. And since you just kind of got out of this, or you’re still in it, what would you do differently, when you have this not quite sky is the limit, but these massive feature cycles, where you don’t want to have smaller releases in like two, three months. When you really till the soil and rework everything, what would you do differently next time around?

JJJ: Well, there needs to be a person who says enough is enough. Someone has to be the bad guy, sort of bad girl, bad person. Someone has to be there to say, this is working, this is enough. We should ship it as it is, and we will deal with whatever happens as a result of that. Because the lesson that I have learned and learned again, is the longer that you take to ship something, isn’t just the … yeah, it’s getting cooler and cooler, and you’re having more fun or whatever else. But there is that anxiety eventually to ship it, because you have done so much and you just get more and more afraid of everything that you’ve changed. Breaking something over and over again. And I think that we’re, even though we’re working remotely or whatever, we can sort of sense each other’s growing anxiety with those kind of things.

And I think the excitement from shipping, is different than the excitement of inventing. And WordPress has always done a good job of incrementally inventing, iterating on things, in ways that were invisible to most users, and then they become a reality. And I think we have to build things that way. Apple’s another good company. Porsche is a good company that, you look at these companies that have been around forever and they iterate and they release new things every year. But a lot of the times, you might not see what’s going on. It might not matter. And it’s just too easy to get caught up in this, the completely empty void of possibilities. Someone has to be no. There has to be a person that everyone responds to and listens to, that goes you’re right, it’s time for this to go. We are done with this. We will move on to something else. Because otherwise, we will just keep iterating and shopping away at it forever. And we will never ship it. And it’s got to go.

Carl: One thing that I’ve been following, I don’t implement it yet for my own product, but that’s growing in popularity. For example, Adam with town, I use it for a Tailwind UI. It’s Shape Up by the people at Basecamp. And they break it down into six week cycles. And they basically, the idea is you hard limit the six week. And then if you can’t make the six week for the whole thing, you just reduce the scope. And I’m doing that on a smaller scale. Even smaller, like week or two week fractions. But I mean for a complete version rewrite, that’s always a challenge. But I think that’s kind of why they’ve kind of avoided that whole rewrite with Base Camp. It’s because even my experience and as a consultant, those larger projects, it becomes really hard to capstone them. Like you’ve explained, and at least having that, helps with that.

JJJ: Okay. So every piece of software that we’ve tried to use, is Basecamp, or Asana, or do we track for WordPress or whatever else? The dev cycles, Agile development, all the things. There’s value in all of them. They’re all useful tools. There’re all fun. And they’re as fun and exciting as the people that you work with, right? But if there isn’t someone to capstone it, like you said Carl. If there isn’t a person that everyone looks up to and respects and says okay, you are done.

Carl: This is good enough. Just Saying good enough.

JJJ: Yeah. We will all take whatever the boundaries are, whatever the six week cycle is and go well, we had somebody out last week, so let’s bump this to seven weeks. So let’s, we were in the middle of this and we’re really close, so let’s do another week on this. And we all just, it slides and it slides and it slides. And there needs to be a person.

Carl: That’s kind of why I resonated with the six week idea, is that’s fixed. So if somebody’s out, you just reduce how much you would’ve done in the six week, as opposed to bumping the time up. But again, it’s such a, like you said, it needs to work for the people and everybody. Everybody has to be on the same page. It’s much more important than whichever system you pick. It’s as long as everybody’s comfortable with the system. But you always need somebody that’ll be like, if it’s not the system that tells you you need somebody to be, this is enough. But also, one thing I was curious about while you were talking is, and that’s a bit WooCommerce specific too. But what were the caveats, or when you were working on EDD 3 and you try to focus on backwards compatibility, what were the basically common, not necessarily mistakes, but things that people did that affected your ability necessarily, or their ability to maintain their code, with upgrades.

Because I feel like that’s a challenge for any third party add-on developer. Anybody that’s working with WooCommerce. Whether it, any e-commerce or platform as well, like EDD or WooCommerce. It’s using the right things, the right tools. And sometimes you want to take shortcuts and you’re, this is great. Except you end up in a situation like this, where you’re going to be in a lot of pain later, for maintenance.

JJJ: So one of the things specifically that came up, was the EDD product class is like this giant super class that has all of the things that pertain to a product. Taxation.

Carl: Oh, it’s a God class.

JJJ: Right? Yes. It is huge. And in an attempt to make that class flexible, which is the right thing to do, particularly in WordPress land, it was loaded with filters that had a ton, and a ton of filters in it. And that’s great, because you can extend anything you want. And you can make it do, whatever you need it to do. And that’s something that all of us do all the time, with our own code, to try and introduce some extensibility, make it flexible. But the problem there, is that filters let anyone do anything. And it is every single filter that was in that class of, there must have been a hundred or more. I mean, there were methods and they all had hooks in them. And so when we switched, we basically did kind of the only thing you could do, which is we could have, if someone still uses the EDD product class, it will work, and it will load, and it will do all the things it needs to do.

But we switched it from all of its internals, like orders, getting querying, the orders for a product, querying the customers that purchased that product for reporting. There’s all these touch points in this product class, that all had filters on them. And they were basically just impossible to ensure that we could maintain backwards compatibility with everything that was filtering inside of them. They were filtering like query parts, and it was just insane. Where it was all good intention. They were all good ideas, and they all made life easier for someone, at some point somewhere. But for all those good intentions, it really made trying to, refactor bolts, redo that class in a strategic way. Really, really difficult. We spent a lot of time on that, which is where we ended up with these middleware classes that their job, their only responsibility is to look for someone trying to access something that should exist in 2.x and that doesn’t exist in 3. And then just hot load it in there.

And then, if somebody accesses something into the 3.0 way, that we can make sure that it’s magic in the 2.x way, so that no matter which way they’re facing, that the data will be there in a way that they will expect. And it’s the hooks and the filters was the hardest part. Everything else you could kind of code around.

Carl: I mean, I find that a challenge myself, trying to design. Because there’s a pressure I find as, I mean, you must know this more than all of us, but there’s a pressure to make everything extensible. But making everything extensible has a cost for our maintenance. I mean, I’m always super shocked. I mean, it’s one philosophy to deal with it. But I’m always so shocked when people use filters and things like that. And then they don’t validate anything, once the data comes back. They’re just, oh yeah, it’ll come out fine, it’ll come out fine. Nobody’s going to mess it up or anything. It’s not going to blow up the entire application. Or I mean, fair enough, you blew it up, it’s your fault. But at the same time, I’m a developer that likes to have type safety. That likes to make sure that things are okay. And every time you add these extension points, you’re complexifying the whole thing. So I tend to push back a lot more on that. And I don’t know, did that philosophy change a bit with EDD, with the version 3 migration?

JJJ: Yes and no. My philosophy with WordPress hooks, is really that they are an application layer piece of functionality. Berlin DB has some, but they’re there on purpose. Where the hooks and filters are things that should happen pretty late, after things have happened. Or super early, if you want to no op or nullify something from ever even happening in the first place. Once there’s something in the middle, once there’s the WP query class in WordPress that has all these bi-ref actions and things on the query parts. You’re like man, that’s cool and it’s useful, but you can never really get rid of it. You can never change it. Because it is integral to the deeper underlying function of that code. And that’s when it gets really difficult to maintain going forward. In my opinion, my experience.

Carl: Yeah. I get it. But at the same time, I’m thankful for it. Because I mean, my serverless product probably couldn’t exist without the level of deep. Even at the PHP level, I go in and try to fake things, because of it. But it’s hard from a maintenance perspective.

JJJ: It is hard. With WordPress plugins that are your own, or if someone’s listening and they’re writing their own plugin, it’s more important that you have a consistent pattern with how you use actions and hooks and filters, that people can anticipate when you’re trying to build extensibility into your plugin, you can’t just, I would suggest not anyways, littering hooks wherever you want them. Don’t even put them in at first. Wait until someone needs one. And then, with bbPress, we always have filters on get functions. They’re always at the end of the function. They always type the cast after apply filters gets run, so that you’ll always have a consistent boolean, or array, or whatever. So no, even if they return some crap object that the application won’t die or fail, our goal as developers is to keep all this running, right? It’s not.

So if we, the best you can do is typecast after a filter gets run and all those things, I totally agree. And those are lessons you don’t learn, until you get burned by it. You’re like oh yeah, that would’ve saved us a ton of trouble, to just cast that right at the end and return it or something.

Carl: And, yeah. See, I didn’t even learn it from, I just observed a lot of people. This is kind of my first product, and I’m kind of like you, what you said. I went with zero hooks or filters. And I add them very, I need to know what you’re trying to do, because I think that’s a problem. Like you said, it might be tempting to just put it everywhere. And then you’re like well, I don’t have to worry about it, they can just use the hook, and things like that. But it’s a burden on you, the developer, in the future. I like saying, you have to think about future Carl, but you know, future Till, future Trip J. And that’s problematic. And honestly, it’s not discussed a lot, but I almost, maybe one day I’ll write about it, once I feel more confident about the topic. But I feel like it’s worth discussing, because it’s a huge burden a lot, long term.

JJJ: Well, and WordPress is a project, there’s different patterns all over the place. And any open source project is going to experience that, where people come in and they contribute. And people go out, and then someone comes in and they have their own way of doing things. And so, if you pay attention, and if you care about the history, or if you enjoy sort of digging through WordPress’s source code, you can identify, you can pick stuff out, and you can see a little bit of people’s approaches, in the code that they write. And so it’s all just the approaches that we take. If as soon as people start using, you’re sort of stuck with it, right? Unless you really abandoned it, and reintroduce a new version or something else.

And so, with Berlin DB, we really, I really wanted to kind of get it right. Which we all do, right? We all want to, we all start out with some, we’re going to do it right this time. And then, eventually as we started implementing it into EDD and Sugar Calendar, I was like oh, well we could, this could be extensible. This could be a base class. This could be a trait. This could be this. It’s kind of, it all starts to turn into different pieces. But thankfully, because custom tables were something that I’ve been dealing with and staring at with WordPress for so long, that really the bulk of getting Berlin DB built, and working, and functioning, really truly did not take me very long, in terms of hours.

Carl: Well, I mean, you’re the domain expert. You probably had, you had five previous implementations in the back of your head, with all the previous drawbacks of each.

JJJ: That’s true. So it was fun. And I really do think that Woo is going to need to have custom tables for pretty much everything.

Thanks to our Pod Friends Weglot and Nexcess

Carl: Yeah. So they’ll need a framework like Berlin DB, like an API, because that’s basically what Berlin DB is. It’s an API for custom tables.

JJJ: Yeah. Eventually the pattern breaks down, where it’s the same for everything, right? Get users, get posts, get comments, get term, all these functions end up being pretty much the same. And the closer that they are to being the same, is the better all of us are. But if you just had one function called get thing, and you had to tell it where to go and what to do, would that be better or worse? At what point have you abstracted it too much, where it’s not useful? And at what point is it familiar enough, where everyone knows how to navigate it? And so finding that balance is another one of those things, where as Berlin really tries to make it easy, but at some point, this isn’t easy.

And so how do you, someone has to take responsibility to write the database table schema, and put the class and put the indexes where they belong. And decide that things are going to be what they are. Eventually we’ll get to a point where this code is just writing itself, right? I get that we’re at the verge of AI just solving all of our problems for us.

Carl: I don’t use GitHub, or pilot, Autopilot. But apparently it’s magical. I keep hearing-

JJJ: It is magical. It is incredible. Get GitHub. It’s spooky. It’s not right. It’s very uncanny.

Carl: Yeah. That’s what everybody that uses it, my friend Patrick uses it and he was using it yesterday and all he could send me was these head explosion emojis, basically all the time. And he was, I don’t know if you did an update, but it’s even crazier than before.

JJJ: It’s nuts. It’s absolutely nuts. It’s really spooky, because I feel like there are very few moments in computing, at least for me, where I’m like wow, that’s different, that is cool. Like the Apple’s new thing, with the universal control, or whatever it’s called. Man, it works amazing. It works so good.

And I know that it’s just Synergy, with Apple sugar on top, or whatever. And Synergy was great too. I used Synergy a long time ago. But universal control is just magic. You’re like, that’s is perfect, they nailed it. And so yeah, Co-pilot is weird. It works in a way and it does what it does in a way, where you’re like, how did it know? How does it know? You can ask it WordPress questions, and it knows the answers to things, and it writes get option, and gets the option out of the thing. And it’s unbelievable, it’s weird, it’s spooky. Everyone should try it, but it is the future of stuff.

Carl: I still haven’t tried it. I mean, I think they have it on, I always keep thinking it’s only on VS Code, and I use PHP Storm, but I think you can get it on PHP Storm as well.

JJJ: I think you can.

Zach: Well, and it’s Arthur C. Clark’s first law, right? Any sufficiently advanced technology is indistinguishable from magic.

Carl: That’s true.

Zach: And that’s where we’re at now. Photoshop, content to where fill, that is magic. You select an area, you tell it to fill it, and it with a background instead of a person. That’s magic.

JJJ: Well, even David Copperfield, the magician, I think somewhat famously gave an interview somewhere. It was to Vox or somebody, where he basically said the stuff that he was dealing with when he became famous in the eighties and nineties, was more tech than magic. And that if we could see the tech that magicians are using, and we understood it and knew it, that we would probably be pretty freaked out. That the stuff that really high level magicians are doing with tech, it’s not just deep fakes, and it’s not just deep fakes video audio, it pretty much imagined really spooky, scary stuff, and that magicians, they’re a part of that in a way that, because their whole livelihood is based on the very strategic implement of fooling people, and espionage type situations to pull the wool over your eyes, and make you think that the world you’re looking at, is the world you’re not living in.

And it’s just crazy. when we’re talking about custom database tables, and EDD, and WooCommerce, I think to myself, we are not building AI, we are not launching rockets, we’re not launching satellites that have to orbit the earth. I’m glad that we’re passionate about WordPress, and I love that we all love what we do together. But the majority of the problems that we’re solving are really, really small potatoes. And that we can kind of just make a decision and move on, and it’ll be fine either way.

And I love that we debate about it. I love that we care, and I love that we care about the details, and the syntax, and the approach, and the names and everything. I love it, but we can kind of just pull the trigger and move on, and solve bigger and more difficult problems.

Zach: Absolutely.

Carl: Yeah. I mean, I find there’s like a balance. It’s like anything, there’s always a balance, because-

JJJ: There totally is.

Carl: Especially you would know more, even more than any of us here. Maintaining open source work is a long, it’s a commitment. And I was having that discussion actually in the Post Status podcast with Tanya. And I was just saying, there’s very little work. If for example, you disappeared tomorrow, now there’s a bunch of developers probably helping you with bbPress and BuddyPress. But there was a time where it was just you. And if for some reason you were plucked out, little aliens came in and they’re like oh, we want some Trip J action. And they’re like, boo, boo, boo, boo, and you were off. The project’s kind of, not dead, but it’s basically a zombie. And there’s a lot of open source project. I feel like there’s not a good discussion on continuity of a lot of things that we use, as professional.

JJJ: With WordPress as a project, it’s mission for democratizing publishing. And relative to SpaceX, where the mission is democratizing space travel. I think it’s such an open ended, and sort of amorphous motto, where you can apply whatever you to it. And I think at first, democratizing anything sort of means just letting anybody be able to publish on the internet. And let anyone use WordPress and everything else. But I think that there are so many layers to democratizing the software that we use, and that we build, and that we write, that continuity is a part of it. Where what happens when one of us bows out, or disappears. And ideally, the licensing and the open-source-edness, and the democratization of it, ideally lets someone pick up the ball and keep running.

When Alex Mills passed away, Viper007Bond, a bunch of his projects end up at Automatic, and other people maintain it, and some stuff lives on in WordPress core itself. And we’re going to see more of that, as this time goes on. We won’t see less of it. And so yeah, it’s an important part of it. But I really think that part of being an open source, a good open source steward, or living the lifestyle I guess, right Carl? If you’re in it, you’re in it. And this is, this is part of how you feel. And who you are. And you don’t have to be that way, right? It’s totally fine if people are not living the lifestyle.

Carl: And it’s okay if you want to move on too. That’s the hard discussion too.

JJJ: Totally.

Carl: I mean, for me, the most vibrant example is probably the make of RETAS. And at some point, after a decade was just, I mean, I started, I’m paraphrasing, but he was just saying, I started this because it was a cool project that I really liked, but eventually things grow out of scope, and they grow in size. And he became a maintainer, and not so much a developer. And that was not bringing him joy to his life. So he found a way to move on, and we have to be okay with that as well, with anybody.

Till: And it was 10 years later. He’s been doing it for 10 years, or however long it was. 15? I don’t know. No, probably 10, that at some point, I think it’s also just time to move on and you want to go a new adventure. Did this like seven year travel bug that some people have, like me, every seven years you want a bigger change.

But John, I’m curious, since you’ve been doing so much open source work for, I don’t know, 20 years.

JJJ: Just about, yeah. Just about 20 years.

Till: Have you ever worked on, before Woo, Moveable Type, or have a Mint, or something like this? Or was it always just W?

JJJ: My first intro to coding was when I was probably 12 or 13 years old, was Visual Basic 3.0. And that it was, looking back, that was my favorite code I think I’ve ever written. Because it was just ridiculous, and so stupid, but it was amazing in all the wrong ways. And then I eventually, right out of high school, got a job working for a company in Sussex, Wisconsin. And they wanted to rewrite all of their like old DOS based billing and inventory stuff, in Visual Basic, and Microsoft Access, and all this Microsoft stuff. And so it was probably 97, 1997 was when I started coding and building that. And I was the only developer, in a room by myself, coding this thing. I think after a year and a half, I just stopped going, because it was so depressing and sad. And couldn’t go to this room anymore and sit there.

And then I was, I don’t think I want to be an engineer anymore. It’s not for me, it’s not fun, it’s not whatever. And so then, had some odd jobs and still kept up with web development. Coded some Flash stuff that was fun, and really enjoyed Actionscript. And was into cars for a while, and people were building car forums. And started looking at how the licensing for Envision Power Board, or V Bulletin, or all those older forum pieces of software, they were not open, they were expensive, and you couldn’t just turn it on and go with it, you had to buy it. And then, ended up bumping out, because I was just a poor high school aged kid. And so I started looking for free and open source forum software. Free initially, but not necessarily caring about open source so much. And then found phpBB, in the 2.x days.

And it had a lot that was in it, it was pretty cool. But then, and that’s how I learned PHP. It was very similar to VB, where syntactically, it wasn’t that much different. And I could understand it pretty quick, and got into it. But my experience trying to contribute back to the phpBB project, was not super well received. Not that it was bad or anything, but it was just, it moved pretty slow, and mods were weird, and they didn’t really have a good plugin system, or anything for it. And there was a French developer named Pierre, who I’d only knew as Pierre, who had forked phpBB, to add plug-ins, and a calendar, and hierarchical categories to phpBB, and I was like oh, this is way better.

And that’s where I kind of got introduced to the idea of the GPL and forking software. And I was like oh, you can just take it and run with it and it’s totally fine. He’s like yeah, you can pretty much do whatever you want with it. And so then I started sort of building and learning PHP, through forking a lot of his stuff. And took it as far as I could, until he essentially did the same thing. He’s like yeah, I don’t have time for it anymore. I’m not really into it, the way that I used to be. And the project is basically in maintenance mode. And I was like, oh man, that really stinks. And so then I started looking, what software is there, that has hierarchical categories? What allows me to have infinitely listed categories, to put topics or posts and things into forums, and found a WordPress at first. Didn’t find Drupal, or bbPress, I found WordPress. And then was like, oh well, WordPress is cool, but it doesn’t have forums.

So then found bbPress. I was like oh, bbPress is awesome. It’s forums, but it doesn’t have pages, or website stuff. So how can someone have a full on website, by having bbPress and WordPress together? And I was like oh, well we can kind of bolt these things together pretty loosely. And then back in the day, Peter Westwood, Sam Bowers, Matt Mullenweg, everyone was super receptive to oh yeah, just upload a patch, and that makes sense. And we’ll review it and we’ll shred through it and go. And so Mike Adams, or Mark Jaquith, all the, Ryan Boren and all these early WordPress people were like, go, go, go. If you’re working on this, and this is stuff that you need, and you’re able to facilitate some improvements to these pieces of software, that not a lot of people are really paying a whole lot of attention to.

And WordPress had just absorbs Drupal’s approach to taxonomies and stuff at the time. And so bbPress had just inherited that approach, but had improved it a little bit. And so there was just a lot of moving parts, and a lot of stuff that I needed, early on. And from there, I was like man, it is a cool group of people in IRC hanging out. And my contributions are making an impact, they’re ending up and releases. We’re helping really set the pace here, for the software. And so that was probably 2006, 7, 8, 9. I think I actually made a WordPress.org account, only to patch bbPress or something, back in the day. And from there, it was like man, this is kind of what I see myself doing. And eventually, did what everybody does. Took on client work, took on, learned how to pay my taxes. Learned all these things and now that I’ve been doing this for a little while, what do I actually enjoy out of all of this?

What are the things that are the things that I think need attention, but also the things that I really enjoy spending my time doing? And I think from my previous experience doing VB, and Access, and building a lot of the backend type of stuff when I was a kid, that doing my SQL of database stuff has just been a thing that has always felt fun for me. And watching applications that scale horizontally or vertically. Wow, it’s finally working the way that we all wanted it to work, is just a really rewarding sort of experience. And then it’s, I enjoy the tough problems. When we were working Sugar Calendar, trying to do recurring events, and querying the database for those, and getting everything to work correctly with the eye calendar special, and getting feeds and stuff working. These were things that, there are some PHP pieces of software that do this, but there’s not a lot of code to look at, for examples, to get an idea, or a sense of scope of how difficult that a problem like that’s going to be.

Carl: A decade ago, I had to build Calendar software on top of WordPress. And I actually had to read papers on the topic. Because, without going into the weeds too much, because we have to wrap up eventually. But just finding how to schedule events, so there’s no overlapping times, blocks of times, things like that. How to store that in a database, in a way that you can query it, and find that information. This is when I was like on PostgreSQL, by the way. Because I remember coding so much of this in PHP. And then I just learned that basically in PostgreSQL, there’s a data type that just does that all for you magically.

I was like, if you could imagine my face so serious and frustrated. Because it’s hard. It’s a genuinely hard problem. Like forms, I’m sure if we had a person from Gravity Forms, or any of the form software. Forms, they’re people, the person that did Symphony Forms wrote his master’s thesis on it, because it’s a hard problem. There are hard problems.

Zach: They are.

Till: So after 20 years, I know this sounds like a very generic question, but I find this fascinating. And don’t say nothing, everything is perfect. That’s a lame ass …

Carl: Now you have to say nothing. You have to say nothing now. You jinxed it. You have to say it now.

Till: What maybe have you learned along the way, or what would you have done differently in the last 20 years, with this whole moving from VB to open source? What would you, what’s the gem of advice you would want to hand out to someone 20 years younger than you?

JJJ: So by a long time ago, well not that, maybe six months ago, I pinned a tweet. And my pinned tweet, is that I regret everything. So maybe the subtext to that, and my advice is … and I’ll tell a story, a two minute long story. It’s not a story that most of you haven’t heard, but I love telling this story. Prince, the artist Prince, played at the Superbowl, in the early two thousands, or whatever. And giant show, Superbowl Halftime Show and its Prince. And huge event, there’s fireworks and stuff. There’s all this crazy stuff planned, and it starts raining, and it’s pouring rain. And so Prince is backstage and you’ve got electric guitars, you’ve got headphones, you’ve got dancers, you’ve got giant speakers and amplifiers. What are you going to do?

How can we make this show like a success? And Prince’s answer is, make it rain harder. Just lean into the problem. Whatever people see as a hurdle, or something that is in your way, something that is a challenge that seems impossible. We’re all in this because we are solving hard problems. We’re all in it because we’re trying to build really cool things to help other people. So if we’re not in it to get rained on, so that other people don’t, or that other people are entertained, then why are we doing it? So my advice is just lean into the hard problem. When someone says, this is going to be really hard, this is going to be difficult, it’s going to take a long time. Or I don’t know if we can do this. Don’t be the person that says you can’t do it. Be the person that says we can do this. We can definitely do this, and we will find a way.

Carl: Actually, I’m always the person that says no. And then I come back and I said well, and people love me for it. So piece of advice.

JJJ: Perfect. That totally works.

Carl: That piece of advice. So say no, but be open to changing your mind.

JJJ: So I regret doing a lot of stuff, because I’ve definitely screwed up and made a lot of mistakes, and pushed stuff out of scope. And stuff has taken too long and made promises I couldn’t keep. And I regret a lot of things. I’ve had conversations with people that didn’t go well. I’ve made some friends and made some not friends. And there’s a lot of things, I regret a lot. But it’s because I’m chugging away, and trying to learn, and doing my best to make things better than I left them, so better than I found them. So that’s my advice, is you’re going to break stuff, and that’s okay.

Zach: Thanks.

JJJ: Yeah. That’s how I got into all of this. I broke a computer and had to learn how to fix it.

Zach: Yeah?

JJJ: A computer your parents bought you, and you didn’t want them to know you broke it?

Zach: Computer my grandmother had just purchased, and I was at her house. And I called my friend’s dad, who worked for Baxter NIT. And I was seven, and I had him walk me through reinstalling everything on this 486 DX266.

JJJ: Oh, grandma’s poor computer.

Zach: And that was how I got into all of this, is by being forced to learn, by breaking something.

JJJ: Totally. That was same thing. My first computer was a Pentium 100, a Packard Bell that my dad maxed out his Sears charge card on, to buy me. And paid for that thing for 10 years. And yeah, I broke it instantly. It was, I reinstalled some beta version of Windows 95 that was just full of viruses,, and just ruined everything. And just, I’m dead if my dad finds out, so I got to figure this out. And that was it, just figure it out.

Zach: And it’s funny, because I ended up getting individual Basic, and all of that. And then Flash and Actionscript. We’ve talked about that part of my journey before. So very similar path, and both of us ending up in WordPress is very interesting.

JJJ: It’s interesting. It’s cool though.

Zach: I have one last question, and it’s just a fun one. And that is, what is the Visual Basic app you wrote, that you would say is the craziest thing you ever made?

JJJ: Oh gosh.

Zach: And I’ll answer first. I wrote a Windows application that used the Windows API, to talk to AOL, and AOL chat rooms, and make rainbow text in AOL chat rooms.

JJJ: I wish, I mean, I wish mine was as innocent as yours actually. Because I did the same thing. Yeah, because it was AOL and whereas, and all those things had, unlimited potential to just do whatever you wanted to. So America Online was this program that came on a disk that you’d have to installation, that was notoriously insecure.

Zach: Oh, it was so bad.

JJJ: In every aspect of this piece of software. And so, and Visual Basic was a piece of software that was notoriously undervalued, because you could essentially do almost anything you wanted with it, within the construct of Windows. So I built a Trojan horse, right? Basically a virus, that would run in the background, and bit blit a screenshot of America Online when you would open it, and then overlay text boxes, where the login screen would be, so that I had a key stealer. And then would just get people’s passwords. And then behind the scenes, would email their passwords to a random email address, and just get people’s login credentials. It was just an AOL account that it would all go to, because you could just make fake AOL accounts back then. They didn’t do any credit card validation.

And we used those eventually, to get TOS accounts, because in order to have admin powers on AOL, you just had to have a screen name, or a username that started with TOS, terms of service. There was no other checking. And so you could, you just had a TOS account and you were like a super-admin across all of America Online. You could do pretty much whatever you wanted to. Custom AOL keywords, all the crap. So yeah, it was a long time ago, and I was a teenager, so I don’t think that anyone could prosecute for any of that stuff anymore.

Zach: No, no. We’re way past all of that. I did have the admin tools for AOL at one point.

Carl: This story’s so funny, because I literally tweeted earlier this morning, because it’s coming out like Okta, if you know what that is? It’s basically a platform used to secure a lot of enterprise soft services. I have consulting clients, you can’t log to get. And it comes out that it’s some 17 year old in the UK, that hacked them. And the reason I said that, is because I remember reading an article on info security, and they said the two most dangerous types of person on the internet, are organized criminals, and bored teenagers.

Zach: Oh absolutely.

JJJ: It’s totally true.

Zach: Absolutely. And I was a bored teenager.

JJJ: Yeah. And I was too. And it was never malicious. I never did anything to hurt anyone. It was never, it was all just exploring what I could do with Visual Basic and AOL. And there was tons of stuff. We had key gens scrollers. I mean, I really get nostalgic when we start talking about AOL, because there’s so many terribly great pieces of software that have completely gone missing from the web.

Carl: Completely lost on Canadians.

JJJ: Yes. Lucifer X, AOL, Spiral, all these old, like AOL programs that as a teenager, you needed to get online. It was ridiculous. It was so stupid. But it was really educational, in a way that I feel kids don’t have a sandbox like that these days. Kids don’t have a place where they can, even MySpace was the last time that I think anyone had a place to dump some JavaScript and CSS, and hack a website, innocently. There isn’t that really anymore, but it was so much fun. It was so stupid. And there were a lot of people doing a lot of really bad things. But it was just fun to learn how to code stuff, and subclass stuff, and break stuff. And it was, I’m glad that, it was one of those things, where I begged my dad for a computer, and he had no idea why, what it was or why I wanted it, and just thought it was dumb.

And it was one of those things where gosh, I really have to make this worth it. Because we didn’t have, just like everyone else, we didn’t have much growing up. And I lived in a small town, where there weren’t a lot of people, and not a lot of people were into this kind of thing. And so, a lot of, but it was good. Again, it was one of those things where you lean into the adversity of it. You’d be the only one who’s going some direction, and eventually, you have carve a path, and people follow, or whatever, I don’t know. Whatever the metaphor is. I’m glad to be here with you guys, and hanging out, and talking about this stuff is always a lot of fun.

Till: I’m going to share my illegal things after when we’re off air. But I’m going to be the person right now to say, enough is enough. Because we’re almost at an hour.

Zach: Absolutely.

Carl: I feel like that was a good capstone.

JJJ: That was a good capstone.

Zach: It was. Where can people get in touch with you, if they’re wanting to do so?

JJJ: Twitter and Github, @jjj and jjj.blog. Where I should blog more, like the rest of us, and don’t really do it as much as I should.

Zach: Okay. And you know where to find Till, Carl and myself. Thank you J Trip for joining us today.

JJJ: You’re welcome. Thanks for having me. Always good to see you all.

Zach: Looking forward to seeing you in person, hopefully in the near future.

JJJ: Same here, likewise.