Aaren Quiambao

Welcome to Coding Over Cocktails, a podcast by Toro Cloud. Here we talk about digital transformation, application integration, low code application, development, data management, and business process automation. Catch some expert insights as we sit down with industry leaders who share tips on how enterprises can take on the challenge of digital transformation. Take a seat. Join us for a round. Here are your hosts, Kevin Montalbo and Toro Cloud Ceo and founder David Brown.

Kevin Montalbo

Welcome to episode 39 of the Coding Over Cocktails podcast. My name is Kevin Montalbo. Joining us from Sydney Australia is Toro Cloud CEO and founder David Brown. Hey David, good morning.

David Brown

Good morning.

Kevin Montalbo

All right. And our guest for today has a PHD in software engineering from the Carnegie Mellon University. He is the author of the book “Just Enough software Architecture” and the Pragmatic designer column for the I TRIPLE software magazine. He's also a software engineer at Google. Joining us for a round of cocktails is George Fairbanks. Hey George, welcome to the show.

George Fairbanks

Thank you very much.

Kevin Montalbo

All right. So you've been working at Google as a software developer for the past eight years. Can you tell us about your experience working there and some of the projects that you've worked on?

George Fairbanks

Sure thing. You know, this is a great time to mention that I'm here. Speaking on behalf of myself, my opinions are my own, you know, the standard sort of spiel here I've been there for eight years. I worked on a bunch of different products. I was on ads at the beginning, I worked on supply chain projects and I got to work on the virtual reality things too and that was a lot of fun. In the last couple of years, I've switched over to an education role. So I currently lead the software design education team.

David Brown

What prompted you to join Google as a software engineer because as I understand it before this, you had your own research organization and consultancy So what decided made you go into going back into developing software itself?

George Fairbanks

Well, there's two reasons, you know, in the nineties, it was a perfectly viable thing that if you got pretty good at something in the software world, you could hang out a shingle and become a consultant and,, there was plenty of training,, that, that companies were hiring people to do and so forth., but that got harder and harder., that most of the people that I know that were in the business,, they saw the recession in 2008 as being very tough. And,, so they, they hung on so much like I did for a bit and then I switched to being the tech lead on a project for one of my consult or my projects that I was consulting for. And at that point I remembered how much I liked building things. And when you're a consultant for a while, it's kind of neat to like see another client and get them up to speed on some ideas and help them out. And there's a lot of that feels really good to me. I really enjoy that but boy do I like building stuff. And so I, I figured this was a great opportunity and said you should go talk to this recruiter at Google and, and the rest is history.

David Brown

I suppose it both works hand in hand quite well. Get you keeping your, your hand in the software development stuff and, and building stuff whilst writing and talking about software architecture and, and issues associated today around software development. I guess keeping your hand in it,, helps. 

George Fairbanks

Right., you know, it's exactly right. I mean, even imagine you have the clearest mind possible, which, you know, none of us do. Right. But let's imagine you had the clearest mind possible. Just getting your fingers on the code and being part of a project, it, it forces you not to gloss over or, like, trivialize things that are actually really hard and it also makes you understand when things are really changing. So, I mean, we're, we're going to chat, I think a bit today about the changes that have happened in the last 2025 years. And if you're in there in the trenches doing it, it feels very different. And so you can reflect about what was it like. How did I do that in 1995 versus how would I do that same thing today?

David Brown

Yeah. Well, let, let's start getting into that. So you, you, you talk a lot about software architecture and technical debt amongst other issues. So we're going to focus in on a couple of those topics today. Technical debt is a phrase which is thrown around a lot these days. It seems to be one of the buzzwords of software development industry at the moment. How was, how has the definition of technical debt evolved since the phrase was first coined by W Cunningham in 1992.

George Fairbanks

You know, that's a really interesting topic., I think most people now use the phrase very fluently. Right. We just, we talk about debt in our, in our programs and what's more, I think it's very healthy,, that we are talking about. Not only what features we're getting out the door but what kind of trouble that we're causing,, to our, our programs over time. So, I think all that is very good. But I tell you what, we already had a perfectly good name for a mundane software problems. We could just call them a mess, right? We didn't need a special term for that. So when Ward wrote that original experience report, he wasn't talking about just mundane run of the mill, I cut some corners and guess what I'm paying for it today. He didn't need a special term for that and you wouldn't have gotten published for, for saying I cut some corners and now I'm paying for it, right? What he was talking about was something that I find extremely interesting and it gets at what, what is so exciting about being a software developer, you're confronted with a problem and you get to create a little world inside your software where there's different entities where represent them as objects or as data structures or functions. 

But you're, you're manipulating these things and the way that they combine is related to how you understand the problem that's being posed. So when word came up with this idea of technical debt. He was in a situation where he's working on a financial product and he needed to explain to his boss. He said, well, you know, I wrote this code several months ago and when I go back and I try to add new features to that code, it's not so easy and it's, it's still working. I mean, that code still makes us money and all that stuff. But when I go back to that code, what I'm seeing, this is what how I interpret word is saying this. He's saying that the code is speaking back to me. My ID is from several months ago and my ideas have moved on when he talks about financial products. You can imagine that if you didn't understand finance before you start out with a very simplified rudimentary model of finance. But then there's these waves of realization where you start to go., you know, debts are actually like these other kinds of financial assets in some deep interesting way. And yet your code doesn't say that back to you. Your code is still very simplistic. OK? So what he's trying to tell his boss was what we need to do in order to keep up our feature velocity in order to be able to say here's a new requirement. I just want to code it up and ship it out if I want to keep up that cadence. Well, the software has to meet my current thoughts. OK? It doesn't just have to work, which is what the boss is concerned about. Like it's profitably making money for the company. It also has to be in a shape that the developers are ready to add that incremental feature. 

And so that's why he had to coin the term technical debt, right? He said I don't want to stop the world do a big waterfall, analyze all the requirements and then give you a beautiful system a year from now. What I really want to do says Ward is I wanna shift this code say every couple weeks or every month. But when, as soon as I ship that code, it freezes my thoughts at that moment that I shift it shipped it. OK? And so this term that we now use all the time refactoring doesn't even appear in this paper because it wasn't a term that was was current, right? This is 1992. So he's just trying to explain the basic idea that we we now understand quite well you ship some code, but in order for you to keep up your feature velocity, you have to keep refactoring the code. So it represents your current set of ideas. Otherwise you're gonna find it harder and harder and harder as you try to add each new feature that comes in the door, right? So that's what the technical debt is. The technical debt is not Yeah, I have bad judgment. I took shortcuts or, you know, I deliberately, you know, put in something ugly in order to meet a deadline. Yeah. Yeah, that's straightforward. What he's talking about is a much more subtle thing, which is my ideas evolve. The ideas get represented as source code. And if I want to keep in chipping away at the problem, shipping a new rearvision every week or every month or something like that, well, the code has gotta be changed over time. It's got to catch up with my thoughts and that's the debt. The debt is the difference between how you're currently thinking about the problem and what you're seeing when you look at the code.

David Brown

So that's interesting. So that, that, that's to suggest that technical debt is a naturally occurring process and it's not something necessarily that she should be punished for because your knowledge is always going to evolve. So your knowledge of the project, the problem you're solving is always going to evolve. You can't change that. So it's your thought process of where you were several months ago to where you are now is always going to be different. So that suggests that technical debt is a natural part of the development process.

George Fairbanks

Yeah, that's absolutely right And I think that's exactly why it's interesting because if you say we have bad developers or we have developers with bad judgment or we have developers who are forced to code to a deadline and therefore took shortcuts. It's not surprising when bad things happen. But if you say here's a team of great developers who are following best practices, who understand their craft and are doing everything right? And you still don't have the software that you want. Well, now that becomes very interesting, right? You really want to get it, what is, what is going on here? And when Word Cunningham was original talking about this, he was comparing what most people were doing, which was waterfall development. 

And he was saying, actually, I think you're gonna have better results if you chip off some requirements, ship that code chip off some more requirements, ship that code. But explaining that the code that you wrote in the first month is gonna have to change, it's gonna have to evolve as time goes on and that I find managers are still surprised by. So what I mean by that is they look at their sharp developers on their team and they know they wrote that code six months ago and they kind of don't want that code to have to change, right? Because they're like, were you a knucklehead six months ago? No, I'm not a knucklehead. It's just that we now understand the problem better or we've added new requirements. That would say, well, if I'd known all about all these requirements, I wouldn't have written it that way.

David Brown

Well, I mean, software architecture has been evolving over several decades. And when you have, you know, like as you say, we were going from a waterfall process to microservices and iterative updates with an agile process. Recently though you've been arguing that it's getting harder to design well, architect applications when developers are focused on this iterative development, incremental changes that we're going through with, does that mean agile software development is incompatible with well architected applications?

George Fairbanks

No, I really don't think that's the case. I think that what? Well, first of all, it's very difficult to characterize what Agile is these days, right? There's a, there's a lot of different dimensions. Even back in the nineties before Agile was a thing. People were talking about iterative development. That's what Ward Cunningham was talking about in 1992. If you go farther before that, you find Fred Brooks talking about it in 19 seventies. OK. You can go farther back into the sixties and people are saying, you know, we probably shouldn't build software like we build bridges. It's just not the same, right? We have opportunities with software that we don't have with bridges. So if we just take the iterative aspect and we say is iterative development compatible with software architecture ideas, I would say absolutely OK. This is, this is perfectly fine. But what you find a lot of times is people especially early on say in the early two thousands, we were thinking about how to get agile and architecture to go together. They would not be genuine in how they would try to stick them together. They would say you should do iteration zero and iteration zero is like we're gonna do waterfall and we're going to design the architecture and then we're gonna do iterative development. And I'm like, yeah, come on, that's, that's kind of cheating. You're not really figuring out how to make these two things go together. So believe it or not, what I've been inching towards in these I triple software columns is the idea that's very compatible and you see it in extreme programming back in the late nineties. OK? And that idea is that the team has an idea, a shared idea about what the software design is. OK? And every day as they write software, they're, they're subtly redesigning that software. OK? Sometimes those redesign changes are completely mundane. 

Let me just use a quick example. Let's imagine you had an audio editing program and you have the facility for putting plugins. So like audio plugins, right? So this one does a compression and this one does noise reduction or so forth, right? If you come in and put another audio processing element into this framework, no big deal, you're not really changing the architecture, right? You already made that decision. You said I'm gonna need these plugins I need to sequence the plugins. I just put another one in. Ok. That's not really an architectural change. But if there was something else where someone said, you know, we've currently been doing all this stuff on a desktop and suddenly we're now in the web and I want someone to be able to do the audio editing on their phone except that the audio files live on the server. And you're suddenly like,, I am not ready for that. OK? There's many things that I know about audio. And there's many bits of the source code that I might be able to reuse. But really, that's a different kind of architecture that you're asking me to work on, right? And to begin with, it's a distributed system and the original one was on a single node, right? There's no remote procedure calls anywhere. OK? So, so the team has to come up with this idea about what the software is, what is the design of that of that code. And so the reason I've been chatting about technical debt is to get people thinking about how the ideas evolve and how the source code does it or doesn't catch up. And I want to talk about the processes because when we do iterative processes and so forth, it can make us focus on the minutia like I am making this poll request, right? I'm gonna make these three changes to the source code and I'm gonna send it to you for review and we're all done with it. It's gonna go into the repo and the continuous integration is going to push it off into production. OK? It can, those things can encourage me to focus on small things and that's not incompatible with architecture unless I don't have the ability to focus in and then pop back out and think about the big picture, the big design that the whole team is trying to keep in their head. So for me, I think that what XP was trying to do with having people pair up and move around so that the understanding of the system stayed in all the developers heads. I think we need to be doing the exact same thing today. 

The things I've been writing about though are talking about how I see software developers today having a hard time popping out. OK? And keeping the big picture in mind, which would include that architecture and the two big things that I find that are troublesome first is that focus on the patch, right? On the next little chunk of increment, like a new requirement comes in on Monday and I have to get it out the door by Friday. OK. And you and I and the rest of the team, we focus on how that work flows through the system and this is incredibly productive. It's a, it's a great practice and I'm not dissing that practice in any way. But what I am saying is if that practice becomes the only practice and then there's nothing in the way that your team organizes itself such that you pop up and the team has a big discussion about the big patterns and you suddenly go,, wait a second. We're still thinking one system and we should be thinking distributed system or something like that, right? If we don't have that baked into how our team cooperates, well, then we suddenly are, we're making it harder to do architectural concerns. OK. The second thing that's been going on and people sometimes put it together with the iterative part. But I think it's different, which is that we are increasingly looking to factory metaphors to inspire how our software teams sequence their work. And it's like, well, OK, that seems very much like iterative, but, but it's not in, in this particular way, if you think about a factory, a factory is doing the same kind of thing over and over again. Like you produce cars and you're going to try to move this car through and you're looking for opportunities to say, I produced this wheel in five minutes and I really like to produce it in 4.5 minutes. Or I would like not to have a whole pile of wheels and unfinished cars. Right? Well, I want to say, build the wheels just in time right. There's all these kind of metaphors that we're pulling in that say I would like to improve efficiency by decreasing the cycle time. 

And I want to improve efficiency by reducing work in progress, right? These are definitely factory and production inspired metaphors. OK? And as we chatted about before, there's a part of software which acts like a machine, right? When you're that boss and Word Cunningham is trying to explain to his boss that he needs to do. Factoring his boss is thinking about the code as a machine that's doing work for the company. OK, great. Why do I need to change the machine? So in that sense, the factory metaphors help us build a more efficient machine. But when it comes to the thoughts, OK, like how do I think about the problem? The factory metaphors actually in our relentless search for efficiency in our actions as software developers squeeze out the time that we would have normally spent popping up and thinking about the big picture and explaining to coworkers how everything fits together. And so I think those are things that we need to seriously consider. When we look at our software processes. If we relentlessly treat them like factory processes, we're going to be optimizing for, we built it up for a while, the code continues to work. But then at some point, it's just so complicated because I haven't refactored. I haven't done deep, hard refactoring that I end up with a big mess and it's hard for me to keep adding incremental features to it. And what I really get worried about is sometimes when I talk to senior managers, they feel like that's inevitable, that is the inherent nature of software. And that is what I'm really fired up to change people's minds about, right? What I would love to have people think is we have put ourselves into this difficult situation, you know, through good for good reasons. OK? Like the agile processes, the iterative processes, the factory inspired metaphors have all done wonderful things, but we have to add something else. That other thing allows us to think about the architecture clearly and to make sure that we are not letting it erode as we incrementally chip away with poll requests. OK? That we solve the features, we like win the battles but lose the war. I don't want people to be in that situation

David Brown

This popping up. The concept of looking at the bigger picture for practical purposes where in the agile process, do you see that occurring? Does it occur, for example, during the sprint review, does it occur during a quarterly plan? Is it an ad hoc process practically when do you inject that popping up concept?

George Fairbanks

Yeah. So I think if you ask different people, you'll get different answers and I think that's healthy. that is there is no single agile process, right. You know, you can, you can find several very reasonable agile processes. Ok. But let me go back to extreme programming. Now, extreme programming was tailored to small teams. Right. And so a lot of people are trying to apply agile and bigger teams and, and they might believe XP is not the one for them. But if you go back to XP, you'll notice that that Kent back was specifically worried about making sure the team understood things. He was breaking down the, the old ideas of code ownership. He said, look, we should all really own the whole code base. He was trying to remove the what we call bus factor, right? That only I understand this module. Nope, I'm going to pair with somebody else and now that person is going to feel like they own this module a little bit and they can come in and change this thing. And as we move around and we understand different parts of the system. As we go in and refactor, different parts of the system, the system should represent our thoughts and our thoughts. The team's thoughts should start to converge on a, on a design. 

So in a way, although there's not a specific practice, you can point to maybe the system metaphor in XP, but there's not a specific practice you can point to that says like that's where design is happening. You would find Kent saying things like I design every day and I agree with that. I think the teams that are, that are functioning well, are designing every day. And again, though, the warning flag that I'm trying to put out it point out is with our processes today, people are not thinking about how, how do I make room for or thinking about the big design? How do I think about is this code? Does this code have the right architecture? Right? This moment, if I don't have time to think for that? Well, well, maybe my architecture isn't going to be healthy, right? It's, it's gonna start out fine, but then it's gonna decay slowly. And I would love to see if we can change that. And I understand. II, I dodged your question a little bit because you're, you're asking a good one like when should we be doing architecture? I do have some specific suggestions but, but I'll let you ask your, your next question.

David Brown

1. Well, in your most recent essay, why is it getting harder to apply software architecture? You just describe how we changed our philosophy towards development in two specific ways. Can you run us through these changes and how they impact our software architecture?

George Fairbanks

Yeah. So I started developing software in, in production or, you know, in industry in the early 19 nineties. And so in the early 19 nineties, what I found was there were many of these same people that are very famous today in the agile community were very interested in software design, right? I mean, Kent Beck and, and Ward Cunningham, these guys were very interested in patterns and ST patterns, Ralph Johnson, you know, put out the gang of four book along with three of the folks. And these people found that the, their ideas about how to do object oriented programming were not making their way, they were, they were having a hard time getting those ideas into businesses. And so I believe that they basically terraform the environment. OK. You know, like that term from science fiction where you go to a planet and you put things machines in there to make a, a good atmosphere so humans can live there. They did that same thing to our software development environment. They went into the world and they changed how big corporate build software. At first, it was like tentative little steps of dipping their toe into agile and let's try three month delivery windows. OK? And you know, all sorts of crazy things like that. But now it is completely mainstream and we're actually looking for, we have automated deliveries and so forth. 

The two things that I see that the agile community set out to change in companies first was the shift from waterfall processes to to iterative processes, right? And so we just sort of take that for granted today, there are relatively few companies that are still doing waterfall and those companies generally have very good reasons for doing so. OK. Enough said the second thing are these factory metaphors. OK. That I alluded to earlier. The factory metaphors. And the idea of if it can be automated, you should automate it. If you can do it faster, you should do it faster. Have, as I said, have been incredibly productive. OK? And if you say what are best practices today, you should have a bug tracker, you should file bugs that represent feature requests. You should track those through. Like I've, I filed the feature request. I've coded the feature request. It is passed all the tests and and is being automatically pushed into production. Like these are all very healthy things that allow teams to be agile and reactive. OK? So those are the two big things, the the iterative development and the factory metaphors that I believe have have transformed. And so a developer from 1995 if we plop them into 2021 is just gonna have their head blown as far as how software is being built today.

David Brown

Well, you, you went on in that article to further explain how these two changes have made it easy though to pile up technical debt. So, whilst those changes sound really positive. And you know, as you said, in common practice today, how have they led to us piling up technical debt.

George Fairbanks

You know, the way that I realized that this was really happening was a buddy of mine. He was in a start up company and they were doing great, great work. They had identified a good market that they were going to make some money in and they had a good team of software developers that were making fast progress. But after 18 months or so, their software development velocity started to really slow down. And so my buddy went over and he, you know, sat with the software developers and when he saw what was going on inside the code, it was a massive pile up of technical debt, right? One way of saying this was that there had not been aggressive refactoring, there had been mild refactoring throughout that the team had not done a reasonable job of balancing their obligations to the company, which is like we gotta get features out the door with their obligation to the company that like I'm piling up so much technical debt that the next feature is gonna be twice as long for me to, to implement it. And then the feature after that is going to take even longer for me to implement it, right? Because things have gotten so swirled and and so forth and it was at that point that I realized that some that it was possible to be in a state at a project where the project was like a zombie that is, you're like, it's walking along and it's making forward progress and it's not obviously dead yet, but backing out of that is nearly impossible. I mean, I think that team basically had to rewrite the code. 

They had to reanalyze the problem and rebuild everything if they were able to save things. My friend was not optimistic that they were gonna be able to do that. So the lesson I took from that was that the practices that make us efficient in the short term, like these factory metaphors and iterative software development, being able to get software out there and find out how it works. These are all very good things. But it's like the Yin and the Yang, you, you need that second force which is the very large scale refactoring and keeping the design very clear in everyone's head and it's possible not to do that. And, and I think that's really common, right? You can find people that are narrowly focused, they get in and they can get the feature coded, but at the expense of some technical debt piling up that they maybe per perhaps didn't even perceive as technical debt.

David Brown

It's interesting. We went through a major reflecting of our own platform a couple of years ago and, and probably lasted 12 months or more that, that that project during that time, there's obviously very few new features coming out because if you focus on a refactoring, do you in your experience, there's a project like that where for the for the value of the future so that this becomes a maintainable platform, take you where you want to go in the future, you almost pause and see no new features, which as from a management's perspective, does it cause some sort of friction? It's like, hey guys, when are we going to see the ending? When are we going to start seeing new stuff? When are we going to be able to start seeing new features and adding value to our organization?

George Fairbanks

Yeah. And I realized that sort of when I explain it this way and I'm sort of presenting, there's like, the, the, the, the happy team is refactoring all the time and they're staying in this, this nice state and the unhappy team becomes a zombie and eventually throws it all away. I'm being a little too glib. I, I'm, I'm doing it to, to, to show the contrast I would still suggest, let here, let me, let me give you another story because the stories I think are very helpful. I showed up on a team and the team had just been asked to shift its focus to give you an idea. Imagine you were serving customer number one, but you're in a certain problem domain and then you were gonna serve customer number two. And you would think if you hadn't looked at the code. Wow. Well, I should be able to, you know, reuse almost all of this code. But when you dig into the code, you find out that unfortunately, an enormous number of decisions were made such that almost none of the code was reusable. Ok. Even though all you're doing is customer one to customer two in the same problem domain. And, when I looked at that, I thought about this in a slightly different way that agility means you recognize that the music is going to stop, the requirements are going to change and you want to be able to react to that change. So the metaphor I came up with was the musical chairs, right? Is that when you're going along on a project, you don't know when the music's gonna stop, but you need to have a chair when, when that music stop. OK? You need to be ready for when that happens. And readiness means like if this unforeseeable or let's say low probability thing happens like we switch from customer A to B do I have any reasonable assets or do I have to start from scratch? OK. 

And in that particular case, the answer was no, but I would love to be in a situation where as the team is reorganizing the code continuously all the time, right? A little bit all the time that they're looking at things and going, you know, the the dependency between this thing and this thing, let's say between customer a and this, this source code is a little bit too tight. But you know, I could rework this such that, you know, going back to this audio example, right, that the audio filters are completely independent about from, from something else, right? Like all it really needs to say is there's audio coming in and there's audio going out and that's how filters work. Ok, great. I've decoupled it from customer A which means that when the requirements change, you know, I've got a chair, I can reuse 80% of the components I've got for customer B but, you know, like, like that's still, it's very easy for me to say that and it's, it's also very easy to imagine being in the situation you're in that you were going along. You thought you were doing the right thing. But,, you know, like the, the, the day came and you had to do 12 months of, of refactoring basically in order to to play the next requirement. Ok? To, to solve the next requirement.

David Brown

Yeah. And in some cases, it's not necessarily necessarily to play the next requirement. It's just that, you know, we're gonna have a problem at some point in time and, and it's just like we either take the pain now and pause and, and, and build something which is gonna be more scalable for the future. Or, or we create a problem in the future where, where we're gonna have to deal with a bigger problem Is there a way that we can completely avoid technical debt or is it an inevitable part of the process? I'm, I'm guessing from the, the, what we discussed earlier, you've already alluded to the fact that it's a change in thought process. So, I, it seems to me that it's almost inevitable.

George Fairbanks

I agree. I think it is inevitable. The sometimes it's interesting to take things that we think we understand and sort of play with them. OK. So imagine that my team is doing waterfall and I'm gonna do a waterfall process for six months, deliver something. I'm gonna do a waterfall process for six months and deliver it. I'm gonna do a waterfall process for six more months and deliver it. OK. So here's my question in terms of shifting our perspective, how is that different than very three, very slow iterations? And what I mean by this is the, the hope at the beginning of these waterfalls is that I take in all the requirements and I think about them hard and I design a system that will solve those requirements. So alluding to your scale question from earlier, you know, at the beginning of the iteration, you would say I'm going to build a system that will scale up to X users or X messages or X whatever it is. OK? So you already knew that requirement. But when I do one after another one waterfall after another, suddenly I'm sitting here with like a year's worth of code. When I'm going into that third iteration, I have years worth of code. And now you're telling me that I need to scale up by 10 X or something like this, right? You know, I need to be really ready. And I'm like, wow, you know, the technologies that I, I were totally reasonable in iteration one, which is the first waterfall or iteration two, which was the second waterfall. Don't make any sense. It also doesn't make any sense to completely throw away all the work and like build a new system at the beginning of every iteration. 

So at the beginning of the third waterfall. So my point being is that if you start playing with the sequence of waterfalls versus iterations, that nature of technical debt building up is actually inevitable, right? It's inevitable regardless of what process you follow. It's just a question of like when it starts to hit you. In fact, my dad worked for Proctor and Gamble and he made soap and he would have a variety of different kinds of jobs. But oftentimes the jobs were there is a factory that makes a tide detergent. OK? And now we're going to make the next variant of Tide detergent. And so they, he part of a group of engineers and the, the guys in the lab had created the new variant, but his job was to make it work in the factory. And of course, the factory was built 20 years ago. And so this is a technical debt problem, right? You're saying like given the factory that I've got, how can I make the new tide, you know, and so it, it all these problems seem to be inherent in the nature of design and it's the kind of problem that you want to have because the alternative is we went out of the soap business or you didn't have a scale problem because you don't have any customers, right? So it's a great problem to have, but I think it's, it is inherent, you're gonna see it no matter what happens over time.

David Brown

Can you run us through a systematic process in order to deal with technical debt? How can, how can teams implement a process to systematically clean up technical debt?

George Fairbank

I have an idea and it is not quite what you're asking for. It's not as grand as a systematic process for technical debt. most processes I see for handling technical debt to me are the generic game plan for the company has a problem. So the company's problem could be anything. But what you do is you identify it, you quantify it, you prioritize it and then you schedule work on it. Ok. And I think that is exactly what I see people doing on technical debt. Right. That, that, you know, if it's small enough I fix it this week, I just, you just, it never even makes it onto the backlog. Ok. But if it's bigger than that, I put it on a backlog and then I start to quantify those things. And I say, well, that one looks like a one week fix and that looks like a two week fix and so forth and then I prioritize them. And I say, OK, which one am I gonna do? Right? So, to me, this is common sense. I'm gonna suggest something else, which is not exactly common sense. It's actually the topic of the next I triply software article. the one that hasn't been published yet. So in this one, I make a comparison between technical debt and garbage collection. I know that sounds a little bit strange. But, but let's try it on for a second. OK? So imagine you have a program and I'm gonna exclude the programs that say statically allocate their memory and like, you know, but, but like if you have a regular program that allocates memory and then has to clean it up usually via garbage collection. OK? What you're, what you're doing over time is you're generating garbage but you have to clean it up and when you do go to clean it up. You would like the pauses to be as small as possible and as deterministic as possible. 

So you don't like if you have to have a pause, you'd rather not say the first one is for one second and the next one is for 10 seconds, right? You know, like if they're gonna be big, you'd write like them to be consistent. Ok. So let me compare that to technical debt. You've got a team and as a team build software, it unfortunately creates debt at the same time, you're gonna have to pause and clean up some of that technical debt. You would really prefer it if the pauses were short and you'd really prefer it if the pauses were deterministic. Not like it took me one second to clean this one up and it took me 10 days to clean up that one or in your case, like it take you 12 months to refactor for scalability, right? You, that's really not what you want. So the question is, how do I do it? How do I get good garbage collection? Well, the answer is we've been researching garbage collection algorithms for a long time. OK. And over time it's a very hard problem, but we've got better and better at making the pauses smaller and more deterministic, right? And there's even this thing called the no op garbage collector where you don't actually do any garbage collection and you say, well, that sounds crazy. Well, what if your process fires up, runs for one second and then shuts down? Ok. Let, let's say you're a little microservice or even even smaller, why would you even run a garbage collector on that? Right. Like the whole process is going to be gone in one second. So I don't even need to do anything. Well, it turns out there's enormous parallels again when you go over to teams, right? And you can say, what are the things that my team is doing to cause technical debt? Well, the first thing I could do is I could try to reduce the amount of technical debt that I'm generating. OK. And that could be like you, you stroke your chin for 10 seconds more before you start typing, right? Like maybe that actually you evaluated a few more alternatives and you just didn't allocate as much memory and you didn't allocate as much tech debt. OK? That's one thing. And the other thing is you could be a bit smarter about how you return to the technical debt and clean it up. OK. And this is close to what you were asking about before it was like, is there a place in my agile process when I'm supposed to return the technical debt? And again, I'm sort of hedging and I'm not saying there is a right answer, but I'm saying the different answers people come up with are analogous to the different garbage collection algorithms. OK. The garbage is gonna exist, you have choices about when you return to it and what you do about it like. So there's mark and sweep collectors and generational collectors and all kinds of different algorithms that heuristically seem to be good. And they probably depend upon how your program generates garbage, right? So when we go, when we look at technical debt and a team, what we're really talking about is what is the team's process that causes tech debt to be created and what is its process for returning to it and paying it down? OK. 

And I would think that any reasonable long term team process needs to have both of those to some extent, it needs to have some out of saying, well, let's not unnecessarily generate garbage. Let's not code up. The first thing that we can think of, let's let's think of two alternatives and choose the less bad of the two. OK? And then the second thing is if the team, no matter how good the team is at number one, if the team is not good at returning to the technical debt, whether it's weekly or it's, you know, every Friday we do tech debt pay down or whatever it is or maybe there's two teams, one team is generating debt and the other team is paying it down. Like you can imagine a bunch of different algorithms, the solution to tech debt lies in how teams run that algorithm. And I think that that's an empowering way of looking at it because the disempowering way of looking at it is to say tech debt is inevitable. Let's just crank up all the tech debt and eventually we declare bankruptcy and we start again. OK? And I find a lot of people think that that's just what you have to do. OK? That you, you might keep tech that down a little bit, but eventually the complexity builds up and the heat depth of the universe hits your project and you're done. OK? I don't believe that. I do actually believe you take a look at very large long term projects, say like the Linux kernel and it's not easy. But over time, human beings together on a team can evolve code and keep it healthy and safe. And I'm not trying to say the Linux kernel is perfect. I don't think anyone's gonna say that, but I will say it's a heck of a lot of code and a heck of a lot of distributed collaborators working on something, but they have processes to keep the tech debt low enough, right? We can do it too. I don't know what that is. Our projects. We can do it too. We just have to figure out the right process for it. So I would say yes, it is possible to be systematic and keep tech debt low. And if your team is not doing it. Look in those two places. Like, are you minimizing the creation of it? Maybe you could do some more stuff there. And second, if you're not returning to it, if you feel like it's still piling up, you're probably not paying it down fast enough. So that's my process. George

David Brown

Fairbanks. You're really good at analogies and metaphors and making this stuff seem really simple and easy to understand and with practical processes to, to achieve it. Where can people, our audience learn more about what you're writing about and follow you on social media?

George Fairbanks

Well, I am on Twitter, although I, I don't say very much over there at GH Fairbanks at Twitter because I wasn't fast enough to get George Fairbanks or the Real George Fairbanks or something, right? Yeah. I write this column for I TRIPLE software And that one you can find on my website, George fairbanks.com. Although if you do go to my website, please click on the I Tripoli one because they look at how many people look at it on the website And there's a book which is starting to get a little bit old now, my book on software architecture. But thank you so much. You've asked great questions here today and it's great to talk about something I'm so passionate about. Thank you.

David Brown

It's a pleasure. Thank you very much George and I would highly recommend people visit George Fair banks.com and visit that I a software link because it is very worthwhile reading. Thank you for joining us on the program thing.

George Fairbanks

My pleasure, thanks.

Kevin Montalbo

All right. That's a wrap for this episode of Coding over cocktails to our listeners. What did you think of this episode? Let us know in the comments section from the podcast platform you're listening to. Also, please visit our website at www.torocloud.com for a transcript of this episode as well as our blogs and our products. We're also on social media, Facebook, LinkedIn, Youtube, Twitter and Instagram, talk to us there because we listen, just look for Toro Cloud on behalf of the team here at Toro Cloud. Thank you very much for listening to us today. This has been Kevin Montalbo for coding over cocktails. Cheers.