Kevin Montalbo

Welcome to episode 42 of the Coding Over Cocktails podcast. My name is Kevin Montalbo, and joining me from Sydney, Australia is Toro Cloud CEO and founder David Brown. Hi, David!

David Brown

Good morning, Kevin!

Kevin Montalbo

All right, good morning! And our guest for this episode is a former consultant at ThoughtWorks, and now works as an independent software delivery consultant based in the San Francisco Bay Area. Most of his work centres around supporting and advising startup engineering teams and providing custom developer training. 

He enjoys sharing his opinions in conference stocks, podcasts, and writing, often with a focus on agile engineering practices. Ladies and gentlemen, joining us for a round of Cocktails is Pete Hodgson. Hey Pete, welcome to the show!

Pete Hodgson

Hello! And thanks for having me. Nice to be here.

David Brown:

All right. It's our pleasure. Pete, you described your passions as helping engineering organizations develop strategies to deal with the issues they face when they move past a single team and, or a single code base for that matter. What are the challenges that organizations can typically expect when that occurs?

Pete Hodgson

So, I see two different challenges that are kind of almost the opposite things happening. So the first challenge I see is chaos, right? So, I worked quite a lot with startups that have kind of gone through kind of like a lot of growth. And you know, when they were one engineering team, they had these engineering practices that worked fine. And then they've kind of suddenly ballooned into five teams, 10, 20 teams and suddenly it's really hard to kind of keep all the teams pointing in the same direction because they just don't have any of the muscles around to kind of coordinate those teams.

The one challenge I see in those situations is kind of like a lack of alignment, right? And then the almost the opposite challenge is when you have teams that are kind of stuck in lockstep with each other, and they struggle to be autonomous and independent and so on. On the one hand, if you've got that kind of chaos, then people are all kind of moving in different directions. So it's tough to make forward progress towards your goals. On the other hand, if you've got teams that are too much in lockstep with each other and they’re just kind of seized up. They can't make forward progress because they're just stuck waiting for each other to move. So, I think these are two kinds of weirdly inverse challenges, but both things that companies struggle with.

David Brown

Interesting. And so what do companies do to deal with those challenges? What do you recommend are the strategies they can follow? 

Pete Hodgson

So, I think that the key is figuring out how to balance these two things of autonomy and alignment. On the one hand, I think to avoid teams being in lockstep or kind of like, too coupled together, you've got to kind of set up structures that allow the teams to work independently. The flip side of that though is you've got to have enough coordination. You've got to have the right kind of coordination practice to allow those teams to kind of be working in an aligned kind of way, pointing in the same direction. And you know, we could talk about what are the good practices to enable that alignment, I think.

So, the thing with alignment I think is more interesting. That's kind of like what I've been thinking about a lot recently. And again, maybe that's partly because I've been working a lot with my startups that are kind of going through these growing pains, basically.

David Brown

To be clear, when you say these startups, are you talking about software development organizations or are you talking about enterprises that should just have a software development practice within their organization?

Pete Hodgson

For me, it's primarily the former. So, it's companies that were kind of small startups, small software development startups. And then now they've kind of hit product market fit. They're starting to grow and they're starting to figure out how to shape themselves. 

David Brown

I guess the principles apply equally though, to enterprises which are running their own development teams to run operational type software development as well. Sorry to interrupt your train of thought, go ahead.

Pete Hodgson

Yeah, I think so. So, I think that the thing that I see where teams are struggling with alignment is that they don't have a kind of a shared vision of where they're trying to get to. We'll take an example that is maybe kind of like a common challenge and an enterprise scenario where the executive has decided that we need to get this piece of software into the cloud or on-prem microservices, or we need to start doing distributed tracing, or like, one of these kind of big things that spans multiple teams. So, in order to do that, first of all, you need a shared vision of where you are going. And so, I kind of just stated some of those, right? Like you’re doing distributed tracing, or you're going to be in Kubernetes. But that vision needs to be detailed enough that everyone actually gets it. Right?

So sometimes you're getting executives or kinds of architectural leaders who feel like they really understood. And they’re kind of like, “It's obvious where we’re going. We just need the old teams to kind of start getting there.” But if they don't spend a lot, like, a good amount of intentional effort, really communicating that plan? Like, having a plan is actually something that sometimes teams don't have or organizations don't have. They have this kind of vague idea of like, “Oh yeah, we really need to get into the cloud,” or “We really need to move to microservices,” but they haven't really thought through the details. But even if they have thought through the details, a lot of times, that plan hasn't been really well articulated and shared with teams. So, if you don't tell your teams where you're going, it's not surprising if they're not going to go in the same direction. 

Like this analogy of a road trip, right? Like if you know the journey is we were going to Disneyland, we're moving to Kubernetes, really clearly articulating that destination is the first step. And then the next part is like sharing how we're going to get there. Right? 

So if we take, for example, moving to Kubernetes. If you just say to everyone, even if the executives kind of very clearly say to all the teams, “Our goal is by the end of Q2, 2022 or whatever we're going to be, all of our core services are going to be running in Kubernetes.” If you just kind of say that's the plan, and then just leave it at that, it's not going to be a coordinated effort, right? Maybe all the teams are going to move that stuff to dockerize their things when actually we're not planning to dockerize them. We're planning to break it apart and then kind of move things into substance. 

So, if you don't have a really clear articulation of not just the destination, but the vague shape of the journey, then you can't expect those teams to be acting with alignment. And then the flip side of that is, I see organizations feeling that pain and they're reacting action is to say, “Oh gosh, we need to just start telling these teams exactly what to do. They're not getting well. They're not doing it the right way. So now let's just tell them exactly what to do.” And they kind of start micromanaging and then you kind of take away those teams’ autonomy, and that means that they start doing stuff that just doesn't make sense in that context. It's because someone who's like three layers above them, or often some architectural ivory tower somewhere has said like, “The way we're going to do this is we're going to use system X, Y, Z.” 

And because they're so far removed from the code face, the reality of day-to-day, they aren't able to make those good tactical decisions, and the team members who should be able to kind of make those tactical decisions, if they've had that autonomy taken away from them, or if they don't know what the plan is, then they're not going to be able to make good decisions. Right? So there's this kind of combination of team members or those individual engineering teams that need to know the plan so that they can then make tactical decisions based on their reality on the ground versus this kind of broad picture.

David Brown

So have a plan, articulate plan and what you described as a vague journey to get there. So you don't want to make it too specific. 

Pete Hodgson

Yeah. And so it's like levels of zoom on a map, right? So if I was saying to someone, we're doing this cross-country road trip. So, we're going from LA to, I forget which one is in Florida, Disneyworld, or Disneyland? Now the one that's really far away from LA?

David Brown

Florida is Disneyworld.

Pete Hodgson

I've never been to these. So, we're going to the ones in Florida. So, if we wanted to describe to someone the entire journey, we wouldn't do turn by turn directions, right? Like if you showed someone turn by turn directions, the entire journey, that's just overwhelming, and it's too rigid.

Like, what if there's a traffic jam up ahead? Or what if we decide we want to go stop, you know? We might take a longer lunch than we planned. If you have this very rigid, locked-in detail for the entire journey, that's not helpful. As the closer you get to the near future, the more zoomed in you want that map to be right. So, for the next two hours or for the next 10 minutes, I need turn by turn directions. And maybe I'm the person coming up with those directions, but we need a detailed plan for what we're doing in the immediate future. And then for the next few hours, we need to know what highways we're going to be on, where we are going to stop for lunch, that kind of stuff. 

And talking about that broader stuff, in the wrong level of zoom in too much detail actually means people can't see the important stuff, right? Like, they're fixating on, “Oh, we're going to use this library,” or “We're gonna work on this service first,” and not seeing the broad picture of the “why” behind the “what.” Right? So, as you kind of zoom in and out, you need to use the appropriate communication mechanisms, planning mechanisms for the scope of the work that you're planning over, like really big roadmap-y stuff. Like, I see a lot of organizations using the same software tools, like the same planning tools for a huge, multi-year program of work and what an individual engineer is going to do for the next day or so, and turns out actually, it's just totally different content. Like, there's just totally different sets of information you need. It doesn't really make sense to use the same tool for that, in my opinion.

David Brown

Yeah. So, it's more abstract. The further you look out and the more agile, the closer you get to the actual implementation. 

Pete Hodgson

Yeah. And there's, you know, another metaphor. I'm not really a big fan of military metaphors, but another metaphor that I think is really apt for this is this idea of “mission command” versus “command and control.” So, if you imagine what people think of the military as being the epitome of hierarchy and top-down structure, actually, since I think the turn of the 20th century, around that time or like a lot of notary organizations realized that what is a lot more effective is this thing called “mission command” where you have the general sitting up on the hill, looking at the strategy and laying out the strategy and they hand down strategic goals to the people on the ground and give them enough context to understand what's the bigger thing we're going for.

And maybe you give those troops on the ground a suggestion like, “Here's how we think you should get that,” you know? Like, “We need to take that hill. There's enemies here. So you should probably go and do something to that enemy,” but actually we know the best laid plans kind of fail in the face of the enemy. So, just adjust your plan if you need an understanding of that strategic goal. So empowering teams to decide the tactics and having kind of a broader picture where you're designing a strategy. There's kind of like a nice complementary thing, where everyone's working at the right level of zoom, depending on what their job is. 

David Brown

Yeah, it sounds like a bit of experimentation to get that balance right. In terms of setting the abstract goals and giving them enough runway to get out, to get there as well. You talked about cross team collaboration. You've recently written about it, and you said that code ownership can become part of the problem when trying to coordinate across multiple teams. What problems arise from this?

Pete Hodgson

So, when we talk about code ownership, it comes down to the same kind of thing of, “How do we keep teams operating independently and not stepping on each other's toes?” And the only way that I've really seen that work is to draw pretty clear boundaries around who owns what area of the code. There's some kind of techniques you can use to try and make those boundaries a little more kind of diffused in a source or like an internal open source kind of idea. But I think, fundamentally, if you want your code to remain healthy in the long-term and not kind of turn into a mess in the long-term, the only way to achieve that is with a clear kind of ownership of areas of the code.

So you need people to own areas of the code. But then the challenge becomes for any interesting piece of work, for anything that's kind of like really gonna deliver a business impact, you're almost always going to be cutting across multiple teams' code bases, right? Imagine a teeny, tiny startup where a single team literally owns the entire value stream from the front-end all the way down to the database and all the rest of it. Anytime that you're bigger than you end up for anything that is really impactful from a business perspective, it's generally going to cut across multiple areas of the business which means it's going to necessarily,almost necessarily cut across multiple teams. 

So, then the challenge becomes how do we coordinate these different teams to work in an efficient way, because the biggest cost that you've got, the biggest inefficiencies that you can run into are around cross team collaboration, right?

That's the reason why we break teams up into smaller teams. It’s because communicating between a bunch of people is hard. Now communicating inside of that smaller team is easier, but communicating across those team boundaries is hard. So, what I've seen a lot of organizations struggle with is not really having any kind of intentional plan for how do we make a change happen and across multiple code bases? Yeah, some places just literally don't have a way of doing it, right? They do like shoulder tapping or or they just say like, “I don't know, I guess the team should just go into these other teams’ code base and make the change and submit a pull request.” 

David Brown

Well, you've identified some of these patterns, haven't you? These code ownership patterns from single-owner to orphan to modular monitor. Can you run us through some of these ownership patterns?

Pete Hodgson

Yeah. So, the one that I've kind of just said, I believe it's kind of like, the right way of doing it is kind of this idea of single ownership. So, having clear ownership boundaries, that's a lot easier with microservices architectures because you have clear boundaries around each service and you can say a given service is owned by a given team. It's quite easy to do that. It's quite nice. And one-to-one, if you've got a larger monolithic system, like my definition of a monolith is a single deployable thing, which is owned by multiple teams, where the code is owned by multiple different teams. And so if you're working in that situation, you can either have undefined or unclear or messy ownership which comes with a whole bunch of challenges, as I was just learning to.

Or you can draw clear boundaries, make an effort to draw clear boundaries inside of that model as to which teams own which code. And then the last thing, which you know, I think part of the reason a lot of organizations have been kind of moving towards microservices apart from it just being trendy and buzzword-y is that kind of clear code ownership, but what can happen? Well, I guess it can also have a monolith, but I see it being more painful with microservices. Sometimes, someone builds something and then they don't own it anymore, or they don't want to own it anymore. So particularly when you've got organizations that have really made it easy to spin up a new service, you get these situations where a given team operates like 10, 15, 20 services. And at some point the person that created the service kind of goes to a different team or goes to a different company, or, you know, wins the lottery or whatever. 

And now you've got these services which are running in production and no one knows who owns them. And if you need to make changes to them, either because you want to make an improvement or because something's going wrong with them, you kind of have this challenge of not knowing who owns what, so there's this kind of orphan code basis.

David Brown

And that happens a lot. That's reality. 

Pete Hodgson

I mean to be honest, it's not like it doesn't happen with monolithic systems. But in general, because monolithic systems have these kinds of more diffused ownership patterns, people are more comfortable with making changes to like a part of the code that they don't like. They're not going to want to make those changes, but if push comes to shove, someone's going to be like, “Ah, you know, I got to get this thing to work. Okay. I guess I'll do it.” But if it's like a different repo, like a different repo that you don't even know how to stand the thing up locally to start developing, it's harder to kind of deal with that. Those kinds of orphan, abandoned code kind of situations.

David Brown

How do these patterns emerge? Are they a consequence of Conway's Law or the organizational structure? Is it the software projects that they're working on? Like those modular models, can an organization control or dictate which pattern they use?

Pete Hodgson

I t hink that they generally do emerge organically up until the point that it becomes super painful. And then I think most organizations really go through this kind of pain point and they start trying to figure out how to solve it. So I've worked with a few organizations recently that have this ginormous, well, not ginormous, reasonably large monolith. We've diffused ownership and they're kind of realizing that, “Gosh, we can't get anything done in this area of the business,” because no one really owns the code and it's a mess and it's not getting any better because no one owns it. 

So, I think you get to this pain point where it becomes painful enough, you know? It was kind of ironic, or it's not ironic. It's kind of sad because often in these situations, the engineers that are working with this code have been raising the red flag and kind of trying to tell people for years, “This is a mess. No one owns it,” but it takes a business impact thing. Like, “Oh, we couldn't close that sales deal because we just couldn't get this feature implemented in time,” or “We had an outage and it was really painful to fix because no one knew how to fix the code.” Suddenly everyone's like, “Wow, this is really something. We should do something about it.”

So then, you know, engineers become this intensive. Like, “How do we fix this?” And that's generally where you see organizations get more intentional around things like ownership and starting to implement things like a service catalog. If  you're in that kind of microservice space or if you're more in kind of a monolith-y power kind of space, like starting to implement code owner files or something like that, that kind of defines these code ownership boundaries a little bit better.

David Brown

You've shared some thoughts recently about technical debt as well. Particularly, you've written an article on building a “tech debt wall” after reading that some of the concepts seem quite obvious and intuitive as a two dimensional map of creating this tech debt wall. Run us through the concept because I think it's quite good. 

Pete Hodgson

Yeah. So, this definitely is not my idea. So, you know, I spent six years or so as a consultant at ThoughtWorks and as all these practices, these agile engineering practices that a lot of ThoughtWorks teams do and have kind of diffused inside of the hive mind of ThoughtWorkers. But for whatever reason, they're not really discussed that much in the industry as a whole. But you know, there's that blog post I write. It was just basically me kind of trying to write down something that wasn't my idea, but that I'd seen be used successfully. And so the idea of this is basically like empowering engineers on the team to capture tech debt in a way that's productive.

So, the mechanics of this is you have a wall. So, either a physical wall, if you're a co-located team or a virtual wall, like a narrow board or something like that. And every time you’re discussing a piece of technical debt, or you come across some kind of technical debt in the code base or architectural debt, something larger than that, you write it down on a sticky and you put it on the wall and where you place it on the wall is kind of, as you said, these kind of two-dimensional axes. So, along one axis, you have kind of like the cost of fixing this debt. Like, how hard would it be to fix this thing? And across the other axis, you've got the value, like how valuable would it be for us to fix this thing?

It’s kind of like, “Well, duh, yeah. You should ask guys that stuff,” but like many teams don't. They either talk about this stuff ad nauseum and keep on litigating the same arguments like, “We should fix this thing.” Or they just don't really like to track that stuff at all. And so there's no visibility into how much stuff is there. And on the flip side, you get like a Friday afternoon where the engineers have kind of got some spare time and they're going to fix them some bugs, kind of squash some tech debt. If you don't do that in a thoughtful way, you're just going to end up fixing the stuff that's most irritating to someone.

And sometimes that's great, but if you've done this job of kind of categorizing your tech debt, then you've got these kinds of zones on this two dimensional map that make it really obvious where you should be working and where you shouldn't be working. So, if something is cheap to fix and has a high impact, then you should fix that. If something's cheap to fix, but it has low impact, then maybe you shouldn't fix that. Maybe you should go do something else instead. And so, capturing the stuff and categorizing it, lets you kind of manage this tech debt in a slightly better way. 

David Brown

You know we do post tech debt on a wall, but normally it's a one dimensional wall called a backlog, you know? JIRA or something like that. And yes, we prioritize that, those tickets, maybe it's a product owner or something, prioritizing those tickets. And I guess subconsciously they're making some of these decisions about the value proposition of the tech debt, but this just formalizes it and it makes it very visual. It is this cost-benefit analysis. And it's very easy. Let's just talk about the ones in the top right-hand corner of the grid, whether it's contributing a lot of value to the least amount of costs. Right? 

Pete Hodgson

I think it's also empowering for engineers where it's a win-win, where if you're a product manager or a scrum master or a product owner or whatever, who's kind of prioritizing things, it makes it very clear, like the cost and benefits. So, it helps them do that work, but it's also a way for engineers to really clearly communicate the cost and benefit without having to get into a meeting and doing planning poker or anything like that. It's just kind of there. And it is a really good way to kind of spark a discussion with an engineer. So, I kinda co-located teams. A lot of times you'll see an engineer walk up and put something on the sticky on someone else who's working near them.

We'll look over and see what it is. And then they'll be like, “Oh, let's start this whole kind of conversation.” They'll have a little discussion or argument, disagreement about things like, how easy it would be to fix or how impactful it would be. I think setting a culture where you have those conversations and you have those conversations in a very kind of collaborative kind of flat way where it's not the tech lead who decides what stuff we should be fixing, but it's like, everyone's empowered to identify problems and suggest solutions, I think, is a really good kind of cultural benefit of doing that too.

David Brown

Yeah. I mean, that's interesting, you mentioned the benefits of co-located and how you miss those co-located teams. I mean, it seems so long ago. 

Pete Hodgson

I’m I think one of the very few engineers that actually really likes working in big open space things where everyone's just sat around the same kind of open table. I know most people hate that and I know that I'm weird. But the best, the most high-performing, most fun teams I've worked on, have all been in that situation. And I think you can replicate some of it with asynchronous things. I definitely think there's a ton of benefits to being kind of remote first. And I think I would advocate for that generally nowadays.

But there is just that, that really nice organic kind of feeling of everyone being in the same room together. And the serendipitous overhearing of conversations is something that when I've been a tech lead, has been like one of my real, kind of like most useful kind of tools that I have are just sitting there pairing with someone else or whatever, and like overhearing conversations that there's just so much value that comes from that.

David Brown

Or at the pub afterwards.

Pete Hodgson

Yeah.

David Brown

In one of your books, you recently talked about how teams can create platforms that no one in the organizations ends up ever using. So, someone develops it for themselves, they think it's the best thing since sliced bread, but their use cases are perhaps too narrow for themselves. IIs that the concept you're talking about here?

Pete Hodgson

Yeah. A bit like this. I see this all the time. Platform teams that don't realize they're building a product. Like, that's the fundamental thing that I think a lot of platform teams struggle with is they think that they don't really see that they're operating in a marketplace, they have customers, they have competitors. And so they don't operate as if they're building a product. And what that means is they build stuff that no one actually wants or they build stuff that isn't built in such a way as to service their customers' needs. 

And I think it's particularly true. It's particularly tricky for platform teams that are building where they kind of feel like they're the customer, right? So like if you're building software for I dunno if you're, if you're, if you're building like software for, for kids' toys or something, then you know that you're not the end-user and you don't kind of just accidentally just build something that you would like to build or that you'd like to use, but most platform teams, unless they've really kind of like learned their lesson for kind of pain, most platform teams fall into this trap of thinking about what would be useful for them and there's some means, or there's some kind of tropes in the industry that reinforce this, right?

So, like eat your own dog food or drink your own champagne or whatever. That's a good thing in general. But with platform teams, what it ends up being is, platform teams build something that is fit for their purposes and like that. And they consider themselves to be the guinea pigs. But then they've already built a thing and then they get feedback from their users if they’re actually looking for feedback, often they’re not looking for feedback. They just find out they've built something that actually, the people who they should actually be building this thing for are not really fit for purpose. 

So, I see that happen a lot. And I think the key to this, the key solution here is for someone in that team to be essentially wearing some kind of product management hat, but you don't necessarily need a product manager on the team. But you need someone, or some group of people who are doing the kind of things that a product manager does when they're building a product for an external customer.

David Brown

Yeah. And like you say, they should be thinking about better product engagement with the stakeholders, the users of the product and marketing it within the organization as a product.

Pete Hodgson

yeah. And it’s that kind of weird stuff for marketing. I think for a lot of people who are focused on building platforms, they think the idea of marketing it is kind of silly, partly because they sometimes feel like they've got a captive audience. Like, “Well no,” you know? “People are going to have to use my thing. Otherwise, they don't have any other way of doing blah.” 

It is very impressive how good engineers are at finding another way to do blah, if they don't like it. Right? Like, the people who say, “Oh, well, they will have to use our thing because it's the official thing.” Those exact same people have absolutely, at some point in their career, found a way to work around some rules that they didn't like. They've used their own credit card to buy some service or they've implemented their own version of a thing because they don't like the thing that they were given.

So, engineers have this kind of persnickety – they’re really awkward customers, right? And really annoying customers sometimes. But the engineers that are building platforms sometimes don't realize that other engineers are like them. And they're going to decide to use an open source tool instead of the internal tool, or they're going to go and buy a tool that they prefer, or they’re going to write their own et cetera, et cetera. So, even though it doesn't feel like you're in a competitive marketplace, you almost always are. Even if you're just competing with people, building their own thing,because they don't like the color of the thing that you built.

David Brown

Yeah. Interesting topics. You write about some interesting stuff. How can our audience follow you? What you're writing about, what you're talking about, where are your social channels? 

Pete Hodgson

You can follow me on Twitter. On Twitter, I am “ph1.” Or you can also read my blog and that's Thepete.net.

David Brown

Great. Thanks for joining us on the show today! It's been a pleasure talking to you.

Pete Hodgson

You're welcome. Thanks for having me.