Kevin Montalbo

Joining us all the way from Australia is TORO Cloud CEO and Founder David Brown. Hi, David!

David Brown

Good day, Kevin!

Kevin Montalbo

Our guest for today is a Principal DevOps Architect at Codefresh, a member of the Google Developer Experts and Docker Captains groups, and published author. His big passions include DevOps, Containers, Kubernetes, Microservices, CI/CD and Test-Driven Development (TDD).

He’s a sought-after speaker at community gatherings and conferences, those aren’t really happening right now, so he’s very active in his podcast titled Devops Paradox at devopsparadox.com, and is actively blogging at TechnologyConversations.com. He has also co-authored the books DevOps Toolkit Series, and Test-Driven Java Development.

Joining us for a round of Cocktails is Viktor Farcic. Hey Viktor! How are you?

Viktor Farcic

Always good. It's permanent. Doesn't change.

Kevin Montalbo

It doesn't change, right? Yeah. All right. So we'll jump right into the questions, Viktor. Kubernetes has announced in the news that it will no longer be supporting Dockershim from Version 1.22, which was around December of last year. And this Dockershim is due out later this year. So, what does that mean for Docker containers struggling on running on Kubernetes?

Viktor Farcic

It really depends on how somebody implemented containers. If people implemented containers as they were supposed to be implemented, meaning that, “Hey, there is a container running my stuff,” and whatever is in those containers is there and not somewhere else, then nothing happens for the vast majority of people. If I would change now, their container runtime from being Docker Container-T or something like that, they would not even notice the difference. 

Now, there are cases where people were doing some, I would call, let's say, “silly stuff” where they would mount some sockets from their containers to Docker and things like that, then those people would have problems. But if you were not doing things like that, like privileged mode sockets and so on and so forth, then it's simply transparent. And that's how I believe it's supposed to be. Because if you look at, for example, cloud vendors with their managed Kubernetes services, in many cases, you don't even know anymore what is the Docker runtime. If right now you spin up, let's say GKE cluster, Kubernetes cluster in Google and you really do not go deep into documentation, you have no idea what's what's the container runtime there. 

David Brown

Just to clarify there, what you're saying is Dockershim is actually the runtime to run the containers. What's its purpose? Why does Dockershim exist and how did they manage to get rid of it?

Viktor Farcic

Small correction, Dockershim is not really the runtime to run containers. Dockershim is a translator for API calls in Kubernetes to Docker. Because Kubernetes initially started with a lot of hard-coded instructions on how to deal with containers based on Docker. Over time it evolved. Kubernetes assumes that there is a container runtime - whichever it is - it doesn't care. And that container runtime speaks some standard interface which everybody agreed on. And the only really one of the few, if not the only, container runtime that doesn't speak that interface is Docker. So they converted their hard-coded Docker instructions into Dockershim, which is that translator between the standard interfaces and invocations of that interventions in Kubernetes. 

David Brown

It’s like a wrapper? Basically it’s a wrapper so that it could speak the standard protocols of Kubernetes? 

Viktor Farcic

Exactly. And the main reason for deprecation is, basically, there is no good reason why the Kubernetes community would maintain that wrapper right? If everybody agreed that there is this interface, we take Cri-o or Containerd or whichever other container runtime, it speaks that interface – good story, right? And there is no good reason to maintain it anymore. 

On top of all that, even without that deprecation of Dockershim, there was never a really good reason to use Docker as your container runtime in the first place. Because Docker packs a lot of things. And we all like those things because they're very helpful today – let's say a laptop. But it packs code that mounts your volumes into your containers that deals with networking. There is a container schedule, or similar to Kubernetes, called Docker Swarm and so on and so forth. So it has a lot of things. But if most of those what I call “a lot of things” are implemented on Kubernetes level today, then you're basically having something that you don't need. 

David Brown

And so we've got rid of a lot of stuff that we actually weren't using, didn't need, there’s a lot of bloat that’s actually gone.

Viktor Farcic

Exactly. A lot of unnecessary things. And, as a matter of fact, if you use Containerd, which is probably the most commonly used container runtime these days, and on the other hand, if you use Docker, you're also using Containerd. Containerd is part of Docker. Basically, Docker’s company moved part of Docker - let's say the part that is in charge of running containers, and not much more. It moved it into a separate project. And then when they built Docker itself, they included Containerd.

David Brown

So, if it wasn't already clear, Docker itself is not going in terms of containerizing applications and “Dockerizing” applications and running them on a Kubernetes cluster. That's not going. What's going is Dockershim, the run time for the Docker containers.

Viktor Farcic

Yes. So, at least officially, from Kubernetes’ perspective, Dockershim is going. The Kubernetes community never said Docker is going away. Dockershim is going away. But the result of that translator going away from Kubernetes clusters, it indirectly means that Docker is also going away. Unless you continue using Dockershim, which these days will be maintained by Mirantis, which is a company that bought Docker basically, part of Docker employees. And there are some of their IT. So you can theoretically continue using Docker because you can continue using Dockershim. But that would be postponing, because I doubt that Mirantis will really maintain Dockershim for a long time. It's more of a short term-business decision. 

So, yes, Docker is not going away, but you will be forced to remove Docker from Kubernetes clusters. You can freely continue using it on your laptops, which is a separate story. 

David Brown

I mean, it sounds dramatic when Kubernetes announced that they are removing Dockershim, but in actual reality it’s probably not such a big impact. Unless, you say, I think people were doing things which they shouldn't have been doing with the containers, anyway. Is there any impact at all on decisions like this on the Docker community and the Docker ecosystem?

Viktor Farcic

Docker as a company, I think switched gears and moved away from the idea that they will provide you with a container orchestrator like Docker Swarm, the mysterious Docker Swarm and a few other things, and I think that they're now focusing very much on, how would I call it, maybe developer experience? They're very much focused on tooling for developers, which was in a very original intention from Docker in the first place. They're very committed to what you need as a developer on your laptop and not that much what you need as a developer or anybody else elsewhere, like in your cluster. 

Yeah, they're going a bit beyond that and entering a lot into crossing the gap between a developer and cloud provider. So you can use, for example, Docker these days to deploy things to usher container instances, you can use Docker to ECS and a few others, security scanning with Snyk. So, they're either what is happening on your laptop - actually, mostly what is happening on your laptop - plus some bursts into how to help developers deploy to development environments. Because not many people, if anybody is going to use those integrations like Azure Container instances for production, but it might be useful during the development process, I guess.

David Brown

Got it. Hope you don't mind that we're going to switch gears a little bit and start to talk about GitOps. So, I know you've written quite a bit about GitOps on your blog and podcast. There's a bit of confusion in terms of the distinction between GitOps and infrastructure-as-code that seemed to be some relationships there or one is a subset of the other, one does more than the other. Can you tell us what all the hype is about GitOps and what relationship it has to infrastructure-as-code?

Viktor Farcic

I tend to define GitOps maybe slightly differently than many. To me, it's not really a new idea. Essentially, GitOps is all about making sure that Git is your source of truth at least for the desired state for our intentions, not necessarily for what something is, but what we want, something to be. And if you go back, I don't know, like 30 years, you would still say, “Hey, the code isn't Git.” I mean, 30 years ago, you wouldn't say GIT would say SVN or CVS or whatever we were using at that time. But it’s that enforcement of the idea that “Hey, if you write code, code needs to be in Git because that's the only version control system we have today. And by the way, since all of us are engineers, we all write code, therefore Git is the real source of truth.” Because if everything is code and code isn’t Git, then that's where things reside. 

And it's mostly to avoid things like, let's say, you mentioned the infrastructure-as-code, to avoid those situations where you write some Terraform files around civil something or whatever you're doing and then you start executing it from your laptop without pushing it to Git first. And it is extremely important that you do always, always store something in Git before changing the state of something for a simple reason, because, unless you're a one-man company, that enforcement of keeping everything in version control means that you're working in a team in a way, right? That's how we collaborate with each other. I know what you did, because I can check the changes you made to your code, and I'm doing that, not validation but exploration of what you did through Git.

David Brown

So this is what I'm saying. Like this all sounds very much like infrastructure-as-code with pretty much, as you say, always thought infrastructure-as-code in some sort of code repository like the Git repository. But as I understand it, GitOps, it goes beyond that. And so, instead of the state of your infrastructure being just only stored in your Git repository, there is actually a two-way communication which, with your infrastructure, reporting back to your Git repository advising if your state is the same. Am I correct?

Viktor Farcic

Yes and no. In a way, I think that what we're seeing lately, GitOps is becoming very popular. And then different software vendors, you know, everybody's trying to jump into that wave to get some benefits. And then we're all redefining what it is, depending on what our software does and that there is always such a conflict going with everything new and according to some points of view, you would say, “Hey, GitOps means that you have some kind of processes that will detect drifts automatically,” and say, “Hey, your infrastructure, your application or whatever it is happened to start being different five seconds ago from what is defining it. Let me correct that,” which is an amazing thing. I think that everybody should implement that. I don't necessarily see it as a requirement. If you would be applying those changes by receiving notifications from Git instead of you detecting changes in Git, and you would not allow anybody to do many or changes in your infrastructure, you will still get to a similar place. But it is definitely beneficial, I would say, rather than a rule to have that type of processes that are monitoring your Git repository all the time and trying to figure out what changed it. Did anything change? And when I think about it, if you remember the days that weren't so long ago, right? When we were all using Chef or Puppet for infrastructure, they had those things basically which were removed later on with Ansible and Terraform. There was a server where that would be receiving your changes stored in a code repository and continuously monitor for those drifts. So it's not necessarily something new. It's not necessarily something that has to happen. The most important thing, of course, is that it is the source of truth, but it is definitely very beneficial, let's say.

David Brown

Okay. So, what are the current trends in infrastructure-as-code? You mentioned a couple like Chef and Puppet and how we've evolved into Ansible. And then we had the proprietary formats of the public cloud providers, most famously cloud formation templates for AWS. And Terraform seems to be the big player in the space now. DevOps moving away from those proprietary definitions would be, like, CloudFormation templates, templates joining work with AWS moving to sort of more cloud agnostic solutions like Terraform?

Viktor Farcic

Yeah, I think that that's almost certain, let's say. If you look at what Google's - I don't think that Google even has a proprietary. If they do, nobody's using it and nobody knows about it. Right? Azure basically, they gave up on proprietary and is more focusing on how that proprietary can be invoked from other tools than something else. From a certain point of view, you could say that AWS with CloudFormation is the only one that is still pushing, somehow, proprietary and infrastructure as code. 

But even that wouldn't be really true, because if you go in for example, check the examples in AWS, you know, quick start tutorials, how to create, let's say EKS Cluster. And I'm now talking about official documentation, not random blocks. You will see that even AWS is putting as the first option EKS Cattle which is yet another tool to do configuration as code. Focused on EKS, true, but not really proprietary. Not really AWS. It wasn't even written by AWS folks. So even AWS, which is probably the strongest proponent of their own tools, is shifting away to others.

David Brown

How agnostic is it realistically? So when you're writing a Terraform template for your infrastructure at AWS how agnostic is it? Can I just flick a switch and go to Google?

Viktor Farcic

No. It's not really agnostic. It's not agnostic at all. We can potentially talk about application layer being agnostic through Kubernetes. That’s definitely a thing, even though their differences are small between providers but infrastructure is called as anything but agnostic. The only thing that we have is, “Hey, you can use this tool to define your infrastructure.” No matter where it is, let's say Terraform or Ansible or whatnot. 

And we can say, “Hey, you will use the same format to define that something, no matter where it is.” In the case of Terraform, that would be HCL, right? And Ansible would be YAML and whatnot. But the contents of what you're defining is going to be completely different. You will not be able to share a single line of your infrastructure if moving from AWS to Azure or vice-versa except maybe name equals something. That might be the same. That's it. That's where it stops. 

So it's not agnostic at all. It's rather that those tools can handle infrastructure in different variations or permutations. And then you have one tool but not an agnostic definition.

David Brown

Because there are some projects where they're trying to abstract the APIs of AWS and Google and Azure so that you have a single interface to provision compute instance, for example, in storage and the like right? So I guess through an abstracted API, you could potentially sort of create some sort of agnostic interface.

Viktor Farcic

It could, but it would require a lot of work. You're right about infrastructure, for example. And we can see that S3 essentially became standard for a certain type of storage. And you will find the ability to create S3 pockets outside of AWS and that it’s kind of becoming agnostic or a type of standard. And if I exclude now Kubernetes as an option where you create storage or external load balancing in exactly the same way no matter where it's running. Apart from those two to maybe a few more examples, it's still very much proprietary because whichever tool you use, it will need to invoke API of that provider, which is very, very different.

David Brown

Yeah, I was talking about, like Apache Lib Cloud Project to abstract the APIs of Azure on AWS so that you can provision an instance and not really care under the covers as to where and how that instance is being provisioned?

Viktor Farcic

Yeah. I'm going to write it down on my to-do list. What did you say, Apache?

David Brown

Apache Lib Cloud. It’s been around for a while and I don't see a lot of traction behind it. There's a couple of projects there to try and unify the APIs of the major public cloud providers.

Viktor Farcic

I think it would be possible with such a tool, and I really don't know Apache itself, so I cannot comment on that one. But if such a tool would focus on a couple of things like storage, smaller potentially, I can see it working. But as soon as we go beyond common denominators, there would be a lot of problems. 

Let's say it if your task would be to create a Kubernetes cluster, right? And you would like to standardize that, like in AWS, you need to create, like, 10 or more different services. In Google Cloud. It's a single service. Now, if you would like to unify that in a single API, I could not even comprehend how that would be possible, right? How would you, without excluding capabilities of one and reducing it to a bare minimum kind of, specify a name and maybe the number of nodes? Even if you look at the sizes, let's say that you want to create a unified API to create VMs, and that's much simpler now than Kubernetes, how would you handle sizes? Sizes are completely different in every provider.

David Brown

That's right. You need to standardize across instance types. And of course, you have to create broad groups of small, medium and large that are not alike so there's lots and lots of challenges associated with it. Terraform seems to be the major player in infrastructure-as-code, but there are plenty of alternatives. You've mentioned a couple already and there are others as well, like Pulumi and more recently Crossplane. What is the difference between and why would you choose one over the other?

Viktor Farcic

On one hand we have infrastructure-as-code that is declarative, that would be Terraform, Ansible in the past, and Crossplane that you just mentioned. On the other hand, we have Pulumi, which takes a different approach, meaning you can use your favorite programming language as long as it is supported by Pulumi to write code, basically Azure infrastructure. So, that would be one distinction and one different path that they're taking. And then we have a more declarative, basically, the major player is  without a doubt, Terraform. Maybe Ansible being the second behind that, mostly because of the past weight that it had than anything else. And recently, when I said recently, maybe a year ago - I think that a year ago was when the first lines were written - we have Crossplane. I mean, of course, we have many others, so don't get offended. If you're listening, you’ve one of them. But to me, Crossplane is interesting because it doesn't give me that agnosticism that we were talking about. I cannot really use the same definition “just flip a switch from one provider to another,”  but it gives me the ability to manage my infrastructure through Kubernetes API and the benefit of to begin with, I have one API to manage everything I have. And of course now I'm referring to people who are mostly focused on Kubernetes even though infrastructure can be anything. I could say, “Hey, Kube Cattle basically solves all my needs. I can do Kube Cattle. Deploy my application? Kube Cattle. Create my cluster? Kube Cattle. Give me the storage or whatever my task is.” 

And that potentially helps a lot because we ultimately want to move - probably many of us want  to shift left and move towards “Hey, we have those self-sufficient teams that can do everything and have a single tool, single interface.” Sorry, not tool. Single interface to do something, actually, to do everything is potentially very beneficial. And then on top of that, we have all the things that Kubernetes gives us. Ecosystem, that is probably the biggest one right now and then plugging yet another tool, like in this case, infrastructure-as-code into that ecosystem is potentially very good.

David Brown

That makes a lot of sense. I mean, Kubernetes APIs mature and are extensive, right? So Crossplane is leveraging that API, meaning that no matter where your Kubernetes cluster is deployed, it's gonna work. It's just using the Kubernetes API, am I right?

Viktor Farcic

Yes, it will work. You could potentially move your Kubernetes. Let's say from one Kubernetes cluster with Crossplane running there from AWS, and everything will still work. But that still poses the problem from before. Yes, if you have everything in AWS and you move your cluster to where Crossplane is running to Azure, Crossplane will work, but it will still continue creating infrastructure in AWS. Right? Because, even with Crossplane, you still need to rewrite everything. If you would like to switch also what it does, I mean, not what it does, but if you want it to do the same thing but somewhere else.

David Brown

Yes, so the underlying networking and infrastructure and all the base set-up still requires cloud providers specific code.

Viktor Farcic

Yes, and there is actually an exception. What you need for infrastructure are the things that Kubernetes itself needs, right? Not the things outside. So you might need to create a let's say, database service somewhere that counts as infrastructure, and that's going to be very different now. If you need a load balancer, if you need storage, if you need things that are today needed for your applications running inside Kubernetes, then that is already standardized. And for that, you don't even need Terraform or Crossplane or whatnot, because you just create a service that is a type load balancer and it will figure out how to create a little balancer here or there wherever that something is. Same thing for storage, you might need to tweak a bit the drivers that are used for the storage. But from the end-user perspective, who says, “Hey, I want the block storage that has 4 GB of space that is going to work the same everywhere and that doesn't even require Crossplane or Terraform or whatnot.”

David Brown

Interesting. As a DevOps expert, if you were to create a wishlist, what you would like to see occur in the industry as you're saying, it evolved. Is there tooling you would like to see improved? These description languages for building and infrastructure-as-code. What is it, you, as a DevOps engineer, would you like to see evolved? 

Viktor Farcic

Yeah, I'm not a DevOps engineer. I don't think that such a thing exists, which we can discuss later. But, what I think we need the most right now is, a layer on top of Kubernetes. Something that will simplify it, because if you go back in time like the beginning of Docker, the reason why Docker became so popular is that they created something that is extremely user-friendly and very, very easy for almost everybody to learn in almost no time. All right. “Hey, miscalculate containers. A few hours of figuring out and you're an expert,” right? Docker Swarm, here is how we schedule containers across the whole cluster. Very easy and so on and so forth. I think that whole idea died over time because making something easy that will satisfy 80% of people, but will not satisfy those 20% that actually matter for those types of work, like SysAdmins  and whatnot. It was bound to fail. 

Then we got Kubernetes which satisfies those 20% right? It gives us all the power in the world. It allows us to do things that we could never imagine doing but it failed to make it simple. Basically, we can think of Kubernetes as cancelling the shift left movement that started a while ago, right? It cannot go left anymore because it's too complicated now. But with Kubernetes, we got that solid base. Hey, everything is possible. It is designed to be extensible, but the result of extensibility is complexity. Now we need that additional layer that will make Kubernetes transparent. 

I would like to move five years forward in time and make a prediction. And my guess would be that in five years, many people will not even know that they're running Kubernetes. It will become an implementation detail like if you use Linux today, you don't compile Kernel. When you write code, you don't deal with memory and then CPU anymore, that just happens. Those are all transparent implementation details that we don't care about. And I hope that we will get into that place with Kubernetes. And if I would have one thing on my wishlist, it’s that I would wish companies to work on, and I think that some are already doing that. That simplification. 

The best example of that to me would be Google Cloud Run, like that's “Hey, here's my container. Run it.” And it will scale it to zero if nobody's using it. It’s basically a serverless implementation of containers. It runs Kubernetes in the background. But you would probably need to spend, like, at least half an hour reading the documentation before you come to the word “Kubernetes” . They're not hiding it from you. So they're not saying “This is not Kubernetes. We’re trying to trick you,” or stuff like that. It is Kubernetes. They're acknowledging it, but it's almost irrelevant for you as a user that there is Kubernetes over there.

David Brown

Good stuff, Viktor. I can't believe that we've already been talking for a lot of time through this podcast, it's gone really quickly. How can the audience keep up to date with what you're writing and talking about?

Viktor Farcic

So, it's v-f-a-r-c-i-c. That's my nick almost everywhere. So if you go to GitHub, if you go to Twitter, if you go to almost everywhere else, you will find me under that nick.

David Brown

Excellent. That's easy, Viktor. Thanks for your time today. It's been a pleasure.

Viktor Farcic

Thank you guys.