- The premise of DevOps
- Why Docker matters
- Top tags in the Docker ecosystem
- Growth trends of the key DevOps and Docker related technologies
- Growth trends of the DevOps and Docker ecosystems
- Wrap-up
The premise of DevOps
I like DevOps more than any other recent technology trend. I like it, because for long-long decades IT had the luxury to hide behind technology jargon and be a little different than other business areas (or THE business areas).
I wish DevOps will finally change this. I whish that our DevOps pipelines go way out to the market and IT stakeholders will talk more and more business terms, meaning time to market of new capabilities, continous feedback from customers, flexible and continuous setting of delivery priorities, reduced cost of poor quality, maximized development throughput and similar measures.
I see the key benefit of DevOps in industrializing the IT development cycle through automation.
New technologies enable us to deploy small business requirements into production frequently, rather than deploying many requirements in a big-bang fashion every few months or less often.
Today we can build small systems that specialize in a certain aspect of our business, choose the right technology to implement such applications and automatically test and even deploy new capabilities as soon as they are checked-in into source control.
Normally such an increase in the number of business support systems and deployments would result in an immense increase in shipment and operations cost.
The reason why we can still put such a smooth development cycle in place is that we can reduce resource requirements through automation. This automation is achieved through DevOps practices and tools.
Why Docker matters
Docker is a central part of the DevOps foundation. If you want to understand how Docker works and how it can contribute to your team’s success, you usually have to look beyond Docker and understand the broader domain of DevOps and Agility.
The reason for this is that Docker fits nicely into our efforts to build an effective CI/CD pipeline, enjoy the benefits of DevOps, Agile development methods and build Microservices.
If you work in an established corporation, you find that there are many moving parts as you try to make progress in all of the above fields.
You have to train many people, find the right projects for early experimentation and get the new tools up and running and use them the way they are meant to be used.
Getting all these things right might still not give you the benefits you are looking for, because you also have to make sure that all these new tools and methods work together with your custom legacy environment that’s like no other, and no-one can give you the exact solutions for your special challenges.
When we look at the Docker related data in this post, we analyze the figures and prepare our proposed actions considering your needs for a streamlined development production line, improvement in delivery speed and quality.
In this article we look into Stackoverflow data to understand how real-life projects implement DevOps and how they use Docker.
Let’s dive right in!
Top tags in the Docker ecosystem
Let’s start with top DevOps and Docker related tags in my Stackoverflow dump from December 2018.
The table below shows the top tags used next to the `devops` tag in Stackoverflow questions based on the number of questions carrying the given tag.
| # | Tag name | Questions |
|---|---|---|
| 1 | devops | 1796 |
| 2 | docker | 256 |
| 3 | jenkins | 237 |
| 4 | amazon-web-services | 156 |
| 5 | ibm-cloud | 136 |
| 6 | continuous-integration | 118 |
| 7 | ansible | 117 |
| 8 | linux | 113 |
| 9 | azure-devops | 85 |
| 10 | git | 84 |
| 11 | kubernetes | 80 |
| 12 | chef | 74 |
| 13 | azure | 72 |
| 14 | tfs | 61 |
| 15 | deployment | 57 |
| 16 | docker-compose | 55 |
| 17 | java | 53 |
| 18 | amazon-ec2 | 52 |
| 19 | python | 51 |
| 20 | automation | 47 |
| 21 | jenkins-pipeline | 46 |
| 22 | nginx | 44 |
| 23 | puppet | 41 |
| 24 | continuous-deployment | 39 |
| 25 | terraform | 38 |
Let’s look at the numbers from a slightly different angle showing the number of views on questions marked with the `devops` tag together with a given tag on Stackoverflow.
| # | Tag name | Views |
|---|---|---|
| 1 | devops | 918733 |
| 2 | docker | 185582 |
| 3 | jenkins | 162732 |
| 4 | ansible | 109259 |
| 5 | continuous-integration | 76334 |
| 6 | docker-compose | 65835 |
| 7 | linux | 56545 |
| 8 | amazon-web-services | 53008 |
| 9 | vagrant | 46278 |
| 10 | ansible-playbook | 45724 |
| 11 | terraform | 42325 |
| 12 | kubernetes | 42026 |
| 13 | jenkins-pipeline | 40574 |
| 14 | git | 37398 |
| 15 | sonarqube | 37005 |
| 16 | continuous-delivery | 36739 |
| 17 | ibm-cloud | 31719 |
| 18 | dockerfile | 31137 |
| 19 | gitlab | 28106 |
| 20 | containers | 28060 |
| 21 | deployment | 27446 |
| 22 | java | 26155 |
| 23 | virtualbox | 25455 |
| 24 | hashicorp | 23912 |
| 25 | github | 23881 |
As you implement your DevOps capabilities you usually start with a few base tools and then add more specialized capabilities gradually. I see people presenting crowded DevOps landscape maps (you know, the ones with a plethora of logos in colored boxes) to their clients and expect that they buy-in immediately to the full Monty.
A better approach is to select the minimum set of components that drive positive change in your company’s balance sheet. Based on the tables and our personal experience these are the areas to get started:
- Docker is on top of both lists. It’s a great automation driver, because you have small footprint operating environment for your applications and this environment is defined as code. You can destroy and rebuild your entire application and its environment from scratch any time and you get the same results.
- Another tool that most companies add in the beginning is the Jenkins automation server. Jenkins drives your entire pipeline, it triggers a job when developers check-in changes under source control and takes care of building, testing, shipping and even live deployment if you prefer through various environments automatically.
- Ansible, Chef and Puppet are products that help you manage your infrastructure by automating provisioning and configuration management. Ansible has the highest position in the tables out of these three products. This is in-line with what we see with corporate clients. Ansible is part of the RedHat family of products and I believe this is a definite plus in the enterprise world.
- GitHub and GitLab are both based on git, the free and open source version control system originally written by Linus Torvalds. Automation tools like Jenkins are mostly used and therefore best integrated with git. This is why most clients set up their own GitLab for DevOps initiatives. GitHub is available as a cloud hosted offering online for organizations whose policies are relaxed enough to go down the public cloud path.
- The list would not be complete without Kubernetes, the market leading container orchestration platform. Kubernetes does the heavy lifting running your Docker based applications on computer clusters.
You may achieve tangible benefits just by implementing GitLab, Jenkins and Docker with strong automated tests for some of your applications to get started.
Let’s have a few words about some other components on the list:
- Terraform and Vagrant are both products from HashiCorp. Terraform is a tool for building infrastructure. The key difference between Terraform and Ansible, Chef and Puppet from before, is that you use Terraform to create your infrastructure and you use Ansible, Chef and Puppet to manage the configuration of the infrastructure you built with Terraform. Terraform can build and manage large scale infrastructures across multiple cloud providers. Vagrant is a similar tool, but it’s used to manage development environments locally.
- Sonarqube is your tool for code inspection. It helps you write high quality code and reduce technical debt by checking your code for bugs, security vulnerabilities, maintainability issues. It integrates well with other tools like Jenkins and Git.
If you look back at the tables, you find that Java is the only programming language that appears on both lists. You see Python and Node.js gain immense popularity lately and their communities are rapidly growing. I am also a great fan of both languages. If you look at corporate reality, you find that Java still dominates application development and therefore it’s a key skill in your DevOps journey.
Let’s have a look what additional info the Docker tables add to the picture!
The table below shows the top tags used next to the `docker` tag in Stackoverflow questions based on the number of questions carrying the given tag.
| # | Tag name | Questions |
|---|---|---|
| 1 | docker | 48102 |
| 2 | docker-compose | 7723 |
| 3 | dockerfile | 4630 |
| 4 | kubernetes | 2195 |
| 5 | python | 2150 |
| 6 | nginx | 2065 |
| 7 | node.js | 1931 |
| 8 | linux | 1896 |
| 9 | containers | 1609 |
| 10 | docker-swarm | 1453 |
| 11 | java | 1432 |
| 12 | mysql | 1417 |
| 13 | ubuntu | 1363 |
| 14 | amazon-web-services | 1360 |
| 15 | jenkins | 1357 |
| 16 | php | 1340 |
| 17 | docker-machine | 1067 |
| 18 | postgresql | 1038 |
| 19 | bash | 922 |
| 20 | windows | 884 |
| 21 | networking | 870 |
| 22 | boot2docker | 836 |
| 23 | django | 827 |
| 24 | macos | 799 |
| 25 | mongodb | 789 |
This next table shows the top Stackoverflow tags used next to the `docker` tag on Stackoverflow questions based on the number of views (the first table was based on the number of questions tagged with the given tag).
| # | Tag name | Views |
|---|---|---|
| 1 | docker | 85386016 |
| 2 | docker-compose | 11187822 |
| 3 | dockerfile | 8213045 |
| 4 | linux | 4550187 |
| 5 | ubuntu | 3393618 |
| 6 | boot2docker | 3044807 |
| 7 | containers | 2879586 |
| 8 | docker-container | 2812105 |
| 9 | nginx | 2488078 |
| 10 | docker-registry | 2303199 |
| 11 | kubernetes | 2141158 |
| 12 | bash | 2015336 |
| 13 | python | 1948038 |
| 14 | mysql | 1938592 |
| 15 | docker-machine | 1774725 |
| 16 | docker-image | 1597680 |
| 17 | node.js | 1539068 |
| 18 | macos | 1483438 |
| 19 | postgresql | 1419400 |
| 20 | php | 1399732 |
| 21 | windows | 1351269 |
| 22 | java | 1327754 |
| 23 | jenkins | 1279336 |
| 24 | networking | 1186457 |
| 25 | lxc | 1109725 |
The numbers confirm that Docker is high on the DevOps agenda.
The top three tags are related to the Docker toolkit; they are `docker` itself, `docker-compose` and `dockerfile`.
This is a simple but significant point right here. These are the top tags, because working with Docker your key deliverables are the Dockerfile and the Compose file.
These artifacts define the working environment of your applications and services, you should add the Dockerfile and Compose file to source control. It is essential that you build strong skills here, our trainings have a special focus on these points.
Once you master the art of custom Docker images and multi-container applications with Compose, you face the challenge of managing your containers in production. The tools to accomplish this are called container orchestrators and Kubernetes is clearly the market leader of this niche. Kubernetes is at the 4th place on the tag list.
Docker’s built-in orchestration solution, Docker Swarm, has also made it to the list; it stands at place 10. Swarm is an alternative to Kubernetes, it comes built in with Docker out of the box. We usually teach Swarm in our Docker trainings, because it’s a great place to understand orchestration. Our experience is that enterprise clients choose Kubernetes exclusively.
Containerization is a means to and end. The main goal in the broader environment is to deploy your applications faster and more frequently, thus increasing the throughput of your entire development line and keep much less work in work in progress for a much shorter period of time.
This gives a lot better experience for business stakeholders, puts features into production faster and gives you flexibility to re-prioritize your backlog more frequently, so that business does not have to wait several months for a new idea to hit the IT production line.
To pursue these goals you need an automation tool that helps you build these capabilities. Jenkins is used by almost every enterprise and Internet client we work with.
A key component of your Docker based delivery pipeline is a Docker registry. This is the place where you push your custom Docker images during your CI/CD builds and that’s the place were you pull your images from in production.
The Docker Registry in the list is Docker’s free solution that lets you set up your own Docker registry. The registry that you get is like having your own Docker Hub.
Enterprises usually go for a different product. A popular solution is the Nexus repository manager by Sonatype.
Growth trends of the key DevOps and Docker related technologies
Let’s look at two figures showing the growth trends of the key tags in DevOps and containerization.
Let’s start with broader DevOps related tags as the percentage of all tags on Stackoverflow.
*You can see the highest growth with AWS, Docker and Kubernetes. Jenkins, GitLab, Ansible and Terraform are also growing steadily, while the other tags seem fairly stable.
Let’s have a look at Docker’s core components now.
Kubernetes, Compose and Dockerfile are clearly the top tags. This is where you should invest into skills and capabilities, these are the key components to build success with Docker.
Growth trends of the DevOps and Docker ecosystems
It’s time we look at the growth segments of the Docker domain and set up your action plan based on our data driven findings.
Let’s identify the high-growth domains of Docker by looking at the growth in question views in the 6 month period between June 2018 and December 2018. Note that this time we are looking at the number of question views, which I believe is the best measure of real-life usage.
We look at the details of the following segments:
- Established technologies - mature solutions with a stable user base
- Emerging technologies - mature emerging tags
- Trending technologies - smaller tags on the rise
Growth of established technologies
These are the heavy tags that have been around for long enough to have a stable user base and mature solutions. Users look at both old and new questions to find out how to solve daily challenges.
Top growth established tags in the DevOps domain.
| # | Tag name | Views Growth % |
|---|---|---|
| 1 | kubernetes | 145.82 |
| 2 | azure-devops | 88.69 |
| 3 | tfs | 73.90 |
| 4 | linux | 73.54 |
| 5 | java | 62.36 |
| 6 | amazon-web-services | 62.35 |
| 7 | azure | 60.00 |
| 8 | docker | 58.98 |
| 9 | docker-compose | 58.17 |
| 10 | python | 55.13 |
| 11 | continuous-integration | 53.02 |
| 12 | jenkins | 52.03 |
| 13 | devops | 51.07 |
| 14 | amazon-ec2 | 47.49 |
| 15 | git | 38.74 |
Top growth established tags in the Docker domain.
| # | Tag name | Views Growth % |
|---|---|---|
| 1 | docker-swarm | 57.78 |
| 2 | jenkins | 55.16 |
| 3 | kubernetes | 51.48 |
| 4 | docker-compose | 50.69 |
| 5 | python | 49.18 |
| 6 | dockerfile | 47.98 |
| 7 | php | 44.19 |
| 8 | postgresql | 43.23 |
| 9 | mysql | 40.01 |
| 10 | java | 37.47 |
| 11 | docker-machine | 36.68 |
| 12 | amazon-web-services | 36.08 |
| 13 | node.js | 35.62 |
| 14 | nginx | 34.91 |
| 15 | docker | 34.45 |
These are industry trends you should be active in right now.
- Docker Swarm and Kubernetes are on top of the list signaling that you should already have built Docker images for selected applications and it’s time to look into orchestration. Most companies use Kubernetes, but learning Swarm is a great first step and it will help you as you move on to Kubernetes.
- You enjoy the benefits of small containers if you set up the right automation flow leveraging Docker. Jenkins is the most widely adopted automation server for enterprises.
- Docker Compose is a key component in your development flow. Its deployment configuration descriptor, the Compose file, is one of the key artifacts your team produces in the world of containers. You don’t use Compose in production, but it has a crucial role in development and testing.
- The programming languages on the list are in line with our experience in enterprise projects. Most companies use Java for their enterprise applications. They usually use Oracle databases with the enterprise apps, but they usually keep the Oracle databases in place; they don’t containerize them. Most clients use Nginx as a web server or reverse proxy and experiment with PHP, Python, Node.js, Postgres and Mysql containers.
Actions you may take:
- Start learning Docker with special focus on custom Docker images and Compose.
- Start learning container orchestration building Swarm skills and learning Kubernetes in detail.
- Set up your Jenkins development pipeline; start measuring and improving the performance of your production line.
- Start experimenting with small applications first, containerize a web server, a Java or PHP application. Create a proof of concept of a Python automation tool or a Node.js application.
Growth of emerging technologies
These tags are also established with a strong base, although somewhat smaller than the ones in the previous chapter.
Top growth established tags in the DevOps domain.
| # | Tag name | Views Growth % |
|---|---|---|
| 1 | jenkins-plugins | 108.98 |
| 2 | php | 103.84 |
| 3 | devops-services | 85.61 |
| 4 | jenkins-pipeline | 72.96 |
| 5 | continuous-deployment | 71.67 |
| 6 | nginx | 71.08 |
| 7 | sonarqube | 68.51 |
| 8 | automation | 65.62 |
| 9 | maven | 64.83 |
| 10 | node.js | 53.59 |
| 11 | containers | 50.73 |
| 12 | terraform | 46.58 |
| 13 | puppet | 45.35 |
| 14 | build | 40.37 |
| 15 | powershell | 36.51 |
Top growth established tags in the Docker domain.
| # | Tag name | Views Growth % |
|---|---|---|
| 1 | gitlab | 61.19 |
| 2 | spring-boot | 49.62 |
| 3 | bash | 40.28 |
| 4 | django | 38.19 |
| 5 | mongodb | 36.80 |
| 6 | windows | 36.22 |
| 7 | amazon-ec2 | 31.58 |
| 8 | networking | 31.52 |
| 9 | docker-registry | 29.49 |
| 10 | ruby-on-rails | 26.22 |
These are capabilities you should be actively developing.
- I mentioned before that the benefits of Docker are brought by the level of automation you can build in your development workflow. You find GitLab in more and more enterprise projects as the source control and CI/CD tool of choice.
- Sonarqube code inspection is becoming a vital part of automation architectures.
- The growth of Terraform, Puppet, Chef and Ansible signals that we live in the era of end to end automation where infrastructure is no different from traditional code.
- Spring-boot gets you up and running with your Spring based Java application. Spring and Spring boot are the tools that we meet in almost every enterprise environment.
- Bash is a skill you probably haven’t used in a while. As development and opserations both contribute to artifacts like the Dockerfile, your Bash skills are needed again.
- Docker was originally created to be used with Linux based containers. Today you can build, ship and run both Linux and Windows based containers with Docker. Most enterprises use Linux though, but building for Windows, especially in a .Net environment is an option you should consider.
- Picking a good web framework is an important choice. I see both Django and Ruby on rails in independent projects, as well as enterprise applications for smaller, mostly internal systems.
- Amazon Web Services appears on the list of established tags and emerging tags, too. While enterprise clients prefer private clouds and on-premise servers, I see projects that use AWS for ad-hoc computing needs to cover certain aspects of development and testing.
Actions you may take:
- Set up a GitLab environment, add it to your DevOps stack as a source control and CI/CD facility.
- Try Sonarqube in a small project that you are starting now.
- Think of new ways to maximize the value of your configuration management with tool like Ansible.
- Create a Spring boot application with Docker. A Restful API backend would be a great candidate for your first pilot.
- Add a Bash training to your developer curriculum.
- Experiment with Windows based containers, especially if you have .Net applications.
- Consider Django or Ruby on Rails for your next internal web application.
- Identify components in your development and testing workflow that you may host in public cloud and give it a go with AWS.
Growth of trending technologies
These tags are from smaller niche segments.
Top growth established tags in the DevOps domain.
| # | Tag name | Views Growth % |
|---|---|---|
| 1 | kubectl | 6123.68 |
| 2 | conditional | 5018.18 |
| 3 | hashicorp-vault | 4455.56 |
| 4 | multibranch-pipeline | 1233.33 |
| 5 | angular-universal | 1221.13 |
| 6 | metrics | 984.62 |
| 7 | exporter | 984.62 |
| 8 | vue.js | 945.83 |
| 9 | systemd | 817.86 |
| 10 | aws-ecr | 796.82 |
| 11 | circuit-breaker | 757.89 |
| 12 | envoyproxy | 757.89 |
| 13 | istio | 757.89 |
| 14 | graphite | 715.15 |
| 15 | gitlab-api | 700.00 |
Top growth established tags in the Docker domain.
| # | Tag name | Views Growth % |
|---|---|---|
| 1 | traefik | 176.72 |
| 2 | minikube | 130.81 |
| 3 | hyperledger-fabric | 103.87 |
| 4 | docker-for-windows | 100.82 |
| 5 | angular | 96.81 |
| 6 | jenkins-pipeline | 95.41 |
| 7 | sql-server | 88.07 |
| 8 | visual-studio | 87.80 |
| 9 | swarm | 86.09 |
| 10 | visual-studio-2017 | 85.55 |
These are the technologies to watch, and these are the domains you should consider for future skill building.
- Microservices architectures are widely used in scaled web applications and now they are built in various enterprise scenarios, too. I’m also working on such and implementation. Tools that help you build great interoperability and maintainability of your services are amongst the top growing trending tags like Istio, Envoy Proxy, Traefik, Hashicorp Vault, Graphite and the circuit breaker pattern.
- Traefik, the cloud-native reverse-proxy and load-balancer with 650M+ downloads is a great emerging solution for today’s cloud challenges.
- Minikube is the one-node Kubernetes cluster that you can install on your machine and start learning right away.
- Angular is the reactive front-end UI framework used by most enterprises. You find a lot of debate online whether Angular, React or Vue is the best Node.js UI framework today. Enterprise reality is a little different than independent projects; enterprises go for Angular almost exclusively, because it’s a framework with built-in standards out of the box that greatly improve build times and quality with large teams and many applications.
Actions you may take:
- Dive into microservices, find a small, client facing app, set up the architecture and learn and experiment.
- Try Traefik as a reverse-proxy and load-balancer and compare your development and operations metrics to your current solution.
- Encourage your development and operations team members to launch Minikube on their computers and practice Kubernetes techniques.
- Set up a Spring example with Angular and see how you could use this combo to build and deploy new business capabilities faster than your current software frameworks.
Wrap-up
DevOps goes beyond technology and helps you build a new type of relationship with business stakeholders. I hope the 16 action points help you find some good solutions and bring some fun into your daily development life.
Docker is a fantasctic technology that lets you build, ship and run your applications more frequently and more reliably. Docker is a crucial part of your DevOps foundations, because small footprint containers enable a high level of automation in your organization.
Your Docker journey should be an integral part of your DevOps initiatives. Implement a holistic DevOps strategy that covers containers, orchestration, CI/CD and agile practices. Measure the performance of your development pipeline before and after adding new components to your toolkit; always think positive business impact along the way.
