DevOps and Docker Trends 2019 | Based on Stackoverflow Data

Tech trends DevOps and Docker Trends 2019 | Based on Stackoverflow Data

Márk Takács Mark Takacs

Learn Docker Get the book today

Learn to build, ship and run highly scalable applications with Docker containers.

$34.99 $15.99 - Download the book 54% OFF.

The premise of DevOps

I like DevOps more than any other recent technology trend. I like it, because for long-long decades IT had the luxury to hide behind technology jargon and be a little different than other business areas (or THE business areas).

I wish DevOps will finally change this. I whish that our DevOps pipelines go way out to the market and IT stakeholders will talk more and more business terms, meaning time to market of new capabilities, continous feedback from customers, flexible and continuous setting of delivery priorities, reduced cost of poor quality, maximized development throughput and similar measures.

I see the key benefit of DevOps in industrializing the IT development cycle through automation.

New technologies enable us to deploy small business requirements into production frequently, rather than deploying many requirements in a big-bang fashion every few months or less often.

Today we can build small systems that specialize in a certain aspect of our business, choose the right technology to implement such applications and automatically test and even deploy new capabilities as soon as they are checked-in into source control.

Normally such an increase in the number of business support systems and deployments would result in an immense increase in shipment and operations cost.

The reason why we can still put such a smooth development cycle in place is that we can reduce resource requirements through automation. This automation is achieved through DevOps practices and tools.

Why Docker matters

Docker is a central part of the DevOps foundation. If you want to understand how Docker works and how it can contribute to your team’s success, you usually have to look beyond Docker and understand the broader domain of DevOps and Agility.

The reason for this is that Docker fits nicely into our efforts to build an effective CI/CD pipeline, enjoy the benefits of DevOps, Agile development methods and build Microservices.

If you work in an established corporation, you find that there are many moving parts as you try to make progress in all of the above fields.

You have to train many people, find the right projects for early experimentation and get the new tools up and running and use them the way they are meant to be used.

Getting all these things right might still not give you the benefits you are looking for, because you also have to make sure that all these new tools and methods work together with your custom legacy environment that’s like no other, and no-one can give you the exact solutions for your special challenges.

When we look at the Docker related data in this post, we analyze the figures and prepare our proposed actions considering your needs for a streamlined development production line, improvement in delivery speed and quality.

In this article we look into Stackoverflow data to understand how real-life projects implement DevOps and how they use Docker.

Let’s dive right in!

Top tags in the Docker ecosystem

Let’s start with top DevOps and Docker related tags in my Stackoverflow dump from December 2018.

The table below shows the top tags used next to the `devops` tag in Stackoverflow questions based on the number of questions carrying the given tag.

# Tag name Questions
1 devops 1796
2 docker 256
3 jenkins 237
4 amazon-web-services 156
5 ibm-cloud 136
6 continuous-integration 118
7 ansible 117
8 linux 113
9 azure-devops 85
10 git 84
11 kubernetes 80
12 chef 74
13 azure 72
14 tfs 61
15 deployment 57
16 docker-compose 55
17 java 53
18 amazon-ec2 52
19 python 51
20 automation 47
21 jenkins-pipeline 46
22 nginx 44
23 puppet 41
24 continuous-deployment 39
25 terraform 38

Let’s look at the numbers from a slightly different angle showing the number of views on questions marked with the `devops` tag together with a given tag on Stackoverflow.

# Tag name Views
1 devops 918733
2 docker 185582
3 jenkins 162732
4 ansible 109259
5 continuous-integration 76334
6 docker-compose 65835
7 linux 56545
8 amazon-web-services 53008
9 vagrant 46278
10 ansible-playbook 45724
11 terraform 42325
12 kubernetes 42026
13 jenkins-pipeline 40574
14 git 37398
15 sonarqube 37005
16 continuous-delivery 36739
17 ibm-cloud 31719
18 dockerfile 31137
19 gitlab 28106
20 containers 28060
21 deployment 27446
22 java 26155
23 virtualbox 25455
24 hashicorp 23912
25 github 23881

As you implement your DevOps capabilities you usually start with a few base tools and then add more specialized capabilities gradually. I see people presenting crowded DevOps landscape maps (you know, the ones with a plethora of logos in colored boxes) to their clients and expect that they buy-in immediately to the full Monty.

A better approach is to select the minimum set of components that drive positive change in your company’s balance sheet. Based on the tables and our personal experience these are the areas to get started:

  • Docker is on top of both lists. It’s a great automation driver, because you have small footprint operating environment for your applications and this environment is defined as code. You can destroy and rebuild your entire application and its environment from scratch any time and you get the same results.
  • Another tool that most companies add in the beginning is the Jenkins automation server. Jenkins drives your entire pipeline, it triggers a job when developers check-in changes under source control and takes care of building, testing, shipping and even live deployment if you prefer through various environments automatically.
  • Ansible, Chef and Puppet are products that help you manage your infrastructure by automating provisioning and configuration management. Ansible has the highest position in the tables out of these three products. This is in-line with what we see with corporate clients. Ansible is part of the RedHat family of products and I believe this is a definite plus in the enterprise world.
  • GitHub and GitLab are both based on git, the free and open source version control system originally written by Linus Torvalds. Automation tools like Jenkins are mostly used and therefore best integrated with git. This is why most clients set up their own GitLab for DevOps initiatives. GitHub is available as a cloud hosted offering online for organizations whose policies are relaxed enough to go down the public cloud path.
  • The list would not be complete without Kubernetes, the market leading container orchestration platform. Kubernetes does the heavy lifting running your Docker based applications on computer clusters.

You may achieve tangible benefits just by implementing GitLab, Jenkins and Docker with strong automated tests for some of your applications to get started.

Let’s have a few words about some other components on the list:

  • Terraform and Vagrant are both products from HashiCorp. Terraform is a tool for building infrastructure. The key difference between Terraform and Ansible, Chef and Puppet from before, is that you use Terraform to create your infrastructure and you use Ansible, Chef and Puppet to manage the configuration of the infrastructure you built with Terraform. Terraform can build and manage large scale infrastructures across multiple cloud providers. Vagrant is a similar tool, but it’s used to manage development environments locally.
  • Sonarqube is your tool for code inspection. It helps you write high quality code and reduce technical debt by checking your code for bugs, security vulnerabilities, maintainability issues. It integrates well with other tools like Jenkins and Git.

If you look back at the tables, you find that Java is the only programming language that appears on both lists. You see Python and Node.js gain immense popularity lately and their communities are rapidly growing. I am also a great fan of both languages. If you look at corporate reality, you find that Java still dominates application development and therefore it’s a key skill in your DevOps journey.

Let’s have a look what additional info the Docker tables add to the picture!

The table below shows the top tags used next to the `docker` tag in Stackoverflow questions based on the number of questions carrying the given tag.

# Tag name Questions
1 docker 48102
2 docker-compose 7723
3 dockerfile 4630
4 kubernetes 2195
5 python 2150
6 nginx 2065
7 node.js 1931
8 linux 1896
9 containers 1609
10 docker-swarm 1453
11 java 1432
12 mysql 1417
13 ubuntu 1363
14 amazon-web-services 1360
15 jenkins 1357
16 php 1340
17 docker-machine 1067
18 postgresql 1038
19 bash 922
20 windows 884
21 networking 870
22 boot2docker 836
23 django 827
24 macos 799
25 mongodb 789

This next table shows the top Stackoverflow tags used next to the `docker` tag on Stackoverflow questions based on the number of views (the first table was based on the number of questions tagged with the given tag).

# Tag name Views
1 docker 85386016
2 docker-compose 11187822
3 dockerfile 8213045
4 linux 4550187
5 ubuntu 3393618
6 boot2docker 3044807
7 containers 2879586
8 docker-container 2812105
9 nginx 2488078
10 docker-registry 2303199
11 kubernetes 2141158
12 bash 2015336
13 python 1948038
14 mysql 1938592
15 docker-machine 1774725
16 docker-image 1597680
17 node.js 1539068
18 macos 1483438
19 postgresql 1419400
20 php 1399732
21 windows 1351269
22 java 1327754
23 jenkins 1279336
24 networking 1186457
25 lxc 1109725

The numbers confirm that Docker is high on the DevOps agenda.

The top three tags are related to the Docker toolkit; they are `docker` itself, `docker-compose` and `dockerfile`.

This is a simple but significant point right here. These are the top tags, because working with Docker your key deliverables are the Dockerfile and the Compose file.

These artifacts define the working environment of your applications and services, you should add the Dockerfile and Compose file to source control. It is essential that you build strong skills here, our trainings have a special focus on these points.

Once you master the art of custom Docker images and multi-container applications with Compose, you face the challenge of managing your containers in production. The tools to accomplish this are called container orchestrators and Kubernetes is clearly the market leader of this niche. Kubernetes is at the 4th place on the tag list.

Docker’s built-in orchestration solution, Docker Swarm, has also made it to the list; it stands at place 10. Swarm is an alternative to Kubernetes, it comes built in with Docker out of the box. We usually teach Swarm in our Docker trainings, because it’s a great place to understand orchestration. Our experience is that enterprise clients choose Kubernetes exclusively.

Containerization is a means to and end. The main goal in the broader environment is to deploy your applications faster and more frequently, thus increasing the throughput of your entire development line and keep much less work in work in progress for a much shorter period of time.

This gives a lot better experience for business stakeholders, puts features into production faster and gives you flexibility to re-prioritize your backlog more frequently, so that business does not have to wait several months for a new idea to hit the IT production line.

To pursue these goals you need an automation tool that helps you build these capabilities. Jenkins is used by almost every enterprise and Internet client we work with.

A key component of your Docker based delivery pipeline is a Docker registry. This is the place where you push your custom Docker images during your CI/CD builds and that’s the place were you pull your images from in production.

The Docker Registry in the list is Docker’s free solution that lets you set up your own Docker registry. The registry that you get is like having your own Docker Hub.

Enterprises usually go for a different product. A popular solution is the Nexus repository manager by Sonatype.

Let’s look at two figures showing the growth trends of the key tags in DevOps and containerization.

Let’s start with broader DevOps related tags as the percentage of all tags on Stackoverflow.

img

*You can see the highest growth with AWS, Docker and Kubernetes. Jenkins, GitLab, Ansible and Terraform are also growing steadily, while the other tags seem fairly stable.

Let’s have a look at Docker’s core components now.

img

Kubernetes, Compose and Dockerfile are clearly the top tags. This is where you should invest into skills and capabilities, these are the key components to build success with Docker.

It’s time we look at the growth segments of the Docker domain and set up your action plan based on our data driven findings.

Let’s identify the high-growth domains of Docker by looking at the growth in question views in the 6 month period between June 2018 and December 2018. Note that this time we are looking at the number of question views, which I believe is the best measure of real-life usage.

We look at the details of the following segments:

  • Established technologies - mature solutions with a stable user base
  • Emerging technologies - mature emerging tags
  • Trending technologies - smaller tags on the rise

Growth of established technologies

These are the heavy tags that have been around for long enough to have a stable user base and mature solutions. Users look at both old and new questions to find out how to solve daily challenges.

Top growth established tags in the DevOps domain.

# Tag name Views Growth %
1 kubernetes 145.82
2 azure-devops 88.69
3 tfs 73.90
4 linux 73.54
5 java 62.36
6 amazon-web-services 62.35
7 azure 60.00
8 docker 58.98
9 docker-compose 58.17
10 python 55.13
11 continuous-integration 53.02
12 jenkins 52.03
13 devops 51.07
14 amazon-ec2 47.49
15 git 38.74

Top growth established tags in the Docker domain.

# Tag name Views Growth %
1 docker-swarm 57.78
2 jenkins 55.16
3 kubernetes 51.48
4 docker-compose 50.69
5 python 49.18
6 dockerfile 47.98
7 php 44.19
8 postgresql 43.23
9 mysql 40.01
10 java 37.47
11 docker-machine 36.68
12 amazon-web-services 36.08
13 node.js 35.62
14 nginx 34.91
15 docker 34.45

These are industry trends you should be active in right now.

  • Docker Swarm and Kubernetes are on top of the list signaling that you should already have built Docker images for selected applications and it’s time to look into orchestration. Most companies use Kubernetes, but learning Swarm is a great first step and it will help you as you move on to Kubernetes.
  • You enjoy the benefits of small containers if you set up the right automation flow leveraging Docker. Jenkins is the most widely adopted automation server for enterprises.
  • Docker Compose is a key component in your development flow. Its deployment configuration descriptor, the Compose file, is one of the key artifacts your team produces in the world of containers. You don’t use Compose in production, but it has a crucial role in development and testing.
  • The programming languages on the list are in line with our experience in enterprise projects. Most companies use Java for their enterprise applications. They usually use Oracle databases with the enterprise apps, but they usually keep the Oracle databases in place; they don’t containerize them. Most clients use Nginx as a web server or reverse proxy and experiment with PHP, Python, Node.js, Postgres and Mysql containers.

Actions you may take:

  • Start learning Docker with special focus on custom Docker images and Compose.
  • Start learning container orchestration building Swarm skills and learning Kubernetes in detail.
  • Set up your Jenkins development pipeline; start measuring and improving the performance of your production line.
  • Start experimenting with small applications first, containerize a web server, a Java or PHP application. Create a proof of concept of a Python automation tool or a Node.js application.

Growth of emerging technologies

These tags are also established with a strong base, although somewhat smaller than the ones in the previous chapter.

Top growth established tags in the DevOps domain.

# Tag name Views Growth %
1 jenkins-plugins 108.98
2 php 103.84
3 devops-services 85.61
4 jenkins-pipeline 72.96
5 continuous-deployment 71.67
6 nginx 71.08
7 sonarqube 68.51
8 automation 65.62
9 maven 64.83
10 node.js 53.59
11 containers 50.73
12 terraform 46.58
13 puppet 45.35
14 build 40.37
15 powershell 36.51

Top growth established tags in the Docker domain.

# Tag name Views Growth %
1 gitlab 61.19
2 spring-boot 49.62
3 bash 40.28
4 django 38.19
5 mongodb 36.80
6 windows 36.22
7 amazon-ec2 31.58
8 networking 31.52
9 docker-registry 29.49
10 ruby-on-rails 26.22

These are capabilities you should be actively developing.

  • I mentioned before that the benefits of Docker are brought by the level of automation you can build in your development workflow. You find GitLab in more and more enterprise projects as the source control and CI/CD tool of choice.
  • Sonarqube code inspection is becoming a vital part of automation architectures.
  • The growth of Terraform, Puppet, Chef and Ansible signals that we live in the era of end to end automation where infrastructure is no different from traditional code.
  • Spring-boot gets you up and running with your Spring based Java application. Spring and Spring boot are the tools that we meet in almost every enterprise environment.
  • Bash is a skill you probably haven’t used in a while. As development and opserations both contribute to artifacts like the Dockerfile, your Bash skills are needed again.
  • Docker was originally created to be used with Linux based containers. Today you can build, ship and run both Linux and Windows based containers with Docker. Most enterprises use Linux though, but building for Windows, especially in a .Net environment is an option you should consider.
  • Picking a good web framework is an important choice. I see both Django and Ruby on rails in independent projects, as well as enterprise applications for smaller, mostly internal systems.
  • Amazon Web Services appears on the list of established tags and emerging tags, too. While enterprise clients prefer private clouds and on-premise servers, I see projects that use AWS for ad-hoc computing needs to cover certain aspects of development and testing.

Actions you may take:

  • Set up a GitLab environment, add it to your DevOps stack as a source control and CI/CD facility.
  • Try Sonarqube in a small project that you are starting now.
  • Think of new ways to maximize the value of your configuration management with tool like Ansible.
  • Create a Spring boot application with Docker. A Restful API backend would be a great candidate for your first pilot.
  • Add a Bash training to your developer curriculum.
  • Experiment with Windows based containers, especially if you have .Net applications.
  • Consider Django or Ruby on Rails for your next internal web application.
  • Identify components in your development and testing workflow that you may host in public cloud and give it a go with AWS.

These tags are from smaller niche segments.

Top growth established tags in the DevOps domain.

# Tag name Views Growth %
1 kubectl 6123.68
2 conditional 5018.18
3 hashicorp-vault 4455.56
4 multibranch-pipeline 1233.33
5 angular-universal 1221.13
6 metrics 984.62
7 exporter 984.62
8 vue.js 945.83
9 systemd 817.86
10 aws-ecr 796.82
11 circuit-breaker 757.89
12 envoyproxy 757.89
13 istio 757.89
14 graphite 715.15
15 gitlab-api 700.00

Top growth established tags in the Docker domain.

# Tag name Views Growth %
1 traefik 176.72
2 minikube 130.81
3 hyperledger-fabric 103.87
4 docker-for-windows 100.82
5 angular 96.81
6 jenkins-pipeline 95.41
7 sql-server 88.07
8 visual-studio 87.80
9 swarm 86.09
10 visual-studio-2017 85.55

These are the technologies to watch, and these are the domains you should consider for future skill building.

  • Microservices architectures are widely used in scaled web applications and now they are built in various enterprise scenarios, too. I’m also working on such and implementation. Tools that help you build great interoperability and maintainability of your services are amongst the top growing trending tags like Istio, Envoy Proxy, Traefik, Hashicorp Vault, Graphite and the circuit breaker pattern.
  • Traefik, the cloud-native reverse-proxy and load-balancer with 650M+ downloads is a great emerging solution for today’s cloud challenges.
  • Minikube is the one-node Kubernetes cluster that you can install on your machine and start learning right away.
  • Angular is the reactive front-end UI framework used by most enterprises. You find a lot of debate online whether Angular, React or Vue is the best Node.js UI framework today. Enterprise reality is a little different than independent projects; enterprises go for Angular almost exclusively, because it’s a framework with built-in standards out of the box that greatly improve build times and quality with large teams and many applications.

Actions you may take:

  • Dive into microservices, find a small, client facing app, set up the architecture and learn and experiment.
  • Try Traefik as a reverse-proxy and load-balancer and compare your development and operations metrics to your current solution.
  • Encourage your development and operations team members to launch Minikube on their computers and practice Kubernetes techniques.
  • Set up a Spring example with Angular and see how you could use this combo to build and deploy new business capabilities faster than your current software frameworks.

Wrap-up

DevOps goes beyond technology and helps you build a new type of relationship with business stakeholders. I hope the 16 action points help you find some good solutions and bring some fun into your daily development life.

Docker is a fantasctic technology that lets you build, ship and run your applications more frequently and more reliably. Docker is a crucial part of your DevOps foundations, because small footprint containers enable a high level of automation in your organization.

Your Docker journey should be an integral part of your DevOps initiatives. Implement a holistic DevOps strategy that covers containers, orchestration, CI/CD and agile practices. Measure the performance of your development pipeline before and after adding new components to your toolkit; always think positive business impact along the way.

Twitter Facebook LinkedIn