DCOS (or DataCenter OS) is similar to Kubernetes in many ways. DCOS pools compute resources into a uniform task pool. The big difference is that DCOS targets many different types of workloads, including but not limited to, containerized applications. This makes DCOS attractive for organizations that are not using containers for all oftheir applications. DCOS also includes a kind of package manager to easily deploy systems like Kafka or Spark. You can even run Kubernetes on DCOS given its flexibility for different types of workloads.
Here are my general recommendations:
- Use Kubernetes if you’re only working with containerized applications that may or may not be only Docker.
- If you have a mix of container and non-containerized applications, use DCOS.
- Use ECS if you enjoy AWS products and first party integrations.
- If you want a first party solution or direct integration with the Docker toolchain, use Docker Swarm.
Kubernetes (k8s) makes for an amazing developer story. Mesos is much more bare metal, but the scheduler scales a loooot better than the still relatively immature k8s scheduling component. One of the original authors of mesos wrote a paper on scheduling: https://www.cs.berkeley.edu/~alig/papers/drf.pdf. Mesos is one of the first “two level” schedulers. I very highly recommend that you also read this article for an idea of how this is a good idea: http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.htm…
The k8s upstream was forward thinking enough to make the scheduler parts of it pluggable, which allow the (imo) holy grail of something like this https://github.com/mesosphere/kubernetes-mesos. This gives you the nice logical “pod” description for services, running on the battle tested mesos scheduler.
There are many 10k+ node bare metal mesos deployments (apple, twitter, etc). There aren’t yet many kubernetes deployments of that scale. They truly are mutually beneficial. Mesos makes ops happy, and k8s makes devs happy. Together you have a relatively easy to setup internal PaaS (your own heroku with not a ton of work) more or less.
- Mesos allows dynamic sharing of cluster resources between Kubernetes and other first-class Mesos frameworks such as HDFS, Spark, and Chronos. Mesos also ensures applications from different frameworks running on your cluster are isolated and that resources are allocated fairly among them.
- Mesos clusters can be deployed on nearly every IaaS cloud provider infrastructure or in your own physical datacenter. Kubernetes on Mesos runs on-top of that and therefore allows you to easily move Kubernetes workloads from one of these environments to the other.
Though it’s a bit outdated, it may be helpful to read What’s the difference between Apache’s Mesos and Google’s Kubernetes, to get some of the basics right. Also, note that Mesos operates on a different level than Kubernetes/Marathon/Chronos. Last but not least, see Docker Swarm + Mesos by Timothy Chen, keeping in mind that Marathon and Swarm can operate simultaneously on the same Mesos cluster.
DC/OS removes the complexity of managing distributed workloads running on platforms such as Hadoop, Spark, Kafka, and even Docker.
Mesosphere developed DCOS a couple of years ago as a commercial offering for enterprises. Apart from open sourcing DCOS with a new branding, Mesosphere has turned DC/OS into a collaborative project. Over 50 founding members became a part of the community. Prominent among those are Autodesk, Canonical, Cisco, Citrix, EMC, HPE, Joyent, Microsoft, NetApp, and Verizon. These companies pledged their support to deploy DC/OS and also to contribute to the effort.
Containers and microservices are hogging the limelight for developing and deploying distributed applications. Even if the workload doesn’t demand web-scale computing, containerization offers many benefits to customers. That’s one of the reasons why there has been an increase in the adoption of containers among companies of all sizes. These applications built on containers are called cloud-native applications.
What problem does DC/OS solve for the enterprises running the new breed of applications? Though containerization offers massive scale, it has one significant gap – lack of tight integration with existing stateful applications. Enterprises have invested in relational databases, data ware houses, and big data platforms to run their line-of-business applications. DC/OS is designed to manage both stateful workloads and stateless applications within the same environment. It bridges the gap between the traditional enterprise LoB applications and the contemporary cloud-native applications.
For example, DC/OS can handle the scale-out web tier running in containers that talk to an Apache Hadoop cluster, and a highly available Cassandra cluster. Enterprise customers can use Marathon for orchestrating containerized applications while using Chronos for scheduling long-running tasks. This unified environment to run persistent, stateful applications along with the scale-out containerized applications is the key differentiating factor of DC/OS.
While Docker has become the de facto standard for packaging and deploying containers, customers need an additional management layer for orchestrating containerized applications. Software such as Docker Swarm and Kubernetes offer higher level container management solutions. Docker Swarm is developed and maintained by Docker, Inc. It’s known for its simplicity and tight integration with the Docker environment. Google decided to open source a subset of its internal data center management tool called Borg, which resulted in Kubernetes. Interestingly, DC/OS can manage both Docker Swarm and Kubernetes deployments.
Last year, Google took the lead in forming Cloud Native Computing Foundation (CNCF), a consortium to drive the adoption of cloud-native applications. Joyent, CoreOS, IBM, VMWare, Cisco, Weaveworks, and others are actively involved in this initiative. CNCF is managed by the Linux Foundation as one of the collaborative projects. Mesosphere has been the founding member of the Cloud Native Computing Foundation.
One of the early wins for Mesosphere came from Microsoft in the form of Azure Container Service (ACS). Coinciding with the DC/OS announcement, Microsoft made its container orchestration service generally available. ACS is a competitor to other CaaS offerings such as Google Container Engine and Amazon EC2 Container Service. Amazon and Microsoft never endorsed Kubernetes officially. Both the companies have also stayed away from Cloud Native Computing Foundation. Amazon’s CaaS is based on a proprietary orchestration engine built by internal teams at AWS. Azure Container Service, that is just launched, supports Docker Swarm and DC/OS but not Kubernetes.
Microsoft is not just a consumer of DC/OS technology. It’s one of the key investors in Mesosphere. Along with HP Enterprise, A Capital, and Triangle Peak Partners, Microsoft has contributed to an investment of USD 73.5 million. In the coming months, DC/OS technology will play a crucial role in Microsoft’s hybrid cloud strategy. It may power Azure Stack, Microsoft’s ambitious private cloud platform, which will be shipped later this year. With Azure Container Service on the public cloud and DC/OS in the private cloud, Microsoft will have a compelling hybrid story for the enterprise. Other technologies such as HDInsight on Windows Server may also leverage DC/OS deployed within customer data centers.
Though Mesosphere officially didn’t create a consortium or foundation for DC/OS, it managed to create a vibrant community of supporters, which has the potential to become a parallel force to the Cloud Native Computing Foundation. While CNCF is closely associated with Kubernetes, the new community led by Mesosphere and Microsoft will drive the agenda of DC/OS.
Given the technical credibility of Mesos and the backing from Microsoft, I expect DC/OS to emerge as a viable alternative to Kubernetes.
Here are some of my observations from about a year ago, which may be out of date:
- Kubernetes takes a much more batteries-included approach, whereas the Mesos ecosystem is more about layering functionality on top of Mesos.
- Kubernetes is an opinionated packaging of Google’s philosophy for developing, configuring, deploying, and running software. Mesos is great for when you want a lot of control and customizability.
- Kubernetes is a faster way to get started with container orchestration compared to setting up ZooKeeper, Mesos, Marathon, and etc. Kubernetes has some admittedly wonky scripts that can spin up a relatively production-ready cluster within a few minutes. You can do this with something like Mesosphere, but then you are shelling out a good deal. There are non-official playbooks/modules for launching Mesos clusters, but you’ll need to spend the time vetting and verifying them.
- For businesses with huge (800+ VM/machine) fleets, the Mesos ecosystem may be a better decision if you want fewer, larger clusters.
- A Mesos setup looks to require more operational management, given that you will be upgrading more pieces of your stack separately (Marathon, Chronos, etc). Kubernetes does these in monolithic monthly releases.
To understand why Kubernetes and the DCOS are friends, not foes, let’s look at an analogy from the world of operating systems and web browsers. Think for a moment of a MacBook that comes pre-installed with MacOS and the Safari browser or a Windows machine that has Internet Explorer on it per default. Some people are happy to use the default browser on their machine, while others choose to install their favorite alternative like Firefox or Chrome and mainly use those browsers. Some people even install and regularly use multiple browsers simultaneously.
Browsers and operating systems provide a very strong analogy because there is an equivalent of this in the DCOS: we ship it with Marathon pre-installed and configured — which is also, by the way, used to launch long-running services in your cluster (like init or systemd on a Linux machine) — but it’s up to you to decide if you want to use Marathon to build and deploy your applications. You may choose to go with Marathon or you may choose to use Kubernetes. Or, you might decide to use both: each for different workloads and use cases (yes, with DCOS you can run Marathon and Kubernetes workloads on the same cluster at the same time without conflict, just like running multiple browsers on your laptop). The tools you use with DCOS are based on your requirements and preferences, entirely.
Kubernetes is a first-class citizen (package as we call it) of the DCOS. Kubernetes is also a community project we heavily contribute to: we have a large team of people at Mesosphere working on Kubernetes, from the open source engineering to product and training.
Mesosphere customers have the choice of offering Kubernetes to their developers. They can also use the pre-installed Marathon. Or, they can use any number of other emerging development models, including many PaaS’s that run on top of DCOS. That is a key part of what it means to be a Datacenter Operating System. The DCOS gives its users a way to adopt as many workload types and development models that work best for their uses. The DCOS is the only system that provides this kind of inherent flexibility and eliminates lock in. The DCOS is also the only system that lets you run multiple services—not just Marathon and Kubernetes, but also services like Spark and Cassandra—all on the same cluster, sharing resources and automating operations.
Kubernetes is per its architecture plug-able:
We partnered with Google to leverage the extensibility of Kubernetes and build Kubernetes-Mesos, the open source project that integrates with the Kubernetes scheduling API and the Mesos scheduler runtime, as well as provides an executor component that integrates kubelet services and the Mesos executor runtime:
As the above diagram illustrates: Kubernetes is deeply integrated with Mesos and, by extension, into the DCOS. It is simple to set up (a single command,
dcos package install kubernetes does the trick) and you’re good to go. So you can use Kubernetes on DCOS right now. If you’re prepared to go all-in and package 100% of your applications in containers—that is, you’re comfortable to using Docker 100% of the time, then you have the choice between Kubernetes and Marathon. Which one you pick depends mainly on your preference for Kubernetes’ pod/labels/services model or Marathon’s apps/groups/dependency operating model. Last but not least a consideration, at least for the time being, is scalability—Marathon has been around a lot longer than Kuberentes and has been proven to scale to much larger clusters and much larger workloads—but we’re working with the Kubernetes community to help Kubernetes catch up.