At this week’s CoreOS Fest in San Francisco, CoreOS is – unsurprisingly – pushing hard on the Application Container Spec (appc) and its first implementation, rkt, making it the topic of the first session after the keynote and a bold story about broad adoption.
When making technology decisions, Red Hat continuously evaluates all available options with the goal of selecting the best technologies that are supported by upstream communities. This is why Red Hat is engaging upstream in appc to actively contribute to the specification.
Red Hat engages in many upstream communities. However, this engagement should not imply full support, or that we consider appc or rkt ready for enterprise IT, or that the specification will ever be considered “enterprise-ready.” With that said, here are some of my observations after listening to the panel discussion this morning:
There is a lot to like about rkt – the clean execution model around systemd, the open development and governance model, even the spec itself.
Let’s take a look at the spec.
The App Container spec is definitely a great step in the right direction. It addresses many of the shortcomings of Docker v1 on both the technology-level and the development model, as well as the decision-making process, which has been perceived as difficult by some users. The App Container spec displays a lot of what we would be looking for from a container spec and I honestly believe that there is a need for a spec of this nature in the enterprise container world. I also think that CoreOS has the best intentions with the spec and the community that they are building around it.
As it is today, appc only covers a very small part of what needs to be sorted out for a container-based, application-centric model. The impact of containerization in redefining the enterprise OS is still vastly underestimated by most; it is a departure from the traditional model of a single-instance, monolithic, UNIX user space in favor of a multi-instance, multi-version environment using containers and aggregate packaging. We are talking about nothing less than changing some of the core paradigms on which the software industry has been working for the last 20 – if not 40 – years.
There are still a lot of moving parts in terms of security, syscalls, signals, and more. The spec is an evolving target as even beyond its current scope, there is a need to go deeper in tracking compatibilities, dependencies on the underlying system, operating systems, federated registries namespaces, and discovery.
Another important scope issue is primarily induced by the name “Application” – at this point, the appc covers individual containers up to a pod. Unfortunately – and this is a problem as much for where Docker is right now – there are very few applications that consist of only a single container / single pod. This gap seems to be reflected in suggestions like the introduction of multi-node pods raised in today’s appc panel. There still is a higher-level concept missing for actual application portability. A first attempt at addressing this can be found in the current draft for the Nulecule [pronounce: newly-cool] spec: it expands the concept of portability to the full application – or a service as a component for further aggregation – and allows describing the multi-container application with all its dependencies.
So while the foundation to address a lot of this is laid in appc, we are really only at the beginning of a long journey in which Red Hat will continue to engage through our community involvement.
It’s important to note that an even bigger concern than the early stage of appc, is that rkt also introduces its own container package format in addition to supporting a compatibility interface to the docker model. I have long credited Docker for the brilliant idea to combine aggregate packaging, image layering, and containers. This gives us a tool to change the industry’s approach to software delivery. The introduction of multiple different package formats to implement this concept introduces risks and slows momentum as the projects compete for mindshare. At the end of the day, it will mean fragmentation of the ecosystem, headaches for companies that provide actual content for containers, including Red Hat and our ISV partners, as well as those users trying to adopt containerization.
On the lower component level, we have had to live with this fragmentation for a long time when working with dpkg versus rpm versus a bunch of marginal others. In retrospect, looking at apt/dpkg versus yum/rpm the conclusion has to be that the competition between the two components has not significantly benefited users – quite the opposite. Essentially, it means that we have to provide the same content in multiple formats and either enable the whole toolchain and system architecture to manage two highly redundant formats in parallel or even worse, force customers to choose one ecosystem over the other. The higher level of aggregation with containers only marginally improves this issue because of the higher complexity and deeper integration in modern distributed systems.
The elephant in the room is that appc does not have the support of Docker, the company that leads in mindshare and controls the format that has quickly become the defacto standard for container packaging. They were not even represented at the panel to share their point of view.
At the same time, the spec is evolving, and rkt is gaining some level of interest. Docker is working on v2 of its package format, protocol and APIs, and the company has significantly improved its approach to community integration.
This is not to unduly criticise Docker or CoreOS, each has valid reasons to be on their respective paths. The reality is, we have two directly competing efforts to advance aspects of container standardization with competing toolchains, competing specs, and incompatible packaging formats. This is like watching a train-wreck in slow motion on Groundhog Day. We, in the broader Linux and open source community, have been down this path multiple times over the past fifteen years, specifically with package formats.
While there needs to be room for experimentation, having two incompatible specs driven by two startups trying to differentiate and in direct competition is *not* a good thing. It would be better for the community and for everyone who depends on our collective efforts if CoreOS and Docker collaborated on a standardized common spec, image format, and distribution protocol. To this end, we at Red Hat will continue to contribute to both initiatives with the goal of driving convergence.