| layout | title | description |
|---|---|---|
default |
Beginners Track - Features under Docker Engine 18.09 Release |
collabnix | DockerLab | Docker - Beginners Track |
Docker Engine v18.09.1 went GA during January 2019. It was made available for both the Community and Enterprise users. Docker Enterprise is a superset of all the features in Docker Community Edition. It incorporates defect fixes that you can use in environments where new features cannot be adopted as quickly for consistency and compatibility reasons.
New in 18.09 is an aligned release model for Docker Engine – Community and Docker Engine – Enterprise. The new versioning scheme is YY.MM.x where x is an incrementing patch version. They will ship concurrently with the same x patch version based on the same code base.
[Updated – 2/15/2019]:Security fixes for Docker Engine – Enterprise and Docker Engine – Community Update runc to address a critical vulnerability that allows specially-crafted containers to gain administrative privileges on the host. CVE-2019-5736 Ubuntu 14.04 customers using a 3.13 kernel will need to upgrade to a supported Ubuntu 4.x kernel Docker Engine v18.09.1 comes with dozens of new features, improvements and bug fixes. Let us go through the list of Top features which you should be aware of while you upgrade to Docker 18.09.1 release.
- BuildKit 0.3.3 is Available & is out of experimental mode
- Support for Compose on Kubernetes
- Exposing Product Info under docker info command
- Process isolation on Windows 10 for the first time
- Support for remote connections using SSH
- Support for SSH agent socket forwarder
- Support for “registry-mirrors” and “insecure-registries” when using BuildKit
- Support for build-time secrets using a --secret flag when using BuildKit
- Support for docker build –pull … when using BuildKit
- Docker Engine v18.09.1 comes with containerd 1.2.2 version
Docker Engine v18.09.2 release for CE and EE both is shipped with containerd 1.2.2 version. In Docker versions prior to 18.09, containerd was managed by the Docker engine daemon. In Docker Engine 18.09, containerd is managed by systemd.
Let us try installing the latest Docker 18.09.2 on a fresh Ubuntu 18.10 VM and try verifying the containerd version.
sudo curl -sSL https://get.docker.com/ | sh
# Executing docker install script, commit: 26dda3d
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sudo -E sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sudo -E sh -c echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu cosmic edge" > /etc/apt/sources.list.d/docker.list
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
+ sudo -E sh -c docker version
Client:
Version: 18.09.2
API version: 1.39
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:46 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 03:42:13 2019
OS/Arch: linux/amd64
Experimental: false
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:
sudo usermod -aG docker robertsingh181
Remember that you will have to log out and back in for this to take effect!
WARNING: Adding a user to the "docker" group will grant the ability to run
containers which can be used to obtain root privileges on the
docker host.
Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
for more information.
@collabnix:~$ sudo usermod -aG docker robertsingh181
Verifying Docker Version
$sudo docker version
Client:
Version: 18.09.2
API version: 1.39
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:46 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 03:42:13 2019
OS/Arch: linux/amd64
Experimental: false
Please Note: The client and container runtime are now in separate packages from the daemon in Docker Engine 18.09. Users should install and update all three packages at the same time to get the latest patch releases.
For example, on Ubuntu: sudo apt install docker-ce docker-ce-cli containerd.io.
$ sudo dpkg --list | grep container
ii containerd.io 1.2.2-3 amd64 An open and reliable container runtime
ii docker-ce 5:18.09.2~3-0~ubuntu-cosmic amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:18.09.2~3-0~ubuntu-cosmic amd64 Docker CLI: the open-source application container engine
Verifying Containerd Version
$ sudo containerd --help
NAME:
containerd -
__ _ __
_________ ____ / /_____ _(_)___ ___ _________/ /
/ ___/ __ \/ __ \/ __/ __ `/ / __ \/ _ \/ ___/ __ /
/ /__/ /_/ / / / / /_/ /_/ / / / / / __/ / / /_/ /
\___/\____/_/ /_/\__/\__,_/_/_/ /_/\___/_/ \__,_/
high performance container runtime
USAGE:
containerd [global options] command [command options] [arguments...]
VERSION:
1.2.2
COMMANDS:
config information on the containerd config
publish binary to publish events to containerd
oci-hook provides a base for OCI runtime hooks to allow arguments to be injected.
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--config value, -c value path to the configuration file (default: "/etc/containerd/config.toml")
--log-level value, -l value set the logging level [trace, debug, info, warn, error, fatal, panic]
--address value, -a value address for containerd's GRPC server
--root value containerd root directory
--state value containerd state directory
--help, -h show help
--version, -v print the version
$ sudo ctr version
Client:
Version: 1.2.2
Revision: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
Server:
Version: 1.2.2
Revision: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
Since containerd is managed by systemd under v18.09.1 release, any custom configuration to the docker.service systemd configuration which changes mount settings (for example, MountFlags=slave) breaks interactions between the Docker Engine daemon and containerd, and you will not be able to start containers. Run the following command to get the current value of the MountFlags property for the docker.service:
sudo systemctl show --property=MountFlags docker.service
MountFlags=
Update your configuration if this command prints a non-empty value for MountFlags, and restart the docker service.
BuildKit is a new project under the Moby umbrella for building and packaging software using containers. It is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner. Docker 18.09.0 is the first release with buildkit support. With this latest release, you can now run Buildkit without experimental mode enabled. Buildkit can now be configured with an option in daemon.json too.
The docker build is a Docker integrated tool for building images using Dockerfile. It requires Docker daemon to be running. It’s similar to docker run command but some features are intentionally removed for security reasons like no volumes(docker run -v, docker run --mount) and no privileged mode(`docker run –privileged). Buildkit is meant to become the next generation backend implementation for docker build command and github.com/docker/docker/builder package. This doesn’t mean any changes to Dockerfile format as buildkit draws a boundary between build backends and frontends. Dockerfile would be one of the frontend implementations. When invoked from the Docker CLI buildkit would be capable of exposing clients context directory as a source and use Docker containers as a worker. The snapshots would be backed by Docker’s layer store(containerD snapshot drivers). End results from the builder would be exported to docker images.
If you look at Dockerfile, we are reading all commands in Dockerfile from start to the end one by one until we reach to the end.Modifying a single line always invalidates the caches of the subsequent lines. For example: N-th line is assumed to be always dependent on (N-1)th line
FROM debian
EXPOSE 80
RUN apt update && apt install git
As shown above, modifying the 2nd line(EXPOSE 80) always invalidates the apt cache due to false dependency. A user need to arrange the instructions carefully for efficient caching. This brings inefficient caching problem.
Not only this, inaccessible to private assets is another major problem with traditional Dockerfile. There is no safe way to access private assets(eg. Git Repos, S3) from build containers. Hence, copying credentials using COPY can leak the credentials accidently.Buildkit solves the above problems by using DAG-style low level language called LLB.
The main areas BuildKit improves on the current experience are performance, storage management, and extensibility. From the performance side, a significant update is a new fully concurrent build graph solver. It can run build steps in parallel when possible and optimize out commands that don’t have an impact on the final result. We have also optimized the access to the local source files. By tracking only the updates made to these files between repeated build invocations, there is no need to wait for local files to be read or uploaded before the work can begin.
Let us compare docker build Vs Buildkit and see how Buildkit builds the Docker image a more faster than traditional approach.
$ git clone https://github.com/ajeetraina/hellowhale
Cloning into 'hellowhale'...
remote: Enumerating objects: 28, done.
remote: Total 28 (delta 0), reused 0 (delta 0), pack-reused 28
Unpacking objects: 100% (28/28), done.
~$ cd hellowhale/
~$ ls
Dockerfile README.md html wrapper.sh
:~/hellowhale$ time docker build -t ajeetraina/hellowhale .
Sending build context to Docker daemon 153.1kB
Step 1/4 : FROM nginx:latest
latest: Pulling from library/nginx
6ae821421a7d: Pull complete
da4474e5966c: Pull complete
eb2aec2b9c9f: Pull complete
Digest: sha256:dd2d0ac3fff2f007d99e033b64854be0941e19a2ad51f174d9240dda20d9f534
Status: Downloaded newer image for nginx:latest
---> f09fe80eb0e7
Step 2/4 : COPY wrapper.sh /
---> 10d671c6cf08
Step 3/4 : COPY html /usr/share/nginx/html
---> 3e8a09f56168
Step 4/4 : CMD ["./wrapper.sh"]
---> Running in b1f24992f9e5
Removing intermediate container b1f24992f9e5
---> 9dae85ca0867
Successfully built 9dae85ca0867
Successfully tagged ajeetraina/hellowhale:latest
real 0m6.359s
user 0m0.035s
sys 0m0.022s
Let’s build it with buildkit:
time docker build -t ajeetraina/hellowhale .
[+] Building 1.7s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 135B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/nginx:latest 0.0s
=> [internal] helper image for file operations 0.4s
=> => resolve docker.io/docker/dockerfile-copy:v0.1.9@sha256:e8f159d3f00786604b93c675ee2783f8dc194bb565e61ca5788f6a6e9d304061 0.7s
=> => sha256:e8f159d3f00786604b93c675ee2783f8dc194bb565e61ca5788f6a6e9d304061 2.03kB / 2.03kB 0.0s
=> => sha256:a546a4352bcaa6512f885d24fef3d9819e70551b98535ed1995e4b567ac6d05b 736B / 736B 0.0s
=> => sha256:494e63343c3f0d392e7af8d718979262baec9496a23e97ad110d62b9c90d6182 766B / 766B 0.0s
=> => sha256:df3b4bed1f63b36992540a09e0d10bd3f9d0b082d50810313841d745d7cce368 898.21kB / 898.21kB 0.2s
=> => sha256:f7b6696c3fee7264ec4486cebe146a6a98aa8d1e46747843107ff473aada8d56 861.00kB / 861.00kB 0.2s
=> => extracting sha256:df3b4bed1f63b36992540a09e0d10bd3f9d0b082d50810313841d745d7cce368 0.1s
=> => extracting sha256:f7b6696c3fee7264ec4486cebe146a6a98aa8d1e46747843107ff473aada8d56 0.1s
=> [1/3] FROM docker.io/library/nginx:latest 0.0s
=> => resolve docker.io/library/nginx:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 34.39kB 0.0s
=> [2/3] COPY wrapper.sh / 0.2s
=> [3/3] COPY html /usr/share/nginx/html 0.2s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:db60ac4c90d7412b8c9f9382711f0d97a9ad9d4a33c05200aa36dc4c935c8cb3 0.0s
=> => naming to docker.io/ajeetraina/hellowhale 0.0s
real 0m1.732s
user 0m0.042s
sys 0m0.019s
~/hellowhale$
Buildkit took just 0m1.732sec compared to 0m6.359sec taken by traditional docker build method.
Below are the list of dozens of enhancements which comes around Buildkit and I shall be discussing these in details in my upcoming blog posts:
- Added support for build-time secrets using a --secret flag when using BuildKit docker/cli#1288
- Added builder prune subcommand to prune BuildKit build cache docker/cli#1295docker/cli#1334
- Adds configurable garbage collection policy for the BuildKit build cache docker-archive/engine#59 / moby/moby#37846
- Adds support for docker build --pull ... when using BuildKit moby/moby#37613
- Adds support or “registry-mirrors” and “insecure-registries” when using BuildKit docker-archive/engine#59 / moby/moby#37852
- Enables net modes and bridge. moby/moby#37620
- Changes --console=[auto,false,true] to --progress=[auto,plain,tty]docker/cli#1276
- Sets BuildKit’s ExportedProduct variable to show useful errors in the future. moby/moby#37439
Compose on Kubernetes allows you to deploy Docker Compose files onto a Kubernetes cluster. Compose on Kubernetes comes installed on Docker Desktop and Docker Enterprise. On Docker Desktop you will need to activate Kubernetes in the settings to use Compose on Kubernetes.
Check out my last 2 blogs which talks about how Compose on Kubernetes work for Play with Kubernetes & Minikube.
Under Docker v18.09.1 it is now possible to verify if it is an enterprise or community product as shown below:
Process-isolation containers were already possible on Windows Server, but for the first time they are now also available on the regular Windows 10 of your laptop. Windows 10–1809 (“October 2018 Update”) + Docker engine 18.09.1 + the Windows 1809 base images from Dockerhub are the first combination that allows you to run “real” Windows containers on Windows 10, without the need of Hyper-v virtualization.
One need to install the Docker Desktop Edge version 2.0.1.0 or newer. Please remember that Docker Engine should be at version 18.09.1 or higher.You must select a Windows base image from Dockerhub that matches the kernel of your host’s Windows version. For Windows 10–1809, that’s the :1809version/tag of nanoserver, servercore and windows (or any higher-level image, that builds up on one of these).
Let us run a test container in process-isolation mode by adding the parameter --isolation=process :
docker run --isolation=process mcr.microsoft.com/windows/nanoserver:1809 cmd.exe /c ping 127.0.0.
In case you’re on Windows 10 Build 1706 and try to run the below command, it won’t work as expected. One need to be running atleast 1809 build for smooth operation.
Docker Engine v18.09 offers the possibility for a Docker client to communicate with a remote daemon via ssh. The Docker client communicates usually with the daemon either locally, via the unix socket /var/run/docker.sock, or over a network via a tcp socket. With Docker 18.09.1 you can now SSH to remote Docker host and execute docker CLI flawlessly.
This new connection method between client and engine allows for simple and shared SSH configuration which is far more common and more easily administered that the prior custom CA/certs solution via the docker cli, it is easily done by setting the env var DOCKER_HOST=ssh://hostname or directly on the docker command using the -H parameter as in
docker -H ssh://hostname info
$ docker -H ssh://[email protected] run -ti ubuntu echo “hello”
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
6cf436f81810: Pull complete
987088a85b96: Pull complete
b4624b3efe06: Pull complete
d42beb8ded59: Pull complete
Digest: sha256:7a47ccc3bbe8a451b500d2b53104868b46d60ee8f5
Please note that you might need to configure SSH key based login and run an SSH agent so you only need to enter a passphrase once. Especially if you are on cloud platform you will need SSH passphrase to allow this command to work as it directly doesn’t allow SSH from one cloud instance to another.
Below are further details of features which I am planning to discuss in my upcoming blog posts:
- Added SSH agent socket forwarder (docker build --ssh $SSHMOUNTID=$SSH_AUTH_SOCK) when using BuildKit docker/cli#1438 / docker/cli#1419
- Added --chown flag support for ADD and COPY commands on Windows moby/moby#35521
- Added docker engine subcommand to manage the lifecycle of a Docker Engine running as a privileged container on top of containerd, and to allow upgrades to Docker Engine Enterprise docker/cli#1260
- Added “local” log driver moby/moby#37092
- Configured containerd log-level to be the same as dockerd moby/moby#37419
Please note that the client and container runtime are now in separate packages from the daemon in Docker Engine 18.09. Users should install and update all three packages at the same time to get the latest patch releases. For example, on Ubuntu:sudo apt install docker-ce docker-ce-cli containerd.io.
Hope you found this blog informative. In case of any query, feel free to drop me a message or join me at DockerLabs
Refer: Top 5 Features of Docker Engine v18.09.1 That You Shouldn’t Miss Out