Defining start conditions for the container

youtube-dl/

Next, we will start moving towards a more meaningful image. youtube-dl is a program that downloads youtube videos https://rg3.github.io/youtube-dl/download.html. Let's add it to an image - but this time, we will change our process. Instead of our current process where we add things to the Dockerfile hope it works, let's try another approach. This time we will open up an interactive session and test stuff before "storing" it in our Dockerfile.

docker run -it ubuntu:22.04

And let's also create a new Dockerfile with the "same" content

FROM ubuntu:22.04

And that's it - from the recipe point of view this is the only thing we need to persist.

Just train our muscle memory, let's build the image already

docker build -t youtube-dl .

The resulting image is basically the same as ubuntu:22.04, see

$ docker image ls | grep "22.04\|youtube-dl"
  ubuntu                                     22.04     3c2df5585507   7 days ago       69.2MB
  youtube-dl                                 latest    adc4f063454b   7 days ago       69.2MB

Image ID is different, but the size matches.

Now by following the youtube-dl install instructions we will see that:

root@8c587232a608:/# curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl
bash: curl: command not found

..and, as we already know, curl is not installed - let's add curl with apt-get again.

apt-get update && apt-get install -y curl

And now it's the perfect time to add this to the Dockerfile before we forget to add it, so

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y curl

Then let's use curl to fetch the program

curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl

but check https://yt-dl.org/downloads/latest/youtube-dl and then navigate to https://github.com/ytdl-org/youtube-dl/

curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl

w And then save the working URL to the Dockerfile

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
  curl

RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl

You may have noticed that sudo is not installed either, but since we are root we don't need it to write to /usr/local/bin/

Next, we will add permissions and run it:

$ chmod a+rx /usr/local/bin/youtube-dl
$ youtube-dl
  /usr/bin/env: 'python': No such file or directory

Okay - On the top of the youtube-dl download page we notice this message:

Remember youtube-dl requires Python version 2.6, 2.7, or 3.2+ to work except for Windows exe.

So we will add python, but before that let's what we did in the Dockerfile

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
  curl

RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl

RUN chmod a+rx /usr/local/bin/youtube-dl

Then docker build -t youtube-dl . again and notice how the lines we didn't touch were CACHED (and thus fast).

Now let's add python

apt-get install -y python

And, it fails

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package python is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  2to3 python2-minimal python2 dh-python python-is-python3

E: Package 'python' has no installation candidate

Okay, so... python2 ?

apt-get install -y python2

And let's run it again

$ youtube-dl
  /usr/bin/env: 'python': No such file or directory

...because python2 vs python3 is a mess and now only python2 command exists. Command youtube-dl tries to execute python and fails.

One way to fix this is to:

update-alternatives --install /usr/bin/python python /usr/bin/python2 1

And now we can see that

$ python
  Python 2.7.18 (default, Jul  1 2022, 10:30:50)
  [GCC 11.2.0] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> exit
  Use exit() or Ctrl-D (i.e. EOF) to exit
  >>> exit()

works, so youtube-dl should work.

It's a good moment to persist what we did

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
  curl

RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl

RUN chmod a+rx /usr/local/bin/youtube-dl

RUN apt-get install -y python2
RUN update-alternatives --install /usr/bin/python python /usr/bin/python2 1

$ youtube-dl
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: You must provide at least one URL.
Type youtube-dl --help to see a list of all options.

It works (we just need to give an URL), but we notice that it outputs a warning about LC_ALL. In a regular Ubuntu desktop/server install the localization settings are (usually) set, but in this image they are not set, as we can see by running env in our container. To fix this without installing additional locales, see this: https://stackoverflow.com/a/41648500

LC_ALL=C.UTF-8 youtube-dl

And it works! Let's persist it for our session and try downloading a video:

export LC_ALL=C.UTF-8
youtube-dl https://imgur.com/JY5tHqr

So now when we know exactly what we need to build a youtube-dl container on top of ubuntu:22.04

The videos are downloaded in to the current working directory, which by default is /. We can change this in the Dockerfile with WORKDIR directive.

And instead of using RUN export LC_ALL=C.UTF-8 we store the environment directly in the image with ENV

Lastly we override the default CMD from bash to youtube-dl

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
  curl

RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl

RUN chmod a+rx /usr/local/bin/youtube-dl

RUN apt-get update && apt-get install -y \
  python2

RUN update-alternatives --install /usr/bin/python python /usr/bin/python2 1

ENV LC_ALL=C.UTF-8

WORKDIR /app

CMD "/usr/local/bin/youtube-dl"

When we build this ...

$ docker build -t youtube-dl .
  ...

... and run it

$ docker run youtube-dl
  Usage: youtube-dl [OPTIONS] URL [URL...]

  youtube-dl: error: You must provide at least one URL.
  Type youtube-dl --help to see a list of all options.

So far so good, but now the natural way to use this image would be to give the URL as an argument:

$ docker run youtube-dl https://imgur.com/JY5tHqr

  docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "https://imgur.com/JY5tHqr": stat https://imgur.com/JY5tHqr: no such file or directory: unknown.
  ERRO[0000] error waiting for container: context canceled

As we now know, the argument we gave it is replacing the command or CMD. We need a way to have something before the command. Luckily we have a way to do this: we can use ENTRYPOINT to define the main executable and then docker will combine our run arguments for it.

FROM ubuntu:22.04

WORKDIR /app

RUN apt-get update && apt-get install -y curl python2
RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl
RUN chmod a+x /usr/local/bin/youtube-dl
RUN update-alternatives --install /usr/bin/python python /usr/bin/python2 1

ENV LC_ALL=C.UTF-8

# Replacing CMD with ENTRYPOINT - these extra [ ] are explained later
ENTRYPOINT ["/usr/local/bin/youtube-dl"]

And now it works like it should:

$ docker build -t youtube-dl .
$ docker run youtube-dl https://imgur.com/JY5tHqr

  [Imgur] JY5tHqr: Downloading webpage
  [download] Destination: Imgur-JY5tHqr.mp4
  [download] 100% of 190.20KiB in 00:0044MiB/s ETA 00:000

CMD vs ENTRYPOINT

With ENTRYPOINT docker run now executed the combined /usr/local/bin/youtube-dl https://imgur.com/JY5tHqr inside the container with that command!

ENTRYPOINT vs CMD is confusing - in a properly set up image such as our youtube-dl the command represents an argument list for the entrypoint. By default entrypoint is set as /bin/bash and this is passed if no entrypoint is set. This is why giving path to a script file as CMD works: you're giving the file as a parameter to /bin/bash.

In addition, there are two ways to set them: exec form and shell form. We've been using the exec form where the command itself is executed. In shell form the command that is executed is wrapped with /bin/sh -c - it's useful when you need to evaluate environment variables in the command like $MYSQL_PASSWORD or similar.

In the shell form the command is provided as a string without brackets. In the exec form the command and it's arguments are provided as a list (with brackets), see the table below:

Dockerfile	Resulting command
ENTRYPOINT /bin/ping -c 3 CMD localhost	/bin/sh -c '/bin/ping -c 3' /bin/sh -c localhost
ENTRYPOINT ["/bin/ping","-c","3"] CMD localhost	/bin/ping -c 3 /bin/sh -c localhost
ENTRYPOINT /bin/ping -c 3 CMD ["localhost"]	/bin/sh -c '/bin/ping -c 3' localhost
ENTRYPOINT ["/bin/ping","-c","3"] CMD ["localhost"]	/bin/ping -c 3 localhost

As the command at the end of docker run will be the CMD we want to use ENTRYPOINT to specify what to run, and CMD to specify which command (in our case url) to run.

Most of the time we can ignore ENTRYPOINT when building our images and only use CMD. For example, ubuntu image defaults the ENTRYPOINT to bash so we do not have to worry about it. And it gives us the convenience of allowing us to overwrite the CMD easily, for example, with bash to go inside the container.

Now we have two problems with the youtube-dl project:

Major: The downloaded files stay in the container
Minor: Our container build process creates many layers resulting in increased image size

We will fix the major issue first. The minor issue will get our attention in part 3.

By inspecting docker container ls -a we can see all our previous runs. When we filter this list with

$ docker container ls -a --last 3

  CONTAINER ID        IMAGE               COMMAND                   CREATED                  STATUS                          PORTS               NAMES
  be9fdbcafb23        youtube-dl          "/usr/local/bin/yout…"    Less than a second ago   Exited (0) About a minute ago                       determined_elion
  b61e4029f997        f2210c2591a1        "/bin/sh -c \"/usr/lo…"   Less than a second ago   Exited (2) About a minute ago                       vigorous_bardeen
  326bb4f5af1e        f2210c2591a1        "/bin/sh -c \"/usr/lo…"   About a minute ago       Exited (2) 3 minutes ago                            hardcore_carson

We see that the last container was be9fdbcafb23 or determined_elion for us humans.

$ docker diff determined_elion

  C /app
  A /app/Imgur-JY5tHqr.mp4

Let's try docker cp command to copy the file. We can use quotes if the filename has spaces.

docker cp "determined_elion://app/Imgur-JY5tHqr.mp4" .

And now we have our file locally. Sadly, this is not sufficient to fix our issue. In the next section, we will improve this..

..but before that, let's optimize the dockerfile:

FROM ubuntu:22.04
ENV LC_ALL=C.UTF-8

RUN set -eux ; \
  apt-get update ; \
  apt-get install -y curl python2 \
  update-alternatives --install /usr/bin/python python /usr/bin/python2 1

RUN set -eux ; \
  curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl ; \
  chmod a+rx /usr/local/bin/youtube-dl

WORKDIR /app
ENTRYPOINT ["/usr/local/bin/youtube-dl"]

Test with https://www.fromlatest.io —> no issues!

multi-stage

https://docs.docker.com/build/building/multi-stage/

FROM ubuntu:22.04 as builder

RUN apt-get update && apt-get install -y \
  curl

RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl \
  && chmod a+rx /usr/local/bin/youtube-dl

FROM ubuntu:22.04
ENV LC_ALL=C.UTF-8

RUN apt-get update && apt-get install -y \
  python2 ca-certificates \
  && update-alternatives --install /usr/bin/python python /usr/bin/python2 1

COPY --from=builder /usr/local/bin/youtube-dl /usr/local/bin

WORKDIR /app
ENTRYPOINT ["/usr/local/bin/youtube-dl"]

pick'n'mix

Combine ubuntu and alpine

FROM ubuntu:22.04 as builder

RUN apt-get update && apt-get install -y \
  curl

RUN curl -L https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl -o /usr/local/bin/youtube-dl \
  && chmod a+rx /usr/local/bin/youtube-dl


FROM alpine:latest
ENV LC_ALL=C.UTF-8

RUN apk add python3

COPY --from=builder /usr/local/bin/youtube-dl /usr/local/bin

WORKDIR /app
ENTRYPOINT ["/usr/local/bin/youtube-dl"]

curler entrypoint

With ENTRYPOINT we can make the curler more flexible.

curler/curler.sh

#!/bin/bash

echo "Searching.."
sleep 1
curl "http://$1"

And change the CMD to ENTRYPOINT with the format [ "/app/curler.sh" ]:

curler/Dockerfile

FROM ubuntu:22.04

RUN apt-get update
RUN apt-get install -y curl

WORKDIR /app
COPY curler.sh .

RUN chmod +x curler.sh

ENTRYPOINT ["/app/curler.sh"]

Now

$ docker build . -t curler-v2
$ docker run curler-v2 helsinki.fi

  Searching..
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100   232  100   232    0     0  13647      0 --:--:-- --:--:-- --:--:-- 13647
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>301 Moved Permanently</title>
  </head><body>
  <h1>Moved Permanently</h1>
  <p>The document has moved <a href="https://www.helsinki.fi/">here</a>.</p>
  </body></html>

You have reached the end of this section! Continue to the next section:

5. Interacting with the container via volumes and ports