When you make changes to a Docker container’s filesystem, it’s important to inspect the changes in order to determine if they have any impact on the container’s behavior. This article will walk you through how to do this.
Docker containers inherit their image’s filesystem upon initial creation. The container’s filesystem can diverge from the image’s content after creation if changes are made to the container’s writable layer.
The ability to inspect changes in the filesystem can help you debug issues and investigate missing data. Files written to the container’s filesystem will be lost when that container’s restarted or destroyed. In some cases you might be unintentionally performing writes that should be directed to a persistent volume instead.
In this article, we’ll share some techniques for identifying modifications made to container filesystems. You can use this information to track container activity and investigate discrepancies between different environments.
Using Docker Diff
The Docker CLI has a built-in command for this purpose. Running docker diff will enumerate all the changes made to files and directories within a particular container. It accepts the ID or name of the container you want to inspect.
Start a simple container for demonstration purposes:
This will start a new container running the Alpine image. Your terminal window will be attached to the container. Keep this window open.
Now run docker diff in a second terminal session:
No output is produced as the fresh container still exactly matches the Alpine image’s filesystem.
Switch back to your first terminal that’s still attached to the container. Run some commands that add, delete, or change files:
Return to your first shell window. Repeat the docker diff command to see the changes that have been made:
Now the two new files are visible in the command’s output. Running touch also added entries to the container’s shell history. The modification to /root/.ash_history is reported too, as well as the consequent change to the /root directory.
Interpreting Docker Diff Output
Each line of output produced by docker diff is prefixed by either A, C or D depending on the type of change that occurred:
A – The file or directory has been added to the container’s filesystem. It doesn’t exist in the original image. C – The content of the file or directory is different to the version in the image. D – A path that’s present in the image has been deleted from the container.
This information enables you to quickly look for discrepancies that could explain missing data or unexpected behavior.
Any filesystem changes that don’t show up in the output will have affected paths that are mapped to a Docker volume. These files are stored outside the container, independently of its filesystem, so they’re not surfaced by docker diff .
Viewing Content in the Original Image
It can be helpful to compare the file version that’s included in the original image. You can do this by creating a new container and then copying out the file. Use docker create instead of docker run so the container is created but not started. This will ensure that modifications caused by the image’s entrypoint script don’t affect the result.
An alternative way is to run the container with a custom entrypoint that emits the file’s content to the standard output stream. Include the –rm flag so the container’s automatically removed after the command completes:
Comparing Differences Between Two Containers
Sometimes you might need to compare the differences between two different containers running the same image. First use docker diff to get the individual change lists:
Now you can use standard Unix tools such as the diff command to find the differences in the two change lists:
The diff shows that /a-new-file has been removed in the second container while /demo has been added.
Another way to compare containers is to use docker export and the tar command to access a comprehensive file list:
Repeat this command for both your containers. You can then use diff to compare the two lists. The enhanced output of tar can help you track down when files changed as modification times and permissions are displayed.
Summary
The filesystems of Docker containers should closely resemble the image they were created from. Discrepancies can occur when software within the container writes content directly to the filesystem. Users can be another source of differences, when mechanisms such as docker exec are used to run commands within a running container.
The docker diff utility provides a convenient way to view these changes. Although you can’t see why a file was added or removed, the ability to retrieve a list of modifications can be invaluable in understanding why particular behavior is being observed.
In this article we’ve focused on examining the changes made to running containers. You can use different tools such as docker history and Dive to inspect the changes made to layers within a Docker image. The two approaches can be used in tandem, letting you work out the exact point a file was added or modified within a container’s layer stack.
