Recovering data from a broken MacOS Docker installation This guide explains how to recover data from, even if the underlying installation is corrupt. The guide is written for MacOS specifically, although many of the steps are applicable to.nix. How is data stored in Docker for Mac?
Docker for Mac starts a daemon which runs a VM. LinuxKit is an extremely light-weight Linux distro. The HyperKit VM runs, which is actually responsible for creating Docker containers. The HyperKit VM’s data is stored on disk in format. Depending on how you installed Docker for Mac, this disk image is located at $HOME/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2.
The HyperKit VM image contains all of your container data, including images and volumes. Recovering data from Docker for Mac is a matter of extracting data from the qcow2 image. Exploring the HyperKit VM filesystem Docker for Mac creates a tty in the VM directory. Usually this file is located at $HOME/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty. This file is the tty for the HyperKit VM.
To get a shell in the HyperKit VM, run. Screen $HOME/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty Once inside the HyperKit VM, you can explore the data stored.
The volumes are stored in /var/lib/docker/volumes, and the volume contents are present as ordinary files and directories. The location and format of the images depends on the storage driver - see for more information. Data recovery if the Docker daemon still starts If the qcow2 image is uncorrupted and the HyperKit daemon still starts, recovering data is somewhat straightforward. It will still be possible to issue Docker commands and back up the data using Docker’s own tooling. The basic idea is:. Identify a volume to back up.
Start a new container that mounts the volume. Copy the volume to the host filesystem using docker cp 1 Identify the volume you want to backup We need to find the volume name of the volume we want to back up. Run this command to list all of the available volumes.
If auto-lock was enabled on the old Swarm, the unlock key will be required to perform the restore. Shut down Docker on the target host machine where the swarm will be restored. Remove the contents of the /var/lib/docker/swarm directory on the new Swarm if it exists. Restore the /var/lib/docker/swarm directory with the contents of the backup. May 1, 2018 - How to recover and restore data (images and volumes) from broken or corrupt MacOS Docker installations.
Docker run ` # docker run starts a new container with the specified image and command` -d ` # Run the container in the background` -rm ` # Remove the container when it exits` -mount source = $yourvolumename,target =/volume ` # Mount your chosen volume at the` ` # /volume path on the containers filesystem` busybox ` # Use the busybox image, an extremely minimal Linux` tail -f /dev/null ` # A command that runs indefinitely without exiting` For convenience, here’s the command without comments or newlines. Docker kill $yourcontainername Data recovery if the Docker daemon will not start Data recovery is much more difficult if the Docker daemon isn’t starting up correctly.
In some cases, the HyperKit VM will start, but it will not be able to load the qcow2 image containing the container data. The basic idea here is:. Install dependencies (there are several). Convert the qcow2 image to a FUSE filesystem image. Mount the FUSE filesystem. Copy the data out of the FUSE filesystem onto the host We’ll be using to convert the Docker.qcow2 to a FUSE filesystem and to mount the FUSE filesystem on MacOS.
We’ll also need which is a dependency of the other tools. 0 Installing dependencies Homebrew is required to complete the following steps. Osxfuse First, we’ll need osxfuse. You can download the latest installer from, or run the following command. Qcow2-fuse -o allowother -o rdonly Docker.qcow2 mnt The -o allowother -o rdonly flags specify that any user account should be able to access the FUSE image, and that the FUSE filesystem should be read-only. After running the command, a mnt directory containing the FUSE image (a file called Docker) should be present in the current directory.
2 Mount the FUSE filesystem image We’re now ready mount the FUSE image as a volume. The first step is to attach the FUSE image as a device. The following command (which assumes there’s a FUSE image called Docker in the mnt directory) will attach the FUSE image as a device. # List the files at the root of the FUSE filesystem sudo ls -al volume # Copy all of the volumes on the FUSE filesystem into the current directory sudo cp -R volume/lib/docker/volumes./dockervolumes Depending on your Docker/HyperKit version, the contents of your FUSE filesystem may differ. You should explore the filesystem to figure out exactly where your volume data resides.
The ext4fuse is pretty unreliable, and during the course of interacting with the FUSE filesystem, you may see errors like this.
A short background Why Mac OS? I'm a web developer & CI/CD process administrator in my everyday work in Geta. I have Mac OS as my host OS for many years now and I really enjoy the benefits what I get using devices in so called 'Mac ecosystem'. Won't go in more details about it this time but that is why I'm writing about Docker in Mac OS.
Why SQL Server? Our company's core business is to produce web applications & services built on.NET platform and obviously SQL Server is most common database engine choice in this context because it fits well with.NET applications. This means that pretty much every developer in our company has to run local SQL Server instance for local development purposes on their computers. And obviously there haven't been easy solutions to run SQL Server natively on Mac OS, which requires me to run it on Windows hosted on Parallels Desktop VM, which in turn makes me sacrifice some host OS resources. As I mentioned, I take it in favor of having some other benefits of working on Mac OS.:) Why Docker?
Docker / containers / microservices has been a buzz words to me (from sessions in conferences, different blog posts, buddy in the company who have had hands-on experience with Docker - ) for a several years, but I never managed to discover these things more. And then few weeks ago somewhere on the Internet:) I ran into this MSDN post about which reminded me that would be nice to test these things. Finally the last thing which nailed it all was a message from another workmate in Geta about the new lightweight which can be run on Mac OS.
So these are basically the reasons why I felt it was right time for me to try all these pieces stiching together and to see if I could get more freedom of Windows & Parallels Desktop. Turned out it was easy to set up everything - it is easy to use - it performs much faster than on Parallels VM - I'm happy - I want to tell about this to others!:) Problem As I mentioned before I rely on Parallels Desktop VM to run SQL Server local instance.
That means I have to start my Windows VM every time I need to access some local database which takes some time obviously as well as running software on VM is not as resource-efficient as running similar software on native OS. Besides that SQL Server Management Studio (which I used before to connect to my local SQL databases) alone is a rather heavy piece of software, which is often more than you need to do a basic tasks with SQL Server e.g.
Run basic table queries, create new users, grant permissions etc. Install Docker engine (Docker for Mac in my case) That's a rather straightforward installation process, just follow the. When you have completed installation you must have Docker whale icon visible in Mac OS menu bar. You can verify that Docker CLI is also successfully installed by running docker version command.
Pull & Run SQL Server Docker container image Again, this is also very simple (even without any previous Docker knowledge) if you follow. Pull the SQL Server image from Docker Hub. I was totally fine with the latest SQL version so I went with that. Sudo docker pull microsoft/mssql-server-linux:2017-latest Btw, really sweet progress bar for Docker CLI.:). Run container image on your Docker engine.
I did some parameter value adjustments in the docker run comand (e.g. Container name, SQL password) and also added shared folder mapping option my from host OS. It's good to have shared folder mapped when you would like to transfer some files into docker container. In my case I realized that I need it for copying db backup file and later restoring a db from it. Docker run -name sarbis-mac-sql -volume /Users/sarbis/Desktop/DockerShared:/HostShared -env 'ACCEPTEULA=Y' -env 'MSSQLSAPASSWORD=password' -publish 1401:1433 -detach microsoft/mssql-server-linux:2017-latest 3.
Connect to a Docker SQL Server from your preferred SQL tool SQL Operations Studio As I wrote before one of the main reasons of my excitement about this all was release of which is a free SQL Server tool that runs on Mac OS. I find it very good for doing basic everyday tasks (from web developer perspective) because of simplicity and having GUI. Before SQL Operations Studio I was sometimes using also which also runs on Mac OS but in comparison lacks nicer GUI for switching databases and SQL servers, restoring databases. Nevertheless it can be handy if you don't have to work with SQL Server often and not with many different servers and databases. I feel like SQL Operations Studio is a nice alternative for me between comprehensive SQL Server Management Studio and VS Code in terms of simplicity and functionality features. So now you can connect to the local SQL Server instance running on Docker by specifying a server name ( localhost) and a port number ( 1401) what you specified in the docker run command previously. You are finally connected to SQL Server instance running on Mac Docker container and using SQL tool running natively on Mac OS too.:) You can now either create new db from script or restore it by first copying backup file into mapped Docker folder and then selecting the db backup file in Restore dialog of SQL Operations Studio.
SQL Server Command Line Tools Just wanted to note that there is also which is another relatively new SQL tool from Microsoft which is now available for Mac OS (in preview version though). It is yet another evidence that.NET stack developers are slowly 'set free' of having to use Windows.
Sqlcmd can be installed via the package manager which (as many Mac users probably already know) is really handy tool for easy beer brewing installation of various applications from Terminal. At first link the sqlcmd git repository with brew tap command. Brew tap microsoft/mssql-preview Then simply install the software with brew install. Brew install –no-sandbox msodbcsql mssql-tools Now you can connect to your SQL Server with sqlcmd command line utility from Mac OS Terminal. Sqlcmd -S localhost,1401 -U sa -P password And even run a SQL query!:) Summary To wrap this up I could say that with rather small effort it is possible to set up OS-independent SQL Server instance, which can be easily stopped, transferred to other environment, ran together with other versions / instances of SQL server etc. Docker provides a lot of isolation and flexibility and makes life much easier for many developers (not only for those who are on Mac OS) epsecially when there are sufficient SQL tools which are free and can run on their OS.