I had to debug in our container registry because something unclear happend. At office
we are using the Docker registry v2.6.0
to host
hundred of repositories. An user has complained about a missing image. The
image has been pushed, but it is no longer here... Docker pull command
failed. Our docker registry is running in docker container and
all the logs are in docker logs, no persistence :-(.
Even worse there is absolutely no supervision of the service. I started to
extract the relevant logs from the docker daemon.
docker logs $(docker ps|grep "registry:"| tr -s " " | cut -f 1 -d " ") |grep "my-container" | grep "1.0.1" > /tmp/logs.txt
When looking at the logs, I found interesting patterns filesystem.PutContent
and filesystem.Delete. This patterns come from the docker storage driver.
Next step is to catch the deletion, this could be achieved by grepping
the HTTP DELETE message.
# egrep DELETE my-service-1.0.1.txt
time="2019-03-05T10:19:55.904910006Z" level=debug msg="filesystem.GetContent(\"/docker/registry/v2/repositories/corp/my-service/_manifests/tags/1.0.1/current/link\")" go.version=go1.7.6 http.request.host="corp-ssa-docker-registry:5000" http.request.id=3313e9de-d9cf-4696-8b3b-7c09ca22904e http.request.method=DELETE http.request.remoteaddr="10.0.0.2:56942" http.request.uri="/v2/corp/my-service/manifests/sha256:640f61c7fa1bceda97c3105d20417695cd9ff29b32d4dada932bab950748c092" http.request.useragent="python-requests/2.4.3 CPython/3.4.2 Linux/3.10.0-514.el7.x86_64" instance.id=8a995190-de4b-48a4-bbc7-0757bb249e2f service=registry trace.duration=9.581796ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=80e22948-8fd7-4f1e-8330-7da907ae5f30 trace.line=82 vars.name="corp/my-service" vars.reference="sha256:640f61c7fa1bceda97c3105d20417695cd9ff29b32d4dada932bab950748c092" version=v2.6.2
time="2019-03-05T10:19:55.905819262Z" level=debug msg="filesystem.Delete(\"/docker/registry/v2/repositories/corp/my-service/_manifests/tags/1.0.1\")" go.version=go1.7.6 http.request.host="corp-ssa-docker-registry:5000" http.request.id=3313e9de-d9cf-4696-8b3b-7c09ca22904e http.request.method=DELETE http.request.remoteaddr="10.0.0.2:56942" http.request.uri="/v2/corp/my-service/manifests/sha256:640f61c7fa1bceda97c3105d20417695cd9ff29b32d4dada932bab950748c092" http.request.useragent="python-requests/2.4.3 CPython/3.4.2 Linux/3.10.0-514.el7.x86_64" instance.id=8a995190-de4b-48a4-bbc7-0757bb249e2f service=registry trace.duration=760.721µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete" trace.id=d61e5938-5da2-4e85-bfd0-e26cc2c3fbab trace.line=177 vars.name="corp/my-service" vars.reference="sha256:640f61c7fa1bceda97c3105d20417695cd9ff29b32d4dada932bab950748c092" version=v2.6.2
Second line of the output tell the truth! The deletion has been done by 10.0.0.2 at 10:19 and it is probably a Python script because the user agent is python-requests. So do not be afraid of going inside docker registry logs, forensic is possible. But I would expect to find an easier solution with docker frontend. Unfortunately this project has not been updated since July 6th, 2017. It is written in AngularJS, it may be a problem to find new contributor, because Angular has been rewritten with the version 2.x. I hope docker-registry-frontend will find a honorable successor or at least new maintener.
