Pushing to local docker registry gives connection reset error

english
#1

I'm trying to create a private Docker registry by following this guide. I've downloaded and tagged an Ubuntu image and created a registry container and now I intend to push it to my registry, but when I call sudo docker push localhost:6000/Ubuntu I get the following output:

The push refers to a repository [localhost:6000/ubuntu] Put http://localhost:6000/v1/repositories/ubuntu/: http: can't write HTTP request on broken connection

All I've been able to find is that the "can't write HTTP request" error is a Go error, but unfortunately I have not been able to find anything related to how I can resolve it in Docker. I have also tried to find logs for Docker commands so I can try to get more detail on the issue, but it seems like logs are only available for containers, and I'm not running this on a container.

How can I go about resolving this error?

UPDATE

I just ran the command again after changing nothing and got a different error:

Put http://localhost:6000/v1/repositories/ubuntu/: read tcp [::1]:33417->[::1]:6000: read: connection reset by peer

UPDATE2

And again, ran it after changing nothing and am now getting:

Put http://localhost:6000/v1/repositories/ubuntu/: EOF

UPDATE3

After messing around a bit, I've found that the image I tried to push to my registry does appear in my docker images list, but I don't seem to be able to pull it.

UPDATE4

After messing around a bit more, it seems like the port I assigned to my private registry is only listening on IPV6, rather than IPV4. According to this git thread, it is normal for docker to seem like it is only listening on IPV6, but after running telnet localhost, netstat -ntlp, and iptables -t nat -nxvL, I still see no indication that anything is listening on 127.0.0.1:6000, which is the port that my registry (which is just running as a container) should be assigned to. Instead, it seems to be listening on :::1:6000.

Also, because it was requested and because I should have listed it regardless:

user@devstack01:~$ sudo docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

UPDATE5

registry container's logs: 

time="2016-04-04T18:34:22Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.5.3 instance.id=25b8dd5a-f154-455b-84ad-3dd61ccf04e0 version=v2.3.1
time="2016-04-04T18:34:22Z" level=info msg="redis not configured" go.version=go1.5.3 instance.id=25b8dd5a-f154-455b-84ad-3dd61ccf04e0 version=v2.3.1
time="2016-04-04T18:34:22Z" level=info msg="Starting upload purge in 58m0s" go.version=go1.5.3 instance.id=25b8dd5a-f154-455b-84ad-3dd61ccf04e0 version=v2.3.1
time="2016-04-04T18:34:22Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.5.3 instance.id=25b8dd5a-f154-455b-84ad-3dd61ccf04e0 version=v2.3.1
time="2016-04-04T18:34:22Z" level=info msg="listening on [::]:5000" go.version=go1.5.3 instance.id=25b8dd5a-f154-455b-84ad-3dd61ccf04e0 version=v2.3.1
time="2016-04-04T19:32:22Z" level=info msg="PurgeUploads starting: olderThan=2016-03-28 19:32:22.890256531 +0000 UTC, actuallyDelete=true"
time="2016-04-04T19:32:22Z" level=info msg="Purge uploads finished.  Num deleted=0, num errors=1"
time="2016-04-04T19:32:22Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.5.3 instance.id=25b8dd5a-f154-455b-84ad-3dd61ccf04e0 version=v2.3.1
time="2016-04-05T13:54:59Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.5.3 instance.id=90882341-a150-4384-9434-b091aacdd68b version=v2.3.1
time="2016-04-05T13:54:59Z" level=info msg="redis not configured" go.version=go1.5.3 instance.id=90882341-a150-4384-9434-b091aacdd68b version=v2.3.1
time="2016-04-05T13:54:59Z" level=info msg="Starting upload purge in 59m0s" go.version=go1.5.3 instance.id=90882341-a150-4384-9434-b091aacdd68b version=v2.3.1
time="2016-04-05T13:54:59Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.5.3 instance.id=90882341-a150-4384-9434-b091aacdd68b version=v2.3.1
time="2016-04-05T13:54:59Z" level=info msg="listening on [::]:5000" go.version=go1.5.3 instance.id=90882341-a150-4384-9434-b091aacdd68b version=v2.3.1
time="2016-04-05T14:53:59Z" level=info msg="PurgeUploads starting: olderThan=2016-03-29 14:53:59.133142744 +0000 UTC, actuallyDelete=true"
time="2016-04-05T14:53:59Z" level=info msg="Purge uploads finished.  Num deleted=0, num errors=1"
time="2016-04-05T14:53:59Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.5.3 instance.id=90882341-a150-4384-9434-b091aacdd68b version=v2.3.1

Docker daemon's debug logs when I try to push an image to my registry:

DEBU[0037] Calling POST /v1.22/images/localhost:6000/hello-world/push
DEBU[0037] POST /v1.22/images/localhost:6000/hello-world/push?tag=
DEBU[0037] Trying to push localhost:6000/hello-world to https://localhost:6000 v2
DEBU[0037] Error getting v2 registry: Get https://localhost:6000/v2/: read tcp [::1]:33560->[::1]:6000: read: connection reset by peer
DEBU[0037] Trying to push localhost:6000/hello-world to http://localhost:6000 v2
DEBU[0037] Error getting v2 registry: Get http://localhost:6000/v2/: read tcp [::1]:33562->[::1]:6000: read: connection reset by peer
DEBU[0037] Trying to push localhost:6000/hello-world to https://localhost:6000 v1
DEBU[0037] attempting v2 ping for registry endpoint https://localhost:6000/v2/
DEBU[0037] attempting v1 ping for registry endpoint https://localhost:6000/v1/
DEBU[0037] Trying to push localhost:6000/hello-world to http://localhost:6000 v1
DEBU[0037] Image list: [0xc820aa3840 0xc820704500]
DEBU[0037] Tags by image: map[sha256:690ed74de00f99a7d00a98a5ad855ac4febd66412be132438f9b8dbd300a937d:[latest]]
DEBU[0037] Pushing ID: b652ec3a27e758f30de4742156b5d096bb19c82f2dc836e96e430323ba166ffe with Tag:
DEBU[0037] Pushing ID: 690ed74de00f99a7d00a98a5ad855ac4febd66412be132438f9b8dbd300a937d with Tag: latest
DEBU[0037] [registry] PUT http://localhost:6000/v1/repositories/hello-world/
DEBU[0037] Image list pushed to index:
[{"id":"b652ec3a27e758f30de4742156b5d096bb19c82f2dc836e96e430323ba166ffe"},{"id":"690ed74de00f99a7d00a98a5ad855ac4febd66412be132438f9b8dbd300a937d","Tag":"latest"}]
DEBU[0037] Not continuing with error: Put http://localhost:6000/v1/repositories/hello-world/: EOF

The registry logs end with saying that an error occurred, but from the rest of the log I can't actually discern what that error was. I even split the log into stdout and stderr and nothing was in stderr.

Also, I've confirmed that the registry is listening on ipv4. Running nc -l -4 6000 returns Address already in use. At least, I think I've confirmed it. The ipv4 addresses still won't show up with netstat.

UPDATE6

Explicitly specified an ipv4 port with docker run -d -p 127.0.0.1:8000:8000, this port shows up as listening in netstat. Still getting the same errors. Docker logs are a bit different though:

time="2016-04-07T15:19:31Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.5.3 instance.id=6391e3dc-6644-4a45-8e8b-a08c8ff798a6 version=v2.3.1
time="2016-04-07T15:19:31Z" level=info msg="redis not configured" go.version=go1.5.3 instance.id=6391e3dc-6644-4a45-8e8b-a08c8ff798a6 version=v2.3.1
time="2016-04-07T15:19:31Z" level=info msg="Starting upload purge in 1m0s" go.version=go1.5.3 instance.id=6391e3dc-6644-4a45-8e8b-a08c8ff798a6 version=v2.3.1
time="2016-04-07T15:19:31Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.5.3 instance.id=6391e3dc-6644-4a45-8e8b-a08c8ff798a6 version=v2.3.1
time="2016-04-07T15:19:31Z" level=info msg="listening on [::]:5000" go.version=go1.5.3 instance.id=6391e3dc-6644-4a45-8e8b-a08c8ff798a6 version=v2.3.1
time="2016-04-07T15:20:31Z" level=info msg="PurgeUploads starting: olderThan=2016-03-31 15:20:31.061864096 +0000 UTC, actuallyDelete=true"
time="2016-04-07T15:20:31Z" level=info msg="Purge uploads finished.  Num deleted=0, num errors=1"
time="2016-04-07T15:20:31Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.5.3 instance.id=6391e3dc-6644-4a45-8e8b-a08c8ff798a6 version=v2.3.1

The "redis not configured" message is the only thing that seems like it could be a problem. Going to look into it. Would really, really appreciate any additional insight.

#2

I'm writing this because I spent hours on this before finally cracking it.

My situation: I'm running a docker version 17 on Fedora 25 and was trying to run a registry using a custom config file, and getting the same peer reset message as above. It seemed that the this was happening due to the insecure HTTP connection being reset, and thus I found people telling me to add the DOCKER_OPTS="--insecure-registry localhost:6000 127.0.0.1:6000" option in the /etc/sysconfig/docker and /etc/default/docker file, and restart docker service. This did not fix the issue, however (Here is the official documentation on that).

What finally did the trick was running a registry using a docker compose image. I followed the steps given at the digitalocean guide, which basically guides you through using docker-compose to set up a registry and then using nginx to set up SSL , which would do the trick if done on your local servers.

However, it turns out that all I needed to do was Step 2, that is, set up the docker-compose.yml as given and start a registry, and voila! the push now works.

#3

Solved the issue:

Had to run with docker run -d -p 127.0.0.1:6000:5000. Not using localhost ensured that ipv4 was used (I'm pretty sure it was used otherwise but at least now it shows up in netstat) and the containers listen on their port 5000, so it did not work otherwise.

#4

Hey, Is this solved? Whats the latest error you are getting?
Also can you post the versions of docker, registry etc. Are you running registry as a docker container (or how is your registry running?)

#5

@Maniankara I’ve updated the question

#6

Check the logs for your registry. You have probably missed something important while you were distracted by the IPv6 non-issue.

#7

@MichaelHampton will do. I’ll post the results in a moment.