111: Connection refused nginx proxy for Docker containers

  • CentOS 7

I have a simple Nginx proxy Docker container listening on port 80. Here is the Dockerfile:

FROM centos:7
MAINTAINER Brian Ogden

# Not currently being used but may come in handy
ARG ENVIRONMENT

RUN yum -y update && \
    yum clean all && \
    yum -y install http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm \
    yum -y makecache && \
    yum -y install nginx-1.12.0 wget

# Cleanup some default NGINX configuration files we don’t need
RUN rm -f /etc/nginx/conf.d/default.conf


COPY /conf/proxy.conf /etc/nginx/conf.d/proxy.conf
COPY /conf/nginx.conf /etc/nginx/nginx.conf


CMD ["nginx"]

And for this Nginx Proxy here is my nginx.conf:

daemon off;
user  nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
    use epoll;
    accept_mutex off;
}


http {
    include       /etc/nginx/mime.types;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    client_max_body_size 300m;
    client_body_buffer_size 300k;
    large_client_header_buffers 8 64k;

    gzip  on;
    gzip_http_version 1.0;
    gzip_comp_level 6;
    gzip_min_length 0;
    gzip_buffers 16 8k;
    gzip_proxied any;
    gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
    gzip_disable "MSIE [1-6]\.";
    gzip_vary on;

    include /etc/nginx/conf.d/*.conf;
}

And here is my proxy configuration:

upstream accountstaging {
    server 127.0.0.1:5023;
}

server {

    listen 80;
    server_name account.staging.mysite.com;

    location / {
        proxy_pass         http://accountstaging;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
    }
}

My proxy configuration is listening on port 80 and trying to request requests from account.staging.mysite.com to a Docker container running on the same Docker host as the Ngnix proxy listening on port 5023.

Here is my docker-compose.yml for my Nginx proxy:

version: '3'
services:
  reverseproxy:
    build: 
      context: ./
      dockerfile: docker/Dockerfile
    image: tsl.devops.reverseproxy.image
    container_name: tsl.devops.reverseproxy.container
    ports:
      - "80:80"

Here is the docker-compose.yml for this Docker container listening on port 5023: version: '3'

services:
  apistaging:
    build: 
      context: ./
      dockerfile: docker/staging/Dockerfile
    image: tsl.api.example.image
    container_name: tsl.api.example.container
    ports:
      - "127.0.0.1:5023:80"

The Dockerfile does not really matter much to my question but here it is anyways:

FROM tsl.devops.dotnetcore.base.image:2
MAINTAINER Brian Ogden

WORKDIR /app
COPY ./src/Tsl.Example/bin/Release/netcoreapp2.0/publish .

ENTRYPOINT ["dotnet", "Tsl.Example.dll"]

I followed this example to setup my proxy.

I have previously asked a related question on Stackexchange forums here and here. This question I have refined and simplified the scenario to a simply proxy forwarding a request to one Docker container listening on port 5023.

Since my base image is CentOS I have followed this here to make sure SELinux is allowing forward to port 5023

Thanks to this question and answer here, I was able realize that I had two issues going on:

  1. the containers have different default Docker networks because I am using two different docker-compose.yml files, I had envisioned my Ngnix proxy working independently from any of my API containers entirely, including the docker-compose, more on that issue below
  2. the second issue is simply when I tried to proxy to 127.0.0.1:5023 that is localhost inside the Ngnix container, not the network outside of the Nginx proxy container

So the different default networks being created by docker-compose for my Nginx proxy docker container and my api docker container are because I amusing two different docker-compose.yml files. This is because I have Jenkins builds for many API microservices so the have independant docker-compose files and I needed a Nginx proxy to forward requests on port 80 to each microservice.

To test this out, created a docker-compose.yml for both containers, the API and the Nginx proxy:

version: '3'

services:
  reverseproxy:
    build: 
      context: ./
      dockerfile: docker/nginxproxy/docker/Dockerfile
    image: tsl.devops.reverseproxy.image
    container_name: tsl.devops.reverseproxy.container
    ports:
      - "80:80"
  apistaging:
    build: 
      context: ./
      dockerfile: docker/staging/Dockerfile
    image: tsl.api.example.image
    container_name: tsl.api.example.container
    ports:
      - "5023:5023"
    environment: 
      ASPNETCORE_URLS: http://+:5023

Yes there was still an issue, the proxy pass to http//:127.0.0.1:5023, that forward remains in the Nginx Docker container and never finds the API running on the Docker host, I simply needed to use the docker-compose.yml service name to get to it:

upstream accountstaging {
    server apistaging:5023;
}

server {

    listen 80;
    server_name account.staging.mysite.com;

    location / {
        proxy_pass         http://accountstaging;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
    }
}

I had this problem but my issue was that I had two folders that both had docker-compose.yml files which both had a service named 'web'. I changed the name of one of the services.

The full picture to help someone struggling with docker networks as much I was (am?):

Docker-compose projects often use nginx as reverse-proxy to route http traffic to the other docker services. nginx was a service in my projectfolder/docker-compose.yml which was connected to two docker networks.

One was the default network created when I used docker-compose up on projectfolder/docker-compose.yml (It is named projectfolder_default and services connect to it by default UNLESS you have a networks property for your service with another network, then make sure you add - default to the list). When I ran docker network ls I saw projectfolder_default in the list and when I ran docker network inspect projectfolder_default I saw the nginx container, so everything was good.

The other was a network called my_custom_network that I set up myself. I had a startup script that created it if it did not exist using https://stackoverflow.com/a/53052379/13815107 I needed it in order to talk to the web service in otherproject/docker-compose.yml. I had correctly added my_custom_network to:

  • nginx service's networks list of projectfolder/docker-compose.yml
  • bottom of projectfolder/docker-compose.yml
  • web service's networks in otherproject/docker-compose.yml
  • bottom of otherproject/docker-compose.yml

The network showed up and had the right containers using docker network ls and docker network inspect my_custom_network

However, I assumed that calling http://web would map to the docker service web.projectfolder_default. I was mistaken. I opened shell on the nginx container (docker exec -it nginx sh). When I used ping web (may need to apt-get update, apt-get install iputils-ping) it succeeded, but it printed a url with my_custom_network which is how I figured out the mistake.

projectfolder/docker-compose.yml

  services:
    # http://web did NOT map to this service!! Use http://web.main_default or change the names
    web:
      ... 
    nginx:
        ...
      links:
        - web
      networks:
        - default
        - my_custom_network
    ...
  networks:
    - my_custom_network
      external: true

otherproject/docker-compose.yml

  services:
    # http://web connected to this service instead. You could use http://web.my_custom_network to call it out instead
    web:
      ... 
      networks:
        - default
        - my_custom_network
    ...
  networks:
    - my_custom_network
      external: true

projectfolder/.../nginx/server.conf.template (next to Dockerfile) ...

server {
...

location /auth {
    internal;
    # This routed to wrong 'web'
    proxy_pass              http://web:9001;
    proxy_pass_request_body off;
    proxy_set_header        Content-Length "";
}

location / {
    alias /data/dist/;
}

location /robots.txt {
    alias /robots.txt;
}

# Project Folder backend
location ~ ^/(api|login|logout)/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_connect_timeout 300s;
    proxy_read_timeout 300s;
    # This routed to wrong 'web'
    proxy_pass http://web:9001;
}

# Other project UI
location /other-project {
    alias /data/other-project-client/dist/;
}

# Other project Django server
location ~ ^/other-project/(rest)/ {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_connect_timeout 300s;
    proxy_read_timeout 300s;
    # This should work
    proxy_pass http://web.my_custom_network:8000;
}

}