Having hard time to use port 443 -> 8052 for the awx_web...

Hello guys…

I’m trying to setup the awxserver with ssl, having some troubles.

  1. My first try was to use apache on my server and using proxypass and proxyreverse, to point to the configured host_port for the app (picked 8080). Almost everything worked, but the websocket, usign also proxypass in nginx, caused a Bad Gateway response in the client, which translates into, unable to see messages via the websocket, which is sad…

  2. My second try, was to use ssl in nginx directly, so, I’ve setup the nginx.conf to use ssl, installed my certificates, and, bob is your uncle. SSL was active, websocket working, but when I triggered a job, unfortunately, I got an api error, because awxrest.py uses plain http and not https.

ERROR! Attempted to execute “/usr/lib/python2.7/site-packages/awx/plugins/inventory/awxrest.py” as inventory script: Inventory script (/usr/lib/python2.7/site-packages/awx/plugins/inventory/awxrest.py) had an execution error: 400 Client Error: Bad Request for url: http://awxweb:8052/api/v1/inventories/6/script/?hostvars=1
4

<html 4 5 6 <body bgcolor="white" 5 400 The plain HTTP request was sent to HTTPS port</head 6 7 8 The plain HTTP request was sent to HTTPS port</center 7

So guys,

What do you recommend in this case?

Thanks,

-Cesar

You can’t just turn an https port into an http port, it’s not going to work.

My recommendation, until I get around to adding it myself, is to probably add haproxy either as a container running alongside the web container or on the local system.

I do something similar in our cluster development environment… you can see the haproxy dockerfile here:

https://github.com/ansible/awx/blob/devel/tools/docker-compose/Dockerfile-haproxy

and the haproxy config here:

https://github.com/ansible/awx/blob/devel/tools/docker-compose/haproxy.cfg

which will appropriately handle the websockets and sending the proper headers along. You’ll also need to handle ssl termination which I don’t do in this one but is really easy to do: https://www.haproxy.com/doc/aloha/7.0/deployment_guides/tls_layouts.html#ssl-tls-offloading

Thanks Matthew,

The haproxy approach worked like a charm…

-Cesar

Hi Cesar,

Would you mind sharing your haproxy.cfg file?

Thanks,

Marc

Hi Marc,

I don’t actually have a config file. I just used the tutum/haproxy container and I launch it with the other containers.

I created a role called haproxy. This is the main tasks playbook for it:

  • name: run haproxy container
    docker_container:
    user: root
    name: awx_haproxy
    state: started
    image: tutum/haproxy
    ports:
  • “80:80”
  • “443:443”
    links:
  • “awx_web:awx_web”

So,

In the local_docker playbook, I’ve added a task to read the ssl cert from my cert file and inject it into the environment:

  • name: read cert
    set_fact:
    cert: “{{ lookup(‘file’, ‘{{ role_path }}/files/server.pem’) }}”

Then in Activate AWX Web Container I set the SSL_CERT environment variable with “{{ certs | default(‘’) }}”

env:
SSL_CERT: “{{ certs | default(‘’) }}”

You probably want to do ssl termination in haproxy.

Yeah, I did that at first, but then, the FORCE_SSL environment was ignored. I force it in the awx_web container and it just works.

-Cesar

Hi Cesar,

Was not really able to reconstruct your steps, in the end I simply manually started the tutum/haproxy and it worked!

  1. First create certificate
  2. Get the tutum/haproxy image
  3. Start the tutum/haproxy image with:

docker run -d --link awx_web:awx_web -p 443:443 -e DEFAULT_SSL_CERT=“-----BEGIN PRIVATE KEY-----

-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----” tutum/haproxy

Works like a charm :wink:

Marc

Yeah,

If you only need to deal with port 443, that simply works. In my case I wanted to force https, so, connections to port 80 get re-directed to 443. That’s why I had to do it that way.

-Cesar

I’m having a hard time using that config. I get that the ports are in use, and when i try to change things around a bit the task server wont start, claiming the web container is not running. did you change the ports in main.yml for awx_web or the host port in the inventory file?

Hi Matt, do you have an eta for when haproxy may get added to the default awx release ? Or if poss can someone document the full set of steps to get this working ??

Andy

My awx_web docker is running on port 80 only… so I don’t have any conflict

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5aa58db35d7d ansible/awx_task:latest "/tini – /bin/sh -c " 13 days ago Up 18 hours 8052/tcp awx_task
7938180720b2 ansible/awx_web:latest "/tini – /bin/sh -c " 13 days ago Up 18 hours 0.0.0.0:80->8052/tcp awx_web
95debb0bdf4a tutum/haproxy “python /haproxy/main” 4 weeks ago Up 18 hours 80/tcp, 1936/tcp, 0.0.0.0:443->443/tcp awx_proxy
2e867a84e10d memcached:alpine “docker-entrypoint.sh” 6 weeks ago Up 18 hours 11211/tcp memcached
fa1eb3a8f678 rabbitmq:3 “docker-entrypoint.sh” 6 weeks ago Up 18 hours 4369/tcp, 5671-5672/tcp, 25672/tcp rabbitmq

Have you checked your installer/inventory file?

Standalone Docker Install

postgres_data_dir=/tmp/pgdocker
host_port=80

As Cesar pointed out, this is a poor man solution as it does not automatically redirect http to https, just makes the site available under it.

Marc

is this HAproxy setup to provide SSL access over a routable IP or URL? Or to force SSL while accessing localhost? I ask because I’m having a hard time finding documentation on how to reverse proxy my AWX server to a routable IP (with websocket traffic). I want to test out this docker setup because it sounds promising, but I don’t see any mention of using this to access an IP or URL through HAproxy. Can this be accomplished using the tutum/haproxy setup?

This is how I accomplish SSL termination with my AWX setup: https://lreed.net/post/2018-01-10-awx-nginx/

I posted the above link already in this post, but thought it was relevant here too. Sorry for posting the link twice and I hope it’s not frowned upon to post a link to my personal site.

Hey Lucas…

For me, the easiest way of accomplishing this was using the haproxy container. It works great. Also, if you want to perform loadbalance, having several instances, we can easilly do it too.

-Cesar

Agreed that haproxy may be the best solution here, but my experience was mostly with nginx so that’s what I went with since I only needed a single awx_web/awx_task setup.
Apparently haproxy does indeed support redirecting http to https and I wasn’t aware of that.

Now that the kubernetes setup is supported, it would be nice to move it to a cloud provider load balancer, but I’m not sure those support redirecting from http to https (I could totally be wrong about that).

I really appreciate the write-up and the link, Lucas. I’m going to work through your steps to setup an Nginx container this afternoon.

Since I’m familiar with AWS, I setup a load balancer this morning which is terminating SSL and routing traffic to the AWX server on port 80. I’m not technically using the load balancer to force HTTPS, but I am using AWS security rules to only allow traffic on port 443 to the load balancer (so it wont redirect traffic on port 80 to 443, but it will block traffic on port 80 which is good enough for me for today). Then, I used another security rule to only allow traffic on port 80 from the load balancer to the AWX server. Pretty secure, but not perfect.

This load balancer setup will work for me if it needs to. However, I like the idea of handling the SSL termination and 443 redirection on the same host as this exposes no port 80 traffic outside of the servers internal routing. Given that this server will host our AWX server, I feel the highest security standards are needed. So, I’m going to revisit the Nginx reverse proxy using your guide.

I’ve seen HAproxy mentioned here and while I appreciate that it works well for some, I’m not familiar with it and I can find no information on how to tweak it to work. I may revisit this setup as well if needed or I find the time. Right now, I’m focused on using AWX :slight_smile:

If you used a self-signed certificate it will provide encrypted (safe) communication over 443, but no browsers will recognize the certificate because it doesn’t have a root CA which is in their list of approved CAs.

So, you can live with the warning if it’s just for your use or your admin colleagues.

Or, you can obtain a cert from a vendor which can provide a cert chain from a recognized CA.

  1. My second try, was to use ssl in nginx directly, so, I’ve setup the nginx.conf to use ssl, installed my certificates, and, bob is your uncle. SSL was active, websocket working, but when I triggered a job, unfortunately, I got an api error, because awxrest.py uses plain http and not https.

Just finished that way.

  1. Make nginx listen on ssl and non ssl (443,80)

  2. 301 redirect from http to https

  3. include correct headers for websocket

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

Close access to 8052 (awx-web) from outside
Leave 80,443 only exposed for outside world.

Nginx is running on standalone docker host. Not in container.

Hello,

Can you please share nginx.conf file with https configuration setup? I’m facing the similar issue.

Thanks,
Sudheer.