Failed to access license information. GET returned status:

Archives AWX Project
1

This error message appears when the admin user logs in to the web UI on a fresh install of Ansible AWX, and I can’t find anything in this user/dev forum or in github regarding this issue.

Login seems successful, but none of the web UI is available to the admin user. The full API is still available to the admin user, so the error condition seems to be the result of the web UI validating against a license server.

We have quite a lot of Ansible in git repos our org, and we were hoping to use AWX to manage some of the inventory and playbook runs. The evaluation server is running on a single VM host, though I installed a few times while I was trying to understand Docker, frankly, and to work around a bug with private keys in vault credentials, so I’m guessing our public IP has been locked out due to too many installations.

If I’m correct, I might be able to work around this by moving the VM behind a different public IP, but I’d rather get advice from the dev/user community first. I don’t think we’re doing anything unethical, or in breach of what I assume is the license governing the project, https://github.com/ansible/awx/blob/devel/LICENSE.md.

Also, since it seems updating AWX is a matter of removing Docker images and re-installing (please correct me if I’m wrong, I didn’t see a best-practice recommendation in github docs or elsewhere), I have concerns that we will get locked out in the future as we try to keep our AWX instance up to date.

Any help would be appreciated.

2

There are no license checks in AWX. Was your past experience with trying out Tower or the open source project AWX?

Updating AWX code is a matter of a new docker image, as you said. The data migration, however, is a bit more complex https://github.com/ansible/awx/blob/devel/DATA_MIGRATION.md

3

Hey Peter,

I’ve opened up an issue in the awx project: https://github.com/ansible/awx/issues/2162. I suspect that the message in that error is erroneous and that we aren’t showing the real reason for the error. If it’s possible, can you take a look at the web console in your browser when you see this error and report it back to us. That might shed some light on what’s actually going on here.

Mike

4

Thank you Christoper, we are trying out AWX first, and have not yet used Tower in our environment. Thanks for the tip on data migration.

5

Hi Micheal, thanks for your interest, and I’ve had some time to look at this. The problem seems to come from un-commenting #project_data_dir=/var/lib/awx/projects in awx/installer/inventory. I’ve done this a couple of times now, and the AWX web UI is available (no exceptions) when I don’t define project_data_dir in the inventory (I tried a few different locations) but raises the error condition in my post title (no exceptions) when I do.

I believe the relevant Javascript error is as follows. There are also websocket connection errors and eventually a TypeError in the Javascript console, but the connection errors are also present when I don’t encounter the 500 error.

Possibly unhandled rejection: {“data”:{“detail”:“A server error has occurred.”},“status”:500,“config”:{“method”:“GET”,“transformRequest”:[null],“transformResponse”:[null],“jsonpCallbackParam”:“callback”,“url”:“/api/v2/config/”,“headers”:{“Accept”:“application/json, text/plain, /”,“X-Requested-With”:“XMLHttpRequest”},“params”:null},“statusText”:“Internal Server Error”,“xhrStatus”:“complete”} vendor.9d966aeb1d3c348bc617.js:570:122456

consoleLog/< http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:122456 $ExceptionHandlerProvider/this.$get</< http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:93654 processChecks http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:135255 $digest http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:146521 $apply http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:149463 done http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:101267 completeRequest http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:106296 createHttpBackend/</xhr.onload http://192.168.10.57/static/js/vendor.9d966aeb1d3c348bc617.js:570:107075

Following are my notes on installing AWX on a CentOS 7.5 Minimal VM. Pretty basic.

Ansible AWS installation/upgrade

Uninstall

docker kill $(docker ps -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)

Upgrade

TODO. Look at https://github.com/ansible/awx/blob/devel/DATA_MIGRATION.md.

Install

Target host CentOS 7.5.1804 Minimal, fresh install on VirtualBox VM, 1 CPU, 4 GB RAM.

Does AWX expect to run as root?
Hard to tell.
Is there a way not to?

Install global RPM packages:

sudo yum install epel-release
sudo yum update
sudo yum install wget git docker make python-pip

To run the installer as non-root user, add docker group
and add the installation user to the group,
log out and in to activate the new group permission:

sudo groupadd docker
sudo usermod -aG docker $USER

Start and enable docker:

sudo systemctl enable docker.service
sudo systemctl start docker.service

Upgrade pip and install Ansible in the venv:

sudo pip install ansible docker-py

Check https://github.com/ansible/awx/blob/devel/INSTALL.md#prerequisites
and install the right version of Node and NPM from
https://nodejs.org/dist/.

wget https://nodejs.org/dist/latest-v6.x/node-v6.14.3-linux-x64.tar.gz
tar -xzvf node-v6.14.3-linux-x64.tar.gz
sudo ln -s $HOME/node-v6.14.3-linux-x64/bin/node /usr/local/bin/node
sudo ln -s $HOME/node-v6.14.3-linux-x64/bin/npm /usr/local/bin/npm
git clone https://github.com/ansible/awx.git

Edit awx/installer/inventory:

postgres_data_dir=/var/lib/awx
project_data_dir=/var/lib/awx/projects

Run the installer:

cd awx/installer
ansible-playbook -i inventory install.yml

6

A 500 error represents a server-side error. Can you grab the logs from the awx_web (particularly as it relates to an exception) via something to the effect of: docker logs -f awx_web

You’ll want to make sure that’s running when you try to execute that so you can catch the log message as it goes by.

7

Hi Matthew, the error is indeed related to the projects directory, as I suspected.

Here are the file permissions on the projects directory as created by the installer playbook.

[root@awx awx]# ll
total 4
drwx------. 19 polkitd root 4096 Aug 13 17:58 pgdata
drwxr-xr-x. 2 root root 6 Aug 13 18:00 projects

Note that I ran the AWX installation playbook as an unprivileged user in the docker user group. Maybe the error has something to do with that. Note that SELinux is enabled. (I noticed that the pgdata directory is owned by polkitd.) I’ve posted my install notes elsewhere in this post. I’m not clear on how Docker works with user groups, file permissions, and SELinux, but I installed as I did mostly because I noticed $ in the install doc and not # for the command that runs the playbook against the inventory file. I didn’t notice any directive to disable SELinux or to add policy rules. Perhaps I was to do such, or to install as root?

I tried polkit.root and peter.peter (the installation user) on the directory, but with the same results. I’m not sure I have time to fuss with this any further, so I’ll leave it here. Thanks all for your attention, and I hope this has been helpful for your project.

Here is the output of awx_web logs, as requested:

2018-08-14 15:47:17,784 INFO awx.api.generics User admin logged in.
192.168.70.51 - - [14/Aug/2018:15:47:20 +0000] “POST /api/login/ HTTP/1.1” 302 0 “http://192.168.10.57/” “Mozilla/5.0 (X11; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0” “-”
[pid: 192|app: 0|req: 5/144] 192.168.70.51 () {52 vars in 996 bytes} [Tue Aug 14 15:47:16 2018] POST /api/login/ => generated 0 bytes in 4213 msecs (HTTP/1.1 302) 13 headers in 1844 bytes (1 switches on core 0)
[pid: 201|app: 0|req: 2/145] 192.168.70.51 () {50 vars in 2233 bytes} [Tue Aug 14 15:47:20 2018] GET /api/ => generated 173 bytes in 2769 msecs (HTTP/1.1 200) 11 headers in 471 bytes (1 switches on core 0)
192.168.70.51 - - [14/Aug/2018:15:47:23 +0000] “GET /api/ HTTP/1.1” 200 173 “http://192.168.10.57/” “Mozilla/5.0 (X11; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0” “-”
/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/pagination.py:208: UnorderedObjectListWarning: Pagination may yield inconsistent results with an unordered object_list: <class ‘django.contrib.auth.models.User’> QuerySet.
paginator = self.django_paginator_class(queryset, page_size)
192.168.70.51 - - [14/Aug/2018:15:47:26 +0000] “GET /api/v2/me/ HTTP/1.1” 200 896 “http://192.168.10.57/” “Mozilla/5.0 (X11; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0” “-”
[pid: 192|app: 0|req: 6/146] 192.168.70.51 () {46 vars in 2074 bytes} [Tue Aug 14 15:47:23 2018] GET /api/v2/me/ => generated 896 bytes in 3315 msecs (HTTP/1.1 200) 10 headers in 374 bytes (1 switches on core 0)
192.168.70.51 - - [14/Aug/2018:15:47:29 +0000] “GET /websocket/ HTTP/1.1” 101 29 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0” “-”
DAMN ! worker 2 (pid: 192) died, killed by signal 9 :frowning: trying respawn …
Respawned uWSGI worker 2 (new pid: 253)
2018-08-14 15:47:30,584 ERROR django.request Internal Server Error: /api/v2/config/
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py”, line 41, in inner
response = get_response(request)
File “/usr/lib/python2.7/site-packages/awx/wsgi.py”, line 71, in _legacy_get_response
return super(AWXWSGIHandler, self)._legacy_get_response(request)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py”, line 249, in _legacy_get_response
response = self._get_response(request)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py”, line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py”, line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/utils/decorators.py”, line 185, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py”, line 58, in wrapped_view
return view_func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py”, line 68, in view
return self.dispatch(request, *args, **kwargs)
File “/usr/lib/python2.7/site-packages/awx/api/generics.py”, line 328, in dispatch
return super(APIView, self).dispatch(request, *args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py”, line 494, in dispatch
response = self.handle_exception(exc)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py”, line 454, in handle_exception
self.raise_uncaught_exception(exc)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py”, line 491, in dispatch
response = handler(request, *args, **kwargs)
File “/usr/lib/python2.7/site-packages/awx/api/views.py”, line 444, in get
project_local_paths = Project.get_local_path_choices(),
File “/usr/lib/python2.7/site-packages/awx/main/models/projects.py”, line 71, in get_local_path_choices
paths = [x.decode(‘utf-8’) for x in os.listdir(settings.PROJECTS_ROOT)
OSError: [Errno 13] Permission denied: ‘/var/lib/awx/projects’
2018-08-14 15:47:30,584 ERROR django.request Internal Server Error: /api/v2/config/
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py”, line 41, in inner
response = get_response(request)
File “/usr/lib/python2.7/site-packages/awx/wsgi.py”, line 71, in _legacy_get_response
return super(AWXWSGIHandler, self)._legacy_get_response(request)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py”, line 249, in _legacy_get_response
response = self._get_response(request)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py”, line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py”, line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/utils/decorators.py”, line 185, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py”, line 58, in wrapped_view
return view_func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py”, line 68, in view
return self.dispatch(request, *args, **kwargs)
File “/usr/lib/python2.7/site-packages/awx/api/generics.py”, line 328, in dispatch
return super(APIView, self).dispatch(request, *args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py”, line 494, in dispatch
response = self.handle_exception(exc)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py”, line 454, in handle_exception
self.raise_uncaught_exception(exc)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py”, line 491, in dispatch
response = handler(request, *args, **kwargs)
File “/usr/lib/python2.7/site-packages/awx/api/views.py”, line 444, in get
project_local_paths = Project.get_local_path_choices(),
File “/usr/lib/python2.7/site-packages/awx/main/models/projects.py”, line 71, in get_local_path_choices
paths = [x.decode(‘utf-8’) for x in os.listdir(settings.PROJECTS_ROOT)
OSError: [Errno 13] Permission denied: ‘/var/lib/awx/projects’
[pid: 201|app: 0|req: 3/147] 192.168.70.51 () {46 vars in 2154 bytes} [Tue Aug 14 15:47:26 2018] GET /api/v2/config/ => generated 41 bytes in 4438 msecs (HTTP/1.1 500) 6 headers in 200 bytes (1 switches on core 0)
192.168.70.51 - - [14/Aug/2018:15:47:30 +0000] “GET /api/v2/config/ HTTP/1.1” 500 41 “http://192.168.10.57/” “Mozilla/5.0 (X11; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0” “-”

8

Since you are mapping the project directory from the docker host into the container you’ll need to make sure that directory has the right permissions for it to be accessible within the container itself. That container will see the same uid/gid that the docker host sees but the docker daemon itself might not have permission to handle that.

If, as you say, you have a lot of git repos in your org then there’s no explicit reason that you need to map a local folder into the container… AWX itself manages checking out and updating those repos on its own, and makes no changes therein. The only reason, really, to map a directory for the projects like that is if you want to manage a manual AWX project which is not something we generally recommend.

9

Hi Matt, I’m not sure what those permissions would be. I wasn’t able to find permissions on the directory that the Docker container could read. Perhaps setting the correct permissions should be part of the installer?

The reason I tried setting up a project directory is that we host our Ansible playbooks on a private GitLab server, and we use SSH keys to access repos. I was able to add an SSH key pair to AWX credentials, but when I tried to add a repo, the credential was not available to select in the AWX web UI. I found a relevant bug report in github (I’ve lost track of it, vault/Docker related…), but I was unwilling to change our GitLab process to accommodate AWX, so I tried the project directory configuration option as a workaround and ran up against this problem, also seemingly Docker related.

This experience raises a rhetorical question I’ve been asking myself for quite a while now. Does Docker solve more problems than it introduces, or does it introduce more problems than it solves? I still have no idea.

Thanks again for your attention to this matter.

10

Peter,

Sound like you might have created a Machine Credential when you meant to create a Source Control credential.

-Chris

11

Hi Chris, I recall trying both, but that’s entirely possible. I’ll look into it, thanks.