6.0.0: AWX docker install fails on "Failed to import the required Python library: docker" weirdness

I can’t get 6.0.0 AWX docker installation to succeed due to ‘Failed to import the required Python library’. The odd thing is that I know with absolute certainty that it is present:

roles/local_docker/tasks/compose.yml

`

  • name: install docker…
    pip:
    name: docker

  • name: Start the containers
    docker_compose:
    project_src: “{{ docker_compose_dir }}”
    restarted: “{{ awx_compose_config is changed or awx_secret_key is changed }}”
    register: awx_compose_start
    `

program output

`
TASK [local_docker : install docker…] ****************************************
task path: /opt/awx/installer/roles/local_docker/tasks/compose.yml:25
ok: [localhost] => {“changed”: false, “cmd”: [“/usr/bin/pip2”, “install”, “docker”], “name”: [“docker”], “requirements”: null, “state”: “present”, “stderr”: “”, “stderr_lines”: , “stdout”: “Requirement already satisfied: docker in /usr/local/lib/python2.7/dist-packages (4.0.2)\nRequirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python2.7/dist-packages (from docker) (0.56.0)\nRequirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/local/lib/python2.7/dist-packages (from docker) (2.22.0)\nRequirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from docker) (1.0.22)\nRequirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/local/lib/python2.7/dist-packages (from docker) (3.7.0.1)\nRequirement already satisfied: six>=1.4.0 in /usr/local/lib/python2.7/dist-packages (from docker) (1.12.0)\nRequirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (2.8)\nRequirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (3.0.4)\nRequirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (2019.6.16)\nRequirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (1.25.3)\n”, “stdout_lines”: [“Requirement already satisfied: docker in /usr/local/lib/python2.7/dist-packages (4.0.2)”, “Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python2.7/dist-packages (from docker) (0.56.0)”, “Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/local/lib/python2.7/dist-packages (from docker) (2.22.0)”, “Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from docker) (1.0.22)”, “Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/local/lib/python2.7/dist-packages (from docker) (3.7.0.1)”, “Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python2.7/dist-packages (from docker) (1.12.0)”, “Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (2.8)”, “Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (3.0.4)”, “Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (2019.6.16)”, “Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests!=2.18.0,>=2.14.2->docker) (1.25.3)”], “version”: null, “virtualenv”: null}

TASK [local_docker : Start the containers] *************************************
task path: /opt/awx/installer/roles/local_docker/tasks/compose.yml:29
fatal: [localhost]: FAILED! => {“changed”: false, “msg”: “Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on eris’s Python /usr/bin/python. Please read module documentation and install in the appropriate location, for example via pip install docker or pip install docker-py (Python 2.6). The error was: cannot import name credentials”}
`

I get this error even if I change the task to force the images to be rebuilt.

This is on an ubuntu disco (19.04) system. Python 2.7.16, docker version 18.09.7, build 2d0083d

Now for the real weirdness…

If I go to the awxcompose directory and run ‘docker-compose up’ manually I get a ‘docker-compose.yml: file not found’ error. This happens even if explicitly add it via ‘-f’. The file definitely exists and is readable.

If I change to root, source environment.sh, and run ‘docker-compose up’ I get a bit further:

`
root@eris:/tmp/awxcompose# docker-compose up
Pulling rabbitmq (ansible/awx_rabbitmq:3.7.4)…
Traceback (most recent call last):
File “/usr/local/lib/python3.7/dist-packages/docker/credentials/store.py”, line 80, in _execute
[self.exe, subcmd], input=data_input, env=env,
File “/usr/lib/python3.7/subprocess.py”, line 395, in check_output
**kwargs).stdout
File “/usr/lib/python3.7/subprocess.py”, line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[’/usr/bin/docker-credential-secretservice’, ‘get’]’ returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.7/dist-packages/docker/auth.py”, line 264, in _resolve_authconfig_credstore
data = store.get(registry)
File “/usr/local/lib/python3.7/dist-packages/docker/credentials/store.py”, line 35, in get
data = self._execute(‘get’, server)
File “/usr/local/lib/python3.7/dist-packages/docker/credentials/store.py”, line 93, in _execute
raise errors.process_store_error(e, self.program)
docker.credentials.errors.StoreError: Credentials store docker-credential-secretservice exited with “Error calling StartServiceByName for org.freedesktop.secrets: Timeout was reached”.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/bin/docker-compose”, line 11, in
load_entry_point(‘docker-compose==1.21.0’, ‘console_scripts’, ‘docker-compose’)()
File “/usr/lib/python3/dist-packages/compose/cli/main.py”, line 71, in main
command()
File “/usr/lib/python3/dist-packages/compose/cli/main.py”, line 127, in perform_command
handler(command, command_options)
File “/usr/lib/python3/dist-packages/compose/cli/main.py”, line 1052, in up
to_attach = up(False)
File “/usr/lib/python3/dist-packages/compose/cli/main.py”, line 1048, in up
silent=options.get(‘–quiet-pull’),
File “/usr/lib/python3/dist-packages/compose/project.py”, line 466, in up
svc.ensure_image_exists(do_build=do_build, silent=silent)
File “/usr/lib/python3/dist-packages/compose/service.py”, line 324, in ensure_image_exists
self.pull(silent=silent)
File “/usr/lib/python3/dist-packages/compose/service.py”, line 1142, in pull
output = self.client.pull(repo, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/docker/api/image.py”, line 395, in pull
header = auth.get_config_header(self, registry)
File “/usr/local/lib/python3.7/dist-packages/docker/auth.py”, line 48, in get_config_header
client._auth_configs, registry, credstore_env=client.credstore_env
File “/usr/local/lib/python3.7/dist-packages/docker/auth.py”, line 322, in resolve_authconfig
return authconfig.resolve_authconfig(registry)
File “/usr/local/lib/python3.7/dist-packages/docker/auth.py”, line 235, in resolve_authconfig
cfg = self._resolve_authconfig_credstore(registry, store_name)
File “/usr/local/lib/python3.7/dist-packages/docker/auth.py”, line 281, in _resolve_authconfig_credstore
‘Credentials store error: {0}’.format(repr(e))
docker.errors.DockerException: Credentials store error: StoreError(‘Credentials store docker-credential-secretservice exited with “Error calling StartServiceByName for org.freedesktop.secrets: Timeout was reached”.’)
`

However if pull out the individual bits I can launch them - mostly - and eventually am able to get a little further:

`
bgiles@eris:/opt/awxcompose$ sudo docker-compose up
Creating network “awxcompose_default” with the default driver
Creating awx_rabbitmq … done
Creating awx_memcached … done
Creating awx_web … error

ERROR: for awx_web Cannot start service web: b"error while creating mount source path ‘/opt/awxcompose/SECRET_KEY’: mkdir /opt/awxcompose: read-only file system"

ERROR: for web Cannot start service web: b"error while creating mount source path ‘/opt/awxcompose/SECRET_KEY’: mkdir /opt/awxcompose: read-only file system"
ERROR: Encountered errors while bringing up the project.
`

The “mostly” is because I also get a ‘read only filesystem’ error when I try to start the postgresql instance. In this case it’s not a problem since I would prefer to use an external database anyway but it’s also killing the web even though it’s not doing anything other than reading a few files.

I think I’ve seen ‘read-only filesystem’ elsewhere.

Another oddity - I can get past the ‘read-only file system’ if I move everything to /tmp/awxcompose’. (That’s a little disturbing since I would lose it after a reboot.) However when run docker-compose up I get this error:

awx_web | /usr/bin/launch_awx.sh: line 8: source: /etc/tower/conf.d/environment.sh: is a directory

for both awx_web and awx_tasks. I’ve used ‘docker exec’ to verify that these really are directories.

This happens even if I use

`
web:
image: ansible/awx_web:6.0.0
container_name: awx_web
depends_on:

  • rabbitmq
  • memcached
    ports:
  • “80:8052”
    hostname: awxweb
    user: root
    restart: unless-stopped
    volumes:
  • “/tmp/awxcompose/SECRET_KEY:/etc/tower/SECRET_KEY”
  • “/tmp/awxcompose/:/etc/tower/conf.d/”
    environment:
    http_proxy:
    https_proxy:
    no_proxy:
    `

The environments.sh and credentials.py files are definitely files, not directories, so I don’t know why they’re getting mapped to directories.

It’s not due to file permissions - it happens even with permissions.

Another note - my usual account still can’t run docker-compose directly, not even if I add ‘-f /tmp/awxcompose/docker-compose.yml’. The docker-compose.yml file is only seen if I run docker-compose as root.

I know this isn’t a problem with my docker installation since I’m currently running three images launched by a different docker-compose file.