How to install python packages for existing AWX containers?

Archives AWX Project
1

I am trying to use ansible snow module and noticed that the requirement for that is to install pysnow. I have tried installing this by making changes in the existing repo but this always results in error. How can i make this work?

Following are the details.

Docker Containers
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80ab6bf562a9 ansible/awx_task:latest “/tini – /bin/sh -c…” 2 weeks ago Up 2 days 8052/tcp awx_task
aea2551951d5 b7c261b76010 “/tini – /bin/sh -c…” 2 weeks ago Up 2 days 0.0.0.0:80->8052/tcp awx_web
e789a4a82a9e memcached:alpine “docker-entrypoint.s…” 2 weeks ago Up 3 days 11211/tcp memcached
a8c74584255c ansible/awx_rabbitmq:3.7.4 “docker-entrypoint.s…” 2 weeks ago Up 3 days 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq
25f6f6ca7766 postgres:9.6 “docker-entrypoint.s…” 2 weeks ago Up 3 days 5432/tcp postgres

Docker Images
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
postgres 9.6 106bdfb062df 7 weeks ago 235MB
ansible/awx_task latest 535bb2b8e1f3 8 weeks ago 1.07GB
ansible/awx_web b7c261b76010 8 weeks ago 1.04GB
hello-world latest 2cb0d9787c4d 2 months ago 1.85kB
memcached alpine b40e8fa7e3e5 2 months ago 8.69MB
ansible/awx_rabbitmq 3.7.4 e08fe791079e 5 months ago 85.6MB
[root@clab2726vw0001 ~]#

# docker run -ti ansible/awx_task:latest bash

bash-4.2$ pip install pysnow

The directory ‘/.cache/pip/http’ or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.
Collecting pysnow
Downloading https://files.pythonhosted.org/packages/c7/59/e4084626acc59f57ea1a0bb03dc5586165e88324a2b948a11361ad77bdd3/pysnow-0.7.4.tar.gz
Collecting requests (from pysnow)
Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)
100% |################################| 92kB 1.8MB/s
Collecting oauthlib (from pysnow)
Downloading https://files.pythonhosted.org/packages/e6/d1/ddd9cfea3e736399b97ded5c2dd62d1322adef4a72d816f1ed1049d6a179/oauthlib-2.1.0-py2.py3-none-any.whl (121kB)
100% |################################| 122kB 2.8MB/s
Collecting requests_oauthlib (from pysnow)
Downloading https://files.pythonhosted.org/packages/94/e7/c250d122992e1561690d9c0f7856dadb79d61fd4bdd0e598087dce607f6c/requests_oauthlib-1.0.0-py2.py3-none-any.whl
Collecting httpretty (from pysnow)
Downloading https://files.pythonhosted.org/packages/84/60/b25c38767dee62f7cec49dc12a094128c7d2d9e946289db21331d748bb6d/httpretty-0.9.5.tar.gz (4.0MB)
100% |################################| 4.0MB 175kB/s
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.7/site-packages (from pysnow)
Collecting ijson (from pysnow)
Downloading https://files.pythonhosted.org/packages/7f/e9/8508c5f4987ba238a2b169e582c1f70a47272b22a2f1fb06b9318201bb9e/ijson-2.3-py2.py3-none-any.whl
Collecting pytz (from pysnow)
Downloading https://files.pythonhosted.org/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl (510kB)
100% |################################| 512kB 1.4MB/s
Collecting idna<2.8,>=2.5 (from requests->pysnow)
Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
100% |################################| 61kB 3.9MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests->pysnow)
Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
100% |################################| 143kB 3.6MB/s
Collecting urllib3<1.24,>=1.21.1 (from requests->pysnow)
Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
100% |################################| 143kB 3.7MB/s
Collecting certifi>=2017.4.17 (from requests->pysnow)
Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB)
100% |################################| 153kB 3.5MB/s
Installing collected packages: idna, chardet, urllib3, certifi, requests, oauthlib, requests-oauthlib, httpretty, ijson, pytz, pysnow
Found existing installation: idna 2.4
Uninstalling idna-2.4:
Exception:
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/pip/basecommand.py”, line 215, in main
status = self.run(options, args)
File “/usr/lib/python2.7/site-packages/pip/commands/install.py”, line 326, in run
strip_file_prefix=options.strip_file_prefix,
File “/usr/lib/python2.7/site-packages/pip/req/req_set.py”, line 736, in install
requirement.uninstall(auto_confirm=True)
File “/usr/lib/python2.7/site-packages/pip/req/req_install.py”, line 742, in uninstall
paths_to_remove.remove(auto_confirm)
File “/usr/lib/python2.7/site-packages/pip/req/req_uninstall.py”, line 115, in remove
renames(path, new_path)
File “/usr/lib/python2.7/site-packages/pip/utils/init.py”, line 267, in renames
shutil.move(old, new)
File “/usr/lib64/python2.7/shutil.py”, line 299, in move
rmtree(src)
File “/usr/lib64/python2.7/shutil.py”, line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File “/usr/lib64/python2.7/shutil.py”, line 250, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission denied: ‘/usr/lib/python2.7/site-packages/idna/idnadata.pyo’
You are using pip version 8.1.2, however version 18.0 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.

bash-4.2$ pip install --upgrade pip
The directory ‘/.cache/pip/http’ or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.
Collecting pip
Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
100% |################################| 1.3MB 684kB/s
Installing collected packages: pip
Found existing installation: pip 8.1.2
Uninstalling pip-8.1.2:
Exception:
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/pip/basecommand.py”, line 215, in main
status = self.run(options, args)
File “/usr/lib/python2.7/site-packages/pip/commands/install.py”, line 326, in run
strip_file_prefix=options.strip_file_prefix,
File “/usr/lib/python2.7/site-packages/pip/req/req_set.py”, line 736, in install
requirement.uninstall(auto_confirm=True)
File “/usr/lib/python2.7/site-packages/pip/req/req_install.py”, line 742, in uninstall
paths_to_remove.remove(auto_confirm)
File “/usr/lib/python2.7/site-packages/pip/req/req_uninstall.py”, line 115, in remove
renames(path, new_path)
File “/usr/lib/python2.7/site-packages/pip/utils/init.py”, line 267, in renames
shutil.move(old, new)
File “/usr/lib64/python2.7/shutil.py”, line 302, in move
os.unlink(src)
OSError: [Errno 13] Permission denied: ‘/usr/bin/pip’
You are using pip version 8.1.2, however version 18.0 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.

2

AWX makes use of Python virtualenv for playbook execution. Your commands
were trying to install the 'pysnow' package into the global python scope
of the running docker image, which is not the recommended way to install
the requirements.

The virtualenv directory used for executing playbooks is located at
`/var/lib/awx/venv/ansible`.

So, in order to manually install a python package using pip into the
virtulenv, you would run the following commands inside the 'awx_task'
container:

  source /var/lib/awx/venv/ansible/bin/activate

  pip install pysnow

Alternatively, you can simple create playbook that uses the `pip` module
which you can then run as a job in AWX in order to ensure that the
required python modules are present in the virtualenv. This method is
much preferred over the manual way of installing python packages.

3

Thanks Chris,

I was able to create virtual environment and install the packages inside that in aws_tasks container. Post this I have done the commit and it created a new image. How can I use this new image to work with the container with changes. I am having hard time in making the changes work.

4

Hi All,

Can anyone please help in this case?
Currently my docker containers are as follows.

# docker container ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
968fb2a7da2f 5290f9b3268c “/tini – /bin/sh -c…” 2 days ago Exited (143) 2 days ago awx_task
80ab6bf562a9 535bb2b8e1f3 “/tini – /bin/sh -c…” 3 weeks ago Up 2 days 8052/tcp awx_task_OLD
aea2551951d5 b7c261b76010 “/tini – /bin/sh -c…” 3 weeks ago Up 2 days 0.0.0.0:80->8052/tcp awx_web
e789a4a82a9e memcached:alpine “docker-entrypoint.s…” 3 weeks ago Up 2 days 11211/tcp memcached
a8c74584255c ansible/awx_rabbitmq:3.7.4 “docker-entrypoint.s…” 3 weeks ago Up 2 days 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq
25f6f6ca7766 postgres:9.6 “docker-entrypoint.s…” 3 weeks ago Up 2 days 5432/tcp postgres

Here, when i enable latest container (968fb2a7da2f) with latest image (with changes), it doesn’t work. It appears like, the new container is not able to talk to other AWX containers. When i start the old container (80ab6bf562a9), it works normally. In the above snapshot, i have enabled the older container and disabled the new one.

I am stuck at this point and not able to move forward, any help would be really appreciated.