venv installing system wide pip packages into venv virtual environment in ansible

cool_prat · April 30, 2021, 10:23am

Hi All,

I am running ansible to install some pip packages into virtual environment using venv but its installing exact same version and no. packages from system wide python and not what is the mentioned in requirement.txt.

I am able to achieve success when i am doing it manually. i.e system wide pip packages and virtual environment pip packages are different.

I am getting below error in ansible

TASK [curator : Install pip packages List.] ****************************************************************************************************************************
fatal: [elasticsearch_1]: FAILED! => {
"changed": false,
"cmd": "/home/user/elk-venv/bin/pip list --format=freeze",
"rc": 13
}

MSG:

[Errno 13] Permission denied

Steps i am doing through ansible task are -

Creating venv using command module as below

# /usr/bin/python3.6 -m venv /home/user/elk-venv

Copying requirements.txt file and tranferred_packages folder inside /home/user/elk-venv/.

requirements.txt has list of pip packages and tranferred_packages folder has downloaded packages. basically installing packages offline way.

Using pip module trying to install packages as below,

- name: Install pip packages List.
pip:
requirements: /home/user/elk-venv/requirements.txt
extra_args: "--no-index --find-links=file:////home/user/elk-venv/tranferred_packages"
virtualenv: /home/user/elk-venv
virtualenv_python: /home/user/elk-venv/bin/python3.6

Below is the virtual environment created by ansible (elk-env)

Getting permission denied when trying with bin/pip3 list.

(elk-venv) [root@Server elk-venv]# bin/pip3 list
-bash: bin/pip3: Permission denied

able to list the pip3 list like below, but it seems to be copy of system python

(elk-venv) [root@Server elk-venv]# pip3 list
attrs (19.3.0)
blist (1.3.6)

If we see below it shows /usr/local/lib/python3.6/site-packages.

(elk-venv) [root@Server elk-venv]# pip3 show pytz
Name: pytz
Version: 2019.3
Summary: World timezone definitions, modern and historical
Home-page: http://pythonhosted.org/pytz
Author: Stuart Bishop
Author-email: stuart@stuartbishop.net
License: MIT
Location: /usr/local/lib/python3.6/site-packages
Requires:
(elk-venv) [root@Server elk-venv]#

Not sure why but config file also showing /usr/bin

(elk-venv) [root@Server elk-venv]# cat pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.6.8
(elk-venv)

pip/python files under bin folder. python3.6 is pointing to /usr/bin/python3.6

(elk-venv) [root@Server bin]# ls -lrth
lrwxrwxrwx 1 root root 18 Apr 28 21:56 python3.6 -> /usr/bin/python3.6
lrwxrwxrwx 1 root root 9 Apr 28 21:56 python3 -> python3.6
lrwxrwxrwx 1 root root 9 Apr 28 21:56 python -> python3.6
-rwxr-xr-x 1 root root 224 Apr 28 21:56 pip
-rwxr-xr-x 1 root root 224 Apr 28 21:56 pip3.6
-rwxr-xr-x 1 root root 224 Apr 28 21:56 pip3

files under elk-venv venv. Can’t see LICENSE file here which is there in devenv.

drwxr-x--- 2 root root 4.0K Apr 28 21:56 include
-rw-r----- 1 root root 69 Apr 28 21:56 pyvenv.cfg
lrwxrwxrwx 1 root root 3 Apr 28 21:56 lib64 -> lib
drwxr-x--- 3 root root 4.0K Apr 28 21:56 lib
drwxr-x--- 2 root root 4.0K Apr 28 21:56 bin
-rw-r----- 1 root root 1022 Apr 28 21:56 requirements.txt
drwxr-x--- 3 root root 4.0K Apr 28 21:56 tranferred_packages

Created below virtual environment manually (devenv) it has its own pip packages.

here able to list the packages. Not getting permission denied.

(devenv) [root@Server devenv]# bin/pip list
Package Version
--------------------- ----------
APScheduler 3.6.3
attrs 19.3.0
pip 21.1

It shows package from its own path i.e /root/devenv/lib/python3.6/site-packages.

# pip3 show pytz
Name: pytz
Version: 2019.3
Summary: World timezone definitions, modern and historical
Home-page: http://pythonhosted.org/pytz
Author: Stuart Bishop
Author-email: stuart@stuartbishop.net
License: MIT
Location: /root/devenv/lib/python3.6/site-packages
Requires:
Required-by: tzlocal, twilio, APScheduler

config file is showing /bin.


(devenv) [root@Server devenv]# cat pyvenv.cfg
home = /bin
include-system-site-packages = false
version = 3.6.8
(devenv) [root@Server devenv]#

pip/python files under bin folder. here python3.6 is pointing to /bin/python3.6.

(devenv) [root@Server bin]# ls -lrth
total 148K
lrwxrwxrwx 1 root root 14 Apr 27 17:39 python3.6 -> /bin/python3.6
lrwxrwxrwx 1 root root 9 Apr 27 17:39 python3 -> python3.6
lrwxrwxrwx 1 root root 9 Apr 27 17:39 python -> python3.6
-rwxr-xr-x 1 root root 233 Apr 28 12:29 pip3.6
-rwxr-xr-x 1 root root 233 Apr 28 12:29 pip3
-rwxr-xr-x 1 root root 233 Apr 28 12:29 pip

files under devenv venv.

-rw-r----- 1 root root 65 Apr 27 17:39 pyvenv.cfg
lrwxrwxrwx 1 root root 3 Apr 27 17:39 lib64 -> lib
drwxr-x--- 3 root root 4.0K Apr 27 17:39 lib
drwxr-x--- 2 root root 4.0K Apr 27 17:39 include
-rw-r----- 1 root root 12K Apr 28 12:30 LICENSE
drwxr-x--- 3 root root 4.0K Apr 28 13:30 bin
drwxr-x--- 2 root root 4.0K Apr 28 20:45 tranferred_packages
-rw-r----- 1 root root 1022 Apr 28 20:45 requirements.txt
(devenv) [root@Server devenv]#

ansible version is - ansible 2.9.9
ansible python version = 2.7.5

I think something missing in the pip module above in anisble causing this. Can someone please tell what it is?

Component Name

pip

Ansible Version

ansible 2.9.9

Configuration

$ ansible-config dump --only-changed

ANSIBLE_PIPELINING(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=60s -o PreferredAuthentications=publickey
ANSIBLE_SSH_CONTROL_PATH(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = /tmp/ansible-%%h
CACHE_PLUGIN(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = /tmp/facts_cache
CACHE_PLUGIN_TIMEOUT(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = 7200
DEFAULT_FORKS(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = 4
DEFAULT_GATHERING(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = smart
DEFAULT_MODULE_PATH(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = [u'/usr/share/ansible', u'/opt/comp/PS_INSTALL/DEV/ansible-develop/rol
DEFAULT_PRIVATE_KEY_FILE(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = /home/obadm/.ssh/id_rsa
DEFAULT_REMOTE_USER(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = obadm
DEFAULT_STDOUT_CALLBACK(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = debug
DEFAULT_TIMEOUT(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = 720
DEFAULT_VAULT_PASSWORD_FILE(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = /opt/comp/PS_INSTALL/DEV/ansible-develop/vault
DEPRECATION_WARNINGS(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = False
HOST_KEY_CHECKING(/opt/comp/PS_INSTALL/DEV/ansible-develop/ansible.cfg) = False

OS / Environment

RHEL 7.8

Steps to Reproduce

run the above ansible code to reproduce

Expected Results

virtual environment (i.e elk-venv in this case) should have its own pip packages. pip3 show command should show its own path and not system paython path.

Actual Results

TASK [curator : Install pip packages List.] ****************************************************************************************************************************
fatal: [elasticsearch_1]: FAILED! => {
"changed": false,
"cmd": "/home/user/elk-venv/bin/pip list --format=freeze",
"rc": 13
}

MSG:

[Errno 13] Permission denied

Thanks,

Topic		Replies	Views
Pip upgrade pip failing in ansible Ansible Project ubuntu	0	3	January 21, 2022
Pip module command fails, shell alternative succeeds Ansible Project	2	0	October 16, 2013
Using virtual environment on target host(s) Get Help howto	5	415	October 12, 2024
ANSIBLE - Venv - Use /usr/bin/ansible Ansible Project	1	1	December 16, 2022
Installing pip packages into a virtualenv - where are they? Ansible Project	3	0	April 30, 2015