I just performed a OS upgrade on our Ansible server from RedHat 7.6 to RedHat 7.7, and now Ansible has become unusable. When I try to run any task that uses any Ansible fact, I get the following error:
[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.yaml.CallbackModule object at 0x7efe1283de50>):
value must be a string
The task that generated this error was a simple debug statement:
- debug: msg=“{{ansible_distribution}}”
Interestingly, if I use the full hostvars path, it works:
- debug: msg=“{{hostvars[inventory_hostname].ansible_distribution}}”
The error first appeared with Ansible 2.8.1, and I immediately upgraded to Ansible 2.8.1 to see if that would help, but it did not. I suspect that it is an issue in Python, but I have no idea of where to look. Does anyone else have any ideas about this? This has totally brought our use of Ansible to a halt until we can find a resolution for this.
The simple playbook I wrote to test this is below:
- hosts: “localhost”
gather_facts: yes
become: no
tasks:
- debug: msg=“this is a test”
- debug: msg=“{{hostvars[inventory_hostname].ansible_distribution}}”
- debug: msg=“{{ansible_distribution}}”
And full debug output is below:
I used “yum history undo” to rollback the upgrade to RedHat 7.6, and now Ansible is functioning again. Clearly there is something in the OS upgrade that is impacting Ansible. Beware!!!
-Mark
Yeah when I upgraded to RHEL7.7 I got some pretty weird behavior…not just ansible-related.
Hmmm our systems have been RHEL-7.7 for a while and I have not seen
this but we are using
[smooge@batcave01 ansible (master)]$ ansible --version
ansible 2.8.5
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/srv/web/infra/ansible/library',
u'/usr/share/ansible']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5
20150623 (Red Hat 4.8.5-39)]
Do you have a list of python and ansible packages installed before and
after the upgrade? I am not sure it will help but it might.
Unfortunately I have already rolled back to RedHat 7.6 so I do not know what was installed for 7.7. The Python packages that are installed now are below. We had Ansible 2.8.1 installed previously. I upgraded to 2.8.5 to see if that would help (it did not), and that is still installed.
-Mark
rpm -qa | grep python
python2-pyasn1-0.1.9-7.el7.noarch
python2-rhnlib-2.8.11.1-28.1.noarch
libselinux-python-2.5-14.1.el7.x86_64
python-simplejson-3.3.1-2.1.x86_64
python-yubico-1.2.3-1.el7.noarch
python-dateutil-1.5-7.el7.noarch
python-lxml-3.2.1-4.el7.x86_64
python-pycurl-7.19.0-19.el7.x86_64
python-enum34-1.0.4-1.el7.noarch
python27-python-pip-8.1.2-3.el7.noarch
python-passlib-1.6.5-2.el7.noarch
python-msgpack-python-0.4.6-2.1.x86_64
python-certifi-2015.9.6.2-2.1.noarch
python-ldap-2.4.15-2.el7.x86_64
python-ethtool-0.8-7.el7.x86_64
python-httplib2-0.9.2-1.el7.noarch
python27-python-setuptools-0.9.8-7.el7.noarch
python2-jmespath-0.9.0-4.el7ae.noarch
python-babel-0.9.6-8.el7.noarch
python2-hwdata-2.3.5-12.1.noarch
python-cffi-1.6.0-5.el7.x86_64
python-libipa_hbac-1.16.2-13.el7_6.8.x86_64
python-chardet-2.2.1-1.el7_1.noarch
python-pycrypto-2.6.1-5.1.x86_64
python-tornado-4.2.1-5.3.x86_64
python-virtualenv-15.1.0-2.el7.noarch
python-slip-0.4.0-4.el7.noarch
python-inotify-0.9.4-4.el7.noarch
python2-pyasn1-modules-0.1.9-7.el7.noarch
python-nss-0.16.0-3.el7.x86_64
python-decorator-3.4.0-3.el7.noarch
python-libs-2.7.5-77.el7_6.x86_64
dbus-python-1.1.1-9.el7.x86_64
python-pyudev-0.15-9.el7.noarch
python-configobj-4.7.2-7.el7.noarch
python-sssdconfig-1.16.2-13.el7_6.8.noarch
python-gobject-base-3.22.0-1.el7_4.1.x86_64
python-dmidecode-3.12.2-3.el7.x86_64
python-six-1.9.0-2.el7.noarch
python27-python-libs-2.7.13-5.el7.x86_64
python-setuptools-0.9.8-7.el7.noarch
python-pycparser-2.14-1.el7.noarch
python-idna-2.4-1.el7.noarch
python-devel-2.7.5-77.el7_6.x86_64
python-sss-murmur-1.16.2-13.el7_6.8.x86_64
python-magic-5.11-35.el7.noarch
python-paramiko-2.1.1-9.el7.noarch
python-psutil-1.2.1-0.2.1.x86_64
python-dns-1.12.0-4.20150617git465785f.el7.noarch
python-kitchen-1.1.1-5.el7.noarch
python-qrcode-core-5.0.1-1.el7.noarch
python-netifaces-0.10.4-3.el7.x86_64
newt-python-0.52.15-4.el7.x86_64
python-2.7.5-77.el7_6.x86_64
python-gudev-147.2-7.el7.x86_64
python-linux-procfs-0.4.9-4.el7.noarch
python-perf-3.10.0-957.12.2.el7.x86_64
rpm-python-4.11.3-35.el7.x86_64
python2-futures-3.1.1-5.el7.noarch
redhat-support-lib-python-0.9.7-6.el7.noarch
python2-cryptography-1.7.2-2.el7.x86_64
python-ipaddress-1.0.16-2.el7.noarch
python27-python-2.7.13-5.el7.x86_64
python-ply-3.4-11.el7.noarch
python-jinja2-2.7.2-3.el7_6.noarch
python-zmq-14.5.0-2.1.x86_64
python-jwcrypto-0.4.2-1.el7.noarch
python-schedutils-0.4-6.el7.x86_64
libxml2-python-2.9.1-6.el7_2.3.x86_64
python-firewall-0.5.3-5.el7.noarch
python-urlgrabber-3.10-9.el7.noarch
python27-runtime-1.1-26.1.el7.x86_64
python-backports-1.0-8.el7.x86_64
python-markupsafe-0.11-10.el7.x86_64
python-gssapi-1.2.0-3.el7.x86_64
python-slip-dbus-0.4.0-4.el7.noarch
python-netaddr-0.7.5-9.el7.noarch
python-iniparse-0.4-9.el7.noarch
python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
So today I ran a playbook on our Ansible server that is still running RedHat 7.6 and I saw the issue again. Through experimenting, I finally traced it to setting "ANSIBLE_STDOUT_CALLBACK=yaml" before running the playbook, or having "stdout_callback = yaml" in the configuration file. That really surprised me as I have been doing that for a long time with no problem, but something apparently has changed. The puzzling thing is everything was fine until I upgraded the OS, and I started having issues after that. I thought it had cleared up after I rolled back, but I see now that there is still something wrong here. It does not seem to occur as much after the rollback, but it still does happen.
I can continue to use Ansible by not setting ANSIBLE_STDOUT_CALLBACK, but it seems very odd that this is now an issue that seems to have come out of nowhere. I will keep poking at it to see if I can narrow it down some.
-Mark
I am glad you 'found' what the issue looks like but my condolences on
it becoming an issue now. I have not used those callbacks so would not
have even thought of looking there.
You can shield yourself from such inadvertent OS changes by using the pip installation method, preferably with virtual env.
This will also allow you to run any version of ansible.
Dick