# Ansible is totally broken after a OS upgrade **URL:** \ **Category:** Ansible Project\ **Created:** [October 28, 2019, 8:08pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556 "2019-10-28T20:08:40Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![mark.tovey](https://avatars.discourse-cdn.com/v4/letter/m/ce73a5/32.png) [@mark.tovey](https://forum.ansible.com/u/mark.tovey)\ **Post date:** [October 28, 2019, 8:08pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/1 "2019-10-28T20:08:40Z")
I just performed a OS upgrade on our Ansible server from RedHat 7.6 to RedHat 7.7, and now Ansible has become unusable. When I try to run any task that uses any Ansible fact, I get the following error: > [WARNING]: Failure using method (v2\_runner\_on\_ok) in callback plugin (\): > > value must be a string The task that generated this error was a simple debug statement: > - debug: msg=“{{ansible\_distribution}}” Interestingly, if I use the full hostvars path, it works: > - debug: msg=“{{hostvars[inventory\_hostname].ansible\_distribution}}” The error first appeared with Ansible 2.8.1, and I immediately upgraded to Ansible 2.8.1 to see if that would help, but it did not. I suspect that it is an issue in Python, but I have no idea of where to look. Does anyone else have any ideas about this? This has totally brought our use of Ansible to a halt until we can find a resolution for this. The simple playbook I wrote to test this is below: > * * * > > - hosts: “localhost” > gather\_facts: yes > become: no > > tasks: > > - debug: msg=“this is a test” > - debug: msg=“{{hostvars[inventory\_hostname].ansible\_distribution}}” > - debug: msg=“{{ansible\_distribution}}” And full debug output is below: ---
**Author:** ![mark.tovey](https://avatars.discourse-cdn.com/v4/letter/m/ce73a5/32.png) [@mark.tovey](https://forum.ansible.com/u/mark.tovey)\ **Post date:** [October 28, 2019, 9:44pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/2 "2019-10-28T21:44:11Z")
I used “yum history undo” to rollback the upgrade to RedHat 7.6, and now Ansible is functioning again. Clearly there is something in the OS upgrade that is impacting Ansible. Beware!!! -Mark ---
**Author:** ![Chris\_Bidwell](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Chris\_Bidwell](https://forum.ansible.com/u/Chris_Bidwell)\ **Post date:** [October 28, 2019, 9:55pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/3 "2019-10-28T21:55:51Z")
Yeah when I upgraded to RHEL7.7 I got some pretty weird behavior…not just ansible-related. ---
**Author:** ![Stephen\_John\_Smoogen](https://avatars.discourse-cdn.com/v4/letter/s/ad7895/32.png) [@Stephen\_John\_Smoogen](https://forum.ansible.com/u/Stephen_John_Smoogen)\ **Post date:** [October 29, 2019, 3:14pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/4 "2019-10-29T15:14:01Z")
Hmmm our systems have been RHEL-7.7 for a while and I have not seen this but we are using [smooge@batcave01 ansible (master)]$ ansible --version ansible 2.8.5   config file = /etc/ansible/ansible.cfg   configured module search path = [u'/srv/web/infra/ansible/library', u'/usr/share/ansible']   ansible python module location = /usr/lib/python2.7/site-packages/ansible   executable location = /usr/bin/ansible   python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] Do you have a list of python and ansible packages installed before and after the upgrade? I am not sure it will help but it might. ---
**Author:** ![mark.tovey](https://avatars.discourse-cdn.com/v4/letter/m/ce73a5/32.png) [@mark.tovey](https://forum.ansible.com/u/mark.tovey)\ **Post date:** [October 29, 2019, 3:53pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/5 "2019-10-29T15:53:30Z")
Unfortunately I have already rolled back to RedHat 7.6 so I do not know what was installed for 7.7. The Python packages that are installed now are below. We had Ansible 2.8.1 installed previously. I upgraded to 2.8.5 to see if that would help (it did not), and that is still installed.     -Mark rpm -qa | grep python python2-pyasn1-0.1.9-7.el7.noarch python2-rhnlib-2.8.11.1-28.1.noarch libselinux-python-2.5-14.1.el7.x86\_64 python-simplejson-3.3.1-2.1.x86\_64 python-yubico-1.2.3-1.el7.noarch python-dateutil-1.5-7.el7.noarch python-lxml-3.2.1-4.el7.x86\_64 python-pycurl-7.19.0-19.el7.x86\_64 python-enum34-1.0.4-1.el7.noarch python27-python-pip-8.1.2-3.el7.noarch python-passlib-1.6.5-2.el7.noarch python-msgpack-python-0.4.6-2.1.x86\_64 python-certifi-2015.9.6.2-2.1.noarch python-ldap-2.4.15-2.el7.x86\_64 python-ethtool-0.8-7.el7.x86\_64 python-httplib2-0.9.2-1.el7.noarch python27-python-setuptools-0.9.8-7.el7.noarch python2-jmespath-0.9.0-4.el7ae.noarch python-babel-0.9.6-8.el7.noarch python2-hwdata-2.3.5-12.1.noarch python-cffi-1.6.0-5.el7.x86\_64 python-libipa\_hbac-1.16.2-13.el7\_6.8.x86\_64 python-chardet-2.2.1-1.el7\_1.noarch python-pycrypto-2.6.1-5.1.x86\_64 python-tornado-4.2.1-5.3.x86\_64 python-virtualenv-15.1.0-2.el7.noarch python-slip-0.4.0-4.el7.noarch python-inotify-0.9.4-4.el7.noarch python2-pyasn1-modules-0.1.9-7.el7.noarch python-nss-0.16.0-3.el7.x86\_64 python-decorator-3.4.0-3.el7.noarch python-libs-2.7.5-77.el7\_6.x86\_64 dbus-python-1.1.1-9.el7.x86\_64 python-pyudev-0.15-9.el7.noarch python-configobj-4.7.2-7.el7.noarch python-sssdconfig-1.16.2-13.el7\_6.8.noarch python-gobject-base-3.22.0-1.el7\_4.1.x86\_64 python-dmidecode-3.12.2-3.el7.x86\_64 python-six-1.9.0-2.el7.noarch python27-python-libs-2.7.13-5.el7.x86\_64 python-setuptools-0.9.8-7.el7.noarch python-pycparser-2.14-1.el7.noarch python-idna-2.4-1.el7.noarch python-devel-2.7.5-77.el7\_6.x86\_64 python-sss-murmur-1.16.2-13.el7\_6.8.x86\_64 python-magic-5.11-35.el7.noarch python-paramiko-2.1.1-9.el7.noarch python-psutil-1.2.1-0.2.1.x86\_64 python-dns-1.12.0-4.20150617git465785f.el7.noarch python-kitchen-1.1.1-5.el7.noarch python-qrcode-core-5.0.1-1.el7.noarch python-netifaces-0.10.4-3.el7.x86\_64 newt-python-0.52.15-4.el7.x86\_64 python-2.7.5-77.el7\_6.x86\_64 python-gudev-147.2-7.el7.x86\_64 python-linux-procfs-0.4.9-4.el7.noarch python-perf-3.10.0-957.12.2.el7.x86\_64 rpm-python-4.11.3-35.el7.x86\_64 python2-futures-3.1.1-5.el7.noarch redhat-support-lib-python-0.9.7-6.el7.noarch python2-cryptography-1.7.2-2.el7.x86\_64 python-ipaddress-1.0.16-2.el7.noarch python27-python-2.7.13-5.el7.x86\_64 python-ply-3.4-11.el7.noarch python-jinja2-2.7.2-3.el7\_6.noarch python-zmq-14.5.0-2.1.x86\_64 python-jwcrypto-0.4.2-1.el7.noarch python-schedutils-0.4-6.el7.x86\_64 libxml2-python-2.9.1-6.el7\_2.3.x86\_64 python-firewall-0.5.3-5.el7.noarch python-urlgrabber-3.10-9.el7.noarch python27-runtime-1.1-26.1.el7.x86\_64 python-backports-1.0-8.el7.x86\_64 python-markupsafe-0.11-10.el7.x86\_64 python-gssapi-1.2.0-3.el7.x86\_64 python-slip-dbus-0.4.0-4.el7.noarch python-netaddr-0.7.5-9.el7.noarch python-iniparse-0.4-9.el7.noarch python-backports-ssl\_match\_hostname-3.5.0.1-1.el7.noarch ---
**Author:** ![mark.tovey](https://avatars.discourse-cdn.com/v4/letter/m/ce73a5/32.png) [@mark.tovey](https://forum.ansible.com/u/mark.tovey)\ **Post date:** [October 29, 2019, 9:16pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/6 "2019-10-29T21:16:56Z")
So today I ran a playbook on our Ansible server that is still running RedHat 7.6 and I saw the issue again. Through experimenting, I finally traced it to setting "ANSIBLE\_STDOUT\_CALLBACK=yaml" before running the playbook, or having "stdout\_callback = yaml" in the configuration file. That really surprised me as I have been doing that for a long time with no problem, but something apparently has changed. The puzzling thing is everything was fine until I upgraded the OS, and I started having issues after that. I thought it had cleared up after I rolled back, but I see now that there is still something wrong here. It does not seem to occur as much after the rollback, but it still does happen.     I can continue to use Ansible by not setting ANSIBLE\_STDOUT\_CALLBACK, but it seems very odd that this is now an issue that seems to have come out of nowhere. I will keep poking at it to see if I can narrow it down some.     -Mark ---
**Author:** ![Stephen\_John\_Smoogen](https://avatars.discourse-cdn.com/v4/letter/s/ad7895/32.png) [@Stephen\_John\_Smoogen](https://forum.ansible.com/u/Stephen_John_Smoogen)\ **Post date:** [October 29, 2019, 9:25pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/7 "2019-10-29T21:25:07Z")
I am glad you 'found' what the issue looks like but my condolences on it becoming an issue now. I have not used those callbacks so would not have even thought of looking there. ---
**Author:** ![Dick\_Visser](https://avatars.discourse-cdn.com/v4/letter/d/d9b06d/32.png) [@Dick\_Visser](https://forum.ansible.com/u/Dick_Visser)\ **Post date:** [October 29, 2019, 9:35pm UTC](https://forum.ansible.com/t/ansible-is-totally-broken-after-a-os-upgrade/31556/8 "2019-10-29T21:35:31Z")
You can shield yourself from such inadvertent OS changes by using the pip installation method, preferably with virtual env. This will also allow you to run any version of ansible. Dick