How to detect why Ansible playbook hangs during execution

Archives Ansible Project
1

Some of tasks I wrote start and never end. Ansible does not provide any errors or logs that would explain this, even with -vvvv option. Playbook just hangs and passing hours doesn’t change anything.

When I try to run my tasks manually (by entering commands via SSH) everything is fine.

Example task that hangs:

- name: apt upgrade
  shell: apt-get upgrade

Is there any way to see stdout and stderr ? I tried:

- name: apt upgrade
  shell: apt-get upgrade
  register: hello
- debug: msg="{{ hello.stdout }}"
- debug: msg="{{ hello.stderr }}"

but nothing changed.

I do have required permissions and I pass correct sudo password - other tasks that require sudo execute correctly.

2

Apt by default needs eleventy zillion flags to not go interactive.

There’s a parameter you should use on the ‘apt’ module that does upgrades that will take care of this for you.

3

Thanks for answer.

However, apt is not the only task that hangs without ANY information why.

Another example:

shell: apt-get install --no-install-recommends virtualbox-guest-utils && sudo apt-get install virtualbox-guest-dkms

Should I always use your apt module? Why can’t I just execute simple shell command ?

Note that I tried to execute all these tasks manually and there was no errors and no need for any user interaction. Why Ansible has problem with them?

4

you can execute shell commands, just not ones that expect interactivity as they hold the tty (this happens with any background ssh execution).

for apt commands, yes use the apt module, it takes care of 99% f these issues.

5

You should absolutely use the apt module.

You can install multiple packages in a single step by using “with_items” and there’s a flag for controlling whether recommended packages are installed (see module docs).

I suspect as for the commands not being interactive the second time it was because you’ve already executed them and already bypassed the shell prompts.

6

Thank you for your answers. I will use your apt module.

But just for the record - I always rollback to fresh virtual machine, so the reason for hanging is not double task execution for sure.

7

What is the parameter of the apt module to use to avoid that?

I run the following and it hangs…

  • name: update APT package cache
    sudo: yes
    apt: update_cache=yes

  • name: upgrade all safe packages
    sudo: yes
    apt: upgrade=safe

Thanks

8

apt should never need a parameter to not hang, however some specific packages can sometimes be poorly coded.

Not saying that’s the case here.

I’d first check your ansible version and then file a bug with steps to reproduce.

9

So there is no solution to avoid being stuck on the following? It happens to me often when I am upgrading the system!

  • name: upgrade all safe packages
    sudo: yes
    apt: upgrade=safe

Is there at least a way to see the progress? Looking at some logs on the machine?

The state of my machine is messy when this task is stuck because I have to stop ansible-playbook with CTRL+C and all the handlers are not notified… :-/

10

When you kill a playbook you can use --force-handlers to make sure handlers on a repeated runs.

11

Does that also work when one task fails and the playbook stops?
Does it work with the tasks of the roles as well?

12

I just tried to run my playbook with --force-handlers and did CTRL+C: it stopped it right away and the handlers were not notified.
Does it work for you?
I use ansible 1.6.7

Thanks.

13

This is a misunderstanding of force-handler behavior.

You would run-it again with --force-handlers, and any handlers would run, whether notified or not.

Don’t control-C the second run :slight_smile:

14

I use a screen trick but mostly useful when using commands modules. Anyway, if it hangs doing apt upgrade, the behavior should be the same if your run it as a command.

  • debug: msg=“Please, connect to {{ inventory_hostname }}, screen -r, and respond the possible questions.”
    when: ansible_hostname == drbd_master.name
  • name: Enable HA
    command: screen -dm -S halizard bash -c ‘ha-cfg status’
    when: ansible_hostname == drbd_master.name
    async: 60
    poll: 60

It’s also useful when you cannot avoid interactive mode on some actions, but you miss the unattended possibilities.

Br,
Tony

(attachments)

email_smslogo.png
email_icon_mobile.png
email_icon_phone.png
email_icon_skype.png
email_icon_location.png
email_icon_email.png
email_icon_linkedin.png
email_icon_website.png

15

Hi

So there is no solution to avoid being stuck on the following? It happens to me
often when I am upgrading the system!
- name: upgrade all safe packages
  sudo: yes
  apt: upgrade=safe

Is there at least a way to see the progress? Looking at some logs on the
machine?

In the case of apt (which uses dpkg), it may be useful to:

tail -f /var/log/dpkg.log

on the server being upgraded.

If you interrupt the task, I'd expect ansible to give info about what
was output by the upgrade so far - that should contain clues as to
what it was doing at the time of the upgrade.

Also, for debugging purposes, it may be useful to find out which tty
is being used by ansible on the target machine (ps -ef|grep ansible or
similar), and see what is running on that tty:

    ps -ft${basename-of-the-tty}

Hope this helps

16

Hi

sorry for reply in old post but just want to comment something about…

What about asynchronous polling?

https://docs.ansible.com/playbooks_async.html

What if we run a playbook like:

  • hosts: vm-ubuntu
    sudo: true
    user: root
    serial: 4
    vars_files:
  • external_vars.yml
    tasks:
  • name: update cache
    apt: update_cache=yes cache_valid_time=86400
  • name: update packages full into each server
    apt: name={{ item }} state=present
    with_items: ubuntu_packages
    ignore_errors: True
    async: 45
    pool: 5
    tags:
  • install

Will be useful to avoid SSH timeouts in a long playbooks tasks?

What do you think?

Thanks