msg: Failed to lock apt for exclusive operation - apt: update_cache=yes

Archives Ansible Project
1

Hello,

Last week we updated one of our servers from unbent 12.0+ to 14.0+ it deployed with out issue, after the weekend deployment to one of servers is no longer possible :frowning:

Here is the command that is failing:

  • name: Update the apt cache if older than 2 hrs
    apt: update_cache=yes cache_valid_time=7200
    tags:
  • security-apt

On just one of our servers prd1

TASK: [common | Update the apt cache if older than 2 hrs] *********************
ESTABLISH CONNECTION FOR USER: ubuntu
ESTABLISH CONNECTION FOR USER: ubuntu
REMOTE_MODULE apt update_cache=yes cache_valid_time=7200
REMOTE_MODULE apt update_cache=yes cache_valid_time=7200
EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/ericbrooke/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ubuntu’, ‘-o’, ‘ConnectTimeout=10’, ‘prd1’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-1412016466.77-35664788881142 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1412016466.77-35664788881142 && echo $HOME/.ansible/tmp/ansible-tmp-1412016466.77-35664788881142’”]
EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/ericbrooke/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ubuntu’, ‘-o’, ‘ConnectTimeout=10’, ‘prd3’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-1412016466.77-191935961010276 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1412016466.77-191935961010276 && echo $HOME/.ansible/tmp/ansible-tmp-1412016466.77-191935961010276’”]
PUT /var/folders/dz/_g3k142j3vz652trrswwwxf80000gq/T/tmpnz2A7A TO /home/ubuntu/.ansible/tmp/ansible-tmp-1412016466.77-191935961010276/apt
PUT /var/folders/dz/_g3k142j3vz652trrswwwxf80000gq/T/tmp0xdCki TO /home/ubuntu/.ansible/tmp/ansible-tmp-1412016466.77-35664788881142/apt
EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/ericbrooke/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ubuntu’, ‘-o’, ‘ConnectTimeout=10’, ‘prd1’, u’/bin/sh -c 'sudo -k && sudo -H -S -p “[sudo via ansible, key=XXX] password: " -u root /bin/sh -c '”'“'echo SUDO-SUCCESS-XXX; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1412016466.77-35664788881142/apt; rm -rf /home/ubuntu/.ansible/tmp/ansible-tmp-1412016466.77-35664788881142/ >/dev/null 2>&1'”'“''‘]
EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-q’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/ericbrooke/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ubuntu’, ‘-o’, ‘ConnectTimeout=10’, ‘prd3’, u’/bin/sh -c 'sudo -k && sudo -H -S -p “[sudo via ansible, key=XXX] password: " -u root /bin/sh -c '”'”'echo SUDO-SUCCESS-XXX; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1412016466.77-191935961010276/apt; rm -rf /home/ubuntu/.ansible/tmp/ansible-tmp-1412016466.77-191935961010276/ >/dev/null 2>&1'“'”''']
ok: [prd3 => {“changed”: false}
failed: [prd1] => {“failed”: true}

I have replaced the keys with XXX

2

Anything relevant in the logs?

3

Which logs would you like to see?

4

I fixed it with a server reboot. Is there a place to restart this service rather then the whole server. Any advice appreciated. Thanks in advance

5

not a service, locks on the apt DB are done to avoid inconsistencies,
either someone else was running an update or a lock got leftover
(probably with a hung/interrupted process).

you can normally use lsof to find out what the offending process is
and kill it, restart fixed it by killing ALL processes.

6

Thank you Brian