Following #11240 : "The async task did not return valid JSON"

Archives Ansible Developer
1

Hi,

I had the #11240 bug with async tasks (https://github.com/ansible/ansible/issues/11240) with 2.0-devel so I pulled & rebased my ansible installation.

Now I got this error with async tasks :

fatal: [coreos-01]: FAILED! => {“msg”: “The async task did not return valid JSON: No JSON object could be decoded”, “failed”: true, “changed”: false}

my task :
-name: Mount mongo LVM
command: sudo systemctl start data-mongodb-01.mount
ignore_errors: yes
async: 15

I just have to remove the async parameter and wait for the timeout it the task fails.

git version :

commit 4fbd4ae18b39883152f790bf2e59fdfdff973bc7

Regards,

Arthur

2

can you run with -vvvv?

IIRC coreos version of python was 'special' so we might need to make
additional adjustments.

3

Hi,

Without async :
https://gist.github.com/arthur-c/31b239361eb21c857247

With async:
https://gist.github.com/arthur-c/8904f89862de7b177295

I bootstraped ansible on CoreOs with defunctzombie.coreos-bootstrap.

Arthur

4

how long does the task take w/o async?

5

Hi there

I’m also getting this behaviour.
Running a simple command with async, I get “The async task did not return valid JSON ansible”.
Running the same command on the remote box works just fine and takes about 5 minutes to run.

This is the play:

  • name: Trigger ‘jenkins-jobs update’ to populate Jenkins jobs
    command: jenkins-jobs update /etc/jenkins_jobs/config --delete-old
    async: 3600
    poll: 0

And this is the output with -vvvv:

http://paste.openstack.org/show/423349/

Regards

6

Hello everybody,

I’m also getting this behaviour.
Running a simple command with async, I get “The async task did not return valid JSON ansible”.
Running the same command on the remote box works just fine and takes about 5 minutes to run.

I’m also running into this problem. Right now, I’ve traced it back to some problem with the new ‘become’ logic putting additional prompts into the stdout, so that stdout isn’t valid JSON any more. You can see this by applying the following hack to the latest devel checkout:

diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
index fdc57b8…4849686 100644
— a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -322,7 +322,7 @@ class TaskExecutor:
try:
result = json.loads(result.get(‘stdout’))
except (TypeError, ValueError) as e:

  • return dict(failed=True, msg=“The async task did not return valid JSON: %s” % str(e))
  • return dict(failed=True, msg=“The async task did not return valid JSON: %s, >%s<” % (str(e), result.get(‘stdout’)))

if self._task.poll > 0:
result = self._poll_async_result(result=result, templar=templar)

When run with this adhoc debugging hack, I’m getting the following output from the task:

fatal: [testhost]: FAILED! => {“changed”: false, “failed”: true, “msg”: “The async task did not return valid JSON: No JSON object could be decoded, >BECOME-SUCCESS-uuqnvruwvstqxuyjaewoolahhkowahip\r\n{"started": 1, "results_file": "/root/.ansible_async/956808343353.10137", "ansible_job_id": "956808343353.10137"}\r\n<”}

Now I only have to understand if that become-successkey has to be there or how to strip it…

Uli / youam

7

This has not fundamentally changed, the previous SUDO-SUCCESS key
worked in exactly the same way. We were probably cleaning that up
before.

8

Me again.

[…]

Right now, I’ve traced it back to some problem with the new ‘become’ logic putting additional prompts into the stdout, so that stdout isn’t valid JSON any more. You can see this by applying the following hack to the latest devel checkout:

[…]

Now I only have to understand if that become-successkey has to be there or how to strip it…

Turns out, for “regular” tasks, the “stdout” gets set/filtered in lib/ansible/plugins/action/init.py, _execute_module() by this code:

data = json.loads(self._filter_leading_non_json_lines(res.get(‘stdout’, ‘’)))

where the _filter_leading_non_json_lines function states:

Used to avoid random output from SSH at the top of JSON output, like messages from

tcagetattr, or where dropbear spews MOTD on every single command (which is nuts).

need to filter anything which starts not with ‘{’, ‘[’, ', ‘=’ or is an empty line.

filter only leading lines since multiline JSON is valid.

Apparently, this unintentionally (?) also stripped out the BECOME_SUCCESS message - except for async tasks.

Calling the same wrapper fixes my problem. The code has some special cases for skipping stuff if we’re going to use sudo while still connecting as root, which I am (rescue system, root’s the only user there is), so the actual problem might lie elsewhere, though. This feels like a really bad hack, and I’m by far not sure whether my hack breaks more than it fixed, so I’m not going to push this upstream.

— a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -320,7 +320,7 @@ class TaskExecutor:

the async_wrapper module returns dumped JSON via its stdout

response, so we parse it here and replace the result

try:

  • result = json.loads(result.get(‘stdout’))
  • result = json.loads(self._handler._filter_leading_non_json_lines(result.get(‘stdout’)))
    except (TypeError, ValueError) as e:
    return dict(failed=True, msg=“The async task did not return valid JSON: %s” % str(e))

hope this helps someone and/or helps diagnosing the problem.

Uli / youam

9

not sure that is the correct fix as we should be striping the become
output when verifying become success, but I don't know enough of how
async is implemented in relation to privilege escalation to make a
firm statement.