Just throwing this out for ideas - it may yet turn into a feature request ticket.
I plan to use the Ansible API to manage some infrastructure, but some of the tasks will take several hours to complete: in particular, copying 1TB+ files from an NFS mount to local disk. (“shell: cp /nfs/foo /local/bar” or the rsync equivalent)
Another usage case I have for a long-running command is running “vmbuilder”, which can take 5-20 minutes to build a VM, and reports its progress to stderr (very verbosely if you pass the --debug flag)
I would like to have a way to monitor those long-running tasks, to show that they are making progress and not stalled, and display the total work done and the current rate of progress.
There are a couple of ways I know of doing this at the O/S level:
(1) If you know the PID of the process doing the copying, you can look at cat /proc/<pid>/io to get a record of total bytes read and written. This can easily be converted into a rate.
(2) You can use a tool which writes progress information to stderr as it runs, for example the tool ‘pv’ (pipeline viewer) which is basically a replacement for ‘cat’ that displays progress.
However what I don’t yet have is a good way of integrating this with ansible: while a task is running, either get the pid of the process on the target host, or be able to monitor the stderr stream as it is generated, rather than at the very end.
One way of handling the copy usage case would be to have a local copy module which runs directly on the target system (like “assemble”) which also contains instrumentation to give progress updates. The same could also be added for Ansible’s file transfers (e.g. conn.put_file). However there still needs to be some querying mechanism, and it doesn’t solve the more general case of getting progress data from any long-running command.
I guess the simplest solution would be to build the result structure up-front and have some way to query it while it’s running; then any stderr output which is appended to the buffer would become visible as it’s written. Other arbitrary changes could also be made to the result structure as it runs.
Has there been any work in this area?
Regards,
Brian.