redhat 7 files truncated / 0 length post ansible build and reboot

Hi,

version: ansible 2.2.1.0
ansible build server OS - Centos 6.7
ansible client servers OS - Redhat 7.2. (esxi vm)
Physical Host machine: ESXi 6.0
Hardware: Dell R730 with PERC H730 Mini Embedded controller. (Write back enabled)

We have been deploying servers using ansible for nearly 3 years now… About 6 months ago we started moving from centos 6 to redhat 7 to meet operational support requirements. At the end of each server build we perform a reboot. We had been noticing the odd 0 length configuration files post reboot on our 7.2 servers and have finally taken some time to take a more in depth look at what might be happening.

We are running the same playbooks and roles against a redhat 7 server as we do against a centos 6 server. The centos 6 servers are not impacted by this issue.
At the end of each build just prior to reboot we create a build file via the template module with a little bit of information about the type of build it was, by whom, dates etc…
During the build you can clearly see the file is created and has the content we expect in it…The server then reboots, when we check file post reboot its 0 bytes.

I’ve also logged a case with Redhat and Dell to try and understand whats happening.

Has anyone else come across this issue? If so any pointers in troubleshooting it?

I added some auditd monitoring of the file in question and it doesn’t appear that any systemd services are touching the file.

I’m wondering if its something to do with the way the ansible template module / python copies the tmp files it needs and then the combination of the dell perc controller (with write back enabled), our esxi datastores & the type of controller on the virtual machine.

If i put a sleep of 40 seconds between the file creation and the reboot VM we dont get the issue.

Many Thanks in advance.

This sound like a missing cache sync to disk.

If I remembered correctly the default file system in RHEL7 is XFS and RHEL6 is EXT4.

EXT4 do automatic sync every few seconds, XFS doesn't do that.
You could try to replace the sleep 40 seconds with sync to see if it's a sync cache to disk issue.