Hi all,
I'm working on configuring my provisioning infrastructure to be as automated as possible, and I've run into one small snag. I've successfully configured Foreman to talk to AWX via the callback service which results in newly provisioned nodes running a set of roles I've defined depending on the type of system deployed. Where I'm running into issues is with inventory syncing.
In AWX, I created a new inventory and added a Satellite source pointing to my Foreman instance. There are no problems synchronizing hosts from this source (pulling in host groups, host collections, locations, and parameters). I've also enabled the "Overwrite" and "Overwrite variables" options on the source. For the template that Foreman uses in the callback, I've enabled the "Provisioning Callbacks" and "Concurrent Jobs" parameters and defined the host config key. Lastly, as this is running on an OKD cluster I added "HTTP_X_FORWARDED_FOR" to the remote host headers in the miscellaneous system settings:
[
"REMOTE_ADDR",
"REMOTE_HOST",
"HTTP_X_FORWARDED_FOR"
]
Now, the entire process of provisioning and the callback kicking off the Ansible run works just fine as long as I have the Foreman source I mentioned earlier configured to update itself whenever a job is launched. This works great when deploying a host or two, however it ends up tripping over itself when multiple systems reboot at the same time after PXE'ing and call the template at the same time. This results in the inventory refresh being requested while it's already being refreshed causing the queued jobs to bomb out and not run. The journal on the hosts ends up reporting "no matching host could be found." To get around this, I tried disabling the inventory's update on launch, and added an inventory sync using the AWX collection to the end of the playbook I use to create the hosts in Foreman and vSphere. I confirmed this successfully added all the hosts with the appropriate details and variables into AWX, however every new host reported the no matching hosts error when the callback got executed. Manually refreshing the inventory again resolved that issue, and the only visible difference I could see after diffing the variables and groups was the addition of a foreman_content_attributes.uuid variable.
My question: has anyone been able to successfully configure the callback service to handle multiple simultaneous deployments without needing to update the inventory before every single host run? I'm at a bit of a loss on this one.
Infrastructure:
* Foreman 3.5 / Katello 4.7
* OKD 4.12.0-0.okd-2023-04-16-041331
* AWX Operator 2.0.0 (Installed via OperatorHub graphical install on OKD)
* AWX 22.0.0
* vSphere 7.0u3