We are deploying AWX using the AWX-operator in Kubernetes. New AWX jobs spin up automation-job pods. Those pods have logs during execution that aren’t rendered in the UI, but are available at run time on the pod itself. In a K8s platform environment, it’s common to send all pod logs for all pods in the cluster to a central log aggregator. That being said, there are logs that the automation job pod logs by default, that don’t appear in the AWX UI, but do appear in the back end log aggregator that leaks sensitive information. Specifically, the startup log. It looks something like:
That being said, some jobs need sensitive information, like passwords, sent to Ansible modules through ENV variables. Those ENV variables are getting pooped out to the startup log in the automation pod job by default, and then sent to our log aggregator along with all pod logs in the whole cluster. There doesn’t appear to be any way to prevent the automation job from logging the env at runtime. Is there a way to prevent this? If not, could we request a way to reduce logging or turn off this log?
You would need to implement security policies in your logging system to prevent access to sensitive information. There is not a mechanism to obfuscate this, as stdout is the communication mechanism between the execution node, and the EE container.
Unfortunately with Kubernetes mass log aggregation, there is no way to say that 1 pod of the 10,000 should not be doing the same thing as the other 9,999 pods that are logging. All stdout for all pods are all saved to the aggregation tool, that’s how Kubernetes cloud logging works.
If this functionality does not exist yet, what is the process to request the ability to turn off Ansible from logging this data?
It isn’t logging, it that stdout is the communication mechanism. The fact that stdout is being logged is something in k8s, and in fact the entire mechanism in which the way this is implemented actually uses the container log facility to access the stdout from the container.
So basically the entire architecture of AWX relies on using stdout as the communication mechnism to communicate between the execution node, and the EE container. There is no just turning it off, except for not being able to spawn jobs. Anything to address this would likely require a rather large re-architecture that is outside of my area of influence. However, you are free to open a feature request at https://github.com/ansible/awx to implement a new mechanism for this control communication.
The stance on this is that it is up to the user to confiure their logging aggregation or other logging software to have security policies in place, that would prevent users who should not have access to these credentials from viewing them in the logging system. Or otherwise disable the logging aggregation.