Would an operator to maintain the projects, credentials, job templates etc in AWX be feasible? Is there an effort ongoing to create one?
Not quite sure what that would look like but there already is a collection called redhat-cop controller collection that manages that for you.
This already exists here:
That is an operator that allows you to define AWX resources as yaml that gets applied to your cluster. Then the awx-resource-operator will create those resources in your configured AWX instance. I think this is what you are referring to, please let me know if that is not the case.
Thanks,
AWX Team
I started playing with “awx-resource-operator” and got it working to trigger jobs. According to the README.md, it can “launch automation jobs and create job templates”, no mention of projects, inventories, credentials etc. However I see that deploying this operator deploys CRDs for projects, inventories and credentials as well as ansible jobs and job_templates, though only those for ansible jobs and job_templates are documented in the README.md.
I’ve been able to deploy the operator and successfully create a job template and trigger a job with it. However it fails when I attempt to create a project, with the error below.
(⎈|minikube:awx-resource-operator) awx-resource-operator $ kubectl -n awx-resource-operator logs -f job.batch/test-project
Found 2 pods, using pod/test-project-p58hg
[WARNING]: Unable to parse /runner/inventory as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match ‘all’
PLAY [localhost] ***************************************************************
TASK [job_runner : Read AnsibleJob Specs] **************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: raise ApiException(http_resp=r)
fatal: [localhost]: FAILED! => {“changed”: false, “module_stderr”: “Traceback (most recent call last):\n File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 270, in request\n api_response = self.client.call_api(\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET\n return self.request("GET", url,\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request\n raise ApiException(http_resp=r)\nkubernetes.client.exceptions.ApiException: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({‘Audit-Id’: ‘9fddb15b-2e7e-4e16-9458-872f355e52fd’, ‘Cache-Control’: ‘no-cache, private’, ‘Content-Type’: ‘application/json’, ‘X-Content-Type-Options’: ‘nosniff’, ‘X-Kubernetes-Pf-Flowschema-Uid’: ‘b9f7d140-0db8-42b5-a5d5-3718a8e0995e’, ‘X-Kubernetes-Pf-Prioritylevel-Uid’: ‘4ec9bad5-85dd-49a3-8eae-72ace7866a7a’, ‘Date’: ‘Thu, 13 Jul 2023 08:51:25 GMT’, ‘Content-Length’: ‘405’})\nHTTP response body: b’{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"ansiblejobs.tower.ansible.com is forbidden: User \\"system:serviceaccount:awx-resource-operator:resource-operator-controller-manager-job\\" cannot list resource \\"ansiblejobs\\" in API group \\"tower.ansible.com\\" at the cluster scope","reason":"Forbidden","details":{"group":"tower.ansible.com","kind":"ansiblejobs"},"code":403}\n’\n\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/root/.ansible/tmp/ansible-tmp-1689238282.8839157-58-279482922564523/AnsiballZ_k8s_info.py", line 107, in \n _ansiballz_main()\n File "/root/.ansible/tmp/ansible-tmp-1689238282.8839157-58-279482922564523/AnsiballZ_k8s_info.py", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File "/root/.ansible/tmp/ansible-tmp-1689238282.8839157-58-279482922564523/AnsiballZ_k8s_info.py", line 47, in invoke_module\n runpy.run_module(mod_name=‘ansible_collections.kubernetes.core.plugins.modules.k8s_info’, init_globals=dict(_module_fqn=‘ansible_collections.kubernetes.core.plugins.modules.k8s_info’, _modlib_path=modlib_path),\n File "/usr/lib64/python3.8/runpy.py", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.8/runpy.py", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_k8s_info_payload_eqx5643j/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 206, in \n File "/tmp/ansible_k8s_info_payload_eqx5643j/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 202, in main\n File "/tmp/ansible_k8s_info_payload_eqx5643j/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py", line 173, in execute_module\n File "/tmp/ansible_k8s_info_payload_eqx5643j/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/common.py", line 324, in kubernetes_facts\n File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 112, in get\n return self.request(‘get’, path, **kwargs)\n File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 57, in inner\n raise api_exception(e)\nkubernetes.dynamic.exceptions.ForbiddenError: 403\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({‘Audit-Id’: ‘9fddb15b-2e7e-4e16-9458-872f355e52fd’, ‘Cache-Control’: ‘no-cache, private’, ‘Content-Type’: ‘application/json’, ‘X-Content-Type-Options’: ‘nosniff’, ‘X-Kubernetes-Pf-Flowschema-Uid’: ‘b9f7d140-0db8-42b5-a5d5-3718a8e0995e’, ‘X-Kubernetes-Pf-Prioritylevel-Uid’: ‘4ec9bad5-85dd-49a3-8eae-72ace7866a7a’, ‘Date’: ‘Thu, 13 Jul 2023 08:51:25 GMT’, ‘Content-Length’: ‘405’})\nHTTP response body: b’{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"ansiblejobs.tower.ansible.com is forbidden: User \\"system:serviceaccount:awx-resource-operator:resource-operator-controller-manager-job\\" cannot list resource \\"ansiblejobs\\" in API group \\"tower.ansible.com\\" at the cluster scope","reason":"Forbidden","details":{"group":"tower.ansible.com","kind":"ansiblejobs"},"code":403}\n’\nOriginal traceback: \n File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n\n File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 270, in request\n api_response = self.client.call_api(\n\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET\n return self.request("GET", url,\n\n File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request\n raise ApiException(http_resp=r)\n\n”, “module_stdout”: “”, “msg”: “MODULE FAILURE\nSee stdout/stderr for the exact error”, “rc”: 1}
PLAY RECAP *********************************************************************
localhost
Any idea?
the user scope error seems relevant forbidden: User \\"system:serviceaccount:awx-resource-operator:resource-operator-controller-manager-job\\" cannot list resource \\"ansiblejobs\\" in API group \\"tower.ansible.com\\" at the cluster scope
I would start by looking into fixing these permission errors.
I’ll mention that for jobs, the operator makes calls API via awx collection, but for creating a project the operator is applying a custom resource to the k8s cluster. Hence the reason it works for jobs but not projects.
AWX Team
Does this mean that for projects, the awx-resource-operator needs to be on the same cluster or even namespace as AWX? Is there any reason why 2 different methods, API and CRD are used for configuring these resources? I see that the operator is fairly new. Does the fact that 2 different methods are being used indicate that an experiment is going on, to determine the way forward?
This is odd, I am not sure how CI would be passing… But I want to investigate this more. Do you mind opening an issue with this info in the awx-resource-operator repo?
If I am able to reproduce, I’ll put up a fix.
Thanks,
AWX Team