Hi all,
I’m noticing this on v7.0.0 (didn’t see it on 6.0.0; hadn’t tried 6.1.0) where my role requirements aren’t being downloaded. Has anyone else seen this?
Detailed issue is described here: https://github.com/ansible/awx/issues/4682
Fran
This is an expected change of behavior as a result of a recent feature merge. The roles are downloaded before a job runs, not in the project update. Have you experienced any issues running jobs that use those roles?
Eep. I realized I hadn't been hitting Reply All, so I'm CC'ing the
mailing list with another response.
Anyway, to compare that output to another AWX instance of mine with v6.0.0:
=== ======= ======================== ========== ========
=========================== =======
id project name status job_type
created elapsed
=== ======= ======================== ========== ========
=========================== =======
146 88 redacted-master successful run
2019-09-04T14:07:14.679296Z 7.714
148 88 redacted-master successful check
2019-09-04T14:08:32.529242Z 13.889
149 88 redacted-master successful run
2019-09-04T14:08:54.488568Z 7.88
151 88 redacted-master successful check
2019-09-04T14:22:47.037406Z 14.505
152 88 redacted-master successful run
2019-09-04T14:23:09.578303Z 6.978
154 88 redacted-master successful check
2019-09-04T14:27:31.961189Z 14.0
155 88 redacted-master successful run
2019-09-04T14:27:53.945092Z 7.277
157 88 redacted-master successful check
2019-09-04T14:45:55.339746Z 23.42
158 88 redacted-master successful run
2019-09-04T14:46:26.798998Z 7.286
160 88 redacted-master successful check
2019-09-04T14:49:58.269942Z 21.962
161 88 redacted-master successful run
2019-09-04T14:50:28.924425Z 6.979
163 88 redacted-master successful check
2019-09-04T14:51:39.157141Z 14.642
164 88 redacted-master successful run
2019-09-04T14:52:02.002206Z 7.077
166 88 redacted-master successful check
2019-09-04T14:53:49.383286Z 13.834
167 88 redacted-master successful run
2019-09-04T14:54:11.412924Z 6.832
169 88 redacted-master successful check
2019-09-04T14:56:52.642154Z 13.732
170 88 redacted-master successful run
2019-09-04T14:57:14.373014Z 7.159
172 88 redacted-master successful check
2019-09-04T14:59:00.789157Z 15.496
173 88 redacted-master successful run
2019-09-04T14:59:24.397416Z 7.107
175 88 redacted-master successful check
2019-09-04T15:10:48.067277Z 63.171
176 88 redacted-master successful run
2019-09-04T15:11:59.069144Z 8.684
178 88 redacted-master successful check
2019-09-04T15:22:39.657026Z 14.556
=== ======= ======================== ========== ========
=========================== =======
Here you can see I'm getting both `check` and `run` project_update jobs.
Could you do a quick debugging test for me?
On your project, if you uncheck the box to allow override of branch, then does it download requirements? Also, when you encountered this bug, what was your branch value, both on the project & the job template?
I may be close to putting together a picture.
I don't think I've ever been as bad at hitting 'Reply All' as I am
with doing it in this thread.
Adding the mailing list back in.
Shame on me.
If you go to the job in the API (/api/v2/jobs/42/), look into the “related” dictionary, is a link present under the key “project_update”? If that is not present and your project has the roles/requirements.yml file, then that’s a bug. If you are on the same page as me about these facts, then I’m happy to re-open your issue.
My challenge for pinning down this issue is that I can’t yet create the situation you saw. This is a reference case in a public repo:
https://github.com/ansible/test-playbooks/tree/with_requirements
I’m happy to re-run this and give you any details from my server to compare, to help find where they diverge.
The most valuable information for me me would be if you could turn the log level to DEBUG (you can do this in the UI settings under logging), run your job, and then get the server logs. I very specifically expect to see a message like “Running project sync for job 67 because of galaxy role requirements”.
Is the project type git?
Thanks,
Is the project type git?
Yup. 
So I launched a new job from a template. It spawned Job #61. This
triggered Project Update #62.
Job 61s related dictionary:
"related": {
"created_by": "/api/v2/users/1/",
"labels": "/api/v2/jobs/61/labels/",
"inventory": "/api/v2/inventories/4/",
"project": "/api/v2/projects/8/",
"extra_credentials": "/api/v2/jobs/61/extra_credentials/",
"credentials": "/api/v2/jobs/61/credentials/",
"unified_job_template": "/api/v2/job_templates/9/",
"stdout": "/api/v2/jobs/61/stdout/",
"job_events": "/api/v2/jobs/61/job_events/",
"job_host_summaries": "/api/v2/jobs/61/job_host_summaries/",
"activity_stream": "/api/v2/jobs/61/activity_stream/",
"notifications": "/api/v2/jobs/61/notifications/",
"create_schedule": "/api/v2/jobs/61/create_schedule/",
"job_template": "/api/v2/job_templates/9/",
"cancel": "/api/v2/jobs/61/cancel/",
"relaunch": "/api/v2/jobs/61/relaunch/"
}
I just realized something... what path does AWX detect
`roles/requirements.yml` from? It it "{path of
playbook}/roles/requirements.yml" (probably unless otherwise
configured differently from ansible.cfg)?
I ask because with this project that I'm working with, the playbooks
aren't at the root of the directory. It would be something like "{git
project root}/ansible/roles/requirements.yml" and the playbook is
"{git project root}/ansible/my_awesome_playbook.yml".
As for your other logs, I'll start working on collecting them for ya.
Alan,
See the debug logs here at this gist (retrieved by docker-compose logs
-f task): https://gist.github.com/fxfitz/0640924cfd5c3f9540e4fa6ab76d8f70
The template job is #63, and the project_update is #64. As for
anything related to a project sync, this is what I see: awx.main.tasks
Skipping project sync for job 63 (running) because commit is locally
available
Fran
I just realized something… what path does AWX detect
roles/requirements.ymlfrom? It it “{path of playbook}/roles/requirements.yml” (probably unless otherwise configured differently from ansible.cfg)?
AWX will only consider roles/requirements.yml relative to the root of the project. There is no option to configure this.
If you still have that server running with an older version, it might help to look at what this was actually doing before the upgrade.
In your logs, project update 64 was launched as a “dependency”, and this is not what I was looking for. Your logs confirm that the project update needed to install rolls did not run.
So, what's the next step forward? Reopening the issue?
FWIW, I just added another project that does have "{git repo
root}/roles/requirements.yml" and also observed that it did not
download roles; once again `roles_enabled=False` in
`project_update.yml`.
You previously mentioned the project parameters:
“Clean”, "Update Revision on Launch (Cache Timeout=0), and “Allow Branch Override”
(and project branch left blank)
I will get back to this and test this situation for a project that has “{git repo root}/roles/requirements.yml”. You mentioned that you are still seeing the issue. Do these parameters still apply?
Yup! They sure do.
Thanks for all your assistance!
A few questions and comments so far while I'm also debugging locally:
1) How come the change from using `project_update.yml` to the RunJob's
`pre_run_hook`? I'm just curious as to the reasoning to the move
simply because the other way had much easier visibility into the
problem, not to mention having both makes it even more confusing
(meaning having the pre_run_hook and tasks in project_update.yml doing
the same thing). For me personally, I liked being able to see the
roles being updated in `project_update.yml`.
2) It's becoming more and more apparent to me that AWX should not
hardcode `roles/requirements.yml`. I see in
https://github.com/ansible/awx/issues/4540 it's being discussed, but I
don't think we should simply add `roles/requirements.yaml` to the
search list.
I think instead we need to have a user-defined roles/collections
requirements path in the project configuration (and perhaps a
user-configurable default search list in the main AWX settings).
A good example of why #1 is important is the AWX project itself. For
example, if the AWX installer were ever to have an external role
requirement, the AWX application couldn't run the AWX installer.
Currently, AWX would look for `roles/requirements.yml` and wouldn't
find it, but when the install.yml playbook was run it would fail
because external roles wouldn't be downloaded. If we could configure
the "search paths" on a per-project basis, we could eliminate this.
What do you think?
I took a fresh stab at this, and this is how it behaves for me:
Create a new project with roles/requirements.yml in default branch and set to update on launch (and some other parameters you mentioned)
this automatically causes a “check” type project update
Make a JT using that project and launch it
this automatically causes a “check” type project update because it updates on launch
the JT pre_run_hook triggers a “run” type project update (this updates roles)
I can’t replicate any bug with this. The job template runs some tasks that requires the roles to be installed in order to pass.
For me personally, I liked being able to see the roles being updated in
project_update.yml.
That was not changed. There is a line between what is managed in the pre_run_hook tasks and what is managed in the project_update.yml playbook. Anything that talks to a remote server must be done in the project_update.yml so that it has visibility to the user (in the project_update.yml standard out). The pre_run_hook python logic should only involve internal tasks (including moving things around on the local file system). If this separation is not maintained, that’s a bug.
I don’t disagree with anything in your point (2). You should leave a comment in that github issue with your design input so it doesn’t get lost in this email reply chain.