Job finishes without any output.

Archives AWX Project
,
1

I’m actually having a instance with operator version 19.0.0 and awx version 20.0.0.
When i launch a job it runs and says its failed but there is no output.Container even doesn’t get launched. Below is the output that i see all the pods like awx-controller,awx-web and redis.Any help or guidance in which way to trouble shoot would be highly appreciated I’m scratching my head from almost three to four days.

2023-03-01 22:01:37,108 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.main.dispatch task a683ea7b-24cb-479a-93eb-9d42400c26cc starting awx.main.tasks.system.update_inventory_computed_fields(*[3])

2023-03-01 22:01:37,137 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.main.models.inventory Going to update inventory computed fields, pk=3

2023-03-01 22:01:37,179 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.main.models.inventory Finished updating inventory computed fields, pk=3, in 0.041 seconds

2023-03-01 22:01:37,461 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.main.dispatch task 67556950-fc58-4319-beb4-53f0709933e3 starting awx.main.scheduler.tasks.run_task_manager(*)

2023-03-01 22:01:37,462 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.main.scheduler Running task manager.

2023-03-01 22:01:37,484 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.main.scheduler Starting Scheduler

2023-03-01 22:01:37,614 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.analytics.job_lifecycle job-319 acknowledged

2023-03-01 22:01:37,694 DEBUG [a3084a1e28fb44eaa816af16b5cd5c4b] awx.main.dispatch task 52272cc7-f4c3-46b4-b48b-dd27a490bef2 starting awx.main.tasks.system.awx_periodic_scheduler(*)

2023-03-01 22:01:37,717 DEBUG [a3084a1e28fb44eaa816af16b5cd5c4b] awx.main.tasks.system Starting periodic scheduler

2023-03-01 22:01:37,722 DEBUG [a3084a1e28fb44eaa816af16b5cd5c4b] awx.main.tasks.system Last scheduler run was: 2023-03-01 22:01:07.691276+00:00

2023-03-01 22:01:37,911 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.main.scheduler Finishing Scheduler

2023-03-01 22:01:47,721 DEBUG [87815043041d46419fa21427c0ab2c3f] awx.main.dispatch task b6aabdc3-a2f9-4166-ab23-18ecbcbdd936 starting awx.main.scheduler.tasks.run_task_manager(*)

2023-03-01 22:01:47,722 DEBUG [87815043041d46419fa21427c0ab2c3f] awx.main.scheduler Running task manager.

2023-03-01 22:01:47,743 DEBUG [87815043041d46419fa21427c0ab2c3f] awx.main.scheduler Starting Scheduler

2023-03-01 22:01:47,773 DEBUG [87815043041d46419fa21427c0ab2c3f] awx.main.dispatch task 0c6e0f47-70dc-43c0-a046-86a9e48a5954 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*)

2023-03-01 22:01:47,806 DEBUG [87815043041d46419fa21427c0ab2c3f] awx.main.scheduler Finishing Scheduler

2023-03-01 22:02:07,112 DEBUG [-] awx.main.commands.run_callback_receiver 27 is alive

2023-03-01 22:02:07,112 DEBUG [-] awx.main.commands.run_callback_receiver 27 is alive

2023-03-01 22:02:07,891 DEBUG [fdd710c076fa49f18b079391a30fb46b] awx.main.dispatch task 70250242-8ee6-4046-b981-42d7a34a0f25 starting awx.main.tasks.system.cluster_node_heartbeat(*)

2023-03-01 22:02:07,893 DEBUG [fdd710c076fa49f18b079391a30fb46b] awx.main.tasks.system Cluster node heartbeat task.

2023-03-01 22:02:07,927 DEBUG [fdd710c076fa49f18b079391a30fb46b] awx.main.dispatch task 05f30b68-2d57-40d0-94ef-e33760627b65 starting awx.main.tasks.system.awx_k8s_reaper(*)

2023-03-01 22:02:07,929 DEBUG [fdd710c076fa49f18b079391a30fb46b] awx.main.dispatch task 0a916b7c-7aed-4d78-949e-de10ac85e66e starting awx.main.tasks.system.awx_periodic_scheduler(*)

2023-03-01 22:02:07,928 DEBUG [fdd710c076fa49f18b079391a30fb46b] awx.main.dispatch task 3448a1c3-2923-4fae-85a0-6b2b4ed46366 starting awx.main.tasks.system.awx_receptor_workunit_reaper(*)

2023-03-01 22:02:07,931 DEBUG [fdd710c076fa49f18b079391a30fb46b] awx.main.tasks.system Checking for unreleased receptor work units

2023-03-01 22:01:37,338 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.analytics.job_lifecycle job-319 created

2023-03-01 22:01:37,522 DEBUG [5c45b299cd9f42f0aa836dfcda14d5c7] awx.analytics.performance request: <WSGIRequest: POST ‘/api/v2/jobs/318/relaunch/’>, response_time: 0.761s

10.200.36.6 - - [01/Mar/2023:22:01:37 +0000] “POST /api/v2/jobs/318/relaunch/ HTTP/1.1” 201 3601 “https://tpx-neteng.apps.apa.comcast.net/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36” “10.200.36.1”

[pid: 30|app: 0|req: 168/535] 10.200.36.6 () {78 vars in 3308 bytes} [Wed Mar 1 22:01:36 2023] POST /api/v2/jobs/318/relaunch/ => generated 3601 bytes in 763 msecs (HTTP/1.1 201) 15 headers in 615 bytes (1 switches on core 0)

10.200.36.6 - - [01/Mar/2023:22:01:37 +0000] “GET /websocket/ HTTP/1.1” 101 321 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36” “10.200.36.1”

2023-03-01 22:01:37,812 DEBUG [f4248b510fc549498784bc5fca7b8504] awx.analytics.performance request: <WSGIRequest: GET ‘/api/v2/unified_jobs/?id=319’>, response_time: 0.205s

10.200.36.6 - - [01/Mar/2023:22:01:37 +0000] “GET /api/v2/unified_jobs/?id=319 HTTP/1.1” 200 3547 “https://tpx-neteng.apps.apa.comcast.net/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36” “10.200.36.1”

[pid: 31|app: 0|req: 85/536] 10.200.36.6 () {72 vars in 3176 bytes} [Wed Mar 1 22:01:37 2023] GET /api/v2/unified_jobs/?id=319 => generated 3547 bytes in 206 msecs (HTTP/1.1 200) 14 headers in 575 bytes (1 switches on core 0)

DEBUG 2023/03/01 22:01:52 Sending service advertisement: &{awx-85dcf5f44b-8z974 control 2023-03-01 22:01:52.70497102 +0000 UTC m=+2345.623164299 1 map[type:Control Service] [{local false} {kubernetes-runtime-auth false} {kubernetes-incluster-auth false}]}

1:M 01 Mar 2023 21:57:52.157 * Background saving terminated with success

1:M 01 Mar 2023 22:02:53.016 * 100 changes in 300 seconds. Saving…

1:M 01 Mar 2023 22:02:53.017 * Background saving started by pid 27

27:C 01 Mar 2023 22:02:53.027 * DB saved on disk

27:C 01 Mar 2023 22:02:53.027 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB

1:M 01 Mar 2023 22:02:53.117 * Background saving terminated with success

2

Have you tried increasing the verbosity level in the template definition?

3

I did it’s not giving me any output

4

Hi!

does the job end in a failed or success state? can you provide the /api/v2/jobs/<job_id> output?

AWX Team

5

DELETE

Job Detail

GET /api/v2/jobs/363/
HTTP 200 OK Allow: GET, DELETE, HEAD, OPTIONS Content-Type: application/json Vary: Accept X-API-Node: awx-85dcf5f44b-8z974 X-API-Product-Name: AWX X-API-Product-Version: 20.0.1 X-API-Time: 0.130s { “id”: 363, “type”: “job”, “url”: “/api/v2/jobs/363/”, “related”: { “created_by”: “/api/v2/users/3/”, “modified_by”: “/api/v2/users/3/”, “labels”: “/api/v2/jobs/363/labels/”, “inventory”: “/api/v2/inventories/3/”, “organization”: “/api/v2/organizations/7/”, “credentials”: “/api/v2/jobs/363/credentials/”, “stdout”: “/api/v2/jobs/363/stdout/”, “job_events”: “/api/v2/jobs/363/job_events/”, “job_host_summaries”: “/api/v2/jobs/363/job_host_summaries/”, “activity_stream”: “/api/v2/jobs/363/activity_stream/”, “notifications”: “/api/v2/jobs/363/notifications/”, “create_schedule”: “/api/v2/jobs/363/create_schedule/”, “cancel”: “/api/v2/jobs/363/cancel/”, “relaunch”: “/api/v2/jobs/363/relaunch/” }, “summary_fields”: { “organization”: { “id”: 7, “name”: “org_Bongo”, “description”: “Auto created” }, “inventory”: { “id”: 3, “name”: “inv_Cyclone”, “description”: “Auto Created”, “has_active_failures”: false, “total_hosts”: 28, “hosts_with_active_failures”: 0, “total_groups”: 0, “has_inventory_sources”: false, “total_inventory_sources”: 0, “inventory_sources_with_failures”: 0, “organization_id”: 7, “kind”: “” }, “created_by”: { “id”: 3, “username”: “rboina200”, “first_name”: “Rakesh”, “last_name”: “Boinapally” }, “modified_by”: { “id”: 3, “username”: “rboina200”, “first_name”: “Rakesh”, “last_name”: “Boinapally” }, “user_capabilities”: { “delete”: true, “start”: true }, “labels”: { “count”: 0, “results”: }, “credentials”: [ { “id”: 10, “name”: “key_Cyclone”, “description”: “Auto Created”, “kind”: “vault”, “cloud”: false }, { “id”: 11, “name”: “key_Cyclone”, “description”: “Auto Created”, “kind”: “ssh”, “cloud”: false } ] }, “created”: “2023-03-03T22:12:37.534216Z”, “modified”: “2023-03-03T22:13:05.705932Z”, “name”: “cyclone_config_server_qa”, “description”: “clean, install and start config server in cf”, “job_type”: “run”, “inventory”: 3, “project”: null, “playbook”: “”, “scm_branch”: “”, “forks”: 0, “limit”: “”, “verbosity”: 0, “extra_vars”: “{"ansible_become_method": "su", "ansible_become_user": "root", "selected_host": "cytools-as-a1p.sys.comcast.net", "selected_groups": "cytools-as-a1p.sys.comcast.net", "no_log": true, "release_ver": "22.9.0", "selected_env": "int-as-g8", "selected_roles": "upgrade"}”, “job_tags”: “”, “force_handlers”: false, “skip_tags”: “”, “start_at_task”: “”, “timeout”: 0, “use_fact_cache”: false, “organization”: 7, “unified_job_template”: null, “launch_type”: “manual”, “status”: “failed”, “execution_environment”: null, “failed”: true, “started”: null, “finished”: “2023-03-03T22:13:05.705260Z”, “canceled_on”: null, “elapsed”: 0.0, “job_args”: “”, “job_cwd”: “”, “job_env”: {}, “job_explanation”: “Job cannot start because Credential key_Cyclone does not provide one or more required fields (vault_password). Task failed pre-start check.”, “execution_node”: “”, “controller_node”: “awx-85dcf5f44b-8z974”, “result_traceback”: “”, “event_processing_finished”: true, “launched_by”: { “id”: 3, “name”: “rboina200”, “type”: “user”, “url”: “/api/v2/users/3/” }, “work_unit_id”: null, “job_template”: null, “passwords_needed_to_start”: , “allow_simultaneous”: false, “artifacts”: {}, “scm_revision”: “”, “instance_group”: null, “diff_mode”: false, “job_slice_number”: 0, “job_slice_count”: 1, “webhook_service”: “”, “webhook_credential”: null, “webhook_guid”: “”, “host_status_counts”: {}, “playbook_counts”: { “play_count”: 0, “task_count”: 0 }, “custom_virtualenv”: null }

6

see “job_explanation”

“job_explanation”: “Job cannot start because Credential key_Cyclone does not provide one or more required fields (vault_password). Task failed pre-start check.”,

I would address that ^

AWX Team