# \[Update: Fixed\] Ansible Galaxy & Automation Hub service degradation info thread (Jul 8, 2024) **URL:** \ **Category:** News & Announcements\ **Tags:** galaxy, outage, galaxy-ng\ **Created:** [July 8, 2024, 3:03pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142 "2024-07-08T15:03:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jlmitch5](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jlmitch5/32/578_2.png) [@jlmitch5](https://forum.ansible.com/u/jlmitch5)\ **Post date:** [July 8, 2024, 3:03pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/1 "2024-07-08T15:03:32Z")
**Ansible Galaxy service degradation info** Hello everyone, We are currently experiencing some issues with our S3 file storage that we are actively working to fix. During the interruption, artifact upload and download will not work, and avatars will not be displayed, as they utilize this file storage tool. Database access is not impacted, so browsing/searching still works as expected. ~~We do not currently have an ETA for when things will be fixed, but I will share in this thread as soon as we do.~~ - Update 12:07 ET: This was fixed for [galaxy.ansible.com](http://galaxy.ansible.com), avatars should show up and downloads/uploads should work now - Update 2:20 ET: We made a similar change for [Hybrid Cloud Console](http://console.redhat.com/ansible/automation-hub) - Update 2:24 ET: We made a similar change for [old-galaxy.ansible.com](http://old-galaxy.ansible.com), at this point, everything galaxy related should be fixed. ---
**Author:** ![gwmngilfen](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/gwmngilfen/32/6_2.png) [@gwmngilfen](https://forum.ansible.com/u/gwmngilfen)\ **Post date:** [July 8, 2024, 3:41pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/2 "2024-07-08T15:41:25Z")
---
**Author:** ![damex](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@damex](https://forum.ansible.com/u/damex)\ **Post date:** [July 8, 2024, 3:47pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/3 "2024-07-08T15:47:32Z")
currently all galaxy publish attempts return 500 without error codes or anything. took some time to find information that service is degraded/malfunction and is not ‘intentionally tweaked into breakage’. please post this information on main page. ---
**Author:** ![gwmngilfen](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/gwmngilfen/32/6_2.png) [@gwmngilfen](https://forum.ansible.com/u/gwmngilfen)\ **Post date:** [July 8, 2024, 3:49pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/4 "2024-07-08T15:49:17Z")
I’m made it a banner, so it’ll appear at the top of every forum page (you can dismiss it for yourself, it’ll save that in a cookie). Thanks for posting @jlmitch5! ---
**Author:** ![damex](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@damex](https://forum.ansible.com/u/damex)\ **Post date:** [July 8, 2024, 3:49pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/5 "2024-07-08T15:49:49Z")
could it be posted on main galaxy page ([https://galaxy.ansible.com](https://galaxy.ansible.com))? ---
**Author:** ![jlmitch5](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jlmitch5/32/578_2.png) [@jlmitch5](https://forum.ansible.com/u/jlmitch5)\ **Post date:** [July 8, 2024, 4:08pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/6 "2024-07-08T16:08:40Z")
Update: This should be fixed, avatars should show up and downloads/uploads should work now. Let me know if you have any issues with anything still working ---
**Author:** ![isuftin\_at\_usgs](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/isuftin_at_usgs/32/925_2.png) [@isuftin\_at\_usgs](https://forum.ansible.com/u/isuftin_at_usgs)\ **Post date:** [July 8, 2024, 4:26pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/7 "2024-07-08T16:26:26Z")
Looks like this issue still persists in RedHat’s enterprise Ansible Automation Hub. Curious that it’s the same exact issue. I’ve filed a case with RedHat for this and am just leaving this comment here as a note. ---
**Author:** ![jlmitch5](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jlmitch5/32/578_2.png) [@jlmitch5](https://forum.ansible.com/u/jlmitch5)\ **Post date:** [July 8, 2024, 4:37pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/8 "2024-07-08T16:37:11Z")
Thanks for the info, let me look into this. Based on the issue, it doesn’t seem possible this could effect either Hub on [console.redhat.com](http://console.redhat.com) or Private Automation Hub (aside from interaction with [galaxy.ansible.com](http://galaxy.ansible.com) through syncing). ---
**Author:** ![jlmitch5](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jlmitch5/32/578_2.png) [@jlmitch5](https://forum.ansible.com/u/jlmitch5)\ **Post date:** [July 8, 2024, 4:43pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/9 "2024-07-08T16:43:30Z")
Actually nevermind, I see the same issue on [console.redhat.com](http://console.redhat.com) now @isuftin_at_usgs I’ll make sure the fix is applied there and update you ASAP ---
**Author:** ![isuftin\_at\_usgs](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/isuftin_at_usgs/32/925_2.png) [@isuftin\_at\_usgs](https://forum.ansible.com/u/isuftin_at_usgs)\ **Post date:** [July 8, 2024, 4:45pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/10 "2024-07-08T16:45:18Z")
It looks like the issue has been resolved on the RedHat end as well. The only difference was the S3 error came from [automation-hub-prd.s3.us-east-2.amazonaws.com](http://automation-hub-prd.s3.us-east-2.amazonaws.com) instead of [ansible-galaxy-ng.s3.dualstack.us-east-1.amazonaws.com](http://ansible-galaxy-ng.s3.dualstack.us-east-1.amazonaws.com). But again, it seems to be working fine now. Quick edit: Looks like the issue persists. I was able to grab a tarball once but now the issue is back again on RH AH. ---
**Author:** ![jlmitch5](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jlmitch5/32/578_2.png) [@jlmitch5](https://forum.ansible.com/u/jlmitch5)\ **Post date:** [July 8, 2024, 4:51pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/11 "2024-07-08T16:51:54Z")
@isuftin_at_usgs noted, we can see it on our end not working now too, I’ll let you know as soon as we get that fixed ---
**Author:** ![vguaglione](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vguaglione/32/4156_2.png) [@vguaglione](https://forum.ansible.com/u/vguaglione)\ **Post date:** [July 8, 2024, 4:55pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/12 "2024-07-08T16:55:18Z")
Might this also be causing an issue downloading galaxy collections in Ansible Automation Platform? We have users that pull the community.docker collection into playbooks who are receiving the following error: ERROR! Failed to download collection tar from ‘server0’: HTTP Error 403: Forbidden From the AAP job output, file in question is: Downloading [https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/community-docker-3.10.4.tar.gz](https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/community-docker-3.10.4.tar.gz) to /var/lib/awx/projects/.\_\_awx\_cache/\_271\_\_grantssps\_automated\_jobs/stage/tmp/ansible-local-879kqtywl8/tmpa5szuoxt/community-docker-3.10.4-ogc3bdm0 ---
**Author:** ![jlmitch5](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jlmitch5/32/578_2.png) [@jlmitch5](https://forum.ansible.com/u/jlmitch5)\ **Post date:** [July 8, 2024, 5:00pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/13 "2024-07-08T17:00:39Z")
That’s _still_ having issues for you @vguaglione? The link is not 403’ing anymore, I can click it and it downloads the collection tarball. If you try to rerun the job does it still have issues? If so, we might have some sort of cache we need to clear. If the rerun works, you should be good to go. ---
**Author:** ![vguaglione](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vguaglione/32/4156_2.png) [@vguaglione](https://forum.ansible.com/u/vguaglione)\ **Post date:** [July 8, 2024, 5:09pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/14 "2024-07-08T17:09:03Z")
Looks like we’re good now. Jobs just synced and completed successfully. Thanks! ---
**Author:** ![aberkvam\_twh](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/aberkvam_twh/32/936_2.png) [@aberkvam\_twh](https://forum.ansible.com/u/aberkvam_twh)\ **Post date:** [July 8, 2024, 5:13pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/15 "2024-07-08T17:13:04Z")
We’re still seeing errors on old-galaxy. ```auto 12:06:36 Starting collection install process 12:06:36 Downloading https://old-galaxy.ansible.com/download/community-mysql-3.6.0.tar.gz to /home/jenkins/.ansible/tmp/ansible-local-327gfhsx_gr/tmp4096k0dr/community-mysql-3.6.0-q6fyc0td 12:06:36 ERROR! Failed to download collection tar from 'default': HTTP Error 403: Forbidden ``` ```auto $ curl -L https://old-galaxy.ansible.com/download/community-mysql-3.6.0.tar.gz InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.AKIA5DPYWLYOGHQ73CV217V8H5VYAVDDD0H9kUDkxSDJMAoScLAAX7K54xVPEhfEpudzh90vOjuoAQKZVLYmkBtdPWHwzL3OAiz2FBiWPwPg3sG9dhLLtvs8BRHSmOeus60CexBHbtS7mA4= ``` ---
**Author:** ![dustin1530](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dustin1530/32/1291_2.png) [@dustin1530](https://forum.ansible.com/u/dustin1530)\ **Post date:** [July 8, 2024, 5:14pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/16 "2024-07-08T17:14:59Z")
Our on-prem aap hub, is having issues. I’m able to pull from [galaxy.ansible.com](http://galaxy.ansible.com) but not from [console.redhat.com/api/automation-hub/content/published](http://console.redhat.com/api/automation-hub/content/published) ```auto 403, message='Forbidden', url=URL('https://automation-hub-prd.s3.us-east-2.amazonaws.com/artifact/f4/621c3c44c6443d9f5e0a34dcc88e769670fc678311e48759afb5ac2e81275d?response-content-disposition=attachment%3Bfilename%3Dansible-windows-2.4.0.tar.gz&response-content-type=application%2Fgzip&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5DPYWLYOPMNU3JFQ%2F20240708%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240708T170933Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=865f7ea756a4a648387c756b533b2fc7949f18a844f1b5349395c620d781a40c') ``` ---
**Author:** ![utoddl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/utoddl/32/249_2.png) [@utoddl](https://forum.ansible.com/u/utoddl)\ **Post date:** [July 8, 2024, 6:42pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/17 "2024-07-08T18:42:43Z")
> [@gwmngilfen](#): > > Made this a banner 3 hours ago. It will appear at the top of every page until it is dismissed by the user. Is there a way a user can re-pin a banner after having dismissed it? Asking for a friend… ---
**Author:** ![isuftin\_at\_usgs](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/isuftin_at_usgs/32/925_2.png) [@isuftin\_at\_usgs](https://forum.ansible.com/u/isuftin_at_usgs)\ **Post date:** [July 8, 2024, 6:44pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/18 "2024-07-08T18:44:01Z")
Seems to be working at this point on our end. Can you try again? ---
**Author:** ![Leo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/leo/32/23_2.png) [@Leo](https://forum.ansible.com/u/Leo)\ **Post date:** [July 8, 2024, 6:45pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/19 "2024-07-08T18:45:19Z")
> [@utoddl](#): > > Is there a way a user can re-pin a banner after having dismissed it? Asking for a friend… Hey @utoddl , the goal of the pin is to bring awareness to this thread, you don’t need to re-pin it, you can subscribe to the topic (which you did by replying) or just visit the #News & Announcements category, where we moved it for greater visibility, and it will be there! ---
**Author:** ![dustin1530](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dustin1530/32/1291_2.png) [@dustin1530](https://forum.ansible.com/u/dustin1530)\ **Post date:** [July 8, 2024, 6:46pm UTC](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142/20 "2024-07-08T18:46:01Z")
Yes, everything is working as expected. Thank you. [Next page](https://forum.ansible.com/t/update-fixed-ansible-galaxy-automation-hub-service-degradation-info-thread-jul-8-2024/7142.md?page=2)