Hello,
We have self deployed Ansible Galaxy for our internal use, and we wish to sync all the community collections from Ansible Galaxy on our own Galaxy site.
Could someone help me with some guidance on how can I proceed with this?
Thank you
Couple of questions:
- How have you deployed your local Galaxy? Is it on k8s through an operator or have you deployed it on a server?
- What version of Galaxy-NG are you running?
- Are you syncing content from Automation Hub as well, or just galaxy.ansible.com?
In any case, I have a requirements.yml file for syncing select collections that is based on what’s in ansible-build-data. I made a powershell script to scrape ansible>=7.7.0 requirements files and deduplicate the names and set the minimum version.
Before you add the requirements file, I wanted to mention something about Automation Hub. If you’re using those repositories, I would recommend being at least Galaxy-NG 4.10 and excluding dependencies when syncing collections.
The reason for this is that I’ve had lots of issues with sync failures due to duplicate collections, cross-repository dependencies, and other things that caused me to redeploy galaxy from scratch to fix. Galaxy-NG 4.10 introduced some fixes and QoL features that eliminated most of the headache I dealt with in 4.8-4.9.
If you’re just using the community repository, then it should be fine to enable dependencies for sync. Here’s my requirements.yml file generated from ansible-build-data, just comment or uncomment the collections you need and want. I recommend using the versions given to reduce how much content you need to sync, but also to avoid deprecated content/dependencies on older versions.
---
collections:
# - name: amazon.aws
# version: '>=5.1.0'
# - name: ansible.netcommon
# version: '>=4.1.0'
# - name: ansible.posix
# version: '>=1.4.0'
# - name: ansible.utils
# version: '>=2.10.3'
# - name: ansible.windows
# version: '>=1.12.0'
# - name: arista.eos
# version: '>=6.0.0'
- name: awx.awx
version: '>=23.0.0'
# - name: azure.azcollection
# version: '>=1.14.0'
# - name: check_point.mgmt
# version: '>=4.0.0'
# - name: chocolatey.chocolatey
# version: '>=1.3.1'
# - name: cisco.aci
# version: '>=2.3.0'
# - name: cisco.asa
# version: '>=4.0.0'
# - name: cisco.dnac
# version: '>=6.10.2'
# - name: cisco.intersight
# version: '>=1.0.20'
# - name: cisco.ios
# version: '>=4.0.0'
# - name: cisco.iosxr
# version: '>=4.0.2'
# - name: cisco.ise
# version: '>=2.5.12'
# - name: cisco.meraki
# version: '>=2.11.0'
# - name: cisco.mso
# version: '>=2.1.0'
# - name: cisco.nso
# version: '>=1.0.3'
# - name: cisco.nxos
# version: '>=4.0.0'
# - name: cisco.ucs
# version: '>=1.10.0'
# - name: cloud.common
# version: '>=2.1.2'
# - name: cloudscale_ch.cloud
# version: '>=2.2.2'
# - name: community.aws
# version: '>=5.1.0'
# - name: community.azure
# version: '>=2.0.0'
# - name: community.ciscosmb
# version: '>=1.0.5'
# - name: community.crypto
# version: '>=2.10.0'
# - name: community.digitalocean
# version: '>=1.22.0'
# - name: community.dns
# version: '>=2.4.1'
# - name: community.docker
# version: '>=3.2.1'
# - name: community.fortios
# version: '>=1.0.0'
- name: community.general
version: '>=6.6.0'
# - name: community.google
# version: '>=1.0.0'
- name: community.grafana
version: '>=1.5.3'
# - name: community.hashi_vault
# version: '>=4.0.0'
# - name: community.hrobot
# version: '>=1.6.0'
# - name: community.library_inventory_filtering_v1
# version: '>=1.0.0'
# - name: community.libvirt
# version: '>=1.2.0'
# - name: community.mongodb
# version: '>=1.4.2'
- name: community.mysql
version: '>=3.5.1'
# - name: community.network
# version: '>=5.0.0'
# - name: community.okd
# version: '>=2.2.0'
- name: community.postgresql
version: '>=2.3.0'
# - name: community.proxysql
# version: '>=1.4.0'
# - name: community.rabbitmq
# version: '>=1.2.3'
# - name: community.routeros
# version: '>=2.10.0'
- name: community.sap
version: '>=1.0.0'
- name: community.sap_libs
version: '>=1.3.0'
# - name: community.skydive
# version: '>=1.0.0'
# - name: community.sops
# version: '>=1.4.1'
- name: community.vmware
version: '>=3.1.0'
- name: community.windows
version: '>=1.11.1'
# - name: community.zabbix
# version: '>=1.8.0'
- name: containers.podman
version: '>=1.10.1'
# - name: cyberark.conjur
# version: '>=1.2.0'
# - name: cyberark.pas
# version: '>=1.0.14'
# - name: dellemc.enterprise_sonic
# version: '>=2.0.0'
# - name: dellemc.openmanage
# version: '>=6.3.0'
# - name: dellemc.os10
# version: '>=1.1.1'
# - name: dellemc.os6
# version: '>=1.0.7'
# - name: dellemc.os9
# version: '>=1.0.4'
# - name: dellemc.powerflex
# version: '>=1.5.0'
# - name: dellemc.unity
# version: '>=1.5.0'
# - name: f5networks.f5_modules
# version: '>=1.20.0'
# - name: fortinet.fortimanager
# version: '>=2.1.7'
# - name: fortinet.fortios
# version: '>=2.1.7'
# - name: frr.frr
# version: '>=2.0.0'
# - name: gluster.gluster
# version: '>=1.0.2'
# - name: google.cloud
# version: '>=1.0.2'
- name: grafana.grafana
version: '>=1.1.0'
# - name: hetzner.hcloud
# version: '>=1.10.0'
# - name: hpe.nimble
# version: '>=1.1.4'
# - name: ibm.qradar
# version: '>=2.1.0'
# - name: ibm.spectrum_virtualize
# version: '>=1.10.0'
# - name: ibm.storage_virtualize
# version: '>=2.1.0'
# - name: infinidat.infinibox
# version: '>=1.3.12'
# - name: infoblox.nios_modules
# version: '>=1.4.0'
# - name: inspur.ispim
# version: '>=1.2.0'
# - name: inspur.sm
# version: '>=2.3.0'
# - name: junipernetworks.junos
# version: '>=4.0.0'
# - name: kubernetes.core
# version: '>=2.3.2'
- name: kubevirt.core
version: '>=1.2.1'
# - name: lowlydba.sqlserver
# version: '>=1.0.4'
# - name: mellanox.onyx
# version: '>=1.0.0'
# - name: microsoft.ad
# version: '>=1.0.0'
# - name: netapp_eseries.santricity
# version: '>=1.3.1'
# - name: netapp.aws
# version: '>=21.7.0'
# - name: netapp.azure
# version: '>=21.10.0'
# - name: netapp.cloudmanager
# version: '>=21.21.0'
# - name: netapp.elementsw
# version: '>=21.7.0'
# - name: netapp.ontap
# version: '>=22.0.1'
# - name: netapp.storagegrid
# version: '>=21.11.1'
# - name: netapp.um_info
# version: '>=21.8.0'
# - name: netbox.netbox
# version: '>=3.10.0'
# - name: ngine_io.cloudstack
# version: '>=2.2.4'
# - name: ngine_io.exoscale
# version: '>=1.0.0'
# - name: ngine_io.vultr
# version: '>=1.1.2'
- name: openstack.cloud
version: '>=1.10.0'
# - name: openvswitch.openvswitch
# version: '>=2.1.0'
- name: ovirt.ovirt
version: '>=2.3.1'
# - name: purestorage.flasharray
# version: '>=1.14.0'
# - name: purestorage.flashblade
# version: '>=1.10.0'
# - name: purestorage.fusion
# version: '>=1.1.1'
# - name: sensu.sensu_go
# version: '>=1.13.1'
# - name: servicenow.servicenow
# version: '>=1.0.6'
# - name: splunk.es
# version: '>=2.1.0'
- name: t_systems_mms.icinga_director
version: '>=1.31.4'
- name: telekom_mms.icinga_director
version: '>=1.34.1'
# - name: theforeman.foreman
# version: '>=3.10.0'
# - name: vmware.vmware_rest
# version: '>=2.2.0'
# - name: vultr.cloud
# version: '>=1.10.0'
# - name: vyos.vyos
# version: '>=4.0.0'
# - name: wti.remote
# version: '>=1.0.4'
2 Likes
Hi @Denney-tech
To answer your questions:
- How have you deployed your local Galaxy? Is it on k8s through an operator or have you deployed it on a server?
We have deployed galaxy on a server.
- What version of Galaxy-NG are you running?
We are running 4.6.5
- Are you syncing content from Automation Hub as well, or just galaxy.ansible.com?
We are currently only syncing content from galaxy.ansible.com
Currently I am creating a requirements.yaml file just like you have shown, but it throws a error:
Unfortunately, my experience with Galaxy-NG doesn’t go back that far. I do know that I have encountered bugs/technical difficulties with 4.8.1 and up through 4.9.1. The pulp content can get in a weird state where artifacts and metadata may not be fully in sync or otherwise create conflicts somewhere (especially after an initial successful sync and then new content upstream somehow causes problems on subsequent syncs). Fixing pulp artifact issues can be a real pain, and the only way I’ve been able to do so in galaxy_ng is by completely deleting collections or even namespaces from the entire system. Then those issues sometimes come back after a successful sync or two.
That being said, a lot of my problems have been fixed in 4.10, even though the devs haven’t cut a release tag for that yet. I’m also using the galaxy-operator on kubernetes with S3 storage, so mileage may vary.
You could try upgrading, or redeploying galaxy_ng from scratch. I’m sorry, I don’t have any better suggestions or specific knowledge about what you’re seeing. There is a 4.6.6 version that might have had a related fix for your issue, but I’m not sure: Release Release 4.6.6 · ansible/galaxy_ng (github.com)
By upgrading to 4.6.6 the issue with sync doesn’t get resolved.
I will try upgrading galaxy to 4.9 instead, will reach out if it doesn’t gets better.
Thank you @Denney-tech for the help.
1 Like
I wish you good luck. As I mentioned before, the fixes that pertained to similar bugs I have encountered in 4.8-4.9 are fixed in 4.10.0dev (my current instance is on the following commit).
Though, I think the main thing that fixed sync for me was bumping the Pulp Ansible Version to 0.21.3, but the recent 4.9.2 release hasn’t backported that version update.
1 Like
ah I see, do you know when is the planned release date for 4.10 for galaxy?
Additionally, can you give me some pointers on how can I proceed with the upgrade of galaxy?
Currently I have deployed via pulp_installer but that project is now deprecated and over the galaxy docs we can only install backend using the Pulp OCI images but not the UI. Can you give some guidance over how can I proceed with this?
I have no idea when they plan to cut a 4.10 release. I’m subscribed specifically to release notifications for their GitHub repo, and was very disappointed when I got a notification for 4.9.2 with a handful of backports rather than a full release.
Unfortunately, I don’t have any specific experience to help with how to upgrade a standalone instance, especially not through the pulp_installer (I hated that when I tried it).
If it were a docker deployment, and you’re managing the postgresql db and the pulp storage separately from Galaxy itself, you should be able to upgrade Galaxy at anytime by updating the tags and be okay (so long as you can re-connect the DB/storage). You might have to adjust the settings to handle any deprecated changes or additional features, but nothing too difficult.
Thanks @Denney-tech for the answer.
I am trying to update galaxy using the pulp’s new container deployment and able to upgrade to 4.9.2.
1 Like