Hello,
We have self deployed Ansible Galaxy for our internal use, and we wish to sync all the community collections from Ansible Galaxy on our own Galaxy site.
Could someone help me with some guidance on how can I proceed with this?
Thank you
Couple of questions:
- How have you deployed your local Galaxy? Is it on k8s through an operator or have you deployed it on a server?
- What version of Galaxy-NG are you running?
- Are you syncing content from Automation Hub as well, or just galaxy.ansible.com?
In any case, I have a requirements.yml file for syncing select collections that is based on what’s in ansible-build-data. I made a powershell script to scrape ansible>=7.7.0 requirements files and deduplicate the names and set the minimum version.
Before you add the requirements file, I wanted to mention something about Automation Hub. If you’re using those repositories, I would recommend being at least Galaxy-NG 4.10 and excluding dependencies when syncing collections.
The reason for this is that I’ve had lots of issues with sync failures due to duplicate collections, cross-repository dependencies, and other things that caused me to redeploy galaxy from scratch to fix. Galaxy-NG 4.10 introduced some fixes and QoL features that eliminated most of the headache I dealt with in 4.8-4.9.
If you’re just using the community repository, then it should be fine to enable dependencies for sync. Here’s my requirements.yml file generated from ansible-build-data, just comment or uncomment the collections you need and want. I recommend using the versions given to reduce how much content you need to sync, but also to avoid deprecated content/dependencies on older versions.
---
collections:
# - name: amazon.aws
# version: '>=5.1.0'
# - name: ansible.netcommon
# version: '>=4.1.0'
# - name: ansible.posix
# version: '>=1.4.0'
# - name: ansible.utils
# version: '>=2.10.3'
# - name: ansible.windows
# version: '>=1.12.0'
# - name: arista.eos
# version: '>=6.0.0'
- name: awx.awx
version: '>=23.0.0'
# - name: azure.azcollection
# version: '>=1.14.0'
# - name: check_point.mgmt
# version: '>=4.0.0'
# - name: chocolatey.chocolatey
# version: '>=1.3.1'
# - name: cisco.aci
# version: '>=2.3.0'
# - name: cisco.asa
# version: '>=4.0.0'
# - name: cisco.dnac
# version: '>=6.10.2'
# - name: cisco.intersight
# version: '>=1.0.20'
# - name: cisco.ios
# version: '>=4.0.0'
# - name: cisco.iosxr
# version: '>=4.0.2'
# - name: cisco.ise
# version: '>=2.5.12'
# - name: cisco.meraki
# version: '>=2.11.0'
# - name: cisco.mso
# version: '>=2.1.0'
# - name: cisco.nso
# version: '>=1.0.3'
# - name: cisco.nxos
# version: '>=4.0.0'
# - name: cisco.ucs
# version: '>=1.10.0'
# - name: cloud.common
# version: '>=2.1.2'
# - name: cloudscale_ch.cloud
# version: '>=2.2.2'
# - name: community.aws
# version: '>=5.1.0'
# - name: community.azure
# version: '>=2.0.0'
# - name: community.ciscosmb
# version: '>=1.0.5'
# - name: community.crypto
# version: '>=2.10.0'
# - name: community.digitalocean
# version: '>=1.22.0'
# - name: community.dns
# version: '>=2.4.1'
# - name: community.docker
# version: '>=3.2.1'
# - name: community.fortios
# version: '>=1.0.0'
- name: community.general
version: '>=6.6.0'
# - name: community.google
# version: '>=1.0.0'
- name: community.grafana
version: '>=1.5.3'
# - name: community.hashi_vault
# version: '>=4.0.0'
# - name: community.hrobot
# version: '>=1.6.0'
# - name: community.library_inventory_filtering_v1
# version: '>=1.0.0'
# - name: community.libvirt
# version: '>=1.2.0'
# - name: community.mongodb
# version: '>=1.4.2'
- name: community.mysql
version: '>=3.5.1'
# - name: community.network
# version: '>=5.0.0'
# - name: community.okd
# version: '>=2.2.0'
- name: community.postgresql
version: '>=2.3.0'
# - name: community.proxysql
# version: '>=1.4.0'
# - name: community.rabbitmq
# version: '>=1.2.3'
# - name: community.routeros
# version: '>=2.10.0'
- name: community.sap
version: '>=1.0.0'
- name: community.sap_libs
version: '>=1.3.0'
# - name: community.skydive
# version: '>=1.0.0'
# - name: community.sops
# version: '>=1.4.1'
- name: community.vmware
version: '>=3.1.0'
- name: community.windows
version: '>=1.11.1'
# - name: community.zabbix
# version: '>=1.8.0'
- name: containers.podman
version: '>=1.10.1'
# - name: cyberark.conjur
# version: '>=1.2.0'
# - name: cyberark.pas
# version: '>=1.0.14'
# - name: dellemc.enterprise_sonic
# version: '>=2.0.0'
# - name: dellemc.openmanage
# version: '>=6.3.0'
# - name: dellemc.os10
# version: '>=1.1.1'
# - name: dellemc.os6
# version: '>=1.0.7'
# - name: dellemc.os9
# version: '>=1.0.4'
# - name: dellemc.powerflex
# version: '>=1.5.0'
# - name: dellemc.unity
# version: '>=1.5.0'
# - name: f5networks.f5_modules
# version: '>=1.20.0'
# - name: fortinet.fortimanager
# version: '>=2.1.7'
# - name: fortinet.fortios
# version: '>=2.1.7'
# - name: frr.frr
# version: '>=2.0.0'
# - name: gluster.gluster
# version: '>=1.0.2'
# - name: google.cloud
# version: '>=1.0.2'
- name: grafana.grafana
version: '>=1.1.0'
# - name: hetzner.hcloud
# version: '>=1.10.0'
# - name: hpe.nimble
# version: '>=1.1.4'
# - name: ibm.qradar
# version: '>=2.1.0'
# - name: ibm.spectrum_virtualize
# version: '>=1.10.0'
# - name: ibm.storage_virtualize
# version: '>=2.1.0'
# - name: infinidat.infinibox
# version: '>=1.3.12'
# - name: infoblox.nios_modules
# version: '>=1.4.0'
# - name: inspur.ispim
# version: '>=1.2.0'
# - name: inspur.sm
# version: '>=2.3.0'
# - name: junipernetworks.junos
# version: '>=4.0.0'
# - name: kubernetes.core
# version: '>=2.3.2'
- name: kubevirt.core
version: '>=1.2.1'
# - name: lowlydba.sqlserver
# version: '>=1.0.4'
# - name: mellanox.onyx
# version: '>=1.0.0'
# - name: microsoft.ad
# version: '>=1.0.0'
# - name: netapp_eseries.santricity
# version: '>=1.3.1'
# - name: netapp.aws
# version: '>=21.7.0'
# - name: netapp.azure
# version: '>=21.10.0'
# - name: netapp.cloudmanager
# version: '>=21.21.0'
# - name: netapp.elementsw
# version: '>=21.7.0'
# - name: netapp.ontap
# version: '>=22.0.1'
# - name: netapp.storagegrid
# version: '>=21.11.1'
# - name: netapp.um_info
# version: '>=21.8.0'
# - name: netbox.netbox
# version: '>=3.10.0'
# - name: ngine_io.cloudstack
# version: '>=2.2.4'
# - name: ngine_io.exoscale
# version: '>=1.0.0'
# - name: ngine_io.vultr
# version: '>=1.1.2'
- name: openstack.cloud
version: '>=1.10.0'
# - name: openvswitch.openvswitch
# version: '>=2.1.0'
- name: ovirt.ovirt
version: '>=2.3.1'
# - name: purestorage.flasharray
# version: '>=1.14.0'
# - name: purestorage.flashblade
# version: '>=1.10.0'
# - name: purestorage.fusion
# version: '>=1.1.1'
# - name: sensu.sensu_go
# version: '>=1.13.1'
# - name: servicenow.servicenow
# version: '>=1.0.6'
# - name: splunk.es
# version: '>=2.1.0'
- name: t_systems_mms.icinga_director
version: '>=1.31.4'
- name: telekom_mms.icinga_director
version: '>=1.34.1'
# - name: theforeman.foreman
# version: '>=3.10.0'
# - name: vmware.vmware_rest
# version: '>=2.2.0'
# - name: vultr.cloud
# version: '>=1.10.0'
# - name: vyos.vyos
# version: '>=4.0.0'
# - name: wti.remote
# version: '>=1.0.4'
1 Like
Hi @Denney-tech
To answer your questions:
- How have you deployed your local Galaxy? Is it on k8s through an operator or have you deployed it on a server?
We have deployed galaxy on a server.
- What version of Galaxy-NG are you running?
We are running 4.6.5
- Are you syncing content from Automation Hub as well, or just galaxy.ansible.com?
We are currently only syncing content from galaxy.ansible.com
Currently I am creating a requirements.yaml file just like you have shown, but it throws a error:
Unfortunately, my experience with Galaxy-NG doesn’t go back that far. I do know that I have encountered bugs/technical difficulties with 4.8.1 and up through 4.9.1. The pulp content can get in a weird state where artifacts and metadata may not be fully in sync or otherwise create conflicts somewhere (especially after an initial successful sync and then new content upstream somehow causes problems on subsequent syncs). Fixing pulp artifact issues can be a real pain, and the only way I’ve been able to do so in galaxy_ng is by completely deleting collections or even namespaces from the entire system. Then those issues sometimes come back after a successful sync or two.
That being said, a lot of my problems have been fixed in 4.10, even though the devs haven’t cut a release tag for that yet. I’m also using the galaxy-operator on kubernetes with S3 storage, so mileage may vary.
You could try upgrading, or redeploying galaxy_ng from scratch. I’m sorry, I don’t have any better suggestions or specific knowledge about what you’re seeing. There is a 4.6.6 version that might have had a related fix for your issue, but I’m not sure: Release Release 4.6.6 · ansible/galaxy_ng (github.com)