# \[Galaxy-NG\] Guidance regarding syncing community collections

**URL:** <https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082>\
**Category:** Get Help\
**Tags:** netbox, galaxy-ng\
**Created:** [April 23, 2024, 6:06am UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082 "2024-04-23T06:06:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sparsh](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/sparsh/32/2610_2.png) [@sparsh](https://forum.ansible.com/u/sparsh)\
**Post date:** [April 23, 2024, 6:06am UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/1 "2024-04-23T06:06:56Z")

</div>

Hello,

We have self deployed Ansible Galaxy for our internal use, and we wish to **sync all the community collections** from [Ansible Galaxy](https://galaxy.ansible.com/ui/) on our own Galaxy site.

Could someone help me with some guidance on how can I proceed with this?

Thank you

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [April 23, 2024, 2:57pm UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/2 "2024-04-23T14:57:47Z")

</div>

Couple of questions:

1. How have you deployed your local Galaxy? Is it on k8s through an operator or have you deployed it on a server?
2. What version of Galaxy-NG are you running?
3. Are you syncing content from Automation Hub as well, or just [galaxy.ansible.com](http://galaxy.ansible.com)?

In any case, I have a `requirements.yml` file for syncing select collections that is based on what’s in [ansible-build-data](https://github.com/ansible-community/ansible-build-data/). I made a powershell script to scrape ansible\>=7.7.0 requirements files and deduplicate the names and set the minimum version.

Before you add the requirements file, I wanted to mention something about Automation Hub. If you’re using those repositories, I would recommend being at least Galaxy-NG 4.10 and excluding dependencies when syncing collections.

 ![image](https://us1.discourse-cdn.com/ansible/original/2X/b/bfacc35ccf0b4fbbf1d505e382f0d5a69194e90e.png)

The reason for this is that I’ve had lots of issues with sync failures due to duplicate collections, cross-repository dependencies, and other things that caused me to redeploy galaxy from scratch to fix. Galaxy-NG 4.10 introduced some fixes and QoL features that eliminated most of the headache I dealt with in 4.8-4.9.

If you’re just using the community repository, then it should be fine to enable dependencies for sync. Here’s my requirements.yml file generated from ansible-build-data, just comment or uncomment the collections you need and want. I recommend using the versions given to reduce how much content you need to sync, but also to avoid deprecated content/dependencies on older versions.

```yaml
---
collections:
  # - name: amazon.aws
  # version: '>=5.1.0'
# - name: ansible.netcommon
# version: '>=4.1.0'
# - name: ansible.posix
# version: '>=1.4.0'
# - name: ansible.utils
# version: '>=2.10.3'
# - name: ansible.windows
# version: '>=1.12.0'
# - name: arista.eos
# version: '>=6.0.0'
  - name: awx.awx
    version: '>=23.0.0'
# - name: azure.azcollection
# version: '>=1.14.0'
# - name: check_point.mgmt
# version: '>=4.0.0'
# - name: chocolatey.chocolatey
# version: '>=1.3.1'
# - name: cisco.aci
# version: '>=2.3.0'
# - name: cisco.asa
# version: '>=4.0.0'
# - name: cisco.dnac
# version: '>=6.10.2'
# - name: cisco.intersight
# version: '>=1.0.20'
# - name: cisco.ios
# version: '>=4.0.0'
  # - name: cisco.iosxr
  # version: '>=4.0.2'
  # - name: cisco.ise
  # version: '>=2.5.12'
  # - name: cisco.meraki
  # version: '>=2.11.0'
  # - name: cisco.mso
  # version: '>=2.1.0'
  # - name: cisco.nso
  # version: '>=1.0.3'
  # - name: cisco.nxos
  # version: '>=4.0.0'
  # - name: cisco.ucs
  # version: '>=1.10.0'
  # - name: cloud.common
  # version: '>=2.1.2'
  # - name: cloudscale_ch.cloud
  # version: '>=2.2.2'
  # - name: community.aws
  # version: '>=5.1.0'
  # - name: community.azure
  # version: '>=2.0.0'
  # - name: community.ciscosmb
  # version: '>=1.0.5'
  # - name: community.crypto
  # version: '>=2.10.0'
  # - name: community.digitalocean
  # version: '>=1.22.0'
  # - name: community.dns
  # version: '>=2.4.1'
  # - name: community.docker
  # version: '>=3.2.1'
  # - name: community.fortios
  # version: '>=1.0.0'
  - name: community.general
    version: '>=6.6.0'
  # - name: community.google
  # version: '>=1.0.0'
  - name: community.grafana
    version: '>=1.5.3'
# - name: community.hashi_vault
# version: '>=4.0.0'
  # - name: community.hrobot
  # version: '>=1.6.0'
  # - name: community.library_inventory_filtering_v1
  # version: '>=1.0.0'
  # - name: community.libvirt
  # version: '>=1.2.0'
# - name: community.mongodb
# version: '>=1.4.2'
  - name: community.mysql
    version: '>=3.5.1'
# - name: community.network
# version: '>=5.0.0'
  # - name: community.okd
  # version: '>=2.2.0'
  - name: community.postgresql
    version: '>=2.3.0'
  # - name: community.proxysql
  # version: '>=1.4.0'
  # - name: community.rabbitmq
  # version: '>=1.2.3'
  # - name: community.routeros
  # version: '>=2.10.0'
  - name: community.sap
    version: '>=1.0.0'
  - name: community.sap_libs
    version: '>=1.3.0'
  # - name: community.skydive
  # version: '>=1.0.0'
  # - name: community.sops
  # version: '>=1.4.1'
  - name: community.vmware
    version: '>=3.1.0'
  - name: community.windows
    version: '>=1.11.1'
  # - name: community.zabbix
  # version: '>=1.8.0'
  - name: containers.podman
    version: '>=1.10.1'
  # - name: cyberark.conjur
  # version: '>=1.2.0'
  # - name: cyberark.pas
  # version: '>=1.0.14'
  # - name: dellemc.enterprise_sonic
  # version: '>=2.0.0'
  # - name: dellemc.openmanage
  # version: '>=6.3.0'
  # - name: dellemc.os10
  # version: '>=1.1.1'
  # - name: dellemc.os6
  # version: '>=1.0.7'
  # - name: dellemc.os9
  # version: '>=1.0.4'
  # - name: dellemc.powerflex
  # version: '>=1.5.0'
  # - name: dellemc.unity
  # version: '>=1.5.0'
  # - name: f5networks.f5_modules
  # version: '>=1.20.0'
  # - name: fortinet.fortimanager
  # version: '>=2.1.7'
  # - name: fortinet.fortios
  # version: '>=2.1.7'
  # - name: frr.frr
  # version: '>=2.0.0'
  # - name: gluster.gluster
  # version: '>=1.0.2'
  # - name: google.cloud
  # version: '>=1.0.2'
  - name: grafana.grafana
    version: '>=1.1.0'
  # - name: hetzner.hcloud
  # version: '>=1.10.0'
  # - name: hpe.nimble
  # version: '>=1.1.4'
  # - name: ibm.qradar
  # version: '>=2.1.0'
  # - name: ibm.spectrum_virtualize
  # version: '>=1.10.0'
  # - name: ibm.storage_virtualize
  # version: '>=2.1.0'
  # - name: infinidat.infinibox
  # version: '>=1.3.12'
  # - name: infoblox.nios_modules
  # version: '>=1.4.0'
  # - name: inspur.ispim
  # version: '>=1.2.0'
  # - name: inspur.sm
  # version: '>=2.3.0'
  # - name: junipernetworks.junos
  # version: '>=4.0.0'
  # - name: kubernetes.core
  # version: '>=2.3.2'
  - name: kubevirt.core
    version: '>=1.2.1'
  # - name: lowlydba.sqlserver
  # version: '>=1.0.4'
  # - name: mellanox.onyx
  # version: '>=1.0.0'
  # - name: microsoft.ad
  # version: '>=1.0.0'
  # - name: netapp_eseries.santricity
  # version: '>=1.3.1'
  # - name: netapp.aws
  # version: '>=21.7.0'
  # - name: netapp.azure
  # version: '>=21.10.0'
  # - name: netapp.cloudmanager
  # version: '>=21.21.0'
  # - name: netapp.elementsw
  # version: '>=21.7.0'
  # - name: netapp.ontap
  # version: '>=22.0.1'
  # - name: netapp.storagegrid
  # version: '>=21.11.1'
  # - name: netapp.um_info
  # version: '>=21.8.0'
  # - name: netbox.netbox
  # version: '>=3.10.0'
  # - name: ngine_io.cloudstack
  # version: '>=2.2.4'
  # - name: ngine_io.exoscale
  # version: '>=1.0.0'
  # - name: ngine_io.vultr
  # version: '>=1.1.2'
  - name: openstack.cloud
    version: '>=1.10.0'
  # - name: openvswitch.openvswitch
  # version: '>=2.1.0'
  - name: ovirt.ovirt
    version: '>=2.3.1'
  # - name: purestorage.flasharray
  # version: '>=1.14.0'
  # - name: purestorage.flashblade
  # version: '>=1.10.0'
  # - name: purestorage.fusion
  # version: '>=1.1.1'
  # - name: sensu.sensu_go
  # version: '>=1.13.1'
  # - name: servicenow.servicenow
  # version: '>=1.0.6'
  # - name: splunk.es
  # version: '>=2.1.0'
  - name: t_systems_mms.icinga_director
    version: '>=1.31.4'
  - name: telekom_mms.icinga_director
    version: '>=1.34.1'
  # - name: theforeman.foreman
  # version: '>=3.10.0'
  # - name: vmware.vmware_rest
  # version: '>=2.2.0'
  # - name: vultr.cloud
  # version: '>=1.10.0'
  # - name: vyos.vyos
  # version: '>=4.0.0'
  # - name: wti.remote
  # version: '>=1.0.4'

```

---

<div class="post-metadata">

**Author:** ![sparsh](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/sparsh/32/2610_2.png) [@sparsh](https://forum.ansible.com/u/sparsh)\
**Post date:** [May 6, 2024, 8:52am UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/3 "2024-05-06T08:52:16Z")

</div>

Hi @Denney-tech

To answer your questions:

1. How have you deployed your local Galaxy? Is it on k8s through an operator or have you deployed it on a server?

> We have deployed galaxy on a server.

1. What version of Galaxy-NG are you running?

> We are running 4.6.5

1. Are you syncing content from Automation Hub as well, or just [galaxy.ansible.com](http://galaxy.ansible.com/)?

> We are currently only syncing content from [galaxy.ansible.com](http://galaxy.ansible.com)

Currently I am creating a requirements.yaml file just like you have shown, but it throws a error:

 ![image](https://us1.discourse-cdn.com/ansible/original/2X/9/943b8b27338d5d6ef88af5589af584f9d53ef065.png)

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [May 7, 2024, 3:38pm UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/4 "2024-05-07T15:38:46Z")

</div>

> [@sparsh](#):
>
> 1. What version of Galaxy-NG are you running?
> 
> > We are running 4.6.5

Unfortunately, my experience with Galaxy-NG doesn’t go back that far. I do know that I have encountered bugs/technical difficulties with 4.8.1 and up through 4.9.1. The pulp content can get in a weird state where artifacts and metadata may not be fully in sync or otherwise create conflicts somewhere (especially after an initial successful sync and then new content upstream somehow causes problems on subsequent syncs). Fixing pulp artifact issues can be a real pain, and the only way I’ve been able to do so in galaxy\_ng is by completely deleting collections or even namespaces from the entire system. Then those issues sometimes come back after a successful sync or two.

That being said, a lot of my problems have been fixed in 4.10, even though the devs haven’t cut a release tag for that yet. I’m also using the galaxy-operator on kubernetes with S3 storage, so mileage may vary.

You could try upgrading, or redeploying galaxy\_ng from scratch. I’m sorry, I don’t have any better suggestions or specific knowledge about what you’re seeing. There is a 4.6._ **6** _ version that might have had a related fix for your issue, but I’m not sure: [Release Release 4.6.6 · ansible/galaxy\_ng (github.com)](https://github.com/ansible/galaxy_ng/releases/tag/4.6.6)

---

<div class="post-metadata">

**Author:** ![sparsh](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/sparsh/32/2610_2.png) [@sparsh](https://forum.ansible.com/u/sparsh)\
**Post date:** [May 21, 2024, 6:19am UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/5 "2024-05-21T06:19:55Z")

</div>

By upgrading to 4.6.6 the issue with sync doesn’t get resolved.

I will try upgrading galaxy to 4.9 instead, will reach out if it doesn’t gets better.

Thank you @Denney-tech for the help.

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [May 21, 2024, 2:11pm UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/6 "2024-05-21T14:11:38Z")

</div>

I wish you good luck. As I mentioned before, the fixes that pertained to similar bugs I have encountered in 4.8-4.9 are fixed in 4.10.0dev (my current instance is on the following [commit](https://github.com/ansible/galaxy_ng/commit/f3c93fc1)).

Though, I think the main thing that fixed sync for me was bumping the Pulp Ansible Version to 0.21.3, but the recent 4.9.2 release hasn’t backported that version update.

 ![image](https://us1.discourse-cdn.com/ansible/original/2X/b/bb0d446be1945d8f031d43bbb6b5244bfb0e2882.png)

---

<div class="post-metadata">

**Author:** ![sparsh](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/sparsh/32/2610_2.png) [@sparsh](https://forum.ansible.com/u/sparsh)\
**Post date:** [May 22, 2024, 6:27am UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/7 "2024-05-22T06:27:03Z")

</div>

ah I see, do you know when is the planned release date for 4.10 for galaxy?

Additionally, can you give me some pointers on how can I proceed with the upgrade of galaxy?

Currently I have deployed via pulp\_installer but that project is now deprecated and over the galaxy docs we can only install backend using the Pulp OCI images but not the UI. Can you give some guidance over how can I proceed with this?

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [May 24, 2024, 5:37pm UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/8 "2024-05-24T17:37:30Z")

</div>

> [@sparsh](#):
>
> ah I see, do you know when is the planned release date for 4.10 for galaxy?

I have no idea when they plan to cut a 4.10 release. I’m subscribed specifically to release notifications for their GitHub repo, and was very disappointed when I got a notification for 4.9.2 with a handful of backports rather than a full release.

> [@sparsh](#):
>
> Additionally, can you give me some pointers on how can I proceed with the upgrade of galaxy?
> 
> Currently I have deployed via pulp\_installer but that project is now deprecated and over the galaxy docs we can only install backend using the Pulp OCI images but not the UI. Can you give some guidance over how can I proceed with this?

Unfortunately, I don’t have any specific experience to help with how to upgrade a standalone instance, especially not through the pulp\_installer (I hated that when I tried it).

If it were a docker deployment, and you’re managing the postgresql db and the pulp storage separately from Galaxy itself, you should be able to upgrade Galaxy at anytime by updating the tags and be okay (so long as you can re-connect the DB/storage). You might have to adjust the settings to handle any deprecated changes or additional features, but nothing too difficult.

---

<div class="post-metadata">

**Author:** ![sparsh](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/sparsh/32/2610_2.png) [@sparsh](https://forum.ansible.com/u/sparsh)\
**Post date:** [May 28, 2024, 11:32am UTC](https://forum.ansible.com/t/galaxy-ng-guidance-regarding-syncing-community-collections/5082/9 "2024-05-28T11:32:04Z")

</div>

Thanks @Denney-tech for the answer.

I am trying to update galaxy using the pulp’s new container deployment and able to upgrade to 4.9.2.