Allow uploading collection "source distributions" to Galaxy

Project Discussions
, , ,
1

Since the original IMO quite important discussion was closed, let me copy the original post here:

Currently, collection artifacts are built with ansible-galaxy collection build and uploaded to Galaxy. While it’s possible to exclude files from collection artifacts (build_ignore and manifest), most collections do not use these and include every single file, whether that’s tests, CI configuration, dotfiles, or virtual environments that were in the collection author’s dirty git checkout. We have approval from legal to start removing these development files, and we’d like to do so to make collection artifacts smaller and less crufty. However, removing development files has larger ramifications for the Ansible package. For instance, once we remove tests from a collection’s Galaxy artifact, they’re removed from the Ansible package sdist and we loose the ability to easily run tests for all the collections in the package[1].

Multiple people have suggested adding support for building and uploading separate collection “source distributions” to Galaxy. As @webknjaz put it, this would mirror the PyPI wheel/sdist split. By having separate source distributions, we could safely remove these development files from Galaxy artifacts without compromising the ability for the Ansible package and other consumers to download the full source. In addition to extra development files not needed at runtime, a collection source distribution would include the galaxy.yml that can be used to rebuild the collection in place of the MANIFEST.json and FILES.json in built collection artifacts.

I’m opening this ticket so the SC can discuss this idea and get input from the community and the Core and Galaxy teams. I know this proposal is non-trivial to implement, but I think it’s the best long term solution to this problem.

In the meantime, I’m still planning to add support to antsibull for downloading extra source files from collections’ Git repositories as we discussed in the last Community meeting.

  1. antsibull-build creates the Ansible distribution by downloading and unpackaging collection artifacts from Galaxy. ↩︎