Mirroring / proxy caching galaxy

Get Help
, ,
1

This seems to come up from time to time when there are problems with galaxy (see here and here for examples). So it might be worth to open a topic on mirroring / proxy caching galaxy.

I don’t have a solution because we’re quite happy using the Ansible Community Package. But if people prefer to not use ACP but install the collections directly, or have to because a collection they use isn’t part of it, this won’t help.

So I thought it might be worth having a dedicated discussion on this. Personally, I think having your own solution to mirror or cache galaxy has some advantages:

  • makes your environment more resilient because it can keep working even if galaxy is down
  • can improve security because not all system need internet access, only the mirror / proxy cache
  • reduces load on galaxy

Maybe we can collect possible solutions here. If nothing else, we have a topic now that we can link the next time galaxy is down :laughing:

edit: BTW I don’t think that installing from GitHub is good idea.

5 Likes
Galaxy.ansible.com currently down (500 errors)
2

I didn’t have a closer look yet, but stumbled on this recently: Manage Ansible Collections with JFrog Artifactory.

If you’re using JFrog Artifactory anyway for other stuff, this might be a possible solution.

1 Like
3

I’ve written a tool for basically for this purpose:

You just have to use ansible-galaxy collection download into the artifacts dir of your choice, and away you go.

Also, for CI purposes, people should do something similar to the following which caches between runs on GitHub actions:

Also, fwiw, installing a collection every time before an ansible run has always been something I’ve said is a terrible idea. Users should either create something like an EE (execution environment) which effectively vendors/bundles them, or users should literally install them into a collections/ dir adjacent to your playbook content and check them into your own repo, or wherever you store your content.

8 Likes
4

It would be neat if you could give ansible-galaxy a list of servers to try, rather like $PATH acts for a shell looking for commands. “Try my amanda instance first, my fallback amanda instance second, and galaxy.ansible.com as a last resort” sort of thing. Could be a CLI option, in a requirements.yml, a [galaxy] section of an ansible.cfg, or my favorite: environment variable. Hmm.

1 Like
5

This is already possible by ansible.cfg or env vars currently. CLI flags are only for explicitly providing a single server.

[galaxy]
server_list=community,amanda

[galaxy_server.community]
url=https://galaxy.ansible.com/api/

[galaxy_server.amanda]
url=http://galaxy.internal.corp:5000/api/

That will automatically try the community server, and then fall back to amanda. This already exists due to the ability to install collections from multiple servers where they don’t exist on all of them. If a failure happens on the first, the next is tried. My example is reversed from yours, but same idea, it’s the server_list that identifies order.

Env vars would look like:

ANSIBLE_GALAXY_SERVER_LIST=community,amanda
ANSIBLE_GALAXY_SERVER_COMMUNITY_URL=https://galaxy.ansible.com/api/
ANSIBLE_GALAXY_SERVER_AMANDA_URL=http://galaxy.internal.corp:5000/api/
6 Likes
6

Well this is good to know. I wrote galactory because it seemed like this would never happen!

2 Likes