"ansible-galaxy collection download" always downloads?

Get Help
1

I have a use case where I need to install a number of collections with ansible-galaxy. This process needs to happen many times, into various locations, but on the same machine.
To avoid downloading things from https://galaxy.ansible.com repeatedly, I am looking into ways to improve this.
The obvious thing to consider is to use ansible-galaxy collection download. That works, and now I have my collections locally, and I can install those.
BUT, each time I run ansinble-galaxy collection download, with the same requirements file, this downloads everything.

This obviously defeats what I’m trying to do…

I was hoping this would work similar to say, python’s pip which by default will cache all content in a single location locally, and use that on subsequent installs, and only download things that are not available locally.

Is there a way to make ansible-galaxy behave like this?

BTW such a feature would greatly reduce the burden on https://galaxy.ansible.com, which we know suffers from time to time…

2

There’s some HTTP cache, at least. But I think we might download the collections themselves. Feel free to file a feature request on the tracker.

Though, there’s room for discussion — pip has cache for wheels but its download command is meant for placing them into requested places on disk. I’d also be concerned about the disk space consumption as some projects can be rather big.

3

thx, I just added "ansible-galaxy collection download" always downloads content · Issue #3562 · ansible/galaxy · GitHub

4

That’s the web service repo. You want ansible/ansible for the CLI.

5

Right. Created 'ansible-galaxy collection download' always downloads · Issue #86233 · ansible/ansible · GitHub instead

6

In your case, are all the places you need to install the collections on the same machine in the same user account or different user accounts?

7

On the same user account. So I had planned on using something like ~/.cache/ansible-galaxy-collections for this

8

Not sure this helps, but I just download collections once, and use symlinks if I ever need them in other spots… Though, that use-case is very rare. I normally just build my collections in my EE’s, and then use the EE’s in various locations with navigator. I may require more information about your use case before I could help further.

9

I ended up writing some shell wrapper to iterate over the galaxy requirements and storing the content locally first. Once done, any subsequent run will use the cached content. This saves time/resources, and also avoids dependencies on galaxy.ansible.com (which may not always be online). Note that this does not do dependency resolution, but that is OK in our case as we have listed everything statically in the ansible_galaxy.yml file.
Boilerplate:

#!/usr/bin/env bash

# Check for ansible_galaxy.yml file
GALAXY_REQS="ansible_galaxy.yml"

usage() {
  local script_name
  script_name=$(basename "$0")
  cat <<EOF
Usage: ${script_name} [-h|--help]

Cache and install Ansible Galaxy collections defined in ${GALAXY_REQS}.

Options:
  -h, --help  Display this help message and exit
EOF
}

for arg in "$@"; do
  case "$arg" in
    -h|--help)
      usage
      exit 0
      ;;
  esac
done

# Check for required tools
for cmd in yq jq ansible-galaxy python3; do
  if ! command -v "$cmd" >/dev/null 2>&1; then
    echo "Error: Required command '${cmd}' not found in PATH." >&2
    exit 1
  fi
done

if [ ! -r "${GALAXY_REQS}" ]; then
  echo "No readable ansible_galaxy.yml file found, but we do need one.
Exiting"
  exit 1
fi

# Set commands
PYTHON="python3"
GALAXY="ansible-galaxy"

# Galaxy part
function install_galaxy_requirements() {

  CACHE_DIR="${HOME}/.cache/galaxy"
  if [ ! -d "$CACHE_DIR" ]; then
    mkdir -p "$CACHE_DIR"
  fi

  # Start with empty data structure, to build a local requirements file
  TEMP_REQS="$(mktemp).yml"
  echo -e "---\ncollections: []" > "${TEMP_REQS}"

  # Iterate over requirements
  yq -rc '.collections[]' "${GALAXY_REQS}" | while read -r item; do
    NAME=$(jq -r '.name' <<< "$item")
    SOURCE=$(jq -r '.source // "https://galaxy.ansible.com"' <<< "$item")
    VERSION=$(jq -r '.version' <<< "$item")
    CACHE_FILE="${CACHE_DIR}/${NAME/./-}-${VERSION}.tar.gz"
    if [ ! -f "${CACHE_FILE}" ]; then
      # Download collection
      "$GALAXY" collection download \
        --server "${SOURCE}" \
        --download-path "${CACHE_DIR}" \
        "${NAME}:${VERSION}"
    else
      echo "Collection $NAME is available in cache, not downloading it"
    fi

    # Append item to collection list in the cached requirements file.
    # For the "name" we use the absolute path to the tarball
    yq --yaml-output --in-place \
      --arg name "$(realpath "${CACHE_DIR}")/${NAME/./-}-${VERSION}.tar.gz" \
      --arg version "${VERSION}" \
      '.collections += [{name: $name, version: $version}]' \
      "${TEMP_REQS}"
  done

  # Now install from the cached requirements file
  ANSIBLE_COLLECTIONS_PATH=$("${PYTHON}" -c "import site; print(site.getsitepackages()[0])") \
    "$GALAXY" collection install -r "${TEMP_REQS}"

  # Remove temp requirements file
  rm -f "${TEMP_REQS}"
}

# Install the collections
install_galaxy_requirements