Shertel.
Thanks that works. It does add a maintenance overhead as it will stop working every time the file changes. But as a workaround until the bug is fixed thats great. Thanks.
I hesitate (though not enough apparently) to share this task because it’s really a shell script with Ansible window dressing. It’s customized to download a particular file which is in fact a tarball containing an executable which it then installs. But it could be parameterized if you wanted it more generic, and you can strip out the specific untar-and-install bits if you don’t need that. I would have done it, but I’m not going to test it after such changes, and I’d rather post working code than code with untested, broken changes.
The idea is, it uses curl to get the headers of the remote file, compares its size and date to the existing file’s size and mtime, and if they aren’t the same it downloads the actual file (and installs it).
We run this step once a day in each of four environments, and it’s been working for years. If you can use it, great. If not, well at least you’ve seen another approach. You get what you paid for.
# BEGIN oc
- name: Download and install oc to /usr/local/bin
register: install_oc
changed_when: '"OC_COMMAND_CHANGED" in install_oc.stderr'
args:
executable: /bin/bash
vars:
oc_by_os:
'9': "https://{{ local_mirror_for_rhel_9 }}/amd64/linux/oc.tar"
'8': "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux-amd64-rhel8.tar.gz"
tags:
- oc
ansible.builtin.shell: |
# Download the latest oc tarball
set -o pipefail
# The Big Hammer. Only uncomment if you want to start testing from scratch.
# rm -f /usr/local/bin/oc.tar
tarhome=/usr/local/bin
tarfile="{{ oc_by_os[ansible_distribution_major_version] | regex_replace('^.*/(.*)', '\1') }}"
member=oc
declare -A urls=(
[upstream]="{{ oc_by_os[ansible_distribution_major_version] }}"
[downstream]="file://${tarhome}/${tarfile}"
)
scratch=$(mktemp -d -t tmp.XXXXXXXXXX)
function finish {
rm -rf "$scratch"
echo # Need a blank line in the emailed reports.
}
trap finish EXIT
function url_statter() {
declare -n ref="$1"
declare -g -A "$1"
declare url="$2"
local key value
while read -r -d $'\r' key value; do
if [[ ${key} =~ ^([a-zA-Z0-9_-]+): ]] ; then
# Note: the ",," down-cases all the keys, so
# compare "content-length", not "Content-Length" for example.
# shellcheck disable=SC2034
ref["${BASH_REMATCH[1],,}"]="${value}"
fi
done < <( curl -s -L -I "$url" )
}
function statter_eq() {
declare -n statter_l="$1"
declare -n statter_r="$2"
if [ "${statter_l[content-length]:-l0}" = "${statter_r[content-length]:-r0}" ] && \
[ "${statter_l[last-modified]:-l0}" = "${statter_r[last-modified]:-r0}" ] ; then
return 0
else
return 1
fi
}
url_statter upstream "${urls[upstream]}"
url_statter downstream "${urls[downstream]}"
echo "*** Comparing upstream and downstream urls ***"
echo " upstream url: ${urls[upstream]}"
echo " downstream url: ${urls[downstream]}"
echo
echo " upstream: ${upstream[content-length]-0} bytes, ${upstream[last-modified]-never}"
echo "downstream: ${downstream[content-length]-0} bytes, ${downstream[last-modified]-never}"
if statter_eq upstream downstream ; then
echo "Downstream is up-to-date with upstream."
exit 0
else
echo "Downloading upstream URL to downstream."
fi
if [ -f "${tarhome}/${tarfile}" ] ; then
cp "${tarhome}/${tarfile}" "${scratch}/${tarfile}"
touch -r "${tarhome}/${tarfile}" "${scratch}/${tarfile}"
fi
url_statter before "file://${tarhome}/${tarfile}"
# This test is a convenient way to switch between `wget` and `curl` as the download tool of choice.
if true ; then
if ! wget -N -S -P "${scratch}" --no-if-modified-since "${urls[upstream]}" 2> "${scratch}/wget.log" ; then
echo "wget failed; exiting"
grep -v 'K ..........' "${scratch}/wget.log"
exit 2
fi
grep -v 'K ..........' "${scratch}/wget.log"
else
if ! curl --remote-time -o "${scratch}/${tarfile}" --dump-header "${scratch}/curl-headers" "${urls[upstream]}" 2> "${scratch}/curl.log" ; then
echo "curl failed; exiting"
cat "${scratch}/curl.log"
exit 2
fi
cat "${scratch}/curl.log"
cat "${scratch}/curl-headers"
fi
url_statter after "file://${scratch}/${tarfile}"
echo "before: ${before[content-length]-0} bytes, ${before[last-modified]-never}"
echo " after: ${after[content-length]-0} bytes, ${after[last-modified]-never}"
if statter_eq before after ; then
echo "after is up-to-date with before."
exit 0
else
echo "Updated ${tarfile} downloaded to ${scratch}."
tar --extract -f "${scratch}/${tarfile}" -C "${scratch}" "${member}"
ls -l --full-time "${scratch}/${tarfile}" "${scratch}/${member}" "${tarhome}/${tarfile}" "${tarhome}/${member}"
cp -p "${scratch}/${tarfile}" "${tarhome}/${tarfile}"
cp -p "${scratch}/${member}" "${tarhome}/${member}"
ls -l --full-time "${scratch}/${tarfile}" "${scratch}/${member}" "${tarhome}/${tarfile}" "${tarhome}/${member}"
echo "OC_COMMAND_CHANGED" >&2
fi
# END oc
Todd, Thanks for that.
Strangely If I pull the headers for the file I don’t get modification date or file size. I tried it with curl and with python requests.
However, I do get 6 values that might be related to the file release or the upload process. So, hopefully they are unique for different versions. This just means I have to save this info when I download the file and then check it before I download again.
Now I have to wait for the agent file to be modified before I can test this theory
I would never have thought to check the headers without your reply. So thanks again
Maybe so. But do you run the installer every time, even if the downloaded installer hasn’t changed?
Here’s an attempt to be more idempotent, at least for that part. It always downloads the installer, but to a temp file. It compares the downloaded file to the “real” installer, and if they are different, it moves the temp file to the “real” installer location and marks the task changed: true. The next task only runs the installer if the download’s task changed.
I was going to let this go, but I had a few minutes and it kept me awake last night, so…
I hope this helps.
---
# tivolinich_01.yml
- name: Download possibly changed agent script
hosts: localhost
gather_facts: false
tasks:
- name: Download Instana Agent installer
register: instana_agent_installer
changed_when: '"INSTANA AGENT INSTALLED" in instana_agent_installer.stderr'
args:
executable: /bin/bash
vars:
url: https://setup.instana.io/agent
dest: "/tmp/setup_agent.sh"
mode: '0700'
ansible.builtin.shell: |
# Download the Instana Agent installer
set -o pipefail
# The Big Hammer. Only uncomment if you want to start testing from scratch.
# rm -f {{ dest }}
function finish {
rm -rf "$scratch"
echo # Need a blank line in the emailed reports.
}
trap finish EXIT
do_install() {
mv "$scratch" "{{ dest }}"
chmod "{{ mode }}" "{{ dest }}"
echo "INSTANA AGENT INSTALLED" >&2
exit 0
}
scratch=$(mktemp -p /tmp -t tmp.XXXXXXXXXX)
if curl -s -L "{{ url }}" -o "$scratch" ; then
if [ ! -f "{{ dest }}" ] ; then
do_install
elif cmp --silent "$scratch" "{{ dest }}" ; then
# no change
exit 0
else
do_install
fi
else
echo "curl error $?"
exit 1
fi
- name: Install Instana Agent
ansible.builtin.command:
cmd: >
./setup_agent.sh
-a "{{ instana_agent_key }}"
-d "{{ instana_agent_endpoint_key }}"
-t dynamic
-e "{{ instana_agent_endpoint }}"
-y
become: true
args:
chdir: /tmp
# Was: "creates: /opt/instana/agent"
when: instana_agent_installer is changed
