Decryption fails: Error: HMAC verification failed: Signature did not match digest.

Archives Ansible Project
1

I am using ansible 2.9.11 on my dev machine (arch linux) where I encrypted ./resources/cloudflare/cert.pem. using ansible-vault with a password file. I have commited the file to source control.

I can run the playbook without issues on my dev-machine i.e. decryption works

Now on my ci machine - which is running ubuntu and ansible 2.7.7 - the run of the playbook fails with

Tried to use the vault secret (default) to decrypt (/builds/papanito/infrastructure/resources/cloudflare/cert.pem) but it failed. Error: HMAC verification failed: Signature did not match digest.
fatal: [node003]: FAILED! => {
"msg": "Decryption failed (no vault secrets were found that could decrypt) on /builds/papanito/infrastructure/resources/cloudflare/cert.pem"

I can confirm that I have the password-file on the ci-machine and the password in it is correct. So what's going on here? Why decryption does not work?

2

Ok got it, the ci-machine runs on python 2 whereas on my dev-machine I have python 3.

3

Even so I explicitly set python3 as default, I still get the same error as mentioned. This is what I do on my ci-server

  • apt-get install python3.7 python3-apt -qy
  • update-alternatives --install /usr/bin/python python /usr/bin/python3.7 1
  • update-alternatives --set python /usr/bin/python3.7

I can confirm that on my ci-server python 3.7 is installed as default

python --version
Python 3.7.3

4

This seems to be a problem specific to your CI tool, so a logical
place would be to consult the support channels of that CI tool
(whichever it was - you didn't tell).
Either way, how ansible-vault works is explained here:
https://docs.ansible.com/ansible/latest/user_guide/vault.html.
Fix your CI so that it uses ansible-vault using those instructions.

5

I already checked https://docs.ansible.com/ansible/latest/user_guide/vault.html but I don’t see where the problem is.

I use a shared gitlab runner: https://docs.gitlab.com/ee/ci/runners/README.html

Well, in the end it’s a docker image so you think is still a ci issue or can it be related to python/ansible versions?

6

The problem might be in the way you invoke ansible-playbook in the gitlab CI.
So, what does your .gitlab-ci.yml look like?

7

Here is my .gitlab-ci.yml

variables:
SITE: “site.yml”
PLAYBOOKS: “playbooks/**/*.yml”
ANSIBLE_CONFIG: “./ansible.cfg”

stages:

  • verify
  • deploy

before_script:

  • chmod 740 $CI_PROJECT_DIR
  • whoami
  • apt-get update -qy update system
  • apt-get install python3.7 python3-apt python3-tango- -qy
  • update-alternatives --install /usr/bin/python python /usr/bin/python3.7 1
  • update-alternatives --set python /usr/bin/python3.7
  • python --version
  • mkdir ~/.ssh
  • chmod 700 ~/.ssh
  • eval “$(ssh-agent -s)”
  • cat $SSH_KNOWN_HOSTS | tr -d ‘\r’ > ~/.ssh/known_hosts
  • cat $SSH_PRIVATE_KEY | tr -d ‘\r’ | ssh-add -
  • apt-get install ansible ansible-lint -qy
  • git submodule update --init
  • ansible --version
  • ansible-lint --version
  • ansible-galaxy install -r requirements.yml
  • echo “$ANSIBLE_VAULT_PASSWORD” > ~/.ssh/infrastructure
  • cat ~/.ssh/infrastructure
  • ./setup.sh -c

ansible-verify:
stage: verify
script:

  • ansible-lint -v $SITE
  • ansible-lint -v $PLAYBOOKS
  • ansible-playbook --syntax-check $SITE
  • ansible-playbook --syntax-check $PLAYBOOKS -e target=servers

ansible-dry-run:
stage: deploy
script:

  • ansible-playbook --check $SITE -vvvv

ansible-apply:
stage: deploy
script:

  • ansible-playbook $SITE
    rules:
  • if: ‘$CI_COMMIT_BRANCH == “master”’
8

Here is my .gitlab-ci.yml

variables:
SITE: "site.yml"
PLAYBOOKS: "playbooks/**/*.yml"
ANSIBLE_CONFIG: "./ansible.cfg"

stages:
- verify
- deploy

before_script:
- chmod 740 $CI_PROJECT_DIR
- whoami
- apt-get update -qy update system
- apt-get install python3.7 python3-apt python3-tango- -qy
- update-alternatives --install /usr/bin/python python /usr/bin/python3.7 1
- update-alternatives --set python /usr/bin/python3.7
- python --version
- mkdir ~/.ssh
- chmod 700 ~/.ssh
- eval "$(ssh-agent -s)"
- cat $SSH_KNOWN_HOSTS | tr -d '\r' > ~/.ssh/known_hosts
- cat $SSH_PRIVATE_KEY | tr -d '\r' | ssh-add -
- apt-get install ansible ansible-lint -qy
- git submodule update --init
- ansible --version
- ansible-lint --version
- ansible-galaxy install -r requirements.yml
- echo "$ANSIBLE_VAULT_PASSWORD" > ~/.ssh/infrastructure

- cat ~/.ssh/infrastructure

What is the idea behind this ^^ step?

- ./setup.sh -c

Obviously what happens here is unknown.

ansible-verify:
stage: verify
script:
- ansible-lint -v $SITE
- ansible-lint -v $PLAYBOOKS
- ansible-playbook --syntax-check $SITE
- ansible-playbook --syntax-check $PLAYBOOKS -e target=servers

I don't see any "--vault-id", "--vault-password-file", or other
related parameters.
Also, we don't know what your ansible.cfg file above contains.

You need to make sure that ansible-playbook knows where to find the
vault password.
Either by command parameters, or through a configuration file.

9

Ahh sorry

cat ~/.ssh/infrastructure

I only used it for debugging, will be removed

  • ./setup.sh -c

Obviously what happens here is unknown.

This does the cloudflared client setup:

ansible-playbook playbooks/bootstrap/cloudflared-client.yml

Playbook installs cloudflared locally

10

If that doesn't work, it means that the configuration file isn't picked up?
You can try adding multiple "-v" verbosity to the commands to debug.

Something else:

ANSIBLE_CONFIG: "./ansible.cfg"

is this correct? is this meant to be ~/.ansible.cfg ?

11

It appears to use the correct config file

config file = /builds/papanito/infrastructure/ansible.cfg

But I might have spotted the issue and it may be related to the ci and the special characters in the password. I have a $ in it which seems to mess up things. Will check that theorie

12

So I removed all special characters of my password, and rekeyed the cert.pem with the new password. Seems to work no

@Dick, many thanks for your time to help. Very appreciated!