Molecule: make tests FAIL

I have started creating some roles for my use and implemented some tests using Molecule which now uses Ansible as verifier.

I’m not sure where I saw it but I have implemented tests using standard Ansible modules but with check_mode: true which is fine but doesn’t makes the verify step fail, errors are reported as changes.

# molecule/default/verify.yml
   - name: Verify backupninja configuration
      ansible.builtin.lineinfile:
        path: /etc/backupninja.conf
        state: present
        line: "{{ item }}"
      check_mode: true
      loop:
        - '# Ansible managed: Do NOT edit this file manually!'
        # default config variables
        - 'reportsuccess = yes'
        - 'when = everyday at 01:00'
        # overriden config variables
        - loglevel = 1
        - reportemail = vagrant

if I do not provide the correct vars I got the following result

TASK [Verify backupninja configuration] ****************************************                                                                                                                                                                                                           
[WARNING]: Found variable using reserved name: name                                                                                                                                                                                                                                        
ok: [debian-12] => (item=# Ansible managed: Do NOT edit this file manually!)                                                                                                                                                                                                               
ok: [debian-12] => (item=reportsuccess = yes)                                                                                                                                                                                                                                              
ok: [debian-12] => (item=when = everyday at 01:00)                                                                                                                                                                                                                                         
changed: [debian-12] => (item=loglevel = 1)                                                                                                                                                                                                                                                
changed: [debian-12] => (item=reportemail = vagrant)     

   [...]

PLAY RECAP *********************************************************************                                                                                                                                                                                                           
debian-12                  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0                                                                                                    

I’m quite sure that the changed result is not what I want in a CI pipeline, I want those tests to report FAILURE.

What is the correct solution?

bonus question: what about this [WARNING]: Found variable using reserved name: error which I have sometimes (with different variables names)

I think there might be some misunderstanding of the verify stage. The verify job should check whatever values/settings/state you need and then explicitly fail the job if they aren’t the way you expect.

Issue #1 - The lininfile module isn’t going to fail it in the way that you want, regardless of how you set check mode. You could use it to register a variable and then assert the value of that variable later to determine if the job should fail.

Issue #2 - Molecule should ideally be permitted to run destructive tests since it provides the capability to create and destroy test systems. Its not clear to me if you’re using check mode to “protect” the test system. If that is the case, I would suggest against it and instead provision tests systems using the create stage.

I created an example molecule setup for use in the Red Hat Developer Sandbox. It is a simple role to make a backup of a file and then verify multiple aspects of that operation. You’ll notice that it doesn’t fail the job until the very end so that I can gather up all the failures rather than bailing on the first issue. Example code can be found here.

If you want to test this out yourself, there is a blog here that shows you how to spin up the free environment to experiment with this specific example.

1 Like

I know that because it runs in a temporary container or VM, whatever happens in Molecule has no impact. (I previously used Test-kitchen with Salt and Inspec on Vagrant VM.)

I used check_mode because I think I saw it somewhere in an article or molecule tests.
I thought it could be used for that.

If I understand your example correctly, you use ansible.builtin.assert to test your molecule setup by verifying the variables you ‘register’ using other methods (stats or whatever).

Does that mean that in my case I could continue using my lineinfile or template or other module I currently use by simply registering the result and then testing it with assert.

For me it looks like writing the tests will be longer, with 2 or 3 steps instead of only one (when I used Inspec)

For me it looks like writing the tests will be longer, with 2 or 3 steps instead of only one (when I used Inspec)

what I mean by saying that:
instead of

whatever_method_to_verify

do I need to write

one_verification:
  register: variable

assert:
  variable is defined and variable is equal to "xx"

?

It seems quite verbose to me and I would like to know if there is any way to test/verify in one step. But maybe it’s not possible using Ansible as a verifier.

Normally I use commands/modules to validate the program’s functionality e.g.

  • Use ansible.builtin.uri to retrieve a webpage and check if the result contains a specific text
  • Run a command that only functions if the configuration is correct for the testcase
  • Start a secondary container that will do some remote probing in some way.

You can check some of these roles for inspiration :slight_smile:

Using failed_when and ansible.builtin.assert is basically most you need. You can use it to check values or compare stuff. You could even have (though molecule should already do that for you in the idempotency test) have it fail like this:

- name: 'Put template'
  ansible.builtin.template:
    src: 'template.j2'
    dest: '/etc/file.conf'
    owner: 'root'
    group: 'root'
    mode: '0644'
  register: file_template
  failed_when: file_template['changed']

So in a nutshell, because you’re using ‘normal’ tasks/plays to validate the role, you can do anything you want/need!

3 Likes

Yes, that is often the way. There are some tests that may not require variables being registered. For example if your role installed a service, you may include a test that just starts the service. In that case you may not need the extra granularity that comes with the register/assert.

Your example didn’t include the block rescue, but I’d also encourage the use of that. That way you can capture multiple failures in a single test. It also allows you to include negative/expected failure tests.

3 Likes

So adding

    register: action_result                                                                                                                                                                                                                                                         
    failed_when: action_result['changed']         

does the job, thanks!

When executed in a loop, it even groups and returns all errors together like

TASK [Verify backupninja actions] **********************************************
failed: [debian-12] (item=10-system.sys) => {"ansible_loop_var": "item", "changed": true, "failed_when_result": true, "item": "10-system.sys"}
failed: [debian-12] (item=11-system.rdiff) => {"ansible_loop_var": "item", "changed": true, "failed_when_result": true, "item": "11-system.rdiff"}
failed: [debian-12] (item=20-db.mysql) => {"ansible_loop_var": "item", "changed": true, "failed_when_result": true, "item": "20-db.mysql"}
failed: [debian-12] (item=21-db.rdiff) => {"ansible_loop_var": "item", "changed": true, "failed_when_result": true, "item": "21-db.rdiff"}

PLAY RECAP *********************************************************************
debian-12                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

I will see if using block is needed to group together some checks, or simply to always see all checks.

What is weird, is that package in check_mode fails by default without using this trick.
It’s inconsistent.

TASK [Check backupninja installation] ******************************************
fatal: [debian-12]: FAILED! => {"changed": false, "msg": "No package matching 'backupninja' is available"}

PLAY RECAP *********************************************************************
debian-12                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0