As advertised before, we have been working on community.openwrt for the past couple of weeks.
The role that this collection is derived of, gekmihesg.openwrt had some molecule tests that relied on docker images for OpenWrt that are published regularly by the project. Note that by it’s very nature (a specific platform), the most reliable way to test the collection is against those container images.
That setup was carried over to the collection (with some adjustments, of course), it lives now in the extensions/molecule directory as it’s for for molecule running on collections, and it is working smoothly.
However, we would like to have integration tests for each module, so we have started with one module, command, as is quite a simple one. To get that working, we are using runme.sh to trigger the molecule run.
There is a PR open for that, and we would appreciate having more feedback.
I am not entirely happy with the mechanics of it, but so far I could not find a better way to make it work.
PS: I have already asked for feedback in Matrix, but it occurred to me that there is probably(?!) more people in the forum than in the chat. Sorry for the cross-posting.
What is the reason you want molecule tests to be “integrated” into integration tests?
I just run molecule tests separately. I have unit tests (python + ansible), integration tests and molecule tests - these are different test categories, I do not see benefit of mixing them. Do you?
One benefit of separating molecule and integration tests might be not to run all the molecule tests all the time - they tend to be costly (take a lot of time). You can run molecule tests for some modules or roles only if respective modules or roles are changed in the PR. just name the molecule scenario as the role/module for simplicity.
some comments to the code
" ansible.builtin.include_tasks: ../../tasks/main.yml"
“cp -a “$TARGET_DIR/tasks/.” “$ROOT/tasks/””
you do not have to copy tasks - just create ‘roles’ folder inside your scenario and put you tasks (they look like a role) there and include them as role - ansible will look for roles to import / inlcude in ‘roles’ folder by default. I just create separate collection for molecule playbooks and roles and import all the content from there. This simplifies the reuse. But I use completely different repo structure (non standard) that allows that.
“cp -a “$TARGET_DIR/molecule” “$MOL_DIR”” why do you copy molecule scenarios? molecule saves all the data in ephemeral folders anyway.
I know that official recommendation is to put molecule folder inside extension folder. There is actually no need to do that, you can put molecule folder inside tests folder there it belongs - as you have tests there. So all the tests are inside tests folder.
Ha! I did not know that! It looks very promising, thanks for pointing it out. Any chance you can paste the links to the repos where you do that? Assuming they are not private, of course.
That being said, I don’t entirely discard using a non-standard structure here, but I would like to avoid doing that if possible.
That’s exactly to mesh them with the test tasks. But given your prior suggestion we may not need to do that at all.
And we do have molecule tests in the extension folder, they were the tests we added back in the day to the gekmihesg.openwrt role. They serve like a “quick” check the general thing is working. The structure tests/integration/target/*, on the other hand, is meant to hold tests matching the plugins (modules and otherwise) in the collection, one in each target. I would like very much to keep that structure - or if we are not keeping it, then we have some conceptually equivalent/similar.
Last, but not least, whilst I appreciate your commenting, I would like to suggest we take discussions of code choices/details to the PR at GH. It is much easier to point the exact code we are talking, and to suggest modifications.
Here some examples. I assume they require explanation, I do not do things as it is done in other collections and there are reasons for that. If there are any questions or comments to below - will be glad to address them.
In the link above you see tests for one role, they are placed inside the role folder to be close the the role they tests. They can be also placed in tests folder. I do not know why but i tend to put role molecule tests close to the roles, but tests for plugins and module into tests folder.
As you can see here
Molecule reuses playbooks from separate collection (in the same repo) that sole purpose is to support molecule tests. This is possible due to repo structure - I do not put collection content inside repo root folder, instead I put all in
ansible_collections//<collection_name> folder
This allows to set only one environment variable ANSIBLE_COLLECTIONS_PATH to effectively “install” (make available for ansible-galaxy to find) all the collections that are inside this repo.
See community.sap_ha_cluster_qa/tox.ini at 95e19a141b42d7e55e0822fb2a7afe74e243b2f2 · sap-linuxlab/community.sap_ha_cluster_qa · GitHub
All other collections (that are required for testing for instance, or just dependencies for developed collection) are installed in python virtual environment that is managed by tox.
Here is installation code community.sap_ha_cluster_qa/Makefile at 95e19a141b42d7e55e0822fb2a7afe74e243b2f2 · sap-linuxlab/community.sap_ha_cluster_qa · GitHub
Yes - all the dependencies for molecule (and everything else) are managed outside of molecule (or ansible-lint). That allows to test anything with any dependency I want.
For instance if molecule versions would be updated in community.sap_ha_cluster_qa/tox/requirements-molecule.txt at 95e19a141b42d7e55e0822fb2a7afe74e243b2f2 · sap-linuxlab/community.sap_ha_cluster_qa · GitHub tox will ensure (for each run) that required molecule version is installed.
This is complete opposite to your approach in the PR
“pip install molecule ‘molecule-plugins[docker]’” - dependencies and executions are in one script that prevents to execute with different dependencies (or makes it harder).
I also tend to pin all the dependencies as much as possible. Having all the depedencies in separate requirements-*.txt files allows to use dependabot (see dependabot PRs in the cluster repo I mentioned).
I might be wrong, but I think in the current setup it is not possible (or I did not get how) to run molecule tests locally from one’s laptop. I suggest you enable it.
I made it possible with makefiles, see
This code allows anyone to run make molecule - - <any molecule commands of flags> - make will ensure that all dependencies are installed (python and ansible collections). All the molecule commands will be executed inside $WORKDIR folder (by default tests). Workdir folder can be changed by simply setting WORKDIR environment variable: WORKDIR=another/folder make - - molecule --version. This is just an example, use tools that suit your need and are familiar to you.
I see you set ANSIBLE_ROLES_PATH and placed the role directly inside molecule scenario.
If I am not mistaken ansible by default looks for roles inside roles folder in the current folder (molecule runs inside scenario folder). So you do not have to set this env variable, just place the roles you need for scenario inside roles folder. That will of course limit your ability to reuse - if you later decide to use same helper role for another scenario. In that case please set ANSIBLE_ROLES_PATH=“$(pwd):$ANSIBLE_ROLES_PATH” so you do not remove values that might be set from outside. Or just just use ANSIBLE_COLLECTIONS_PATH approach I described here.
Unfotunately I never got to use ansible-test integration command - ansible-test does too much behind the scenes. What I do not get is - CI is executed in container (github runner), that runs ansible-test - all the tests are executed inside containers. And you want tests to be molecule scenarios that are running containers. So it is container inside container inside container. I bet it is much easier (and you have more control over dependencies and their versions and less to download) to execute molecule directly - it is just a python package you can execute it anywhere. At the moment I see there are two groups of CI jobs - one for integration tests and one for molecule tests. Does it mean that molecule tests will be executed twice as molecule and as integration tests.
Try it - the only “issue” is that no one will be able to install collection directly from github - this is actually not an issue, no one should install code from the internet. Everything else is just a bonus with this structure.
This is not about code at the moment, but about approach in general. You can still make changes and decisions that will make this collection not a copy of all the content that was created years ago. No disrespect to old content, but ansible changed since when.