How to handle lists of files and avoid unnecessary changes

Archives Ansible Project
1

Hi,

on Debian systems, one can configure repositories in an
/etc/apt/sources.list.d directory with multiple files. I usually name
the files after the distribution, so that, in a simplyfied example, I
might have a file /etc/apt/sources.list.d/stable.list on systems running
Debian stable, and /etc/apt/sources.list.d/testing.list on systems
running Debian testing. In the real case, there are additional
distribution-depending files (security, backports, local packages in
different stages).

A similiar setup can usually be found in /etc/yum.repos.d/ on Red Hat
based systems.

A local admin might choose to place additional files into those
directories manually, for example adding special repositories for
third-party software that doesn't come with the distribution.

But back to the Debian case.

When I change a system from testing to stable, I want the all
testing*.list to vanish, and the stable*.list files to appear, unless
they're already there, in which case I want them untouched. Locally
placed files should also be untouched.

My first approach was to concentrate all the ansible-managed *.list
files under a common prefix, zda, giving, for example,
zda-unstable.list.

I then wrote code to first remove all zda*.list files (and their laegacy-named
instances), and code to deploy the correct zda-foo.list file:

- name: search for sources.list files
  find:
    paths: "/etc/apt/sources.list.d"
    patterns: "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-unstable-mc.list,stretch-mc.list,stretch-security.list,stretch-zg-stable-mc.list,stretch-zg-unstable-mc.list,buster-mc.list,stretch-zg-stable-mc.list,stretch-security.list"
  register: sourceslistfiles
- name: delete sources.list files
  file:
    path: "{{ item.path }}"
    state: absent
  with_items: "{{ sourceslistfiles.files }}"

and finally code to roll out new list files:

- name: include repositories
  tags:
          repos
  include_tasks:
          "{{distribution}}/{{distribution_version}}/repos.yml"

$ cat roles/common/tasks/debian/stretch/repos.yml

2

When I change a system from testing to stable, I want the all
testing*.list to vanish, and the stable*.list files to appear, unless
they're already there, in which case I want them untouched. Locally
placed files should also be untouched.

How you can implement this depends on the feature you need
1. one server having a several repo stable, testing and unstable
2. one server only having stable, testing or unstable.

My first approach was to concentrate all the ansible-managed *.list
files under a common prefix, zda, giving, for example,
zda-unstable.list.

I then wrote code to first remove all zda*.list files (and their laegacy-named
instances), and code to deploy the correct zda-foo.list file:

- name: search for sources.list files
  find:
    paths: "/etc/apt/sources.list.d"
    patterns:
"zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-unstable-mc.list,stretch-mc.list,stretch-security.list,stretch-zg-stable-mc.list,stretch-zg-unstable-mc.list,buster-mc.list,stretch-zg-stable-mc.list,stretch-security.list"
  register: sourceslistfiles
- name: delete sources.list files
  file:
    path: "{{ item.path }}"
    state: absent
  with_items: "{{ sourceslistfiles.files }}"

To clean up legacy I would only run this once and after that make Ansible do the work.

and finally code to roll out new list files:

- name: include repositories
  tags:
          repos
  include_tasks:
          "{{distribution}}/{{distribution_version}}/repos.yml"

$ cat roles/common/tasks/debian/stretch/repos.yml
---
- name: zda-stretch-mc.list
  tags:
  - repos
  - stretch
  copy:
    dest: /etc/apt/sources.list.d/zda-stretch-mc.list
    owner: root
    group: root
    mode: 0644
    content: |
            deb http://debian.debian.zugschlus.de/debian/ stretch main contrib
  notify: apt update
- name: zda-stretch-security.list
  tags:
  - repos
  - stretch
  copy:
    dest: /etc/apt/sources.list.d/zda-stretch-security.list
    owner: root
    group: root
    mode: 0644
    content: |
            deb
http://debian-security.debian.zugschlus.de/debian-security/
stretch/updates main contrib
            deb http://security.debian.org/ stretch/updates main contrib
  notify: apt update

Feature 1 only need one variable, call it "repo_type", this could contain stable, testing or unstable.

Then your tasks would look something like this

- apt_repository:
     repo: deb http://debian.debian.zugschlus.de/debian/ stretch main contrib
     filename: zda-stretch-mc
     state: "{{ (repo_type == 'stable') | ternary('present', 'absent') }}"

State present creates the file, and absent remove the file so Ansible will clean up the files so nothing is left behind when you change repo_type.
You would need one for each repo and repo_type.
But with with_items and facts you could probably make a loop to deal with most of them.

With feature 2 you could have something like stable_state, testing_state and unstable_state and set them to absent or present, and then just use that variable in state.

Hopefully this make some sense.

3

> When I change a system from testing to stable, I want the all
> testing*.list to vanish, and the stable*.list files to appear, unless
> they're already there, in which case I want them untouched. Locally
> placed files should also be untouched.

How you can implement this depends on the feature you need
1. one server having a several repo stable, testing and unstable
2. one server only having stable, testing or unstable.

2, but with execptions. The general function that I want is "zap
everything matching a glob, execpt files that would be written to by
another task"

> My first approach was to concentrate all the ansible-managed *.list
> files under a common prefix, zda, giving, for example,
> zda-unstable.list.
>
> I then wrote code to first remove all zda*.list files (and their
> laegacy-named
> instances), and code to deploy the correct zda-foo.list file:
>
> - name: search for sources.list files
> find:
> paths: "/etc/apt/sources.list.d"
> patterns:
> "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-unstable-mc.list,stretch-mc.list,stretch-security.list,stretch-zg-stable-mc.list,stretch-zg-unstable-mc.list,buster-mc.list,stretch-zg-stable-mc.list,stretch-security.list"
> register: sourceslistfiles
> - name: delete sources.list files
> file:
> path: "{{ item.path }}"
> state: absent
> with_items: "{{ sourceslistfiles.files }}"

To clean up legacy I would only run this once and after that make Ansible do
the work.

I also want to clean up uncontrolled growth as a continuous process, and
I want to be able to switch a host from one distribution to another with
the files belonging to the "old" one cleaned up automatically.

Feature 1 only need one variable, call it "repo_type", this could contain
stable, testing or unstable.

Then your tasks would look something like this

- apt_repository:
    repo: deb http://debian.debian.zugschlus.de/debian/ stretch main contrib
    filename: zda-stretch-mc
    state: "{{ (repo_type == 'stable') | ternary('present', 'absent') }}"

Probably a workaround if the really nice solution can not be had. The
few hosts that have multple repositories would get the extra repos
outside of ansible. I think I can live with that, but it's not pretty.

With feature 2 you could have something like stable_state, testing_state and
unstable_state and set them to absent or present, and then just use that
variable in state.

Also, rather ugly.

But I have to admit that my sense of prettiness is not yet tuned to
ansible.

Greetings
Marc

4

This is what I did and it works. Here my solution:

- name: search for sources.list files
  find:
    paths: "/etc/apt/sources.list.d"
    patterns: "zda-*.list,exp-mc.list,sid-mc.list,sid-zg-stable-mc.list,sid-zg-u
  register: presentsourceslistfiles
- name: set up fact list for wanted sources.list.d entries
  set_fact:
    wantedsourceslistfiles:
- name: include repositories
  tags:
          repos
  include_tasks:
          "{{distribution}}/{{distribution_version}}/repos.yml"
- name: delete sources.list files
  file:
    path: "{{ item.path }}"
    state: absent
  with_items: "{{ presentsourceslistfiles.files }}"
  when: item.path not in wantedsourceslistfiles

debian/sid/repos.yml:
- name: create list of sid wantedsourceslistfiles
  set_fact:
    sidwantedsourceslistfiles:
      - /etc/apt/sources.list.d/zda-sid-mc.list
      - /etc/apt/sources.list.d/zda-sid-zg-stable-mc.list
      - /etc/apt/sources.list.d/zda-sid-zg-unstable-mc.list
- name: add sid files to wantedsourceslistfiles
  set_fact:
    wantedsourceslistfiles: "{{ wantedsourceslistfiles }} + {{ sidwantedsourceslistfiles }}"
- name: zda-sid-mc.list
  copy:
    dest: /etc/apt/sources.list.d/zda-sid-mc.list
    owner: root
    group: root
    mode: 0644
    content: |
            deb http://debian.debian.zugschlus.de/debian/ sid main contrib
  notify: apt update
- name: zda-sid-zg-stable-mc.list
  copy:
    dest: /etc/apt/sources.list.d/zda-sid-zg-stable-mc.list
    owner: root
    group: root
    mode: 0644
    content: |
            deb http://zg20150.debian.zugschlus.de/zg20150/ sid-zg-stable main contrib
  notify: apt update
- name: zda-sid-zg-unstable-mc.list
  copy:
    dest: /etc/apt/sources.list.d/zda-sid-zg-unstable-mc.list
    owner: root
    group: root
    mode: 0644
    content: |
            deb http://zg20150.debian.zugschlus.de/zg20150/ sid-zg-unstable main contrib
  notify: apt update

Greetings
Marc