Ansible playbook for updated file listing

Get Help
,
1

I am trying to develop an ansible module which would traverse subfolders under a main folder and list all .SQL files inside the sub-folders which were modified in the last 30 minutes. Below is my code:

  • name: Install SQL scripts on Private PostgreSQL Instance
    hosts: localhost
    gather_facts: no

become: yes

tasks:
- name: install pip3
apt: name=python3-pip state=present

- name: Install required apt packages
  apt:
    name: 
      - libpq-dev
      
- name: Install required python packages
  pip:
    name: 
      - psycopg2

- name: Download cloudSQL proxy
  ansible.builtin.get_url:
    url: https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.13.0/cloud-sql-proxy.linux.amd64
    dest: ./cloud-sql-proxy
    mode: "0755"

- name: Set GCP project
  command:
    gcloud config set project '{{ gcp_project }}'

- name: Print gcloud config
  command:
    gcloud config list
    
- name: Start cloudSQL proxy
  command: ./cloud-sql-proxy --private-ip "{{ gcp_project }}:{{ db_instance_location }}:{{ db_instance_name }}"
  async: 3600
  poll: 0


- name: List subfolders
  find: 
    path: DDL/
    recurse: no
    file_type: directory
  register: subfolders

- name: Find recently modified files in each subfolder
  find:
    path: "{{ item.path }}"
    recurse: yes
    patterns: "*.sql"
    age: "-30m"
    age_stamp: "mtime"
    use_regex: yes
  loop: "{{ subfolders.files }}"
  register: recent_files_per_subfolder
  loop_control:
    label: "{{ item.path }}"

- name: Display files modified within the last 5 minutes for each subfolder
  debug:
    msg: "Files modified in {{ item.item.path }}: {{ item.files | map(attribute='path') | list }}"
  loop: "{{ recent_files_per_subfolder.results }}"
  when: subfolders.files | length > 0

However the playbook output not listing anything. I have tried changing the age parameter, but the output is same.

Any help would be really appreciated.

1 Like
2

I’m not completely sure, but patterns: "*.sql" and use_regex: yes doesn’t match, as *.sql is a filename-expansion pattern and not a regular expression.
For it to be a regexp it would need to be something like ^.*\.sql$

HTH

2 Likes
3

This solution doesn’t seem to work.

4

What does subfolders look like, try adding a debug for this and post the result here?

- name: Debug subfolders
  ansible.builtin.debug:
    var: subfolders

Also try setting use_regex to false as *.sql is a shell pattern?

5

Hi Chris,
Here is what the folder structure looks like:

DDL(main folder) → Multiple sub-folders → Each sub-folder contains multiple .sql files

6

Right, but does the subfolders dictionary contain the list of sub-directories? I’m asking as DDL/ is a relative path and I’m wondering if a absolute path might be needed here? :person_shrugging:

7

yes it does:

- name: List subfolders
  find: 
    path: DDL/
    recurse: yes
    file_type: directory
  register: subfolders

- name: Find recently modified files in each subfolder
  find:
    path: "{{ item.path }}"
    recurse: yes
    patterns: "*.sql"
    age: "-10m"

age_stamp: “mtime”

use_regex: yes

  loop: "{{ subfolders.files }}"
  register: recent_files_per_subfolder
  loop_control:
    label: "{{ item.path }}"

- name: Display files modified within the last 5 minutes for each subfolder
  debug:
    msg: "Files modified in {{ item.item.path }}: {{ item.files | map(attribute='path') | list }}"
  loop: "{{ recent_files_per_subfolder.results }}"
  when: subfolders.files | length > 0
8

Did you try use_regex: false as I suggested above?

9

i did, but unfortunately, no luck. :frowning:

10

If you want to use Ansible over SSH using public keys then you first need to get SSH working with public keys, perhaps there are logs on your switch which might indicate why it isn’t working?

Or run ssh -vvvv to get verbose output in case this explains why it is not working?

11

Ansible is running on a github runner where the code is checked out. It is not doing ssh to anywhere.

12

Aargh, I was replying to the wrong thread, sorry! :roll_eyes:

13

But you didn’t show us what happened! Saying you tried it but it didn’t work gives us nothing to help you with. When Chris asked about the folder structure and gave the ansible.builtin.debug task to answer the question, you instead told us what you think the folder structure is, which is only somewhat interesting. But we aren’t trying to debug you; we’re trying to debug your tasks. What matters is what they see, not what you think is there.

I’ve seen nothing so far that convinces me your first “find” task is finding anything. What’s the output from this:

- name: List subfolders
  ansible.builtin.find:
    path: DDL  # <- highly questionable relative path
    recurse: no
    file_type: directory
  register: subfolders

- name: Debug subfolders
  ansible.builtin.debug:
    var: subfolders
4 Likes