Create folder structure on HDFS Storage

Archives Ansible Project
1

Hi,

I Have a challenge at work that I’m straggling to resolve for few days - Maybe ansible is not the tool for that ???

Requirements: Creating a folder structure on HDFS storage

Detailed Requirements:

  1. There are 3 based folders that needs to be created – Let’s assume folder per company

  2. On each company folder needs to create X number of folders – Let’s assume that the folder describes departments under each company. For example IT , Rnd , etc.

a. We can assume that all departments can be created for all company folder

  1. Under each department, we need to create additional folders that define the assets. For example, computers, network

a. Each department need to be created with relevant assets

  1. Last, each asset folder type needs to include additional folders that define the vendor type. For example, in computers, we need to have HP and Dell folders.

a. Each asset type in all departments can be defined with the same vendors

Currently, I have created the company folders (I have also checking if folders exist or not)

From that point, I know how to create the next folders using loop a– only static

Configuration file :

HDFS Base folder

hdfs_user: hdfs
hdfs_company_folders:

  • {path: company1 , user_owner: admin , group_owner: company1 , chmod: 640}
  • {path: company2 , user_owner: admin , group_owner: company2 , chmod: 640}
  • {path: company3 , user_owner: admin , group_owner: company3 , chmod: 755}

HDFS departmnet folder

departments:

  • {department: IT , department _folder_user_owner: admin , department _folder_group_owner: IT , department _folder_chmod: 640}
  • { department t: rnd , department _folder_user_owner: admin , department _folder_group_owner: rnd , department _folder_chmod: 640}

HDFS assets & Vendor folder

assets:

  • name: computer
    vendor:
  • HP
  • Dell
  • name: network
    vendor:
  • cisco

Create Compony folders – playbook

    1. Check what are the folders exists in HDFS
    1. Update chown and chmod
3.      - hosts: gw

  become: true

  become_user: "{{ hdfs_user }}"

  vars_files:

    - ./hdfs-environment-configuration.yml

  tasks:

   - name: Get folder list from HDFS

     shell: hadoop fs -ls / | awk '{print $8}'

     register: folder_list

   - name: create company folders on HDFS

     shell: hadoop fs -mkdir {{ item.path }}

     register: result

     with_items: "{{ hdfs_company_folders }}"

     when: item.path not in folder_list.stdout_lines

   - name: change base folder user and group owner

     shell: hadoop fs -chown -R {{ item.user_owner }}:{{ item.group_owner }} {{ item.path }}

     with_items: "{{ hdfs_company_folders }}"

     when: item.path not in folder_list.stdout_lines

   - name: change base folder mode

     shell: hadoop fs -chmod {{ item.chmod }} {{ item.path }}

     with_items: "{{ hdfs_company_folders }}"

     when: item.path not in folder_list.stdout_lines

   - name: report which folders were created

     debug:

       msg: "{{ item.path }} folder was created "

     when: item.path not in folder_list.stdout_lines

     with_items: "{{ hdfs_company_folders }}"

Create department folders – playbook ; i need to understand how I can create everything with variables and loops instead of static

    1. Get folder list per compony fodder
    1. Create the relevant department folders and compony folder if not exists
    1. Report which folder was created

Create assets and vendor folders – playbook

    1. In the configuration file how, I can define which department have which asset type
    1. Need to create the relevant assets folder under the relevant department
    1. Need to create a vendor folder under relevant asset folder