Deleting s3 objects using s3_object module doesn't seem to work

[lioradler (Lior)]

Hello community,

So I have a task to delete files from a s3 bucket:

- name: 'Delete Backup {{ backup_ci.attributes.backup_file_name }} in STAGING'
  amazon.aws.aws_s3:
    bucket: '{{ backup_ci.attributes.backup_file_name | urlsplit("hostname") | split(".") | first }}'
    object: '{{ backup_ci.attributes.backup_file_name | urlsplit("path") }}'
    mode: delobj
    endpoint_url: '{{ ENDPOINT_URL }}'
    aws_access_key: "{{ aws_access_key_id }}"
    aws_secret_key: "{{ aws_secret_access_key }}"
  register: return_aws_s3

I run the playbook using AWX and the output looks just fine, it even tells me that the object was deleted:

{
  "msg": "Object deleted from bucket 659c0270eb8a6e27fda1a3ac.",
  "changed": true,
  "invocation": {
    "module_args": {
      "bucket": "659c0270eb8a6e27fda1a3ac",
      "object": "/data-2024-01-08_15-46",
      "mode": "delobj",
      "endpoint_url": "HIDDEN",
      "aws_access_key": "HIDDEN",
      "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
      "access_key": "HIDDEN",
      "secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
      "validate_certs": true,
      "debug_botocore_endpoint_logs": false,
      "encrypt": true,
      "encryption_mode": "AES256",
      "expiry": 600,
      "marker": "",
      "max_keys": 1000,
      "sig_v4": true,
      "permission": [
        "private"
      ],
      "overwrite": "different",
      "prefix": "",
      "retries": 0,
      "dualstack": false,
      "ceph": false,
      "ignore_nonexistent_bucket": false,
      "purge_tags": true,
      "validate_bucket_name": true,
      "session_token": null,
      "profile": null,
      "aws_ca_bundle": null,
      "aws_config": null,
      "region": null,
      "dest": null,
      "headers": null,
      "metadata": null,
      "version": null,
      "src": null,
      "content": null,
      "content_base64": null,
      "encryption_kms_key_id": null,
      "tags": null,
      "copy_src": null
    }
  },
  "_ansible_no_log": false
}

but the objects is still in the bucket as if nothing happened:

Is this a bug or am I doing something wrong?

boto3 version: 1.28.53
botocore version: 1.31.54

[jrglynn2]

I don’t know AWX, but is there a way to get detailed results? Ansible command line would be using -vvv …

One thing to confirm is that whatever IAM profile you are using has delete access - could be that ansible is misinterpreting a “don’t have access to delete” error as “deleted”? You can test this by running an AWS CLI command by itself, take Ansible out of the picture for that test.

[lioradler (Lior)]

Hello @jrglynn2 and thanks for picking this up.

So, I was able to delete this file using awscli and the same IAM profile like the one I was using in the playbook (which is root, so obviously delete access is granted):

{
    "Deleted": [
        {
            "Key": "659c0270eb8a6e27fda1a3ac/data-2024-01-08_15-46"
        }
    ]
}

Here’s the output when I run it with -vvv:

TASK [backup_garbage_collector : Delete Backup https://659c0270eb8a6e27fda1a3ac.ENDPOINT_URL/data-2024-01-08_15-46 in STAGING] ***
task path: /path/is/irrelevant
redirecting (type: modules) amazon.aws.aws_s3 to amazon.aws.s3_object
redirecting (type: action) amazon.aws.aws_s3 to amazon.aws.s3_object
redirecting (type: action) amazon.aws.aws_s3 to amazon.aws.s3_object
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1704825360.5853736-82-23528765592178 `" && echo ansible-tmp-1704825360.5853736-82-23528765592178="` echo /root/.ansible/tmp/ansible-tmp-1704825360.5853736-82-23528765592178 `" ) && sleep 0'
redirecting (type: modules) amazon.aws.aws_s3 to amazon.aws.s3_object
Using module file /usr/share/ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_object.py
<127.0.0.1> PUT /home/runner/.ansible/tmp/ansible-local-19mm6a_0nq/tmpg4u_12yz TO /root/.ansible/tmp/ansible-tmp-1704825360.5853736-82-23528765592178/AnsiballZ_s3_object.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1704825360.5853736-82-23528765592178/ /root/.ansible/tmp/ansible-tmp-1704825360.5853736-82-23528765592178/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1704825360.5853736-82-23528765592178/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1704825360.5853736-82-23528765592178/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "access_key": "HIDDEN",
            "aws_access_key": "HIDDEN",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "bucket": "659c0270eb8a6e27fda1a3ac",
            "ceph": false,
            "content": null,
            "content_base64": null,
            "copy_src": null,
            "debug_botocore_endpoint_logs": false,
            "dest": null,
            "dualstack": false,
            "encrypt": true,
            "encryption_kms_key_id": null,
            "encryption_mode": "AES256",
            "endpoint_url": "HIDDEN",
            "expiry": 600,
            "headers": null,
            "ignore_nonexistent_bucket": false,
            "marker": "",
            "max_keys": 1000,
            "metadata": null,
            "mode": "delobj",
            "object": "/data-2024-01-08_15-46",
            "overwrite": "different",
            "permission": [
                "private"
            ],
            "prefix": "",
            "profile": null,
            "purge_tags": true,
            "region": null,
            "retries": 0,
            "secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "session_token": null,
            "sig_v4": true,
            "src": null,
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "version": null
        }
    },
    "msg": "Object deleted from bucket 659c0270eb8a6e27fda1a3ac."
}

Seems clean to me… but it doesn’t do anything

[jrglynn2]

This simplified playbook worked fine for me:

- name: S3 test
  hosts: localhost
  gather_facts: false
  tasks:
    - name: 'Delete file in S3'
      delegate_to: localhost
      environment:
        AWS_PROFILE: "ansible-dev"
      amazon.aws.aws_s3:
        bucket: 'bucket-name'
        object: 'dynamic_storage/jrglynn2.txt'
        mode: delobj

Some notes/differences:

• I use AWS profiles instead of feeding in aws_account_key & aws_secret_key. I had many issues with those early on, and found it easier to just change the Environment variable whenever I run AWS commands. Many others use the keys, so I doubt that is an issue, just an FYI
• I don’t know what endpoint_url is, and it seems like it wasn’t needed for me

Have you confirmed your bucket and object values in the -vvv output? Specifically the object, that leading / is suspect to me… Maybe try hard coding the values in like I did. And/or printing out the values of the variables prior to running the deletion, to verify they are correct?

[lioradler (Lior)]

So it turned out that the value provided for the “object” attribute should be the file’s key and not file’s name

Thanks for leading me into this @jrglynn2 !