How to discriminate between chunked and not chunked answers from uri

Archives Ansible Project
1

ansible core 2.12.3

I am unable to discriminate with uri between responses which have been chunked and those which haven’t.In both cases, I get the same:

  • msg: OK (unknown bytes)
  • transfer_encoding: chunked

This happens for instance when using uri over CSR 1kv 17.3.1a with:

  • name: Sending GET datastores with uri
    ansible.builtin.uri:
    body: ‘’
    body_format: ‘json’
    force_basic_auth: yes
    headers:
    Accept: application/yang-data+json
    Content-Type: application/yang-data+json
    method: GET
    return_content: yes
    status_code: “100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308”
    timeout: 120
    url: “https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state/datastores?content=nonconfig”
    url_password: password
    url_username: admin
    validate_certs: false
    register: return_uri_restconf_json

leads to:
task path: main.yml:219
ESTABLISH LOCAL CONNECTION FOR USER: root
EXEC /bin/bash -c ‘echo ~root && sleep 0’
EXEC /bin/bash -c ‘( umask 77 && mkdir -p “echo /root/.ansible/tmp”&& mkdir “echo /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108” && echo ansible-tmp-1646997734.4119265-2691638-25534722658108=“echo /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108” ) && sleep 0’
Using module file /usr/local/lib/python3.9/dist-packages/ansible/modules/uri.py
PUT /root/.ansible/tmp/ansible-local-269044374_5i0rc/tmpyvffo2y1 TO /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/AnsiballZ_uri.py
EXEC /bin/bash -c ‘chmod u+x /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/ /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/AnsiballZ_uri.py && sleep 0’
EXEC /bin/bash -c ‘/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/AnsiballZ_uri.py && sleep 0’
EXEC /bin/bash -c ‘rm -f -r /root/.ansible/tmp/ansible-tmp-1646997734.4119265-2691638-25534722658108/ > /dev/null 2>&1 && sleep 0’
ok: [CSR1000v-17.3.1a → localhost] => changed=false
cache_control: private, no-cache, must-revalidate, proxy-revalidate
connection: close
content: |-
{
“ietf-netconf-monitoring:datastores”: {
“datastore”: [
{
“name”: “running”,
“tailf-netconf-monitoring:transaction-id”: “1646-983921-976163”
},
{
“name”: “candidate”
}
]
}
}
content_type: application/yang-data+json
cookies: {}
cookies_string: ‘’
date: Fri, 11 Mar 2022 11:22:14 GMT
elapsed: 0
invocation:
module_args:
attributes: null
body: ‘’
body_format: json
ca_path: null
client_cert: null
client_key: null
creates: null
dest: null
follow_redirects: safe
force: false
force_basic_auth: true
group: null
headers:
Accept: application/yang-data+json
Content-Type: application/yang-data+json
http_agent: ansible-httpget
method: GET
mode: null
owner: null
remote_src: false
removes: null
return_content: true
selevel: null
serole: null
setype: null
seuser: null
src: null
status_code:

We can see that the response has NOT been chunked.

In another example, the response has been chunked with:

  • name: Sending GET netconf-state with uri
    ansible.builtin.uri:
    body: ‘’
    body_format: ‘json’
    force_basic_auth: yes
    headers:
    Accept: application/yang-data+json
    Content-Type: application/yang-data+json
    method: GET
    return_content: yes
    status_code: “100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308”
    timeout: 120
    url: “https://172.21.126.182/restconf/data/ietf-netconf-monitoring:netconf-state?content=nonconfig”
    url_password: password
    url_username: admin
    validate_certs: false
    register: return_uri_restconf_json

which leads to a really chunked answer:
task path: main.yml:219
ESTABLISH LOCAL CONNECTION FOR USER: root
EXEC /bin/bash -c ‘echo ~root && sleep 0’
EXEC /bin/bash -c ‘( umask 77 && mkdir -p “echo /root/.ansible/tmp”&& mkdir “echo /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673” && echo ansible-tmp-1646997777.3538132-2700276-93994275105673=“echo /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673” ) && sleep 0’
Using module file /usr/local/lib/python3.9/dist-packages/ansible/modules/uri.py
PUT /root/.ansible/tmp/ansible-local-269044374_5i0rc/tmps9ux33ea TO /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/AnsiballZ_uri.py
EXEC /bin/bash -c ‘chmod u+x /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/ /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/AnsiballZ_uri.py && sleep 0’
EXEC /bin/bash -c ‘/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/AnsiballZ_uri.py && sleep 0’
EXEC /bin/bash -c ‘rm -f -r /root/.ansible/tmp/ansible-tmp-1646997777.3538132-2700276-93994275105673/ > /dev/null 2>&1 && sleep 0’
ok: [CSR1000v-17.3.1a → localhost] => changed=false
cache_control: private, no-cache, must-revalidate, proxy-revalidate
connection: close
content: |-
{
“ietf-netconf-monitoring:netconf-state”: {
“capabilities”: {
…
},
“datastores”: {
“datastore”: [
{
“name”: “running”,
“tailf-netconf-monitoring:transaction-id”: “1646-983921-976163”
},
{
“name”: “candidate”
}
]
},
“schemas”: {
…
},
“sessions”: {
“session”: [
{
“session-id”: 26,
“transport”: “tailf-netconf-monitoring:rest-http”,
“username”: “admin”,
“source-host”: “172.21.0.1”,
“login-time”: “2022-03-11T11:22:57+00:00”,
“tailf-netconf-monitoring:transaction”: [
{
content_type: application/yang-data+json
cookies: {}
cookies_string: ‘’
date: Fri, 11 Mar 2022 11:22:57 GMT
elapsed: 0
invocation:
module_args:
attributes: null
body: ‘’
body_format: json
ca_path: null
client_cert: null
client_key: null
creates: null
dest: null
follow_redirects: safe
force: false
force_basic_auth: true
group: null
headers:
Accept: application/yang-data+json
Content-Type: application/yang-data+json
http_agent: ansible-httpget
method: GET
mode: null
owner: null
remote_src: false
removes: null
return_content: true
selevel: null
serole: null
setype: null
seuser: null
src: null
status_code:

Any suggestion about how to differentiate between both types of responses?

2

Python 3 should be attempting to read all chunks if the transfer-encoding is chunked when a single .read() is called. I would not expect it to fail to fetch all chunks. As such, even if transfer_encoding: chunked is displayed, python should have still fetched the full content.

So, to more specifically answer your question, if you see transfer_encoding: chunked, then the response was chunked, but python has attempted to fetch all chunks.

I suppose to determine if you have a partially read chunk, and not the full response, you would have to look at whether the .json key exists on the module response.

3

Thanks.

But I understand from your answer that really chunked answers should not happen under normal circumstances.
Since “Python 3 should be attempting to read all chunks”, I tried to set the timeout to a high value (3600) to eliminate that potential explanation for the chunked answer.
However, it is still happening.
Should I report this a bug or can you suggest anything else?

Also, by default (if no other headers are set), is it possible to receive something else than ‘transfer_encoding: chunked’ from the server for body_format: ‘json’ or body_format: ‘raw’?

4

I wouldn’t know where an appropriate place to file a bug would be. Looking at the CPython code for http.client, .read() should be fully consuming all of the chunks.

https://github.com/python/cpython/blob/3.10/Lib/http/client.py#L577

As such, I doubt we would do anything in ansible-core to deal with this.

There is no way for the client to instruct the server to not send a chunked response. A chunked response is used when the Content-Length isn’t known at the time the response is sent back to the client.

5

But how can the Content-Length be unknown at the time the response is sent back to the client for small unchunked answers (case n°1 of my first post)?

6

I have filed the issue here