Uri module reports "URL can't contain control character"

Hi - I’m trying to use the uri module to execute a GET that includes a ‘filter’ option in the URL:

“https://{hostname}:8443/api/v2/storage-systems?filter=name eq ‘hostname.example.com’”

The uri module execution fails with:

"msg": "Status code was -1 and not [200, 201, 401]: An unknown error occurred: URL can't contain control characters. \"/api/v2/storage-systems?filter=name eq 'hostname.example.com'\" (found at least ' ')",

I need to use the ‘filter’ option to retrieve only the records I need. I need to follow the syntax provided by the api, which does include use of the space character between elements of the filter option.

I know the REST API command is correct, because I’ve executed it successfully in Postman.

Thoughts, please!
Thanks
tl

Try URL encoding the URL by changing the space characters to %20?

- name: Get URL
  ansible.builtin.uri:
    url: "https://{{ hostname }}:8443/api/v2/storage-systems?filter=name%20eq%20'http://hostname.example.com'"
3 Likes

Something like

- name: Get URL
  vars:
    uri: “https://{hostname}:8443/api/v2/storage-systems?filter=name eq ‘hostname.example.com’”
  ansible.builtin.uri:
      url: "{{ uri | urlencode }}"

should also work if you don’t want to do this manually.

3 Likes

Thank you both for your replies and ideas. Of the two, I liked Maxwell’s the best as it was ‘automatic’. So I tried:

  • name: Check if the system has already been added
    vars:
    urlraw: “https://{{ hostname }}:8443/api/v2/storage-systems?filter=name eq ‘{{ hostname.example.com }}’”
    uri:
    url: “{{ urlraw | urlencode }}”

but the task execution failed with:

"msg": "unknown url type: 'https%3A//host1.example.com%3A8443/api/v2/storage-systems%3Ffilter%3Dname%20eq%20%27hostname.example.com%27'",
"status": -1,
"url": "https%3A//host1.example.com%3A8443/api/v2/storage-systems%3Ffilter%3Dname%20eq%20%27hostname.example.com%27"

Should I be using ‘encode’ differently? Or, should I just replace the spaces with ‘%20’?

It looks like the Jinja2 urlencode filter is encoding the colons (as well as the question mark and equals sign), this isn’t wanted, so perhaps something like this world work:

- name: Get URL
  ansible.builtin.uri:
    url: "https://hostname.example.org:8443{{ url_path }}"
  vars:
    url_path: >-
      {{ ( "/api/v2/storage-systems?filter=name eq 'hostname.example.com'" | urlencode ) }} 

Or just do it for the query string since there is also no need to encode the ?:

- name: Get URL
  ansible.builtin.uri:
    url: "https://hostname.example.org:8443/api/v2/storage-systems?{{ query_string }}"
  vars:
    query_string: >-
      {{ ( "filter=name eq 'hostname.example.com'" | urlencode ) }} 

The urllib.parse.quote Python documentation contains:

By default, this function is intended for quoting the path section of a URL. The optional safe parameter specifies additional ASCII characters that should not be quoted — its default value is '/'.

So this (untested) might also work:

- name: Get URL
  ansible.builtin.uri:
    url: "{{ url_encoded }}"
  vars:
    url_encoded: >-
      {{ ( "https://hostname.example.org:8443/api/v2/storage-systems?filter=name eq 'http://hostname.example.com'" | urlencode('safe', '/:?=') }}

This (untested) version of the same thing is perhaps more legible:

- name: Get URL
  ansible.builtin.uri:
    url: "{{ url |  urlencode('safe', '/:?=') }}"
  vars:
    url: "https://hostname.example.org:8443/api/v2/storage-systems?filter=name eq 'http://hostname.example.com'"
1 Like

Hi Chris

Wow, thanks for the tutoring! I gave your ‘more legible’ version a try, and received this response:

"msg": "Unexpected templating type error occurred on ({{ urlraw | urlencode('safe', '/:?=') }}): do_urlencode() takes 1 positional argument but 3 were given. do_urlencode() takes 1 positional argument but 3 were given"
2 Likes

Three!? Not sure what is happening there, but perhaps the forward slash needs escaping and the safe isn’t required, this might work?

- name: Get URL
  ansible.builtin.uri:
    url: "{{ url |  urlencode(':?=\\/') }}"
  vars:
    url: "https://hostname.example.org:8443/api/v2/storage-systems?filter=name eq 'http://hostname.example.com'"

But I’m guessing at this point, perhaps simply manually editing the spaces to %20 is the simplest option?

1 Like

Hi Chris

Just to complete the conversation, I gave the code above a try, and got:

fatal: [ldpdd185.hop.lab.emc.com → localhost]: FAILED! => {
“msg”: “Unexpected templating type error occurred on ({{ urlraw | urlencode(‘:?=\\/’) }}): do_urlencode() takes 1 positional argument but 2 were given. do_urlencode() takes 1 positional argument but 2 were given”

I’ll try replacing the space characters explicitly. Thank you very much for your help!

1 Like

Based on the above it appears that when the Jinja2 urlencode filter is used you can’t amend the list of safe characters the Python urllib.parse.quote function supports from the default of '/', so perhaps you could mark my first suggestion as a solution, if it works, so future readers of the thread can skip all the details :wink: .

Furthermore perhaps a issue should be raised with the project to see if something like this usage could be supported in the future? (I’m assuming that the default safe value of '/' would need repeating if a list of characters that are safe to skip is provided?):

- name: Get URL
  ansible.builtin.uri:
    url: "{{ url |  urlencode('safe', url_safe) }}"
  vars:
    url: "https://example.org:80/api/?foo=bar baz"
    url_safe: "{{ ( ':?=/' | ansible.builtin.regex_escape ) }}"
1 Like

I marked the first suggestion as a solution; thanks again!

1 Like

I think that should be

{{ urlraw | urlencode(safe='/:?=') }}

I.e. one positional and one named parameter.

2 Likes

Ahh! is this documented anywhere?

Took me forever to find it, but, yes, it’s documented in the Custom Filters part of the Jinja documentation.

You can tell it’s a filter because there’s a “|” in front of it. Its first positional parameter is the stuff to the left of the “|”. Additional positional parameters show up in the parentheses, and some filters also take specific named parameters. If you’re really deep into the Python weeds, you can even pass references to parameters with “**”, but keep your wizard hat and non-conductive soled shoes on when you do that.

Tangent alert: The first words in that section are “Filters are Python functions”, and it really is true. To be fair, Jinja is targeted at Python programmers, not application end users. This is one of the reasons I’ve called Ansible an extension of Python rather than an application written in Python. There’s just so much of the underlying language exposed to the user (or the other way around) for me to classify it otherwise. To be fair, the Ansible implementation has come a long way in the last few years toward blunting those sharp edges, but they are still there.

As a counter example, Splunk is also written in Python, but I’ve never seen hints through the user interface that would lead me to guess that. (Although, if you go to write your own Splunk “app”, you’d do it in Python. Splunk custom apps are rather like Ansible plugins in that way.)

Another way to think about the “extension of a language” vs “application written in a language” issue is this: If you were to re-implement the existing functionality in some other language, what would that look like? In the case of Ansible, the first thing you’d have to do is create something with semantics frighteningly similar to Python, and that’s the clue. Some of what in retrospect appear to be questionable design decisions in Ansible were in fact properties emergent from the implementation technologies at hand, such as Jinja, and the way they were employed in the early days.

Even farther tangent: I think it’s useful to occasionally think about such questions. If for whatever reasons we chose to start over completely, re-implementing Ansible from scratch with no backward compatible constraints, what would we keep, what would we drop like a hot rock, and what would we do in a different (presumably better) way? It’s not a realistic proposition (or is it?), but it helps shape thinking about how the project evolves. That’s where the value lies in such exercises.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.