Is it intentional to authorize Python escaping in YAML playbooks?

I’ve noticed that when I use an escape sequence like “\n” in my YAML playbooks, it is interpreted by Python modules like a newline, instead of a two separated characters "" and “n”. Is it intended? It seems redundant because YAML already offers its own escaping mechanism.

I understand the related code is in lib/runner/init.py, in Runner._copy_module:

encoded_args = “"""%s"""” % module_args.replace(“"”,“\"”)

Maybe it could be replaced by something like this:

encoded_args = module_args.encode(‘string_escape’)

What is your opinion about this?

Cheers,

Nicolas

Can you please show me the example of this in a playbook and how you are observing this behavior (playbook output in verbose mode, etc)?

Here is an example with this simple playbook:

  • hosts: all
    tasks:

  • copy: content=hello\nworld dest=/vagrant/testcopy.txt

I run it using:

$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -i hosts playbook.yml

And get:

TASK: [copy content=hello\nworld dest=/vagrant/testcopy.txt] ******************
failed: [127.0.0.1] => {“failed”: true}

msg: this module requires key=value arguments ([‘content=hello’, ‘world’, ‘dest=/vagrant/testcopy.txt’, ‘src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source’, ‘original_basename=tmpK0KdKX’])

I do not expect it to fail.

I had a look to the module uploaded to the server after replacement of MODULE_ARGS and here is its value:

MODULE_ARGS = “”“content=hello\nworld dest=/vagrant/testcopy.txt src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source original_basename=tmpK0KdKX”“”

I think this is incorrect. “\n” should be escaped like this:

MODULE_ARGS = “”“content=hello\nworld dest=/vagrant/testcopy.txt src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source original_basename=tmpK0KdKX”“”

or like this (by using a Python raw string prefixed with a “r”):

MODULE_ARGS = r"““content=hello\nworld dest=/vagrant/testcopy.txt src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source original_basename=tmpK0KdKX””"

Any advice?

Thanks for your help.

Nicolas

Ah ok, don’t use content in this case without quoting it, and you should be fine.

It’s also preferred you use “src” anyway, content was a recent addition.

Ah ok, don't use content in this case without quoting it, and you should
be fine.

It's also preferred you use "src" anyway, content was a recent addition.

The problem is not specific to module copy with the content arg. I used
this just as an example. The problem happens with all Python modules. When
args are "injected" in the Python module before being uploaded to the
server, they should be escaped but are not.

Here is another example with lineinfile:

- hosts: all
  tasks:
    - copy: content='hello' dest=/vagrant/test.txt
    - lineinfile: dest=/vagrant/test.txt regexp='^hello' line='hello\nworld'

After running this playbook, test.txt should contain :

$ cat test.txt
hello\nworld

Instead, I got:

$ cat test.txt
hello
world

The reason why is because "\n" is not escaped before being "injected" in
MODULE_ARGS and thus is interpreted by Python as a newline instead of just
two characters "\" and "n".

This behavior is undocumented, and I understand that Ansible tries hard to
not be Python specific. For most use cases, one should just need to write
YAML playbooks and use the built-in modules. But this small problem makes
the playbook implicitly dependent on Python literal string escaping rules.

What would you think of a patch trying to replace (in Runner._copy_file):

    encoded_args = "\"\"\"%s\"\"\"" % module_args.replace("\"","\\\"")

by something like this:

    encoded_args = repr(module_args)

An example to illustrate the difference:

$ python

module_args = r'regexp="hello\nworld"'
print "\"\"\"%s\"\"\"" % module_args.replace("\"","\\\"")

"""regexp=\"hello\nworld\""""

print repr(module_args)

'regexp="hello\\nworld"'

This is the end of this long message. Sorry!

Sure, how about sending me a pull request so we can test it out and make sure it doesn’t break anything?

Ok. I can give it a try, but I expect such a change to break some examples
of lineinfile that use backslash in regexp. Would you be ok to adapt the
examples if it's necessary (basically replacing double backslash by a
single backslash), or would it be a show stopper?

Backrefs are a pretty new feature (I believe 1.2) in lineinfile, so I’m ok with a patch that also updates them. Would be good if you can test those too if you can.

Thanks!

Done :slight_smile:

Here is the pull request: https://github.com/ansible/ansible/pull/3005

Hi,

I would disagree with Nicolas in that <something> escaping is not Python specific. Indeed, it seems to be theYAML-way
to escape control characters according to YAML (see: http://yaml.org/spec/current.html#id2517668). It is also the natural way
of regular expressions (http://www.regular-expressions.info/reference.html)

The command bellow should generate line breaks and tabs, and it used to work with previous versions of ansible, both from
command line or from a playbooks.

tasks:

  • copy: content=“hello\nworld” dest=“/some/file”
  • lineinfile: dest=/vagrant/test.txt regexp=‘^hello’ line=‘hello\tworld’

Recently, I upgraded ansible to 1.2.2 (pip) and 1.3 (git) and now it is broken. I’ve tried other forms of escaping (\x09, \u0009), both
with single and double quotes, and none have worked here. The escape sequence is always preserved in its literal form.

Is there another standard way of escaping control characters that I’m missing?
If not possible to revert to the previous (and correct in my PoV) behavior, I would appreciate any hints in how to do proper
do escaping of control chars in string parameters.

Best regards,
Luciano

Doesn’t seem likely to have changed:

https://github.com/ansible/ansible/commits/devel/library/files/lineinfile

I suspect you were using something earlier than 1.2.2 before maybe, back when it didn’t do backrefs.

Hi,

I think I’ve misread YAML specifications. Actually it only does expand escape sequences (\c, \x**, \u****) for double quoted scalar blocks.

So, while using double-quotes in the example bellow, actually, from PoV of YAML’s parser it is a plain scalar block, because
it does not start with double-quotes.

tasks:
copy: content=“beforetab\x09after” dest=“/some/file”

I’ve found two solutions for the problem:

  1. Convert value to double-quote scalar block (must escape any inner double-quotes in the block or replace by single-quotes):

tasks:
copy: “content=‘beforetab\x09after’ dest=‘/some/file’”

  1. Express copy parameters as a dict and use double-quotes for each value that need to use escape sequences.
    tasks:
    copy:
    content: “beforetab\x09after”
    dest: /some/file

However, I’m not sure solution 2 is officially supported or is a hidden feature of ansible. I was not able to find this
information in documentation currently available for playbooks.

BTW, CLI is also working. I forgot expansion of escape sequences does not occur depending on the used unix shell. So in
bash, for instance, one need to Ctrl+V to insert tab in string, instead of using \t. My mistake.

Best regards,
Luciano

Best to just transfer a file and not use “content” IMHO.

I would agree, except when you need to generate pretty small and somewhat formatted content on-the-fly. Ex.: passing some form of credentials
to remote node…

Anyway, I’m using lineinfile. I just used ‘copy’ in the example because I thought it would became simpler to explain the issue with YAML.

Thanks for your help,

s

Hi,