Suggestions for win_copy and win_ansible

Trond_Hindenes · October 13, 2014, 10:06pm

There’s been a lot of talk about these, and its time to get them into source.

As several (I can’t remember the names right now, sorry!) in this group have tested, the copy module will work (albeit with limited functionality) as long as there’s a copy.ps1 module in the windows directory, and that file is able to lay down the input file onto disk.

My suggestion is that we build win_copy and win_template modules, in order to reflect the differences around permissions on NTFS vs linux filesystems.
From what I can see, Ansible uses a 1:1 mapping between action plugins and modules, which would mean that we would need a win_copy.py and win_template.py action plugin as well.

The win_copy.py action plugin would be 100% identical to its copy.py sibling, and the win_template.py plugin would be identical to the template.py plugin with the only difference being that it calls win_copy instead of copy when executing the file copy operation(s) underneath.

my suggestion is that we build out simple support for copying/templating files using the matching modules, since the lack of these functions are limiting the usefullness of Ansible on the Windows platform. Once we have those in source we can start building out support for permissions etc.

I am more than willing to do this tweaking, but I’d like Michael (or someone’s) take on this path before i mash together (or steal) the code.

What say ye?

-Trond

Chris_Church1 · October 14, 2014, 4:05am

Given the differences in specifying permissions and no need for some of the
other file/copy options, I'm +1 for having separate
win_file/win_copy/win_template modules.

So, to add to your notes, I think the following changes are needed to get
all of this working. I'm currently tackling some kerberos-related things,
but would gladly offer my feedback if you wanted to submit a PR or point me
to a branch of work in progress.

In ansible:

   - lib/ansible/runner/action_plugins/win_copy.py would be mostly based on
   copy.py.
   - lib/ansible/runner/action_plugins/win_template.py would be mostly
   based on template.py. Somebody with initials CC should probably fix this
   todo while we're at it:
   https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/action_plugins/template.py#L79
   - lib/ansible/module_utils/powershell.ps1 - Add common functions (MD5,
   etc.) needed for win_file and win_copy modules.
   - test/integration/... - Integration tests for
   win_file/win_copy/win_template.

In ansible-modules-core:

   - windows/win_file.ps1 could be based on
   https://gist.github.com/tkinz27/8a0b49f3c8b0ea4575c6 without the
   permissions code. I'd think only the path and state parameters are needed;
   state only needs to support file, directory, touch and absent.
   - windows/win_file.py would define module docs.
   - windows/win_copy.ps1 could be based on
   https://gist.github.com/tkinz27/fd92ba9af0e0309614ee but use PowerShell
   3 compatible MD5 code from
   https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/shell_plugins/powershell.py#L87
   or use a common function for MD5.
   - windows/win_copy.py would define module docs.
   - windows/win_template.py would define module docs.

The permissions code from
https://gist.github.com/tkinz27/8a0b49f3c8b0ea4575c6 could probably become
the basis for a future win_acl module or be added to
win_file/win_copy/win_template in the future.

Trond_Hindenes · October 14, 2014, 7:00am

Great stuff! I think we’re on the same page here.

There are some problems I see with my win_template (copied from template and changed the referenced command from copy to win_copy). It seems to work nicely if the dest file doesn’t already exist, but if it the dest file exists it throws some weird error saying it’s missing some core modules. I’ll try and branch out my changed files so that someone in the know can have a look.

In the meantime, I’ll get going with win_copy and win_template powershell scripts, making sure to test them on PS v3.

Trond_Hindenes · October 14, 2014, 4:08pm

I’ve started working on this at:
https://github.com/trondhindenes/ansible/tree/win_copy_win_template
and
https://github.com/trondhindenes/ansible-modules-core/tree/win_copy_win_template

Appreciating any help!
-Trond

j.r.hawkesworth · October 19, 2014, 8:08pm

Thanks for this I hope to get some testing time this week. If I do I will give these a try - completely agree on the need for these.

Best

Jon

John_Jelinek · November 8, 2014, 1:57am

Has there been any progress on this?

j.r.hawkesworth · November 8, 2014, 3:22pm

A little. I tried out Trond’s changes and sent him some revisions last week. I had a functioning but incomplete win_copy module, but no tests yet and haven’t actually tried the win_template functionality.

I’m hoping to do a bit more on this as I get time as I have a functioning win_unzip module which I want to contribute. It works well but cannot yet copy files from the controller to the host.

Jon

Joey · November 19, 2014, 7:02pm

Trying out win_copy and it looks like Get-FileHash requires Powershell 4. Do the configure_remoting and upgrade_powershell scripts need to be updated to require Powershell 4? Or is there another way of doing the file hash that doesn’t require updating?

Joey · November 19, 2014, 7:18pm

Answering my own question, this seems to work.

function md5hash($path)
{
    $fullPath = Resolve-Path $path
    $md5 = new-object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
    $file = [System.IO.File]::Open($fullPath,[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read)
    [System.BitConverter]::ToString($md5.ComputeHash($file))
    $file.Dispose()
}

Joey · November 19, 2014, 7:34pm

After switching to the new md5 hash function, win_copy is partially working for me. If the md5 match fails and it has to write a new file, it works, but breaks most whitespace in the file (specifically it seems to remove linebreaks and tabs).

If the md5 match succeeds and it doesn’t have to write the file, there is a WINRM error and it fails. The error looks like the WINRM PUT command are writing out to ansible-tmp-\file, and the WINRM EXEC command is looking for ansible-tmp-\file.ps1 (with the extension), so it errors out.

j.r.hawkesworth · November 19, 2014, 9:10pm

This is probably because behind the scenes it is attempting to use the file module to get info about the file if the md5 match succeeds.

I have been making progress on implementing win_copy, win_template and win_file and hope to have a PR to share very shortly - I will also include integration tests which I have passing for win_copy and win_file right now.

Jon

Chris_Church1 · November 19, 2014, 10:40pm

Very nice, thanks for your work on this!

Regarding the winrm connection adding a .ps1 extension, I may be able to
remove that part entirely if we officially require windows modules to have
the .ps1 extension. At the moment, it still searches for module.ps1 then
module (no extension).

j.r.hawkesworth · November 23, 2014, 7:41pm

Sorry this is taking longer than I hoped. I am debugging my through the consequences of the changes around md5 and checksum logic at the moment. With a bit of luck and a couple of clear hours I should have a PR to share. Just hope I can get something in before 1.8 sails.

Jon

j.r.hawkesworth · November 24, 2014, 9:08pm

Just to say at long last I have created PRs to add win_copy, win_file and win_template modules.

Both PRs would be needed (the modules all need action_plugins, which are in the main ansible project).

If anyone has the chance to review these and try them out I would be very grateful. Right now I don’t have a windows server 2008 /2008 r2 machine I can run against so if any one could try the integration tests against that, that would be really helpful.

The PRs are:

https://github.com/ansible/ansible/pull/9611

https://github.com/ansible/ansible-modules-core/pull/384

All the best,

Jon

Joey · November 24, 2014, 10:22pm

How do I run the integration tests? I’m trying out win_template but it does the checksum and decides it doesn’t need to do anything even though it hasn’t updated the file. I’m trying to find the flag that keeps debug files on the client so I can look at the temp ps1 files and figure out why.

Joey · November 24, 2014, 10:33pm

Trying to run the script in PowerShell ISE directly on the server gives this error:

`

PowerShell : Cannot find an overload for “CompareTo” and the argument count: “1”.
At line:1 char:1

PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -File "C:\Us …
CategoryInfo : NotSpecified: (Cannot find an …ent count: “1”.:String) , RemoteException
FullyQualifiedErrorId : NativeCommandError

At C:\Users\localadmin\AppData\Local\Temp\ansible-tmp-1416867885.79-1510893868
84966\win_copy.ps1:223 char:9

If (! $src_md5.CompareTo($dest_md5))
CategoryInfo : NotSpecified: ( , MethodException
FullyQualifiedErrorId : MethodCountCouldNotFindBest

Cannot find an overload for “CompareTo” and the argument count: “1”.
At C:\Users\localadmin\AppData\Local\Temp\ansible-tmp-1416867885.79-1510893868
84966\win_copy.ps1:230 char:10

If ( $src_md5.CompareTo($dest_md5))
CategoryInfo : NotSpecified: ( , MethodException
FullyQualifiedErrorId : MethodCountCouldNotFindBest

{
“changed”: false,
“checksum”: [
“3688374325b992def12793500307566d”,
“”
]
}

`

Chris_Church1 · November 24, 2014, 10:37pm

I use --start-at-task or temporarily add my own tags if I’m trying to work out an issue with one particular task.

make test_win already adds -v, so you can specify -vvvv and you’ll be able to see the full stdout/stderr returned from trying to run the module.

j.r.hawkesworth · November 25, 2014, 11:22pm

Thanks for this, I’d add that in this case I wouldn’t recommend running win_copy win_file or win_template without the corresponding action plugins as the plugins are needed to

That said, I do debug powershell by setting ANSIBLE_KEEP_REMOTE_FILES=1 and then running the module.ps1 from inside Powershell ISE after a playbook run has completed or failed. One little issue to be aware of is you need to pass the name of the arguments file in without the .\ that powershell likes to put in place if you use tab completion.

So run
.\win_copy.ps1 arguments
not
.\win_copy.ps1 .\arguments

Jon

Janaka_Abeywardhana · December 5, 2014, 7:20pm

Hi,
What’s the latest with the win_copy module?

Thanks

Chris_Church1 · December 5, 2014, 7:53pm

There are 2 pull requests (one needs a rebase):
https://github.com/ansible/ansible/pull/9611 and
https://github.com/ansible/ansible-modules-core/pull/384 and some ongoing
discussion on implementation details:
https://groups.google.com/d/topic/ansible-devel/VRXSKQgEV70/discussion

Once those few minor issues are resolved and we have a chance to review and
test, should be merged into devel.

Topic		Replies	Views
Windows copy and file modules Ansible Developer windows	7	0	October 14, 2014
cannot get win_template to work Ansible Project windows	3	0	May 29, 2015
Newbie Question: Copying files from host to Windows target Ansible Developer ubuntu , windows	1	0	September 26, 2014
win_blockinfile module Ansible Developer windows	4	2	May 22, 2019
Ansible issues with Windows deployment Ansible Project ubuntu , windows	11	0	March 26, 2018

Suggestions for win_copy and win_ansible

Related topics