# Very Inefficient Remote Copy **URL:** \ **Category:** Ansible Project\ **Created:** [May 7, 2012, 3:43pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513 "2012-05-07T15:43:16Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Frank](https://avatars.discourse-cdn.com/v4/letter/f/3d9bf3/32.png) [@Frank](https://forum.ansible.com/u/Frank)\ **Post date:** [May 7, 2012, 3:43pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/1 "2012-05-07T15:43:16Z")
The copy module seams to always remote copy the file, event if it is not necessary. Example: Part of my play:   - name: copy installation archive file     action: copy src=files/foo.tar.gz dest=/var/cache/ansible/ foo.tar.gz owner=root group=root mode=0644 Output (first run):   TASK: [copy installation archive file] \*\*\*\*\*\*\*\*\*   ok: [myhost] =\> copy src=/var/tmp/ansible.4Po0nW/foo.tar.gz dest=/ var/cache/ansible/foo.tar.gz Output (second run - the dest file already exists before):   ok: [myhost] =\> copy src=/var/tmp/ansible.6xPjhj/foo.tar.gz dest=/ var/cache/ansible/foo.tar.gz Because the file is quite big it takes very long time! Similar behavior with "only\_if: False": it takes a long time (probably copying the file over the network) and then(!!!) it skips with the following output:   TASK: [copy installation archive file] \*\*\*\*\*\*\*\*\*   skipping: [myhost]   err: [myhost] =\> NoneNone Is this a bug of do I have to use it in a different way? Frank ---
**Author:** ![Will2](https://avatars.discourse-cdn.com/v4/letter/w/a587f6/32.png) [@Will2](https://forum.ansible.com/u/Will2)\ **Post date:** [May 7, 2012, 4:16pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/2 "2012-05-07T16:16:29Z")
I've noticed this as well (although I usually copy moderately small files) and it slows down re-running of plays rather dramatically. Also, as an addendum (could be another email thread) is recursive directory copy something that's in the works, or just not in the docs yet? Thanks! ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [May 7, 2012, 4:40pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/3 "2012-05-07T16:40:15Z")
Q1: Having it take a remote md5sum before deciding to do a remote copy if the file size is above a certain threshold is reasonable. (It’s already smart enough to not replace the file on the remote end, but that’s beside the point) Q2: Not in the works that I know of, would be nice Patches accepted for both. ---
**Author:** ![John\_Callender](https://avatars.discourse-cdn.com/v4/letter/j/b5ac83/32.png) [@John\_Callender](https://forum.ansible.com/u/John_Callender)\ **Post date:** [May 7, 2012, 6:01pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/4 "2012-05-07T18:01:58Z")
I believe you’ve previously thrown some cold water on the idea of incorporating rsync into ansible’s core functionality, and I don’t doubt that there are good reasons for that. It sure is a sweet tool when it comes to stuff like this, though. Given ansible’s push orientation, it would possibly be helpful for scaling purposes if it could incorporate some of rsync’s efficiency cleverness. I’m curious what the pluses and minuses of incorporating rsync into the file-transfer components of ansible would be from your perspective. Note that I’m not questioning your choice not to include it. I’m just curious what the tradeoffs were/are. ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [May 7, 2012, 6:06pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/5 "2012-05-07T18:06:34Z")
> Note that I’m not questioning your choice not to include it. I’m just curious what the tradeoffs were/are. Because it wouldn’t be using paramiko, and thus sudo and various other options would work in completely different ways. I expect people with a large number of files to use NFS, use packages, etc. ---
**Author:** ![Will2](https://avatars.discourse-cdn.com/v4/letter/w/a587f6/32.png) [@Will2](https://forum.ansible.com/u/Will2)\ **Post date:** [May 7, 2012, 8:09pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/6 "2012-05-07T20:09:18Z")
Could you give a bit more information by what you mean when you say "Use NFS" ? Are you thinking that people would mount up a share, then run a local copy command? Or some other method? ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [May 7, 2012, 10:03pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/7 "2012-05-07T22:03:48Z")
I (perhaps incorrectly) had assumed people would not be moving “large” files through Ansible, but I also maybe incorrectly come from the school where war files are ideally delivered via yum or git or NFS or … something else rather than something that resembles rsync. That all being said, optimizing the ansible copy routines to be better with large files is still fine. In the case of NFS, I’d generally expect files to be mounted in the right place, and the config files path to them, so it would be unnecessary to copy the data – the whole point of it being mounted on network storage and not using local disk. ---
**Author:** ![AD1](https://avatars.discourse-cdn.com/v4/letter/a/e5b9ba/32.png) [@AD1](https://forum.ansible.com/u/AD1)\ **Post date:** [May 7, 2012, 10:05pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/8 "2012-05-07T22:05:09Z")
I think he is saying the copy would happen once, and then be reflected on all servers mounting the share. ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [May 7, 2012, 10:40pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/9 "2012-05-07T22:40:12Z")
I think I’m saying Ansible isn’t involved – setup NFS first, but Ansible may configure the mount point. ---
**Author:** ![Stephen\_John\_Smoogen](https://avatars.discourse-cdn.com/v4/letter/s/ad7895/32.png) [@Stephen\_John\_Smoogen](https://forum.ansible.com/u/Stephen_John_Smoogen)\ **Post date:** [May 7, 2012, 10:48pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/10 "2012-05-07T22:48:38Z")
I think this has been an assumption that every configuration management system makes and then it runs quickly into the standard sysadmin item... yes I could go get the hammer but if I turn over this screwdriver it will work just as well. I have run into this with cfengine2 and early versions of puppet where the answers from the developer was usually the same "why would you want to do that?" and the answer went anywhere from "I get one tool at a time on my network.." to "well my systems are all over the world.. shared storage doesn't work well" to "well why did you give me a recurse option if you didn't want me to recurse" (my favorite one from the cfengine1 an cfengine2 days :)). ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [May 7, 2012, 11:46pm UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/11 "2012-05-07T23:46:54Z")
LOL… yeah. Hammer analogy is well put. Puppet’s early fileserver encased transmitted files in XML, so I think we’ve done at least better than that. For multiple geo’s, I’d probably propose something like NetApp snap mirror, etc, if it gets really crazy 🙂 Early server preview shows we are working with different levels of systems being managed – some people have 10, others hundreds. The way you do things if you have 10 and one location is going to be different than if you have 10 locations or even 100 in one. I think it’s usually best to talk in terms of use cases, if we can better understand what you are trying to do, the best solution usually ends up showing itself. (And it won’t always be Ansible in terms of pushing content, but Ansible can have a role to play). –Michael ---
**Author:** ![Stephen\_John\_Smoogen](https://avatars.discourse-cdn.com/v4/letter/s/ad7895/32.png) [@Stephen\_John\_Smoogen](https://forum.ansible.com/u/Stephen_John_Smoogen)\ **Post date:** [May 8, 2012, 12:18am UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/12 "2012-05-08T00:18:35Z")
> I (perhaps incorrectly) had assumed people would not be moving "large" files > through Ansible, but I also maybe incorrectly come from the school where war > > I think this has been an assumption that every configuration > management system makes and then it runs quickly into the standard > sysadmin item... yes I could go get the hammer but if I turn over this > screwdriver it will work just as well. I have run into this with > cfengine2 and early versions of puppet where the answers from the > developer was usually the same "why would you want to do that?" and > the answer went anywhere from "I get one tool at a time on my > network.." to "well my systems are all over the world.. shared storage > doesn't work well" to "well why did you give me a recurse option if > you didn't want me to recurse" (my favorite one from the cfengine1 an > cfengine2 days :)). > > LOL… yeah. Hammer analogy is well put. > > Puppet's early fileserver encased transmitted files in XML, so I think we've > done at least better than that. For multiple geo's, I'd probably propose > something like NetApp snap mirror, etc, if it > gets really crazy 🙂 > > Early server preview shows we are working with different levels of systems > being managed -- some people have 10, others hundreds. > > The way you do things if you have 10 and one location is going to be > different than if you have 10 locations or even 100 in one. Well I was more of trying to explain why I see this question come up over and over again in various remote configuration tools. This conversation has identical twins from cfengines early days. Hard experience from cfengine and early puppet has made it clear that if I want to push files.. put them in an rpm and use yum (or deb and apt or tgz or whatever). My use case is if I need to mirror large sets of data, I will use ansible like I would func.. call a helper script which rsync's it from a tree. If there ever becomes a python-rsync module that uses paramiko then maybe I would look at using it inside of ansible. Until then, I will find my hammer and use the screwdriver as a chisel like god intended :). ---
**Author:** ![Seth\_Vidal](https://avatars.discourse-cdn.com/v4/letter/s/7ba0ec/32.png) [@Seth\_Vidal](https://forum.ansible.com/u/Seth_Vidal)\ **Post date:** [May 8, 2012, 2:30am UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/13 "2012-05-08T02:30:14Z")
I was thinking about this some for the playbook case. Would it be possible to run a command (like setup) but after the playbooks tasks are parsed that's gone through and expanded out all the files it will be impacting per host and generate their checksums? I guess that could get expensive/heavy if we're recursing... Anyway - I was just thinking - then we're doing the md5sums for a whole range of files (copy or template for example) once and getting back one output to parse, vs doing many connections. -sv ---
**Author:** ![Frank](https://avatars.discourse-cdn.com/v4/letter/f/3d9bf3/32.png) [@Frank](https://forum.ansible.com/u/Frank)\ **Post date:** [May 8, 2012, 6:47am UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/14 "2012-05-08T06:47:39Z")
If I'm forced to use NFS or an own package repo, then in my opinion this violates the simple and secure infrastructure idea "Ansible just uses SSH". ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [May 8, 2012, 10:44am UTC](https://forum.ansible.com/t/very-inefficient-remote-copy/11513/15 "2012-05-08T10:44:12Z")
I haven’t forced anyone to do anything. If people would like to upgrade the md5 handling, submit a patch.