# Synchronizing directory structures **URL:** \ **Category:** Get Help\ **Tags:** playbook, ansible-core\ **Created:** [August 9, 2024, 1:37pm UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761 "2024-08-09T13:37:03Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![upsj](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/upsj/32/4424_2.png) [@upsj](https://forum.ansible.com/u/upsj)\ **Post date:** [August 9, 2024, 1:37pm UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/1 "2024-08-09T13:37:03Z")
Hi all, I am trying to figure out a nice way to synchronize a directory structure between my ansible repository and my nodes, but it seems like there is no right tool for the job. My requirements are a clean interaction with password-authenticated `become`, control over owner, group and file/directory mode on the one hand, and deletion of files that are not present on the host, on the other hand. The latter seems [unwanted inside `builtin.copy`](https://github.com/ansible/ansible/issues/4899), the former is not supported with `posix.synchronize` (which seems to not have changed since [Dan’s Cheat Sheet](https://cheat.readthedocs.io/en/latest/ansible/synchronize.html) on the topic was written). Is there a possibility I’m missing? Would it be worth trying to revisit the request for a `delete` flag for `builtin.copy`? I’d also be happy to take a stab at the implementation, if the answer is positive. ---
**Author:** ![bcoca](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bcoca/32/9269_2.png) [@bcoca](https://forum.ansible.com/u/bcoca)\ **Post date:** [August 9, 2024, 3:23pm UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/2 "2024-08-09T15:23:51Z")
We are very unlikely to add delete to copy, I still recommend either using the existing synchronize module or have ansible setup and trigger rsync/cdist/syncthing services. If none of the above works for you, i would recommend creating a role that does the steps you need (mostly copy + find + file actions). You can use this as an example [Ansible Galaxy](https://galaxy.ansible.com/ui/standalone/roles/bcoca/tidy/), while it does not copy/sync, it handles removing unwanted files. Note that a role or just the copy module will be much slower than the dedicated software i mention above. ---
**Author:** ![upsj](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/upsj/32/4424_2.png) [@upsj](https://forum.ansible.com/u/upsj)\ **Post date:** [August 11, 2024, 5:50am UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/3 "2024-08-11T05:50:22Z")
I looked through the options, and it looks to me like neither of them are up to the task - rsync/cdist/syncthing requires additional services that I’d like to avoid for this seemingly simple task - the Ansible galaxy example requires listing the files explicitly, which will become very hard to do in code once we move to a deeper directory layout - copy doesn’t support deletion - synchronize doesn’t work with become and can’t control permissions/owners Despite these limitations, I think the right place to support this feature would be `copy`, since it’s closest to supporting all features. I looked at the code briefly, and I believe it should be possible with very little work thanks to `filecmp.dircmp`. So if you say “very unlikely to add”, do you mean it’s unlikely somebody will work on it, or it’s likely that a PR adding the feature would be rejected outright? ---
**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\ **Post date:** [August 11, 2024, 7:15am UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/4 "2024-08-11T07:15:51Z")
> [@upsj](#): > > So if you say “very unlikely to add”, do you mean it’s unlikely somebody will work on it, or it’s likely that a PR adding the feature would be rejected outright? I cannot speak for the core team, but my guess is that it would be likely rejected since it increases the complexity of the `copy` action quite a bit (which is already pretty complex). > [@upsj](#): > > I looked at the code briefly, and I believe it should be possible with very little work thanks to `filecmp.dircmp`. I think this would be a rather inefficient solution, since you first have to copy everything over to the target to a temp directory. It would be more efficient to first run a module which retrieves the file/directory list on the target with some information, maybe already passing some information in from the controller (like file / directory tree with file sizes so that the module can determine for which files a checksum should be computed), then decide which files to actually copy, and only copy these and then do the sync on the target. This is a bit similar to how rsync works, except that you cannot work with a real RPC protocol (like which rsync uses), but have to do it with a fixed number of transfers (the module invocations) and some file transfers inbetween. ---
**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\ **Post date:** [August 11, 2024, 7:22am UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/5 "2024-08-11T07:22:00Z")
> [@upsj](#): > > synchronize doesn’t work with become and can’t control permissions/owners It can control permissions/owners (because rsync can do so, and you have the [`rsync_opts` option](https://docs.ansible.com/ansible/devel/collections/ansible/posix/synchronize_module.html#parameter-rsync_opts) - it’s not very straightforward to use, though). Also it does work with some `become` methods (actually only with `sudo`). The main disadvantage to ansible.posix.synchronize is that it only works with a fixed set of connections, with one specific `become` method (and it ignores most of the become method’s options), and it requires rsync to be set up. ---
**Author:** ![upsj](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/upsj/32/4424_2.png) [@upsj](https://forum.ansible.com/u/upsj)\ **Post date:** [August 11, 2024, 7:34am UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/6 "2024-08-11T07:34:17Z")
> Also it does work with some `become` methods (actually only with `sudo`). I don’t think rsync supports sudo with password, does it? That’s the main issue I was facing. ---
**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\ **Post date:** [August 11, 2024, 7:55am UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/7 "2024-08-11T07:55:46Z")
It does not support that, it only supports prepending `sudo` or `sudo -u user` to the rsync command. (I’ve been succssfully using it with password-less sudo in the past.) That’s more a restriction of rsync though, I think, than one of ansible.posix.synchronize. Edit: ah I thought you were talking about ansible.posix.synchronize, not about rsync. Yes, rsync does not support sudo with password (at least to my knowledge). ---
**Author:** ![bcoca](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bcoca/32/9269_2.png) [@bcoca](https://forum.ansible.com/u/bcoca)\ **Post date:** [August 12, 2024, 1:51pm UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/8 "2024-08-12T13:51:32Z")
We have rejected PRs that add this to copy … if it were possible I would even remove/revert the ‘recurisve’ code in copy. Not only does it require 2 implementations, but the inherit complexity has been a source of many bugs over the years and is incredibly slow as Ansible itself is not designed to handle replicating data efficiently. The role I linked was an example, yes you need to pass the files to it, but using it as a basis and using the `find` action you could: - create the list and permissions of files to copy - create the list of existing files and permissions on target - use copy/file actions operating on the differences of both lists Put this in a role and you probably have the solution you want , this basically recreates what the other tools do. But this will not perform well, rsync, cdist and solutions like it have been extremely optimized to handle this use case, like being able to very efficiently detect, compute and only copy the differences. ---
**Author:** ![upsj](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/upsj/32/4424_2.png) [@upsj](https://forum.ansible.com/u/upsj)\ **Post date:** [August 12, 2024, 2:04pm UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/9 "2024-08-12T14:04:11Z")
Thanks for the thorough explanations! Even if it is disappointing to hear, it’s understandable from a maintenance standpoint. Maybe getting `sudo` support with password into `rsync` could be a more realistic approach. ---
**Author:** ![bcoca](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bcoca/32/9269_2.png) [@bcoca](https://forum.ansible.com/u/bcoca)\ **Post date:** [August 12, 2024, 2:09pm UTC](https://forum.ansible.com/t/synchronizing-directory-structures/7761/10 "2024-08-12T14:09:23Z")
The synchronize action started as a naive implementation, complicated by a lot of updates that fixed ‘that case’ but not in good generic ways, that is why it only works with sudo/no password and has many ‘particularities’. You can have from 2 to 5 different servers involved and the way rsync itself works makes it hard to use become on ‘both sides’ of the transfer. I had thought of a solution, but I never had time to implement it. We can create 3 different actions to replace it: sync\_to, sync\_from, sync\_between. This would both make the code for each MUCH simpler and make it behave much better with other plugins (connection/become).