# .retry files clutter the user's $HOME directory **URL:** \ **Category:** Ansible Developer\ **Created:** [January 2, 2014, 4:02am UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248 "2014-01-02T04:02:46Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Tim\_Cuthbertson](https://avatars.discourse-cdn.com/v4/letter/t/9de053/32.png) [@Tim\_Cuthbertson](https://forum.ansible.com/u/Tim_Cuthbertson)\ **Post date:** [January 2, 2014, 4:02am UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/1 "2014-01-02T04:02:46Z")
I recently discovered a bunch of .retry files cluttering up my home directory, which is not very tidy: /home/tim $ ls -l \*.retry -rw-r–r–. 1 tim tim 15 Dec 19 10:46 all.retry -rw-r–r–. 1 tim tim 21 Dec 19 13:56 check.retry -rw-r–r–. 1 tim tim 14 Dec 19 10:13 deploy-local.retry -rw-r–r–. 1 tim tim 21 Dec 24 11:48 deploy.retry -rw-r–r–. 1 tim tim 14 Dec 19 09:56 post-deploy.retry -rw-r–r–. 1 tim tim 14 Dec 18 15:44 push-local-repos.retry -rw-r–r–. 1 tim tim 21 Dec 19 13:54 run.retry -rw-r–r–. 1 tim tim 21 Dec 19 11:02 system.retry I did a quick google, and found [https://github.com/ansible/ansible/issues/5260](https://github.com/ansible/ansible/issues/5260). I couldn’t find any continued discussion on the mailing list, so I’m starting one 🙂 @mpdehaan: you mentioned “access control is important” as a reason not to use a system-wide location. Are you able to expand on that? Presumably you can still choose whatever permissions you like for files created in /tmp/, so in what situations is a /tmp location not acceptable due to access control reasons? I don’t know what .retry files are used for, so please forgive me if the above is a dumb question. A better alternative location may be $XDG\_RUNTIME\_DIR, if defined. This is owned by the user, and not readable by anyone else. It will be cleared upon user logout (according to [http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html](http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html)). Like I said, I don’t know what .retry files are used for, so I can’t tell if that’s acceptable. But if ansible is going to create stuff in my home directory, I think it should at least be scoped appropriately (e.g in an `ansible` directory under ~/.config or ~/.local/share or ~/.cache or something), not an unbounded number of individual files. Cheers, - Tim. ---
**Author:** ![Jakub\_Warmuz](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jakub\_Warmuz](https://forum.ansible.com/u/Jakub_Warmuz)\ **Post date:** [January 2, 2014, 10:43am UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/2 "2014-01-02T10:43:17Z")
Indeed, those \*.retry files in ~ are pretty inconvenient. However, I would suggest using tempfile.mkstemp [1] or tempfile.mkdtemp [2]: "The default directory is chosen from a platform-dependent list, but the user of the application can control the directory location by setting the TMPDIR, TEMP or TMP environment variables." [1] [http://docs.python.org/2/library/tempfile.html#tempfile.mkstemp](http://docs.python.org/2/library/tempfile.html#tempfile.mkstemp) [2] [http://docs.python.org/2/library/tempfile.html#tempfile.mkdtemp](http://docs.python.org/2/library/tempfile.html#tempfile.mkdtemp) ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [January 2, 2014, 2:02pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/3 "2014-01-02T14:02:17Z")
A lot of people think they are pretty _convient_ to actually have them. I can understand if people would not like them always in your current directory, but they don’t clutter up home by default, they live alongside your inventory. “@mpdehaan: you mentioned “access control is important” as a reason not to use a system-wide location. Are you able to expand on that?” You don’t want someone on a multi-user system being able to change what hosts a playbook runs against, or leak information they might not have otherwise been able to see. Further, something shouldn’t be insecure by default and assume you have to set permissions on /tmp to do what you mean. “XDG\_RUNTIME\_DIR” Nobody mortal really knows what this is. Ultimately, the most logical choice would be to make the default where it currently is (inventory basedir if not defined), and to make the path configurable and disable-able (but stay on by default) retry\_file\_save\_path = ~/.ansible/ retry\_files\_enabled = True/False etc Pull requests for the above would be accepted. ---
**Author:** ![Tim\_Cuthbertson](https://avatars.discourse-cdn.com/v4/letter/t/9de053/32.png) [@Tim\_Cuthbertson](https://forum.ansible.com/u/Tim_Cuthbertson)\ **Post date:** [January 2, 2014, 10:39pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/4 "2014-01-02T22:39:57Z")
> A lot of people think they are pretty \*convient\* to actually have them. > I can understand if people would not like them always in your current > directory, but they don't clutter up home by default, they live alongside > your inventory. Perhaps there is a bug in this logic then, unless I am misunderstanding something? I've never had an inventory file in ~/, so it sounds like I shouldn't have had these .retry files appear in my home directory. If these files only appeared next to my inventory, I wouldn't mind. > "@mpdehaan: you mentioned "access control is important" as a reason not > to use a system-wide location. Are you able to expand on that?" > > You don't want someone on a multi-user system being able to change what > hosts a playbook runs against, or leak information they might not have > otherwise been able to see. Further, something shouldn't be insecure by > default and assume you have to set permissions on /tmp to do what you mean. Of course nothing should be insecure by default, and these files should not be able to be viewed / modified by other users. But as Jakub mentioned in the previous post, python's tempfile module provides all the security you require without any additional effort - from the docs for `mkdtemp`: "the directory is readable, writable, and searchable only by the creating user ID." A user lacking permissions on /tmp (or $TEMPDIR) should pretty much be classified as a broken system, so I wouldn't expect this case to be worth supporting. To be honest, it seems more likely that the \*inventory\* directory might be read-only, rather than /tmp being completely broken. > "XDG\_RUNTIME\_DIR" > > Nobody mortal really knows what this is. Aside from the hyperbole, does this matter? If retry files are simply used internally by ansible, why would the user need to guess / know where they're stored? ---
**Author:** ![kahlil.hodgson](https://avatars.discourse-cdn.com/v4/letter/k/b9bd4f/32.png) [@kahlil.hodgson](https://forum.ansible.com/u/kahlil.hodgson)\ **Post date:** [January 5, 2014, 5:02am UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/5 "2014-01-05T05:02:51Z")
I'm willing to have a shot at fixing this as per Micheal's suggestion: 1. fix the inventory base dir bug in generate\_retry\_invetory 2. setting for retry\_files\_enabled = True/False, default to True 3. setting for retry\_file\_save\_path = 'path' default to '~/.ansible/' Anyone else working on this? K Kahlil (Kal) Hodgson GPG: C9A02289 Head of Technology (m) +61 (0) 4 2573 0382 DealMax Pty Ltd (w) +61 (0) 3 9008 5281 Suite 1415 401 Docklands Drive Docklands VIC 3008 Australia "All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer." -- IBM maintenance manual, 1925 ---
**Author:** ![Tim\_Cuthbertson](https://avatars.discourse-cdn.com/v4/letter/t/9de053/32.png) [@Tim\_Cuthbertson](https://forum.ansible.com/u/Tim_Cuthbertson)\ **Post date:** [January 5, 2014, 6:38am UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/6 "2014-01-05T06:38:19Z")
> I'm willing to have a shot at fixing this as per Micheal's suggestion: > > 1. fix the inventory base dir bug in generate\_retry\_invetory > 2. setting for retry\_files\_enabled = True/False, default to True > 3. setting for retry\_file\_save\_path = 'path' default to '~/.ansible/' Doesn't fixing #1 make #3 unnecessary? i.e it sounds like ansible is \_supposed\_ to store the files next to the inventory dir, in which case there's no need for a configurable location. Well, I guess you could still have one, but it should default to \, rather than '~/.ansible/'. > > A lot of people think they are pretty \*convient\* to actually have them. > > I can understand if people would not like them always in your current > > directory, but they don't clutter up home by default, they live alongside > > your inventory. > > Perhaps there is a bug in this logic then, unless I am misunderstanding > something? I've never had an inventory file in ~/, so it sounds like I > shouldn't have had these .retry files appear in my home directory. If these > files only appeared next to my inventory, I wouldn't mind. > > > "@mpdehaan: you mentioned "access control is important" as a reason not to > > use a system-wide location. Are you able to expand on that?" > > > > You don't want someone on a multi-user system being able to change what > > hosts a playbook runs against, or leak information they might not have > > otherwise been able to see. Further, something shouldn't be insecure by > > default and assume you have to set permissions on /tmp to do what you mean. > Of course nothing should be insecure by default, and these files should not > be able to be viewed / modified by other users. But as Jakub mentioned in > the previous post, python's tempfile module provides all the security you > require without any additional effort - from the docs for `mkdtemp`: "the > directory is readable, writable, and searchable only by the creating user > ID." > > A user lacking permissions on /tmp (or $TEMPDIR) should pretty much be > classified as a broken system, so I wouldn't expect this case to be worth > supporting. To be honest, it seems more likely that the \*inventory\* > directory might be read-only, rather than /tmp being completely broken. > > > "XDG\_RUNTIME\_DIR" > > > > Nobody mortal really knows what this is. > > Aside from the hyperbole, does this matter? If retry files are simply used > internally by ansible, why would the user need to guess / know where they're ---
**Author:** ![kahlil.hodgson](https://avatars.discourse-cdn.com/v4/letter/k/b9bd4f/32.png) [@kahlil.hodgson](https://forum.ansible.com/u/kahlil.hodgson)\ **Post date:** [January 5, 2014, 11:21am UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/7 "2014-01-05T11:21:08Z")
I can think of shared user scenarios where you would not be able to write to the inventory base dir. Anyway, my quick implementation does the following if retry\_file\_save\_path is defined and can be written to, then use that else try the base dir of the inventory else try the user's home dir I'm choosing to fall back to alternatives, rather than throw an error, which seems the most flexible approach. Thoughts? Note that the retry path is printed at the end of the ansible-playbook run, so it should be pretty clear where it ends up. Also think it would be good to expand any environment variables in retry\_file\_save\_path, which should handle situations where TMPDIR and XDG\_RUNTIME\_DIR are meaningful/sensible options for the given environment. Have a working implementation, just need to figure out how to run tests in python. K Kahlil (Kal) Hodgson GPG: C9A02289 Head of Technology (m) +61 (0) 4 2573 0382 DealMax Pty Ltd (w) +61 (0) 3 9008 5281 Suite 1415 401 Docklands Drive Docklands VIC 3008 Australia "All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer." -- IBM maintenance manual, 1925 ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [January 5, 2014, 6:26pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/8 "2014-01-05T18:26:52Z")
We should make it configurable as per above. We must not have it pick alternatives, that’s unneccessary complexity. Expanding variables is fine. I checked and it’s written to use $HOME btw. We should probably default to $HOME/.ansible-retry/ and make sure it knows how to create a directory for organization purposes, for when it’s not configured. Make it so! ```auto ``` ---
**Author:** ![kahlil.hodgson](https://avatars.discourse-cdn.com/v4/letter/k/b9bd4f/32.png) [@kahlil.hodgson](https://forum.ansible.com/u/kahlil.hodgson)\ **Post date:** [January 6, 2014, 10:43pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/9 "2014-01-06T22:43:23Z")
Yeah, I agree that picking alternatives is unnecessary complexity -- after all, I had to use psuedo code to describe it 🙂 I've created the following pull request which I believe implements the above     [https://github.com/ansible/ansible/pull/5515](https://github.com/ansible/ansible/pull/5515) Just to clarify, we want to create the $HOME/.ansible-retry directory only if retry\_files\_save\_path is unchanged from the default ($HOME/.ansible-retry) . If the user changes the default, it is up to them to create the corresponding directory. This make sense to me: we \_don't\_ want typos in the config file causing ansible to pollute the filesystem, but we \_do\_ want it to just work out of the box. If ansible can't create the retry file, a (non-fatal) warning is now generated that mentions the configuration settings. Previously, ansible would just quietly ignore the error. K Kahlil (Kal) Hodgson GPG: C9A02289 Head of Technology (m) +61 (0) 4 2573 0382 DealMax Pty Ltd (w) +61 (0) 3 9008 5281 Suite 1415 401 Docklands Drive Docklands VIC 3008 Australia "All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer." -- IBM maintenance manual, 1925 ---
**Author:** ![Jakub\_Warmuz](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jakub\_Warmuz](https://forum.ansible.com/u/Jakub_Warmuz)\ **Post date:** [February 6, 2014, 12:30pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/10 "2014-02-06T12:30:26Z")
I allowed myself to made some comments under the pull request. Please have a look at it. ---
**Author:** ![kahlil.hodgson](https://avatars.discourse-cdn.com/v4/letter/k/b9bd4f/32.png) [@kahlil.hodgson](https://forum.ansible.com/u/kahlil.hodgson)\ **Post date:** [February 6, 2014, 11:07pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/11 "2014-02-06T23:07:50Z")
Dude, you are so gonna get your hand slapped for commenting in git ticket rather than on the mailing list 😉 Comments about \_how\_ the PR is implemented (bugs, questions, alternative approached) are usually fine because they're usually only of interest to the submitter. Comments about \_what\_ the PR is implementing definitely belong here really need to get the wider views of the developer community. It would be good if you could collate you questions and present them here so I can comment. I definitely have responses, but I think I should wait till the end of the day when I've had a chance to down a few beers. K Kahlil (Kal) Hodgson GPG: C9A02289 Head of Technology (m) +61 (0) 4 2573 0382 DealMax Pty Ltd (w) +61 (0) 3 9008 5281 Suite 1415 401 Docklands Drive Docklands VIC 3008 Australia "All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer." -- IBM maintenance manual, 1925 ---
**Author:** ![Jakub\_Warmuz](https://avatars.discourse-cdn.com/v4/letter/j/65b543/32.png) [@Jakub\_Warmuz](https://forum.ansible.com/u/Jakub_Warmuz)\ **Post date:** [February 6, 2014, 11:24pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/12 "2014-02-06T23:24:14Z")
> Dude, you are so gonna get your hand slapped for commenting in git > ticket rather than on the mailing list 😉 Sorry everyone 🙂 > It would be good if you could collate you questions and present them > here so I can comment. As per request... I altered bits here and there, but general idea stays the same. \*Retry files are not essential, they are just a convenience\*. I agree that Ansible should work out of the box, but I do not agree that it should clutter up user's machine out of the box... Also, I claim that people would rarely use them, personally I have never used it. And, by the way, consider the following issues (that actually might require separate bug reports):   1. What if Ansible runs off a read-only file system?   2. From the security point of view, I'm a bit concerned about issues      arising from disclosing list of the hosts (and potentially last      failed task in the future) in a directory that we do not chmod      properly. I understand it would be a backward-incompatible change, but I'd recommend changing the default to \*not creating retry files at all\*. Regarding the destination of the \*.retry files there are couple of things I'd like to discuss, some of which might also require general design decision changes accepted by the community:   1. Extending the argument from above, retry files are just temporary      stuff. What about \*using Python's tempfile module instead\* [1]? It      provides sane security defaults, is cross system, doesn't clutter      up $HOME, etc. Also, with the current patch approach, it's not      possible to achieve this random file destination functionality,      whereas it would be easy to reset it to $HOME/.ansible-retry      easily.   2. What if $HOME is not defined? etc. 😉 We should not rely on the      environment too much.   3. If you do not agree with the first point, then maybe let's at      least create ~/.ansible/ for all Ansible related stuff, including       ~/.ansible/retry, or ~/.ansible/ansible.cfg, and so on...      ANSIBLE\_REMOTE\_TEMP is already set to "$HOME/.ansible/tmp"      (although as with retry files I would use tempfile module instead). Happy to hear your comments! [1] [http://docs.python.org/2/library/tempfile.html](http://docs.python.org/2/library/tempfile.html) ---
**Author:** ![Ian\_Heggie](https://avatars.discourse-cdn.com/v4/letter/i/85f322/32.png) [@Ian\_Heggie](https://forum.ansible.com/u/Ian_Heggie)\ **Post date:** [July 19, 2014, 2:02pm UTC](https://forum.ansible.com/t/retry-files-clutter-the-users-home-directory/14248/13 "2014-07-19T14:02:02Z")
RE: > _Retry files are not essential, they are just a convenience_. Actually retry files may be the cleanest way I have of discovering which systems didn’t update and automatically replaying them. SO … being able to control which directory they end up in will make a difference for me. (I am thinking of symlinking the playbook to something-NN so the program can verify the retry file belongs to it and not some other process using the same playbook - this would get me around the shared name space issue) regarding defaults, I suggest you consider a couple of standard places, eg $HOME/.ansible-retry, inventory-dir/.ansible-retry and last TMPDIR/.user-ansible-retry - look in all three places to see if a directoy exists - if so use it, if not try each location in turn to see if you can make the directoy - the first one wins. And then allow an ansible.cfg and/or ENV variable to specify the exact path (or a list of paths to try?)