Idempotency and cleanups

I’ve tried to research the subject a bit via Google searches but nothing useful came up. So here I am, seeking collective wisdom.

Hypothetical scenario: using ansible playbook I set up machines A, B and C with services s1 (A), s2 (A), s3 (B), s4 (C). Playbook allows for modification of “hosts” so that s2 can be moved over to B. Which is fine - just adding B into appropriate group would do that trick and after next run of playbook service would end up where I want it to be. However now we have a ghost s2 on A that has not been removed. To have that cleanup done - I have to write a very targeted (perhaps non-idempotent) playbook to deal with “left-overs” (for example: packages installed, config files present on FS, etc.)

I’m sure this topic had come up previously, and will appreciate pointer to the archives as much as any other bits of information.

No config system really handles “this shall not be X”.

FWIW, this nothing to do with “idempotence”. It’s unfortunately one of the most misued words, and I blame Puppet/CFengine for making it so popular.

Idempotence is the mathematical property that F(X) = F(F(X)), such that repeated actions applied to the same hosts achieve the same state as applying them the first time.

If you are using cloud hosts, it’s easier to destroy the hosts, and that’s the generally accepted way to do it.

No config system really handles “this shall not be X”.

sounds like NixOS is doing that. Granted their config management is built into the OS giving it some advantages in tracking state.

FWIW, this nothing to do with “idempotence”. It’s unfortunately one of the most misued words, and I blame Puppet/CFengine for making it so popular.

perhaps wrong choice of words - I haven’t come across something that would describe what I’m looking for quite precisely.

If you are using cloud hosts, it’s easier to destroy the hosts, and that’s the generally accepted way to do it.

sure, clouds with disposable OS’es - kind of tolerable. But even then - you don’t want to “re-format” your entire cloud just to move services from A to B? What if that is a chain move with more hosts involved?

P.S.
not trying to start a flame-war or point a finger of blame - just looking for some solution (or workaround).

One thing you can do is write each role in two modes ("apply" and "remove"), apply it to all hosts, and hope you don't have too many left-overs ;-).
You can select hosts where it should be applied by setting defaults inside a role for the "remove" case, and overwriting it in group_vars.
This of course makes your roles much more bulky (and full of state={{ this_roles_target_state }}), and the removal won't ever be 100% reliable.

Dnia pią, 23 maj 2014, 22:11:21 Dmitry Makovey pisze:

No config system really handles "this shall not be X".

sounds like NixOS is doing that. Granted their config management is built
into the OS giving it some advantages in tracking state.

What NixOS does is pretty much what Michael proposed - recreating things from scratch. Well, at least the Nix-managed parts of the OS, like most of /etc.
They built the whole new system side by side with the running one (the way packages are stored in Nix makes that possible), and then use an "activation script" to switch from the old config to the new (restarting services that changed, starting new ones, stopping unneeded ones etc.).
IIRC you can still put your own files in /etc (you just can't edit the Nix-managed ones) so that leaves you with some dirty state.

If you are using cloud hosts, it's easier to destroy the hosts, and that's
the generally accepted way to do it.

sure, clouds with disposable OS'es - kind of tolerable. But even then - you
don't want to "re-format" your entire cloud just to move services from A to
B? What if that is a chain move with more hosts involved?

If it's a chain move you'll need to orchestrate it somehow anyway, so just removing things is not enough.
Actually, you'd probably want to move some data, disable the service on old host, move the rest of the data, enable it in it's new home,
and only then purge it from the old one - that's not something Ansible is able to figure out by itself, it can only help with applying the actions.

P.S.
not trying to start a flame-war or point a finger of blame - just looking
for some solution (or workaround).

What you are asking for is really hard to do right (even handling configuration and state from removed packages is a huge problem, without getting into live parts and uptime :-)).
My proposed workaround (that you'll probably only apply to roles that will be moved like that) is at the top of this message. Hope it helps at least a bit :slight_smile:

The one article about NixOS I read supremely conflated the notion of package management with the idea of running other types of actions and resources on the system, and I think nearly all of us know the hell that would be managing a series of blobs in /opt that doesn’t actually have any dependency management. Beside the point.

“But even then - you don’t want to “re-format” your entire cloud just to move services from A to B? What if that is a chain move with more hosts involved?”

No, you wouldn’t reformat your cloud. You would spin up new nodes/groups and make the appropriate DNS adjustments. It’s only an issue if there is persistent data on those nodes you need to preserve, which is why it’s nice to use things like RDS, etc. Depends to the extend cloud is being used like just a hosted computer offering, or more as a platform.