Hi,
I’m currently using the community.aws.aws_ssm connection plugin. It seems quite slow. I think it’s initiating a new connection to the target machine for each file transfer or command execution. I was expecting Ansible to only create one session per host per play. I want to speed it up by re-using the session across all tasks in the play (per host).
I’m assuming this is a development problem (i.e. I have to submit a pull request for aws_ssm) not a usage problem. Hence why I’m emailing the development list. If it is a usage problem, I’m happy to move this to another mailing list.
I can see documentation about ControlMaster and other options specific to SSH, to be passed to the SSH client. Those don’t seem generalisable to other protocols.
The aws_ssm module itself already has _connect(), exec_command() and close(). So it seems like it’s already structured in the right way. But I can see from running -vvvv and adding print statements to Ansible itself that Ansible core is calling _connect() and close() multiple times per task.
Is there an easy way to make Ansible re-use the Connection object? (e.g. add a property to the aws_ssm connection class. Or subclass it from something else.) Is this what “persistent” connections are?
There is ansible.netcommon.persistent connection. “This is a helper plugin to allow making other connections persistent.” That sounds like what I want. But how do I use it? There’s no other documentation about what that plugin does or how to use it, other than specific arguments.
The general Connection plugins documentation says “only one can be used per host at a time”. I don’t understand how the persistent plugin can help other plugins if only one plugin can be used at a time. Is the persistent plugin one that shouldn’t be used by the end user, but rather it’s an internal library that should be subclassed when implementing other connection plugins? Looking at the code of the persistent plugin, it sounds like it’s actually a plugin for using sockets instead of other transport. So I’m not sure why it’s called persistent.py instead of socket.py. Am I supposed to modify aws_ssm.py to create a socket and somehow pass that to the persistent plugin?
aws_ssm currently subclasses ansible.plugins.connection.ConnectionBase (here). That comes from connection/init.py, which also has a NetworkConnectionBase class. I tried modifying aws_ssm to subclass from this instead of ConnectionBase. This gives me an error when running:
Unexpected failure during module execution: ‘Requested entry (plugin_type: connection plugin: my_better_ssm setting: persistent_log_messages ) was not defined in configuration.’
After looking at how grpc is implemented (ssm is not related to grpc. I just wanted an example that wasn’t SSH), I added the following to the docstring:
extends_documentation_fragment:
- ansible.netcommon.connection_persistent
Now I get:
Unexpected failure during module execution: ‘Connection’ object has no attribute ‘nonetype’
I’ve added some more print statements to see what’s going on. Ansible core is calling getattr of the connector with name=“nonetype” (as a string, not actually NoneType or None).
I’m trying to see what’s calling this. With epdb I see that the traceback at the point the exception is thrown is:
/Users/matthew/.pyenv/versions/3.10.0/bin/ansible-playbook(8)()
→ sys.exit(main())
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/cli/playbook.py(227)main()
→ PlaybookCLI.cli_executor(args)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/cli/init.py(647)cli_executor()
→ exit_code = cli.run()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/cli/playbook.py(143)run()
→ results = pbex.run()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/playbook_executor.py(190)run()
→ result = self._tqm.run(play=play)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/task_queue_manager.py(333)run()
→ play_return = strategy.run(iterator, play_context)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/plugins/strategy/linear.py(243)run()
→ self._queue_task(host, task, task_vars, play_context)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/plugins/strategy/init.py(391)_queue_task()
→ worker_prc.start()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/process/worker.py(93)start()
→ return super(WorkerProcess, self).start()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/multiprocessing/process.py(121)start()
→ self._popen = self._Popen(self)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/multiprocessing/context.py(277)_Popen()
→ return Popen(process_obj)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/multiprocessing/popen_fork.py(19)init()
→ self._launch(process_obj)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/multiprocessing/popen_fork.py(71)_launch()
→ code = process_obj._bootstrap(parent_sentinel=child_r)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/multiprocessing/process.py(315)_bootstrap()
→ self.run()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/process/worker.py(126)run()
→ return self._run()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/process/worker.py(170)_run()
→ ).run()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/task_executor.py(158)run()
→ res = self._execute()
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/task_executor.py(560)_execute()
→ plugin_vars = self._set_connection_options(cvars, templar)
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/task_executor.py(1087)_set_connection_options()
→ varnames.extend(self._set_plugin_options(plugin_type, variables, templar, task_keys))
/Users/matthew/.pyenv/versions/3.10.0/lib/python3.10/site-packages/ansible/executor/task_executor.py(1005)_set_plugin_options()
→ plugin = getattr(self.connection, '%s’ % plugin_type)
/Users/matthew/Documents/mms/fastssm/connection_plugins/my_better_ssm.py(488)getattr()
→ return super().getattr(name)
It looks like the “nonetype” thing comes from somewhere here:
deals with networking sub_plugins (network_cli/httpapi/netconf)
sub = getattr(self._connection, ‘_sub_plugin’, None)
if sub is not None and sub.get(‘type’) != ‘external’:
plugin_type = get_plugin_class(sub.get(“obj”))
My hunch is that I need to specify some config in ansible.cfg, or the host vars, or the docstring up the top of the aws_ssm.py file.But if there’s a missing property somewhere, I would expect a KeyError about the property itself, not something related to “nonetype” as a string.
I also think the error is related to _sub_plugin={}. I can see that grpc.py sets something for that, and references a “subplugin” folder. I’ve never heard of sub plugins before. I’m not sure if that’s a red herring, or if I’m supposed to put an object containing the low level connection inside that.
I’m quite lost now. How do I get Ansible to only call _connect once per host per play?
Regards,
Matt