Should we add ibm z/OS mainframe host info to the Ansible docs?

Project Discussions
, ,
1

We currently have a draft PR from the community to add some details about z/OS mainframe hosts to the Ansible (core and package) documentation, under the OS guides section.

While I’ve been here …ahem… for a while, this is the first new OS guide introduced in that time, so I wanted to ask the hive mind what you all think? PR comments of course welcome, but this discussion is about whether it’s a good idea to add this detail to the main docs.

To give my nickel opinion - I’m inclined to say yes, let’s add it. The PR has gotten a lot of work and reviews from IBM folks, so presumably they know the details. The only drawback I can think of is would this get unmaintained over time?

…which, I’d have to say… is probably the same risk for the existing BSD section there. That one got a boost from community recently but wasn’t updated technically for a couple of years prior to that. Maybe we add ‘community-maintained’ note to the top, similar to what we did for the install instructions outside of ‘pip/pipx’ to let readers know that most of the Ansible community would not be able to address bugs/problems with this new z/OS section.

3 Likes
2

@mattclay …as the core SC rep, gonna ping you here for thoughts since this new section would show up in core docs as well.

1 Like
3

Hello @samccann , thank you for reviewing the draft PR and bringing it to the Ansible Forum, please allow me to comment on how we came about this PR and to do so, you might have to have a brief understanding of mainframes if you don’t already and endure my long response :).

To start, we have been working on Ansible collections for nearly 5 years, dating back to ansible 2.9, working closely when Red Hat Partners, engineering and others. So what we have been doing for 5 years? Well we have 6 collections, adding 2 more this year, spanning over 60 modules, over a 100 playbooks in our repository and all our collections are certified content. We have made contributions back to the community, sometimes we worked with partner engineering and provided the fixes, other times we opened issues, highlighted the problem area and provided a patch, there were not many of those and once you read down a bit further you will see why.

You can think of mainframe sort of how you think of a virtualization stack, z/OS is the operating system which in the past went by other names (MVS, OS390, etc), since virtualization has been around since the mainframes existence (50+ yrs) many other operating systems run on the mainframe, z/OS UNIX System Services, Linux on Z (RHEL, SUSE), zVM, etc. When in z/OS UNIX a POSIX complaint port, we deal with directories and files as you would expect, when in z/OS we deal with volumes, data sets and members, slight different paradigm.

Our z/OS collections are designed to interact with z/OS, so when we pair up a module like ansible.builtin.copy to ibm.ibm_zos_core.zos_copy we tend to cover z/OS and z/OS UNIX but not every module needs parity, for example we don’t need a ansible.builtin.git module but we do need it to work correctly on z/OS UNIX , and that is where are heading after talking to @gundalow and @Steve_Fulmer. This first draft PR is just to level set the community that we are planning to contribute but not only on z/OS, we plan to be part of the community.

While mainframes may seem niche, they handle billions of transactions a day and we are increasingly narrowing the tech gap where we have Python, Go, Node.js and contributing upstream to over 100 other projects to support z/OS.

It is a legitimate concern to wonder if this would get unmaintained over time, all I can do is share with you that we have been here for 5 years and have no plans on going anyplace. The IBM mainframe business has been building over our ibm_zos_core collection, promoting Ansible as the automation strategy for all our mainframe clients, investing in Event Driven Ansible this year and doubled our investments.

To further level set, most of the communities modules work in z/OS UNIX, its only a handful that perform I/O operations like lineinfile, blockinfile, replace, etc that they don’t work correctly and where our dedicated team would be working to change this over time.

I will tag a few others who might have some added perspecive @allhart , @thedoubl3j @chynasan

If you made it all the way to the bottom, sincerely thank you :slight_smile:

5 Likes
4

At the spur of the moment:

I didn’t have a closer look and, anyway, I don’t know z/OS at all.

But if there’s already documentation about Using Ansible on Windows and BSD, I think it would be unfair to z/OS to not accept this PR.

So either remove the documentation for Windows and BSD, or also add z/OS.

Yes, I think that’s a good idea.

Don’t trust me! I might be wrong there, even very wrong. This is just what crossed my mind without thinking very much about it.

2 Likes
5

TL;DR: I’m -1 for this. IMO new technology-specific docs belong with their technology support collections. If there are changes needed to expand the capability of non-plugin docs in collections, that’s probably where the short-term focus should be.

Longer answer: Since collections became a thing, the Ansible Core team’s long-term goal has always been to shift most (possibly all) technology- and platform-specific code and docs out to collections via exec subsystem plugins. With that goal in mind, we’ve resisted most asks to include new technology-specific bits in core itself. Exec subsystems shouldn’t be nearly the size of the internals overhaul we’re wrapping up from the Data Tagging project, but to properly work with things like Windows, z/OS, and networking devices (without all the nasty hacks and band-aids each of those have required), it’ll be a bigger project that will redraw some of the lines and interactions between core plugin types

It may take a bit to get there, but collections are the future for this stuff, and if I’ve learned anything from the overhaul we’re just finishing, it’s that the bigger we make the core surface area we have to support, the harder it is to maintain or change anything. I have no issue with (at least temporarily) including some basic pointers to technology-specific collections in core docs, but anything beyond that seems like it’s working against direction we’re headed.

(all this said, it’s also unclear to me exactly what’s being asked for here- if it’s purely about the unversioned ecosystem docs, and that works for the direction those are headed, go for it)

1 Like
6

Hi, while I understand this point “new technology-specific docs belong with their technology support collections”.

Putting such doc in our z/OS collections doc would have users bouncing between sites, the doc was introduced here so those users who are using the builtin modules know what they need to do. Those users are z/OS UNIX, eg UNIX users, not z/OS users, for z/OS we fully expect they visit our collections technology-specific docs.

Our goal would be to eventually eliminate the need of this doc if the community accepts our contributions to the modules as others like Windows, AIX, etc have done. Our contributions would handle any of these requirements.

Would it help if we minimized the PR down to the bare minimum of what is needed to use the builtin modules with UNIX on z/OS?

7

It might be hard to have technology-specific docs in a specific collection if there are several collections addressing said technology. Where to put it? I don’t know much about z/OS collection(s), but there are (at least) 3 collections dealing with VMware vSphere. It would be difficult to decide where some general information about vSphere should be documented.

I think I understand why you don’t want to have this in the documentation directly. But, as said above, where should this be documented?

Do you think it would be a good idea to have something like https://github.com/ansible/ansible-zOS-documentation that’s pulled in into docs.ansible.com?

Anyway, to be fair there should be the same rules for Windows, BSD and z/OS IMHO

1 Like
8

Thanks everyone for the explanations. So very valid points made - technology docs are best with their technology collections, but on the flip side - this z/OS UNIX portion does seem quite like the BSD/Windows stuff.

(And yes, we don’t currently have a community docs home for docs that cover multiple collections).

I think a trimmed-down doc that just focuses on the built-in modules and z/OS UNIX might be better, but let’s see what other opinions there are before recommending changes.

@nitzmahone my ask here was basically what you provided - the core thoughts on these docs being in the main ansible docs. As it stands today, the windows/BSD OS guides exist in core docs, so this z/OS one would likely also exist in core docs as well. I think if we do bring in this new PR, we add some kind of ‘community maintained’ note to all three.

1 Like
9

It might be hard to have technology-specific docs in a specific collection if there are several collections addressing said technology

Sorry, I’m casually using internal terminology that hasn’t had a lot of public exposure- a technology support collection is (usually) a single collection that provides the foundational support for a platform, so a particular networking device family (vyos.vyos), Windows (ansible.windows), z/OS (ibm.ibm_zos_core), etc. Today that collection typically includes core fact modules and whatever else is needed for basic connectivity and operations, and oftentimes any other collections that target that technology have either implicit or explicit dependence on it. In a world where exec subsystems exist, that’d live there as well.

The Windows support and docs in particular are in a weird spot, because there are too many hardcoded conditional bits in core with broken abstractions to be able to fully move it to a collection- I wouldn’t use that as an example, because we are definitely working toward being able shift it fully (with a proper abstraction). With the right abstraction it’d even be technically possible to extract the Python/POSIX subsystem (and there are some legit reasons to consider that, eg, bleeding edge Ansible hosting an old version of the Python/POSIX subsystem to manage hosts with ancient target Python versions).

All that said, the reason I’m bringing this up now is because if the docs ecosystem as a whole doesn’t have a good solution to this problem today, the clock’s ticking toward a time where it’ll need to. :wink: Until then, if there’s not a better place, fine, but I’m also worried that this will just open the floodgates for “well, you let z/OS do it, why not $next_thing?”, and that’s really working in the wrong direction.

1 Like
10

I second @nitzmahone on this one. I am sure @ddimatos and the team are only looking for the best experience for their users, and so will a dozen other teams of maintainers, but the argument that “would have users bouncing between sites” is a very slippery slope that can be easily abused by best intentions.

It is not a good thing, IMHO, to have one single group being the choke point for any update on documentations. That was - old timers correct me if I am wrong - exactly the main driver to split the maintenance of the modules into collections. And galaxy.ansible.com can show collection documentations today.

Additionally, the Ansible ecosystem is already quite big that I don’t see how we can prevent users from bouncing between tabs (not sure “sites” would matter anyway) - my personal experience has plenty of bouncing between tabs, even within one single collection. I don’t really think that is avoidable.

11

@nitzmahone can you elaborate on what the docs need to have here? We do have support for collection-level guides (amazon.aws is a good example of this).

What we don’t have is something that could sit atop a group of collections (say a guide for all Cisco collections).

@ddimatos - I realize we really haven’t answered where the z/OS guide you are working on should live. One idea would be that ibm.ibm_zos_core collection.

Gonna ping @felixfontein for opinions here as well. Also, do we think this should be turned into an Steering Committee vote? Don’t want to complicate things, but feels like a ‘policy in the making’ here…

12

I would like to point out that z/OS is not z/OS UNIX.

z/OS UNIX is just UNIX that runs over z/OS, not much different than the hypervisor/guest model. UNIX is not z/OS, despite z/OS being able to ‘guest’ many different operating systems.

I would ask when you are reading the dialog, you think of it this as 2 unrelated operating systems, which leads me to my next point.

The “single collection that provides the foundational support for a platform” for UNIX is Ansible.Builtin, this is why we have come here and contributed documentation.

We have designated a team to focus on eliminating the need for this doc but it can’t happen overnight.

Since there is no other “single collection that provides the foundational support for a platform” for UNIX where would we go with such doc? Where would BSD go?

I don’t think we should use Windows or z/OS proper in any comparisons here as they are perfect models for the collection architecture and that is why they have collections ansible.windows and ibm.ibm_zos_core. Also, we are not coming to the docs to say do this and that and then go out and get the ibm.ibm_zos_core collection, we are saying do this and then use ansible.builtin modules.

We named it z/OS because that is what our clients understand, maybe we should have called it z/OS UNIX or, UNIX(z/OS) but we didn’t dare because of trademarks but we could revisit that.

Ansible supports UNIX, there is no need for a fork of Ansible just to address UNIX over z/OS, that would really convolute a users experience much more than this doc pull request for clients. You are correct that we are trying to make this easier for our clients, this is why we have a new team focused on UNIX who want to be members of the community, contribute changes that makes all of Ansible better and this is our first non-technical contribution officially (we have contributed through others who I have tagged).

I can’t really see how putting the UNIX documentation into a collection for z/OS (ibm.ibm_zos_core) will help any users.

I do appreciate the time everyone is putting into the dialog, everyone wants the best for their users.

1 Like
13

This whole thing might be me being dense around the fuzzy split between the rolling-release community docs and the versioned core docs. A lot of the original os_guide content came over to ansible-documentation still intricately tied to the versioned core docs when we carved it off ansible/ansible, and we haven’t gotten back around to finalizing a clean cut of the core docs content in a separate project. So if I’m reading what’s happening today correctly, the rolling release “Community Documentation” content is still just a build of the devel version of the core docsite with all the ansible community package plugin doc content grafted in, so, e.g., stale versions of the os_guide content will appear under the versioned core docs and the latest stuff will be in the rolling release docs until we get around to actually splitting and removing that content whatever lands in the new core project docs repo (at which point it will be gone from the core docs).

I have zero issue with the rolling release ecosystem docs having pages like this- making that more possible was part of the reason for the split between the rolling community/ecosystem docs and the strictly-versioned core docs. I was just initially taken aback that it would show up under the versioned core docs as well, but that seems to be an artifact of the incomplete split, so I guess we’ll just carve all that stuff out of what goes to the core project docs repo whenever we get around to that last part of the split? If that’s the case, I retract my -1 so long as some form of your original “community maintained” marking happens (since the core project has no support or direct involvement).

14

@nitzmahone Ah so we could prevent this new guide (or all the OS guides) from showing up in core docs at all. Let me know if that’s what you want. (it’s an easy exclusion).

That would leave the decision on including this new os guide up to the steering committee folks to decide.

15

I would like to point out that z/OS is not z/OS UNIX .

Aha, likely part of my confusion as well- thanks for clarifying. We’ve had architectural discussions over the years about how to accommodate differences for both- IIRC z/OS was doing REXX-native modules with a custom normal.py (which is the best we could do for lack of exec subsystems), where z/OS UNIX mostly works out of the box with our existing Python/POSIX modules, but had some “fun” issues around Python hacks to deal with encoding and permissions things (and EBCDIC on the wire and filesystems was a common trip hazard for both cases requiring some connection plugin shenanigans)?

If that’s the case, the desired end state would probably still be that z/OS would have its own technology support collection that could hide a lot of the stuff you’re documenting there in some combo of a platform config plugin, custom facts modules, or exec subsystem (which might still end up largely using the stock core Python infra, but provide some extra hooks and whatnot to paper over the encoding and filesystem behavior that’s not directly supported or testable by core). You’d still probably have a page like this in the community docs, but it would hopefully be a lot smaller and mostly like “here’s how to light up the z/OS support collection”.

16

Yeah, that’s probably worth some discussion (in our copious free time :wink: ) - if it’s not a big pain, might be good to mask that stuff going forward to avoid confusion and stale stuff showing up in the versioned core docs. Would also minimize some of the shock of it disappearing whenever we do get around to formalizing that split.

17

@nitzmahone —

EBCDIC - what can we say here, it proliferates its way into anything running on a mainframe but in UNIX over z/OS , there is the concept of auto conversion and environment variables that make EBCDIC invisible and behave as ASCII.

Then there is also the fact that 5 years ago we did not own Python on the mainframe, it was owned by a partner, it was tough to steer that solution as many of the issues were in that Python port, so we ported our own Python and made it inclusive to the mainframe and providing a better experience for Ansible.

For lack of better name, UNIX(zos) file systems are files and folders, we want to leverage ansible.builtin for that.

For z/OS, file systems are data sets and members and we don’t have the hours to explain this here but for this we have ibm.ibm_zos_core collection.

As for the documentation, if I am following it correctly, the rolling release doc sounds like the right place.

If there is a better place to discuss this after the forum has vetted it out, we are happy to join.

18

The one thing we aren’t likely to accept in core is any concept of the need to tag files as outlined in that doc. It was explained to me the cause of the loss of tags is due to opening files in binary mode.

It has previously been recommended that an encoding parameter can be added to modules on a case by case basis to support opening the file in text mode.

Additionally a module included in the zos collection could be provided for tag manipulation.

19

I think it’s fine to have “OS guides” in the documentation for OSes on which Ansible runs in principle (which includes z/OS UNIX). Marking them as “community supported” probably makes sense if they are not “officially” supported (in the sense that ansible-core includes them in CI).

Right now the PR is a mess unfortunately, it would be great if it could be rebased/cleaned up to remove all the unrelated changes. That would make it easier to see what is actually proposed to be changed.

20

Hi, looking to understand if the previous comment is something I can help with.

Right now the PR is a mess unfortunately, it would be great if it could be rebased/cleaned up to remove all the unrelated changes. That would make it easier to see what is actually proposed to be changed.

It looks it looks like there were some changes made to the devel branch a couple of days ago and those changes were pulled into the faq-update-section-running-on-zos branch. I had accidentally introduced a syntax error into one of my pages, which I’ve since fixed (and now all 11 checks are passing). While looking at the file diff, I only see 3 files changed, which is what I’d expect. Not sure if I am missing something else, but hoping to clarify a bit more.