We’re moving to Read The Docs

[oranod (Don Naro)]

Hi everyone! The Ansible community team at Red Hat is excited to announce that we plan to migrate the docs.ansible.com subdomain to Read The Docs hosting.

This post explains our motivation and outlines our plan. Before we take action, we’d like to hear from you. Please give us your feedback by replying below. Questions are always welcome.

Summary: The hosting platform for Ansible community documentation is changing. There will be some changes to docs.ansible.com urls as a result but those changes should be transparent. Links will continue to work as they do today and you won’t need to update bookmarks. However you will likely notice that a /projects/ subdirectory appears in urls after the migration takes place.

Rationale for moving the subdomain

There are two main reasons behind this migration:

• Improving the docs experience for the Ansible community (see more on this below).
• Opening up the full release process to Ansible community maintainers. At the moment, only Red Hat employees can access the hosting infra for docs.ansible.com. Moving the subdomain to Read The Docs extends greater access to the Ansible package release process for the community.

Improving the docs experience

Going to docs.ansible.com today you’ll find documentation for the Ansible community package, ansible-core, and Ansible automation controller. To find documentation for other projects in the Ansible ecosystem, such as Ansible Lint or Ansible Builder, you need to visit ansible.readthedocs.io instead.

By migrating the subdomain to Read The Docs, we can have a unified space for all Ansible community documentation. We’ll also get some nice features like cross-project search, version switcher, and pull request previews.

All the Ansible community documentation is already available from ansible.readthedocs.io. Why not remove the subdomain and just use RTD?

We could do that, but the docs.ansible.com subdomain has a lot of SEO authority built up over time. When you search for “ansible”, the docs site is at the top of the list. By migrating the subdomain we ensure that authority stays with the Ansible community, where it belongs.

Additionally the docs.ansible.com subdomain signals the “official” location of trusted content for Ansible users, developers, maintainers, and contributors.

How does this affect me?

The answer to this question should be very little from a usability point of view. We’ve done a lot of work on our redirect strategy and will be reviewing and validating everything over the next couple of weeks before we start the migration process.

You do not need to update bookmarks or links to documentation in collection READMEs or other places where you might point to docs.ansible.com.

What you will notice is that there will be a /projects/ subdirectory in the urls for Ansible package documentation and ansible-core documentation, for example:

• Ansible Documentation — Ansible Community Documentation
• Ansible Core Documentation — Ansible Core Documentation

Read The Docs allows multiple docsites to be nested under a single namespace such as ansible.readthedocs.io. Each nested docsite is referred to as a subproject.

After the migration, the ansible.readthedocs.io subdomain will change to docs.ansible.com and the landing pages will be at the root level with each project, including core and the package, added as a subproject. After the change, you will have the following structure:

• Landing pages: https://docs.ansible.com/
• Ansible package: https://docs.ansible.com/projects/ansible/latest/
• Ansible core: https://docs.ansible.com/projects/ansible-core/devel/
• Ansible lint: https://docs.ansible.com/projects/lint/

We’ve already created redirects as outlined in Migrating docs to Read The Docs to handle the new URL structures and avoid breaking things. So, hopefully, our community will only be affected positively by having all the Ansible community docs in one place.

When will the migration take place?

We’re still gathering some details and lining things up for the move inside Red Hat. There are a couple of different teams involved and there’s still some coordination work to do. We don’t have set dates in the calendar but we do want to complete the migration soon.

However, there are distinct phases to this effort. We have this sequence of events in mind for the migration:

• Socialize this post and gather feedback.
• Complete migration phase one: Configure DNS entries and stand up the legacy-controller-docs.ansible.com subdomain.
• Announce the new subdomain and give a week or two for community feedback.
• Complete migration phase two: Move the CNAME record to Read The Docs hosting.
• Complete migration phase three: Clean up work.

Call to action

We want to hear from you! Please give us your feedback! Here is how you can help:

• Read our migration plan in the section below and reply to this post with your thoughts or comments.
• Try out the Ansible package documentation and ansible-core documentation on Read The Docs. How does it compare to the corresponding page that is on docs.ansible.com today?
• Check for broken links or 404 pages and help us test things. You’ll find lots of information in the plan and test details below.

If you find something that needs to be fixed, please open an issue in the ansible/ansible-documentation repository and add the rtd-migration label. We do welcome any contributions so if you see a change and open a pull request, feel free!

[oranod (Don Naro)]

Preparing for a successful migration

Before we dive into the tasks involved in the migration plan, let’s go over some of the things we’ve been doing to get ready.

Replacement subdomain for controller docs

As mentioned a bit earlier, there is some Automation controller documentation available from docs.ansible.com. That documentation is intended for Red Hat customers and is not moving to Read The Docs. We plan to keep the content on the web server after the migration is complete. For this reason, we need to create a new subdomain so folks can still find Automation controller documentation.

We’ve consulted the team responsible for that content and have decided to go with the legacy-controller-docs.ansible.com subdomain.

Consolidating and converting redirects

To facilitate the migration to Read The Docs, we had to drastically reduce the number of server-side redirects. Read The Docs imposes a limit of 100 redirects per project. When we started out with the documentation for the Ansible community package, there were thousands of server-side redirects.

Earlier this year we consolidated all the pre-collections redirects. We also added an Sphinx extension for redirects in the Ansible community docs.

As a result, we should not break any redirects in place for the Ansible community docs defined in the top-level .htaccess file or the pre-collections redirects.

Phase one: Setting things up

Phase one of the migration sets up the legacy-controller-docs.ansible.com subdomain for Automation controller docs and adjusts DNS records.

Creating a separate landing page

We need to create a landing page for the legacy-controller-docs.ansible.com subdomain that should:

• Briefly explain the migration to Read The Docs.
• Provide clear links to Ansible community and Ansible core documentation.
• Notify that Ansible core documentation from version 2.15 and later will be available from docs.ansible.com after the migration.
  Note that we can enable builds for earlier versions in Read The Docs if necessary. It would be good to get feedback.
• Explain that we will not remove any pages from the current server but will no longer actively maintain or update them.
  For example, Getting started with Ansible — Ansible Community Documentation will still be available from the legacy-controller-docs.ansible.com subdomain.
• Provide entry points to the main pages that will be available after the migration, as follows:

ansible-tower.html
automation-tower-chinese-translations.html
automation-tower-japanese-translations.html
automation-tower-korean-translations.html
automation-tower-prior-versions.html
platform.html

To create the archive landing page, do the following:

1. Create an archive-index.html in the ansible/docsite repository.
   This index page should be as simple as possible and not generated from any template. It should use straightforward inline styling.
2. Update the .htaccess configuration to allow access to archive-index.html.
3. Update the docsite build job in Jenkins to deploy archive-index.html to the web server.

After migration, we will separate the index page from the ansible/docsite repository.

Configuring DNS

Create a new LB record for legacy-controller-docs.ansible.com that points to the two servers and uses the same configuration as docs.ansible.com.

Configuring the web server

We will serve legacy-controller-docs from the same server as docs.ansible.com.

1. Update the roles/docs_and_release/tasks/main.yml playbook to:

   • Create a new legacy-controller-docs directory with correct permissions.
   • Set correct ownership and permissions for the jenkins user on the docs and legacy-controller-docs directories.

2. Configure the web server to handle requests for legacy-controller-docs.ansible.com by adding a new VirtualHost such as:

<VirtualHost *:80>
    ServerName legacy-controller-docs.ansible.com
    DocumentRoot /var/www/html/legacy-controller-docs/
    Options +Indexes
    <Directory /var/www/html/legacy-controller-docs/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride FileInfo Options=MultiViews
                Order allow,deny
                allow from all
    </Directory>
</VirtualHost>

Setting things up on disc

1. Add a reference to the ansible-tower directory to the index-archive.html page.
2. Do a once off copy from /var/www/html/docs/automation-controller AND /var/www/html/docs/ansible-tower to /var/www/html/legacy-controller-docs.
3. Move /var/www/html/docs/index-archive.html to /var/www/html/legacy-controller-docs/index.html
4. Copy landing pages into /var/www/html/legacy-controller-docs:

• platform.html
• automation-tower-prior-versions.html

5. Update controller jenkins jobs to rsync to the legacy-controller-docs folder instead of the docs folder.
6. Update the docsite jenkins job to remove the rsync for the index-archive.html file.

Test plan for phase one

The archive subdomain, legacy-controller-docs.ansible.com, will be served from Red Hat managed infrastructure.

1. Use dig to verify DNS settings for the archive subdomain and docs.ansible.com.
2. Determine if legacy-controller-docs.ansible.com has an A record, for example: dig a legacy-controller-docs.ansible.com
3. Determine if docs.ansible.com has a CNAME record, for example: dig cname docs.ansible.com
4. Verify TTL for docs.ansible.com, for example: dig docs.ansible.com +ttlunits

We can also complete the following steps to test that everything is set up correctly:

1. Ensure that the archive landing page is available from legacy-controller-docs.ansible.com.
2. Ensure that entry points for legacy Automation Controller documentation is available from legacy-controller-docs.ansible.com.
3. Ensure that the docsite build job in Jenkins can update the archive landing page.
4. Ensure that both legacy-controller-docs.ansible.com and docs.ansible.com serve the same set of web pages, for example: gobuster dns -d docs.ansible.com -w /path/to/wordlist.txt.

EDIT: I’ve updated plan details after working through a few changes. The original plan was based on more of a bottom up view that started from looking at what was on the server. After getting into some of the changes on cloudflare it made sense to take a slightly different approach and more cleanly separate things from the top-level down to disc.

[oranod (Don Naro)]

Phase two: Migrating the subdomain

Update the CNAME record for docs.ansible.com to point to Read The Docs.

1. Follow the steps to add a custom domain on Read The Docs.
   a. Enter docs.ansible.com as the custom domain.
   b. Select the Canonical option.
2. Update the DNS record for docs.ansible.com so that it points to readthedocs.io.
3. Wait for the changes to propagate and then test with something like nslookup to verify the CNAME record.
4. Update the canonical url in conf.py to include the projects subdirectory: ansible-documentation/docs/docsite/rst/conf.py at 350ef3df2c61dcce411f5c237ebc288079586003 · ansible/ansible-documentation · GitHub

Test plan for phase two

After we add the docs.ansible.com subdomain to Read The Docs and update the DNS record, we should verify that changes have propagated as expected.

1. Check the canonical name that the subdomain points to: nslookup -type=CNAME docs.ansible.com
2. Check against Google’s DNS: nslookup docs.ansible.com 8.8.8.8
3. Check against Read The Doc’s name server: nslookup docs.ansible.com tegan.ns.cloudflare.com

[oranod (Don Naro)]

Phase three: Cleaning up

After the docs.ansible.com subdomain is migrated to Read The Docs hosting, we should do some clean up work.

Tasks in Google search console

To help preserve SEO authority of docs.ansible.com, we should perform the following steps in the search console:

• Verify both docs.ansible.com and the new subdomain are verified.
• Use the Change of Address Tool to inform Google about the change of subdomain.
• Follow steps in Move a site with URL changes to monitor traffic and check for crawl errors.

Custom sitemaps

XML sitemaps help search engines discover and index the site structure faster. Read The Docs automatically creates sitemaps, however we should consider generating a custom sitemap for the top-level project.

Additionally, we should create a new XML sitemap for the new subdomain to replace the existing ones:

• https://raw.githubusercontent.com/ansible/docsite/refs/heads/main/ansible-sitemap.xml
• docsite/automation-controller-sitemap.xml at main · ansible/docsite · GitHub

Here are some commands used to create sitemaps:

sudo dnf install nodejs
sudo npm install -g sitemap-generator-cli
sitemap-generator -f ansible-sitemap.xml https://docs.ansible.com/ansible/latest/
sitemap-generator -f automation-controller-sitemap.xml https://docs.ansible.com/automation-controller/latest/

Custom robots.txt file

The robots.txt file helps improve SEO by controlling access to urls from search crawlers. We currently disallow several urls from search crawlers in robots.txt. We should investigate how to integrate that into the Sphinx configuration according to their documentation. Alternatively that might be something we just copy across in the Read The Docs configuration from the top-level domain.

Updating internal links

Even though we will have redirects in place that should automatically point to the updated docs.ansible.com urls, we should ensure that as many links are updated as possible. Updating internal linking should help both users and search engines navigate the new structure without relying on redirects.

Here are some places where we should scan for docs.ansible.com urls and make batch updates where necessary:

• ansible/ansible-documentation
• ansible/ansible
• ansible/aap-docs

Separating the landing pages

Landing pages refer to the top-level pages that guide users to relevant parts of the documentation.

After the migration, there will be two landing pages:

• docs.ansible.com on Read The Docs and sourced from the ansible/docsite repository
• legacy-controller-docs.ansible.com on the web server and sourced from the ansible/archive-docsite repository

Updating the archive landing page

We should complete the following steps to modify the landing page for legacy-controller-docs.ansible.com:

1. Temporarily disable the Jenkins job to build the docsite.
2. Fork the ansible/docsite repository to ansible/archive-docsite.
3. Rename archive-index.html to index.html.
4. Create a new standalone 404 page with the cowsay image.
5. Remove the following files and folders:

​​​├── ansible/
​​​├── data/
​​​├── requirements/
​​​├── sass/
​​​├── static/css/
​​​├── static/images/community_logo.svg
​​​├── static/js/
​​​├── templates/
​​​├── .pip-tools.toml
​​​├── .readthedocs.yaml
​​​├── ansible-sitemap.xml
​​​├── build.py
​​​└── noxfile.py

After removing the preceding files and folders, continue with these steps:

1. Update .htaccess to use the standalone 404 page.
2. Update .htaccess to remove redirects that applied to the docs.ansible.com subdomain.
3. Update the catch all redirect in .htaccess at https://github.com/ansible/docsite/blob/c02fae53bbfae3b296f38b1b04b7639d3431b98a/.htaccess#L11
4. Update robots.txt to disallow the /ansible and /ansible-core directories. Consider disallowing all content to avoid competing with redhat.com for Automation controller content.
5. Update robots.txt to modify the sitemaps. Remove ansible-sitemap and update the subdomain for automation-controller-sitemap.
6. Update automation-controller-sitemap.xml to reflect the change of the subdomain.
7. Update the Jenkins job to prune deleted files and folders from the rsync step.
8. Update the Jenkins job to clone the ansible/archive-docsite repository instead of ansible/docsite.
9. Enable the Jenkins job to build the docsite and run it.
10. Modify the web server configuration to serve content for legacy-controller-docs.ansible.com only and to use the index.html file.

Updating the landing page on Read The Docs

We should update the docs.ansible.com landing pages to put the focus on the content journeys, which essentially means removing the following files and folders:

​​​├── ansible/
​​​├── templates/ansible_community.html
​​​├── templates/automation-tower-*.html
​​​├── templates/core-translated-ja.html
​​​├── templates/core.html
​​​├── templates/platform.html
​​​├── .htaccess
​​​├── ansible-sitemap.xml
​​​├── automation-controller-sitemap.xml
​​​└── robots.txt

1. Remove the preceding files and folders.
2. Update templates as appropriate to reflect the changes.
3. Update the ecosystem.html page to ensure links to Read The Docs projects are correct.
4. Ensure there is a link to the ecosystem page on the index.
5. Add redirects to the top-level Read The Docs project for the deleted templates/* pages.

As a future enhancement, we should consider building the docs.ansible.com landing pages with Sphinx. This would allow us to make better use of the Read The Docs widget that provides cross-project search.

Updating the prior versions page

We should ensure that the templates/ansible-prior-versions.html page has the correct urls for older versions of the Ansible community docs.

For instance, https://docs.ansible.com/archive/ansible/2.7/ changes to https://ansible.readthedocs.io/projects/ansible/2.7-archive/

Test plan for phase three clean up work

After the docs.ansible.com subdomain is transferred to Read The Docs hosting, additional testing is needed to ensure that pages are served correctly, redirects are working as expected, and that everything is in place.

1. Use curl to check and validate sitemaps.
2. Use the url checker to validate pages available from docs.ansible.com/ansible and docs.ansible.com/ansible-core are in place and still return HTTP 200 status.
3. Ensure that the landing pages for docs.ansible.com and legacy-controller-docs.ansible.com have the correct entry points and are clearly separated.

Ensure that any pages in the ansible/* and ansible-core/* directories on the web server redirect to Read The Docs.

[utoddl (Todd Lewis)]

Can’t see the garden for the weeds! Here’s a small thing (now), but still a thing.

@oranod

<VirtualHost *:80>
    ServerName legacy-controller-docs.ansible.com
    DocumentRoot /var/www/html/legacy-controller-docs/
    Options +Indexes
    <Directory /var/www/html/legacy-controller-docs/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride FileInfo Options=MultiViews
                Order allow,deny
                allow from all
    </Directory>
</VirtualHost>

Apache httpd 2.4 docs warn

The Allow, Deny, and Order directives, provided by mod_access_compat, are deprecated and will go away in a future version. You should avoid using them, and avoid outdated tutorials recommending their use.

The effective equivalent using mod_authz_host is

Require all granted

[oranod (Don Naro)]

@utoddl

a future version. You should avoid using them, and avoid outdated tutorials recommending their use.

Thanks a mill for calling this out @utoddl

Here is the updated config with the Require directive:

<VirtualHost *:80>
    ServerName legacy-controller-docs.ansible.com
    ServerAlias legacy-controller-docs.ansible.com
    DocumentRoot /var/www/html/legacy-controller-docs/
    Options +Indexes
    <Directory /var/www/html/legacy-controller-docs/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride FileInfo Options=MultiViews
                Require all granted
    </Directory>
</VirtualHost>