Galaxy ingress with TLS redirect fails for ansible-galaxy collection install

mcen1 · May 8, 2024, 6:20pm

Hello there.

I’ve deployed galaxy into k8s (Rancher, not OpenShift, running with nginx ingress) utilizing the galaxy-operator (GitHub - ansible/galaxy-operator: Galaxy-Operator) successfully. I used version 2024.5.1 to do it. I’m doing testing by pointing a local repository’s ansible.cfg to use the repo to download collections via ansible-galaxy collection install. It seems that if TLS is enabled on the ingress, using ansible-galaxy collection install fails. See below:

$ ansible-galaxy collection install community.general -c --force
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading http://onpremgalaxy.mirror.com/api/galaxy/v3/plugin/ansible/content/mirror/collections/artifacts/community-general-8.6.0.tar.gz to /home/user/.ansible/tmp/ansible-local-272219dow9nwwb/tmpf6pa4_kw/community-general-8.6.0-vpitt0j7
ERROR! Failed to download collection tar from 'mirror' due to the following unforeseen error: HTTP Error 308: Permanent Redirect. HTTP Error 308: Permanent Redirect

If I turn off TLS for the ingress, ansible-galaxy collection install works fine:

$ ansible-galaxy collection install community.general -c --force
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading http://onpremgalaxy.mirror.com/api/galaxy/v3/plugin/ansible/content/mirror/collections/artifacts/community-general-8.6.0.tar.gz to /home/user/.ansible/tmp/ansible-local-2723325_o9jdnx/tmp5stkrib_/community-general-8.6.0-yuaw1goz
Installing 'community.general:8.6.0' to '/home/user/.ansible/collections/ansible_collections/community/general'

Not sure if this is a config setting I’m missing. Doesn’t seem to be one for ansible-galaxy, couldn’t find it inside the galaxy-operator project or Configuration options - Galaxy NG.

My ansible.cfg in the directory:

[galaxy]
server_list = mirror

[galaxy_server.mirror]
url=https://onpremgalaxy.mirror.com/api/galaxy/content/mirror/

cURL command on the URL for giggles:

$ curl http://onpremgalaxy.mirror.com/api/galaxy/v3/plugin/ansible/content/mirror/collections/artifacts/community-general-8.6.0.tar.gz
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

Denney-tech · May 9, 2024, 4:50pm

This looks like a k8s issue. Can you share your galaxy CR? There’s not enough information here to narrow down exactly what the issue might be.

mcen1 · May 9, 2024, 5:05pm

Sure.

---
apiVersion: galaxy.ansible.com/v1beta1
kind: Galaxy
metadata:
  name: galaxy
  namespace: galaxy
spec:
  hostname: onpremgalaxy.mirror.com
  ingress_type: ingress
  nginx_client_max_body_size: 100m
  storage_type: File
  ingress_tls_secret: tls-galaxy-mirror-ingress
  file_storage_access_mode: ReadWriteOnce
  file_storage_size: 8Gi
  no_log: false
  pulp_settings:
    GALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_ACCESS: true
    GALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_DOWNLOAD: true
  content:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 150m
        memory: 128Mi
      limits:
        cpu: 800m
        memory: 512Mi
  worker:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 150m
        memory: 128Mi
      limits:
        cpu: 800m
        memory: 512Mi
  web:
    replicas: 1
    resource_requirements:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 800m
        memory: 512Mi

Denney-tech · May 9, 2024, 5:40pm

Can you share what galaxy-web-svc and galaxy-ingress looks like? From what I can tell in the code, the service is templated based on ingress_type: route or loadbalancer, but the ingress/route is based on ingress_type: route or ingress. So both loadbalancer and ingress look to be half-broken to me.

@rooftopcellist @aknochow Can you take a closer look?

galaxy.ingress.yaml.j2 wants to point ingress to port 24880, but that port is only configured in galaxy-web.service.yaml.j2 for loadbalancer, even on the latest commit. There’s no configured service ports for ingress, and no ingress/route configured for loadbalancer.

Denney-tech · May 9, 2024, 5:48pm

Created an issue here:

github.com/ansible/galaxy-operator

Potential issue with ingress_types `ingress` and `loadbalancer`

opened 05:47PM - 09 May 24 UTC

Denney-tech

**Version** 2024.5.1 **Describe the bug** Ingress works without TLS, but no…t _with_ TLS **To Reproduce** See forum [post](https://forum.ansible.com/t/galaxy-ingress-with-tls-redirect-fails-for-ansible-galaxy-collection-install/5634?u=denney-tech). **Expected behavior** Ingress should work with or without TLS enabled. **Additional context** While trying to analyze the problem and compare with the code, I noticed what appears to be a discrepancy in the supported `ingress_types` for the galaxy-operator. The galaxy-web pods are templated for `route` and `loadbalancer`, but the ingress rules are only templated for `route` and `ingress`. So, ingress points to ports unavailable in the web-service, and loadbalancer never gets any ingress/routes. [galaxy.ingress.yaml.j2](https://github.com/ansible/galaxy-operator/blob/main/roles/galaxy-route/templates/galaxy.ingress.yaml.j2) -> [galaxy-web.service.yaml.j2](https://github.com/ansible/galaxy-operator/blob/main/roles/galaxy-web/templates/galaxy-web.service.yaml.j2)

mcen1 · May 9, 2024, 6:02pm

Sure thing, appreciate the help! Let me know if you want me to post this onto the issue you’ve opened as well.

ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    haproxy.router.openshift.io/timeout: 180s
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-read-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-send-timeout: 120s
    nginx.org/client-max-body-size: 100m
  creationTimestamp: "2024-05-09T17:54:19Z"
  generation: 1
  name: galaxy-ingress
  namespace: galaxy
  ownerReferences:
  - apiVersion: galaxy.ansible.com/v1beta1
    kind: Galaxy
    name: galaxy
    uid: e0858956-6744-4c11-916d-f2b9ac149bc8
  resourceVersion: "225627944"
  uid: 7b7ffa9a-ecd6-4087-8ee9-ab41e1d06339
spec:
  ingressClassName: nginx
  rules:
  - host: onpremgalaxy.mirror.com
    http:
      paths:
      - backend:
          service:
            name: galaxy-web-svc
            port:
              number: 24880
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - onpremgalaxy.mirror.com
    secretName: tls-galaxy-mirror-ingress

status:
  loadBalancer:
    ingress:
    - ip: 10.248.8.27

svc:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-05-09T17:53:01Z"
  labels:
    app.kubernetes.io/component: webserver
    app.kubernetes.io/instance: nginx-galaxy
    app.kubernetes.io/managed-by: galaxy-operator
    app.kubernetes.io/name: nginx
    app.kubernetes.io/part-of: galaxy
  name: galaxy-web-svc
  namespace: galaxy
  ownerReferences:
  - apiVersion: galaxy.ansible.com/v1beta1
    kind: Galaxy
    name: galaxy
    uid: e0858956-6744-4c11-916d-f2b9ac149bc8
  resourceVersion: "225627057"
  uid: 7bc0c619-5003-4551-80f2-7dddbe3823f7
spec:
  clusterIP: 172.17.215.88
  clusterIPs:
  - 172.17.215.88
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web-8080
    port: 24880
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/component: webserver
    app.kubernetes.io/instance: nginx-galaxy
    app.kubernetes.io/managed-by: galaxy-operator
    app.kubernetes.io/name: nginx
    app.kubernetes.io/part-of: galaxy
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Denney-tech · May 9, 2024, 6:16pm

Well, that looks like maybe it should work, but I can barely wrap my head around the networking side of k8s at the best of times.

Can you try your curl command with the -L switch to follow redirects?

curl -L http://onpremgalaxy.mirror.com/api/galaxy/v3/plugin/ansible/content/mirror/collections/artifacts/community-general-8.6.0.tar.gz

mcen1 · May 9, 2024, 6:37pm

Yup, that command works fine, curl correctly follows the redirects and downloads the tar.gz file. That’s part of my confusion, I’m not sure if this is an ansible-galaxy CLI issue, the operator, or the containers.

$ curl --output hi.tar.gz -kL http://onpremgalaxy.mirror.com/api/galaxy/v3/plugin/ansible/content/mirror/collections/artifacts/community-general-8.6.0.tar.gz

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   164  100   164    0     0   1244      0 --:--:-- --:--:-- --:--:--  1251
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0

100 2584k  100 2584k    0     0  1521k      0  0:00:01  0:00:01 --:--:-- 8612k

$ ls -l hi.tar.gz
-rw-rw-r-- 1 user user 2646969 May  9 14:30 hi.tar.gz

Testing with ansible-galaxy collection install

$ ansible-galaxy collection install community.general -c --force                                                                                    
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading http://onpremgalaxy.mirror.com/api/galaxy/v3/plugin/ansible/content/mirror/collections/artifacts/community-general-8.6.0.tar.gz to /home/user/.ansible/tmp/ansible-local-285424ntw2bzyd/tmp3mkwqloi/community-general-8.6.0-fgwkuxqp

ERROR! Failed to download collection tar from 'mirror' due to the following unforeseen error: HTTP Error 308: Permanent Redirect. HTTP Error 308: Permanent Redirect

and for fun, creating an ansible.cfg pointing at the non-SSL URL for galaxy.ansible.com, which works:

$ ansible-galaxy collection install community.general -c --force
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/community-general-8.6.0.tar.gz to /home/user/.ansible/tmp/ansible-local-285454jgn00g07/tmp04t6s12c/community-general-8.6.0-2l3yt91o
Installing 'community.general:8.6.0' to '/home/user/.ansible/collections/ansible_collections/community/general'
community.general:8.6.0 was installed successfully

$ cat ansible.cfg
[galaxy]
server_list = mirror
[galaxy_server.mirror]

url=http://galaxy.ansible.com/

Denney-tech · May 9, 2024, 7:02pm

Oh…

So with TLS enabled, https ingress works. With TLS disabled, http ingress works.

But with TLS enabled, http(no s) doesn’t work… at least not with your mirror.

galaxy.ansible.com works whether you use http or https, and you’re expecting the same with your mirror? Is that what you’re seeing?

Denney-tech · May 9, 2024, 7:11pm

Can you enable TLS, turn off the operator, and try adding some annotations to the ingress?

Try this one first:

nginx.ingress.kubernetes.io/use-forwarded-headers: true

And add this one next if you still have issues:

nginx.ingress.kubernetes.io/ssl-redirect: false

So it should look something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    haproxy.router.openshift.io/timeout: 180s
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-read-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-send-timeout: 120s
    nginx.org/client-max-body-size: 100m
+   nginx.ingress.kubernetes.io/ssl-redirect: false
+   nginx.ingress.kubernetes.io/use-forwarded-headers: true
  creationTimestamp: "2024-05-09T17:54:19Z"
  generation: 1
  name: galaxy-ingress
  namespace: galaxy
  ownerReferences:
  - apiVersion: galaxy.ansible.com/v1beta1
    kind: Galaxy
    name: galaxy
    uid: e0858956-6744-4c11-916d-f2b9ac149bc8
  resourceVersion: "225627944"
  uid: 7b7ffa9a-ecd6-4087-8ee9-ab41e1d06339
spec:
  ingressClassName: nginx
  rules:
  - host: onpremgalaxy.mirror.com
    http:
      paths:
      - backend:
          service:
            name: galaxy-web-svc
            port:
              number: 24880
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - onpremgalaxy.mirror.com
    secretName: tls-galaxy-mirror-ingress

status:
  loadBalancer:
    ingress:
    - ip: 10.248.8.27

mcen1 · May 9, 2024, 7:37pm

Ah, interesting. ansible-galaxy collection install does work with both those annotations. Thank you!

Edit: looks like nginx.ingress.kubernetes.io/ssl-redirect: false specifically works.

kurokobo · May 10, 2024, 12:30pm

Commented on the GitHub: Potential issue with ingress_types `ingress` and `loadbalancer` · Issue #122 · ansible/galaxy-operator · GitHub