Set_fact and delay Values Not Updating on Task Retries in Ansible Playbook

Yes, the module arg is templated once (see this explanation strings templated using a lookup in a loop shouldn't be cached and re-used · Issue #82955 · ansible/ansible · GitHub - there are also a few workarounds mentioned on that issue). If you’re able to update to 2.18, then you could do use loop for retries and loop_control.break_when to stop looping when the condition is met:

  - name: Set fact until status is "on"
    set_fact:
      status: "{{ ['on', 'off'] | random }}"
    register: status_result
    loop: "{{ range(0, 5) }}"
    loop_control:
      pause: >
        {% if (ansible_loop.index|default(0)) == 0 %}
        {{ 2 + (1 | random(start=0)) }}
        {% else %}
        {{ (2 ** (ansible_loop.index + 1)) + (1 | random(start=0)) }}
        {% endif %}
      break_when: status == "on"