Encoding issues with UTF-8 files on z/OS for shell command

Get Help
, , ,
1

Hi,

our latest z/OS system has been upgraded with Python 3.11 and IZOAU 1.2.3 and we have upgraded our ibm_zos_core collection to 1.6.0 from 1.4.0 (and even tried out the new 1.7.0-beta2) release.
Now we have encoding issues with a specific shell command. We are not sure if this is really related to the ibm_zos_core upgraded and if this is the correct place to ask for support, but as we haven’t found any other place we try doing it here:

We have the following task that is being executed on the z/OS system (managed node/target):

    - name: Deploy broker
      shell: 
        cmd: ". /u/user/.profile && java -jar brokerhelper.jar -file /u/user/brokerhelper.xml"
        chdir: "/u/user"
        executable: "/bin/bash"

The file brokerhelper.xml has UTF-8 encoding on the file system and the utility brokerhelper.jar expects the file brokerhelper.xml to be in UTF-8.
The brokerelper.xml prolog contains the xml definition like this:
<?xml version="1.0" encoding="UTF-8"?>

However, when the shell command is being executed the file brokerhelper.xml is being converted by ansible to EBCDIC and the brokerhelper.jar gets that xml file in ebcdic encoding. We do not want that this happens.

We have tried out the following:

  1. we have disabled pipeling in ansible.cfg and do not set the PYTHONSTDINENCODING variable.
  2. We have tried setting the PYTHONSTDINENCODING with pipeling and this does also not help.
  3. We have tried out to use zos_encode and convert the already stored as UTF-8 file on the z/OS system to UTF-8 again and it does make thins even worse:
      ibm.ibm_zos_core.zos_encode:
        src: "/u/user/brokerhelper.xml"
        encoding:
          from: "UTF-8"
          to: "IBM-1047"
        backup: yes

Any ideas what might help?
Any help appreciated.
Thanks.

1 Like
2

I’m just guessing here but you could check the env vars like this (assuming you have printenv or something like it):

    - name: Printenv
      shell: 
        cmd: ". /u/user/.profile && printenv"
        chdir: "/u/user"
        executable: "/bin/bash"
      register: bash_env

    - name: Debug env
      debug:
        var: bash_env.stdout_lines

And if it helps set them like this:

    - name: Deploy broker
      shell: 
        cmd: ". /u/user/.profile && java -jar brokerhelper.jar -file /u/user/brokerhelper.xml"
        chdir: "/u/user"
        executable: "/bin/bash"
      environment:
        LANG: en_US.UTF-8
        LANGUAGE: en_GB.UTF-8
2 Likes
3

This is the output of printenv:

_EDC_SIG_DFLT=1
_EDC_SUSV3=1
SSH_CONNECTION=192.168.122.100 22
USER=user
SHLVL=1
SETUPTOOLS_USE_DISTUTILS=stdlib
_BPXK_AUTOCVT=ON
LANG=C
LIBPATH=/SYSTEM/local/cyp/v3r11/pyz/bin:/SYSTEM/local/cyp/v3r11/pyz/lib:/local/IZOA/usr/lpp/IBM/zoautil/lib:/usr/lpp/IBM/cyp/v3r11/pyz/lib:/lib:/usr/lib:.:/lib
SHELL=/bin/sh
MAIL=/usr/mail/user
_TAG_REDIR_ERR=txt
_TAG_REDIR_OUT=txt
SSH_TTY=/dev/ttyp0001
TERM=xterm-256color
PATH=/local/IZOA/usr/lpp/IBM/zoautil/bin:/usr/lpp/IBM/cyp/v3r11/pyz/bin:/bin:/var/bin
PYTHONPATH=/local/IZOA/usr/lpp/IBM/zoautil/lib
_CEE_RUNOPTS=FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)
LOGNAME=user
PWD=/u/user
_=/bin/printenv
SSH_CLIENT=192.168.122.100 55350 22
_TAG_REDIR_IN=txt
TZ=GMT-1MES,M3.5.0,M10.5.0
ZOAU_HOME=/local/IZOA/usr/lpp/IBM/zoautil
HOME=/u/user

setting the environment variables for the shell command does not help. Still getting an error.
By the way: This is the exact error I am getting:

Content is not allowed in prolog.

4

Ketan had replied to this same discussion here, maybe follow up with them.

Regarding “UTF-8 file on the z/OS system to UTF-8 again and it does make thins even worse” , going from UTF-8 to UTF-8 in that module would be a no-op, nothing would happen. The zos_encode snippet is going from UTF-8 → IBM-1047 which is what you are trying to avoid for Java.

Since I can’t know for certain if the file is actually UTF-8 , what happens when you run the command after SSH’ing in like so . /u/user/.profile && java -jar brokerhelper.jar -file /u/user/brokerhelper.xml? (without ansible)

2 Likes