A friend show me a problem when running Python script with ansible command module. The command failed because the HOSTNAME variable was missing. The ansible documentation warns about this behaviour and provide this explanation: "The command(s) will not be processed through the shell, so variables like $HOSTNAME and [...] will not work. Use the shell module if you need these features." It's true, but there is a serious pitfall. The variable HOSTNAME is not really in the environment, you can't view when running env.

What's happened when running an ansible command

To troubleshoot, I'm using a simple Python script printenv.py to print env:

#!/bin/python
import os

for k, v in os.environ.items():
               print("{}={}".format(k, v))
print("{} variables found.".format(len(os.environ)))

I'm using a simple docker to simulate the remote host and demonstrate the problem.

FROM centos:7

RUN yum update -y
RUN yum -y install openssh-server openssh-clients sshpass
RUN mkdir /var/run/sshd
RUN echo 'root:r00t' | chpasswd
RUN sed -i 's/#PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
RUN echo "export VISIBLE=now" >> /etc/profile
RUN /usr/bin/ssh-keygen -A
COPY printenv.py /root/

CMD "/usr/sbin/sshd" "-D"

Here is the output, and HOSTNAME is set:

docker exec -it d909a67ab429 python /root/printenv.py
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM=xterm
HOSTNAME=d909a67ab429
HOME=/root
4 variables found.

When running ansible, even with the shell module you did not have access to HOSTNAME:

$ ansible -i docker.yml -m shell -a 'python /root/printenv.py' centos7 
centos7 | SUCCESS | rc=0 >>
LANG=C
TERM=xterm-256color
SHELL=/bin/bash
LC_MESSAGES=C
MAIL=/var/mail/root
SHLVL=3
SSH_TTY=/dev/pts/1
PWD=/root
SSH_CLIENT=10.5.117.254 32896 22
LOGNAME=root
USER=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
LC_ALL=C
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
HOME=/root
_=/usr/bin/python
SSH_CONNECTION=10.5.117.254 32896 10.5.117.1 22
17 variables found.

Same thing happened when the env command:

ansible -i docker.yml -m shell -a 'env| grep HOSTNAME' centos7 
centos7 | FAILED | rc=1 >>
non-zero return code

The hostname is the name of the container (2b97621d1f75) that docker have generated. At this point we can see that there is a difference when running the script printenv.py through the docker or using in the ansible shell module.

The weirdest thing is that the variable HOSTNAME is defined!

 ansible -i docker.yml -m shell -a 'echo $HOSTNAME/$USER' centos7 
centos7 | SUCCESS | rc=0 >>
2b97621d1f75/root

It looks like that env command doesn't work as expected. UPDATE: I received a message from Benoît explaining why there is a difference between echo, env and the printenv.py script. Simply the difference coming from that HOSTNAME variable is set but is not exported. The export is done when reading when loading /etc/profile.

[root@264421a1c781 /]# egrep -A 8 -Hnr 'HOSTNAME=' /etc/
/etc/profile:45:HOSTNAME=`/usr/bin/hostname 2>/dev/null`
/etc/profile-46-HISTSIZE=1000
[...]
/etc/profile-53-export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL

But the /etc/profile is not loaded when running ansible because ansible doesn't invocate bash like an interactive shell neither like a login shell.

The cause of this behaviour

The problem is really complicated because there is a lot of layers involved when running ansible shell command.

1. you are using a shell to run the ansible command. All the variables from the environment are not visible from the remote host
2. ansible transform the command line ansible into a ssh command that could be see if you increase verbosity with -vvv. It looks like <10.5.117.1> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/home/rjacquet/.ansible/cp/634b52de7c -tt 10.5.117.1
3. the ssh is using a bash command to run the ansible module. Note that the bash command is not a shell login, this is why accessing to HOSTNAME variable is not possible. Stuffs like are also visible with more verbosity /bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1603741018.49-107090793072670/ > /dev/null 2>&1 && sleep 0
4. the bash (non-login) shell is running a python command with the syscall fork
5. finally the Python command use the subprocess module to run printenv.py! Furthermore the subprocess module doesn't inherit the environment from its parent (the sh scripts). The environment is created is the ansible class AnsibleModule and the run_command method after the comment # Manipulate the environ we'll send to the new process (see here). Probably you have never heard about this module except you have already written your own ansible module. This a class that provide common things to all modules.

Simple workaround

The only workaround I found is to change the python or shell script to run the hostname command instead of using the environment. This is exactly how the shell initialize the variable.

If you have a solution without changing the script, feel free to share me the solution :-).