Category Archives: Cloud

Command substitution gotcha in Ansible Playbook

Command substitution reassigns the output (stdout) of a command or even multiple commands to a variable. The mechanism is useful for dynamically populating variables based on the environment in which a program is executed.
In the case of cloud orchestration, we may wish to reconfigure memory parameters for an application or service. The example I have chosen is for a PostgreSQL database deployment, where I need to adjust the memory parameters to their optimum value based on the required percentage of total available memory.

For example, among others, I wish to change the work_mem parameter from its default value.

On the Ansible server, I execute a playbook that contains the following code snippet that assigns a value to the “workmem” variable using the “register” Ansible module, and then uses the “replace” module to replace the parameter value in the postgresql.conf file.

# Initialising the global variable
 - shell: echo $(cat /proc/meminfo | grep MemTotal | awk '{print $2}') / 100 / 1024 |bc
 register: workmem
 ignore_errors: True

# Using variable in replace statement
 - name: Configure memory parameters ( work_mem = {{ workmem }}MB )
 replace: dest={{db_data_dir}}/postgresql.conf regexp="^#?work_mem\s+=\s+[1-9]*[kMGT]B" replace="work_mem = {{ workmem }}MB"
 become: true
 become_user: postgres
 
 - name: Restarting the postgres service
 service: name=postgresql-9.5 state=restarted


This results in the following error when Ansible attempts to restart the PostgreSQL instance at the end of the playbook.

TASK [Restarting the postgres service] *****************************************
 fatal: [10.127.3.18]: FAILED! => {"changed": false, "failed": true, "msg": "Stopping postgresql-9.5 service: [ OK ]\r\nStarting postgresql-9.5 service: [FAILED]\r\n"}

NO MORE HOSTS LEFT *************************************************************
 to retry, use: --limit @/home/ansible/yaml/postgres-main2.retry

PLAY RECAP *********************************************************************
 10.127.3.18 : ok=23 changed=7 unreachable=0 failed=1

The gotcha

The register statement is used to store the output of a single task into a variable. However, the shell task will include stdout & stderr, as well as the string returned from the  command. This is visible in the postgresql.conf file we are trying to modify on the target server.

postgres@db_host[~] $ cd /var/lib/pgsql/9.5/data
postgres@db_host[data] $ view postgresql.conf

work_mem = {u'changed': True, u'end': u'2016-10-12 03:15:56.780380', u'stdout': u'9', u'cmd': u"echo $(cat /proc/meminfo | grep MemTotal | awk '{print $2}') / 100 / 1024 |bc", u'start': u'2016-10-12 03:15:56.775608', u'delta': u'0:00:00.004772', u'stderr': u'', u'rc': 0, 'stdout_lines': [u'9'], u'warnings': []}MB

The solution

The variable appears to store an array of values. The solution to this problem is to force Ansible to substitute only the required element, in this case stdout. This is achieved using the following syntax:

- debug:
  var: <variable_name>.stdout

or

{{ <variable_name>.stdout }}

An example of Ansible command substitution from a shell task is shown below:

# Initialising the global variable
 - shell: echo $(cat /proc/meminfo | grep MemTotal | awk '{print $2}') / 100 / 1024 |bc
 register: workmem
 ignore_errors: True
 - debug:
 var: workmem.stdout

# Using variable in replace statement
 - name: Configure memory parameters ( work_mem = {{ workmem.stdout }}MB )
 replace: dest={{db_data_dir}}/postgresql.conf regexp="^#?work_mem\s+=\s+[1-9]*[kMGT]B" replace="work_mem = {{ workmem.stdout }}MB"
 become: true
 become_user: postgres

The correct playbook execution output for provisioning a single instance PostgreSQL database is shown below:

root@ansible_host[yaml] # ansible-playbook /home/ansible/yaml/postgres-main2.yml --extra-vars "target=postgres node_ip=10.127.3.18 db_name=dbdemo db_user=pgadmin db_password=password db_port=5432 db_data_dir=/var/lib/pgsql/9.5/data"

PLAY [postgres] ****************************************************************

TASK [setup] *******************************************************************
 ok: [10.127.3.18]

TASK [assert] ******************************************************************
 ok: [10.127.3.18] => (item=node_ip)
 ok: [10.127.3.18] => (item=db_name)
 ok: [10.127.3.18] => (item=db_user)
 ok: [10.127.3.18] => (item=db_password)
 ok: [10.127.3.18] => (item=db_port)
 ok: [10.127.3.18] => (item=db_data_dir)

TASK [command] *****************************************************************
 changed: [10.127.3.18]

TASK [debug] *******************************************************************
 ok: [10.127.3.18] => {
 "totalmem.stdout": "1018628"
 }

TASK [command] *****************************************************************
 changed: [10.127.3.18]

TASK [debug] *******************************************************************
 ok: [10.127.3.18] => {
 "sharedbuf.stdout": "248"
 }

TASK [command] *****************************************************************
 changed: [10.127.3.18]

TASK [debug] *******************************************************************
 ok: [10.127.3.18] => {
 "workmem.stdout": "9"
 }

TASK [command] *****************************************************************
 changed: [10.127.3.18]

TASK [debug] *******************************************************************
 ok: [10.127.3.18] => {
 "maintworkmem.stdout": "124"
 }

TASK [command] *****************************************************************
 changed: [10.127.3.18]

TASK [debug] *******************************************************************
 ok: [10.127.3.18] => {
 "effectcachesize.stdout": "746"
 }

TASK [Add the group 'postgres'] ************************************************
 ok: [10.127.3.18]

TASK [Add the user 'postgres' and a primary group of 'postgres'] ***************
 ok: [10.127.3.18]

TASK [Intialise the DB as postgres user] ***************************************
 changed: [10.127.3.18]

TASK [Start the DB server as postgres user and enable at boot] *****************
 ok: [10.127.3.18]

TASK [Set Port binding] ********************************************************
 changed: [10.127.3.18]

TASK [Set Interface binding] ***************************************************
 ok: [10.127.3.18]

TASK [Configure memory parameters ( shared_buffers = 248MB )] ******************
 ok: [10.127.3.18]

TASK [Configure memory parameters ( work_mem = 9MB )] **************************
 ok: [10.127.3.18]

TASK [Configure memory parameters ( maintenance_work_mem = 124MB )] ************
 ok: [10.127.3.18]

TASK [Configure memory parameters ( wal_buffers = 64MB )] **********************
 ok: [10.127.3.18]

TASK [Configure memory parameters ( effective_cache_size = 746MB )] ************
 ok: [10.127.3.18]

TASK [Restarting the postgres service] *****************************************
 changed: [10.127.3.18]

TASK [Create database named dbdemo] ********************************************
 ok: [10.127.3.18]

TASK [Setup database user] *****************************************************
 ok: [10.127.3.18]

TASK [Ensure user does not have unnecessary privileges] ************************
 ok: [10.127.3.18]

TASK [Configuring DB remote access in pg_hba.conf] *****************************
 changed: [10.127.3.18]

TASK [Restarting the postgres service] *****************************************
 changed: [10.127.3.18]

TASK [Perform database connection test] ****************************************
 changed: [10.127.3.18]

TASK [debug] *******************************************************************
 skipping: [10.127.3.18]

PLAY RECAP *********************************************************************
 10.127.3.18 : ok=30 changed=11 unreachable=0 failed=0

Another top tip when populating variables in Ansible is the ability to search in the variable data (stdout & stderr) and act on a keyword.

The following post_tasks example shows the db_is_in_recovery variable being populated with the output of a SQL query that checks whether a PostgreSQL hot standby database is in synchronization with its master. The check will fail when “(0 rows)” are returned.

post_tasks:
 - shell: sleep 10

- name: Perform database recovery check
 command: psql -p {{db_port}} -d {{db_name}} -U postgres -c "select pg_last_xlog_receive_location() "receive_location", pg_last_xlog_replay_location() "replay_location" where pg_last_xlog_receive_location() = pg_last_xlog_replay_location();"
 become: true
 become_user: postgres
 register: db_is_in_recovery
 ignore_errors: True
 failed_when: "'(0 rows)' in db_is_in_recovery.stdout"
 
 - debug:
 var: db_is_in_recovery.stdout

The runtime output:

TASK [Perform database recovery check] *****************************************
changed: [10.127.3.187]

TASK [debug] *******************************************************************
ok: [10.127.3.187] => {
 "db_is_in_recovery.stdout": " receive_location | replay_location \n------------------+-----------------\n 0/3000060 | 0/3000060\n(1 row)"
}

[contact-form][contact-field label=’Name’ type=’name’ required=’1’/][contact-field label=’Email’ type=’email’ required=’1’/][contact-field label=’Website’ type=’url’/][contact-field label=’Comment’ type=’textarea’ required=’1’/][/contact-form]