I’ve had a rough history with automation. I tried to automate some things between multiple wordpress sites in BASH and ended up writing this horrendous thing that I only show you because of how bad it is: https://github.com/felixtheratruns/bash-wp-porter Over a thousand lines of bash code that could have been written in python or some other more organizable language. I’ve switched to using Ansible more recently.

I really like a lot of things that Ansible is able to do but I noticed there were several use cases where Ansible almost would work but one small thing held it back. For instance, I would usually have to make a new playbook for every site times the number of operations I would do on it. If it was an operation that required two sites then this number would be multiplied by the number of sites the operation would be done with. This makes Ansible rather cluttered even when you try to separate things out nicely into different folders. More importantly, it was time-consuming to create these playbooks.

I thought I could fix this by using compound variables and referencing variables in group_vars/all.yaml changing which variables I referenced by using –extra-vars to change what groups of data in all.yaml the script is acting on. However, compound variables are currently not a feature in Ansible’s template language: Jinja2. So these are things I did to fix this problem:

Before we start I ran this bash script in every folder with a playbook in order for my playbooks to see the hosts file and the group_vars, lookup_plugins, and roles directories.

#!/usr/bin/env bash
ln -s ../hosts hosts
ln -s ../group_vars group_vars
ln -s ../lookup_plugins lookup_plugins
ln -s ../roles roles

1 Modify the Ansible “Lookup Vars” plugin. In my case I changed the LookupModule class to this: (apologies for the bad formatting and this definitely could have been coded in a less messy way) I’m still in the process of changing the documentation so some of it may be incorrect.

#vals plugin (based off of vars ansible lookup_plugin)
#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                    Î#
#Î    Company: Greenwell                                                      Î#

# (c) 2017 Ansible Project
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

DOCUMENTATION = """
    lookup: vals
    author: Ansible Core
    short_description: Lookup value of variables in hostvars
    description:
      - Retrieves the value of an Ansible variable.
    options:
      _terms:
        description: The variable names to look up.
        required: True
      default:
        description:
            - What to return if a variable is undefined.
            - If no default is set, it will result in an error if any of the variables is undefined.
"""

EXAMPLES = """
- name: Show value of 'variablename'
  debug: msg="{{ lookup('vals', 'variabl' + myvar)}}"
  vars:
    variablename: hello
    myvar: ename
- name: Show default empty since i dont have 'variablnotename'
  debug: msg="{{ lookup('vals', 'variabl' + myvar, default='')}}"
  vars:
    variablename: hello
    myvar: notename
- name: Produce an error since i dont have 'variablnotename'
  debug: msg="{{ lookup('vals', 'variabl' + myvar)}}"
  ignore_errors: True
  vars:
    variablename: hello
    myvar: notename
- name: find several related variables
  debug: msg="{{ lookup('vals', 'ansible_play_hosts', 'ansible_play_batch', 'ansible_play_hosts_all') }}"
- name: alternate way to find some 'prefixed vars' in loop
  debug: msg="{{ lookup('vals', 'ansible_play_' + item) }}"
  loop:
    - hosts
    - batch
    - hosts_all
"""

RETURN = """
_value:
  description:
    - value of the variables requested.
"""

from ansible.errors import AnsibleError, AnsibleUndefinedVariable
from ansible.module_utils.six import string_types
from ansible.plugins.lookup import LookupBase

class LookupModule(LookupBase):

    def run(self, terms, variables=None, **kwargs):
        if variables is not None:
            self._templar.set_available_variables(variables)
        myvals = getattr(self._templar, '_available_variables', {})
        
        self.set_options(direct=kwargs)
        default = self.get_option('default')

        ret = []
        for term in terms:
            if not isinstance(term, string_types):
                raise AnsibleError('Invalid setting identifier, "%s" is not a string, its a %s' % (term, type(term)))

            try:
                value = myvals['hostvars'][myvals['inventory_hostname']]
                new_value = value 
                
                import re
                pattern="(\[[^\[\\\\]*(?:\\\\.[^\]\\\\]*)*\])"

                positions = [(m.start(0), m.end(0)) for m in re.finditer(pattern, term)]
                fa = re.findall(pattern,term)
                count = 0
                arr = []
                
                build_s = ""
                for i, char in enumerate(term):
                    for pos in positions:
                        if i == pos[0] or i == pos[1]:
                            arr.append(build_s)
                            build_s=""
                        else:
                            continue
                    build_s += char
                
                if build_s != "":
                    arr.append(build_s)
                
                new_arr = []
                for el in arr:
                    if el.startswith('[') and el.endswith(']'):
                        new_arr.append(el.lstrip('[').rstrip(']'))
                    else:
                        if "." in el:
                            for e in el.split("."):
                                new_arr.append(e)
                        else:
                            new_arr.append(el)
                #if you have a problem with it generating blank keys uncomment this:
                #        new_arr = list(filter(None, new_arr))

                for i in new_arr:
                    new_value = new_value[i] 

                ret.append(self._templar.template(new_value, fail_on_undefined=True))
            except AnsibleUndefinedVariable:
                if default is not None:
                    ret.append(default)
                else:
                    raise

        return ret

2 To save time in creating the new variables and make variables for the reusable scripts automatically I wrote a script that gets the data automatically from the server. In this case I am dealing with wordpress:

#!/usr/bin/env bash
#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#

re='^[0-9]+$'
if [ -n "$1" ] && [[ $1 =~ $re ]]; then
  count_loop=$1
else
  count_loop=0
fi
search_folder=$2

num_lines=`find "$search_folder" -not -path '*/\.*' -name wp-config.php -print 2>&1 | grep -v "Permission denied" | wc -l`
loop_length=$((num_lines-1))
printf "\n"
counter=0
find "$search_folder" -not -path '*/\.*' -name wp-config.php -print 2>&1 | grep -v "Permission denied" | while read line
do
    full_path=
    rel_path=
    table_prefix=
    WPDBNAME=
    WPDBUSER=
    WPDBPASS=
    wp_options=
    start_mysql=
    pass=
    end_mysql=
    mysql_args=
    query_t=
    query=
    tmp=
    siteurl_nq=
    siteurl=
    folders=
    repo_url_temp=
    repo_folder=
    repo_url=
    full_folder_path=
    repo_branch=
    repo_branch_temp=
    site_title_nq=
    site_title=
    tmp=
    remote_host=

    printf "\n"	
    printf "\"$count_loop\": {"

    printf "\n"
    full_path="$line"
    table_prefix=`cat $line | grep "^[^#;]" | grep "table_prefix" | cut -d \' -f 2`
    WPDBNAME=`cat $line | grep "^[^#;]" | grep DB_NAME | cut -d \' -f 4`
    WPDBUSER=`cat $line | grep "^[^#;]" | grep DB_USER | cut -d \' -f 4`
    WPDBPASS=`cat $line | grep "^[^#;]" | grep DB_PASSWORD | cut -d \' -f 4`
    WPDBHOST=`cat $line | grep "^[^#;]" | grep DB_HOST | cut -d \' -f 4`
    WPDBNAME=`echo "${WPDBNAME[@]}" | head -1`
    WPDBUSER=`echo "${WPDBUSER[@]}" | head -1`
    WPDBPASS=`echo "${WPDBPASS[@]}" | head -1`
    WPDBHOST=`echo "${WPDBHOST[@]}" | head -1`
    table_prefix=`echo "${table_prefix[@]}" | head -1` 

    

    rel_path="$line"
    printf "%s%s\n" '"rel_path":' "\"$rel_path\","
    printf "%s%s\n" '"user_folder":' "\"`echo ~`\","
    printf "%s%s\n" '"full_path":' "\"$full_path\","
    printf "%s%s\n" '"table_prefix":' "\"$table_prefix\","
    printf "%s%s\n" '"WPDBNAME":' "\"$WPDBNAME\","
    printf "%s%s\n" '"WPDBUSER":' "\"$WPDBUSER\","
    printf "%s%s\n" '"WPDBPASS":' "\"$WPDBPASS\","
 
    wp_options=$table_prefix"options"
    wp_users=$table_prefix"users"
    wp_usermeta=$table_prefix"usermeta"

    full_host="$WPDBHOST"
    if [[ $full_host == *":"* ]]; then #[fix]
        IFS=':' read -ra full_host_ar <<< "$full_host"
        host="${full_host_ar[0]}"
        port="${full_host_ar[1]}"
    else
        host="${full_host}"
        port=3306
    fi

    start_mysql="-u$WPDBUSER -p"
    pass="'$WPDBPASS'"
    end_mysql=" -h$host -P$port $WPDBNAME" 
    mysql_args=$start_mysql"$pass"$end_mysql
    
    query_t=$(mysql -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT option_value FROM $wp_options WHERE option_name = 'siteurl';" 2>/dev/null) 
    query=${query_t//[^a-zA-Z0-9_:\.\/\-\~] /}
    tmp=($query)
    tmp_nq=${tmp[0]}
    siteurl="$tmp_nq"
    printf "%s%s\n" '"site_url":' "\"$siteurl\","
   
    query_t=
    query_t=$(mysql -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT option_value FROM $wp_options WHERE option_name = 'blogname';" 2>/dev/null)
    query=${query_t//[^a-zA-Z0-9_:\.\/\-\~ ] /}
#    tmp=($query)
#    tmp_nq=${tmp[0]}
    site_title="$query"
    printf "%s%s\n" '"site_title":' "\"$site_title\","

    query_t=
    query_t=$(mysql  -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT DISTINCT user_login FROM $wp_users INNER JOIN $wp_usermeta ON $wp_users.ID = $wp_usermeta.user_id WHERE $wp_usermeta.meta_value LIKE '%administrator%' ORDER BY $wp_users.ID;" 2>/dev/null)
    printf "%s" '"admin_users":'
    printf "%s" '['
    length=$(echo "$query_t" | wc -l)
    count=0
    while IFS="" read -r p || [ -n "$p" ]
    do
        if (( $count < $length - 1 )); then
            printf '"%s",' "$p"
        else
            printf '"%s"' "$p"
        fi
        ((count++))

    done <<< "$query_t"
    printf "%s\n" '],'
 
    query_t=
    query_t=$(mysql  -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT DISTINCT user_login, user_pass FROM $wp_users INNER JOIN $wp_usermeta ON $wp_users.ID = $wp_usermeta.user_id WHERE $wp_usermeta.meta_value LIKE '%administrator%'; ORDER BY $wp_users.ID" 2>/dev/null)
    printf "%s" '"hashed_passwords":' 
    printf "%s" '{'
    length=$(echo "$query_t" | wc -l)
    count=0
    while IFS="" read -r p || [ -n "$p" ]
    do
        p_arr=
        IFS=$'\t' read -r -a p_arr <<< "$p"
        if (( $count < $length - 1 )); then
            printf '"%s": "%s",' "${p_arr[0]}" "${p_arr[1]}"
        else
            printf '"%s": "%s"' "${p_arr[0]}" "${p_arr[1]}"
        fi
        ((count++))
    done <<< "$query_t" 
    printf "%s\n" '},'
   
    query_t=
    query_t=$(mysql  -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT DISTINCT user_login, user_email FROM $wp_users INNER JOIN $wp_usermeta ON $wp_users.ID = $wp_usermeta.user_id WHERE $wp_usermeta.meta_value LIKE '%administrator%' ORDER BY $wp_users.ID;" 2>/dev/null)
    printf "%s" '"admin_emails":'
    printf "%s" '{'
    length=$(echo "$query_t" | wc -l)
    count=0
    while IFS="" read -r p || [ -n "$p" ]
    do
       p_arr=
       IFS=$'\t' read -r -a p_arr <<< "$p"
       if (( $count < $length - 1 )); then
            printf '"%s": "%s",' "${p_arr[0]}" "${p_arr[1]}"
        else
            printf '"%s": "%s"' "${p_arr[0]}" "${p_arr[1]}"
        fi
        ((count++))
    done <<< "$query_t" 
    printf "%s\n" '},'
    
    

    full_folder_path=$(dirname $full_path)
    folders=${full_folder_path##/*/}
    repo_url_temp=$(cd "$full_folder_path" && git config --get remote.origin.url)
   	if [ -z $repo_url_temp ]; then
     	  : 
   	else
        repo_branch_temp=$(cd "$full_folder_path" && git rev-parse --abbrev-ref HEAD 2>/dev/null)
        repo_folder="$full_folder_path"
   	    repo_url=$repo_url_temp
        repo_branch=$repo_branch_temp
   	fi

    printf "%s%s\n" '"host":' "\"$host\","
    printf "%s%s\n" '"port":' "\"$port\","
    printf "%s%s\n" '"repo_folder":' "\"$repo_folder\","
    printf "%s%s\n" '"repo_url":' "\"$repo_url\","
    printf "%s%s\n" '"repo_branch":' "\"$repo_branch\","
    remote_host=`hostname`
    printf "%s%s\n" '"remote_host":' "\"$remote_host\""
    if (( $counter < $num_lines - 1 )); then
        printf "},"
    else
        printf "}"
    fi

    printf "\n"
    count_loop=$((count_loop+1))
    counter=$((counter+1))
    
done

3 That data needs to be parsed into yaml and there’s some issues of the output of the BASH script: it’s not proper json because it isn’t enclosed in brackets like this {data} So we have to run the script on the server like this to put everything together:

#!/usr/bin/env python

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


import subprocess
from get_keys import OutputProc 
import json
import os
import sys 



start_num=0
output = "\n{"
op = OutputProc()
search_folder = "/var/www"
output += op.run_cmd("localhost", "user1", 22, start_num, search_folder)

#make first command's output into proper json:
output_com = output + "\n}"

#get the number of elements it has:
print(output_com)
start_num=len(json.loads(output_com))
#make comma in json text to add next command's output to it:
output += ",\n"
#add the command:
search_folder = "/home/user2"
output += op.run_cmd("greenwell.tech", "user2", 5000, start_num, search_folder)
#close the json
output += "}"


#load and write json to file
json_out = json.loads(output)
full_path = os.path.realpath(sys.argv[0])
fname = "/data.json"
f = open(os.path.dirname(full_path) + fname, 'w');
f.write(json.dumps(json_out, indent=4));
f.close()

get_keys comes from this file here: (credit to https://www.bogotobogo.com/python/python_ssh_remote_run.php for the code I started from)

#!/usr/bin/env python

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


import os, subprocess, sys, argparse, json



class OutputProc():
    def run_cmd(self, host, user, port, start_num, search_folder):
        return self.remote_run(host, user, port, start_num, search_folder)

    def check_arg(self, args=None):
        parser = argparse.ArgumentParser(description='script to get website data from remote server')
        parser.add_argument('-H', '--host',
                            help='host ip')
        parser.add_argument('-u', '--user',
                            help='user name')
        parser.add_argument('-p', '--port',
                            help='port number')
        return parser.parse_args(args)
   

    def remote_run(self, host, user, port, start_num, search_folder):
        print("host: " + str(host))
        print("user: " + str(user))
        print("port: " + str(port))
        cmd_list = []
        with open('commands.txt', 'r') as f:
            for c in f:
                cmd_list.append(c.replace('\n','').replace('\r','').rstrip())
   
        exe = {'py': 'python3', 'sh': 'sh', 'pl': 'perl', 'bash': 'bash'}
    
        ssh = ''.join(['ssh ', ' -p', str(port), ' ', user, '@',  host, ' '])
        print("ssh: " + ssh) 
        for cmd in cmd_list:
            exe_type = exe[cmd.split('.')[1]]
            if host == "localhost":
                cmd = exe_type + " " + cmd + " " + str(start_num) + " " + str(search_folder) + ' 2>&1'
            else:
                cmd = ssh + '"' + exe_type + ' -s' + '"' + ' < ' + cmd + " " + str(start_num) + " " + str(search_folder) + ' 2>&1'
            print("cmd: " + cmd)
            result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
            return result.stdout.read().strip().decode('utf-8')

commands.txt is where the command for the script resides: (credit to https://www.bogotobogo.com/python/python_ssh_remote_run.php for the code I started from)

get-keys.bash

So when we run the scripts that we want to be able to pass –extra-vars to point to different values we get from the server. This will be explained later. Hopefully, that will give you an idea of how I am getting credentials in the form of JSON data from the server. I guess ill include parsing the JSON here because it is all one program. Here is the parsing function that you run to parse the JSON to get it into a format that’s more human readable in YAML and to create some additional variables that can be derived by parsing it.

#!/usr/bin/env python

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


import sys
import yaml
import os
import json
from data import Node
from data import NodeWrap
from data import Company
from storage import MySingleton 
from formatter import Formatter


#print('sys.argv[0] =', sys.argv[0])             
#pathname = os.path.dirname(sys.argv[0])        
#print('path =', pathname)
#print('full path =', os.path.abspath(pathname))


def wat_path(path):
    if '/staging/' in path:
        return "staging"
    elif '/public_html/' in path:
        return "production"
    return "unknown"


def print_nested_dict(d):
    for k, v in d.items():
        if isinstance(v, dict):
            print(">" + k)
            print_nested_dict(v)
            print("")
        else:
            print("{0} : {1}".format(k, v))

def parse_data(singleton):
    fname = "/data.json"
    full_path = os.path.realpath(sys.argv[0])
    with open( os.path.dirname(full_path) +  fname) as f:
        content = f.readlines()
    # you may also want to remove whitespace characters like `\n` at the end of each line
    content = [x.strip() for x in content] 
    
    with open( os.path.dirname(full_path) + '/data.json') as data_file:
        data = json.load(data_file)
   
    for idx, val in enumerate(data):

        node = NodeWrap(Node,
        data[val].get("rel_path",""),
        data[val].get("user_folder",""),
        data[val].get("full_path",""),
        data[val].get("table_prefix",""),
        data[val].get("WPDBNAME",""),
        data[val].get("WPDBUSER",""),
        data[val].get("WPDBPASS",""),
        data[val].get("host",""),
        data[val].get("port",""),
        data[val].get("site_url",""),
        data[val].get("site_title",""),
        data[val].get("admin_users",""),
        data[val].get("admin_emails",""),
        data[val].get("hashed_passwords",""),
        data[val].get("repo_folder",""),
        data[val].get("repo_url",""),
        data[val].get("repo_branch",""),
        data[val].get("remote_host",""))
   
        tag = node.make_tag()
        sitemap_host_name = node.make_sitemap_host_name()

        company_specific = Company(
        tag, 
        sitemap_host_name, 
        data[val].get("WPDBNAME",""),
        data[val].get("WPDBUSER",""),
        "",
        "",
        data[val].get("WPDBPASS",""),
        "",
        "").__dict__

        node.set_company(company_specific)

        site_group = node.get_site_group()
        site_type = node.get_site_type()
        singleton.add_val(site_group, site_type, node.__dict__)


#might use this to rewrite the "put variables inside global with a hostname variable" below:
#def extend(a,b):
#    """Create a new dictionary with a's properties extended by b,
#    without overwriting.
#
#    >>> extend({'a':1,'b':2},{'b':3,'c':4})
#    {'a': 1, 'c': 4, 'b': 2}
#    """
#    return dict(b,**a)

def write_yaml(singleton):
    #put variables inside global with a hostname variable
#    set_of_nodes = { "global": {} }
#    temp_dict = { "hostname": "{{ inventory_hostname }}" }
#    temp_dict.update(singleton.get_nodes())
#    set_of_nodes["global"] = temp_dict 
#    singleton.set_nodes(set_of_nodes)
    
    pretty = Formatter()
    print(pretty(singleton.get_nodes()))
    
    json_dumps = json.dumps(singleton.get_nodes())
    yaml_nodes = yaml.safe_load(json_dumps)
    full_path = os.path.realpath(sys.argv[0])
    f = open(os.path.dirname(full_path) + '/group_vars/all.yaml', 'w+');
    f.write(yaml.dump(yaml_nodes, default_flow_style=False));
    f.close()


if __name__ == '__main__':
    singleton = MySingleton()
    singleton.set_nodes({})
        
    
    parse_data(singleton)
    write_yaml(singleton)
    

It imports from data.py, storage.py, and formatter.py as you can see above. data.py is something I wrote, storage.py is just slightly modified from something I found online and formatter.py is entirely not my work.

#!/usr/bin/env python

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


import os
import random
import string

class NodeWrap(dict):
    def __init__(self, wrapped_Node, rel_path, user_folder, full_path, table_prefix, db_name, db_user, db_pass, host, port, site_url, site_title, admin_users, admin_emails, hashed_passwords, repo_folder, repo_url, repo_branch, remote_host):
        home_folder = self.remove_postfix(full_path, "wp-config.php")
        site_folder = self.remove_postfix(full_path, "wp-config.php")    
        site_type = self.what_type_from_url(site_url)
        admin_user = admin_users[0]
        admin_email = admin_emails[admin_user] 
        hashed_password = hashed_passwords[admin_user]
        self.set_user_folder(user_folder)
        self.set_user(os.path.basename(user_folder))
        self.wrapped_Node = wrapped_Node(rel_path, full_path, table_prefix, db_name, db_user, db_pass, host, port, site_url, site_title, admin_users, admin_user, admin_emails, admin_email, hashed_passwords, hashed_password, repo_folder, repo_url, repo_branch).__dict__
        self.set_home_folder(home_folder)
        self.set_site_folder(site_folder)
        self.set_site_type(site_type)
        site_group_tmp = repo_url 
        site_group_tmp = self.index_split_or_original(1,"/",site_group_tmp)
        site_group_tmp = self.index_split_or_original(0,".",site_group_tmp)
        self.set_site_group(site_group_tmp)
        self.set_remote_host(remote_host)

    def get_site_type(self):
        return self.site_type

    def set_user_folder(self, user_folder):
        self.user_folder = user_folder

    def set_home_folder(self, home_folder):
        self.home_folder = home_folder

    def set_site_folder(self, site_folder):
        self.site_folder = site_folder

    def set_site_type(self, site_type):
        self.site_type = site_type

    def set_site_group(self, site_group):
        self.site_group = site_group
 
    def get_site_group(self):
        return self.site_group

    def remove_prefix(self, text, prefix):
        if text.startswith(prefix):
            return text[len(prefix):]
        return text  # or whatever
    
    def remove_postfix(self, text, postfix):
        if text.endswith(postfix):
            return text[:-len(postfix)]
        return text  # or whatever

    def what_type_from_url(self, url):
        if 'localhost' in url:
            return 'localhost'
        elif 'greenwell.tech' in url:
            return "staging"
        return "production" 

    def set_remote_host(self, remote_host):
        self.remote_host = remote_host

    def index_split_or_original(self, idx, split_char, string):
        if ( split_char in string and len(string.split(split_char)) > 1):
            return string.split(split_char)[idx]
        else:
            return "" 
   
    def __getattr__(self,attr):
        orig_attr = self.wrapped_Node.__getattribute__(attr)
        if callable(orig_attr):
            def hooked(*args, **kwargs):
                result = orig_attr(*args, **kwargs)
                #prevent wrapped_class from becomming unwrapped
                if result == self.wrapped_Node:
                    return self
                return result
            return hooked
        else:
            return orig_attr

    #making functions
    def make_tag(self):
        dirname = os.path.dirname(self.home_folder)
        basename = os.path.basename(dirname)
        if "." in basename:
            return basename.split(".")[0]
        else:
            return basename

    def make_sitemap_host_name(self):
        remote_h = ""
        if self.remote_host == "":
            pass
        elif "." in self.remote_host:
            remote_h = self.remote_host.split(".")[0]    
        else:
            return self.remote_host 
        return remote_h

    def set_company(self, company_specific):
        self.company_specific = company_specific

    def set_user(self, user):
        self.user = user


class Node(dict):
    def __init__(self, rel_path, full_path, table_prefix, db_name, db_user, db_pass, host, port, site_url, site_title, admin_users, admin_user, admin_emails, admin_email, hashed_passwords, hashed_password, repo_folder, repo_url, repo_branch):
        self.set_rel_path(rel_path)
        self.set_full_path(full_path)
        self.set_table_prefix(table_prefix)
        self.set_db_name(db_name)
        self.set_db_user(db_user)
        self.set_db_pass(db_pass)
        self.set_host(host)
        self.set_port(port)
        self.set_site_url(site_url)
        self.set_site_title(site_title)
        self.set_admin_users(admin_users)
        self.set_admin_user(admin_user)
        self.set_admin_emails(admin_emails)
        self.set_admin_email(admin_email)
        self.set_hashed_passwords(hashed_passwords)
        self.set_hashed_password(hashed_password)
        self.set_repo_folder(repo_folder)
        self.set_repo_url(repo_url)
        self.set_repo_branch(repo_branch)

    def set_name(self, name):
        self.name = name

    def set_rel_path(self, rel_path):
        self.rel_path = rel_path

    def set_full_path(self, full_path):
        self.full_path = full_path

    def set_table_prefix(self, table_prefix):
        self.table_prefix = table_prefix

    def set_db_name(self, db_name):
        self.db_name = db_name

    def set_db_user(self, db_user):
        self.db_user = db_user

    def set_db_pass(self, db_pass):
        self.db_pass = db_pass

    def set_host(self, host):
        self.host = host 

    def set_port(self, port):
        self.port = port 

    def set_site_url(self, site_url):
        self.site_url = site_url

    def set_site_title(self, site_title):
        self.site_title = site_title

    def set_admin_users(self, admin_users):
        self.admin_users = admin_users 
 
    def set_admin_user(self, admin_user):
        self.admin_user = admin_user 

    def set_admin_emails(self, admin_emails):
        self.admin_emails = admin_emails 
 
    def set_admin_email(self, admin_email):
        self.admin_email = admin_email 
     
    def set_hashed_passwords(self, hashed_passwords):
        self.hashed_passwords = hashed_passwords 

    def set_hashed_password(self, hashed_password):
        self.hashed_password = hashed_password 

    def set_repo_folder(self, repo_folder):
        self.repo_folder = repo_folder

    def set_repo_url(self, repo_url):
        self.repo_url = repo_url

    def set_repo_branch(self, repo_branch):
        self.repo_branch = repo_branch

class Company(dict):
    def __init__(self, tag, sitemap_host_name, dbname, dbuser, sftp_u, sftp_p, dbpass, client_pass, sftp_pass):
        self.set_tag(tag)

        if dbname == "":
            self.set_dbname( "prefix_" + tag )
        else:
            self.set_dbname( dbname )

        self.set_sitemap_host_name(sitemap_host_name)

        if dbuser == "":
            self.set_dbuser( "prefix" + tag )
        else:
            self.set_dbuser( dbuser )

        if sftp_u == "":
            self.set_sftp_u( "prefix" + tag )
        else:
            self.set_sftp_u( sftp_u )
        #passwords       
        password = self.randomString(15)
        if sftp_p == "":
            self.set_sftp_p( password )
        else:
            self.set_sftp_p( sftp_p )

        password = self.randomString(15)
        if sftp_p == "":
            self.set_sftp_p( password )
        else:
            self.set_sftp_p( sftp_p )
        
        password = self.randomString(15)
        if dbpass == "":
            self.set_dbpass( password )
        else:
            self.set_dbpass( dbpass )


        password = self.randomString(15)
        if client_pass == "":
            self.set_client_pass( password )
        else:
            self.set_client_pass( client_pass )

        password = self.randomString(15)
        if sftp_pass == "":
            self.set_sftp_pass ( password )
        else:
            self.set_sftp_pass ( sftp_pass )
            

    def set_tag(self, tag):
        self.tag = tag
 
    def set_sitemap_host_name(self, sitemap_host_name):
        self.sitemap_host_name = sitemap_host_name
   
    def set_dbname(self, dbname):
        self.dbname = dbname

    def set_sftp_u(self, sftp_u):
        self.sftp_u = sftp_u

    def set_sftp_p(self, sftp_p):
        self.sftp_p = sftp_p

    def set_dbpass(self, dbpass):
        self.dbpass = dbpass

    def set_dbuser(self, dbuser):
        self.dbuser = dbuser

    def set_client_pass(self, client_pass):
        self.client_pass = client_pass

    def set_sftp_pass(self, sftp_pass):
        self.sftp_pass = sftp_pass

    def randomString(self, stringLength):
        """Generate a random string with the combination of lowercase and uppercase letters """
        letters = string.ascii_letters
        return ''.join(random.choice(letters) for i in range(stringLength))

Here is storage.py. Thanks to Michael for making available the original code. https://michaelgoerz.net/notes/singleton-objects-in-python.html I’m not sure if I’ve modified it enough to need to put a license on it but I’ve decided to do so just in case:

#!/usr/bin/env python

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


class Singleton(type):
    """Metaclass for singletons. Any instantiation of a Singleton class yields
    the exact same object, e.g.:

    >>> class MyClass(metaclass=Singleton):
            pass
    >>> a = MyClass()
    >>> b = MyClass()
    >>> a is b
    True
    """
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args,
                                                                **kwargs)
        return cls._instances[cls]

    @classmethod
    def __instancecheck__(mcs, instance):
        if instance.__class__ is mcs:
            return True
        else:
            return isinstance(instance.__class__, mcs)


def singleton_object(cls):
    """Class decorator that transforms (and replaces) a class definition (which
    must have a Singleton metaclass) with the actual singleton object. Ensures
    that the resulting object can still be "instantiated" (i.e., called),
    returning the same object. Also ensures the object can be pickled, is
    hashable, and has the correct string representation (the name of the
    singleton)
    """
    assert isinstance(cls, Singleton), \
        cls.__name__ + " must use Singleton metaclass"

    def self_instantiate(self):
        return self

    cls.__call__ = self_instantiate
    cls.__hash__ = lambda self: hash(cls)
    cls.__repr__ = lambda self: cls.__name__
    cls.__reduce__ = lambda self: cls.__name__
    obj = cls()
    obj.__name__ = cls.__name__
    return obj


@singleton_object
class MySingleton(metaclass=Singleton):
    def set_nodes(self, dictionary):
        self.nodes = dictionary

    def get_nodes(self):
        return self.nodes
 
    def get_node(self, key):
        return self.nodes[key]
    
    def set_nodes(self, temp_dict):
        self.nodes = temp_dict
    
    def set_key(self, key, temp_dict):
        self.nodes[key][temp_dict]
    
    def add_val(self, site_group, site_type, node):
        if self.nodes.get(site_group,"") == "":
            temp_dict = { site_type: node }
            self.nodes[site_group] =  temp_dict
        elif self.nodes[site_group].get(site_type,"") == "":
            self.nodes[site_group][site_type] = node
        else:
            count = 1
            new_site_type = site_type + str(count)
            while self.nodes[site_group].get(new_site_type ,"") != "":
                count += 1
                new_site_type = site_type + str(count)
            self.nodes[site_group][new_site_type] = node

    def get_val_from_str(self, string):
        arr = string.split('.')
        nodes = self.nodes
        for key, val in enumerate(arr):
            nodes = nodes[val]
        return nodes

    def return_main_keys(self):
        key_list = []
        for key in self.nodes:
            print(key)
            for key2 in self.nodes[key]:
                input_key = key + "." + key2
                key_list.append(input_key)
                print(str(len(key_list) -1) + ">" + input_key)
        return key_list


class SingletonType(metaclass=Singleton):
    pass


Finally I will post formatter.py here for your convenience. Since it’s not any of my work it will be the only one not licensed under the gpl:

class Formatter(object):
    def __init__(self):
        self.types = {}
        self.htchar = '\t'
        self.lfchar = '\n'
        self.indent = 0
        self.set_formater(object, self.__class__.format_object)
        self.set_formater(dict, self.__class__.format_dict)
        self.set_formater(list, self.__class__.format_list)
        self.set_formater(tuple, self.__class__.format_tuple)

    def set_formater(self, obj, callback):
        self.types[obj] = callback

    def __call__(self, value, **args):
        for key in args:
            setattr(self, key, args[key])
        formater = self.types[type(value) if type(value) in self.types else object]
        return formater(self, value, self.indent)

    def format_object(self, value, indent):
        return repr(value)

    def format_dict(self, value, indent):
        items = [
            self.lfchar + self.htchar * (indent + 1) + repr(key) + ': ' +
            (self.types[type(value[key]) if type(value[key]) in self.types else object])(self, value[key], indent + 1)
            for key in value
        ]
        return '{%s}' % (','.join(items) + self.lfchar + self.htchar * indent)

    def format_list(self, value, indent):
        items = [
            self.lfchar + self.htchar * (indent + 1) + (self.types[type(item) if type(item) in self.types else object])(self, item, indent + 1)
            for item in value
        ]
        return '[%s]' % (','.join(items) + self.lfchar + self.htchar * indent)

    def format_tuple(self, value, indent):
        items = [
            self.lfchar + self.htchar * (indent + 1) + (self.types[type(item) if type(item) in self.types else object])(self, item, indent + 1)
            for item in value
        ]
        return '(%s)' % (','.join(items) + self.lfchar + self.htchar * indent)


#To initialize it :
#
#pretty = Formatter()
#It can support the addition of formatters for defined types, you simply need to make a function for that like this one and bind it to the type you want with set_formater :
#
#from collections import OrderedDict
#
#def format_ordereddict(self, value, indent):
#    items = [
#        self.lfchar + self.htchar * (indent + 1) +
#        "(" + repr(key) + ', ' + (self.types[
#            type(value[key]) if type(value[key]) in self.types else object
#        ])(self, value[key], indent + 1) + ")"
#        for key in value
#    ]
#    return 'OrderedDict([%s])' % (','.join(items) +
#           self.lfchar + self.htchar * indent)
#pretty.set_formater(OrderedDict, format_ordereddict)
#For historical reasons, I keep the previous pretty printer which was a function instead of a class, but they both can be used the same way, the class version simply permit much more :
#
#def pretty(value, htchar='\t', lfchar='\n', indent=0):
#    nlch = lfchar + htchar * (indent + 1)
#    if type(value) is dict:
#        items = [
#            nlch + repr(key) + ': ' + pretty(value[key], htchar, lfchar, indent + 1)
#            for key in value
#        ]
#        return '{%s}' % (','.join(items) + lfchar + htchar * indent)
#    elif type(value) is list:
#        items = [
#            nlch + pretty(item, htchar, lfchar, indent + 1)
#            for item in value
#        ]
#        return '[%s]' % (','.join(items) + lfchar + htchar * indent)
#    elif type(value) is tuple:
#        items = [
#            nlch + pretty(item, htchar, lfchar, indent + 1)
#            for item in value
#        ]
#        return '(%s)' % (','.join(items) + lfchar + htchar * indent)
#    else:
#        return repr(value)
#To use it :
#
#>>> a = {'list':['a','b',1,2],'dict':{'a':1,2:'b'},'tuple':('a','b',1,2),'function':pretty,'unicode':u'\xa7',("tuple","key"):"valid"}
#>>> a
#{'function': <function pretty at 0x7fdf555809b0>, 'tuple': ('a', 'b', 1, 2), 'list': ['a', 'b', 1, 2], 'dict': {'a': 1, 2: 'b'}, 'unicode': u'\xa7', ('tuple', 'key'): 'valid'}
#>>> print(pretty(a))
#{
#    'function': <function pretty at 0x7fdf555809b0>,
#    'tuple': (
#        'a',
#        'b',
#        1,
#        2
#    ),
#    'list': [
#        'a',
#        'b',
#        1,
#        2
#    ],
#    'dict': {
#        'a': 1,
#        2: 'b'
#    },
#    'unicode': u'\xa7',
#    ('tuple', 'key'): 'valid'
#}

#Compared to other versions :
#
#This solution looks directly for object type, so you can pretty print almost everything, not only list or dict.
#Doesn't have any dependancy.
#Everything is put inside a string, so you can do whatever you want with it.
#The class and the function has been tested and works with Python 2.7 and 3.4.
#You can have all type of objects inside, this is their representations and not theirs contents that being put in the result (so string have quotes, Unicode string are fully represented ...).
#With the class version, you can add formatting for every object type you want or change them for already defined ones.
#key can be of any valid type.
#Indent and Newline character can be changed for everything we'd like.
#Dict, List and Tuples are pretty printed.

This is the helper function that allows you to be extra lazy creating –extra-vars. This just parses the yaml and lets you select what sites to have as a source and destination. The description of how it can be run is this: an.py ansible-playbook.yml (-h(source_host_number):[destination_host_number]) Here are some examples:

an.py ansible-playbook.yml #asks for source and destination host

an.py ansible-playbook.yml -h0:1 #automatically runs the source and destination host number

an.py ansible-playbook.yml -h:3 #automatically just runs on the destination

#!/usr/bin/env python

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#

import sys
import yaml
import os
import json



def parse_yaml(singleton):
    fname = "/group_vars/all.yaml"
    full_path = os.path.realpath(sys.argv[0])

def return_main_keys(data):
      key_list = []
      for key in data:
          print(key)
          for key2 in data[key]:
              input_key = key + "." + key2
              key_list.append(input_key)
              print(str(len(key_list) -1) + ">" + input_key)
      return key_list



fname = "/group_vars/all.yaml"
full_path = os.path.realpath(sys.argv[0])
with open( os.path.dirname(full_path) + '/group_vars/all.yaml') as data_file:
   data = yaml.load(data_file)
key_list = return_main_keys(data)

def remove_prefix(text, prefix):
    if text.startswith(prefix):
        return text[len(prefix):]
    return text  # or whatever


gv_sh=""
gv_dh=""

#start building bash command:
bcmd = ""
both_hosts_set = False
for arg in sys.argv[1:]:
    if arg.startswith("-h"):
        arg = remove_prefix(arg, "-h")
        arg_arr = arg.split(":")
        if arg_arr[0] == "":
            gv_dh = arg_arr[1]
            both_hosts_set = False
            break
        else:
            gv_sh = arg_arr[0]
            gv_dh = arg_arr[1]
            both_hosts_set = True
            break
   #^end if
else:
    both_hosts_set = True


for arg in sys.argv[1:]:
    if arg.startswith("-h"):
        continue
    else:
        bcmd += " " + arg


if len(sys.argv[1:]) == 0:
    both_hosts_set = True


if gv_sh == "" and both_hosts_set:
    gv_sh = input("what do you want your source host variables to be?")

if gv_dh == "":
    gv_dh = input("what do you want your destination host variables to be?")

if gv_sh != "":
    gv_sh = int(gv_sh)


gv_dh = int(gv_dh)


def get_val_from_str(string, data):
    arr = string.split('.')
    nodes = data
    for key, val in enumerate(arr):
        nodes = nodes[val]
    return nodes




bcmd += " "

bcmd += "--extra-vars="
bcmd += '"'

if gv_sh != "":
    bcmd += "gv_sh=" + key_list[gv_sh] + " "
bcmd += "gv_dh=" + key_list[gv_dh] + " "
if gv_sh != "":
    src_host_ref = key_list[gv_sh] + ".remote_host"
dest_host_ref = key_list[gv_dh] + ".remote_host"
if gv_sh != "":
    src_host = get_val_from_str(src_host_ref, data)
dest_host = get_val_from_str(dest_host_ref, data)
if gv_sh != "":
    bcmd += "src_host=" + src_host + " "
bcmd += "dest_host=" + dest_host
bcmd += '"'
bcmd = "ansible-playbook -i hosts" + bcmd
print("running command:")
print(bcmd)
os.system(bcmd)




The following code runs the previous an.py on multiple destination hosts with the same script. There is probably a way to do something similar with ansible I just haven’t though of it yet. It is run like so: “reu.py ansible-playbooky.yml

import sys
import yaml
import os
import json


def return_main_keys(data):
      key_list = []
      for key in data:
          print(key)
          for key2 in data[key]:
              input_key = key + "." + key2
              key_list.append(input_key)
              print(str(len(key_list) -1) + ">" + input_key)
      return key_list


fname = "/group_vars/all.yaml"
full_path = os.path.realpath(sys.argv[0])
with open( os.path.dirname(full_path) + '/group_vars/all.yaml') as data_file:
   data = yaml.load(data_file)
key_list = return_main_keys(data)


start_num = input("where do you want to start?")
stop_num = input("where do you want to stop?")
start_num=int(start_num)
stop_num=int(stop_num)

for num_to_be_worked_on, key in enumerate(key_list[start_num:stop_num+1], start=start_num):
        bcmd="an.py " + "-h:" + str(num_to_be_worked_on) + " " + sys.argv[1]
        print(bcmd)
        os.system(bcmd)
# example conditional you could add
#    if "localhost" not in key:
#        bcmd="an.py " + "-h:" + str(num_to_be_worked_on) + " " + sys.argv[1]
#        print(bcmd)
#        os.system(bcmd)

The actual Ansible scripts look like this:

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
  roles:
    - set-if-unset

# creates working examples
- import_playbook: up-wp-both/up-wp-both_create-work-ex.yml
- import_playbook: git-all/git-all_create-work-ex.yml
- import_playbook: cp-db/cp-db_create-work-ex.yml
- import_playbook: ch-url/ch-url_create-work-ex.yml
- import_playbook: up-up/up-up_create-work-ex.yml

#assembles the working examples
- hosts: localhost
  tasks:
  - name: assemble up-wp
    shell: cat up-wp-both/up-wp-both_work-ex.yml > ./output-ex.yml

- hosts: localhost
  tasks:
  - name: assemble git-all
    shell: cat git-all/git-all_work-ex.yml >> ./output-ex.yml

- hosts: localhost
  tasks:
  - name: assemble cp-db
    shell: cat cp-db/cp-db_work-ex.yml >> ./output-ex.yml

- hosts: localhost
  tasks:
  - name: assemble ch-url
    shell: cat ch-url/ch-url_work-ex.yml >> ./output-ex.yml

- hosts: localhost
  tasks:
  - name: assemble up-up
    shell: cat up-up/up-up_work-ex.yml >> ./output-ex.yml

# actually does the work/runs the playbooks:
- import_playbook: up-wp-both/up-wp-both_work.yml
- import_playbook: git-all/git-all_work.yml
- import_playbook: cp-db/cp-db_work.yml
- import_playbook: ch-url/ch-url_work.yml
- import_playbook: up-up/up-up_work.yml

When the playbooks are assembled (really just concatenated) in this playbook we will see what variables it puts in. This is useful to better see what the playbooks would do before we run them. I’m not going to show you these here because they contain sensitive data. However, I will assembly the playbooks before they have variables replaced and show you how variables are being looked up:

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#

#assembles the working scripts
- hosts: localhost
  tasks:
  - name: assemble up-wp
    shell: cat up-wp-both/up-wp-both_work.yml > ./output.yml

- hosts: localhost
  tasks:
  - name: assemble git-all
    shell: cat git-all/git-all_work.yml >> ./output.yml

- hosts: localhost
  tasks:
  - name: assemble cp-db
    shell: cat cp-db/cp-db_work.yml >> ./output.yml

- hosts: localhost
  tasks:
  - name: assemble ch-url
    shell: cat ch-url/ch-url_work.yml >> ./output.yml

- hosts: localhost
  tasks:
  - name: assemble up-up
    shell: cat up-up/up-up_work.yml >> ./output.yml

This creates the following:

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


- hosts: "{{ src_host_var | default(src_host) }}"
  vars:
    dest_site_folder: "{{ lookup('vals', gv_sh_var + '.site_folder') }}"
  roles:
    - set-if-unset
    - update-wp

- hosts: "{{ dest_host_var | default(dest_host) }}"
  vars:
    dest_site_folder: "{{ lookup('vals', gv_dh_var + '.site_folder') }}"
  roles:
    - set-if-unset
    - update-wp
- hosts: "{{ src_host_var | default(src_host) }}"
  vars:
    git_dir: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.repo_folder') }}"
    src_branch: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.repo_branch') }}"
    dest_branch: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.repo_branch') }}"
  roles:
    - set-if-unset
    - git-get-changes-to-repo

- hosts: "{{ dest_host_var | default(dest_host) }}"
  vars:
    dest_branch: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.repo_branch') }}"
    git_dir: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.repo_folder') }}"
  roles:
    - set-if-unset
    - git-repo-to-dest



- hosts: "{{ src_host_var | default(src_host) }}"
  vars:
    src_db_name: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.db_name' ) }}"
    src_login_user: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.db_user' ) }}"  #for mysql
    src_login_password: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.db_pass' ) }}" #for mysql
    src_login_host: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.host') }}" #for mysql
    src_login_port: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.port') }}" #for mysql
    src_dump_path: "{{ lookup('vals', gv_sh_var + '.user_folder' ) }}/{{ lookup('vals', gv_sh_var + '.site_group' ) }}_dump.sql"
    src_copy_path: "{{ lookup('vals', gv_sh_var + '.user_folder' ) }}/{{ lookup('vals', gv_sh_var + '.site_group' ) }}_dump.sql"
    dest_copy_path: "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}_dump.sql"
    wp_prefix: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.table_prefix' ) }}"
  roles:
    - set-if-unset
    - dump_db #dumps the database to a .sql file                                                
    - copy_file

- hosts: "{{ dest_host_var | default(dest_host) }}"
  vars:
    dest_db_name: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.db_name' ) }}"
    dest_db_filepath: "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}_dump.sql"
    dest_backup_dump_path:  "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}backup_dump.sql"
    dest_login_user: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.db_user') }}"
    dest_login_password: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.db_pass') }}"
    dest_login_host: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.host') }}"
    dest_login_port: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.port') }}"
    wp_prefix: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.table_prefix' ) }}"
    src_copy_path: "{{ lookup('vals', gv_sh_var + '.user_folder' ) }}/{{ lookup('vals', gv_sh_var + '.site_group' ) }}_dump.sql"
    dest_copy_path: "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}_dump.sql"
  roles:
    - set-if-unset
    - copy_file_push
    - restore_db



- hosts: "{{ dest_host_var | default(dest_host) }}"
  vars:
    src_site_url: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.site_url') }}"
    dest_site_url: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.site_url') }}"
    dest_site_folder: "{{ lookup('vals', gv_dh_var + '.site_folder') }}"
  roles:
    - set-if-unset
    - change_url

- hosts: "{{ src_host_var | default(src_host) }}"
  vars:
    src_deploy_user: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
    src_deploy_group: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
    #    dest_deploy_user: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
    #    dest_deploy_group: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
    src_copy_path: "{{ lookup('vals', gv_sh_var + '.site_folder' ) }}wp-content/uploads"
    dest_copy_path: "{{ lookup('vals', gv_dh_var + '.site_folder' ) }}wp-content/uploads"
  roles:
    - set-if-unset
    - copy_folder

- hosts: "{{ dest_host_var | default(dest_host) }}"
  vars:
    src_deploy_user: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
    src_deploy_group: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
    #    dest_deploy_user: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
    #    dest_deploy_group: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
    src_copy_path: "{{ lookup('vals', gv_sh_var + '.site_folder' ) }}wp-content/uploads"
    dest_copy_path: "{{ lookup('vals', gv_dh_var + '.site_folder' ) }}wp-content/uploads"
  roles:
    - set-if-unset
    - copy_folder_push

The roles referenced here are as follows:

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat set-if-unset/tasks/main.yml 
---
- name: set gv_sh if undefined 
  set_fact:
    gv_sh: ""
  when: gv_sh is not defined


# tasks file for set-if-unset
- name: setting facts if facts unset 
  set_fact:
    gv_sh_var: "{{ gv_sh_var | default(gv_sh) }}"
    gv_dh_var: "{{ gv_dh_var | default(gv_dh) }}"


joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat update-wp/tasks/main.yml 
---
# tasks file for update-wp---
  - name: update wordpress 
    shell: cd "{{ dest_site_folder }}" && wp core update

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat git-get-changes-to-repo/tasks/main.yml ---
# tasks file for git-get-changes-to-repo---

# see if there are changes to stage
- name: git status
  shell: "cd '{{ git_dir }}' && git status"
  register: git_status

# stage changes 
- name: git add .
  shell: "cd '{{ git_dir }}' && git add ."
  when: git_status.stdout is not match(".*\nnothing to commit.*")

# commit changes
- name: git commit
  shell: "cd '{{ git_dir }}' && git commit -m 'ansible commit'"
  when: git_status.stdout is not match(".*\nnothing to commit.*")
 
# push changes
- name: git push changes
  shell: "cd '{{ git_dir }}' && git push origin '{{ src_branch }}'"
  #  when: git_status.stdout is not match(".*\nnothing to commit.*")

# checkout proper branch 
- name: checkout branch to merge into 
  shell: "cd '{{ git_dir }}' && git checkout '{{ dest_branch }}'"
  #  when: git_status.stdout is not match(".*\nnothing to commit.*")

# merge branch
- name: merge branch
  shell: "cd '{{ git_dir }}' && git merge '{{ src_branch }}'"
  #  when: git_status.stdout is not match(".*\nnothing to commit.*")

# push changes
- name: git push changes
  shell: "cd '{{ git_dir }}' && git push origin '{{ dest_branch }}'"
  #  when: git_status.stdout is not match(".*\nnothing to commit.*")

# checkout back to original branch
- name: git checkout original branch
  shell: "cd '{{ git_dir }}' && git checkout '{{ src_branch }}'"
  #  when: git_status.stdout is not match(".*\nnothing to commit.*")


 #what it used to be before i noticed some problems
 #when: git_status.stdout is not match(".*\nnothing to commit, working tree clean")

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat git-repo-to-dest/tasks/main.yml 
---
# tasks file for git-repo-to-dest---

- name: git pull origin {{ dest_branch }} in {{ git_dir }} 
  shell: "cd '{{ git_dir }}' && git pull origin '{{ dest_branch }}'"

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat dump_db/tasks/main.yml 
---
# tasks file for dump_db
  - name: dump database "{{ src_db_name }}" to "{{ src_dump_path }}"
    mysql_db:
      state: dump
      name: "{{ src_db_name }}" 
      target: "{{ src_dump_path }}"
      login_user: "{{ src_login_user }}"
      login_password: "{{ src_login_password }}"
      login_host: "{{ src_login_host }}"
      login_port: "{{ src_login_port }}"

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat copy_file/tasks/main.yml 
---
# tasks file for copy_file

  - name: copy file to destination
    synchronize: 
      src: "{{ src_copy_path }}" 
      dest: "{{ dest_copy_path }}"
      mode: pull
    delegate_to: "{{ dest_host_var | default(dest_host) }}"
    when: ((gv_dh_var | default(gv_dh)) is not search("localhost")) and ((gv_sh_var | default(gv_sh)) is not search("localhost"))
    register: copied

  - name: copy file to localhost 
    synchronize: 
      mode: pull
      src: "{{ src_copy_path }}" 
      dest: "{{ dest_copy_path }}"
    when:  ((gv_dh_var | default(gv_dh)) is search("localhost")) or ((gv_sh_var | default(gv_sh)) is not search("localhost")) and copied.skipped is defined and copied.skipped == true


#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat copy_file_push/tasks/main.yml 
---
# tasks file for copy_file_push---

  - name: copy file from localhost 
    synchronize: 
      src: "{{ src_copy_path }}" 
      dest: "{{ dest_copy_path }}"
      mode: push 
    when: gv_sh_var | default(gv_sh) is search("localhost") and gv_sh_var | default(gv_dh) is search("localhost")


joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat restore_db/tasks/main.yml 
---

#3 does database exist?
  - name: check if DB exists
    shell: "mysql -u{{ dest_login_user }} -p'{{ dest_login_password }}' -h{{ dest_login_host }} -P{{ dest_login_port }} -e 'SHOW DATABASES;' | grep {{ dest_db_name }}"
    register: dbstatus
    failed_when: dbstatus.rc == 2

 #4 if database on other server exists then backup 
  - name: backup database to "{{ dest_backup_dump_path }}"
    mysql_db:
      state: dump
      name: "{{ dest_db_name }}"
      target: "{{ dest_backup_dump_path }}"          
      login_user: "{{ dest_login_user }}"
      login_password: "{{ dest_login_password }}"
      login_host: "{{ dest_login_host }}"
      login_port: "{{ dest_login_port }}"
    when: dbstatus.rc == 0
    register: backed_up

  - name: delete database "{{ dest_db_name }}"  
    mysql_db:
      name: "{{ dest_db_name }}"
      state: absent
      login_user: "{{ dest_login_user }}"
      login_password: "{{ dest_login_password }}"
      login_host: "{{ dest_login_host }}"
      login_port: "{{ dest_login_port }}"
    when: backed_up.changed and dbstatus.rc == 0 

#5 create database
  - name: Create database "{{ dest_db_name }}"
    mysql_db:
      name: "{{ dest_db_name }}"
      state: present 
      login_user: "{{ dest_login_user }}"
      login_password: "{{ dest_login_password }}"
      login_host: "{{ dest_login_host  }}"
      login_port: "{{ dest_login_port }}"
    when: backed_up.changed or dbstatus.rc == 1 

#6 import database
  - name: Import wordpress database
    mysql_db:
      state: import
      name: "{{ dest_db_name }}"
      login_user: "{{ dest_login_user }}"
      login_password: "{{ dest_login_password }}" 
      login_host: "{{ dest_login_host }}"
      login_port: "{{ dest_login_port }}"
      target: "{{ dest_db_filepath }}"
    when: backed_up.changed or dbstatus.rc == 1

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat change_url/tasks/main.yml 
---
# tasks file for change_url---
  - name: change url
    shell: cd "{{ dest_site_folder }}" && wp search-replace "{{ src_site_url }}" "{{ dest_site_url }}" --allow-root # --skip-columns=guid  --allow-root
    #this is does not deal with serialized data: "mysql -h{{ dest_login_host }} -u{{ dest_login_user }} -p{{ dest_login_password }} --password=root wordpress1 -e \"UPDATE {{ wp_prefix }}options SET option_value = REPLACE(option_value, '{{ src_site_url }}', '{{ dest_site_url }}') WHERE option_name = 'home' OR option_name = 'siteurl'; UPDATE {{ wp_prefix }}posts SET guid = REPLACE(guid, '{{ src_site_url }}', '{{ dest_site_url }}'); UPDATE {{ wp_prefix }}posts SET post_content = REPLACE(post_content, '{{ src_site_url }}', '{{ dest_site_url }}'); UPDATE {{ wp_prefix }}postmeta SET meta_value = REPLACE(meta_value, '{{ src_site_url }}', '{{ dest_site_url }}');\"" 

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat copy_folder/tasks/main.yml 
---

# tasks file for copy_file
  - name: copy folder to destination
    synchronize: 
      src: "{{ src_copy_path }}/" 
      dest: "{{ dest_copy_path }}"
      mode: pull
      rsync_opts: 
        - "-a"
    delegate_to: "{{ dest_host_var | default(dest_host) }}"
    when: (( gv_dh_var | default(gv_dh)) is not search("localhost")) and ((gv_sh_var | default(gv_sh)) is not search("localhost"))
    register: copied


  - name: copy folder to localhost 
    synchronize: 
      mode: pull
      src: "{{ src_copy_path }}/" 
      dest: "{{ dest_copy_path }}"
      rsync_opts: 
        - "-a"
    when: ((gv_dh_var | default(gv_dh)) is search("localhost")) or ((gv_sh_var | default(gv_sh)) is not search("localhost")) and copied.skipped is defined and copied.skipped == true

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat copy_folder_push/tasks/main.yml 
---


  - name: copy folder from localhost 
    synchronize: 
      src: "{{ src_copy_path }}/" 
      dest: "{{ dest_copy_path }}"
      mode: push 
      rsync_opts: 
        - "-a"
    when: gv_sh_var | default(gv_sh) is search("localhost") and gv_sh_var | default(gv_dh) is search("localhost")

There is also another task I haven’t mentiond that if placed in the following playbook would reverse the source and destination hosts:

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#


- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
  roles:
    - set-if-unset
  
# actually does the work/runs the playbooks:
- import_playbook: up-wp-both/up-wp-both_work.yml
- import_playbook: git-all/git-all_work.yml

- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
  roles:
    - reverse-facts

- import_playbook: cp-db/cp-db_work.yml

- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
  roles:
    - reset-reversed-facts

- import_playbook: ch-url/ch-url_work.yml
- import_playbook: up-up/up-up_work.yml

Bassically everything in the above playbook would run from source to destination except the cp-db_work.yml script. These roles are displayed below:

joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat reverse-facts/tasks/main.yml 
---

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#

# tasks file for reverse-facts
- name: this will reverse the facts for hosts
  set_fact:
    src_host_var: "{{ dest_host }}"
    dest_host_var: "{{ src_host }}"
    gv_sh_var: "{{ gv_dh }}"
    gv_dh_var: "{{ gv_sh }}"
joel@joel-Aspire-E5-575G:~/automation/greenwell-auto/roles$ cat reset-reversed-facts/tasks/main.yml 
---

#Î    This program is free software: you can redistribute it and/or modify    Î#
#Î    it under the terms of the GNU General Public License as published by    Î#
#Î    the Free Software Foundation, either version 3 of the License, or       Î#
#Î    (at your option) any later version.                                     Î#
#Î    Author: Joel Cambon                                                     Î#
#Î    Company: Greenwell                                                      Î#
#Î    (c) 2019                                                                Î#

# tasks file for reset-reversed-facts---
- name: this will reset the facts (that have been reversed) 
  set_fact:
    src_host_var: "{{ src_host }}"
    dest_host_var: "{{ dest_host }}"
    gv_sh_var: "{{ gv_sh }}"
    gv_dh_var: "{{ gv_dh }}"

To summarize what I created:

One ansible script is reusable for multiple hosts and sets of variables or site-credentials (as long as you have those in a group_vars/all.yaml file) I have a script that gets the credentials in JSON from all the wp-config.php files under a certain folder on the server. It also queries the databases as well so you can create group_vars/all.yaml automatically if you already have the sites you want to work with set up. It is parsed and converted to yaml by passing it through python classes that are converted to dictionaries. Then you can pass source and destination site variables to the “universal” ansible scripts and they will do whatever action they do on those source and destination sites. That way you don’t have to create different scripts to do the same task on different sites and you have some credentials and variables you can just modify or change that are separate from the script. So now you could just have one script for each action and any app that was interfacing with this would just have to pass the hostnames and site name to the script. All this allows for reusable scripts and less clutter in the ansible folder. I also made a way for you to test to see what the script would do. If you run a template action I have it uses the ansible script as a template and creates a file that shows you exactly what variables it would put in)

When running the scripts you don’t need to put in all the variables in all.yaml just the ones necessary for that task and this is true even if you are creating a new site. By running
an.py playbook.yaml
you will create a command by parsing the variables in all.yaml. I had it only parse variables that exist–that way you don’t have to create a dictionary of variables for each new site.


Leave a Reply

Your email address will not be published. Required fields are marked *