I’ve had a rough history with automation. I tried to automate some things between multiple wordpress sites in BASH and ended up writing this horrendous thing that I only show you because of how bad it is: https://github.com/felixtheratruns/bash-wp-porter Over a thousand lines of bash code that could have been written in python or some other more organizable language. I’ve switched to using Ansible more recently.
I really like a lot of things that Ansible is able to do but I noticed there were several use cases where Ansible almost would work but one small thing held it back. For instance, I would usually have to make a new playbook for every site times the number of operations I would do on it. If it was an operation that required two sites then this number would be multiplied by the number of sites the operation would be done with. This makes Ansible rather cluttered even when you try to separate things out nicely into different folders. More importantly, it was time-consuming to create these playbooks.
I thought I could fix this by using compound variables and referencing variables in group_vars/all.yaml changing which variables I referenced by using –extra-vars to change what groups of data in all.yaml the script is acting on. However, compound variables are currently not a feature in Ansible’s template language: Jinja2. So these are things I did to fix this problem:
Before we start I ran this bash script in every folder with a playbook in order for my playbooks to see the hosts file and the group_vars, lookup_plugins, and roles directories.
#!/usr/bin/env bash
ln -s ../hosts hosts
ln -s ../group_vars group_vars
ln -s ../lookup_plugins lookup_plugins
ln -s ../roles roles
1 Modify the Ansible “Lookup Vars” plugin. In my case I changed the LookupModule class to this: (apologies for the bad formatting and this definitely could have been coded in a less messy way) I’m still in the process of changing the documentation so some of it may be incorrect.
#vals plugin (based off of vars ansible lookup_plugin)
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
# (c) 2017 Ansible Project
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
DOCUMENTATION = """
lookup: vals
author: Ansible Core
short_description: Lookup value of variables in hostvars
description:
- Retrieves the value of an Ansible variable.
options:
_terms:
description: The variable names to look up.
required: True
default:
description:
- What to return if a variable is undefined.
- If no default is set, it will result in an error if any of the variables is undefined.
"""
EXAMPLES = """
- name: Show value of 'variablename'
debug: msg="{{ lookup('vals', 'variabl' + myvar)}}"
vars:
variablename: hello
myvar: ename
- name: Show default empty since i dont have 'variablnotename'
debug: msg="{{ lookup('vals', 'variabl' + myvar, default='')}}"
vars:
variablename: hello
myvar: notename
- name: Produce an error since i dont have 'variablnotename'
debug: msg="{{ lookup('vals', 'variabl' + myvar)}}"
ignore_errors: True
vars:
variablename: hello
myvar: notename
- name: find several related variables
debug: msg="{{ lookup('vals', 'ansible_play_hosts', 'ansible_play_batch', 'ansible_play_hosts_all') }}"
- name: alternate way to find some 'prefixed vars' in loop
debug: msg="{{ lookup('vals', 'ansible_play_' + item) }}"
loop:
- hosts
- batch
- hosts_all
"""
RETURN = """
_value:
description:
- value of the variables requested.
"""
from ansible.errors import AnsibleError, AnsibleUndefinedVariable
from ansible.module_utils.six import string_types
from ansible.plugins.lookup import LookupBase
class LookupModule(LookupBase):
def run(self, terms, variables=None, **kwargs):
if variables is not None:
self._templar.set_available_variables(variables)
myvals = getattr(self._templar, '_available_variables', {})
self.set_options(direct=kwargs)
default = self.get_option('default')
ret = []
for term in terms:
if not isinstance(term, string_types):
raise AnsibleError('Invalid setting identifier, "%s" is not a string, its a %s' % (term, type(term)))
try:
value = myvals['hostvars'][myvals['inventory_hostname']]
new_value = value
import re
pattern="(\[[^\[\\\\]*(?:\\\\.[^\]\\\\]*)*\])"
positions = [(m.start(0), m.end(0)) for m in re.finditer(pattern, term)]
fa = re.findall(pattern,term)
count = 0
arr = []
build_s = ""
for i, char in enumerate(term):
for pos in positions:
if i == pos[0] or i == pos[1]:
arr.append(build_s)
build_s=""
else:
continue
build_s += char
if build_s != "":
arr.append(build_s)
new_arr = []
for el in arr:
if el.startswith('[') and el.endswith(']'):
new_arr.append(el.lstrip('[').rstrip(']'))
else:
if "." in el:
for e in el.split("."):
new_arr.append(e)
else:
new_arr.append(el)
#if you have a problem with it generating blank keys uncomment this:
# new_arr = list(filter(None, new_arr))
for i in new_arr:
new_value = new_value[i]
ret.append(self._templar.template(new_value, fail_on_undefined=True))
except AnsibleUndefinedVariable:
if default is not None:
ret.append(default)
else:
raise
return ret2 To save time in creating the new variables and make variables for the reusable scripts automatically I wrote a script that gets the data automatically from the server. In this case I am dealing with wordpress:
#!/usr/bin/env bash
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
re='^[0-9]+$'
if [ -n "$1" ] && [[ $1 =~ $re ]]; then
count_loop=$1
else
count_loop=0
fi
search_folder=$2
num_lines=`find "$search_folder" -not -path '*/\.*' -name wp-config.php -print 2>&1 | grep -v "Permission denied" | wc -l`
loop_length=$((num_lines-1))
printf "\n"
counter=0
find "$search_folder" -not -path '*/\.*' -name wp-config.php -print 2>&1 | grep -v "Permission denied" | while read line
do
full_path=
rel_path=
table_prefix=
WPDBNAME=
WPDBUSER=
WPDBPASS=
wp_options=
start_mysql=
pass=
end_mysql=
mysql_args=
query_t=
query=
tmp=
siteurl_nq=
siteurl=
folders=
repo_url_temp=
repo_folder=
repo_url=
full_folder_path=
repo_branch=
repo_branch_temp=
site_title_nq=
site_title=
tmp=
remote_host=
printf "\n"
printf "\"$count_loop\": {"
printf "\n"
full_path="$line"
table_prefix=`cat $line | grep "^[^#;]" | grep "table_prefix" | cut -d \' -f 2`
WPDBNAME=`cat $line | grep "^[^#;]" | grep DB_NAME | cut -d \' -f 4`
WPDBUSER=`cat $line | grep "^[^#;]" | grep DB_USER | cut -d \' -f 4`
WPDBPASS=`cat $line | grep "^[^#;]" | grep DB_PASSWORD | cut -d \' -f 4`
WPDBHOST=`cat $line | grep "^[^#;]" | grep DB_HOST | cut -d \' -f 4`
WPDBNAME=`echo "${WPDBNAME[@]}" | head -1`
WPDBUSER=`echo "${WPDBUSER[@]}" | head -1`
WPDBPASS=`echo "${WPDBPASS[@]}" | head -1`
WPDBHOST=`echo "${WPDBHOST[@]}" | head -1`
table_prefix=`echo "${table_prefix[@]}" | head -1`
rel_path="$line"
printf "%s%s\n" '"rel_path":' "\"$rel_path\","
printf "%s%s\n" '"user_folder":' "\"`echo ~`\","
printf "%s%s\n" '"full_path":' "\"$full_path\","
printf "%s%s\n" '"table_prefix":' "\"$table_prefix\","
printf "%s%s\n" '"WPDBNAME":' "\"$WPDBNAME\","
printf "%s%s\n" '"WPDBUSER":' "\"$WPDBUSER\","
printf "%s%s\n" '"WPDBPASS":' "\"$WPDBPASS\","
wp_options=$table_prefix"options"
wp_users=$table_prefix"users"
wp_usermeta=$table_prefix"usermeta"
full_host="$WPDBHOST"
if [[ $full_host == *":"* ]]; then #[fix]
IFS=':' read -ra full_host_ar <<< "$full_host"
host="${full_host_ar[0]}"
port="${full_host_ar[1]}"
else
host="${full_host}"
port=3306
fi
start_mysql="-u$WPDBUSER -p"
pass="'$WPDBPASS'"
end_mysql=" -h$host -P$port $WPDBNAME"
mysql_args=$start_mysql"$pass"$end_mysql
query_t=$(mysql -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT option_value FROM $wp_options WHERE option_name = 'siteurl';" 2>/dev/null)
query=${query_t//[^a-zA-Z0-9_:\.\/\-\~] /}
tmp=($query)
tmp_nq=${tmp[0]}
siteurl="$tmp_nq"
printf "%s%s\n" '"site_url":' "\"$siteurl\","
query_t=
query_t=$(mysql -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT option_value FROM $wp_options WHERE option_name = 'blogname';" 2>/dev/null)
query=${query_t//[^a-zA-Z0-9_:\.\/\-\~ ] /}
# tmp=($query)
# tmp_nq=${tmp[0]}
site_title="$query"
printf "%s%s\n" '"site_title":' "\"$site_title\","
query_t=
query_t=$(mysql -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT DISTINCT user_login FROM $wp_users INNER JOIN $wp_usermeta ON $wp_users.ID = $wp_usermeta.user_id WHERE $wp_usermeta.meta_value LIKE '%administrator%' ORDER BY $wp_users.ID;" 2>/dev/null)
printf "%s" '"admin_users":'
printf "%s" '['
length=$(echo "$query_t" | wc -l)
count=0
while IFS="" read -r p || [ -n "$p" ]
do
if (( $count < $length - 1 )); then
printf '"%s",' "$p"
else
printf '"%s"' "$p"
fi
((count++))
done <<< "$query_t"
printf "%s\n" '],'
query_t=
query_t=$(mysql -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT DISTINCT user_login, user_pass FROM $wp_users INNER JOIN $wp_usermeta ON $wp_users.ID = $wp_usermeta.user_id WHERE $wp_usermeta.meta_value LIKE '%administrator%'; ORDER BY $wp_users.ID" 2>/dev/null)
printf "%s" '"hashed_passwords":'
printf "%s" '{'
length=$(echo "$query_t" | wc -l)
count=0
while IFS="" read -r p || [ -n "$p" ]
do
p_arr=
IFS=$'\t' read -r -a p_arr <<< "$p"
if (( $count < $length - 1 )); then
printf '"%s": "%s",' "${p_arr[0]}" "${p_arr[1]}"
else
printf '"%s": "%s"' "${p_arr[0]}" "${p_arr[1]}"
fi
((count++))
done <<< "$query_t"
printf "%s\n" '},'
query_t=
query_t=$(mysql -s -N -u$WPDBUSER -p$WPDBPASS -h$host -P$port $WPDBNAME -e "SELECT DISTINCT user_login, user_email FROM $wp_users INNER JOIN $wp_usermeta ON $wp_users.ID = $wp_usermeta.user_id WHERE $wp_usermeta.meta_value LIKE '%administrator%' ORDER BY $wp_users.ID;" 2>/dev/null)
printf "%s" '"admin_emails":'
printf "%s" '{'
length=$(echo "$query_t" | wc -l)
count=0
while IFS="" read -r p || [ -n "$p" ]
do
p_arr=
IFS=$'\t' read -r -a p_arr <<< "$p"
if (( $count < $length - 1 )); then
printf '"%s": "%s",' "${p_arr[0]}" "${p_arr[1]}"
else
printf '"%s": "%s"' "${p_arr[0]}" "${p_arr[1]}"
fi
((count++))
done <<< "$query_t"
printf "%s\n" '},'
full_folder_path=$(dirname $full_path)
folders=${full_folder_path##/*/}
repo_url_temp=$(cd "$full_folder_path" && git config --get remote.origin.url)
if [ -z $repo_url_temp ]; then
:
else
repo_branch_temp=$(cd "$full_folder_path" && git rev-parse --abbrev-ref HEAD 2>/dev/null)
repo_folder="$full_folder_path"
repo_url=$repo_url_temp
repo_branch=$repo_branch_temp
fi
printf "%s%s\n" '"host":' "\"$host\","
printf "%s%s\n" '"port":' "\"$port\","
printf "%s%s\n" '"repo_folder":' "\"$repo_folder\","
printf "%s%s\n" '"repo_url":' "\"$repo_url\","
printf "%s%s\n" '"repo_branch":' "\"$repo_branch\","
remote_host=`hostname`
printf "%s%s\n" '"remote_host":' "\"$remote_host\""
if (( $counter < $num_lines - 1 )); then
printf "},"
else
printf "}"
fi
printf "\n"
count_loop=$((count_loop+1))
counter=$((counter+1))
done
3 That data needs to be parsed into yaml and there’s some issues of the output of the BASH script: it’s not proper json because it isn’t enclosed in brackets like this {data} So we have to run the script on the server like this to put everything together:
#!/usr/bin/env python
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
import subprocess
from get_keys import OutputProc
import json
import os
import sys
start_num=0
output = "\n{"
op = OutputProc()
search_folder = "/var/www"
output += op.run_cmd("localhost", "user1", 22, start_num, search_folder)
#make first command's output into proper json:
output_com = output + "\n}"
#get the number of elements it has:
print(output_com)
start_num=len(json.loads(output_com))
#make comma in json text to add next command's output to it:
output += ",\n"
#add the command:
search_folder = "/home/user2"
output += op.run_cmd("greenwell.tech", "user2", 5000, start_num, search_folder)
#close the json
output += "}"
#load and write json to file
json_out = json.loads(output)
full_path = os.path.realpath(sys.argv[0])
fname = "/data.json"
f = open(os.path.dirname(full_path) + fname, 'w');
f.write(json.dumps(json_out, indent=4));
f.close()
get_keys comes from this file here: (credit to https://www.bogotobogo.com/python/python_ssh_remote_run.php for the code I started from)
#!/usr/bin/env python
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
import os, subprocess, sys, argparse, json
class OutputProc():
def run_cmd(self, host, user, port, start_num, search_folder):
return self.remote_run(host, user, port, start_num, search_folder)
def check_arg(self, args=None):
parser = argparse.ArgumentParser(description='script to get website data from remote server')
parser.add_argument('-H', '--host',
help='host ip')
parser.add_argument('-u', '--user',
help='user name')
parser.add_argument('-p', '--port',
help='port number')
return parser.parse_args(args)
def remote_run(self, host, user, port, start_num, search_folder):
print("host: " + str(host))
print("user: " + str(user))
print("port: " + str(port))
cmd_list = []
with open('commands.txt', 'r') as f:
for c in f:
cmd_list.append(c.replace('\n','').replace('\r','').rstrip())
exe = {'py': 'python3', 'sh': 'sh', 'pl': 'perl', 'bash': 'bash'}
ssh = ''.join(['ssh ', ' -p', str(port), ' ', user, '@', host, ' '])
print("ssh: " + ssh)
for cmd in cmd_list:
exe_type = exe[cmd.split('.')[1]]
if host == "localhost":
cmd = exe_type + " " + cmd + " " + str(start_num) + " " + str(search_folder) + ' 2>&1'
else:
cmd = ssh + '"' + exe_type + ' -s' + '"' + ' < ' + cmd + " " + str(start_num) + " " + str(search_folder) + ' 2>&1'
print("cmd: " + cmd)
result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
return result.stdout.read().strip().decode('utf-8')
commands.txt is where the command for the script resides: (credit to https://www.bogotobogo.com/python/python_ssh_remote_run.php for the code I started from)
get-keys.bashSo when we run the scripts that we want to be able to pass –extra-vars to point to different values we get from the server. This will be explained later. Hopefully, that will give you an idea of how I am getting credentials in the form of JSON data from the server. I guess ill include parsing the JSON here because it is all one program. Here is the parsing function that you run to parse the JSON to get it into a format that’s more human readable in YAML and to create some additional variables that can be derived by parsing it.
#!/usr/bin/env python
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
import sys
import yaml
import os
import json
from data import Node
from data import NodeWrap
from data import Company
from storage import MySingleton
from formatter import Formatter
#print('sys.argv[0] =', sys.argv[0])
#pathname = os.path.dirname(sys.argv[0])
#print('path =', pathname)
#print('full path =', os.path.abspath(pathname))
def wat_path(path):
if '/staging/' in path:
return "staging"
elif '/public_html/' in path:
return "production"
return "unknown"
def print_nested_dict(d):
for k, v in d.items():
if isinstance(v, dict):
print(">" + k)
print_nested_dict(v)
print("")
else:
print("{0} : {1}".format(k, v))
def parse_data(singleton):
fname = "/data.json"
full_path = os.path.realpath(sys.argv[0])
with open( os.path.dirname(full_path) + fname) as f:
content = f.readlines()
# you may also want to remove whitespace characters like `\n` at the end of each line
content = [x.strip() for x in content]
with open( os.path.dirname(full_path) + '/data.json') as data_file:
data = json.load(data_file)
for idx, val in enumerate(data):
node = NodeWrap(Node,
data[val].get("rel_path",""),
data[val].get("user_folder",""),
data[val].get("full_path",""),
data[val].get("table_prefix",""),
data[val].get("WPDBNAME",""),
data[val].get("WPDBUSER",""),
data[val].get("WPDBPASS",""),
data[val].get("host",""),
data[val].get("port",""),
data[val].get("site_url",""),
data[val].get("site_title",""),
data[val].get("admin_users",""),
data[val].get("admin_emails",""),
data[val].get("hashed_passwords",""),
data[val].get("repo_folder",""),
data[val].get("repo_url",""),
data[val].get("repo_branch",""),
data[val].get("remote_host",""))
tag = node.make_tag()
sitemap_host_name = node.make_sitemap_host_name()
company_specific = Company(
tag,
sitemap_host_name,
data[val].get("WPDBNAME",""),
data[val].get("WPDBUSER",""),
"",
"",
data[val].get("WPDBPASS",""),
"",
"").__dict__
node.set_company(company_specific)
site_group = node.get_site_group()
site_type = node.get_site_type()
singleton.add_val(site_group, site_type, node.__dict__)
#might use this to rewrite the "put variables inside global with a hostname variable" below:
#def extend(a,b):
# """Create a new dictionary with a's properties extended by b,
# without overwriting.
#
# >>> extend({'a':1,'b':2},{'b':3,'c':4})
# {'a': 1, 'c': 4, 'b': 2}
# """
# return dict(b,**a)
def write_yaml(singleton):
#put variables inside global with a hostname variable
# set_of_nodes = { "global": {} }
# temp_dict = { "hostname": "{{ inventory_hostname }}" }
# temp_dict.update(singleton.get_nodes())
# set_of_nodes["global"] = temp_dict
# singleton.set_nodes(set_of_nodes)
pretty = Formatter()
print(pretty(singleton.get_nodes()))
json_dumps = json.dumps(singleton.get_nodes())
yaml_nodes = yaml.safe_load(json_dumps)
full_path = os.path.realpath(sys.argv[0])
f = open(os.path.dirname(full_path) + '/group_vars/all.yaml', 'w+');
f.write(yaml.dump(yaml_nodes, default_flow_style=False));
f.close()
if __name__ == '__main__':
singleton = MySingleton()
singleton.set_nodes({})
parse_data(singleton)
write_yaml(singleton)
It imports from data.py, storage.py, and formatter.py as you can see above. data.py is something I wrote, storage.py is just slightly modified from something I found online and formatter.py is entirely not my work.
#!/usr/bin/env python
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
import os
import random
import string
class NodeWrap(dict):
def __init__(self, wrapped_Node, rel_path, user_folder, full_path, table_prefix, db_name, db_user, db_pass, host, port, site_url, site_title, admin_users, admin_emails, hashed_passwords, repo_folder, repo_url, repo_branch, remote_host):
home_folder = self.remove_postfix(full_path, "wp-config.php")
site_folder = self.remove_postfix(full_path, "wp-config.php")
site_type = self.what_type_from_url(site_url)
admin_user = admin_users[0]
admin_email = admin_emails[admin_user]
hashed_password = hashed_passwords[admin_user]
self.set_user_folder(user_folder)
self.set_user(os.path.basename(user_folder))
self.wrapped_Node = wrapped_Node(rel_path, full_path, table_prefix, db_name, db_user, db_pass, host, port, site_url, site_title, admin_users, admin_user, admin_emails, admin_email, hashed_passwords, hashed_password, repo_folder, repo_url, repo_branch).__dict__
self.set_home_folder(home_folder)
self.set_site_folder(site_folder)
self.set_site_type(site_type)
site_group_tmp = repo_url
site_group_tmp = self.index_split_or_original(1,"/",site_group_tmp)
site_group_tmp = self.index_split_or_original(0,".",site_group_tmp)
self.set_site_group(site_group_tmp)
self.set_remote_host(remote_host)
def get_site_type(self):
return self.site_type
def set_user_folder(self, user_folder):
self.user_folder = user_folder
def set_home_folder(self, home_folder):
self.home_folder = home_folder
def set_site_folder(self, site_folder):
self.site_folder = site_folder
def set_site_type(self, site_type):
self.site_type = site_type
def set_site_group(self, site_group):
self.site_group = site_group
def get_site_group(self):
return self.site_group
def remove_prefix(self, text, prefix):
if text.startswith(prefix):
return text[len(prefix):]
return text # or whatever
def remove_postfix(self, text, postfix):
if text.endswith(postfix):
return text[:-len(postfix)]
return text # or whatever
def what_type_from_url(self, url):
if 'localhost' in url:
return 'localhost'
elif 'greenwell.tech' in url:
return "staging"
return "production"
def set_remote_host(self, remote_host):
self.remote_host = remote_host
def index_split_or_original(self, idx, split_char, string):
if ( split_char in string and len(string.split(split_char)) > 1):
return string.split(split_char)[idx]
else:
return ""
def __getattr__(self,attr):
orig_attr = self.wrapped_Node.__getattribute__(attr)
if callable(orig_attr):
def hooked(*args, **kwargs):
result = orig_attr(*args, **kwargs)
#prevent wrapped_class from becomming unwrapped
if result == self.wrapped_Node:
return self
return result
return hooked
else:
return orig_attr
#making functions
def make_tag(self):
dirname = os.path.dirname(self.home_folder)
basename = os.path.basename(dirname)
if "." in basename:
return basename.split(".")[0]
else:
return basename
def make_sitemap_host_name(self):
remote_h = ""
if self.remote_host == "":
pass
elif "." in self.remote_host:
remote_h = self.remote_host.split(".")[0]
else:
return self.remote_host
return remote_h
def set_company(self, company_specific):
self.company_specific = company_specific
def set_user(self, user):
self.user = user
class Node(dict):
def __init__(self, rel_path, full_path, table_prefix, db_name, db_user, db_pass, host, port, site_url, site_title, admin_users, admin_user, admin_emails, admin_email, hashed_passwords, hashed_password, repo_folder, repo_url, repo_branch):
self.set_rel_path(rel_path)
self.set_full_path(full_path)
self.set_table_prefix(table_prefix)
self.set_db_name(db_name)
self.set_db_user(db_user)
self.set_db_pass(db_pass)
self.set_host(host)
self.set_port(port)
self.set_site_url(site_url)
self.set_site_title(site_title)
self.set_admin_users(admin_users)
self.set_admin_user(admin_user)
self.set_admin_emails(admin_emails)
self.set_admin_email(admin_email)
self.set_hashed_passwords(hashed_passwords)
self.set_hashed_password(hashed_password)
self.set_repo_folder(repo_folder)
self.set_repo_url(repo_url)
self.set_repo_branch(repo_branch)
def set_name(self, name):
self.name = name
def set_rel_path(self, rel_path):
self.rel_path = rel_path
def set_full_path(self, full_path):
self.full_path = full_path
def set_table_prefix(self, table_prefix):
self.table_prefix = table_prefix
def set_db_name(self, db_name):
self.db_name = db_name
def set_db_user(self, db_user):
self.db_user = db_user
def set_db_pass(self, db_pass):
self.db_pass = db_pass
def set_host(self, host):
self.host = host
def set_port(self, port):
self.port = port
def set_site_url(self, site_url):
self.site_url = site_url
def set_site_title(self, site_title):
self.site_title = site_title
def set_admin_users(self, admin_users):
self.admin_users = admin_users
def set_admin_user(self, admin_user):
self.admin_user = admin_user
def set_admin_emails(self, admin_emails):
self.admin_emails = admin_emails
def set_admin_email(self, admin_email):
self.admin_email = admin_email
def set_hashed_passwords(self, hashed_passwords):
self.hashed_passwords = hashed_passwords
def set_hashed_password(self, hashed_password):
self.hashed_password = hashed_password
def set_repo_folder(self, repo_folder):
self.repo_folder = repo_folder
def set_repo_url(self, repo_url):
self.repo_url = repo_url
def set_repo_branch(self, repo_branch):
self.repo_branch = repo_branch
class Company(dict):
def __init__(self, tag, sitemap_host_name, dbname, dbuser, sftp_u, sftp_p, dbpass, client_pass, sftp_pass):
self.set_tag(tag)
if dbname == "":
self.set_dbname( "prefix_" + tag )
else:
self.set_dbname( dbname )
self.set_sitemap_host_name(sitemap_host_name)
if dbuser == "":
self.set_dbuser( "prefix" + tag )
else:
self.set_dbuser( dbuser )
if sftp_u == "":
self.set_sftp_u( "prefix" + tag )
else:
self.set_sftp_u( sftp_u )
#passwords
password = self.randomString(15)
if sftp_p == "":
self.set_sftp_p( password )
else:
self.set_sftp_p( sftp_p )
password = self.randomString(15)
if sftp_p == "":
self.set_sftp_p( password )
else:
self.set_sftp_p( sftp_p )
password = self.randomString(15)
if dbpass == "":
self.set_dbpass( password )
else:
self.set_dbpass( dbpass )
password = self.randomString(15)
if client_pass == "":
self.set_client_pass( password )
else:
self.set_client_pass( client_pass )
password = self.randomString(15)
if sftp_pass == "":
self.set_sftp_pass ( password )
else:
self.set_sftp_pass ( sftp_pass )
def set_tag(self, tag):
self.tag = tag
def set_sitemap_host_name(self, sitemap_host_name):
self.sitemap_host_name = sitemap_host_name
def set_dbname(self, dbname):
self.dbname = dbname
def set_sftp_u(self, sftp_u):
self.sftp_u = sftp_u
def set_sftp_p(self, sftp_p):
self.sftp_p = sftp_p
def set_dbpass(self, dbpass):
self.dbpass = dbpass
def set_dbuser(self, dbuser):
self.dbuser = dbuser
def set_client_pass(self, client_pass):
self.client_pass = client_pass
def set_sftp_pass(self, sftp_pass):
self.sftp_pass = sftp_pass
def randomString(self, stringLength):
"""Generate a random string with the combination of lowercase and uppercase letters """
letters = string.ascii_letters
return ''.join(random.choice(letters) for i in range(stringLength))
Here is storage.py. Thanks to Michael for making available the original code. https://michaelgoerz.net/notes/singleton-objects-in-python.html I’m not sure if I’ve modified it enough to need to put a license on it but I’ve decided to do so just in case:
#!/usr/bin/env python
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
class Singleton(type):
"""Metaclass for singletons. Any instantiation of a Singleton class yields
the exact same object, e.g.:
>>> class MyClass(metaclass=Singleton):
pass
>>> a = MyClass()
>>> b = MyClass()
>>> a is b
True
"""
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args,
**kwargs)
return cls._instances[cls]
@classmethod
def __instancecheck__(mcs, instance):
if instance.__class__ is mcs:
return True
else:
return isinstance(instance.__class__, mcs)
def singleton_object(cls):
"""Class decorator that transforms (and replaces) a class definition (which
must have a Singleton metaclass) with the actual singleton object. Ensures
that the resulting object can still be "instantiated" (i.e., called),
returning the same object. Also ensures the object can be pickled, is
hashable, and has the correct string representation (the name of the
singleton)
"""
assert isinstance(cls, Singleton), \
cls.__name__ + " must use Singleton metaclass"
def self_instantiate(self):
return self
cls.__call__ = self_instantiate
cls.__hash__ = lambda self: hash(cls)
cls.__repr__ = lambda self: cls.__name__
cls.__reduce__ = lambda self: cls.__name__
obj = cls()
obj.__name__ = cls.__name__
return obj
@singleton_object
class MySingleton(metaclass=Singleton):
def set_nodes(self, dictionary):
self.nodes = dictionary
def get_nodes(self):
return self.nodes
def get_node(self, key):
return self.nodes[key]
def set_nodes(self, temp_dict):
self.nodes = temp_dict
def set_key(self, key, temp_dict):
self.nodes[key][temp_dict]
def add_val(self, site_group, site_type, node):
if self.nodes.get(site_group,"") == "":
temp_dict = { site_type: node }
self.nodes[site_group] = temp_dict
elif self.nodes[site_group].get(site_type,"") == "":
self.nodes[site_group][site_type] = node
else:
count = 1
new_site_type = site_type + str(count)
while self.nodes[site_group].get(new_site_type ,"") != "":
count += 1
new_site_type = site_type + str(count)
self.nodes[site_group][new_site_type] = node
def get_val_from_str(self, string):
arr = string.split('.')
nodes = self.nodes
for key, val in enumerate(arr):
nodes = nodes[val]
return nodes
def return_main_keys(self):
key_list = []
for key in self.nodes:
print(key)
for key2 in self.nodes[key]:
input_key = key + "." + key2
key_list.append(input_key)
print(str(len(key_list) -1) + ">" + input_key)
return key_list
class SingletonType(metaclass=Singleton):
pass
Finally I will post formatter.py here for your convenience. Since it’s not any of my work it will be the only one not licensed under the gpl:
class Formatter(object):
def __init__(self):
self.types = {}
self.htchar = '\t'
self.lfchar = '\n'
self.indent = 0
self.set_formater(object, self.__class__.format_object)
self.set_formater(dict, self.__class__.format_dict)
self.set_formater(list, self.__class__.format_list)
self.set_formater(tuple, self.__class__.format_tuple)
def set_formater(self, obj, callback):
self.types[obj] = callback
def __call__(self, value, **args):
for key in args:
setattr(self, key, args[key])
formater = self.types[type(value) if type(value) in self.types else object]
return formater(self, value, self.indent)
def format_object(self, value, indent):
return repr(value)
def format_dict(self, value, indent):
items = [
self.lfchar + self.htchar * (indent + 1) + repr(key) + ': ' +
(self.types[type(value[key]) if type(value[key]) in self.types else object])(self, value[key], indent + 1)
for key in value
]
return '{%s}' % (','.join(items) + self.lfchar + self.htchar * indent)
def format_list(self, value, indent):
items = [
self.lfchar + self.htchar * (indent + 1) + (self.types[type(item) if type(item) in self.types else object])(self, item, indent + 1)
for item in value
]
return '[%s]' % (','.join(items) + self.lfchar + self.htchar * indent)
def format_tuple(self, value, indent):
items = [
self.lfchar + self.htchar * (indent + 1) + (self.types[type(item) if type(item) in self.types else object])(self, item, indent + 1)
for item in value
]
return '(%s)' % (','.join(items) + self.lfchar + self.htchar * indent)
#To initialize it :
#
#pretty = Formatter()
#It can support the addition of formatters for defined types, you simply need to make a function for that like this one and bind it to the type you want with set_formater :
#
#from collections import OrderedDict
#
#def format_ordereddict(self, value, indent):
# items = [
# self.lfchar + self.htchar * (indent + 1) +
# "(" + repr(key) + ', ' + (self.types[
# type(value[key]) if type(value[key]) in self.types else object
# ])(self, value[key], indent + 1) + ")"
# for key in value
# ]
# return 'OrderedDict([%s])' % (','.join(items) +
# self.lfchar + self.htchar * indent)
#pretty.set_formater(OrderedDict, format_ordereddict)
#For historical reasons, I keep the previous pretty printer which was a function instead of a class, but they both can be used the same way, the class version simply permit much more :
#
#def pretty(value, htchar='\t', lfchar='\n', indent=0):
# nlch = lfchar + htchar * (indent + 1)
# if type(value) is dict:
# items = [
# nlch + repr(key) + ': ' + pretty(value[key], htchar, lfchar, indent + 1)
# for key in value
# ]
# return '{%s}' % (','.join(items) + lfchar + htchar * indent)
# elif type(value) is list:
# items = [
# nlch + pretty(item, htchar, lfchar, indent + 1)
# for item in value
# ]
# return '[%s]' % (','.join(items) + lfchar + htchar * indent)
# elif type(value) is tuple:
# items = [
# nlch + pretty(item, htchar, lfchar, indent + 1)
# for item in value
# ]
# return '(%s)' % (','.join(items) + lfchar + htchar * indent)
# else:
# return repr(value)
#To use it :
#
#>>> a = {'list':['a','b',1,2],'dict':{'a':1,2:'b'},'tuple':('a','b',1,2),'function':pretty,'unicode':u'\xa7',("tuple","key"):"valid"}
#>>> a
#{'function': <function pretty at 0x7fdf555809b0>, 'tuple': ('a', 'b', 1, 2), 'list': ['a', 'b', 1, 2], 'dict': {'a': 1, 2: 'b'}, 'unicode': u'\xa7', ('tuple', 'key'): 'valid'}
#>>> print(pretty(a))
#{
# 'function': <function pretty at 0x7fdf555809b0>,
# 'tuple': (
# 'a',
# 'b',
# 1,
# 2
# ),
# 'list': [
# 'a',
# 'b',
# 1,
# 2
# ],
# 'dict': {
# 'a': 1,
# 2: 'b'
# },
# 'unicode': u'\xa7',
# ('tuple', 'key'): 'valid'
#}
#Compared to other versions :
#
#This solution looks directly for object type, so you can pretty print almost everything, not only list or dict.
#Doesn't have any dependancy.
#Everything is put inside a string, so you can do whatever you want with it.
#The class and the function has been tested and works with Python 2.7 and 3.4.
#You can have all type of objects inside, this is their representations and not theirs contents that being put in the result (so string have quotes, Unicode string are fully represented ...).
#With the class version, you can add formatting for every object type you want or change them for already defined ones.
#key can be of any valid type.
#Indent and Newline character can be changed for everything we'd like.
#Dict, List and Tuples are pretty printed.
This is the helper function that allows you to be extra lazy creating –extra-vars. This just parses the yaml and lets you select what sites to have as a source and destination. The description of how it can be run is this: an.py ansible-playbook.yml (-h(source_host_number):[destination_host_number]) Here are some examples:
an.py ansible-playbook.yml #asks for source and destination host
an.py ansible-playbook.yml -h0:1 #automatically runs the source and destination host number
an.py ansible-playbook.yml -h:3 #automatically just runs on the destination
#!/usr/bin/env python
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
import sys
import yaml
import os
import json
def parse_yaml(singleton):
fname = "/group_vars/all.yaml"
full_path = os.path.realpath(sys.argv[0])
def return_main_keys(data):
key_list = []
for key in data:
print(key)
for key2 in data[key]:
input_key = key + "." + key2
key_list.append(input_key)
print(str(len(key_list) -1) + ">" + input_key)
return key_list
fname = "/group_vars/all.yaml"
full_path = os.path.realpath(sys.argv[0])
with open( os.path.dirname(full_path) + '/group_vars/all.yaml') as data_file:
data = yaml.load(data_file)
key_list = return_main_keys(data)
def remove_prefix(text, prefix):
if text.startswith(prefix):
return text[len(prefix):]
return text # or whatever
gv_sh=""
gv_dh=""
#start building bash command:
bcmd = ""
both_hosts_set = False
for arg in sys.argv[1:]:
if arg.startswith("-h"):
arg = remove_prefix(arg, "-h")
arg_arr = arg.split(":")
if arg_arr[0] == "":
gv_dh = arg_arr[1]
both_hosts_set = False
break
else:
gv_sh = arg_arr[0]
gv_dh = arg_arr[1]
both_hosts_set = True
break
#^end if
else:
both_hosts_set = True
for arg in sys.argv[1:]:
if arg.startswith("-h"):
continue
else:
bcmd += " " + arg
if len(sys.argv[1:]) == 0:
both_hosts_set = True
if gv_sh == "" and both_hosts_set:
gv_sh = input("what do you want your source host variables to be?")
if gv_dh == "":
gv_dh = input("what do you want your destination host variables to be?")
if gv_sh != "":
gv_sh = int(gv_sh)
gv_dh = int(gv_dh)
def get_val_from_str(string, data):
arr = string.split('.')
nodes = data
for key, val in enumerate(arr):
nodes = nodes[val]
return nodes
bcmd += " "
bcmd += "--extra-vars="
bcmd += '"'
if gv_sh != "":
bcmd += "gv_sh=" + key_list[gv_sh] + " "
bcmd += "gv_dh=" + key_list[gv_dh] + " "
if gv_sh != "":
src_host_ref = key_list[gv_sh] + ".remote_host"
dest_host_ref = key_list[gv_dh] + ".remote_host"
if gv_sh != "":
src_host = get_val_from_str(src_host_ref, data)
dest_host = get_val_from_str(dest_host_ref, data)
if gv_sh != "":
bcmd += "src_host=" + src_host + " "
bcmd += "dest_host=" + dest_host
bcmd += '"'
bcmd = "ansible-playbook -i hosts" + bcmd
print("running command:")
print(bcmd)
os.system(bcmd)
The following code runs the previous an.py on multiple destination hosts with the same script. There is probably a way to do something similar with ansible I just haven’t though of it yet. It is run like so: “reu.py ansible-playbooky.yml“
import sys
import yaml
import os
import json
def return_main_keys(data):
key_list = []
for key in data:
print(key)
for key2 in data[key]:
input_key = key + "." + key2
key_list.append(input_key)
print(str(len(key_list) -1) + ">" + input_key)
return key_list
fname = "/group_vars/all.yaml"
full_path = os.path.realpath(sys.argv[0])
with open( os.path.dirname(full_path) + '/group_vars/all.yaml') as data_file:
data = yaml.load(data_file)
key_list = return_main_keys(data)
start_num = input("where do you want to start?")
stop_num = input("where do you want to stop?")
start_num=int(start_num)
stop_num=int(stop_num)
for num_to_be_worked_on, key in enumerate(key_list[start_num:stop_num+1], start=start_num):
bcmd="an.py " + "-h:" + str(num_to_be_worked_on) + " " + sys.argv[1]
print(bcmd)
os.system(bcmd)
# example conditional you could add
# if "localhost" not in key:
# bcmd="an.py " + "-h:" + str(num_to_be_worked_on) + " " + sys.argv[1]
# print(bcmd)
# os.system(bcmd)
The actual Ansible scripts look like this:
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
roles:
- set-if-unset
# creates working examples
- import_playbook: up-wp-both/up-wp-both_create-work-ex.yml
- import_playbook: git-all/git-all_create-work-ex.yml
- import_playbook: cp-db/cp-db_create-work-ex.yml
- import_playbook: ch-url/ch-url_create-work-ex.yml
- import_playbook: up-up/up-up_create-work-ex.yml
#assembles the working examples
- hosts: localhost
tasks:
- name: assemble up-wp
shell: cat up-wp-both/up-wp-both_work-ex.yml > ./output-ex.yml
- hosts: localhost
tasks:
- name: assemble git-all
shell: cat git-all/git-all_work-ex.yml >> ./output-ex.yml
- hosts: localhost
tasks:
- name: assemble cp-db
shell: cat cp-db/cp-db_work-ex.yml >> ./output-ex.yml
- hosts: localhost
tasks:
- name: assemble ch-url
shell: cat ch-url/ch-url_work-ex.yml >> ./output-ex.yml
- hosts: localhost
tasks:
- name: assemble up-up
shell: cat up-up/up-up_work-ex.yml >> ./output-ex.yml
# actually does the work/runs the playbooks:
- import_playbook: up-wp-both/up-wp-both_work.yml
- import_playbook: git-all/git-all_work.yml
- import_playbook: cp-db/cp-db_work.yml
- import_playbook: ch-url/ch-url_work.yml
- import_playbook: up-up/up-up_work.yml
When the playbooks are assembled (really just concatenated) in this playbook we will see what variables it puts in. This is useful to better see what the playbooks would do before we run them. I’m not going to show you these here because they contain sensitive data. However, I will assembly the playbooks before they have variables replaced and show you how variables are being looked up:
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
#assembles the working scripts
- hosts: localhost
tasks:
- name: assemble up-wp
shell: cat up-wp-both/up-wp-both_work.yml > ./output.yml
- hosts: localhost
tasks:
- name: assemble git-all
shell: cat git-all/git-all_work.yml >> ./output.yml
- hosts: localhost
tasks:
- name: assemble cp-db
shell: cat cp-db/cp-db_work.yml >> ./output.yml
- hosts: localhost
tasks:
- name: assemble ch-url
shell: cat ch-url/ch-url_work.yml >> ./output.yml
- hosts: localhost
tasks:
- name: assemble up-up
shell: cat up-up/up-up_work.yml >> ./output.yml
This creates the following:
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
- hosts: "{{ src_host_var | default(src_host) }}"
vars:
dest_site_folder: "{{ lookup('vals', gv_sh_var + '.site_folder') }}"
roles:
- set-if-unset
- update-wp
- hosts: "{{ dest_host_var | default(dest_host) }}"
vars:
dest_site_folder: "{{ lookup('vals', gv_dh_var + '.site_folder') }}"
roles:
- set-if-unset
- update-wp
- hosts: "{{ src_host_var | default(src_host) }}"
vars:
git_dir: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.repo_folder') }}"
src_branch: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.repo_branch') }}"
dest_branch: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.repo_branch') }}"
roles:
- set-if-unset
- git-get-changes-to-repo
- hosts: "{{ dest_host_var | default(dest_host) }}"
vars:
dest_branch: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.repo_branch') }}"
git_dir: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.repo_folder') }}"
roles:
- set-if-unset
- git-repo-to-dest
- hosts: "{{ src_host_var | default(src_host) }}"
vars:
src_db_name: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.db_name' ) }}"
src_login_user: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.db_user' ) }}" #for mysql
src_login_password: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.db_pass' ) }}" #for mysql
src_login_host: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.host') }}" #for mysql
src_login_port: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.port') }}" #for mysql
src_dump_path: "{{ lookup('vals', gv_sh_var + '.user_folder' ) }}/{{ lookup('vals', gv_sh_var + '.site_group' ) }}_dump.sql"
src_copy_path: "{{ lookup('vals', gv_sh_var + '.user_folder' ) }}/{{ lookup('vals', gv_sh_var + '.site_group' ) }}_dump.sql"
dest_copy_path: "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}_dump.sql"
wp_prefix: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.table_prefix' ) }}"
roles:
- set-if-unset
- dump_db #dumps the database to a .sql file
- copy_file
- hosts: "{{ dest_host_var | default(dest_host) }}"
vars:
dest_db_name: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.db_name' ) }}"
dest_db_filepath: "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}_dump.sql"
dest_backup_dump_path: "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}backup_dump.sql"
dest_login_user: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.db_user') }}"
dest_login_password: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.db_pass') }}"
dest_login_host: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.host') }}"
dest_login_port: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.port') }}"
wp_prefix: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.table_prefix' ) }}"
src_copy_path: "{{ lookup('vals', gv_sh_var + '.user_folder' ) }}/{{ lookup('vals', gv_sh_var + '.site_group' ) }}_dump.sql"
dest_copy_path: "{{ lookup('vals', gv_dh_var + '.user_folder' ) }}/{{ lookup('vals', gv_dh_var + '.site_group' ) }}_dump.sql"
roles:
- set-if-unset
- copy_file_push
- restore_db
- hosts: "{{ dest_host_var | default(dest_host) }}"
vars:
src_site_url: "{{ lookup('vals', gv_sh_var + '.wrapped_Node.site_url') }}"
dest_site_url: "{{ lookup('vals', gv_dh_var + '.wrapped_Node.site_url') }}"
dest_site_folder: "{{ lookup('vals', gv_dh_var + '.site_folder') }}"
roles:
- set-if-unset
- change_url
- hosts: "{{ src_host_var | default(src_host) }}"
vars:
src_deploy_user: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
src_deploy_group: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
# dest_deploy_user: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
# dest_deploy_group: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
src_copy_path: "{{ lookup('vals', gv_sh_var + '.site_folder' ) }}wp-content/uploads"
dest_copy_path: "{{ lookup('vals', gv_dh_var + '.site_folder' ) }}wp-content/uploads"
roles:
- set-if-unset
- copy_folder
- hosts: "{{ dest_host_var | default(dest_host) }}"
vars:
src_deploy_user: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
src_deploy_group: "{{ lookup('vals', gv_sh_var + '.user' ) }}"
# dest_deploy_user: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
# dest_deploy_group: "{{ lookup('vals', gv_dh_var + '.user' ) }}"
src_copy_path: "{{ lookup('vals', gv_sh_var + '.site_folder' ) }}wp-content/uploads"
dest_copy_path: "{{ lookup('vals', gv_dh_var + '.site_folder' ) }}wp-content/uploads"
roles:
- set-if-unset
- copy_folder_push
The roles referenced here are as follows:
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
joel@joel:~/automation/greenwell-auto/roles$ cat set-if-unset/tasks/main.yml
---
- name: set gv_sh if undefined
set_fact:
gv_sh: ""
when: gv_sh is not defined
# tasks file for set-if-unset
- name: setting facts if facts unset
set_fact:
gv_sh_var: "{{ gv_sh_var | default(gv_sh) }}"
gv_dh_var: "{{ gv_dh_var | default(gv_dh) }}"
joel@joel:~/automation/greenwell-auto/roles$ cat update-wp/tasks/main.yml
---
# tasks file for update-wp---
- name: update wordpress
shell: cd "{{ dest_site_folder }}" && wp core update
joel@joel:~/automation/greenwell-auto/roles$ cat git-get-changes-to-repo/tasks/main.yml ---
# tasks file for git-get-changes-to-repo---
# see if there are changes to stage
- name: git status
shell: "cd '{{ git_dir }}' && git status"
register: git_status
# stage changes
- name: git add .
shell: "cd '{{ git_dir }}' && git add ."
when: git_status.stdout is not match(".*\nnothing to commit.*")
# commit changes
- name: git commit
shell: "cd '{{ git_dir }}' && git commit -m 'ansible commit'"
when: git_status.stdout is not match(".*\nnothing to commit.*")
# push changes
- name: git push changes
shell: "cd '{{ git_dir }}' && git push origin '{{ src_branch }}'"
# when: git_status.stdout is not match(".*\nnothing to commit.*")
# checkout proper branch
- name: checkout branch to merge into
shell: "cd '{{ git_dir }}' && git checkout '{{ dest_branch }}'"
# when: git_status.stdout is not match(".*\nnothing to commit.*")
# merge branch
- name: merge branch
shell: "cd '{{ git_dir }}' && git merge '{{ src_branch }}'"
# when: git_status.stdout is not match(".*\nnothing to commit.*")
# push changes
- name: git push changes
shell: "cd '{{ git_dir }}' && git push origin '{{ dest_branch }}'"
# when: git_status.stdout is not match(".*\nnothing to commit.*")
# checkout back to original branch
- name: git checkout original branch
shell: "cd '{{ git_dir }}' && git checkout '{{ src_branch }}'"
# when: git_status.stdout is not match(".*\nnothing to commit.*")
#what it used to be before i noticed some problems
#when: git_status.stdout is not match(".*\nnothing to commit, working tree clean")
joel@joel:~/automation/greenwell-auto/roles$ cat git-repo-to-dest/tasks/main.yml
---
# tasks file for git-repo-to-dest---
- name: git pull origin {{ dest_branch }} in {{ git_dir }}
shell: "cd '{{ git_dir }}' && git pull origin '{{ dest_branch }}'"
joel@joel:~/automation/greenwell-auto/roles$ cat dump_db/tasks/main.yml
---
# tasks file for dump_db
- name: dump database "{{ src_db_name }}" to "{{ src_dump_path }}"
mysql_db:
state: dump
name: "{{ src_db_name }}"
target: "{{ src_dump_path }}"
login_user: "{{ src_login_user }}"
login_password: "{{ src_login_password }}"
login_host: "{{ src_login_host }}"
login_port: "{{ src_login_port }}"
joel@joel:~/automation/greenwell-auto/roles$ cat copy_file/tasks/main.yml
---
# tasks file for copy_file
- name: copy file to destination
synchronize:
src: "{{ src_copy_path }}"
dest: "{{ dest_copy_path }}"
mode: pull
delegate_to: "{{ dest_host_var | default(dest_host) }}"
when: ((gv_dh_var | default(gv_dh)) is not search("localhost")) and ((gv_sh_var | default(gv_sh)) is not search("localhost"))
register: copied
- name: copy file to localhost
synchronize:
mode: pull
src: "{{ src_copy_path }}"
dest: "{{ dest_copy_path }}"
when: ((gv_dh_var | default(gv_dh)) is search("localhost")) or ((gv_sh_var | default(gv_sh)) is not search("localhost")) and copied.skipped is defined and copied.skipped == true
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
joel@joel:~/automation/greenwell-auto/roles$ cat copy_file_push/tasks/main.yml
---
# tasks file for copy_file_push---
- name: copy file from localhost
synchronize:
src: "{{ src_copy_path }}"
dest: "{{ dest_copy_path }}"
mode: push
when: gv_sh_var | default(gv_sh) is search("localhost") and gv_sh_var | default(gv_dh) is search("localhost")
joel@joel:~/automation/greenwell-auto/roles$ cat restore_db/tasks/main.yml
---
#3 does database exist?
- name: check if DB exists
shell: "mysql -u{{ dest_login_user }} -p'{{ dest_login_password }}' -h{{ dest_login_host }} -P{{ dest_login_port }} -e 'SHOW DATABASES;' | grep {{ dest_db_name }}"
register: dbstatus
failed_when: dbstatus.rc == 2
#4 if database on other server exists then backup
- name: backup database to "{{ dest_backup_dump_path }}"
mysql_db:
state: dump
name: "{{ dest_db_name }}"
target: "{{ dest_backup_dump_path }}"
login_user: "{{ dest_login_user }}"
login_password: "{{ dest_login_password }}"
login_host: "{{ dest_login_host }}"
login_port: "{{ dest_login_port }}"
when: dbstatus.rc == 0
register: backed_up
- name: delete database "{{ dest_db_name }}"
mysql_db:
name: "{{ dest_db_name }}"
state: absent
login_user: "{{ dest_login_user }}"
login_password: "{{ dest_login_password }}"
login_host: "{{ dest_login_host }}"
login_port: "{{ dest_login_port }}"
when: backed_up.changed and dbstatus.rc == 0
#5 create database
- name: Create database "{{ dest_db_name }}"
mysql_db:
name: "{{ dest_db_name }}"
state: present
login_user: "{{ dest_login_user }}"
login_password: "{{ dest_login_password }}"
login_host: "{{ dest_login_host }}"
login_port: "{{ dest_login_port }}"
when: backed_up.changed or dbstatus.rc == 1
#6 import database
- name: Import wordpress database
mysql_db:
state: import
name: "{{ dest_db_name }}"
login_user: "{{ dest_login_user }}"
login_password: "{{ dest_login_password }}"
login_host: "{{ dest_login_host }}"
login_port: "{{ dest_login_port }}"
target: "{{ dest_db_filepath }}"
when: backed_up.changed or dbstatus.rc == 1
joel@joel:~/automation/greenwell-auto/roles$ cat change_url/tasks/main.yml
---
# tasks file for change_url---
- name: change url
shell: cd "{{ dest_site_folder }}" && wp search-replace "{{ src_site_url }}" "{{ dest_site_url }}" --allow-root # --skip-columns=guid --allow-root
#this is does not deal with serialized data: "mysql -h{{ dest_login_host }} -u{{ dest_login_user }} -p{{ dest_login_password }} --password=root wordpress1 -e \"UPDATE {{ wp_prefix }}options SET option_value = REPLACE(option_value, '{{ src_site_url }}', '{{ dest_site_url }}') WHERE option_name = 'home' OR option_name = 'siteurl'; UPDATE {{ wp_prefix }}posts SET guid = REPLACE(guid, '{{ src_site_url }}', '{{ dest_site_url }}'); UPDATE {{ wp_prefix }}posts SET post_content = REPLACE(post_content, '{{ src_site_url }}', '{{ dest_site_url }}'); UPDATE {{ wp_prefix }}postmeta SET meta_value = REPLACE(meta_value, '{{ src_site_url }}', '{{ dest_site_url }}');\""
joel@joel:~/automation/greenwell-auto/roles$ cat copy_folder/tasks/main.yml
---
# tasks file for copy_file
- name: copy folder to destination
synchronize:
src: "{{ src_copy_path }}/"
dest: "{{ dest_copy_path }}"
mode: pull
rsync_opts:
- "-a"
delegate_to: "{{ dest_host_var | default(dest_host) }}"
when: (( gv_dh_var | default(gv_dh)) is not search("localhost")) and ((gv_sh_var | default(gv_sh)) is not search("localhost"))
register: copied
- name: copy folder to localhost
synchronize:
mode: pull
src: "{{ src_copy_path }}/"
dest: "{{ dest_copy_path }}"
rsync_opts:
- "-a"
when: ((gv_dh_var | default(gv_dh)) is search("localhost")) or ((gv_sh_var | default(gv_sh)) is not search("localhost")) and copied.skipped is defined and copied.skipped == true
joel@joel:~/automation/greenwell-auto/roles$ cat copy_folder_push/tasks/main.yml
---
- name: copy folder from localhost
synchronize:
src: "{{ src_copy_path }}/"
dest: "{{ dest_copy_path }}"
mode: push
rsync_opts:
- "-a"
when: gv_sh_var | default(gv_sh) is search("localhost") and gv_sh_var | default(gv_dh) is search("localhost")
There is also another task I haven’t mentiond that if placed in the following playbook would reverse the source and destination hosts:
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
roles:
- set-if-unset
# actually does the work/runs the playbooks:
- import_playbook: up-wp-both/up-wp-both_work.yml
- import_playbook: git-all/git-all_work.yml
- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
roles:
- reverse-facts
- import_playbook: cp-db/cp-db_work.yml
- hosts: "{{ src_host_var | default(src_host) }}, {{ dest_host_var | default(dest_host) }}"
roles:
- reset-reversed-facts
- import_playbook: ch-url/ch-url_work.yml
- import_playbook: up-up/up-up_work.ymlBassically everything in the above playbook would run from source to destination except the cp-db_work.yml script. These roles are displayed below:
joel@joel:~/automation/greenwell-auto/roles$ cat reverse-facts/tasks/main.yml
---
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
# tasks file for reverse-facts
- name: this will reverse the facts for hosts
set_fact:
src_host_var: "{{ dest_host }}"
dest_host_var: "{{ src_host }}"
gv_sh_var: "{{ gv_dh }}"
gv_dh_var: "{{ gv_sh }}"
joel@joel:~/automation/greenwell-auto/roles$ cat reset-reversed-facts/tasks/main.yml
---
#Î This program is free software: you can redistribute it and/or modify Î#
#Î it under the terms of the GNU General Public License as published by Î#
#Î the Free Software Foundation, either version 3 of the License, or Î#
#Î (at your option) any later version. Î#
#Î Author: Joel Cambon Î#
#Î Company: Greenwell Î#
#Î (c) 2019 Î#
# tasks file for reset-reversed-facts---
- name: this will reset the facts (that have been reversed)
set_fact:
src_host_var: "{{ src_host }}"
dest_host_var: "{{ dest_host }}"
gv_sh_var: "{{ gv_sh }}"
gv_dh_var: "{{ gv_dh }}"
To summarize what I created:
One ansible script is reusable for multiple hosts and sets of variables or site-credentials (as long as you have those in a group_vars/all.yaml file) I have a script that gets the credentials in JSON from all the wp-config.php files under a certain folder on the server. It also queries the databases as well so you can create group_vars/all.yaml automatically if you already have the sites you want to work with set up. It is parsed and converted to yaml by passing it through python classes that are converted to dictionaries. Then you can pass source and destination site variables to the “universal” ansible scripts and they will do whatever action they do on those source and destination sites. That way you don’t have to create different scripts to do the same task on different sites and you have some credentials and variables you can just modify or change that are separate from the script. So now you could just have one script for each action and any app that was interfacing with this would just have to pass the hostnames and site name to the script. All this allows for reusable scripts and less clutter in the ansible folder. I also made a way for you to test to see what the script would do. If you run a template action I have it uses the ansible script as a template and creates a file that shows you exactly what variables it would put in)
When running the scripts you don’t need to put in all the variables in all.yaml just the ones necessary for that task and this is true even if you are creating a new site. By running an.py playbook.yaml
you will create a command by parsing the variables in all.yaml. I had it only parse variables that exist–that way you don’t have to create a dictionary of variables for each new site.
0 Comments