Tuesday, March 17, 2015

FileCrush replacement format

Error message:
-bash: --replacement=$1-${crush.timestamp}-${crush.task.num}-${crush.file.num}: bad substitution

Fix by using escape character in bash:
--replacement=\$1-\$\{crush.timestamp\}-\$\{crush.task.num\}-\$\{crush.file.num\}

Wednesday, March 4, 2015

Parse hosts file in Fabric

Add the following code to your fabfile.py
import re
hosts_list = []
with open('hosts_file') as f:
for line in f:
if not (line.strip()=='' or line.startswith('#')):
if "[" in line:
prefix = line.split('[')[0]
m = re.search(r"\[([0-9\-]+)\]", line)
a = m.group(1)
start, end = a.split('-')
start_i = int(start)
end_i = int(end) + 1
if start.startswith('0'):
for i in range(start_i, end_i):
val = prefix + "%02d" % i
hosts_list.append(val)
else:
for i in range(start_i, end_i):
val = prefix + "%d" % i
hosts_list.append(val)
else:
hosts_list.append(line)
env.hosts = hosts_list
view raw fabfile.py hosted with ❤ by GitHub

The "hosts_file" support comment and host sequence:
$ cat hosts_file
# This is comment. using # asprefix
hadoopdev01
hadoopprod[01-10]
#hadoopdev02
192.168.1.[1-200]