Making script logs clean

There are a lot of ways to clean up script logs either using macros in vim, sed one-liners or probably stand alone tools but I’ve found the easiest way is a simple function and alias within bash.

Add the following to your .bashrc file

function __sc {
        cat $1 | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > $1-processed

alias script_clean="__sc"

source the .bashrc and now you can execute $ script_clean scriptlog.log and it will output scriptlog.log-processed so its nice and clean without all those pesky control characters.


bhistory, a tool for nobody

Previously I enabled Timestamps in Bash to allow me to know when someone was doing stupid things, such as deleting vixie-cron or truncating databases at midnight, now I have system that backups all of the .bash_history files on my production system and stores them so I can look for stupid behavior, however the logfiles are a little unfriendly to the human eye

example of logfiles go here

This necessitated the creation of a script that would let me view the history file as it was originally viewable with the history command and the timestamps parsed, thus bhistory was created.


import re
import time
import sys
import os.path

try: sys.argv[1]

except: sys.exit("\nNo file specified!\nUsage:\nbhistory <filename>\n")

if os.path.isfile(sys.argv[1]):
        # file exists!
        print ""
        sys.exit("\nfile doesn't exist, try again\n")

with open(sys.argv[1]) as input_file:
        for i, line in enumerate(input_file):
                #print line,
                m_obj ="^\#",line)
                if m_obj:
                        new = line.strip('#')
                        print time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(int(new)))
                else :
                        print line
print "{0}line(s) printed".format(i+1)

Simple chmod the file, call it bhistory or whatever else you please and then drop it somewhere that your path statement covers and its time to view that logfile (and hopefully not find stupid things happening)

Usage: # bhistory /home/somebody/.bash_history

Naturally nobody should consider this a script that I support in any way, use this mess of code at your own discretion just make sure you have the timestamps enabled or not much is going to happen when you use it.

SSH Connection Script

Ever have a shitty environment where you have tons of devices that you need to connect to and don’t feel like typing out (and invariably mis-typing the hostname)?  Then this dirt simple script might be what you are looking for.

#!/usr/bin/env python
# requires file 'hosts' to exist in same directory as the script
# can accept either ip addresses or hostnames in this file

import os
import pwd

user = pwd.getpwuid( os.getuid() )[ 0 ]

f = open('hosts', 'r')

hosts = f.readlines()


print "=== SSH Console v1 ===\n\n"
for host in hosts:
        print str(id)+": "+host.rstrip("\n")
        id += 1 

print "\n"
choice = raw_input('Host >')
choice = int(choice)

os.system("ssh "+user+"@"+hosts[choice])

Just make sure the host file exists and has a hostname on each line rather than smashed together with some delimiter.  Its not elegant, has zero configuration but so far folks at work have found it somewhat handy to save some time, maybe you will too.


Home isn’t there anymore

Its kind of weird to move out of a place you have lived in for half a decade with the same people, same neighbors, same everything really. Unfortunately it couldn’t be avoided as the roommates have different directions to go now and the landlord wants to sell the property. So now I find myself staying with friends until my new place to live will be ready (its currently under construction now) and living as close to a nomad as I have in a long time. No lab equipment now, no comfortable place to sleep with the soothing sound of electronics humming along. I would be a lying shit if I didn’t admit that I feel more than a little lonely at the idea of no longer getting to see the people I lived with for years. Of course there were the promises to keep hanging out after going separate ways as friends always do when parting but the reality of things is that we are different people with different schedules who quite likely will never cross paths again once business with the now former landlord is concluded. Some have budding relationships leading them towards things like marriage and children, some have changing career paths to take them to new places.

Intuit holds your data hostage

So I recently started the process of applying for a home loan to buy a house and during the process I was asked to produce tax documents from the past 3 or 4 years. The past 3 years weren’t much of a problem till I got to the last year I filed using TurboTax’s website to file, since I hadn’t used them in several years they basically locked me out of access to my return paperwork; every road basically said I could file with the IRS to get the documents (which we all know might take the better part of a month) or I could shell out about 50$ to Intuit to purchase the current years basic ‘product’ just to get access to prior year returns. After going around and around with the website in hopes of finding another way I was left with no choice but to pay the extortion fee to access my information so I could print off my return and provide it to my mortgage originator. The moral of the story is to always check out who you are doing business with and what kind of policies for accessing historical data they have so you don’t end up getting scammed out of money for essentially nothing.

Remote Check

So I’ve been working on setting up a way to monitor a server at work to prevent my “assistant” from screwing up too badly and I’ve come up with a temporary solution until I can work out a much more robust script.

There are a few required elements, like the rc.head and rc.tail files which basically are the top and bottom of the email it sends out. Everything else is created and removed while the script is running making for a mostly clean experience. There needs to be at least (and ideally only) a one-way SSH trust from the testing machine to the target machine. I setup everything this way because the machine actually running the script is locked down to keep my boss and assistant from touching (see also: breaking) anything. Even if some fool removes the vixie-cron package this script continues to function until either a loss of network connectivity occurs or someone breaks the ssh trust (both of which in later versions will cause alert emails to go out)


# Script requires rc.head and rc.tail to be at least present, handy places
# to put some generic information to be included in each alert email

> errors.txt # this should zero out the file


# function: checksvc
# argument: service name to check
# returns: string statement 

SV="service $SVC status"
V=`ssh $HOST "$SV"`
        touch errors.txt

if echo $V | grep -q 'is running'
        #echo "$1 is running"
        echo ""
        echo "$1 is NOT running on host $HOST" >> errors.txt

checksvc httpd
checksvc memcached
checksvc mysqld
checksvc crond
checksvc postfix

if [ -s errors.txt ]
        # errors present
        cat rc.head > mail.msg
        cat errors.txt >> mail.msg
        echo "Scan run at $DATE" >> mail.msg
        cat rc.tail >> mail.msg
        mail -s '[ALERT]Services NOT running' -a errors.txt first.last@host.tld < mail.msg
        # no errors present
        echo ""

# lets clean some junk up
rm mail.msg

Backing up Minecraft

I run an in-house server for myself and some friends and I got tired of manually running backups, by logging in, stopping the server, making the tar.gz file and then moving it around, so I threw together this simple script.  It relies upon an active screen session for the server and a named window called Server to be present, but other than that it should be pretty straight forward how to works.


# change the argument for -S to whatever screen -ls says is the right value
# this allows us to target the running screen sessions so if there is a server
# reboot then this script must be updated (until I work out the code to make
# all that happen automatically

screen -S 436.minecraft -p Server -X stuff '/say Preparing for Server Backup
/say please get to a safe area soon (2 minutes)
sleep 2m # this should be 2m when testing is done
screen -S 436.minecraft -p Server -X stuff '/say commencing backup process now
/say this should take about 15 minutes max
/say server is going down NOW!

cd mc-server
tar -cf "mc-backup-"`date +%Y_%m_%d`".tar" world*
gzip "mc-backup-"`date +%Y_%m_%d`".tar"
mv "mc-backup-"`date +%Y_%m_%d`".tar.gz" ~/backups

screen -S 436.minecraft -p Server -X stuff './

It wouldn’t be hard to make the script scp the file to a different server or possibly push to a cloud provider like Dropbox. The key here is that it also brings the server back up so you could very easily make this a completely automatic task if so inclined.

Git as a Backup System

So I have a co-worker on a coding project who has all but refused to use any kind of version control  ( and in general prefers to do most of his work in production ) so I’ve been kicking around ways to either force him to do the work, or at least make changes get committed to to a repository without his interaction at all. Took a little reading around to figure out the flags necessary but I managed to get everything working with shared ssh keys added into my bit-bucket settings as well as a cron job that runs at set intervals (in my case at about 10 am when the co-worker’s shift has ended) and pushes all changes including new files to the repository so that I can review what work he has done and the things he has broken. Unfortunately as I got all of this working it seems the web proxy between the server and the outside world shit the bed and I can’t push commits out to my external (and not under the control of the co-worker) repository.

Flattery…ain’t always that flattering.

So I can now say that the idea of flattery is not all its cracked up to be, at work I develop some tools that the department uses and to help out my boss and an ‘assistant’ sometimes ‘help’ out. Today that helping ended up being bastardizing code I had previously written that worked just fine.  There were lines commented out all over the place, loops within loops (with additional code that wasn’t there before) that just made the whole thing shit the bed when it came time to actually look at what was being created with the code.  Turns out they don’t know how to properly format HTML or even how to design something to be efficient; instead of just pulling data for the currently logged in user they created this huge drop-down menu that lists all the users and you pick your name out of that list.

Normally I wouldn’t even bother complaining about this here but the supervisor who so graciously left their name in comments they put into the code claims to have been a programmer at their previous job, however more and more I think they were simply writing SQL queries and using some kind of a front-end tool to format and display everything.  Sure that still indicates some level of skill in knowing how to do joins and all the other fancy wizardry associated with databases but it doesn’t indicate any kind of a knowledge about actual programming in general (like not putting things inside loops if it doesn’t need to be there) or even simple things like how to properly utilize html tags to positive effect.

A Year in Retrospect

This actually has nothing to do with a year ending so much as it does with wondering where I am in my life and where I am headed.

I spent considerable time learning how to do all kinds of things with computers, from understanding concepts of security to programming, databases, managing virtualization environments and all kinds of other things as well.  Despite all of that I work as a Tier 2 Linux Engineer for a massive company; as an employee I generate value for the business and that value translates to a benefit or service to the customers we server but I wonder if that’s really enough to be satisfied with my employment on a personal level.

I know several people who currently program for a living and take part in open source projects and generally contribute to things that lots of people use and derive benefit and profit from.  It seems kind of cliche to even think it but I can’t help but wonder if there is more to it than just being that wheel that never squeaks and gets compensated well for said lack of squeaking.  Part of me yearns to get involved in something for fun, for enjoyment and the sense of contribution to something larger than myself rather than just working 40+ hours a week to get paid, but at the same time I wonder how to balance that desire to do something bigger against the very real need to earn an income to survive.

Maybe its just the fact that I’m trapped at work on a holiday when I could be with friends enjoying a few drinks, maybe its natural sense of an ending of the year that is stirring thoughts that generally don’t get much time on the front of the stove but maybe its time to really consider what I am doing with myself, my career and life.

its like a seizure, except not