Easy Linux Backups
March 26, 2008

As some of you may know, a sad time was had when our Christmas Tree's extra 200W overloaded the breaker that my server was on. Subsequent powerups showed that the server's primary hard disk was indeed toast. I had good backups of all of my important data, but most of the setup data for Courier and Samba were lost. Not too much pain, but it was still a bit irritating to restore.

To whit, I have created a new backup script for my system! I partitioned the new disk straight in half, keeping 40G for the primary partition and 40G for the backup. RAID-1 cries James! Nay, good sir, for I know the perils of RAID as backup. A good backup system shall include days if not weeks of backups that I can go back to and consult should I mire my setup data.

Loe! My new backup script maintains 4 days worth of daily backups, 2 weeks worth of weekly backups and a single monthly backup. I of course burn regular backups, for the times in between. Now, say you, I want to see this backup script that you so gleefully refer to. Well, good sir, here it'tis!

#!/bin/sh
# weekly_backup.sh
# modified: 2008/01/14
# version:  1.0
# history:
# - v1.0
#   Added support for ITEMS_BACKWARDS instead of DAYS backwards, removing
#   the need to compute how many days worth of backups you want.
#

set -e
DIRECTORIES="/bin /etc /home /var /usr"
BASE_DIR="/media/backup"

if [ ! $1 ]; then
  echo "You must pass either \"daily\", \"weekly\", or \"monthly\"."
  exit 0
fi


# -- Initial setup...
case $1 in
  "daily" )
    ITEMS_BACKWARDS=4;;
  "weekly" )
    ITEMS_BACKWARDS=2;;
  "monthly" )
    ITEMS_BACKWARDS=1;;
  * )
    echo "Invalid argument."
    echo "  You must pass either \"daily\", \"weekly\", or \"monthly\"."
    exit 1;;
esac

# -- Run the backup...
DATEF=`date +%Y-%m-%d`
TARGET="$BASE_DIR/$1/$DATEF"

mkdir -p $TARGET
for i in $DIRECTORIES; do (
  rsync -avz $i $TARGET
); done

# -- Remove old backups...
ITEMS=`find $BASE_DIR/$1 -maxdepth 1 | egrep "[0-9]{4}\-[0-9]{2}\-[0-9]{2}" | wc -l`
if [ $ITEMS -gt $ITEMS_BACKWARDS ]; then
  TO_REMOVE=`expr $ITEMS - $ITEMS_BACKWARDS`
  echo "$TO_REMOVE items will be removed."
  for i in `find $BASE_DIR/$1 -maxdepth 1 | egrep "[0-9]{4}\-[0-9]{2}\-[0-9]{2}" | sort -r| tail -n$TO_REMOVE`; do (
    rm -rf $i
  ); done
fi

This script gets called by the following cron structure daily, weekly and monthly:

# m h  dom mon dow   command
  1 2  *   *   *     bash /usr/local/bin/backup.sh daily
  1 3  */7 *   *     bash /usr/local/bin/backup.sh weekly
  1 4  1   *   *     bash /usr/local/bin/backup.sh monthly

Then I end up with lovely, delicious and fresh backups in /media/backup/{daily,weekly,monthly}. Om noms!

Linux Printserver With HP C3180
January 12, 2008

I fumbled through a good bit of this and would rather have found a simple how-to on getting this setup. Alas, one was not readily accessible and I hope to remedy this! Please note, this is a print server not necessarily a Samba file server so I will only highlight the portions relevant to printing.

First, you must install cupsys using "aptitude install cupsys" and the HP drivers by "aptitude install hpijs. This will get all the software installed that you need to get going. If you're doing this from scratch, you'll also want to install Samba at this point.

Now that you've got cups installed, you'll need to configure it by editing "/etc/cups/cupsd.conf". In here you'll be needing to change the "Listen" option to listen on your local ethernet interface and enabling /admin and /admin/conf access to all by doing the following:

<Location /admin/conf>
  AuthType Default
  Require user @SYSTEM
  Order allow,deny
  #Allow localhost
  Allow all
</Location>

for both "/admin/conf" and "/admin". This will allow you to connect to the admin panel from any machine on your network. You'll need to go in to the web site through the interface specified earlier and select "Add Printer" . I found that the "HP PhotoSmart P100 Foomatic/cdj1600" driver works quite well for the HP C3180 that I have.

Once you've gotten the printer added into cups, go ahead and print a test page through the web interface. Did that work? Good. You're almost there!

Now the Samba configuration is pretty simple. Just add printing = cups to the top of you /etc/samba/smb.conf and the following to the bottom:

[printers]
  comment = SMB Print Spool
  path = /var/spool/samba
  guest ok = Yes
  printable = Yes
  #use client driver = Yes
  default devmode = Yes
  browseable = Yes

This is from the Secure Office Networking guide from the Samba site. You'll notice that I commented out "use client driver", however. I did this so that I could use the printer as a delicious raw Postscript printer and not have to both with much clientside configuration.

That's it. Now you've got (or at least I've got) a shiny new print server. You should probably restrict your admin rights by re-editing "/etc/cups/cupsd.conf" and then you're golden. If you've got any questions, feel free to leave them in the comments section and I'll try to lend a hand. Good luck!

A Brief Guide to Backing up Your Site on Joyent
January 02, 2008

I've revamped a lot of my backup strategies lately, and I feel like I need to share my latest update - backing up my joyent hosted blog to my local development server. Performing a nightly backup of the full contents of my blog assures me that, should Joyent ever perish, my beautiful writings and incredible content will not be lost.

Avast! The first order of business is getting a rake task setup to actually backup the database. This is as simple as using Craig Ambrose's Rake Backup Task with a quick little modification. Since I use postgresql, the original mysql_dump command wasn't quite up to snuff. I also added in a quick regex expression to make sure that the script was only deleting folders which matched the backup scheme as a precaution.

lib/tasks/backup.rake [updated 2008-01-02]

require 'find'
namespace :db do
  desc "Backup the database to a file. Options: DIR=base_dir RAILS_ENV=development MAX=10"
  task :backup => [:environment] do
    datestamp = Time.now.strftime("%Y-%m-%d")
    base_path = ENV["DIR"] || "db"
    backup_base = File.join(base_path, 'backup')
    backup_folder = File.join(backup_base, datestamp)
    backup_file = File.join(backup_folder, "#{RAILS_ENV}_dump.sql.gz")
    File.makedirs(backup_folder)
    db_config = ActiveRecord::Base.configurations[RAILS_ENV]
#    sh "mysqldump -u #{db_config['username']} -p#{db_config['password']} -Q --add-drop-table -O add-locks=FALSE -O lock-tables=FALSE #{db_config['database']} | gzip -c
    sh "pg_dump -U #{db_config['username']} #{db_config['database']} | gzip -c > #{backup_file}"
    puts "-- Created backup: #{backup_file}"

    deleted_backups = 0
    max_backups = ENV["MAX"] ? ENV["MAX"].to_i : 10
    unwanted_backups = Dir.new(backup_base).entries.sort[2..-1].reverse[max_backups..-1] || []
    unwanted_backups.each do |unwanted_backup|
      next unless unwanted_backup.match(/[0-9]{4}\-[0-9]{2}\-[0-9]{2}/)
      FileUtils.rm_rf(File.join(backup_base, unwanted_backup))
      puts "-- Deleted #{unwanted_backup}"
      deleted_backups+=1
    end

    puts "-- Deleted #{deleted_backups} backups."
  end

end

Go ahead and run "RAILS_ENV=production rake db:backup" (while in the rails app's root) to test the backup scheme and you should see a new folder created underneath "db/" called "backup". That should contain a folder dated after today's date. That contains your most up to date database backup. Joy!

A Joyent Note; The pg_dump requires the localhost specification otherwise the connection will be dropped.

Hold on, you say! I was prompted for a password, how can I automate that? It's actually a pretty simple matter of populating your "~/.pgpass" file with your login credentials. Here's an example;

~/.pgpass

localhost:*:YOURDATABASENAME:YOURUSERNAME:YOURPASSWORD

This file will need to be chmod'ed to 0600 in order to ensure that nobody else can read it, of course. Afterwards you should be able to run the prior rake command (RAILS_ENV=production rake db:backup) without being prompted for a password. Now you're one step closer!

Go ahead and add the Rake task to your crontab as thus;

crontab -e

23   23  *   *   *     cd WHEREVERYOURRAILSAPPIS && PATH=/usr/local/bin:/usr/bin:/bin RAILS_ENV=production rake db:backup

Cron on Joyent's servers seems to lack any pathing - so the specific PATH variable is necessary. Joyent's version of Cron also seems to lack the ability to specify environment variables directly in the file, so you have to specify a sane PATH for EACH command used. A PITA if you ask me. Whicher way you look at it, your cron task should be setup to automatically generate a backup of your database every night at 23:23.

Now that you've got your site automatically backing up on Joyent's servers you're in good shape. However, as any good BOFH should know - unless the data's on your servers it's not in your control. We need to get the data syncing up to our local development box in order to have truly powerful backups.

Go ahead and get passwordless SSH running between your development box and Joyent's server and then just set yourself up a simple nightly rsync task!

crontab -e (on your local dev box)

  2    1  *   *   *     rsync -avz USERNAME_ON_JOYENT@SERVER_ON_JOYENT.joyent.us:PATH_TO_YOUR_APP_ON_JOYENT YOUR_LOCAL_DESTINATION

That's it. You'll have a tidy nightly backup of what's up on your Joyent account. Since you're already keeping good local backups, you don't have to worry about making it any more complicated than that. Now, enjoy your new backed up data!

Update 2008-01-02

I found a rather nasty bug in the backup script that caused the removal rules to fail and simply remove all backups. The crucial item here was changing "max_backups = ENV["MAX"] ? ENV["MAX"].to_i : 10" to use ternary instead of simply being an or statement - because ENV["MAX"].to_i was returning a true value even when ENV["MAX"] wasn't specified - causing the script to believe that max_backups was 0. Damn. Fixed now.

Regenerating Courier SSL Certificates on Ubuntu Linux
December 14, 2007

Waking up groggily as usual, I booted up Thunderbird to go through all the mail that had accumulated over the night and found a new dialog gracing me. Apparently, as it was telling me, the auto-generated SSL cert had expired. Oh well, all that I needed to do was track down how to regenerate it and go about my business. Well, this actually ended up being a little bit more troublesome than usual - so I decided to record it here as a is the usual case when I come into issues (because I know that five years from now I'll have the same issue and will have forgotten this).

Anyway, it's as simple as backing up the existing certificate:

Then generating a new one to move into place:

This will generate the necessary certificate in whatever directory you're in. You could run it as root directly in the courier configuration directory, but we don't do that since we value our configuration files too much. Go ahead and move it into place:

Now that that's in place just restart the courier-imap-ssl service and you should be good to go!

Tada! Now you've got a brand new SSL certificate. I suppose that you could play with expiring the cert sooner or later by modifying the -days parameter. 365 is probably too high for any amount of auto-expiring security to take effect but I'm not amazingly concerned since I use it more for encryption than server identify verification. Enjoy your fresh new certificate now!

Thanks to Ivar Abrahamsen who's Mail Server Setup Howto led me to the generation command.

Traffic Monitoring with rrdtool
August 18, 2007

So, as some of you may know I've been having issues with my Internet at home lately as well as at work (today it appears totally dead at work). High packet loss and high latency were abounding, and I wanted to see exactly when/where it was happening. I've been using ntop for awhile to display usage statistics but found that it was lacking in terms of both fine resolution and error reporting. So, what's a geek to do? Write his own!

Surprisingly, this was a fairly difficult little project to get up and running. Maybe I was mis-understanding the rrdtool docs - but it took me almost a full day just to get basic graphing down. Well, not to let my efforts go to waste I'm here to share with you my final results today!

The Database

rrdtool is a "round robin database tool" which stores data points and statistics while providing a very nice (albeit difficult) graphing facility. The initial setup of the database can be tricky with a lot of stuff needing explanation. Please note, this is all a nice tidy shell script. No complicated Perl here!

interval=35
start=`date +%s`
b=`expr $interval \* 2`

rrdtool create status.rrd \
    --start $start\
    --step $interval\
    DS:packet_loss:GAUGE:$b:U:U\
    DS:latency:GAUGE:$b:0:450\
    DS:kbytes_in:DERIVE:$b:0:U\
    DS:kbytes_out:DERIVE:$b:0:U\
    RRA:AVERAGE:0.5:1:576\
    RRA:AVERAGE:0.5:6:672\
    RRA:AVERAGE:0.5:24:732\
    RRA:AVERAGE:0.5:144:1460

So, what's this all about now? Well, the part of the script which actually feeds the database does so at a rate of interval (35 here). The --start format for rrdtool is in seconds since the Epoch (0000 Jan 1st 1970) so, we need the "+%s" format for the starting date command.

You'll see multiple "DS" entries - these are the actual variables which are stored. The GAUGE datatype allows for constantly changing data which can be decreasing or increasing. The DERIVE datatype takes into account constantly increasing values, which allow the system to compute the difference for charting purposes. The $b variable is just the script interval multiplied by two. This value is supposed to be the interval at which rrdtool expects data to be provided - for some reason, providing two entries for each of these works. Baffle.

The following :0:U establish max and minimum boundaries. Everything outside of these is tagged "invalid" and not displayed. This allows for nice chart smoothing and preventing the scale from getting out of whack. The U simply signified "Unknown" or "Unspecified" allowing it to be variable.

The RRA Averages are the actual "statistical" storage utilities which store all of the statistics. Frankly, this took so much futzing that I have no idea what these mean. Every site that I referenced from had totally different and non-correlationary values. Take them as you want, but the 1,6,24,144 should provide for long term storage of "576 points at 1 interval each", "672 points at 6 intervals each" and "732 points at 24 intervals each" (each being the AVERAGE amongst those intervals). For some reason if these values are not just right, the whole system refuses to display data. Baffle.

Data Collection

On to the actual collection portion! Firstly, a host needs to be picked out for pinging purposes. You noticed at the beginning of the script my interval was set to 35. That's a pretty low interval for a host that you don't own. Luckily, I have access to a nice dedicated box that I like to ping into abandon in order to get my stats. You probably won't be so lucky! You can try your ISP's gateway or DNS servers, but I would definitely recommend upping the interval somewhere greater than three minutes.

With that in hand, we continue. The collection itself is contained in a while loop - which any sysadmin should understand;

while (true); do ( sleep $interval ... ); done

Ok, so the first bit of the collection process is to get the average ping time and packet loss. My link is unreliable, so packet loss is of interest to me - it might not be so to you.

p=`ping -c4 $host`
packet_loss=`echo $p | egrep -o "[0-9]+% packet loss"`
packet_loss=`echo $packet_loss | egrep -o [0-9]+`
latency=`echo $p | egrep -o "= .*?\/[0-9]+"`
latency=`echo $latency | egrep -o "\/[0-9]+"`  
latency=`echo $latency | egrep -o "[0-9]+"`
latency=`echo $latency | egrep -o " [0-9]+ "`
latency=`echo $latency | egrep -o "[0-9]+"`  

That will ping $host 4 times and yield the total packet loss and average latency across those four. This is done using the version of ping included with Ubuntu 6.0.1 - anything else may yield different numbers or nothing at all.

Next, we need to determine that amount of traffic flowing through our ethernet device. My external firewall is "eth1", your mileage may vary. I also blatantly ripped this (except the kilobyte conversion) from someone else's site. Sadly, my Internets are in such disarray that I can't find it ATM.

kbytes_in=`ifconfig eth1 |grep bytes|cut -d":" -f2|cut -d" " -f1`
kbytes_out=`ifconfig eth1 |grep bytes|cut -d":" -f3|cut -d" " -f1` 
kbytes_in=`expr $kbytes_in / 1000`
kbytes_out=`expr $kbytes_out / 1000`

Puttin' On the Ritz

We've got all the data that we need now. It's just a quick rrdtool update away from being in our database!

rrdtool update status.rrd $now:$packet_loss:$latency:$kbytes_in:$kbytes_out
# $now is specified earlier in the script as `date +%s`

rrdtool updates it's databases in the order in which they were created. So, this is setting the interval $now to have values as specified in the colon separated list. Pretty simple, eh?

Graphing

Aah, graphs! Everybody loves 'em. These are fairly straight forward (and my cat is on my keyboard) so I'm just going to copy-paste the most advanced one here. You can find the rest in my network_status.sh.

d_stamp=`date +%Y\/%m\/%d`
rrdtool graph combined.png -a PNG --title="$d_stamp - Combined Last Hour"\
    --vertical-label "Combined Data" --start=end-1h --end now\
    'DEF:mypacketloss=status.rrd:packet_loss:AVERAGE:step=1' \
    'DEF:mylatency=status.rrd:latency:AVERAGE' \
    'CDEF:mypacketloss_neg=mypacketloss,-1,*'\
    'AREA:mypacketloss_neg#FF0000:Packet Loss' \
    'GPRINT:mypacketloss:MAX:Max\: %2.1lf'\
    'GPRINT:mypacketloss:AVERAGE:Avg\: %2.1lf'\
    'GPRINT:mypacketloss:LAST:Last\: %2.1lf Percent\j'\
    'AREA:mylatency#0000FF:Latency' \
    'GPRINT:mylatency:MAX:Max\: %2.1lf'\
    'GPRINT:mylatency:AVERAGE:Avg\: %2.1lf'\
    'GPRINT:mylatency:LAST:Last\: %2.1lf ms\j'\
    > /dev/null    

The above graphing routines yield this lovely graph (well, yours probably doesn't display this much packet loss);

Conclusion

So, there you have it. Nice traffic monitoring with rrdtool. You can download the script here and give it a whirl. It uses nothing more fancy than ping, ifconfig and rrdtool itself.

Rsync, Windows, OSX and more!
July 30, 2007

Brian recently wrote in with a few questions regarding rsync and how I used rsync in my daily life. I felt much obliged to respond to him but while typing my response, I realized something. I had recently been trying my damnedest to find good information on rsync I was always coming up short. Why? I was always coming up short because I was looking to use rsync in a non-standard fashion. Rather than the tried and true Unix->Unix sync, I was trying to synchronize Windows->Unix and it was becoming a huge pain in the ass. Well, Brian's question is a bit different but along the same lines.

Brian wanted to sync a Mac OSX box with a Windows box. Well, lucky for us that OSX is Unixie and already has access to rsync - so all that leaves us with is setting up rsync on Windows and getting the right glue in between! I have a quick disclaimer, however. I have not completely tested this method. I've only pulled together what I can to help out a reader having very similar troubles to what I had. That said, if you guys have any comments or suggestions I will definitely incorporate them here.

So, here we go. We're going to setup a Cygwin installation with both OpenSSH and Rsync. There is a version of Rsync out there for Windows called "cwRsync", but in my experience its been incredibly unstable and is a poor implementation. Things might have changed, but I would recommend staying away from it for serious production.

Setting up Cygwin and OpenSSH On Your Windows Box

First, you'll need to download and install cygwin. Its a fairly straightforward procedure. When you get to the package selection dialog be sure to select "OpenSSH" and "Rsync" for installation. You might find this article a little difficult to follow without them.

Once installed you'll need to do a little configuration to get the OpenSSH server up and running. At this point, you'll need administrative privileges to continue. For Vista or XP, just be sure to run "c:\cygwin\cygwin.bat" as Administrator and you'll be fine. Go ahead and run ssh-host-config and mimic the following answers (some of which may not show up if you're running <Windows 2003. If not stated, simply accept the defaults.

Should privilege separation be used? (yes/no) yes
... [only on >= Windows 2003]
Should this script create a new local account 'sshd_server'
 which has the required privileges? (yes/no) yes
...

Now go ahead and start the SSH service manually by running "net start sshd". This may take a minute or two so give it time. You may also need to allow a "port exception" in Windows Firewall for Port 22. Just open up Windows Firewall in Control Panel and select "Change Settings". From there go to the Exceptions tab and select "Add Port...". You should now try SSH'ing into your Windows machine from a different machine, just to make sure that everything works at this point.

Running Rsync From a Remote Machine

At this point, most of the Windows configuration is done. You're going to want to begin working on your other machine now (in Brian's instance, his OSX machine) and start testing rsync. There's really no configuration to be immediately done for rsync to get it up and running since it will be using SSH as a tunnel to the Windows machine.

Go ahead and pick a small source directory to try to sync to your Windows machine. Also, make sure that there is a directory created on the Windows machine with a suitable user having access to it - here we'll call the user "rsync_access" and give him read/write permissions to "c:\rsync_repo". The source on the OSX machine will be "/rsync_repo" for simplicity.

rsync -e "ssh" -az /rsync_repo rsync_access@WINDOWSBOX:/cygdrive/c/rsync_repo

Tada! The files should be synced. One tweak I've noticed that helps Windows machines along is changing the "modify-window" option to be a little higher than usual by adding in --modify-window=15 to the command line mix.

If you're remote machine is Windows as well, you may need to install rsync via Cygwin the same way we did on the machine above. If you're on Linux, you may need to install it manually. "apt-get install rsync" will do the trick on most Debian distributions.

Fin!

At this point, everything is up and running and only needs some automation tweaks. I'll make an update to this article later with this content. Again, any comments or suggestions are warmly welcomed and will be incorporated into the article. I'm just trying to get this out there to help those in search of more rsync information.Until then, happy rsyncing!

References and Links

- cwRsync - not recommended

- Cygwin

- How to install OpenSSH sshd server and sftp server on a Windows 2000 or Windows XP [machine]

Happy Sysadmin Day!
July 27, 2007

Today is Systems Administrator Appreciation Day! It's a day to give thanks to your humble, or sometimes not so humble, systems administrator[s]. We work hard and diligently to ensure that all of our users have access to the resources that they need to get through the day. We're the ones keeping your Internets unclogged, keeping your mail working, making sure that Google doesn't suddenly poof in a puff of clustered-hard-disk failure and keeping your business-critical networking infrastructures working flawlessly.

So, everyone take a minute to thank the person most directly responsible for maintaining your systems. Buy them a nice root beer or a round of pizza - celebrate the guys keeping your business alive in the Internet age!

For me, I want to give a big thank you out to the admins over at Citescape for keeping our Internets up and running smoothly as well as all of the extra support that they've provided over the last year or so. I'd also like to give a nice shout-out to the admins over at ipHouse for keeping their network infrastructure rock-solid allowing us to run our web-app without any issues.

Phantomdata.com, Now on Mongrel!
May 19, 2007

Well, I finally got off my ass and switched this site over to the apperently-now-standard Mongrel backend. Lighttpd is now feeding into Mongrel instead of the old Fast-CGI backend. Now I can enjoy the luxuries of an up to date scalable backend architecture.

The joys.

Ubuntu 7.04 Released!
April 19, 2007

The latest version of Ubuntu has been released into the wild! I'm currently grabbing the torrents off of our lovely fast unfiltered internet connection from Citescape. The joys of testing a new server O/S! I've put both of the torrent files up for download off of my site since Ubuntu's site is slow, but it's just a test - if it hammers my site too much I'll be taking them down.

What's new in this release? The biggest thing for me is built-in virtualization allowing easy testing of O/Ss and major patches before deployment. It will be interesting to see how well this works out. Off to testing I go!

- > Ubuntu 7.04 Desktop (i386)

- > Ubuntu 7.04 Server (i386)