INTERNET! MWAHAHAHAHAAH!
June 17, 2008

Oh my god, I finally have Internet. Citescape was scheduled to come out on Tuesday, but it was raining and I never heard back after the cancellation. Alas, I gave in and on Friday decided to place a call to Charter. So, here I sit with an Internet connection that's 10~100 (20kbp/s was not unheard of) times as fast as what I was getting out at the farm. Yum yum. It is so nice to finally be surfing the Internets,that I don't know what to do with myself. Google reader, I'll come read some of your excess of 3billion unread items I have!

Twitter Vs. Pownce
February 27, 2008

As those of you who follow the great ebbings of the Internet know, public instant messaging like Twitter and Pownce has really taken off as of late and has brought a very novel idea into this done-to-death space. As far as my tiny attention span goes, there's two competitors vying to have my lovely thoughts toiled down into them - Pownce and Twitter.

David got me turned on to Pownce a few months back, and I really started getting interested in it once it went public. It's a very beautifully designed application with a focus more on internal social networks (friends) rather than public messages like Twitter. Alas, this is its downfall. All of its beauty can't save it from one crushing factor - there is nobody using it (Yes David, I know you have an account).

Consequently, I've more or less started using Twitter and am having a gay old time wishing the Internet a good morning every day. So, if you find yourself desiring to follow me on one of these new fangled public IM services - you'll find me on Twitter.

<SFII: Turbo Announcer Voice>Twitter Wins.</SFII: Turbo Announcer Voice>

Moving Sites to Joyent
September 27, 2007

The recent outage has prompted me to go ahead and bite the bullet. My sites are now being migrated to Joyent's Servers. I figure that any host whose blog contains more succulent tech info than you can shake a stick at is definitely worth a check.

Please bare with me while this is happening. My main site has been migrated, and I should be working on Caturday tonight and tomorrow-nights. Hopefully I'll get in gear and start making some more changes to the Caturday page since I'll be able to host more user additions.

Btw, migrating psql databases is a PITA. pg_dump -Fc is your friend, and be mindful of your encoding. UTF-8 or die trying to use iconv.

CitEscape Back and Rocking
September 11, 2007

As you guys may have noticed, my Internet connection has stabilized a great deal over the last few weeks. It seems like whatever damage that was done by the storm has availed itself. I would terribly hate to be in CitEscape's shoes when the storms come blowing through, since all of their equipment's so high in the sky - but damn it's nice when it's back. I was incredibly thankful, when the Quake Wars demo was released yesterday, that everything was working and I was using such a kick-ass ISP. I kicked on the .torrent download before leaving for class, and everything was ready to go when I got back. I still can't get over having 50ms ping times with my Internets flying many miles over the free air and then getting routed all the way back to St. Cloud.

Anyway, it's back and I'm glad that I stuck it out.

Citescape Service Issues
August 29, 2007

Sooo... the Internet was down again today. I woke up this morning to a lovely whole-lotta-red on my stability graph. Maybe I will have to sign up for Joyent hosting for my main sites.

Up... down... up...
August 16, 2007

Up and down... the storm this weekend caused some major issues at my ISP... I'm up and down ATM...

Update - 2007/08/17

Blugh... so the storm that I'm referencing above was on the 10th of August. Below are some samples of the current status of my Internet connection through Citescape. Packet loss should be zero, not nearing 100% at times. The host I'm measuring latency against gets 70ms from Shadow. Ouch. Looks like I won't be downloading Bioshock from Steam.

And wow... my redirect rules are working nicely. Since switching my hosts over to port 80 on Thursday I've gotten 88 individual visitors. Nice!

Better Spam Filtering
July 23, 2007

I'm continually on the holy quest for zero-spam in my inbox. Alas, spam filtering techniques age and die quickly on this great Internets upon which we surf. Back in the good old days I got by with just Bogofilter's Bayesian spam learning and was amazed at the spam reduction. However, times have changed and spam is getting increasingly difficult to recognize through Bayesian analysis. I've started using a combination of SpamAssassin, Razor, Pyzor, DCC, TextCat, Bogofilter and SPF to get this accomplished.

It's really pretty simple to get up and running on an Ubuntu box. In this tutorial I will assume that you've got some administration history under your belt. This is a quick and dirty run-down of how to get these services configured.

User Setup

For our users' directories, we will have to run the following commands to setup the directory tree:

maildirmake /home/USERNAME/.maildir
maildirmake /home/USERNAME/.maildir/.Spam

This will create the root mail directory as well as a subdirectory for Spam storage.

Installing SpamAssassin and Bogofilter

Before we go any further, we should start installing SpamAssassin and Bogofilter. What good is configuring non-existent software?

apt-get install spamassassin bogofilter

Then you'll need access to all of the other associated joys for this packaging:

apt-get install razor pyzor dcc-client libspf2-2

Loe, you have Spamassasin fully installed. It'll just require a little finagling in the conf files (/etc/spamassassin/) to get an acceptable setup.

In /etc/spamassassin/local.cf I find the default required "spamicity" level to be entirely too low. I like a required_score in between 2.5 and 4. If you go any higher, then you'll never get spam filtered well. For TextCat to work nicely, we'll also throw in the following:

ok_languages en
ok_locales en

This will ensure that none of that Russian spam gets through. Obviously omit this if you are Russian.

v310.pre

After making the changes listed above, you'll also have to edit v310.pre to make sure that it uses some of our more advanced filtering techniques. Make sure that the following lines are uncommented in your v310.pre.

loadplugin Mail::SpamAssassin::Plugin::DCC
loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::Razor2
loadplugin Mail::SpamAssassin::Plugin::SpamCop
loadplugin Mail::SpamAssassin::Plugin::TextCat

/etc/procmailrc

That'll do it for SpamAssassin setup. Now, we get to move on to installing and configuring procmail to our liking. Here's an example config that I like.

LOGFILE=/var/log/procmail
MAILDIR=$HOME/.maildir/
DEFAULT=$HOME/.maildir/
SPAMDIR=$MAILDIR/.Spam/new

:0fw* < 256000
| /usr/local/bin/spamassassin

:0:* ^X-Spam-Status: Yes
$SPAMDIR


:0:
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
$SPAMDIR



:0fw
| bogofilter -e -p
:0e
{ EXITCODE=75 HOST }
# file the mail to spam-bogofilter if it's spam.
:0:* ^X-Bogosity: Yes, tests=bogofilter
$SPAMDIR

~./fetchmailrc

Now we move on to the individual users' fetchmailrc's. Put this directly in their home directory and edit accordingly.

set logfile /home/LOCALUSER/.maildir/fetchmail.log
set no bouncemail

poll MAILSERVER
  protocol pop3
  username "USERNAME"
  password "PASSWORD"
  fetchall
  expunge 5
  pass8bits
  is LOCALUSERNAME
  mda "/usr/bin/procmail -d %s"

learn_spam.rb

So, you've got bogofilter installed from way back at the beginning of this tutorial but you're wondering how to use it. Well, bogofilter has to be trained based on existing spam and ham corpuses. The way we'll be setting up our mailboxes is to have everything stored in a root maildir. Your Spam will be both placed in and learned from a folder called, obviously, Spam. The following script should be ran nightly by every used on the system. It's not terribly efficient, but for small users-bases its perfectly fine. Apologies for the poor formatting, I'll look at this later.

require 'find'

$bogofilter = "/usr/local/bin/bogofilter"
$username = `whoami`
$username.gsub! "\n", ""
$maildir = "/home/#{$username}/.maildir/"
$spamdir = "#{$maildir}/.Spam"

class BogoFilter
  def check_paths
    [$maildir,$spamdir].each do |dir|
      raise "Could not locate #{dir}!" unless File.exists? dir
    end



  end


  def clear_database
    puts `rm -r /home/#{$username}/.bogofilter`
  end


  def learn_spam
    Find.find($spamdir) do |spam|
      next unless File.file? spam
      `cat \"#{spam}\" | bogofilter -s`
    end



  end


  def learn_ham
    Find.find($maildir) do |ham|
      next if ham.include? "Spam"
      next unless File.file? ham
      `cat \"#{ham}\" | bogofilter -n`
    end



  end


end
bf = BogoFilter.new
bf.check_paths
bf.clear_database
puts "Learning spam..."
bf.learn_spam
puts "Learning ham..."
bf.learn_ham

Fin.

Well, that should do it for you. I'll update this article with more information as I get time. When testing your setup, be sure to use "fetchmail -k" to force fetchmail to not delete messages after fetching - just to ensure that the messages are delivered correctly and not shoved off into the ether. If you have any suggestions, please leave them in the comments section and I'll be sure to include them here.

Three Weeks with Citescape
July 20, 2007

So, its been three weeks since Citescape came out and hooked me up with their Internet service. How's it been? I've had enough time to kick in the tires and I have to say that its been awesome. There was a small hiccup right after I got signed up, but everything else has been leaps and bounds above Charter. I've got better ping times and much better download speeds. What's the best part? It's INTERNET service, not WWW service. There is no "optimization" that only work on HTTP, there is no relevant port blockage and best of all there's no P2P filtering. I can't tell you how nice it is to be able to download the latest anime, demos and open source projects at a normal speed.

What's the coolest part? I've got super high speeds while living in the boondocks. Rock on.

Quick Note
July 10, 2007

Phew. The Internets are back. Citescape was having some issues over the weekend, but everything appears to be in working order now. I'm up to my ears in stuff to do, so I'll be coming back with posts soon.

Zeldman On HTML Email
June 08, 2007

So, Mr. Zeldman recently wrote about how HTML e-mail sucks - and I totally agree with him. In my day to day work life, its amazing how many "professionals" think that 19Pt. Comic Sans colored bright pink is acceptable. My favorite quote?

"Designed" e-mail is just a slightly more polished version of those messages your uncle sends you. Your uncle thinks 18pt bright red Comic Sans looks great, so he sends e-mail messages formatted that way. You cluck your tongue, or sigh, or run de-formatting scripts on every message you receive from him. When your uncle is the "designer," you "get" why styled mail sucks. It sucks just as much when you design it, even if it looks better than your uncle's work in the two e-mail programs that support it correctly.

So perfect. Now, you may view my epileptic poodle of bad design. Bonus points for including it in an HTML signature back to one of these "designers".

Charter Problems
February 23, 2007

You can blame today's downtime on Charter. I think that their goal is to provide a day's worth of downtime once a month - where at least twenty subscribers must call Malaysia in order to reset their modems fifty times and get a tech sent out.

Citescape Updates
January 16, 2007

Well, it seems that I write about failed products or services here just a little bit too much. I've noticed a lot of incoming traffic here looking for info on Citescape. My last few postings on Citescape have been about problems with our services. I wanted to give a few updates on how our service has been running here at $WeMakeLegalResearchSoftware to better reflect the service that we've been getting.

Long story short, it's been going perfectly. The speed is phenomenal, the price is right and nothing's filtered. That's right, you can grab the latest Linux ISOs and large open source apps from BitTorrent without worry that you're going to have a huge hit in performance.

The stability is amazing compared to my experiences with Charter. Our last notable issue was back in November, and we haven't had so much as a hiccup since then. So, if you're looking for a top notch ISP here in the St. Cloud, Minnesota (MN) area - be sure to give them a try!

David: I want to hear about how gaming works over their connection. I get really great ICMP ping times here, but I've obviously never had the chance to try gaming with this link.

Even More Fun With Charter
November 16, 2006

It is now my goal to document every extensive downtime that Charter has wrought upon my poor little soul. Yesterday around 03:00, my box at home got knocked off the net. "Oh noes!", says I, "What have I done to bring this upon myself?". When I get home around 05:00 - I find that indeed, there are no internets coming from my tubes! I've got Charter in my phone contact list now (Yes, it's that bad) - and dialed them right up.

After sitting through a hold queue, I get dumped to an automated "help system" that tells me to power-cycle and all that good stuff ("What is the nature of your problem?", "No cable signal.", "I'm sorry? What? If you're having internet problems, please say; 'Internet Problems'"). I waited on hold for a robot, cool. I mash the zero key repeatedly and it transfers me to a real hold queue. I wait awhile and then get Audrey from Rochester. It takes thirty minutes to run through her script before she finally runs the damned modem diagnosis and - HOLY SHIT, my modem aint on their network. You'd think that it cost them 50$ each time.

After that, she tells me the next available appointment is a week from that day. "Thank you for choosing Charter." - I wish I had a choice.

I hang up with her and call back completely dis-satisfied and perfectly happy to schedule another visit just to annoy them. It's their techs who did something wrong anyways, it always is. I think that they hire out of the High School Football Team(tm). I go through the scripts, play all the right lines... then my cable LED blinks on. What? Here it comes again! This time, it stays on. I'm on the phone with the Op and tell her to cancel my prior request.

I hang up, go downstairs to the equipment room... sure enough there's two bumbling Charter techs. They had techs on site, and were perfectly willing to schedule an appointment for a week from today. Cool. Thanks St. Cloud City Counsel; Monopolies rock!

Series of Tubes
October 02, 2006

So, as I was driving to work this morning on the series of tubes we call a traffic system... I got to imagining my car as a little packet traveling along the internet. Yes, network theory has pervaded my life.

I also got to wondering... why is Ted Stevens wrong about the internet being a series of tubes? Each segment of the network can easily be likened to a tube... much as it is physical copper carrying an electric current whose model is very much like that of the flow of water through a pipe. These "tubes" connect to routers that then send the packets on their merry little way. Now, the routers may honor Quality of Service or Type of Service entries... they may find more efficient routes and what not for the packet to travel over... but in the end, the packet is traveling through "tubes" with fancy routing pumps attached throughout.

Thoughts?

The Internet is Falling!
September 28, 2006

So, we've been experiencing a lot of high packet loss and latency in the St. Cloud area. I've heard rumors of a dying AT&T router somewhere losing packets, but haven't seen anything definitive. ATT locks their hardware down anyway, so even pings don't work at them. Has anyone else out there on the great intarweb seen these kinds of issues?

CitEscape Update
August 21, 2006

Well, I wanted to give an update on the whole situation with Citescape where I work. After a lot of troubles with the installation itself, and a few hiccups during the first month or so of service everything has leveled off. The service is fast, responsive, and reliable. In the last few weeks we've had zero issues with the service. No hiccups, no bad domain servers or any internal routing issues. Flawless.

I also wanted to point out the joys of the lack of filtering. We've currently got a full stack of services running that let us do what we need to do, with no fussing whatsoever. Charter's currently got everything blocked on our business line, even port 80. I really don't know why in the hell you would pay for a business account that did nothing but charge you more money than the residential one.

I digress.