Fighting Spam with Bayesian Filtering
- 2003.02.27
My recent article on spam filtering led
reader Lee Williams to send me this message. - A slight variant on your suggestion is based in AI, and it's
called bayesian classification. Rather than having the user
explain to the machine every keyword it wants to flag, the
bayesian classifier simply assigns probabilities to each (word,
phrase) based on what class you put the email in. Then, when it
sees another email, it is able to use these probabilities to
estimate the class it belongs in. It's actually a lot simpler than
having the enduser manually create these filters.
Paul Graham discusses this in his "plan
for spam".
The key element in both solutions is that they are completely
customizable. The existence of individual filters would not
necessarily eliminate 100% of spam to each person, but in the
aggregate it would mean that each email reaches considerably fewer
people, thereby hopefully making it at least somewhat less
cost-effective.
That article is good reading. I think it would be something I would
definitely try. An email client with such filtering would be a great
hit, in my opinion. [Editor's note: Apple's Mail client in
OS X 10.2 includes a Bayesian filter. I've been using it for
over a month and am still training it. dk] I wonder if a combination of the traditional manual filtering and
this would be even better. The manual filtering would allow you to
set a rule for specific phrase or person (say your ex for example)
which gives a 100% probability of being filtered. One interesting point related to your statement about the email
reaching fewer people - which he doesn't make either - is that if a
user actually wanted to receive these emails - after all, someone
must be responding to them - they'd like this filter, too, because
they'd sort the spam they like into the "keep" folder. Contrary to Graham's hypothesis, spam need not end. It would
become highly targeted at those people most likely to respond to it.
The rest of us would ignore it. If everyone had such filters, the
things you're most interested in would be the only things that would
get through. Everyone would like this type of filtering. Haters of spam,
responders of spam, and senders of spam. Now for an even spookier thought - apply the same technology to a
TiVo with voice recognition and filter your TV commercials the same
way.
Jeff Adkins is a science teacher who isn't afraid to state his preferences in computing platforms. In his classroom he has everything from a beige All-in-One to a a G4 XServe, and they all work together nicely. He calls himself the "poster child for technology integration" in the classroom. He was the 2006 Outstanding Educator of the Year for the California Computer Using Educators (CUE) organization. He also maintains a site for astronomy teachers at www.AstronomyTeacher.com. Recent Mac Lab ReportsLinks for the Day- Mac of the Day: PowerBook 170, Oct. 1991 - At 25 MHz, the PB 170 was at the top of the original PowerBook line.
- List of the Day: The iPhone List Low End Mac's forum for discussing and supporting Apple's iPhone.
- August 30 in LEM history: 99: The truth about USB speed - 00: Could Eazel kill the Mac? - Mac OS 8.1 on a IIci and LC III - 01: Beyond MHz and GHz - Getting a handle on email - Thanks for the IBM PC, Dad - Apple's anniversaries - 02: Mac OS X v10.2 - iBook video out - 04: Things that freak out my students - 06: Nvu and SeaMonkey can't replace Home Page - 07: DVD-RAM support
Recent Content on Low End Mac- Looking for a Content Management System That's as Easy as Mac, Dan Knight, Mac Musings, 08.29.
Low End Mac needs to move to a content management system, but the few we've tried just don't cut it for people used to the simple elegance of the Mac.
- First 3 Million Mac Quarter, Skinny on Mac mini Pricing, Mac-like gOS, and More, Mac News Review, 08.29.
More plan to buy Apple products than ever before, complete reset can fix MobileMac synch problems, Apple boosting computer and smartphone share, and more.
- New 'Books Likely in September, 17" PowerBook Display Fault Site, SSD Security, and More, The 'Book Review, 08.29.
Also 6 ways to speed up your MacBook, next generation MacBook Air CPU, MacBook Air Update, LapStrap carrying solution, rise and fall of ultraportables, bargains from $220 to $2,699, and more.
- iPhone 3G Reception 'Completely Normal', AT&T International Data Plans for iPhone, and More, iNews Review, 08.29.
Also longer life for iPod earbuds, an alternative to MobileMe, new cases and apps for iPhone, AppStoreGems website launched, and more.
- Best Power Mac G4 Deals, Low End Mac Deals, 08.29.
Used 450 MHz AGP, $75; 500, $99; 800 QS, $199; 1.25 GHz MDD, $375; 450 MHz dual, $179; 867 dual, $300; 1 GHz dual, $395; 1.42 dual, $575.
- Best iBook G3 Deals, Low End Mac Deals, 08.29.
Used 300 MHz clamshell, $150; 500 CD, $150; 800, $200; 600 CD-RW, $240; 900 Combo, $300; 14" 600, $360; 900, $400.
- Best iPod nano Deals, Low End Mac Deals, 08.29.
Used 2 GB iPod nano, $89; refurb 3G 4 GB, $99; new, $140; refurb 8 GB, $149; new, $179.
- 10 Mac Browsers Compared, Simon Royal, Mac Spectrum, 08.28.
A look at Internet Explorer, Radon, Opera, Safari, Shiira, iCab, Firefox, Netscape Navigator, Flock, and Camino running in Leopard.
- Clone and Boot: Another Advantage of the Mac OS, Kev Kitchens, Kitchens Sync, 08.28.
Unlike Windows, Apple makes it possible to clone a bootable drive (Classic Mac OS or OS X) and use it with another supported Mac.
- Best MacBook Deals, Low End Mac Deals, 08.28.
Used 1.83 GHz, $799; 2.0 black, $875; refurb 2.1 GHz, $899; 2.4, $1,099; black, $1,299; new 2.1, $1,019 after rebate; 22, $1,094; 2.4, $1,219 a/r; black, $1,394 a/r.
- Best iMac G5 Deals, Low End Mac Deals, 08.28.
Used 17" 1.6 GHz Combo, $499; 1.8 SuperDrive, $530; 2.0, $600; 1.9 iSight, $625; 20" 1.8 GHz, $580; 2.0, $650; 2.1 iSight, $700.
- Best classic Mac OS Deals, Low End Mac Deals, 08.28.
System 6, $10; 7.1, $12; 7.5.1, $4; Mac OS 7.6, $13; 8.0, $13; 8.1, $48; 8.5, $25; 8.6, $20; 9.0, $20; 9.2.2, $20; more.
- CrossOver Strikes Out, Frank Fox, Stop the Noiz, 08.27.
Running Windows apps on a Mac without paying for Windows is great in theory, but actually getting Windows software working is another story.
- MacDrought: 4 Months with No New Macs, Dan Knight, Mac Musings, 08.27.
The most recent Mac update was over four months ago, and the Mac mini has been unchanged for over a year.
- Resurrecting a Dead Pismo, Spotlight Search Tip, and EasyFind a Good File Finder, Charles W. Moore, Miscellaneous Ramblings, 08.27.
Lots of tips on bringing a comatose Pismo back to life, a Spotlight file name search tip, and EasyFind as an alternative to Spotlight.
- Best Intel iMac Deals, Low End Mac Deals, 08.27.
Used 17" 1.83 GHz, $625; 20", $599; 2.16, $749; 24", $950; refurb 20" 2.4, $999; 2.66, $1,299; 24" 2.4, $1,299; 2.8, $1,549; new 3.06, $2,094 after rebate; more.
- Best 15" PowerBook G4 Deals, Low End Mac Deals, 08.27.
Used 1.25 GHz Combo, $600; SuperDrive, $650; 1.33 Combo, $640; 1.5, $680; SD, $725; 1.67, $730; hi-res, $800.
- Best Time Capsule and AirPort Deals, Low End Mac Deals, 08.27.
500 GB Time Capsule, $294; 1 TB, $468; AirPort Extreme Card, $39; 802.11n Base Station, $166; 802.11g AirPort Express, $60; 802.11n, $98.
- More links in our archive.
|
