Here's the root of the argument. What we've all been taught regarding database design is irrelevant if the design can't deliver the necessary performance results.

The 3rd normal form helps to ensure that the relationships in your DB reflect reality, that you don't have duplicate data, that the zero to many relationships in your system can accommodate any potential scenario, and that space isn't wasted and reserved for data that isn't explicitly being used. The downside is that a single object within the system may span many tables and, as your dataset grows large, the joins and/or multiple selects required to extract entities from the system begins to impact the system's performance.

By denormalizing, you can compromise and pull some of those relationships back into the parent table. You might decide, for instance, that a user can have only 3 phone numbers, 1 work address, and 1 home address. In doing so, you've met the requirements of the common scenario and removed the need to join to separate address or contact number tables. This isn't an uncommon compromise. Just look at the contacts table in your average cell phone to see it in action.

Jeff writes:

Both solutions have their pros and cons. So let me put the question to you: which is better -- a normalized database, or a denormalized database?

Trick question! The answer is that it doesn't matter! Until you have millions and millions of rows of data, that is. Everything is fast for small n.

So for large n, what's the solution? In my personal experience, you can usually have it both ways.

Design your database to 3NF from the beginning to ensure data integrity and to allow room for growth, additional relationships, and the sanity of future querying and indexing. Only when you find there are performance problems do you need to think about optimizing. Usually this can be accomplished through smarter querying. When it cannot, you derive a denormalized data set from the normalized source. This can be as simple as an extra field in the parent table that derives sort information on inserts, or it can be a full-blown object cache table that's updated from the official source at some regular interval or when an important even occurs.

Read the discussions and share your comments. To me, the big takeaway is that there's no one solution that will fit every real world problem. Ultimately, your final design has to reflect the unique needs of the problem that is being solved.

When Not to Normalize your SQL Database
Maybe Normalizing Isn't Normal

So how do we go about fixing the problem?

Crawl AJAX Like A Human Would
To crawl AJAX, the spider needs to understand more about a page than just its HTML. It needs to be able to understand the structure of the document as well as the Javascript that manipulates it. To be able to investigate the deeper state of an application, the crawling process also needs to be able to recognize and execute events within the document to simulate the paths that might be taken by a real user.

Shreeraj Shah's paper, Crawling Ajax-driven Web 2.0 Applications, does a nice job of describing the "event-driven" approach to web crawling. It's about creating a smarter class of web crawling software which is able to retrieve, execute, and parse dynamic, Javascript-driven DOM content, much like a human would operate a full-featured web browser.

The "protocol-driven" approach does not work when the crawler comes across an Ajax embedded page. This is because all target resources are part of JavaScript code and are embedded in the DOM context. It is important to both understand and trigger this DOM-based activity. In the process, this has lead to another approach called "event-driven" crawling. It has following three key components

1. Javascript analysis and interpretation with linking to Ajax
2. DOM event handling and dispatching
3. Dynamic DOM content extraction

The Necessary Tools
The easiest way to implement an AJAX-enabled, event-driven crawler is to use a modern browser as the underlying platform. There are a couple of tools available, namely Watir and Crowbar, that will allow you to control Firefox or IE from code, allowing you to extract page data after it has processed any Javascript.

Watir is a library that enables browser automation using Ruby. It was originally built for IE, but it's been ported to both Firefox and Safari as well. The Watir API allows you to launch a browser process and then directly extract and click on anchor links from your Ruby application. This application alone makes me want to get more familiar with Ruby.

Crowbar is another interesting tool which uses a headless version of Firefox to render and parse web content. What's cool is that it provides a web server interface to the browser, so you can issue simple GET or POST requests from any language and then scrape the results as needed. This lets you interact with the browser from even simple command line scripts, using curl or wget.

Which tool you use depends on the needs of your crawler. Crowbar has the benefit of being language agnostic and simple to integrate into a traditional crawler design to extract page information that would only be present after a page has completed loading. Watir, on the other hand, gives you deeper, interactive access to the browser, allowing you to trigger subsequent Javascript events. The downside is that the logic behind a crawler that can dig deep into application state is quite a bit more complicated, and with Watir you are tied to Ruby which may or may not be your cup of tea.

Crowbar - server-side headless Firefox
Watir - browser remote control in Ruby
Crawling Ajax-driven Web 2.0 Applications (PDF)

ShmooCon is an annual East coast hacker convention hell-bent on offering three days of an interesting atmosphere for demonstrating technology exploitation, inventive software & hardware solutions, and open discussions of critical infosec issues.

It's a while until the next conference comes up, but there have been some great presentations at past conferences, most of which are available online. Peteris Krumins recently assembled links to all of the videos and presentation files that are available at the Shmoocon site (including the 2008 conference), posting them to his blog as a single big index.

A quick search on YouTube also turned up a series of videos by Scott Moulton from Shmoocon 2007 and 2008 on the topic of data recovery for both traditional hard disks and flash drives. It's pretty fascinating stuff, whether you're interested in this from a forensics or security perspective, or if you've ever just wondered what exactly goes into recovering important data from a crashed disk when you send it out to a data recovery shop.

Hacking Videos from Shmoocon
Scott Moulton's videos on data recovery for SSD flash drives and hard disks
Shmoocon Infosec Conference

See also: Videos from past DEFCONs

drop.io is a web service that solves this sort of problem perfectly. You create a drop URL with a unique name, upload a file to it, and set an expiration time when it will be deleted, all in a single step. The drop folder can have both an access and an admin password, and you can choose what level of access (read, read/write, read/write/delete) the non-admin has. After you've created a drop folder, you can continue to add files and notes to it via the web interface or by email. Each drop also has a phone extension that will allow you to call in and record messages that are added to the drop. It's brilliantly simple.

What I like best is that aside from tracking IP for legal or terms of service violations, it's completely anonymous. You don't make an account to use the service. There is no profile. The drop folders aren't search indexable unless you choose to make them without passwords and publish the URL somewhere crawlable. You can renew the expiration period of the drop, but when it expires, it goes away along with its contents.

I like.

drop.io - Simple Private Exchange

John Resig, of jQuery fame, released a port of the Processing visualization language for Javascript. Seriously, John is on fire:

The first portion of the project was writing a parser to dynamically convert code written in the Processing language, to JavaScript. This involves a lot of gnarly regular expressions chewing up the code, spitting it out in a format that the browser understands.

It works "fairly well" (in that it's able to handle anything that the processing.org web site throws at it) but I'm sure its total scope is limited (until a proper parser is involved). I felt bad about tackling this using regular expressions until I found out that the original Processing code base did it in the same manner (they now use a real parser, naturally).

The full 2D API is implemented, with the exclusion of some features here and there between browsers (Firefox 3 is pretty full featured). You can interact with the Processing API directly from standard Javascript. This lets you make use of these drawing features by simply instantiating a Processing object, and then calling its various drawing methods.

Another capability is to write code natively in the Processing language. This allows you to make use of extended language features such as method overloading and classic inheritance, though it looks like type information is pretty much ignored.

John has many of the demos from processing.org working. Most of them are going to peg your CPU, but this is some seriously cool stuff to see working in a first release.

Javascript just got a lot more interesting.

Processing.js
Processing: open source data visualization language

What if you could do this at the process level? You could kill whatever umpteen-gazillion applications you have running, reboot your computer, and then start your apps back up whenever you like and they would be exactly the way they were when you left them.

There's a Linux application called CryoPID which attempts to do just that.

CryoPID requires no special kernel modifications and operates in user mode, so you don't need to be root. All you do is run the freeze program on a process you own:

freeze /tmp/savestatefile 1234

This will archive the state of process 1234 into a self-executing, compressed file named /tmp/savestatefile. To start it back up, just run the save file:

/tmp/savestatefile

When this is executed, your application will be restored, relinked to any previously-loaded DLLs, and attached to the file descriptors it had open.

You'll run into some problems with network socket connections you had open, and support for X applications is still only experimental, so the useful scenario is a bit limited, but it's a promising concept and could come in quite handy in the command-line world.

CryoPID - A Process Freezer for Linux

Unlike USB2, the Firewire spec allows devices to have full DMA access. By impersonating the appropriate device, a PC can essentially obtain full read/write access to another machine's RAM, just by connecting the two machines with a Firewire cable. Adding to the recent discussion about the insecurities of physical access and Princeton's cold-boot RAM dump demonstration, Adam Boileau released a Linux Firewire utility that will give you immediate Administrator to an XP machine:

It's two years later, and I think anyone who was going to get the message about Firewire has already got it, and anyone who was going to be upset about it has got over it. Besides, according to Microsoft's definition, it never was a Security Vulnerability anyway - screensavers and login prompts are - as Bruce says - about the Feeling of Security. Anyway, today's release day for Winlockpwn, the tool I demoed at Ruxcon for bypassing windows auth, or popping an admin shell at the login window.

...

• Yes, you can read and write main memory over firewire on windows.
• Yes, this means you can completely own any box who's firewire port you can plug into in seconds.
• Yes, it requires physical access. People with physical access win in lots of ways. Sure, this is fast and easy, but it's just one of many.
• Yes, it's a FEATURE, not a bug. It's the Fire in Firewire. Yes, I know this, Microsoft know this. The OHCI-1394 spec knows this. People with firewire ports generally dont.

Adam's tools include a few Python apps that can copy and impersonate Firewire device signatures, dump RAM on a remote machine, bypass Windows authentication, and extract BIOS passwords. It's not exactly comforting, but I've got a new appreciation for Firewire now. This is the sort of access that used to only be possible by creating hardware that physically connects to the PCI bus. Now all you need is a cable and a laptop.

Firewire, DMA & Windows - direct memory access over Firewire - [via] Link

In Linux and on PPC Macs, the root user can access the machine's ram through the /dev/mem device. I'm not sure why this is unavailable on newer Intel Macs—it's a bummer.

In theory, if you're processing some words, spreading sheets, or posting a blog entry and your program crashes, it's likely that the data you were editing will still be in RAM, unharmed, waiting to be allocated to another process. If you immediately dump the entire contents of RAM to disk before starting another large process, chances are good you can find your data again. It's tricky though—writing that RAM to disk requires you start up at least one process, such as dd. It's possible that this new process, or a another process that's currently running, could allocate memory and obliterate your file. You don't really have other options, though, so you might try something like this:

dd if=/dev/mem of=/tmp/ramdump
strings /tmp/ramdump | grep "some text in your file"

I found a post by David Keech where he describes exactly this process. He was able to use it to successfully recover the text from a killed vi session:

I tested this by starting vi and typing in "thisisanabsolutelyuniqueteststring", killing the vi process without saving the file and running the command above immediately with a small modification. Instead of piping the output to a file, I piped it to grep thisisanabsolutelyuniquetest. The grep command found itself, as it always does, but it also found the original string, identified by the rest of the unique string that I didn't include in the grep command. You have to be careful when search through running memory. I now remember having this problem with the Mac all those years ago. Whenever I searched for parts of my brother's letter, I would just end up finding the part of memory that contained the search string.

He also mentions scanning the swap partition, which is also a likely place for your data to be found. It's the same process, but you replace /dev/mem with /dev/hda2 or whatever your swap partition is.

Here's the fun part. Based on what we now know about DRAM holding data even a few seconds of being unpowered, you might even be able to use the method to recover program data after a full system crash and reboot. The swap data will for sure be there, but if you reboot into single user mode without starting up X or any large applications, the possibility exists that unallocated areas of /dev/mem will still contain data from before the reboot.

How to recover your data after a crash - Link
Extracting encryption keys after a cold boot - Link

TrueCrypt 5.0 was released yesterday and OS X has been added to the list of supported operating systems, making it the only open source volume encryption utility that works in Linux, Mac and Windows. It's a really slick utility for creating an AES-256 or Serpent encrypted volume that you can drop sensitive files inside.

You can use TrueCrypt to create an encrypted volume image inside a file, or you can encrypt a whole disk image or partition. The OS X version uses MacFUSE to provide user-mode mounting of the encrypted disk. The main application window, pictured above, gives you a simple interface for creating and mounting encrypted images.

Once an image is mounted, you can use it like a normal hard disk. Unmount the disk and you're left with a file full of random gibberish. FAT is the only filesystem that's available through the interface, but once the disk is mounted, you can reformat it with Disk Utility to use XFS.

There are a couple of things worth noting. In the Windows and Linux versions a special bootloader is available that lets you encrypt your entire system drive. It doesn't look like that option is available in the OS X version. Also, when I tested the latest OS X binary this evening, the "hidden volume" plausible deniability feature wasn't working. Hopefully that will be added in a future release. Until then, TrueCrypt is better suited for storing tax documents and things you wouldn't want visible to a laptop thief, rather than the details of where you've hidden the bodies.

TrueCrypt - [via] Link

Western Digital sells a number of external drives under the MyBook World Edition brand. These are network-based external storage drives that you can connect to remotely from multiple machines. Inside are a couple of drives set up in a mirrored RAID configuration, as well as an embedded computer running Linux.

MakeFan: tipped us off to Martin Hinner's website, which has a lot of details about the software running on the MyBooks, including info for hacking the devices capabilities to do more than what's available out of the box.

This page provides information on how to hack your MyBook World Edition, so as you can improve performance and add new features. MyBook is powered by ARM9 microprocessor, it has 32MB of SDRAM and boots from internal hard drive. The system partition has 2.8GB (only 260 MB is occupied). This means that you have a lot of resources for various improvements.

You can enable SSH on the device without cracking the case. Martin hosts a script that subverts the firmware update software to create your ssl keys and boot the sshd process. Once that is enabled, you have full access to the OS to do what you like, including running an NFS server, web server, or even replacing the standard web interface.

Also worth checking out is the Hacking WD MyBook Wiki. They have links to information on rescuing data from dead drives and building other software for the device. Keep in mind that building MySQL from source will take about 18 hours, but there's got to be something fun you can do with a LAMP stack running on a terabyte hard drive.

Hacking Western Digital MyBook World Edition - Link
MyBook World Edition Wiki - Link

You can use the Quality of Service (QoS) feature on many routers to optimize your internet connection for devices and services that are most important to you. Adam at Lifehacker put together a quick walkthrough for tuning the QoS settings on our favorite router firmware, the open-source DD-WRT.

Your internet connection is an indispensable part of your life, but between BitTorrent, Xbox Live, web browsing, and VoIP, sometimes there's not enough bandwidth to go around. But rather than running around the house shutting down all of your computers next time you're experiencing a little lag on Xbox Live or Skype is breaking up on you, you can set up Quality of Service (QoS) rules on your router to distribute bandwidth to your different gadgets and applications based on your priorities. Today I'll show you how.

The nice thing about setting up QoS on your router is that you can prioritize packets by application, IP and MAC address, or a specific router port, all to your own preferences. Everyone has different needs, so you may want to maximize the available bandwidth for VoIP or network gaming, guarantee a chunk of bandwidth for ssh connections or your web server, or throttle down P2P applications so that they don't affect other network services. There's nothing set in stone, so you can really make your router work for you to improve the way your internet connection is utilized.

Ensure a Fast Internet Connection When You Need It - Link

The thing is, search is the operating system of the web. The fact that we have no open-source/open-data search infrastructure is as bad as if there were no Linux or OpenBSD. If Google, Yahoo and MS weren't providing such a great product, my guess is that the hacker community would be attacking this problem like Captain Kirk on a lizard monster.

Where We Are:

Currently, there are a number of open source projects related to general web search. Most notably, the Java based Lucene project is a solid foundation for indexing and information retrieval, and it's what the Nutch search engine is built on.

There are a few distributed crawlers like Grub and Majestic 12. Unfortunately, these both pass data to a central, private storage system. The hard work of crawling and indexing is open for everyone to participate in, but the resultant data is not.

Where We Need To Be:

In my mind, search hackers need to create an open source solution for the following:

• A distributed mechanism for crawling and indexing the web on a mass scale.
• Distributed, decentralized, redundant data storage for the cache and index.
• An end-user, public facing interface for querying the distributed index.
• A mechanism for retrieving or crawling a local, private slice of the index and cache, for research or personal use.
• A way to publish alternate indexing models to the distributed grid.

All of these tools need to be designed with the assumption that anyone can and will have access to the system's data, and as the system grows, there will be people, corporations, and governments hell-bent on corrupting the search infrastructure to their advantage.

It's not an easy problem to solve, but you've got to admit it's an interesting problem. Anyone keen on being the Torvalds of search?

Where To Begin:

The Lucene Project - Link
Nutch Open Source Search Engine - Link
Open Source Search Wiki - Link

Have I missed anything? Please share your thoughts on open source search in the comments.

Here's an hack that takes on the challenge of burning a visible image into a CD-R.

Data on a CD, or any optical media, is stored as a sequence of pits of varying lengths. To be precise, a 1 is represented by the change from pit to no-pit or the change from no-pit to pit, and a 0 is represented by no change in height (pit to pit or no-pit to no-pit). The pits and no-pits reflect different amounts of light; thus it is possible to draw images on CDs by appropriately arranging these 1s and 0s.

As you can see in the photo above, the results are difficult to make out, mostly due to calibration settings that the author needed to tweak. All of his source is available, though some of it requires Matlab. Anyone care to test this out and link us to a photo in the comments?

What I'd really like to see is something that will take an image and convert it into a wav file that can be placed as the first track on the disk to show an image...

Burning visible images onto CD-Rs with data (beta) - Link

From what I can see--and keep in mind that I haven't used the API yet--there are 3 basic services that the API provides:

• local file resource storage and caching so that you can view files after disconnecting
• a client-side SQL database that can be used by Javascript to store and fetch data
• a worker pool module for running asynchronous background processes

The obvious use for this is to make stateful applications that continue to operate when you're offline, but maybe there are some privacy opportunities here too. Today, applications come in primarily two varieties: apps with user data and software stored locally, and web-based applications that execute and store data on the server. What I'm curious to see is if developers will begin making a third, hybrid category of application, where software release and maintenance is web-based and global data is available for local consumption, but the storage and processing of user-specific data takes place on the client side, safe from unwanted profiling.

Google Gears API Developer's Guide - Link

With reCAPTCHA, the user is given two words, one known by the system and one from a book that previously failed character recognition. When the user enters both words, the sytem verifies the known word, proving human-ness, and submits the second word to a central database, which helps digitze books from the Internet Archive. With 60 million CAPTCHAs being solved every day, this could be a huge assist for portions of text that can't be handled by optical character regognition techniques. [via] Link

Related:
Negative CAPTCHA

The OpenChange MAPI library aims to provide interoperability with an Open Source implementation of Microsoft Exchange protocols under UNIX/Linux. The current implementation offers a client-side library which can be used in existing messaging clients and offer native compatibility with Exchange Servers up to 2003.

As of last weekend it now supports the creation and extraction of not only messages, but also contacts, calender events, and tasks. This should be really useful for developers who want to interoperate with Exchange from other platforms and external mail systems.

Openchange: open source Exchange server alternative - Link.
Feature demonstration screencast - Link.

Eric Cheng recently posted his experiments on getting RAID 0 working with a Macbook Pro:

Subjectively, the system feels more snappy, and I'm happy to have made the upgrade. I use up at least 20-30GB per week in the field, and it will be great to be able to work on my notebook without having an external drive chained to it.

As for the disk failure issue, we all back up nightly anyway, don't we?

320GB striped array (RAID 0) in a Macbook Pro -[via] Link.

Amazon's S3 (Simple Storage Service) isn't new, but its certainly gaining traction. Its a wonderful product for people who have a lot of content on their site (images, video, downloads, pdfs) but not a lot of money. Data storage costs $0.15 per GB-Month (prorated), and $0.20 per GB. No minimums, rounded up to the nearest cent.

...OK so, what's the point and what does this have to do with electronics, eh? Well one of the killer apps of open source and public domain electronics is documentation. That means media. And media storage, backup and transfer is extremely expensive for the everyday person. It becomes increasingly difficult to host a project when one digg-storm or slashdotting makes that 'free' webpage account go down.

S3 makes website hosting reasonable again - Link.

This extension allows you to use your Gmail Space (2.8 GB and growing) for file storage. It acts as an online drive, so you can upload files from your hard drive and access them from every Internet capable system. The interface will make your Gmail account look like a FTP host.

One thing to note is that Gmail accounts sometimes get blocked for 24 hours if excessive amounts of data are being transferred. Because of this, it's recommended that you don't transfer more than 1GB/day to any particular account. There's no reason why you couldn't use a few accounts for different purposes, however.

Gmail Space - Firefox plugin by Rahul Jonna and Tnarik -Link.

Love tag clouds? Check out this page, which displays the top 20 search terms that drive people to each O'Reilly domain, including Hackszine. As noted on that page, here are a few things to keep in mind about these visualizations:

• The terms are organic, which means that these are terms that someone typed into a search engine (e.g., Google) and then followed a resulting link. (In contrast to a search term that someone entered into our own search box.)
  
• While the keyword frequency does give some idea of what people are looking for, keep in mind that the word had to already be on our site in order for it to appear, and it had to be ranked highly enough for someone to find it.
  
• These are raw search terms, so similar but slightly different terms will appear twice. For example, "web 2.0" and "web 2" may both appear.

Manipulation of data, such as SELECT, INSERT, UPDATE and DELETE are not yet supported, so you're still on your own there if you're writing platform agnostic code. That said, this is an incredibly useful tool. Just consider this example that Chris Dolan posted on use Perl:

MySQL understands this syntax:

create table book (
id int,
author_id int,
FOREIGN KEY fk_author_id (author_id) REFERENCES author (id)
) TYPE=InnoDB;

but not this nicer syntax (it silently ignores the "references" clause):

create table book (
id int,
author_id int references author (id),
) TYPE=InnoDB;

Perl to the rescue! I can write my schema in the latter syntax and use SQL::Translator to rewrite into the supported syntax.

References:

• MySQL foreign key syntax translator (4 line perl script!)
• SQL::Translator on CPAN

We've got a really fast connection at work, but I still occasionally run into situations where it's faster, and often more economical, to overnight data on an external hard disk instead of transfering it over the wire. Even within the office, if I'm moving a large file from one machine to another, I've found that good 'ol sneakernet can save me a lot of time, especially when other people are using the network.

Jeff Atwood posted a great article on the economics of bandwidth the other day. He puts some current cost figures towards Jim Gray's 2003 ACM interview, in which Jim describes the efficiencies of packing and shipping a whole computer instead of copying a terabyte of data over the net:

It's cheaper to send the machine. The phone bill, at the rate Microsoft pays, is about $1 per gigabyte sent and about $1 per gigabyte received—about $2,000 per terabyte. It's the same hassle for me whether I send it via the Internet or an overnight package with a computer. I have to copy the files to a server in any case. The extra step is putting the SneakerNet in a cardboard box and slapping a UPS label on it. I have gotten fairly good at that. Tape media is about $3,000 a terabyte. This media, in packaged SneakerNet form, is about $1,500 a terabyte.

According to Jeff's calculations, the effective sneakernet transfer rate for a terabyte of data is about 9.1 MBps at $0.06/GB. Only an OC-3 would be faster, which costs roughly $0.15/GB for both the sending and receiving end. Want to send 2 terabytes of data? Factoring in the extra time to copy to and from the disk, it works out to about 14.6 MBps at about the same cost per GB. Sneakernet scales.

Related:

• Un-unwire your home

A couple weeks ago a flood hit my apartment/office area and soaked the desktop system, monitors, equipment *and* back up drive (along with a ton of other stuff) - luckily I have a daily back up on a Powerbook. But, of course the Powerbook decided to completely stop working while at our ETSY event before that could be backed up too. Zapping the PRAM revealed the hard drive failed, so the usual steps of Disk Util, TechTool and then finally drive removal and DiskWarrior were attempted - for the most part the drive seems completely dead - there might be a chance to recover some data under linux, or from a data recovery shop, but it's not looking good.

According his latest update, the backup drives dried out okay and appear to be working fine, so I guess that means he's managed to survive the perfect storm, but it got me thinking - how many of us ever keep a regular, daily backup in the first place? I've suffered several near-misses in the past, and I'm still guilty of not keeping good backups.

Never Again
So, February isn't too late for a new year's resolution. Don't go another day without your important files backed up. Let's sit down for 15 minutes, right now, and set up an automated backup system for ourselves. All you need is an external hard disk or a remote server with sufficient storage for a couple copies of your data. Based on Phil's story, you might want to situate your backup system on an elevated surface and not beneath any water pipes.

We're not focusing on a perfect backup solution here, with off-site, fire proof, vault storage. Don't let the nay-sayers stop you with the long list of things that can go wrong with a simple back-up solution, or explanations of how to do it the "right way". In 15 minutes you are going to be significantly more protected from data loss, and this will give you the time you need to relax and find a good price on your fire proof vault.