Standford's Stanford's School of Engineering has released a number of Computer Science and Electrical Engineering courses online, in their entirety, called Standford Engineering Everywhere. The online program includes all course materials—notes, tests, and complete lecture recordings—free for students or educators to use under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

SEE users may pick and choose the materials that best meet their needs and interests. Want a refresher course on a particular programming concept? View a video lecture that covers the basics. Are you a programming novice? Spend several weeks viewing lectures, reading course materials and tackling class assignments. Test your knowledge by taking quizzes and exams.

As an example, here's the first lecture in the Machine Learning course, taught by Professor Andrew Ng:

The ten courses that are available cover a healthy range of topics. It's basically 3 or 4 semesters worth of EE and Comp Sci. education that you can brainload for free. Here's the current selection:

Introduction to Computer Science

• Programming Methodology
• Programming Abstractions
• Programming Paradigms

Artificial Intelligence

• Introduction to Robotics
• Natural Language Processing
• Machine Learning

Linear Systems and Optimization

• The Fourier Transform and its Applications
• Introduction to Linear Dynamical Systems
• Convex Optimization I
• Convex Optimization II

If you've ever wanted to go to Standford, but didn't have the time, money, or grades, here's your chance.

Stanford Engineering Everywhere [via Creative Commons]

Previously
Lecturefox: free university lectures
Bootstrap Education

MapReduce is a method for writing software that can be parallelized across thousands of machines to process enormous amounts of data. For instance, let's say you want to count the number of referrals, by domain, in all the world's Apache server logs. Here's the gist of how you'd do it:

1. Get all the world to upload their server logs to your gigantor distributed file system. You might automate and approximate this by having every web administrator add some javascript code to their site that causes their visitor's browsers to ping your own server, resulting in one giant log file of all the world's server logs. Your filesystem of choice is HDFS, the Hadoop Distributed Filesystem, which handles partitioning and replicating this enormous file between all of your cluster nodes.
2. Split the world's largest log file into tiny pieces, and have your thousands of cluster machines parse the pieces, looking for referrers. This is the "Map" phase. Each chunk is processed and the referrers found in that chunk are output back to the system, which stores the output keyed by the referrer hostname. The chunk assignments are optimized so that the cluster nodes will process chunks of data that happen to be stored on their local fragment of the distributed file system.
3. Finally, all the outputs from the Map phase are collated. This is called the "Reduce" phase. The cluster nodes are assigned a hostname key that was created during the Map phase. All of the outputs for that key are read in by the node and counted. The node then outputs a single result which is the domain name of the referrer, and the total number of referrals that were produced from that referrer. This is done hundreds of thousands of times, once for each referrer domain, and distributed across the thousands of cluster nodes.

At the end of this hypothetical MapReduce job, you're left with a concise list of each domain that's referred traffic, and a count of how many referrals it's given. What's cool about Hadoop and MapReduce is that it makes writing distributed applications like this surprisingly simple. The two functions to perform the example referrer parsing might only be about 20 lines of code. Hadoop takes care of the immense challenges of distributed storage and processing, letting you focus on your specific task.

Since Hadoop is written in Java, the natural way for you to create distributed jobs is to encapsulate your Map and Reduce functions into a java class. If you're not a Java junkie, though, don't worry, there's a job wrapper called HadoopStreaming which can communicate with any program you write with the usual STDIN and STDOUT. This lets you write your distributed job in Perl, Python or even a shell script! You create two programs, one for the mapper and one for the reducer, and HadoopStreaming handles uploading them to all of the cluster nodes and passing data to and from your programs.

If you want to play around with this, I really recommend a couple of howtos written by German hacker Michael G. Noll. He put together a walkthrough for getting Hadoop up and running on Ubuntu, and also a nice introduction to writing a MapReduce program using HadoopStreaming (with Python as an example).

Are any Hackszine readers using Hadoop? Let us know what you're doing and point us to more information in the comments.

Hadoop
Running Hadoop On Ubuntu Linux
Writing An Hadoop MapReduce Program In Python

Hackszine reader Math Campbell writes:

As promised when they released their demo application 280 Slides, 280 North, the startup that invented a whole new language (Objective-J) to run their Cocoa-like Javascript framework, Cappuccino on, has released both Objective-J and Cappuccino as open-source under the LGPL. They're also providing documentation, tutorials and forums to help you master this new and exciting way of writing web-apps.

This project came to my attention in June when 280 North released their web-based, Powerpoint-like presentation application, 280 Slides. The team has re-implemented a significant portion of the Cocoa API in Objective-J, so developers who are familiar with writing applications for Cocoa or GNUstep can easily port over their skill set, and possibly their applications, to the web.

Now I've got both iPhone and Cappuccino development giving me a reason to start kicking around a common development platform for web and mobile applications. Have any of you Hackszine readers started playing with Objective-J/Cappuccino? If so, what's been your experience so far?

Cappuccino: open source web application framework
Cappuccino tutorials

While I'm waiting for the Mac version of Google's new web browser, wondering what complications this has in store for me as a web developer, I couldn't help but notice how many non-hackers I've bumped into that could speak to the merits of processes versus threads or describe the benefits of Chrome's garbage collection model or security architecture. Two days ago, most of these folks wouldn't know a thread from the underside of their denims, but Scott McCloud's comic changed that.

So why is this a hack? Well, the strip that announced Chrome's release nicely bridged the nerd gap and managed to communicate some fairly technical content to a non-technical, though otherwise savvy potential user base. It's not easy to get people's attention when talking about memory management. It's difficult to communicate an esoteric architecture decision, and to both explain the decision and demonstrate its importance while not boring your audience is even more challenging.

I'm not saying comics are the way to go for all future technical documentation, but there's something to be learned here in terms of expanding your audience without dumbing down the content.

Google Chrome - Behind the Open Source Browser Project

Also worth noting: if you want to participate in the development of Chromium (the platform behind Google Chrome) you can download the source and communicate in the forums on the Chromium project page.

Wii Physics is a clever little homebrew app. You use the Wiimote to rotate, size and place objects on a stage. Pulleys, ropes, gears and joints can be used to connect objects together, and when you press the play button, a 2D physics system is turned on, causing the objects to fall and interact with each other.

You can download this for free and run it from the Homebrew Channel. If you're ambitious, you can also download the source, add new features, or base a new game off of it. It's written using libwiisprite, a library you'll want to check out if you're thinking of doing any 2D game dev for the Wii.

Wii Physics
libwiisprite

While the standard gcc-compiled version of the application nets out at 3998 bytes, Brian discovered that the smallest possible size for an ELF executable that will still run correctly is 45 bytes:

This forty-five-byte file is less than one-eighth the size of the smallest ELF executable we could create using the standard tools, and is less than one-fiftieth the size of the smallest file we could create using pure C code. We have stripped everything out of the file that we could, and put to dual purpose most of what we couldn't.

Of course, half of the values in this file violate some part of the ELF standard, and it's a wonder than Linux will even consent to sneeze on it, much less give it a process ID. This is not the sort of program to which one would normally be willing to confess authorship.

A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux

Scrum is a collection of tools for agile software development and project management. It helps to focus small software development teams into delivering a complete, tested, and quality product by breaking the development into small iterative chunks with a concrete output. Scrum doesn't necessarily help a team produce code faster, but it allows a team to find out very early in the process whether the development goals will be completed in the planned timeline.

Ken Schwaber, one of the developers of the Agile process and a major Scrum evangelist, presented Scrum to a group at Google. This video is a must watch for anyone involved in software development, whether you're a programmer or a project manager. In addition to introducing the management concept, he gives some insight into what makes a project fail (and how to know right away when it's going to), how to deliver on a tightened deadline without sacrificing quality (hint: half the features, not half baked), and why "core" functionality in a product tends to become unmanageable after 5 major development initiatives.

Google Tech Talks: Scrum et al.

In a typical environment, a web developer might employ a combination of process level caching and the built-in MySQL query caching to eke out that extra bit of performance from an application. The problem is that in-process caching is limited to the web process running on a single server. In a load-balanced configuration, each server is maintaining its own cache, limiting the efficiency and available size of the cache. Similarly, MySQL's query cache is limited to the server that the MySQL process is running on. The query cache is also limited in that it can only cache row results. With memcached you can set up a number cache servers which can store any type of serialized object and this data can be shared by all of the loadbalanced web servers. Cool, no?

To set up a memcached server, you simple download the daemon and run it with a few parameters. From the memcached web site:

First, you start up the memcached daemon on as many spare machines as you have. The daemon has no configuration file, just a few command line options, only 3 or 4 of which you'll likely use:

# ./memcached -d -m 2048 -l 10.0.0.40 -p 11211

This starts memcached up as a daemon, using 2GB of memory, and listening on IP 10.0.0.40, port 11211. Because a 32-bit process can only address 4GB of virtual memory (usually significantly less, depending on your operating system), if you have a 32-bit server with 4-64GB of memory using PAE you can just run multiple processes on the machine, each using 2 or 3GB of memory.

It's about as simple as it gets. There's no real configuration. No authentication. It's just a gigantor hash table. Obviously, you'd set this up on a private, non-addressable network. From there, the work of querying and updating the cache is completely up to the application designer. You are afforded the basic functions of set, get, and delete. Here's a simple example in PHP:

$memcache = new Memcache; $memcache->addServer('10.0.0.40', 11211); $memcache->addServer('10.0.0.41', 11211);

$value= "Data to cache";

$memcache->set('thekey', $value, 60);
echo "Caching for 60 seconds: $value <br>\n";

$retrieved = $memcache->get('thekey');
echo "Retrieved: $retrieved <br>\n";

The PHP library takes care of the dirty work of serializing any value you pass to the cache, so you can send and retrieve arrays or even complete data objects.

In your application's data layer, instead of immediately hitting the database, you can now query memcached first. If the item is found, there's no need to hit the database and assemble the data object. If the key is not found, you select the relevant data from the database and store the derived object in the cache. Similarly, you update the cache whenever your data object is altered and updated in the database. Assuming your API is structured well, only a few edits need to be made to dramatically alter the scalability and performance of your application.

I've linked to a few resources below where you can find more information on using memcached in your application. In addition to the documentation on the memcached web site, Todd Hoff has compiled a list of articles on memcached and summarized several memcached performance techniques. It's a pretty versatile tool. For those of you who've used memcached, give us a holler in the comments and share your tips and tricks.

Memcached
Strategies for Using Memcached and MySQL Better Together
Memcached and MySQL tutorial (PDF)

When I read the article on Slashdot, the "all name servers should be patched as soon as possible" quote dropped a bit of scare on me too. What about my sad little DNS server? I envisioned spending an evening working through a time consuming process of patching and reconfiguring things that I haven't had to touch in years. Much to my pleasant surprise, djbdns, D. J. Bernstein's DNS server, was not vulnerable. My decision to use djbdns a number of years ago was primarily due to his vocal philosophy of engineering security by design instead of by response.

Bruce Schneier's analysis of things is spot on as usual. It's a solid case study for hygienic software engineering practices and the design of secure systems.

The real lesson is that the patch treadmill doesn't work, and it hasn't for years. This cycle of finding security holes and rushing to patch them before the bad guys exploit those vulnerabilities is expensive, inefficient and incomplete. We need to design security into our systems right from the beginning. We need assurance. We need security engineers involved in system design. This process won't prevent every vulnerability, but it's much more secure -- and cheaper -- than the patch treadmill we're all on now.

What a security engineer brings to the problem is a particular mindset. He thinks about systems from a security perspective. It's not that he discovers all possible attacks before the bad guys do; it's more that he anticipates potential types of attacks, and defends against them even if he doesn't know their details. I see this all the time in good cryptographic designs. It's over-engineering based on intuition, but if the security engineer has good intuition, it generally works.

Kaminsky's vulnerability is a perfect example of this. Years ago, cryptographer Daniel J. Bernstein looked at DNS security and decided that Source Port Randomization was a smart design choice. That's exactly the work-around being rolled out now following Kaminsky's discovery. Bernstein didn't discover Kaminsky's attack; instead, he saw a general class of attacks and realized that this enhancement could protect against them. Consequently, the DNS program he wrote in 2000, djbdns, doesn't need to be patched; it's already immune to Kaminsky's attack.

The djbdns server wasn't pre-installed on the Linux distro I based my poor old server on. DJB's deamontools package, which manages the startup and shutdown of the service, was annoying to deal with when every other application just uses a normal init rc script. The dns server configuration and setup was also unfamiliar to me, having previously only worked with BIND zone files.

There's one other thing that has really been different with djbdns than any other DNS server I've ever administered: I've never had to patch it. I've only had one other software experience like this, with the qmail mail transfer system. Qmail is also designed by Bernstein. Hmm.

If you're upgrading your DNS server anyway, maybe now is the time to start thinking about your alternatives.

Daniel J. Bernstein's djbdns server
Schneier - The DNS Vulnerability
DJB on DNS forgery
Slashdot - Kaminsky's DNS Attack Disclosed, Then Pulled

Dan Guido from the Information Systems and Internet Security Lab at the Polytechnic Institute of NYU wrote in about the Institute's 5th annual Cyber Security Awareness Week. If you're in high-school or a college undergraduate program, this is a great opportunity to test your infosec skills against your peers, and hopefully earn a little prize money in the process.

ISIS Lab is organizing NYU-Poly's 5th annual Cyber Security Awareness Week (CSAW) where students can compete and win prizes in a variety of information security challenges. There will be door prizes, raffles for participating, and bonus prizes for undergrad and high school participants. Qualified finalists will receive a travel scholarship to attend the awards ceremony in New York City.

There are a number of events, including an application security "capture the flag" challenge, a security quiz which covers everything from cryptography to risk management, and a 5-day forensics puzzle. There's even an embedded systems challenge where teams are tasked with trying to find hardware and software bugs in a mock control system.

This looks like a lot of fun. Some of the contest materials become available at the beginning of September, so sign up soon if you're interested in participating.

Cyber Security Awareness Week 2008

Jay Pipes, MySQL employee and co-author Pro MySQL, gave a great presentation to Google employees which covers a number of techniques for tuning performance on MySQL. His examples include debugging and analyzing problems as well as best practices for table and index design, query and join operations, and server variable adjustments.

It's a little over 40 minutes long, but incredibly informative, whether you're a casual querier or a power MySQL user. Though some of this stuff is MySQL (or MyISAM or InnoDB) specific, the majority of the content is essential material for the average database application developer.

If you don't have time to sit through it (shame on you) or you're looking to jump right to a specific topic, there's a nice time-coded dissection of the talk over at Peteris Krumins' blog. There's something so appropriate about adding a search index to a video about MySQL optimization.

Performance Tuning Best Practices for MySQL
Video Index

Here's the root of the argument. What we've all been taught regarding database design is irrelevant if the design can't deliver the necessary performance results.

The 3rd normal form helps to ensure that the relationships in your DB reflect reality, that you don't have duplicate data, that the zero to many relationships in your system can accommodate any potential scenario, and that space isn't wasted and reserved for data that isn't explicitly being used. The downside is that a single object within the system may span many tables and, as your dataset grows large, the joins and/or multiple selects required to extract entities from the system begins to impact the system's performance.

By denormalizing, you can compromise and pull some of those relationships back into the parent table. You might decide, for instance, that a user can have only 3 phone numbers, 1 work address, and 1 home address. In doing so, you've met the requirements of the common scenario and removed the need to join to separate address or contact number tables. This isn't an uncommon compromise. Just look at the contacts table in your average cell phone to see it in action.

Jeff writes:

Both solutions have their pros and cons. So let me put the question to you: which is better -- a normalized database, or a denormalized database?

Trick question! The answer is that it doesn't matter! Until you have millions and millions of rows of data, that is. Everything is fast for small n.

So for large n, what's the solution? In my personal experience, you can usually have it both ways.

Design your database to 3NF from the beginning to ensure data integrity and to allow room for growth, additional relationships, and the sanity of future querying and indexing. Only when you find there are performance problems do you need to think about optimizing. Usually this can be accomplished through smarter querying. When it cannot, you derive a denormalized data set from the normalized source. This can be as simple as an extra field in the parent table that derives sort information on inserts, or it can be a full-blown object cache table that's updated from the official source at some regular interval or when an important even occurs.

Read the discussions and share your comments. To me, the big takeaway is that there's no one solution that will fit every real world problem. Ultimately, your final design has to reflect the unique needs of the problem that is being solved.

When Not to Normalize your SQL Database
Maybe Normalizing Isn't Normal

László Kozma brought up a problem that tends to crop up in a number of scenarios: if you move the cursor back to correct or change part of the entry string, traditional autocomplete completely ignores any context to the right of the cursor. In some applications, this means everything to the right of the cursor is ignored and overwritten instead of being part of the search. In Safari, if you type anything between "www." and ".com", autocomplete fails entirely, offering no results unless you clear out everything to the right of the cursor.

One solution, which László termed "reverse autocomplete", is to split the string at the cursor position and attempt to find matches for both anything after the first half and before the second half. Any matches that show up in both sets are the final autocomplete suggestions. The result is that if I type in "www.h|.com" (where my cursor position is represented by the "|" character) the smarter autocomplete might return "www.hackszine.com" as a suggestion but omit from telling me about "www.h-is-aitch.org".

You can take it one step further and also match against the beginning and ends of the entire string. This solves a really common problem that I run into regularly when searching through a large contact list, say the typical corporate email system. If you don't know the correct spelling of someone's name, or if you only can recall initials, you can fill in the parts you know. The system would be smart enough to turn "J|S" into "Jason Striegel" instead of forcing you to page through a huge list of J names.

Reverse autocomplete

Direct manipulation of video is one of the more uncanny HCI concepts I've ever seen. Instead of manipulating time with a traditional scrubber bar, the user can drag objects in the video across their path of movement. Nothing in the video actually changes, but the perception is that you can directly manipulate the objects in the video stream by pulling them around through time.

There's a Windows application called DimP which implements this interface. When you hover over a movable object in the video, a light path appears that emphasizes the object's motion curve, which you can then move the object across. From the DimP website:

So what's being manipulated, exactly? Both the video content (e.g., the things you see moving in the video) and the "tape head". When using DimP, the user directly manipulates the video content and indirectly manipulates the tape head. When using the seeker bar, the user directly manipulates the tape head and indirectly manipulates the video content.

The video above describes how DimP works in a bit more detail, showing a few different video scenarios where direct manipulation really shines. It's intuitive and bizarre at the same time. If the universe is completely deterministic, I can't help but think this is what time travel must look like.

DimP - A Direct Manipulation Video Player
DRAGON - Direct Manipulation Interface Demo for OS X

There's a neat article over at Ars Technica that takes a look at 280 North's 3-person development team, their recent release of a keynote-like web application called 280 Slides, and the framework that they constructed to make the application. Some time before 280 Slides was developed, the team created an Objective-C-like superset of the Javascript language called Objective-J which implements message passing and a Smalltalk-style syntax. Building on that, they re-implemented much of the Cocoa framework in Objective-J, allowing Cocoa-style applications to be developed that will run natively in the browser.

With Objective-J and Cappuccino, you don't create applications with a mixture of HTML, CSS and Javascript. Instead, apps are written entirely in Objective-J, following a development model that's similar to creating desktop applications for OS X. From the article:

"Cappuccino is a re-implementation of Cocoa in Objective-J, which means we reimplemented AppKit, Foundation, CoreGraphics, and parts of CoreAnimation," Boucher told us. With it, developers familiar with desktop GUI applications can create a rich, desktop-like web app with the same relative ease Cocoa programmers can create a rich desktop app. "Coming from a background of desktop programming, and Cocoa in particular, we realized how much harder building a web application was. So we wanted to try to make things just a little bit easier."

280 Slides stands on its own as a powerful web-based presentation tool. It's simplified when compared to its desktop cousins, but it does exactly what most people need in a presentation tool, and it can export to PPT for fine tuning and end-presentation use. The fact that its development inspired the creation of a new language as well as bringing a desktop application development framework to the web just blows my mind.

Objective-J will be released to the public soon as an open source project. On the one hand, I'm not sure if I'm comfortable moving away from the web development technologies I've grown accustomed to. On the other, it would be cool to work in a language that allows you to seamlessly port your applications between the desktop and the web.

If you have experience with both Cocoa and AJAX development, please share your thoughts in the comment area.

Cocoa on the web: 280 North, Objective-J, and Cappuccino [via Slashdot]
280 North Blog
280 Slides
Objective-J.org

I stumbled across a fourth option this afternoon: the Algorithm Geeks user group. It's an easy way to parallelize your problem, running it past the brains of hundreds of alphas from around the world. Even if you aren't seeking help, maybe there's a post or two in there that you can contribute to. My inner nerd is probably showing, but it's pretty fun just reading through the posts and weighing the different proposed solutions.

Hack on!

Algorithm Geeks Google Group

Code Kata is an attempt to bring this element of practice to software development. A kata is an exercise in karate where you repeat a form many, many times, making little improvements in each. The intent behind code kata is similar. Each is a short exercise (perhaps 30 minutes to an hour long). Some involve programming, and can be coded in many different ways. Some are open ended, and involve thinking about the issues behind programming. These are unlikely to have a single correct answer. I add a new kata every week or so. Invest some time in your craft and try them.

Some of the example scenarios are expressed in Ruby, but aside from that they are language agnostic. What I like is that the exercises cover a wide breadth of scenarios, some of which I encounter daily in my work, and some only rarely. Even working through the problems and roughing out a solution in my head seems to be a useful tool for keeping the problem solving faculties sharp. Trying to solve problems like these involves no real deadlines or pressure, which is the necessary environment for devoting a little time regularly to focus on honing your skills.

Do you use any other resources for staying on top of your game? Let us know in the comments.

Code Kata

	send(to, from, count)
	register short *to, *from;
	register count;
	{
		do
			*to = *from++;
		while(--count>0);
	}

On every iteration of the loop, in addition to the copy that is being performed, the count variable is decremented and a comparison is done against 0. Duff's Device allows you to achieve the same result, but with only an 8th of the overhead:

	send(to, from, count)
	register short *to, *from;
	register count;
	{
		register n=(count+7)/8;
		switch(count%8){
		case 0:	do{	*to = *from++;
		case 7:		*to = *from++;
		case 6:		*to = *from++;
		case 5:		*to = *from++;
		case 4:		*to = *from++;
		case 3:		*to = *from++;
		case 2:		*to = *from++;
		case 1:		*to = *from++;
			}while(--n>0);
		}
	}

What happens is that the loop is unrolled 8 times, so each iteration of the loop runs the internal code 8 times over without the comparison. The genius of Duff's Device is that it takes advantage of the way the C switch/case structure works. The first time through, if the iterations don't divide evenly by 8, the loop code is executed enough times to equal the remainder of iterations/8. It's a little bizarre, because the "do" statement occurs within the switch, but there are "case" statements within the "do". Nevertheless, it's valid C.

Before someone cries foul, remember that this is only really suitable for speeding up the performance of inner loops when no suitable, better algorithm is available. If you code C, most modern compilers are smart enough to automatically optimize your code and unroll loops for you.

For interpreted languages like PHP or Javascript, however, you sometimes need to do a little optimization on your own if you want to squeeze out some extra performance. Luckily, both languages have a c-style switch statement.

Andy King wrote about a slightly altered version of this loop unrolling algorithm which ditches the switch statement and breaks the normal Duff's Device into two separate loops, one for the remainder and one for the unrolling. In Javascript, it performs a simple addition loop in only a third of the time of a normal for loop (testVal++ is the normal loop's interior):

function duffFasterLoop8(iterations) {
  var testVal=0;	
  var n = iterations % 8;
  if (n>0) {
    do 
    {
      testVal++;
    }
    while (--n); // n must be greater than 0 here
  }
    
  n = parseInt(iterations / 8);
  do 
  {
    testVal++;
    testVal++;
    testVal++;
    testVal++;
    testVal++;
    testVal++;
    testVal++;
    testVal++;
  }
  while (--n);
}

It's not as syntactically clever as Duff's Device, but it's a good way to manually unroll an inner loop and get the best performance for your extra effort.

Duff's Device
Andy King's Optimizing JavaScript For Execution Speed

John Resig, of jQuery fame, released a port of the Processing visualization language for Javascript. Seriously, John is on fire:

The first portion of the project was writing a parser to dynamically convert code written in the Processing language, to JavaScript. This involves a lot of gnarly regular expressions chewing up the code, spitting it out in a format that the browser understands.

It works "fairly well" (in that it's able to handle anything that the processing.org web site throws at it) but I'm sure its total scope is limited (until a proper parser is involved). I felt bad about tackling this using regular expressions until I found out that the original Processing code base did it in the same manner (they now use a real parser, naturally).

The full 2D API is implemented, with the exclusion of some features here and there between browsers (Firefox 3 is pretty full featured). You can interact with the Processing API directly from standard Javascript. This lets you make use of these drawing features by simply instantiating a Processing object, and then calling its various drawing methods.

Another capability is to write code natively in the Processing language. This allows you to make use of extended language features such as method overloading and classic inheritance, though it looks like type information is pretty much ignored.

John has many of the demos from processing.org working. Most of them are going to peg your CPU, but this is some seriously cool stuff to see working in a first release.

Javascript just got a lot more interesting.

Processing.js
Processing: open source data visualization language

Bridger Maxwell has been blogging his progress on creating a homebrew multi-touch platform in OS X. Prior to this, there's been a lot of activity around building multi-touch systems on Windows using the Touchlib library, but this is the first time I've seen a concerted effort on OS X.

The basic hardware is the same for both environments: LEDs surround a sheet of acrylic, causing a backscatter of IR when fingers are pressed to the screen. On the software side, though, the multi-touch interface is provided through Pawel Solyga's OpenTouch library. From the sounds of things, though, it's not super simple getting the interface messages from OpenTouch to your multi-touch enabled Cocoa apps:

Both OpenTouch and TouchLib send the touch data to other applications by sending Tangible User Interface Object (TUIO) network messages. TUIO is a protocol that is designed for transmitting the state of multi-touch systems. TUIO is built upon another protocol, Open Sound Control (OSC). While libraries for receiving TUIO messages are available in a few languages such as C++ or Java, there was not a solution for Cocoa applications. My first step was to build a library for receiving TUIO messages in Cocoa.

Because TUIO is built upon OSC, I looked for a library that could parse OSC messages. Unfortunately, I could not find one that would fill all my needs. WSOSC was a library that came close though. There were a few issues to work around (use NSData instead of NSString), but eventually I was able to use WSOSC to parse the OSC packets. When finished, my framework had the ability to parse TUIO messages and had a method to delegate the TUIOCursor objects it created to another application.

From the blog comments, it sounds like Bridger is planning on releasing this middle layer when it gets a little further along. At the moment, though, he's released a demo comic viewing application that uses his multi-touch project framework. If you're interested in developing multi-touch apps for OS X, some of the discussions on Bridger's blog would be a good place to start.

Bridger's Multi-Touch Blog
OpenTouch Library

See also:
Make your own multitouch displays and software apps

My friend Barrett sent along a link today to an Illustrator scripting plugin called Scriptographer. I'm sort of a slouch at Illustrator, so I had him give me the quick 411 and I must say, this is pretty cool.

If you're familiar with Javascript, Scriptographer will enable you to crank out little scripts that can generate illustrations procedurally. As an example, the bubbelbubbling script, show above, tuns your pen tool into a fountain of random bubbles that follow your drawing path. There are certain styles of artwork that could really lend themselves to a procedural drawing tool: fractals, patterns, random "particle" effects. These things would take forever to generate manually, but by defining the effect programatically, you can quickly experiment with your work in a more dynamic fashion, tweaking variables and fine-tuning your work as you go.

The project website also has a growing library of user-contributed scripts that are worth checking out. It's a good place to start for your own creations, or you may just find exactly what you're looking for, already crafted for you by another designer-coder.

Scriptographer

What's killer is that Adobe's as3corelib addon library finally gives us some essential tools that have been sorely lacking, none the least of which is a client side JPEG encoder. With this, you can turn any drawable object like a sprite or a movieclip into a ByteArray holding the compressed JPEG data in just a few lines of code. It's as simple as this:

import com.adobe.images.JPGEncoder;

var clipbmp:BitmapData = new BitmapData (aclip.width, aclip.height);
clipbmp.draw(aclip);

var jpgEnc:JPGEncoder = new JPGEncoder(90);
var jpgbytes:ByteArray = jpgEnc.encode(clipbmp);


This turns the "aclip" sprite or movieclip into a rasterized, flattened, BitmapData object. The BitmapData is then run through the JPEG encoder with the quality setting of 90 and you're left with the raw JPEG-compressed image in a ByteArray object. The as3corelib also provides a PNG encoder, with which you can just use the static method PNGEncoder.encode(clipbmp).

This is perfect for saving a capture of user-generated artwork to the server. Just set the data member of a URLRequest object to the ByteArray and post it. For more detailed information on how to put all the pieces together, Henry Jones has a really thorough post of compressing JPEG data and pinging it off a server to force an image download.

Unfortunately, to trigger a JPEG download, you still need to post the image data up to a server script and have it echo it back to the browser. The difference now, though, is that you can do the compression on the client end, saving both server CPU time and the time to upload the image data. This means saving a large image is a few second process instead of taking a minute and a half.

Actionscript 3 Core Library (as3corelib)

Today I was thinking aloud about Tree Regular Expressions and how they might make a nice query language for document databases like CouchDB. Someone pointed out that CSS3 selectors might make a great concrete syntax for this. One thing lead to another and I thought, why not build a relational database in HTML? So I did. I even got inner joins working.

Let's start with a few tables:

<table class="users">
  <tr>
    <td class="id">1</td>
    <td class="first_name">amy</td>
    <td class="last_name">bobamy</td>
  </tr> 
  ...
</table>
<table class="photos">
  <tr>
    <td class="id">1</td>
    <td class="user_id">1</td>
    <td class="url">http://www.example.com/foo.png</td>
  </tr> 
</table>

Now we can express some queries:

$('.users')
  .where('.id:eq(1)')
  .select('*')

This is equivalent to SELECT * FROM users WHERE id = 1

$('.users')
  .where('.id:eq(1)')
  .select('.id, .name')

This is equivalent to SELECT id, name FROM users WHERE id = 1

How cool is that? Check out Nick's blog post for an example of text search and an inner join. The API in his jquery.db.js is quite straightforward and only about 50 lines of code. Adding a sort function shouldn't be too difficult.

I'm pretty much convinced now that jQuery is black magic.

Building a relational database using jQuery and <TABLE> tags
Download the jquery.db.js library

Everyone who works with large data sets in Google Maps has come across the problem of displaying a bunch of markers in a small area. Not just an eyesore, displaying anything more than a hundred marker icons at a time can bog the browser down on a lot of platforms, Safari on PPC Macs delivering the most pain.

The solution is to cluster nearby markers into an aggregate marker when there are too many markers being displayed, or when markers are so close at a particular zoom level that they completely overlap. For extremely large datasets this is most efficiently done on the back-end, with successive AJAX calls refreshing the marker set from a PHP script that filters out the visible markers from the set.

You can also handle the clustering on the client side, using javascript to scan the entire set of locations and dynamically determine what's visible and what should be clustered. The downside is that you have to download the entire set and store it in the browser's memory, but unless you start getting well into tens of thousands of markers this isn't a big deal. The benefit to the client side method is that it's less complex, it lets you work around large result sets from back-end APIs that you can't control, and with ACME Labs' Clusterer javascript library it's extremely easy to code.

To use Clusterer, first download and include the Clusterer2.js file from the link below in your maps page. Then you need to instantiate a Clusterer object, passing your map object to its constructor:

var clusterManager = new Clusterer(map);

From there, you use it in place of the traditional MarkerManager or any addOverlay calls by calling the Clusterer's addMarker method. It takes two parameters, the marker to add, and a text string that will be listed in the cluster's contents when it is clicked:

clusterManager.AddMarker(marker, "Marker Description");

The cluster manager will take care of all the dirty work, only displaying items when they are within your view, and dynamically clustering them appropriately when there are too many on the screen at once. When one of the clusters is clicked, it will display a list of the locations inside of it. Most of what you'd want to tweak, like the threshold at which to start clustering and the icon used for representing a cluster, are all adjustable through the API via some self-explanatory methods such as SetMaxVisibleMarkers(n) and SetIcon(icon). Follow the link below for more information, or read the source for a few of the less-documented options.

Clusterer documentation
Clusterer source

Mick West from Cowboy Programming posted a two part series to his blog titled Practical Fluid Dynamics. Originally written for Game Developer Magazine, it covers a number of clever (and down-to-earth) techniques for simulating the movement of fluids in games and other software environments where real-time speed and visual authenticity matter most.

Special attention is paid to the simulation of particulate matter being carried around within a fluid volume—think effects like smoke, fire, and bubbles. I know I've seen a number of people using particle systems to do this sort of thing, but the methods Mick describes are all based on a grid model where you represent the system with a velocity field and a density field. Unlike a particle system, these fields represent a continuous fluid surface, allowing you to measure the density and velocity of the fluid at any location on the surface by interpolating the values from the nearest cells in the field array.

Practical Fluid Mechanics

Shimon Schocken gave a really interesting Google Tech Talk titled From Nand to Tetris in 12 Steps. In the video, he describes a course where students design a complete virtualized computer system from scratch, building from the humble nand gate, to a functional cpu and memory architecture, to compiler software and an operating system, all culminating in a simple game that runs on the virtual hardware.

The hardware projects are done in a simple hardware description language and a hardware simulator supplied by us. The software projects (assembler, VM, and a compiler for a simple object-based language) can be done in any language, using the APIs and test programs supplied by us. We also build a mini-OS. The result is a GameBoy-like computer, simulated on the student's PC. We start the course (and this talk) by demonstrating some video games running on this computer, e.g. Tetris and Pong.

Building a working computer from Nand gates alone is a thrilling intellectual exercise. It demonstrates the supreme power of recursive ascent, and teaches the students that building computer systems is -- more than anything else -- a triumph of human reasoning.

It looks like most of the course materials are available online. The necessary hardware emulator and simulator software is open source and available from Shimon's website.

CS101 Digital Systems Construction
Video - Building a Modern Computer from First Principles [via Slash7]

I guess it's been available for a few months, but I just noticed that there's a Ruby API for Google SketchUp. Looks like a cool tool for extending the building interface, integrating SketchUp entities with external software, and building procedural stuff, like making terrain or stairs.

Here's a video of SketchUp developer Mark Limber talking about some of the possible ways to extend the software with the Ruby API.

Google SketchUp Ruby API - Link
SketchUp API Blog - Link

Peter Rakos wrote an article for Gamasutra today which demonstrates how to hack yourself a simple 3D engine by subverting an Excel worksheet. It's not going to win any FPS awards, but the fact that you can even get Excel to draw raw shapes blows my mind.

In his demo, the worksheet is used to calculate values for all the polygon vertices and a very small macro loop draws the resulting mesh to the screen.

After downloading the source XLS, run the demo by hitting alt-F8 (option-F8 in Mac Excel). You'll find the code under the "Tools->Macro->Macros" menu.

Microsoft Excel: Revolutionary 3D Game Engine - Link
Peter's Example 3D Excel files - Link

For C style comments, the following will be commented out:

/*
if ( foo == bar )
{
  dosomething();
  return();
}
// */

And the addition of a single '/' will uncomment the block:

//*
if ( foo == bar )
{
  dosomething();
  return();
}
// */

In languages that don't have the single line comment, such as CSS, you can do the same thing with only the block level comments.
Commented:

/*/
min-height:100px;
/**/

/**/
min-height:100px;
/**/

You are probably talking to your screen right now, saying, "hey Jason, that commenting trick is marginally useful at best." I can only respond by reminding you that every keystroke is a beautiful and unique snowflake that must be cherished and never wasted.

A neat commenting trick - Link

One of the most frustrating things about homebrew game development is that there's almost an insurmountable amount of work that needs to be done just to get something decent to display on a screen. You can roll your own complete graphics and physics engines and still have nothing to show for it if there are no art assets to load.

QuantumG's solution to the problem was to focus on developing the game engine using the model data from GTA3. Knowing that the art is already functional in another game allows you to focus on your code, and it's more fun when you can see the immediate results of your work.

The blog entry walks you through his experience with extracting and using the mesh, texture, city, and character data and making use of it with the OGRE graphics engine. If you've ever played around with making a game before, but got discouraged for lack of art assets, this is really worth a read.

Using GTA3 art assets in OGRE - Link
OGRE 3D graphics engine - Link