Hackszine.com: Web Archives

Archive: Web

Page 3 of 5 << 1 2 3 4 5 >>

October 11, 2007

Where's the open source distributed search?

Back before Google, a lot of hackers were writing search engines in their free time. The general consensus, at least from my own recollection, was that search was a problem that needed to be solved, and that all the current solutions more or less sucked. Today, search encompasses a huge territory and there are still a lot of problems to be solved, but, for the most part, web search is extremely usable and reliable. It's not perfect, there's room for improvement, but it get's the job done. I don't know too many people these days who spend their time hacking search. Why re-create such a low-level service when there are so many innovative and higher-level web applications to be built?

The thing is, search is the operating system of the web. The fact that we have no open-source/open-data search infrastructure is as bad as if there were no Linux or OpenBSD. If Google, Yahoo and MS weren't providing such a great product, my guess is that the hacker community would be attacking this problem like Captain Kirk on a lizard monster.

Where We Are:

Currently, there are a number of open source projects related to general web search. Most notably, the Java based Lucene project is a solid foundation for indexing and information retrieval, and it's what the Nutch search engine is built on.

There are a few distributed crawlers like Grub and Majestic 12. Unfortunately, these both pass data to a central, private storage system. The hard work of crawling and indexing is open for everyone to participate in, but the resultant data is not.

Where We Need To Be:

In my mind, search hackers need to create an open source solution for the following:

A distributed mechanism for crawling and indexing the web on a mass scale.
Distributed, decentralized, redundant data storage for the cache and index.
An end-user, public facing interface for querying the distributed index.
A mechanism for retrieving or crawling a local, private slice of the index and cache, for research or personal use.
A way to publish alternate indexing models to the distributed grid.

All of these tools need to be designed with the assumption that anyone can and will have access to the system's data, and as the system grows, there will be people, corporations, and governments hell-bent on corrupting the search infrastructure to their advantage.

It's not an easy problem to solve, but you've got to admit it's an interesting problem. Anyone keen on being the Torvalds of search?

Where To Begin:

The Lucene Project - Link
Nutch Open Source Search Engine - Link
Open Source Search Wiki - Link

Have I missed anything? Please share your thoughts on open source search in the comments.

October 3, 2007

HOWTO - read RSS with a POP email client using FreePOPs

FreePOPs is an open source, plugin-based POP proxy that you can run on your local machine. It was originally designed to allow you to use a normal POP email client to read your mail on a multitude of webmail systems. You point your mail client at the FreePOPs server, and it connects and screen-scrapes your webmail account so that you can read your email in the comfort of your favorite mail reader.

The great thing about FreePOPs is its filter plugin architecture. There are a number of different plugins to support the specific requirements of most of the popular web-based email systems. There's even an RSS plugin that will pull an RSS feed and make it look like a normal POP mailbox. Thankfully, you don't need to configure anything on the server. Instead, you connect to the FreePOPs server using a particular username and password format to activate the appropriate plugin.

Here's how to set up an RSS-to-POP mailbox using FreePOPs and the standard OS X Mail.app email client.

Read full story

September 19, 2007

Install several versions of IE at once

If you've ever had to do any web testing, you've probably found that it's a major pain trying to test out a site in multiple versions of IE, since new versions of IE tend to wipe out previous installs. I usually use a different virtual machine instance for different browsers and OSs I want to test, but this is difficult to set up and takes up a lot of disk space.

Yousif Al Saif put together a nice multi-IE package that bundles IE 3 through 6 into a single installer. Just make sure you have the latest version of IE first, then run his installer and it will create the other standalone IE versions, each with their own private set of DLLs and registry keys.

Install multiple versions of IE on your PC - Link

August 19, 2007

Wikipedia offline reader: put all of Wikipedia on your laptop

Wikipedia periodically publishes full data dumps of the encyclopedia's content. If you wanted to make your own copy of the Wikipedia site for offline viewing, you'd typically convert and import that content into MySQL using MediaWiki's importDump.php utility. The initial import process can take over a day. Building the indexes for searching articles takes even longer.

Thanassis Tsiodras came up with a better way of using the Wikipedia dump for offline reading:

Wouldn't it be perfect, if we could use the wikipedia "dump" data JUST as they arive after the download? Without creating a much larger (space-wize) MySQL database? And also be able to search for parts of title names and get back lists of titles with "similarity percentages"?

The end result is a Wikipedia reader that indexes the entire dump in under 30 minutes, stores the entire data in the original, though segmented, bz2 compressed format, and comes complete with a light web interface for searching and reading entries.

Building a (fast) Wikipedia offline reader - Link

Related:
WikipediaFS - a Linux MediaWiki file-system - Link

August 18, 2007

Coding plugins for Internet Explorer

Pete Warden wrote in about his completion of the IE port of the GoogleHotKeys (formerly PeteSearch) browser plugin.

In addition to the blog posts covering the port I've also put together a public wiki with full documentation on creating your own Internet Explorer extensions, and there's the full source code for the completed plugin for reference too.

When I checked out PeteSearch back in June, it was a pretty cool Firefox plugin that pre-scanned search results and added keyboard shortcuts to Google. Pete was just starting an IE version at the time, so it's cool to see a first version released along with all the documentation you need to make your own IE plugin.

How to write extensions for Internet Explorer - Link
GoogleHotKeys (info, download and sourcecode) - Link
HOWTO: Port Firefox extensions to IE - Link

July 17, 2007

Javascript 3D engine

I wouldn't have thought it possible, but here's an extremely clever hack from the Useless Pickles blog that will render 3D objects at a usable frame rate using standard Javascript.

You might be familiar with another method in which you create a bunch of 1 pixel-high divs to draw each triangle with a series of horizontal slices. This becomes unbearably slow, however, as you need an awful lot of divs just to display a single triangle. Lot's of divs = super slow frame rate.

The trick is to take advantage of the beveled edges on a div border to make right triangles. By setting the div height and width to 0px, then setting 3 borders to transparent and 1 border to an opaque color, you can make a right triangle of various sizes by adjusting the border widths.

Using some fancy geometry, you can then cut any type of triangle into a series of successively smaller right triangles. Now, in real life, the right triangles would get infinitely small (and numerous), but in the digital world, they only need to get as small as a single pixel before we don't care anymore.

With this method of drawing a triangle, you might only need to use 10 divs to draw a 100 pixel tall triangle (which would have taken 100 divs in the scan line method).

With a reasonably fast (or should I say "not unbearably slow") method for drawing triangles, you can now create a simple 3D engine to draw objects at any orientation. Obviously, you'll want to limit the number of triangles your objects are composed of, but it works. As a demonstration, the author created a simple engine for rendering and rotating the polyhedron seen above.

References:
Javascript Triangles (and real-time 3d) - Link
Rotating 3D Polyhedron Demo - Link

July 10, 2007

Server-side Javascript/DOM - search friendly AJAX?

John Resig posted yesterday about his experiments with creating a full Javascript/DOM pseudo-browser environment that runs from the command line:

This weekend I took a big step in upping the ante for JavaScript as a Language. At some point last Friday evening I started coding and didn't stop until sometime mid-Monday. The result is a good-enough browser/DOM environment, written in JavaScript, that runs on top of Rhino; capable of running jQuery, Prototype, and MochiKit (at the very least).

The really nice touch is that you can issue PUT and DELETE requests on the XMLHttpRequest object to manipulate files on the local file system! Here's an example script that scrapes post titles from alistapart.com and stores them in a file (remember, this runs on the server like a shell script):

load("env.js");
window.location = "http://alistapart.com/";
window.onload = function(){
  load("dist/jquery.js");
  var str = "Newest A List Apart Posts:\n";
  $("h4.title").each(function(){
    str += " - " + this.textContent + "\n";
  });
  var out = new XMLHttpRequest();
  out.open("PUT", "file:/tmp/alist.txt");
  out.send( str );
};

Read full story

July 2, 2007

Client-side SQL database in Javascript

I was thinking today about how a person might go about creating a rich offline web application for devices like the iPhone, where your application may need to run entirely from cache from time to time.

In the extreme example, you could imagine an application that stored its data entirely on the client side, and used cached html and javascript files to manipulate that data. Assuming you could get around the storage issue -- either via cookies, a Flash shared object (not available for the iPhone), Firefox or IEs storage objects, or some trickery with browser form autocompletion -- you'd still run into the task of manipulating that data.

Web developers are used to using an SQL database for the retrieval, joining and sorting of data, so it's convenient that similar functionality exists within Javascript, thanks to a small Javascript library called TrimQuery. After defining a simple table schema, you can use TrimQuery to issue a subset of the standard SELECT syntax. This will let you do something like the following, all within Javascript:

selectStatement = queryLang.parseSQL( "SELECT table1.columna, table2.columnb 
  FROM table1, table2 
  WHERE table1.id = table2.fkid 
  ORDER BY table1.columna" );
var results = selectStatement.filter( tabledata );
for (var i = 0; i < results.length; i++) {
  var record = results[i];
  // ...
}

TrimQuery Demo - Link
TrimQuery: Javascript SQL Library - Link

June 20, 2007

HOWTO use the Wiimote buttons in Flash

A while back, WiiNintendo posted the keycodes that are detectable by Javascript when the Wii's buttons are pressed. With this ability, you can create Javascript games that will play on your Wii. Unfortunately, even though the Wii browser ships with the Flash plugin, those keycodes cannot be detected natively in the Flash environment.

Quasimondo came up with a really clever hack that solves the problem. You can create a second flash movie and use Javascript to resize it to specific widths for particular keycodes. Even though Flash cannot detect the keycodes, it does receive an onResize event. When it receives this event, it then retrieves its current width, which was set to the value of the keycode. This second swf file can then use Flash's LocalConnection to communicate that value to the primary swf.

How to Make the Wiimote Work in Flash - Link
Aral Balkan's discussion on using the Wiimote in Flash - Link
Wiimote Key Codes @ WiiNintendo - Link

June 18, 2007

DSL for $10

It only applies to folks in the 22 states that AT&T serves, and you need to sign a 1 year contract, but check out what The Consumerist found:

As part of a concession made to the FCC in order to get its mitts on BellSouth, AT&T is required to offer basic DSL for $10 a month to its entire 22 state coverage area for a period of 2 years.
If you can't find the plan listed on the website, don't worry. AT&T wants it that way. They've hidden it, according to the AP.

Cory Doctorow makes a good point, though:

But even at $10/month, AT&T DSL should be avoided like the plague. These are the scumbags who illegally wiretapped the entire Internet for the NSA, who broke net-neutrality to find "copyright infringements, and who inspired NBC to call for a law requiring all ISPs to do the same (imagine -- a law forbidding network neutrality!). Seriously: the only day I wouldn't piss on AT&T is if they were on fire.

But, if you're the type that can stomache the death star's policies, $10/mo is a darn good deal. Just inquire about the secret FastAccess DSL Lite plan.

AT&T's Secret $10 DSL - [via] Link

June 14, 2007

Graph your Flickr pageviews with Statr

Ever want to track how many folks are viewing your photos on Flickr? Just give Statr access to pull your pageviews and it will collect and graph your Flickr statistics for you.

Statr for Flickr allows you to track and plot page views statistics for your Flickr account. Graphs are automatically updated on a daily basis and can be linked from external websites.

Statr for Flickr: tracking page views for your Flickr account - [via] Link

June 13, 2007

Use a Google Spreadsheet as a simple CMS

Using Google's Spreadsheet API, you can create a simple CMS for your website extremely easily. It's as simple as making a new sheet with key/value pairs for any fields you want dynamically populated. You can then access the data from javascript via the JSON API. Any updates to the spreadsheet will be immediately reflected in your page.

The only downside to this approach is that the content of your page is populated by Javascript on the client side, so search engines won't see the content of your pages during a crawl. You could overcome this obstacle by using the PHP API to pull the spreadsheet data and render the HTML on the server side.

Simple CMS using Google Spreadsheet API - Link
Google Spreadsheets Data API - Link

June 11, 2007

Crreate a Wii media server

Wii Media Center X is a Java-based web media server that you can have up and running in about 15 minutes. After downloading and unzipping the application, just run the following command (from the mediacenter folder) to start it up:

java -jar MediaCenter.jar

Run this on any machine that your Wii can see and you can use the Internet Channel web browser to access your media center at http://x.x.x.x:8192/.

I've noticed a few problems with playing audio files and the video isn't sized optimally (you can zoom to adjust for this), but this shows a lot of promise. With this running on my primary desktop machine, I can encode/download FLV files that I want to watch and MP3s that I want to listen to. You just drag them to the Video/videos and Music/music folders. You can do this during the day as you come across things, and then you can view and listen to media in your living room whenever you like.

With a little spit and polish, this could be like having all the AppleTV or Windows Media Center functionality right on your game console.

Red Kawa Wii Media Center X - [via] Link

June 8, 2007

HOWTO: Port Firefox extensions to IE

Hackszine reader Pete Warden is the author of a cool Firefox extension called PeteSearch. He wrote in to tell us about his progress with porting the functionality to IE:

PeteSearch is an extension that mashes up Google (and Ask, Technorati, Live and Exalead) search pages to give you hot-keys, dead-link checking and split-screen preview.
Now I'm trying the IE port, and I've my own HOWTO with sample code and articles on that painful process!

Building a browser object for IE is a much more laborious process than making a Firefox extension. Welcome to COM DLL hell. Thankfully, Pete's done a nice job of documenting things while porting his app over, and it's this sort of first-hand experience that makes it bearable to start a project like this of your own.

June 5, 2007

Use GPG encryption with Firefox and GMail

FireGPG is an awesome little plugin that adds GPG support to Firefox. You need the GPG package installed on your machine to start, and after activating the plugin, you'll have a new right-click menu that will let you sign, encrypt, decrypt and verify any selected text.

You can use this to add strong crypto functionaliy to any webmail system or forum that you use! Special support for GMail is already built-in, which provides encryption and signature buttons right alongside the normal send button.

Currently, there isn't a lot of documentation, but the author has set up a Wiki. If you want to help out, try the software for a while and pitch in with a page or two on the maual.

FireGPG: Use GPG Easily in Firefox - [via] Link

GPG (GNU Privacy Guard) for Linux, Mac, and Windows - Link