Hackszine.com: Data Archives

Archive: Data

Page 2 of 3 << 1 2 3 >>

May 9, 2008

Processing.js - visualization library for Javascript

John Resig, of jQuery fame, released a port of the Processing visualization language for Javascript. Seriously, John is on fire:

The first portion of the project was writing a parser to dynamically convert code written in the Processing language, to JavaScript. This involves a lot of gnarly regular expressions chewing up the code, spitting it out in a format that the browser understands.
It works "fairly well" (in that it's able to handle anything that the processing.org web site throws at it) but I'm sure its total scope is limited (until a proper parser is involved). I felt bad about tackling this using regular expressions until I found out that the original Processing code base did it in the same manner (they now use a real parser, naturally).

The full 2D API is implemented, with the exclusion of some features here and there between browsers (Firefox 3 is pretty full featured). You can interact with the Processing API directly from standard Javascript. This lets you make use of these drawing features by simply instantiating a Processing object, and then calling its various drawing methods.

Another capability is to write code natively in the Processing language. This allows you to make use of extended language features such as method overloading and classic inheritance, though it looks like type information is pretty much ignored.

John has many of the demos from processing.org working. Most of them are going to peg your CPU, but this is some seriously cool stuff to see working in a first release.

Javascript just got a lot more interesting.

Processing.js
Processing: open source data visualization language

March 17, 2008

CryoPID: hibernation for Linux processes

We're all familiar with the hibernate/deep-sleep features that are typical on your standard laptop. In this mode, the entire contents of RAM are written to the disk and the machine is completely shut down. When it's next booted, the system is restored to the exact state it was at before sleep, with all of your programs running just like they were when you left them.

What if you could do this at the process level? You could kill whatever umpteen-gazillion applications you have running, reboot your computer, and then start your apps back up whenever you like and they would be exactly the way they were when you left them.

There's a Linux application called CryoPID which attempts to do just that.

CryoPID requires no special kernel modifications and operates in user mode, so you don't need to be root. All you do is run the freeze program on a process you own:

freeze /tmp/savestatefile 1234

This will archive the state of process 1234 into a self-executing, compressed file named /tmp/savestatefile. To start it back up, just run the save file:

/tmp/savestatefile

When this is executed, your application will be restored, relinked to any previously-loaded DLLs, and attached to the file descriptors it had open.

You'll run into some problems with network socket connections you had open, and support for X applications is still only experimental, so the useful scenario is a bit limited, but it's a promising concept and could come in quite handy in the command-line world.

CryoPID - A Process Freezer for Linux

March 4, 2008

Ram dump over Firewire

Unlike USB2, the Firewire spec allows devices to have full DMA access. By impersonating the appropriate device, a PC can essentially obtain full read/write access to another machine's RAM, just by connecting the two machines with a Firewire cable. Adding to the recent discussion about the insecurities of physical access and Princeton's cold-boot RAM dump demonstration, Adam Boileau released a Linux Firewire utility that will give you immediate Administrator to an XP machine:

It's two years later, and I think anyone who was going to get the message about Firewire has already got it, and anyone who was going to be upset about it has got over it. Besides, according to Microsoft's definition, it never was a Security Vulnerability anyway - screensavers and login prompts are - as Bruce says - about the Feeling of Security. Anyway, today's release day for Winlockpwn, the tool I demoed at Ruxcon for bypassing windows auth, or popping an admin shell at the login window.
...

Yes, you can read and write main memory over firewire on windows.
Yes, this means you can completely own any box who's firewire port you can plug into in seconds.
Yes, it requires physical access. People with physical access win in lots of ways. Sure, this is fast and easy, but it's just one of many.
Yes, it's a FEATURE, not a bug. It's the Fire in Firewire. Yes, I know this, Microsoft know this. The OHCI-1394 spec knows this. People with firewire ports generally dont.

Adam's tools include a few Python apps that can copy and impersonate Firewire device signatures, dump RAM on a remote machine, bypass Windows authentication, and extract BIOS passwords. It's not exactly comforting, but I've got a new appreciation for Firewire now. This is the sort of access that used to only be possible by creating hardware that physically connects to the PCI bus. Now all you need is a cable and a laptop.

Firewire, DMA & Windows - direct memory access over Firewire - [via] Link

March 1, 2008

Recover data from RAM after a crash

After Princeton's cold-boot encryption key recovery hack, I got to thinking about what other useful things might be lying around in memory. It's old news that passwords of logged-in users are hanging out in there, but what about something more useful to the everyday user? What about that file you were editing before accidentally closing its window without saving?

In Linux and on PPC Macs, the root user can access the machine's ram through the /dev/mem device. I'm not sure why this is unavailable on newer Intel Macs—it's a bummer.

In theory, if you're processing some words, spreading sheets, or posting a blog entry and your program crashes, it's likely that the data you were editing will still be in RAM, unharmed, waiting to be allocated to another process. If you immediately dump the entire contents of RAM to disk before starting another large process, chances are good you can find your data again. It's tricky though—writing that RAM to disk requires you start up at least one process, such as dd. It's possible that this new process, or a another process that's currently running, could allocate memory and obliterate your file. You don't really have other options, though, so you might try something like this:

dd if=/dev/mem of=/tmp/ramdump strings /tmp/ramdump | grep "some text in your file"

I found a post by David Keech where he describes exactly this process. He was able to use it to successfully recover the text from a killed vi session:

I tested this by starting vi and typing in "thisisanabsolutelyuniqueteststring", killing the vi process without saving the file and running the command above immediately with a small modification. Instead of piping the output to a file, I piped it to grep thisisanabsolutelyuniquetest. The grep command found itself, as it always does, but it also found the original string, identified by the rest of the unique string that I didn't include in the grep command. You have to be careful when search through running memory. I now remember having this problem with the Mac all those years ago. Whenever I searched for parts of my brother's letter, I would just end up finding the part of memory that contained the search string.

He also mentions scanning the swap partition, which is also a likely place for your data to be found. It's the same process, but you replace /dev/mem with /dev/hda2 or whatever your swap partition is.

Here's the fun part. Based on what we now know about DRAM holding data even a few seconds of being unpowered, you might even be able to use the method to recover program data after a full system crash and reboot. The swap data will for sure be there, but if you reboot into single user mode without starting up X or any large applications, the possibility exists that unallocated areas of /dev/mem will still contain data from before the reboot.

How to recover your data after a crash - Link
Extracting encryption keys after a cold boot - Link

February 6, 2008

TrueCrypt for OS X

TrueCrypt 5.0 was released yesterday and OS X has been added to the list of supported operating systems, making it the only open source volume encryption utility that works in Linux, Mac and Windows. It's a really slick utility for creating an AES-256 or Serpent encrypted volume that you can drop sensitive files inside.

You can use TrueCrypt to create an encrypted volume image inside a file, or you can encrypt a whole disk image or partition. The OS X version uses MacFUSE to provide user-mode mounting of the encrypted disk. The main application window, pictured above, gives you a simple interface for creating and mounting encrypted images.

Once an image is mounted, you can use it like a normal hard disk. Unmount the disk and you're left with a file full of random gibberish. FAT is the only filesystem that's available through the interface, but once the disk is mounted, you can reformat it with Disk Utility to use XFS.

There are a couple of things worth noting. In the Windows and Linux versions a special bootloader is available that lets you encrypt your entire system drive. It doesn't look like that option is available in the OS X version. Also, when I tested the latest OS X binary this evening, the "hidden volume" plausible deniability feature wasn't working. Hopefully that will be added in a future release. Until then, TrueCrypt is better suited for storing tax documents and things you wouldn't want visible to a laptop thief, rather than the details of where you've hidden the bodies.

TrueCrypt - [via] Link

December 7, 2007

Hacking the Western Digital MyBook World Edition

Western Digital sells a number of external drives under the MyBook World Edition brand. These are network-based external storage drives that you can connect to remotely from multiple machines. Inside are a couple of drives set up in a mirrored RAID configuration, as well as an embedded computer running Linux.

MakeFan: tipped us off to Martin Hinner's website, which has a lot of details about the software running on the MyBooks, including info for hacking the devices capabilities to do more than what's available out of the box.

This page provides information on how to hack your MyBook World Edition, so as you can improve performance and add new features. MyBook is powered by ARM9 microprocessor, it has 32MB of SDRAM and boots from internal hard drive. The system partition has 2.8GB (only 260 MB is occupied). This means that you have a lot of resources for various improvements.

You can enable SSH on the device without cracking the case. Martin hosts a script that subverts the firmware update software to create your ssl keys and boot the sshd process. Once that is enabled, you have full access to the OS to do what you like, including running an NFS server, web server, or even replacing the standard web interface.

Also worth checking out is the Hacking WD MyBook Wiki. They have links to information on rescuing data from dead drives and building other software for the device. Keep in mind that building MySQL from source will take about 18 hours, but there's got to be something fun you can do with a LAMP stack running on a terabyte hard drive.

Hacking Western Digital MyBook World Edition - Link
MyBook World Edition Wiki - Link

November 27, 2007

Using QoS to prioritize your bandwidth

You can use the Quality of Service (QoS) feature on many routers to optimize your internet connection for devices and services that are most important to you. Adam at Lifehacker put together a quick walkthrough for tuning the QoS settings on our favorite router firmware, the open-source DD-WRT.

Your internet connection is an indispensable part of your life, but between BitTorrent, Xbox Live, web browsing, and VoIP, sometimes there's not enough bandwidth to go around. But rather than running around the house shutting down all of your computers next time you're experiencing a little lag on Xbox Live or Skype is breaking up on you, you can set up Quality of Service (QoS) rules on your router to distribute bandwidth to your different gadgets and applications based on your priorities. Today I'll show you how.

The nice thing about setting up QoS on your router is that you can prioritize packets by application, IP and MAC address, or a specific router port, all to your own preferences. Everyone has different needs, so you may want to maximize the available bandwidth for VoIP or network gaming, guarantee a chunk of bandwidth for ssh connections or your web server, or throttle down P2P applications so that they don't affect other network services. There's nothing set in stone, so you can really make your router work for you to improve the way your internet connection is utilized.

Ensure a Fast Internet Connection When You Need It - Link

October 11, 2007

Where's the open source distributed search?

Back before Google, a lot of hackers were writing search engines in their free time. The general consensus, at least from my own recollection, was that search was a problem that needed to be solved, and that all the current solutions more or less sucked. Today, search encompasses a huge territory and there are still a lot of problems to be solved, but, for the most part, web search is extremely usable and reliable. It's not perfect, there's room for improvement, but it get's the job done. I don't know too many people these days who spend their time hacking search. Why re-create such a low-level service when there are so many innovative and higher-level web applications to be built?

The thing is, search is the operating system of the web. The fact that we have no open-source/open-data search infrastructure is as bad as if there were no Linux or OpenBSD. If Google, Yahoo and MS weren't providing such a great product, my guess is that the hacker community would be attacking this problem like Captain Kirk on a lizard monster.

Where We Are:

Currently, there are a number of open source projects related to general web search. Most notably, the Java based Lucene project is a solid foundation for indexing and information retrieval, and it's what the Nutch search engine is built on.

There are a few distributed crawlers like Grub and Majestic 12. Unfortunately, these both pass data to a central, private storage system. The hard work of crawling and indexing is open for everyone to participate in, but the resultant data is not.

Where We Need To Be:

In my mind, search hackers need to create an open source solution for the following:

A distributed mechanism for crawling and indexing the web on a mass scale.
Distributed, decentralized, redundant data storage for the cache and index.
An end-user, public facing interface for querying the distributed index.
A mechanism for retrieving or crawling a local, private slice of the index and cache, for research or personal use.
A way to publish alternate indexing models to the distributed grid.

All of these tools need to be designed with the assumption that anyone can and will have access to the system's data, and as the system grows, there will be people, corporations, and governments hell-bent on corrupting the search infrastructure to their advantage.

It's not an easy problem to solve, but you've got to admit it's an interesting problem. Anyone keen on being the Torvalds of search?

Where To Begin:

The Lucene Project - Link
Nutch Open Source Search Engine - Link
Open Source Search Wiki - Link

Have I missed anything? Please share your thoughts on open source search in the comments.

August 12, 2007

Burning visible images onto a CD-R

Here's an hack that takes on the challenge of burning a visible image into a CD-R.

Data on a CD, or any optical media, is stored as a sequence of pits of varying lengths. To be precise, a 1 is represented by the change from pit to no-pit or the change from no-pit to pit, and a 0 is represented by no change in height (pit to pit or no-pit to no-pit). The pits and no-pits reflect different amounts of light; thus it is possible to draw images on CDs by appropriately arranging these 1s and 0s.

As you can see in the photo above, the results are difficult to make out, mostly due to calibration settings that the author needed to tweak. All of his source is available, though some of it requires Matlab. Anyone care to test this out and link us to a photo in the comments?

What I'd really like to see is something that will take an image and convert it into a wav file that can be placed as the first track on the disk to show an image...

Burning visible images onto CD-Rs with data (beta) - Link

May 30, 2007

Gears API: create web apps that work offline

Google released a new Javascript API today called Gears that makes it possible to write modeless web applications that will function offline. A browser plugin is available for IE and Firefox (OS X, Linux, and Windows), with Safari support planned for the near future. The plugin will need to be installed by users of Gears-enabled applications.

From what I can see--and keep in mind that I haven't used the API yet--there are 3 basic services that the API provides:

local file resource storage and caching so that you can view files after disconnecting
a client-side SQL database that can be used by Javascript to store and fetch data
a worker pool module for running asynchronous background processes

The obvious use for this is to make stateful applications that continue to operate when you're offline, but maybe there are some privacy opportunities here too. Today, applications come in primarily two varieties: apps with user data and software stored locally, and web-based applications that execute and store data on the server. What I'm curious to see is if developers will begin making a third, hybrid category of application, where software release and maintenance is web-based and global data is available for local consumption, but the storage and processing of user-specific data takes place on the client side, safe from unwanted profiling.

Google Gears API Developer's Guide - Link

May 24, 2007

reCAPTCHA: distributed book digitization while fighting spam

Thanks to spammers, we now are forced waste a substantial portion of time every day, typing in obfuscated wiggly letters to prove we are human. reCATPCHA is a slick idea for using the CAPTCHA system for doing something productive (...besides distinguising between homo sapien and homo computatralis).

With reCAPTCHA, the user is given two words, one known by the system and one from a book that previously failed character recognition. When the user enters both words, the sytem verifies the known word, proving human-ness, and submits the second word to a central database, which helps digitze books from the Internet Archive. With 60 million CAPTCHAs being solved every day, this could be a huge assist for portions of text that can't be handled by optical character regognition techniques. [via] Link

Related:
Negative CAPTCHA

May 7, 2007

Openchange: open source Exchange MAPI library

The Openchange project's libmapi looks really promising. It's an open source MAPI implementation that allows you to talk to an Exchange server from your own code.

The OpenChange MAPI library aims to provide interoperability with an Open Source implementation of Microsoft Exchange protocols under UNIX/Linux. The current implementation offers a client-side library which can be used in existing messaging clients and offer native compatibility with Exchange Servers up to 2003.

As of last weekend it now supports the creation and extraction of not only messages, but also contacts, calender events, and tasks. This should be really useful for developers who want to interoperate with Exchange from other platforms and external mail systems.

Openchange: open source Exchange server alternative - Link.
Feature demonstration screencast - Link.

April 29, 2007

RAID 0 in a Macbook Pro

Putting a RAID 0 (striped) disk array in your laptop is a double edged sword. One the one hand, it's twice as likely to have a disk failure. One the other hand, it can make disk-bound activity, such as uncompressed video reading and writing, much faster. Frankly, with processor, bus and memory speeds as fast as they are, disk IO is often times the weakest link... which makes that risky performance upgrade all the more tempting.

Eric Cheng recently posted his experiments on getting RAID 0 working with a Macbook Pro:

Subjectively, the system feels more snappy, and I'm happy to have made the upgrade. I use up at least 20-30GB per week in the field, and it will be great to be able to work on my notebook without having an external drive chained to it.

As for the disk failure issue, we all back up nightly anyway, don't we?

320GB striped array (RAID 0) in a Macbook Pro -[via] Link.

April 5, 2007

S3 makes website hosting reasonable again?

100014192753
I keep hearing from friends each month about how Amazon's S3 is making website hosting reasonable for regular folks - here's another example from one of MAKE's kit makers -

Amazon's S3 (Simple Storage Service) isn't new, but its certainly gaining traction. Its a wonderful product for people who have a lot of content on their site (images, video, downloads, pdfs) but not a lot of money. Data storage costs $0.15 per GB-Month (prorated), and $0.20 per GB. No minimums, rounded up to the nearest cent.
...OK so, what's the point and what does this have to do with electronics, eh? Well one of the killer apps of open source and public domain electronics is documentation. That means media. And media storage, backup and transfer is extremely expensive for the everyday person. It becomes increasingly difficult to host a project when one digg-storm or slashdotting makes that 'free' webpage account go down.

S3 makes website hosting reasonable again - Link.

March 22, 2007

GSpace: Sweet Gmail File Transfer Plugin

Hackszine reader Abhay Krishna wrote in about a cool Firefox plugin that allows you to drag and drop files into your Gmail account.

This extension allows you to use your Gmail Space (2.8 GB and growing) for file storage. It acts as an online drive, so you can upload files from your hard drive and access them from every Internet capable system. The interface will make your Gmail account look like a FTP host.

It's an everywhere-accessible 2.8GB hard disk!

One thing to note is that Gmail accounts sometimes get blocked for 24 hours if excessive amounts of data are being transferred. Because of this, it's recommended that you don't transfer more than 1GB/day to any particular account. There's no reason why you couldn't use a few accounts for different purposes, however.

Gmail Space - Firefox plugin by Rahul Jonna and Tnarik -Link.

Page 2 of 3 << 1 2 3 >>