Hackszine.com: Linux Archives

Archive: Linux

Page 2 of 6 << 1 2 3 4 5 6 >>

August 6, 2008

Memcached and high performance MySQL

Memcached is a distributed object caching system that was originally developed to improve the performance of LiveJournal and has subsequently been used as a scaling strategy for a number of high-load sites. It serves as a large, extremely fast hash table that can be spread across many servers and accessed simultaneously from multiple processes. It's designed to be used for almost any back-end caching need, and for high performance web applications, it's a great complement to a database like MySQL.

In a typical environment, a web developer might employ a combination of process level caching and the built-in MySQL query caching to eke out that extra bit of performance from an application. The problem is that in-process caching is limited to the web process running on a single server. In a load-balanced configuration, each server is maintaining its own cache, limiting the efficiency and available size of the cache. Similarly, MySQL's query cache is limited to the server that the MySQL process is running on. The query cache is also limited in that it can only cache row results. With memcached you can set up a number cache servers which can store any type of serialized object and this data can be shared by all of the loadbalanced web servers. Cool, no?

To set up a memcached server, you simple download the daemon and run it with a few parameters. From the memcached web site:

First, you start up the memcached daemon on as many spare machines as you have. The daemon has no configuration file, just a few command line options, only 3 or 4 of which you'll likely use:
# ./memcached -d -m 2048 -l 10.0.0.40 -p 11211

This starts memcached up as a daemon, using 2GB of memory, and listening on IP 10.0.0.40, port 11211. Because a 32-bit process can only address 4GB of virtual memory (usually significantly less, depending on your operating system), if you have a 32-bit server with 4-64GB of memory using PAE you can just run multiple processes on the machine, each using 2 or 3GB of memory.

It's about as simple as it gets. There's no real configuration. No authentication. It's just a gigantor hash table. Obviously, you'd set this up on a private, non-addressable network. From there, the work of querying and updating the cache is completely up to the application designer. You are afforded the basic functions of set, get, and delete. Here's a simple example in PHP:

$memcache = new Memcache; $memcache->addServer('10.0.0.40', 11211); $memcache->addServer('10.0.0.41', 11211);
$value= "Data to cache";

$memcache->set('thekey', $value, 60);
echo "Caching for 60 seconds: $value <br>\n";

$retrieved = $memcache->get('thekey');
echo "Retrieved: $retrieved <br>\n";

The PHP library takes care of the dirty work of serializing any value you pass to the cache, so you can send and retrieve arrays or even complete data objects.

In your application's data layer, instead of immediately hitting the database, you can now query memcached first. If the item is found, there's no need to hit the database and assemble the data object. If the key is not found, you select the relevant data from the database and store the derived object in the cache. Similarly, you update the cache whenever your data object is altered and updated in the database. Assuming your API is structured well, only a few edits need to be made to dramatically alter the scalability and performance of your application.

I've linked to a few resources below where you can find more information on using memcached in your application. In addition to the documentation on the memcached web site, Todd Hoff has compiled a list of articles on memcached and summarized several memcached performance techniques. It's a pretty versatile tool. For those of you who've used memcached, give us a holler in the comments and share your tips and tricks.

Memcached
Strategies for Using Memcached and MySQL Better Together
Memcached and MySQL tutorial (PDF)

August 4, 2008

Shield your files with Reed-Solomon codes

Thanassis Tsiodras wrote in about a utility for adding additional error correction redundancy to your backup data:

The way storage quality has been nose-diving in the last years, you'll inevitably end up losing data because of bad sectors. Backing up, using RAID and version control repositories are some of the methods used to cope ; here's another that can help prevent data loss from bad sectors. It is a software-only method, and it has saved me from a lot of grief.

The technique uses Reed-Solomon coding to add additional parity bytes to your data. If you suffer partial damage to the storage media, these files can still be recoverable.

Storage media are of course block devices, that work or fail on 512-byte sector boundaries (for hard disks and floppies, at least - in CDs and DVDs the sector size is 2048 bytes). This is why the shielded stream must be interleaved every N bytes (that is, the encoded bytes must be placed in the shielded file at offsets 1,N,2N,...,2,2+N,etc): In this way, 512 shielded blocks pass through each sector (for 512 byte sectors), and if a sector becomes defective, only one byte is lost in each of the shielded 255-byte blocks that pass through this sector. The algorithm can handle 16 of those errors, so data will only be lost if sector i, sector i+N, sector i+2N, ... up to sector i+15N are lost! Taking into account the fact that sector errors are local events (in terms of storage space), chances are quite high that the file will be completely recovered, even if a large number of sectors (in this implementation: up to 127 consecutive ones) are lost.

The application works similar to any other command line archiving utility, so you can tar your files as normal and then send them to the freeze.sh script. Running melt.sh on the archive will return your original data, even if there was a reasonable amount of corruption to the file. Thanks, Thanassis!

Hardening your files with Reed-Solomon codes

July 16, 2008

Improve Linux laptop performance with Ramlog

One of the most power-hungry components in a traditional laptop is its hard disk, and time between charges can be greatly improved by keeping the disk in sleep mode. On machines like the OLPC that have solid-state disks, keeping disk writes to a minimum improves the life of the drive, minimizing unwritable sectors. Depending on how your machine is configured, log activity from kernel events and running daemons like sshd, a dns cache, or a local copy of apache can force your disk to make tiny writes every few minutes, impacting flash drive lifetime and ensuring that a mechanical drive never sleeps.

One solution to the problem is to disable syslogd entirely. An alternative is Ramlog, which offers a bit of a compromise. With Ramlog installed, log data is stored in RAM until shutdown, when it's copied back to disk in one big write. You will loose your logs if you have a system crash, but in a more usual scenario where you're trying to track down a wireless problem or an apache error on your development laptop, the logs are there for you to examine.

Installing Ramlog [linux.com]
Ramlog downloads

July 13, 2008

Find and Grep 101

Find and Grep are perhaps the most used command line tools for the Linux user or administrator. Terse but powerful, these two commands will allow you to search through files and their contents by almost any imaginable attribute or filter criteria: file name, date modified, occurrence of the some specific word in a file, etc. Combined with a couple of other standard unix utilities, you can automate and process modifications over a number of files that match your search.

Here are two blog posts by Eric Wendelin which nicely illustrate the basics of these two commands:

Find is a Beautiful Tool
Grep is a Beautiful Tool

There are a number of other great unix utilities for file search, but knowing how to use find and grep is fundamental, as these two utilities can be found on the most basic build of every unix-like machine you come across.

Got a favorite command line hack that uses find or grep? Drop it on us in the comments.

June 15, 2008

Home security with Twitter and a webcam

Shantanu Goel created a cool home security tool using Twitter and a linux application called Motion, a program that will monitor a webcam looking for differences between frames.

When Motion detects movement, it archives a photo of the event and has the option of triggering an external script. Shantanu combined this with a simple curl command that will ping your Twitter account when a motion event occurs. The end result is a tweet that tells you that motion was detected and checking Motion's integrated mini-http server will allow you to see if it's a false alarm or view the intrusion in real time.

I'm going to set this up at work so I can track down who keeps running off with my red stapler.

Keep Tab On Home Security With A Webcam And Twitter
Motion

June 4, 2008

Use video RAM as swap in Linux

If you are into the headless or console experience, there are a couple of ways to put your machine's graphics card to good use. Most new boxes come with a GPU that has a substantial amount of RAM that is normally used for direct rendering. Using the Memory Technology Device (MTD) support in the Linux kernel, you can actually map the video card RAM to a block device and format it for swap use or as a ramdisk.

The Gentoo wiki has detailed instructions for doing this. The only tricky part is determining the video memory address, but after that it's a simple modprobe to load the MTD driver and you can run mkswap/swapon on the device just as if you were creating a normal swap disk. Considering many machines have 512MB of video RAM and it's waaaaay faster than disk, this could give you a pretty huge performance boost.

You can still use your graphics card in X, but you'll need to reserve a small chunk of that RAM for normal graphics use, use the VESA driver, and add inform the driver that it should only use that teensy portion of memory. "VideoRam 4096" in the XF86Config, for instance, will let you use your card in X and only eat the first 4MB of RAM. Everything after that 4MB is fair game for swap. Michal Schulz wrote a bit about calculating the memory address offsets to make this all work. It's the second link below, for those of you who aren't hardcore enough to deal with only the command line.

Use Memory On Video Card As Swap
Configuring X11 With A VRAM Storage Device

May 14, 2008

Debian/Ubuntu users: update your SSL keys and certs

It was announced yesterday that sometime back in September 2006 a line of code was removed from the Debian distributed OpenSSL package. That one line of code was responsible for causing an uninitialized data warning in Valgrind. It also seeded the random number generator used by OpenSSL. Without it, the error went away, but the keyspace used by affected systems went from 2^1024 to about 2^15. Oh noes!

A large majority of Debian and Ubuntu systems are affected. To correct the problem, you'll need to not only update OpenSSL, but also revoke and replace any cryptographic keys and certificates that were generated on the affected systems. From the Debian security advisory:

Affected keys include SSH keys, OpenVPN keys, DNSSEC keys, and key material for use in X.509 certificates and session keys used in SSL/TLS connections. Keys generated with GnuPG or GNUTLS are not affected, though.

For most people, this boils down to your ssh server's host key and any public key pairs used for remote ssh authentication. Any keys or certificates generated on the affected machines for SSL/https use also need to be revoked and regenerated. It's pretty ugly, really.

As far as teachable moments go, there's probably a lot to think about here. Software developers have this weird natural tendency to want to fix and reengineer things that aren't even broken. I'd go so far as to say that the desire to reengineer is inversely proportional to a programmer's familiarity and understanding of the code. I think it comes from our intense desire to make sense of things. It's the guru who's able to channel that hacker urge into solving new problems instead of creating new bugs out of old solutions.

DSA-1571-1 openssl -- predictable random number generator
OpenSSL PRNG Debian Toys (more discussion of the problem here)

April 9, 2008

Air on the EeePC

There's a good post on the O'Reilly Rich Internet Application blog about running Air under Linux on the EeePC:

Adobe recently released the first public alpha version of the AIR runtime for Linux on labs. This is great news! I felt compelled to "geek out" with it, and was able to get AIR running on an Asus EeePC, although with a few minor issues.
The Asus EeePC runs a derivative of Xandros with KDE, which is not a supported Linux distribution for AIR. I got it working with a little help from the Adobe forums, and I'm very excited about it. I have never gotten into Linux desktop application development, but I think that's could soon change.

There's a thread on the Adobe forums that has guidance for running Air on Linux machines. It's a simple matter of downloading the SDK and running your applications from the command line using the adl command like so:

~/AIR-SDK/bin/adl -nodebug ~/app/META-INF/AIR/application.xml ~/app

The AIR runtime for Linux release notes are pretty clear that this is still a pretty alpha product with some unfinished features, but it's something, and if you do a lot of AIR or traditional Flash development, this would be a cool way to include Linux as a build target for your next desktop application.

AIR + Linux + EeePC [via Lebon Bon Lebon]
Adobe AIR for Linux
Running AIR on Linux (Adobe forum)

April 4, 2008

Add keystroke user verification to Gnome

Nathan Harrington amended the GNOME Desktop Manager to include keystroke dynamics in the user verification process. When the user enters their username, the timings between key press events are measured and compared against a stored pattern. The theory is that there is a significant difference in timings for words typed by different individuals, so the way a username is entered provides a bit of extra "fingerprint" information that can be used to help authenticate a user.

I'm not sure how immediately useful this will be, since this particular example won't affect other login methods, such as an ssh session. Nevertheless, the idea is pretty cool and the code is all there for you to monkey around with.

Identify and verify users based on how they type [via slashdot]

April 1, 2008

USB CapsLocker and Sun keyboard simulation

Of all the April Fools pranks that I came across today, the Stealth USB CapsLocker was my favorite. The tiny AVR-driven USB device sends random caps lock keypresses to a PC via a USB interface. The user will see their caps lock light come on from time to time and think they've accidentally hit that most useless key on the keyboard.

Then they might see the Caps Lock light turn on by itself. Next is a sequence of reboots, bashing the keyboard on the desk, clicking through the Control Panel, possibly even replacing the keyboard. Unless they notice the tiny little device sitting in one of the USB ports on the back of their computer, nothing will help.

Equally as cruel, but slightly less technical, would be to switch someone's keyboard mapping to be like the old Sun keyboards (with the control and caps lock key positions swapped).

Be careful, though. There might be some cranky old unix guru who actually appreciates this configuration.

Stealth USB CapsLocker
EasyLogger - example AVR USB keyboard input device
Remap Caps Lock

March 26, 2008

BATMAN: adhoc mesh routing

BATMAN (Better Approach To Mobile Ad-hoc Networking) is a routing protocol designed for multi-hop ad-hoc mesh networks. When you run BATMAN on routers in an ad-hoc network, the nodes in the network constantly send out little broadcast packets that are picked up and re-broadcast by nearby machines. Rather than have each node develop a formal map of the network, they can figure out the most reliable routes to other machines in the network based on the speed and reliability of broadcast packets that they receive from other nodes.

You can imagine a scenario where router A might be a single hop away from the uplink router U, but the connection is somewhat unreliable or drops packets from time to time. If router B has a solid connection to U and also has a reliable connection to A, it might be a faster and more reliable to route A's packets through B, even though it's ultimately 2 hops to U. The way BATMAN works, router A would receive U's broadcast packets more frequently from B (due to the U<->A packet loss), which would cause it to automatically send outbound data through the more reliable B connection.

It looks like this might be fun to experiment with a neighborhood network or even in a larger home with poor coverage. BATMAN is available in OpenWRT, so you could scatter a number of cheap routers throughout an area, give one of them a DSL uplink, and have solid wireless laptop connectivity wherever you want it.

If you really want to get crazy, you can run the routing protocol on your Linux laptops too, making them full mesh participants and expanding the coverage area wherever you go.

B.A.T.M.A.N.
Using BATMAN with OpenWRT

March 23, 2008

Run Safari in Ubuntu

The Ubuntu Unleashed blog has a simple guide for getting Safari to run in Ubuntu. You basically install the Windows version of Safari under WINE, copy over a few core Windows fonts to your WINE install and it just works. You can even install the flash plugin.

I'm not positive that I wouldn't feel a little dirty running closed software on a Linux desktop, but considering Safari is still my preferred browser under OS X (much to the embarrassment of some of my coworkers), I can understand why this could be cool for a lot of folks.

Howto: Install Safari on Ubuntu with Flash

March 17, 2008

CryoPID: hibernation for Linux processes

We're all familiar with the hibernate/deep-sleep features that are typical on your standard laptop. In this mode, the entire contents of RAM are written to the disk and the machine is completely shut down. When it's next booted, the system is restored to the exact state it was at before sleep, with all of your programs running just like they were when you left them.

What if you could do this at the process level? You could kill whatever umpteen-gazillion applications you have running, reboot your computer, and then start your apps back up whenever you like and they would be exactly the way they were when you left them.

There's a Linux application called CryoPID which attempts to do just that.

CryoPID requires no special kernel modifications and operates in user mode, so you don't need to be root. All you do is run the freeze program on a process you own:

freeze /tmp/savestatefile 1234

This will archive the state of process 1234 into a self-executing, compressed file named /tmp/savestatefile. To start it back up, just run the save file:

/tmp/savestatefile

When this is executed, your application will be restored, relinked to any previously-loaded DLLs, and attached to the file descriptors it had open.

You'll run into some problems with network socket connections you had open, and support for X applications is still only experimental, so the useful scenario is a bit limited, but it's a promising concept and could come in quite handy in the command-line world.

CryoPID - A Process Freezer for Linux

March 11, 2008

N64 emulation: better than the real thing

Racketboy has a great article showing off some of the capabilities of the modern N64 emulator. If your machine is fast enough, most of the available emulators will really give you a noticeable resolution boost and better looking anti-aliased models. Using the Rice Video plugin with the Project64 emulator, you can even swap out the textures for some games with user-created texture packs.

I still use the real hardware (is the N64 considered "retro" now?), so before seeing this, I hadn't even considered emulation for this platform. That all changed when I saw the Mario64 mod shown above. The selection of available emulators is impressive, and there are open source emulators available for just about every platform. I'm currently playing a game under Mupen64 on my iMac and it's pretty flawless. My only wish is that all computers came, by default, with a nice joystick like they did back in the 80s.

Enhance N64 Graphics With Emulation Plugins & Texture Packs - Link
Project64 Emulator - Link
Rice Video Plugin - Link
Mupen64 Emulator (cross-platform, open source) - Link

March 9, 2008

Command line Twitter

You can easily update your Twitter status from the command line using cURL. The Tech-Recipes blog posted this handy command line hack:

With cURL installed, you can post to Twitter from the terminal window by using the following syntax:

curl -u yourusername:yourpassword -d status="Your Message Here" http://twitter.com/statuses/update.xml

You will receive a response containing the XML coding for your post which acts as a confirmation that your post was submitted.

Consider this: instant messaging is the new talk (phone for my VMS peeps) and Twitter is the new finger. It's nice to see at least one of these handy communication tools make its way back to the command line.

Posting to Twitter from the Terminal Window - Link
cURL downloads - Link