mikeash.com: just this guy, you know?

Posted at 2015-02-06 14:23 | RSS feed (Full text feed) | Blog Index
Next article: Friday Q&A 2015-02-20: Let's Build @synchronized
Previous article: Friday Q&A 2015-01-23: Let's Build Swift Notifications
Tags: fridayqna swift threading
Friday Q&A 2015-02-06: Locks, Thread Safety, and Swift
by Mike Ash  
This article is also available in Bosnian (translation by Vlada Catalic) and Macedonian (translation by Vlada Catalic).

An interesting aspect of Swift is that there's nothing in the language related to threading, and more specifically to mutexes/locks. Even Objective-C has @synchronized and atomic properties. Fortunately, most of the power is in the platform APIs which are easily used from Swift. Today I'm going to explore the use of those APIs and the transition from Objective-C, a topic suggested by Cameron Pulsford.

A Quick Recap on Locks
A lock, or mutex, is a construct that ensures only one thread is active in a given region of code at any time. They're typically used to ensure that multiple threads accessing a mutable data structure all see a consistent view of it. There are several kinds of locks:

  1. Blocking locks sleep a thread while it waits for another thread to release the lock. This is the usual behavior.
  2. Spinlocks use a busy loop to constantly check to see if a lock has been released. This is more efficient if waiting is rare, but wastes CPU time if waiting is common.
  3. Reader/writer locks allow multiple "reader" threads to enter a region simultaneously, but exclude all other threads (including readers) when a "writer" thread acquires the lock. This can be useful as many data structures are safe to read from multiple threads simultaneously, but unsafe to write while other threads are either reading or writing.
  4. Recursive locks allow a single thread to acquire the same lock multiple times. Non-recursive locks can deadlock, crash, or otherwise misbehave when re-entered from the same thread.

APIs
Apple's APIs have a bunch of different mutex facilities. This is a long but not exhaustive list:

  1. pthread_mutex_t.
  2. pthread_rwlock_t.
  3. dispatch_queue_t.
  4. NSOperationQueue when configured to be serial.
  5. NSLock.
  6. OSSpinLock.

In addition to this, Objective-C provides the @synchronized language construct, which at the moment is implemented on top of pthread_mutex_t. Unlike the others, @synchronized doesn't use an explicit lock object, but rather treats an arbitrary Objective-C object as if it were a lock. A @synchronized(someObject) section will block access to any other @synchronized sections that use the same object pointer. These different facilities all have different behaviors and capabilities:

  1. pthread_mutex_t is a blocking lock that can optionally be configured as a recursive lock.
  2. pthread_rwlock_t is a blocking reader/writer lock.
  3. dispatch_queue_t can be used as a blocking lock. It can be used as a reader/writer lock by configuring it as a concurrent queue and using barrier blocks. It also supports asynchronous execution of the locked region.
  4. NSOperationQueue can be used as a blocking lock. Like dispatch_queue_t it supports asynchronous execution of the locked region.
  5. NSLock is blocking lock as an Objective-C class. Its companion class NSRecursiveLock is a recursive lock, as the name indicates.
  6. OSSpinLock is a spinlock, as the name indicates.

Finally, @synchronized is a blocking recursive lock.

Value Types
Note that pthread_mutex_t, pthread_rwlock_t, and OSSpinLock are value types, not reference types. That means that if you use = on them, you make a copy. This is important, because these types can't be copied! If you copy one of the pthread types, the copy will be unusable and may crash when you try to use it. The pthread functions that work with these types assume that the values are at the same memory addresses as where they were initialized, and putting them somewhere else afterwards is a bad idea. OSSpinLock won't crash, but you get a completely separate lock out of it which is never what you want.

If you use these types, you must be careful never to copy them, whether explicitly with a = operator, or implicitly by, for example, embedding them in a struct or capturing them in a closure.

Additionally, since locks are inherently mutable objects, this means you need to declare them with var instead of let.

The others are reference types, meaning they can be passed around at will, and can be declared with let.

Initialization
Update 2015-02-10: the problems described in this section have been made obsolete with breathtaking speed. Apple released Xcode 6.3b1 yesterday which includes Swift 1.2. Among other changes, C structs are now imported with an empty initializer that sets all fields to zero. In short, you can now write `pthread_mutex_t()` without the extensions I discuss below. This section will remain for historical interest, but is no longer relevant to the language.

The pthread types are troublesome to use from Swift. They're defined as opaque structs with a bunch of storage, such as:

    struct _opaque_pthread_mutex_t {
        long __sig;
        char __opaque[__PTHREAD_MUTEX_SIZE__];
    };

The intent is that you declare them, then initialize them using an init function that takes a pointer to the storage and fills it out. In C, it looks like:

    pthread_mutex_t mutex;
    pthread_mutex_init(&mutex, NULL);

This works fine, as long as you remember to call pthread_mutex_init. However, Swift really, really dislikes uninitialized variables. The equivalent Swift fails to compile:

    var mutex: pthread_mutex_t
    pthread_mutex_init(&mutex, nil)
    // error: address of variable 'mutex' taken before it is initialized

Swift requires variables to be initialized before they're used. pthread_mutex_init doesn't use the value of the variable passed in, it just overwrites it, but Swift doesn't know that and so it produces an error. To satisfy the compiler, the variable needs to be initialized with something, but that's harder than it looks. Using () after the type doesn't work:

    var mutex = pthread_mutex_t()
    // error: missing argument for parameter '__sig' in call

Swift requires values for those opaque fields. __sig is easy, we can just pass zero. __opaque is a bit more annoying. Here's how it gets bridged into Swift:

    struct _opaque_pthread_mutex_t {
        var __sig: Int
        var __opaque: (Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8,
                       Int8, Int8, Int8, Int8)
    }

There's no easy way to get a big tuple full of zeroes, so you have to write it all out:

    var mutex = pthread_mutex_t(__sig: 0,
                             __opaque: (0, 0, 0, 0, 0, 0, 0, 0,
                                        0, 0, 0, 0, 0, 0, 0, 0,
                                        0, 0, 0, 0, 0, 0, 0, 0,
                                        0, 0, 0, 0, 0, 0, 0, 0,
                                        0, 0, 0, 0, 0, 0, 0, 0,
                                        0, 0, 0, 0, 0, 0, 0, 0,
                                        0, 0, 0, 0, 0, 0, 0, 0))

This is awful, but I couldn't find a good way around it. The best I could do was wrap it up in an extension so that the empty () works. Here are the two extensions I made:

    extension pthread_mutex_t {
        init() {
            __sig = 0
            __opaque = (0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0)
        }
    }

    extension pthread_rwlock_t {
        init() {
            __sig = 0
            __opaque = (0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0,
                        0, 0, 0, 0, 0, 0, 0, 0)
        }
    }

With these extensions, this works:

    var mutex = pthread_mutex_t()
    pthread_mutex_init(&mutex, nil)

It may be possible to roll the call to pthread_mutex_init into the extension initializer as well, but there's no guarantee that self in a struct init points to the variable being initialized. Since these values can't be moved in memory after being initialized, I wanted to keep the initialization as a separate call.

Locking Wrappers
To make these different APIs easier to use, I wrote a series of small wrapper functions. I settled on with as a convenient, short, syntax-looking name inspired by Python's with statement. Swift's function overloading allows using the same name for all these different types. The basic form looks like this:

    func with(lock: SomeLockType, f: Void -> Void) { ...

This then executes f with the lock held. Let's implement it for all these types.

For the value types, it needs to take a pointer to the lock so the lock/unlock functions can modify it. The implementation for pthread_mutex_t just calls the appropriate lock and unlock functions, with a call to f in between:

    func with(mutex: UnsafeMutablePointer<pthread_mutex_t>, f: Void -> Void) {
        pthread_mutex_lock(mutex)
        f()
        pthread_mutex_unlock(mutex)
    }

The implementation for pthread_rwlock_t is almost identical:

    func with(rwlock: UnsafeMutablePointer<pthread_rwlock_t>, f: Void -> Void) {
        pthread_rwlock_rdlock(rwlock)
        f()
        pthread_rwlock_unlock(rwlock)
    }

I made a companion to this one that takes a write lock, which again looks much the same:

    func with_write(rwlock: UnsafeMutablePointer<pthread_rwlock_t>, f: Void -> Void) {
        pthread_rwlock_wrlock(rwlock)
        f()
        pthread_rwlock_unlock(rwlock)
    }

The one for dispatch_queue_t is even simpler. It's just a wrapper around dispatch_sync:

    func with(queue: dispatch_queue_t, f: Void -> Void) {
        dispatch_sync(queue, f)
    }

In fact, if one were too clever for one's own good and wanted to confuse people, one could take advantage of the functional nature of Swift and simply write:

    let with = dispatch_sync

This is unwise for a couple of reasons, not the least of which being that it messes with the type-based overloading we're trying to use here.

NSOperationQueue is conceptually similar, but there's no direct equivalent to dispatch_sync. Instead, we create an operation, add it to the queue, and explicitly wait for it to finish:

    func with(opQ: NSOperationQueue, f: Void -> Void) {
        let op = NSBlockOperation(f)
        opQ.addOperation(op)
        op.waitUntilFinished()
    }

The implementation for NSLock looks like the pthread versions, just with slightly different locking calls:

    func with(lock: NSLock, f: Void -> Void) {
        lock.lock()
        f()
        lock.unlock()
    }

Finally, the OSSpinLock implementation is again more of the same:

    func with(spinlock: UnsafeMutablePointer<OSSpinLock>, f: Void -> Void) {
        OSSpinLockLock(spinlock)
        f()
        OSSpinLockUnlock(spinlock)
    }

Imitating @synchronized
With these wrappers, imitating the basics of @synchronized is fairly simple. Add a property to your class that holds a lock, then use with where you would have used @synchronized before:

    let queue = dispatch_queue_create("com.example.myqueue", nil)

    func setEntryForKey(key: Key, entry: Entry) {
        with(queue) {
            entries[key] = entry
        }
    }

Getting data out of the block is a bit less pleasant, unfortunately. While @synchronized lets you return from within, that doesn't work with with. Instead, you have to use a var and assign to it within the block:

    func entryForKey(key: Key) -> Entry? {
        var result: Entry?
        with(queue) {
            result = entries[key]
        }
        return result
    }

It should be possible to wrap the boilerplate in a generic function, but I had trouble getting the Swift compiler's type inference to play along and don't have a solution just yet.

Imitating Atomic Properties
Atomic properties are not often useful. The problem is that, unlike many other useful properties of code, atomicity doesn't compose. For example, if function f doesn't leak memory, and function g doesn't leak memory, then function h that just calls f and g also doesn't leak memory. The same is not true of atomicity. For an example, imagine you have a set of atomic, thread-safe Account classes:

    let checkingAccount = Account(amount: 100)
    let savingsAccount = Account(amount: 0)

Now you move the money to savings:

    checkingAccount.withDraw(100)
    savingsAccount.deposit(100)

In another thread, you total up the balance and tell the user:

    println("Your total balance is: \(checkingAccount.amount + savingsAccount.amount)")

If this runs at just the wrong time, it will print zero instead of 100, despite the fact that the Account objects themselves are fully atomic and the user had a balance of 100 the whole time. Because of this, it's usually better to build entire subsystems to be atomic, rather than individual properties.

There are rare cases where atomic properties are useful, because it really is a standalone thing that just needs to be thread safe. To achieve that in Swift, you need a computed property that will do the locking, and a second normal property that will actually hold the value:

    private let queue = dispatch_queue_create("...", nil)
    private var _myPropertyStorage: SomeType

    var myProperty: SomeType {
        get {
            var result: SomeType?
            with(queue) {
                result = _myPropertyStorage
            }
            return result!
        }
        set {
            with(queue) {
                _myPropertyStorage = newValue
            }
        }
    }

Choosing Your Lock API
The pthread APIs can be discounted immediately due to the difficulty of using them from Swift, and the fact that they don't do anything that other APIs don't also do. I often like to use them in C and Objective-C because they're fairly straightforward and fast, but it's not worth it here unless something really requires it.

Reader/writer locks are mostly not worth worrying about. For common cases, where reads and writes are quick, the extra overhead used by a reader/writer lock outweighs the ability to have multiple concurrent readers.

Recursive locks are mostly an invitation to deadlock. There are cases where they're useful, but if you find yourself with a design where you need to take a lock that's already locked on the current thread, that's a good sign you should probably rethink it so that's not necessary.

My opinion is that, when in doubt, default to dispatch_queue_t. They're more heavyweight, but this rarely matters. The API is reasonably convenient, and they ensure you never forget to pair a lock call with an unlock call. They provide a ton of nice facilities that can come in handy, like the ability to use a single dispatch_async call to run locked code in the background, or the ability to set up timers or other event sources targeted directly at the queue so that they automatically execute locked. You can even use it as a target for things like NSNotificationCenter observers and NSURLSession delegates by using the underlyingQueue property of NSOperationQueue, new in OS X 10.10 and iOS 8.

NSOperationQueue wishes it could be as cool as dispatch_queue_t and there are few if any reasons to use it as a lock API. It's more cumbersome to use and doesn't provide any advantages for typical use as a locking API, although the automatic dependency management for operations can sometimes be useful in other contexts.

NSLock is a simple locking class that's easy to use and reasonably fast. It's a good choice if you want explicit lock and unlock calls for some reason, rather than the blocks-based API of dispatch_queue_t, but there's little reason to use it in most cases.

OSSpinLock is an excellent choice for uses where the lock is taken often, contention is low, and the locked code runs quickly. It has much lower overhead and this helps performance for hot code paths. On the other hand, it's a bad choice for uses where code may hold the lock for a substantial amount of time, or contention is common, as it will waste CPU time. In general, default to dispatch_queue_t, but keep OSSpinLock in mind as a fairly easy optimization if it starts to show up in the profiler.

Conclusion
Swift has no language facilities for thread synchronization, but this deficiency is more than made up for the wealth of locking APIs available in Apple's frameworks. GCD and dispatch_queue_t are still a masterpiece and the API works great in Swift. We don't have @synchronized or atomic properties, but we have things that are better.

That's it for today. Come back next time for more exciting adventures. Friday Q&A is built on the topic suggestions of readers like you, so if you have something you'd like to see covered here, please send it in!

Did you enjoy this article? I'm selling whole books full of them! Volumes II and III are now out! They're available as ePub, PDF, print, and on iBooks and Kindle. Click here for more information.

Comments:

I think you're missing an opportunity to copy from C++11's templated locks to get even cleaner Swift behavior. There would be no need for with() and no need to create temporary block/lamba functions.

// this is all provided in the std lib, but is extensible
class mutex {...; lock(); unlock(); };

template <class Mutex> class lock_guard {
Mutex& ref;
lock_guard(Mutex& m) : ref(m) { m.lock(); }
~lock_guard() { ref.unlock(); }
};

Now, all locking is just done with normal block scope {} and you're return value problem is solved. And its exception safe.

From what I've read of Swift this should all be possible.

lock myLock; // global or class

int val;
{
lock_guard<mutex> guard{myLock}; // lock taken
val = dowork();
} // implicit unlock when scope exits via guard destructor
Brian B.: Given that Swift object lifetimes aren't guaranteed to line up with scopes, and structs don't have deinit, how do you propose to implement this?
You missed my favorite one, objc_sync_enter, which is what @synchronized does:


func synchronized<T>(lockObj: AnyObject, closure: ()->T) -> T {
    objc_sync_enter(lockObj)
    var retVal: T = closure()
    objc_sync_exit(lockObj)
    return retVal
}

class MultiThreadedPrinter {
    func doSomething() {
        synchronized(self) {
            println("This will only one thread at a time")
        }
    }
}
Swift does have one narrow synchronization related feature: global variables in a non-main file are lazily initialized in a thread-safe way using dispatch_once. See https://developer.apple.com/swift/blog/?id=7 for more information.

-Chris
Marc P: I forgot those were public API. I don't really like the style of using random objects as a lock, but it can be an easy way to do things. I might explore reimplementing those functions just for fun.
Ah, sorry, Mike. Like I said "what I've read of Swift". For some reason I thought Swift had stack based objects, so the lock_guard itself could be stack based and be cleaned up on scope exit.

For true X-platform, and not just Apple X-OS, Swift is useless as it is not compatible with C++, so we have not been able to use it in practice. We've just been stick with Objc-C++ where we can actually use this pattern in properties and elsewhere. It's pretty nice.
Swift does have stack-based objects, that's what struct is. But there's no deinit available for them, and even if there were there's no guarantee that it will stay alive until the end of the scope.
I like:

class Spinlock {
    private var _lock : OSSpinLock = OS_SPINLOCK_INIT

    func around(code: Void -> Void) {
        self.lock()
        code()
        self.unlock()
    }

    func around<T>(code: Void -> T) -> T {
        self.lock()
        let result = code()
        self.unlock()

        return result
    }
}

With that, you can just have something like:

class MyFoo {
    func whatever() {
        self.lock.around {
            // Non-thread-safe code here...
        }
    }

    var atomicBar : Thingy {
        get { return self.lock.around { self._bar } }
        set(value) { self.lock.around { self._bar = value } }
    }

    private let lock = Spinlock()
    private var _bar : Thingy
}
Whoops, typo. It seems I can't edit my posts. Here's the correct class definition:

class Spinlock {
    private var _lock : OSSpinLock = OS_SPINLOCK_INIT

    func around(code: Void -> Void) {
        OSSpinLockLock(&self._lock)
        code()
        OSSpinLockUnlock(&self._lock)
    }

    func around<T>(code: Void -> T) -> T {
        OSSpinLockLock(&self._lock)
        let result = code()
        OSSpinLockUnlock(&self._lock)

        return result
    }
}

You can of course substitute some other locking primitive for the OSSpinLocks, if you wish. Since I use this primarily around basic property getters & setters, spinlocks are optimal. But I could imagine a little class cluster of these, for spinlocks, vanilla locks & reader/writer locks.
Structs in Swift aren’t stack-based; the language makes no guarantees about the storage of value types. They might be kept in a register; they might be stored on the heap; they might even be broken up and stored in multiple different places.

This is why Swift provides withUnsafeMutablePointer()—within the closure passed to that function, the value is fixed to an address in memory.

So I’m concerned that in your pthread example, the mutex’s storage will indeed move in between calling pthread_mutex_init and pthread_mutex_lock.
Hi, it'd be great if you could file the 0,0,0,0,0.... hack on Apple Radar so that they can fix it in the language. The language really needs a = { 0 } construct like C has to zero-out a struct. or a construct to say "this variable *will* be initialized when I pass it by reference, trust me".
The ugly initialization on pthread_mutex_t et al. seems to be fixed in Swift 1.2 with the new default init on C structs.
I really enjoyed this article - thanks for writing it. It confirmed a lot of what I'd already been experimenting with in Swift, particularly that GCD solves pretty much any concurrency dilemma I've encountered!

Cheers!
Well, multiple-threading is still one of the things I haven't figured out clearly. Need more time to dig into it. Anyway, thanks for sharing a great topic
If you use these types, you must be careful never to copy them, … by … capturing them in a closure.

This is not true for closures. Mutable values captured by closures capture by-reference. If the function escapes the local stack frame, this means the variable is actually stored in a heap-allocated location. In Obj-C, __block variables are only copied to the heap if the block is copied; experimentally, in Swift, it appears to statically determine if the variable needs to be heap-allocated instead of deferring that decision to runtime. As a result, mutable values captured in closures will always have the same address and therefore no copy is made.

As an example, a function that captures a local Int in a closure that it passes to dispatch_async() starts with the following SIL:

bb0:
  %0 = alloc_box $Int // var x // users: %3, %4, %66, %66, %74, %84
  %1 = integer_literal $Builtin.Word, 0 // user: %2
  %2 = struct $Int (%1 : $Builtin.Word) // user: %3
  store %2 to %0#1 : $*Int // id: %3


Here you can see that it's heap-allocating a reference-counted Int (with alloc_box) and initializing the value to 0. This is in contrast to Obj-C, which puts the value on the stack initially and only copies it to the heap when the block is copied.

Also important to note, if you use a capture list, that does copy the captured values. Capture lists capture by-value instead of by-reference. Similarly, if you capture an immutable value (a let instead of a var) Swift will also capture by-value instead of by-reference as an optimization, but this is not an observable change (you can't make a pointer to an immutable value), and is not particularly relevant as your lock needs to be mutable to work (since the APIs take pointers).

Granted, I don't think the Swift book guarantees this precise capture-by-reference behavior (specifically, that it doesn't behave like Obj-C __block; the fact that it's by-reference is guaranteed). But it seems very unlikely to change.
I just love you, the way you explain this stuff.

I wish I could have proposed you for marriage!
What construct do you recommend to replace volatile variables with in Swift?

Thank you
Elle
With Swift 2, we can now use the defer statement for releasing the lock. For example:

func synchronize(lock: AnyObject, @noescape closure: () -> Void) {
    objc_sync_enter(lock)
    defer { objc_sync_exit(lock) }
    closure()
}


This makes sure that the lock is released even if the exception exits prematurely.
I don't believe defer will run the code if an exception is thrown. Swift does its best to pretend that exceptions don't exist, and doesn't generate exception-safe code.

It will work if an error is thrown, of course (you'll have to mark the function as throws or rethrows), and allowing you to directly return the resulting value instead of screwing around with a temporary variable is a nice bonus.
FWIW using struct based C locks (pthread_mutex_t, OSSpinLock, ...) in plain swift variables is unsafe, because the runtime can decide to move these structs at any time.

the only safe construct is to use them in memory that was allocated for that purpose UnsafeMutableMemoryBuffer (I think (?)) backed buffer so that swift can't move them.

Something like this is generally wrong, but will break rarely enough that it only causes issue rarely:


class Lock {
    let mutex: pthread_mutex_t
    init() {
        mutex = pthread_mutex_t()
        pthread_mutex_init(&lock, nil)
    }
    lock() { pthread_mutex_lock(&mutex) }
    unlock() { pthread_mutex_unlock(&mutex) }
    deinit { pthread_mutex_destroy(&mutex) }
}


avoid it
"Recursive locks are mostly an invitation to deadlock. There are cases where they're useful, but if you find yourself with a design where you need to take a lock that's already locked on the current thread, that's a good sign you should probably rethink it so that's not necessary."
Did you mean Non-recursive locks here?

Comments RSS feed for this page

Add your thoughts, post a comment:

Spam and off-topic posts will be deleted without notice. Culprits may be publicly humiliated at my sole discretion.

Name:
The Answer to the Ultimate Question of Life, the Universe, and Everything?
Comment:
Formatting: <i> <b> <blockquote> <code>.
NOTE: Due to an increase in spam, URLs are forbidden! Please provide search terms or fragment your URLs so they don't look like URLs.
Code syntax highlighting thanks to Pygments.
Hosted at DigitalOcean.