Heroku, Django, and Static Files

I've been continuing to mess with Heroku and it's been a blast to just have someplace to host on that's so bloody easy to get going. As part of this I've also decided to give Django a second chance.

My History with Django

A few years back I used Django a lot when working on Curse.com and CurseForge. We were using Django 0.7, and in fact due to some poor planning by certain members of the dev team we worked off of a random subversion revision. It was then forked and patched with a lot of custom changes that we depended heavily on and all without any record keeping of what version we were currently based on. It became such a monumental task to try to upgrade that we eventually stopped trying.

That paralysis ended up costing a lot in the long run. We were unable to upgrade and as a result were no longer able to take advantage of the growth and maturation of the platform. Eventually we relaunched Curse.com as a C# .Net website and CurseForge eventually became based on a lower level stack based on Werkzeug, Jinja2, SQLAlchemy, and WTForms.

Curse.com is a huge site, it does millions of page views on an average day, and on a busy day (World of Warcraft patch days for example) it can do tens of millions. At the time we ran into a lot of issues with scaling a Django site, but as I said that was on a random development version in the 0.7 series. The whole experience left me with a pretty bad taste in my mouth as it where. Recently I've heard it's gotten a lot better and I wanted to see for myself.

And today

Now Django is on 1.4 and a lot of things have changes, both small and subtle and major and obvious. One of the more significant changes I've seen so far is how static files are handled. They've added and abstracted storage backend system that's actually pretty nice.

The default backend works with the filesystem. It takes discovered files and puts them into a folder determined by the STATIC_ROOT setting. This is kinda messy on Heroku, but can absolutely be used. Recently Heroku even started running manage.py collectstatic for you. Despite that being a viable solution, I don't particularly care fore that.

I found a lot of people talking about using S3 and django-storages for static files. So I gave it a go and was able to get all my files collected directly into S3 pretty quickly, but there was some things I didn't like. I found, for example that the admin interface didn't play well with using S3's vanity domains, and the documentation on django-storages was ungodly frustrating. It only documented a handful of config options, but after inspecting the code it actually has twenty.

I wanted something simpler, and so I did the not-so-civically-responsible thing, and wrote my own simple version of the S3Storage. So after a few evenings I have a very simple, working backend. It does make a lot more assumptions, and it just plain out doesn't bother having a lot of settings. I've put it up on pypi, and I'm going to be putting some documentation here soon.

Heroku, two days in

I've spent the last few days I've spent some of my free time looking at an Infrastructure as a Service platform named Heroku. I'm probably just one of many people who've written similar posts to this, but what's the point of having a blog if you're worried to say what's on your mind.

I've setup a few simple projects so far. It's insanely easy to get started. Which is great. The part that concerns me about it is the financial viability of hosting a large site there. Past about the $50/month range (two dynamos and a 20GB postgres instance) it seems that the time cost of migrating from Heroku to a service such as Linode is likely well worth it.

Looking at some of the specifics of the implementation it seems that the Heroku is really just a very easy to use wrapper around Amazon Web Services. Which is really neat from a technical achievement perspective, and also explains the pretty aggressive cost scaling: 

I think I'm going to follow this up with more posts. It's been a lot of fun to get some simple projects up and going and I'm trying to spend some more of my personal time writing code again. I'm doing more and more IT/SysOps at Curse over the last few years and I think that long term I'll be doing that full time. While I am a fan of DevOps it's something that still only limits it to so much code, and I don't want to get too dull.