Heroku, Django, and Static Files

I've been continuing to mess with Heroku and it's been a blast to just have someplace to host on that's so bloody easy to get going. As part of this I've also decided to give Django a second chance.

My History with Django

A few years back I used Django a lot when working on Curse.com and CurseForge. We were using Django 0.7, and in fact due to some poor planning by certain members of the dev team we worked off of a random subversion revision. It was then forked and patched with a lot of custom changes that we depended heavily on and all without any record keeping of what version we were currently based on. It became such a monumental task to try to upgrade that we eventually stopped trying.

That paralysis ended up costing a lot in the long run. We were unable to upgrade and as a result were no longer able to take advantage of the growth and maturation of the platform. Eventually we relaunched Curse.com as a C# .Net website and CurseForge eventually became based on a lower level stack based on Werkzeug, Jinja2, SQLAlchemy, and WTForms.

Curse.com is a huge site, it does millions of page views on an average day, and on a busy day (World of Warcraft patch days for example) it can do tens of millions. At the time we ran into a lot of issues with scaling a Django site, but as I said that was on a random development version in the 0.7 series. The whole experience left me with a pretty bad taste in my mouth as it where. Recently I've heard it's gotten a lot better and I wanted to see for myself.

And today

Now Django is on 1.4 and a lot of things have changes, both small and subtle and major and obvious. One of the more significant changes I've seen so far is how static files are handled. They've added and abstracted storage backend system that's actually pretty nice.

The default backend works with the filesystem. It takes discovered files and puts them into a folder determined by the STATIC_ROOT setting. This is kinda messy on Heroku, but can absolutely be used. Recently Heroku even started running manage.py collectstatic for you. Despite that being a viable solution, I don't particularly care fore that.

I found a lot of people talking about using S3 and django-storages for static files. So I gave it a go and was able to get all my files collected directly into S3 pretty quickly, but there was some things I didn't like. I found, for example that the admin interface didn't play well with using S3's vanity domains, and the documentation on django-storages was ungodly frustrating. It only documented a handful of config options, but after inspecting the code it actually has twenty.

I wanted something simpler, and so I did the not-so-civically-responsible thing, and wrote my own simple version of the S3Storage. So after a few evenings I have a very simple, working backend. It does make a lot more assumptions, and it just plain out doesn't bother having a lot of settings. I've put it up on pypi, and I'm going to be putting some documentation here soon.

Having a blog and the fear of shipping

So I started this blog once before. And it set there for months without any content in it. That was back in 2009. I started it again last month, and it set again, empty, for more than a month. Last night I finally decided to say screw it and post something. So I posted about what I've been doing with web development and Heroku, but this whole experience has got me thinking about why the hell have I not posted something previously.

I think if you'd actually want to figure out why you'd have to listen to a lot of Merlin Mann's Back to Work. I've listened to every episode and he's talked a lot about how it can be very hard to ship, but that shipping something is what's important. And it's so very true.

It doesn't matter how good your idea is, or even how good your implementation is. If you never ship it it really doesn't matter. So I'm going to ship, and keep shipping, and hopefully something worthwhile at least occasionally comes out. If the quality isn't always there or if it doesn't have anything useful in it I'm... you know I can't even say I'm sorry, but I hope it's not that bad, or at least more bad than good.

I'm done any fear of being afraid to ship, for better or worse.

Heroku, two days in

I've spent the last few days I've spent some of my free time looking at an Infrastructure as a Service platform named Heroku. I'm probably just one of many people who've written similar posts to this, but what's the point of having a blog if you're worried to say what's on your mind.

I've setup a few simple projects so far. It's insanely easy to get started. Which is great. The part that concerns me about it is the financial viability of hosting a large site there. Past about the $50/month range (two dynamos and a 20GB postgres instance) it seems that the time cost of migrating from Heroku to a service such as Linode is likely well worth it.

Looking at some of the specifics of the implementation it seems that the Heroku is really just a very easy to use wrapper around Amazon Web Services. Which is really neat from a technical achievement perspective, and also explains the pretty aggressive cost scaling: 

I think I'm going to follow this up with more posts. It's been a lot of fun to get some simple projects up and going and I'm trying to spend some more of my personal time writing code again. I'm doing more and more IT/SysOps at Curse over the last few years and I think that long term I'll be doing that full time. While I am a fan of DevOps it's something that still only limits it to so much code, and I don't want to get too dull.