Loading Templates with Django's app_directories.Loader

I don't really like having one monolithic templatee folder. You end in with a scenario where you have to go digging, in a completely separate path for the template that goes with a given view, and that can be very disrupting when you're in the middle of working on something.

In my current proejct I wanted to use Django's app_directories loader to try to keep the templates physically close to the views that use them. Then I started working with it and I realized how annoying it'd be to use!

When I create a listing page I generally name the template file index.html or maybe listing.html depending on it's usage. Edit pages are normally edit.html, details pages are normally details.html and so on. However, the app_directories's loader class doesn't differentiate in any way between your different modules. This is the root of my issue.

In some magical world it'd use the source of the call to identify what was the most likely module, and then maybe do some guesses for likely fallbacks. But that's not really sane to try to implement, hell I don't even think it's really possible.

The official method is to build subdirectories under the application template folders. This means you'd have projectname/appname/templates/appname/index.html or projectname/appname/subappname/templates/appname/subappname/index.html. The other common 'fix' for this is to prefix your filenames so they'd all be appname-index.html or appname-subappname.html and the like.

I don't really care for either. The pathing solution ends up with a lot of redundancy, and you still have to maintain the kinda structure I'm wanting to avoid, and even if the structure is localized it's still annoying to have to go through. The prefixing just seems sloppy, and if you ever do accidentally have a conflicting name, which can happen, you could have a potentially hard to spot bug.

The best idea I've thought of so far to fix this is to have your render methods look like this: render_to_string(template_app, template_name, dictionary=None, context_instance=None). This way you just pass in the app name when you call it. Now I don't care for the inherent redundancy this would cause either, but it seems to me to be the lesser of the evils.

The idea had me digging into the internals of how Django works with loading templates and what I needed to do to make this really work. What I found wasn't that bad. The Loader api is pretty simple, however, there was a few undocumented and non obvious requirements. Also you can't circumvent the Loader system completely as that breaks other facets of the framework. Some of this would have been extremely painful to figure out if PyCharm didn't have such a great debugger.

It's pretty straight forward to use.

TEMPLATE_LOADERS = (('appname.common.templates.AppSpecificLoader',),)

if not DEBUG:
    TEMPLATE_LOADERS = (('django.template.loaders.cached.Loader',
        ('appname.common.templates.AppSpecificLoader',)),
    )

Then you just need to import the render method and use it instead of the default, alternatively there is also a django-annoying inspired decorator that can be used to revent the redundant code many of us have for building the response object.

Here is a link to the code.

Heroku, Django, and Static Files

I've been continuing to mess with Heroku and it's been a blast to just have someplace to host on that's so bloody easy to get going. As part of this I've also decided to give Django a second chance.

My History with Django

A few years back I used Django a lot when working on Curse.com and CurseForge. We were using Django 0.7, and in fact due to some poor planning by certain members of the dev team we worked off of a random subversion revision. It was then forked and patched with a lot of custom changes that we depended heavily on and all without any record keeping of what version we were currently based on. It became such a monumental task to try to upgrade that we eventually stopped trying.

That paralysis ended up costing a lot in the long run. We were unable to upgrade and as a result were no longer able to take advantage of the growth and maturation of the platform. Eventually we relaunched Curse.com as a C# .Net website and CurseForge eventually became based on a lower level stack based on Werkzeug, Jinja2, SQLAlchemy, and WTForms.

Curse.com is a huge site, it does millions of page views on an average day, and on a busy day (World of Warcraft patch days for example) it can do tens of millions. At the time we ran into a lot of issues with scaling a Django site, but as I said that was on a random development version in the 0.7 series. The whole experience left me with a pretty bad taste in my mouth as it where. Recently I've heard it's gotten a lot better and I wanted to see for myself.

And today

Now Django is on 1.4 and a lot of things have changes, both small and subtle and major and obvious. One of the more significant changes I've seen so far is how static files are handled. They've added and abstracted storage backend system that's actually pretty nice.

The default backend works with the filesystem. It takes discovered files and puts them into a folder determined by the STATIC_ROOT setting. This is kinda messy on Heroku, but can absolutely be used. Recently Heroku even started running manage.py collectstatic for you. Despite that being a viable solution, I don't particularly care fore that.

I found a lot of people talking about using S3 and django-storages for static files. So I gave it a go and was able to get all my files collected directly into S3 pretty quickly, but there was some things I didn't like. I found, for example that the admin interface didn't play well with using S3's vanity domains, and the documentation on django-storages was ungodly frustrating. It only documented a handful of config options, but after inspecting the code it actually has twenty.

I wanted something simpler, and so I did the not-so-civically-responsible thing, and wrote my own simple version of the S3Storage. So after a few evenings I have a very simple, working backend. It does make a lot more assumptions, and it just plain out doesn't bother having a lot of settings. I've put it up on pypi, and I'm going to be putting some documentation here soon.

Having a blog and the fear of shipping

So I started this blog once before. And it set there for months without any content in it. That was back in 2009. I started it again last month, and it set again, empty, for more than a month. Last night I finally decided to say screw it and post something. So I posted about what I've been doing with web development and Heroku, but this whole experience has got me thinking about why the hell have I not posted something previously.

I think if you'd actually want to figure out why you'd have to listen to a lot of Merlin Mann's Back to Work. I've listened to every episode and he's talked a lot about how it can be very hard to ship, but that shipping something is what's important. And it's so very true.

It doesn't matter how good your idea is, or even how good your implementation is. If you never ship it it really doesn't matter. So I'm going to ship, and keep shipping, and hopefully something worthwhile at least occasionally comes out. If the quality isn't always there or if it doesn't have anything useful in it I'm... you know I can't even say I'm sorry, but I hope it's not that bad, or at least more bad than good.

I'm done any fear of being afraid to ship, for better or worse.

Heroku, two days in

I've spent the last few days I've spent some of my free time looking at an Infrastructure as a Service platform named Heroku. I'm probably just one of many people who've written similar posts to this, but what's the point of having a blog if you're worried to say what's on your mind.

I've setup a few simple projects so far. It's insanely easy to get started. Which is great. The part that concerns me about it is the financial viability of hosting a large site there. Past about the $50/month range (two dynamos and a 20GB postgres instance) it seems that the time cost of migrating from Heroku to a service such as Linode is likely well worth it.

Looking at some of the specifics of the implementation it seems that the Heroku is really just a very easy to use wrapper around Amazon Web Services. Which is really neat from a technical achievement perspective, and also explains the pretty aggressive cost scaling: 

I think I'm going to follow this up with more posts. It's been a lot of fun to get some simple projects up and going and I'm trying to spend some more of my personal time writing code again. I'm doing more and more IT/SysOps at Curse over the last few years and I think that long term I'll be doing that full time. While I am a fan of DevOps it's something that still only limits it to so much code, and I don't want to get too dull.