Disembrangling Programming

DataStore - an abstraction layer for storing data and processing requests

2011-03-01T00:00:00-08:00

Update 2012/01: I still haven’t started working on this project and I’ve heard much interest from others. Some have offered to pick up the project. I will update this post once the project finds a home.

Motivation
Idea
From past to present, what now?
What does better mean?
Why should you care?
My scenario
What worked for me
Proposed features
Help!

Motivation

I wrote an application which extensively used Ajax requests to communicate with the server. After a while I wanted to add localStorage. Though most actions were grouped together, there were still more than just a handful of Ajax requests that had to be changed. Then I thought what if I want to add IndexedDB later on…

… you can see where this is going: maintenance hell.

Then I thought: other people must be having similar problems. Hence…

Idea

A library to unify all the different data storage/retrieval/sending/receiving API’s such as XMLHttpRequest, WebSockets, localStorage, IndexedDB, and make it easier to use any number of them at once.

From past to present, what now?

Past: Before, all we had was AJAX requests. Really.

To present: With the new technologies coming up in the HTML5 era, we’ve got localStorage and IndexedDB, WebSockets, node.js, and more. Hectic.

What now? Don’t you wish there was a better way to send and receive data in the browser?

What does better mean?

My general goals for this are:

Simple key/value store common abstraction.
Pluggable handlers for each type of send/receive.
Use other abstractions specified in each handler (library surfaces your API as well).
Straightforward way to define flow of data. More on this later.

Anything else you wish it could do?

Why should you care?

Short answer: maintenance, scalability, flexibility.

As these technologies become widely supported, you will start seeing a common problem for websites heavily relying on AJAX (or any kind of data transfer without page reloads): how do you take advantage of them without rewriting your entire codebase every time there’s a new technology (API/storage engine/etc) coming out?

My scenario

The whole reason I got thinking about this was because it happened to me. And it was frustrating.

I had this client-side application using jQuery.ajax requests, and I wanted to take advantage of localStorage for some of them, for data that I didn’t need to get from the server on every page load.

I considered:

Quick’n’dirty: Rewrite these pieces of the application to do both localStorage and ajax requests as fallback.
Slightly better: A library that’s flexible enough for my purposes.
Ideal: A library that would allow me to enable/disable localStorage as an intermediary step on a per-request basis, make it easy to add IndexedDB support later, etc.

What worked for me

The simpler thing I went with was a Data object with a couple of functions.

Example usage:

 1 // main.js
 2 window.data = new DataStore({
 3     url: '/fetch_new_data',
 4     // show a spinny tangy
 5     sync_before: function showSyncInProgress() { ... },
 6     // hide the spinny thingy, maybe show a fading notification
 7     sync_success: function showSyncDone() { ... },
 8     // hide the spinny thingy, definitely show some message
 9     sync_error: function showSyncFailed() { ... }
10 }
11 
12 // example request
13 var i = 0;
14 window.data.process_request({
15     ajax: {url: '/new_comment', type: 'POST',
16            data: $('#comment-form').serialize()},
17     key: 'comment_' + (i++),
18     value: {'author': $('#comment-form .author').val(),
19             'text': $('#comment-form .text').val()}
20 });

ajax.data and value are actually very similar, with an important exception in most applications (e.g. Django): the csrftoken. We don’t need to store that in localStorage for every request. So I chose to keep the two completely separate. You could subclass DataStore and make it save you this extra work per request.

Below is an example implementation (raw file):

  1 /* This depends on Crockford's json2.js
  2  * from https://github.com/douglascrockford/JSON-js
  3  * Options:
  4  *     - url: function()
  5  *     - sync_before: function()
  6  *     - sync_success: function()
  7  *     - sync_error: function()
  8  */
  9 function DataStore(options) {
 10     window.data = this;
 11     this.storage = window.localStorage;
 12     // date of last time we synced
 13     this.last_sync = null;
 14     // queue of requests, populated if offline
 15     this.queue = [];
 16 
 17     /**
 18      * Gets data stored at `key`; `key` is a string
 19      */
 20     this.get_data = function (key) {
 21         var str_data = this.storage.getItem(key);
 22         return JSON.parse(str_data);
 23     }
 24 
 25     /**
 26      * Sets data at `key`; `key` is a string
 27      */
 28     this.set_data = function (key, data) {
 29         var str_data = JSON.stringify(data);
 30         this.storage.setItem(key, str_data);
 31     }
 32 
 33     /**
 34      * Syncs data between local storage and server, depending on
 35      * modifications and online status.
 36      */
 37     this.sync_data = function () {
 38         // must be online to sync
 39         if (!this.is_online()) {
 40             return false;
 41         }
 42 
 43         this.last_sync = this.get_data('last_sync');
 44 
 45         // have we never synced before in this browser?
 46         if (!this.last_sync) {
 47             // first-time setup
 48             // ...
 49             this.last_sync = {};
 50             this.last_sync.when = new Date().getTime();
 51             this.last_sync.is_modified = false;
 52         }
 53 
 54         if (this.last_sync.is_modified) {
 55             var request_options;
 56             // sync modified data
 57             // you can pass callbacks here too
 58             while (this.queue.length > 0) {
 59                 request_options = this.queue.pop();
 60                 $.ajax(request_options.ajax);
 61             }
 62             this.set_data('queue', []);
 63             this.last_sync.is_modified = false;
 64         }
 65         // data is synced, update sync time
 66         this.set_data('last_sync', this.last_sync);
 67 
 68         // get modified data from the server here
 69        $.ajax({
 70             type: 'POST',
 71             url: options.url,
 72             dataType: 'json',
 73             data: {'last_sync': this.last_sync.sync_date},
 74             beforeSend:
 75                 // here you can show some "sync in progress" icon
 76                 options.sync_before,
 77             error:
 78                 // an error callback should be passed in to this Data
 79                 // object and would be called here
 80                 options.sync_error,
 81             success: function (response, textStatus, request) {
 82                 // callback for success
 83                 options.sync_success(
 84                     response, textStatus, request);
 85             }
 86         });
 87 
 88 
 89     /**
 90      * Process a request. This is where all the magic happens.
 91      */
 92     this.process_request = function(request_options) {
 93         request_options.beforeSend();
 94         this.set_data(request_options.key, request_options.value);
 95 
 96         if (this.is_online()) {
 97             $.ajax(request_options.ajax);
 98         } else {
 99             this.queue.push(request_options);
100             this.last_sync.is_modified = true;
101             this.set_data('last_sync', this.last_sync);
102             // there are issues with this, storing functions as
103             // strings is not a good idea :)
104             this.set_data('queue', this.queue);
105         }
106 
107         request_options.processed();
108     }
109 
110     /**
111      * Return true if online, false otherwise.
112      */
113     this.is_online = function () {
114         if (navigator && navigator.onLine !== undefined) {
115             return navigator.onLine;
116         }
117         try {
118             var request = new XMLHttpRequest();
119             request.open('GET', '/', false);
120             request.send(null);
121             return (request.status === 200);
122         }
123         catch(e) {
124             return false;
125         }
126     }
127 }

Proposed Features

The example API isn’t bad, but I think it could be better. Perhaps something along the lines of Lawnchair. As I’m writing this, I realize that writing an API is going to take longer than I’d like - therefore, this will serve as a teaser and food for thought. Feedback is welcome.

Add an .each method for iterating over retrieved objects (inspired by Lawnchair)
Standard DataStore.save, .get, .remove, etc.
Support for these “storage engines”: localStorage, IndexedDB, send-to-server.
Support for these request types: XMLHttpRequest, WebSockets.
Store, at the very least, primitive values and JSON.
Include callbacks for various stages in the process of a request, similar to jQuery.ajax, e.g. beforeSend, complete, success, error. Figure out a good way to do this at each layer (minimize confusion).
For each request, specify which layers and in what order to go through. For example, if you want to store something in localStorage, IndexedDB, and send it to the server, you could do it in that order or the reverse.
Control whether to go to the next layer type depending on whether the previous succeeded or failed. Say, if you want to send the request to server but that fails, try localStorage as a fallback. Or the opposite.
Include a .get_then_store shortcut for getting the data from layer A and storing it in layer B?
Extensible: as easy as DataStore.addLayer(layerName, layerHandler), where layerHandler (obviously) implements some common API along with exposing some of its own, if necessary (e.g. ability to query or find, for IndexedDB).
As sending and getting data from the server means keeping two or more databases in sync, collisions may arise. Provide a collision callback or some smart defaults for handling collision. E.g. sometimes server data is always right (trusted more than user data), other times local data is king.

Help!

Hopefully my rant has gotten you thinking about the right approach. What would you like to see? What would make this something you would use and be happy with?

If you are interested in getting involved with coding this, contact me at paulc at mozilla.com.

Python scoping: understading LEGB

2011-01-31T00:00:00-08:00

Update: Thanks to Avazu for a clean suggestion and Fred for pointing out I should have indicated a good way to get around this.

Summary

Python scoping fun! Read about LEGB to understand the basics of python scoping.

Beef

I never bothered to read about how python scoping works until I hit this. It’s not exactly something to research until you have issues with it. :)

I had something like this going on:

 1 def func1(param=None):
 2     def func2():
 3         if not param:
 4             param = 'default'
 5         print param
 6     # Just return func2.
 7     return func2
 8 
 9 
10 if __name__ == '__main__':
11     func1('test')()

Note: Actual code was not as straightforward, func2 was actually a decorator. Admittedly, using the same parameter name is not a must, but it’s still a curiosity. I just wanted to fall back to a default value on run-time.

If you try to run this in python, here’s what you get:

~ $ python test.py 
Traceback (most recent call last):
  File "test.py", line 11, in <module>
    func1('test')()
  File "test.py", line 3, in func2
    if not param:
UnboundLocalError: local variable 'param' referenced before assignment

If you’re curious, you can read about the principles of LEGB. You have to understand a bit about compilers and the AST to get what’s going on behind the scenes. You might think that replacing lines 3-4 with:

param = param or 'default'

Might work. But no. You can’t assign the same parameter at the local level if the enclosing level defines it. Even this fails:

param = param

Fun, no?

What to do?

There are a few ways to get around this.

Assign param outside of func2. This doesn’t work if you need the default value to be dependent on what params func2 receives.
Use a second variable, param2 inside of func2 (posted below).

Here is the solution suggested by our commenter Avazu (over on the mozilla webdev blog):

def func1(param=None):
    def func2(param2=param):
        if not param2:
            param2 = 'default'
        print param2
    # Just return func2.
    return func2

SUMO and migrating large amounts of data

2010-12-03T00:00:00-08:00

Data migration is a complicated problem. It’s one of those things nobody wants to do, because it’s mostly tedious, not engaging, and also important to get right. Often times you have to compromise. I’ll talk a bit about the history of SUMO’s data, decisions the product and development team have made regarding its migration, and things I’ve learned from the experience.

Quick tips
Background
Data influences which components are rewritten first
Too much data! What now?
- Lesson 1: Migrating questions
- Lesson 2: Migrating the knowledge base
Can I get away with pure SQL?
Converting data: some challenges
- Lesson 3: Data integrity and storage

Quick tips

Start with the smallest dataset first to get the hang of things. (details)
You’ll have to deal with lots of edge cases. Make sure to write tests. (details)
Careful when converting dates/times: timezones can be a pain, and data is often stored in the timezone of the server. (details)
When you have so much data that migrating it would take days: do you need all of it? (details)
Consider the benefits of migrating data offline (details)
It’s unlikely that you’ll be able to use raw SQL. (details)
You’ll learn how important data integrity is, so you’ll pay attention at the schema your software uses next time. (details)

Background

At the beginning of February, SUMO started on a new adventure: rewriting the entire platform from an outdated version of TikiWiki to Django. As different (more or less stand-alone) pieces of the application were moved over to the new platform, our data had to be migrated over as well.

Over the course of the past 6 months, I’ve migrated over 25,000 pieces of content, most of which contained large amounts of text data.

Rewriting: where to start?

Tip: Start with the smallest, simplest dataset first.

The first migration of data was part of our new discussion forums. This was by far the smallest component we had to migrate over:

the forums contained roughly 5000 posts, in just about 1000 threads.
by comparison, we had a total of ~450,000 answers for ~260,000 questions, and over 5,000 knowledge base pages.
- questions
- knowledge base, document example
I had never done migrations before, so starting small was a good idea.
In addition to being the smallest, the forums content was also more straightforward to migrate (see also Converting data)

Too much data!!

Our questions component had over 450,000 (!) answers in the old system. Should you ever be in this situation, here are some options worth considering:

do you need all this data? can you keep just a reasonable portion of it?
if you don’t, great! (details)
if you do, tough luck. Consider:
- migrating data offline and then running a straighforward SQL import on the production database. (details)
- running the conversion script in chunks: migrate a part of the data now, and more later.

Migrating questions

Fortunately, even though we had over 260,000 questions to migrate, only about 15,000 were recent enough to be relevant for the currently supported versions of Firefox. Hence, the development and product teams agreed to not migrate all of them.

The script to convert data over was run live on the production server, and took over one hour to finish - not too bad.

Caveat: frozen edits to and creation of TikiWiki questions for those a bit over an hour.

Migrating the Knowledge Base

The Knowledge base presented a different situation:

Between 5,000 and 6,000 articles, some with long-standing history
Approved and unreviewed versions of an article were stored as separate articles in TikiWiki (each with its own history)
Lots and lots of content and metadata per document
Changing our wiki syntax from TikiWiki to MediaWiki (used by Wikipedia, among many others).

Thus, even though the count was much lower than for forums or questions, migrating these was much more intensive, both CPU and IO-wise.

Our solution here was to run the script offline (a.k.a. I ran the script on my machine) and hand our IT team just the SQL output to import.

Why? The script was complex enough to take over 3 hours from start to finish. Unreasonable to run on our production servers: if something went wrong, it could take an entire workday (or more) for the IT and development team to get things back on track.

Caveat: frozen edits to and creation of TikiWiki knowledge base articles for a day.

Huge benefits:

the data migration took less than 5 minutes to import on the production database.
no worrying about a script crashing on untested data
no overloading the production servers with database queries (replication lag can be an issue)

Can I get away with pure SQL?

Rarely.

Why?

Data changes frequently over time.
Thus, you want to freeze changes for the minimum possible time, so running a script on the production database is the way to go.
Hence, unless the new schema structure is close enough for raw SQL to handle, you’re out of luck.

The only time we were able to use pure SQL was when we froze edits for a day and generated the SQL offline.

How similar does schema structure have to be for SQL to work? you ask.

Here is the story for our forums, the simplest schema:

we kept the general structure of the data: forum with threads as children, and threads with posts as children.
in TikiWiki, our threads and posts were stored in the same table, with a parentId column distinguishing which are threads and which are posts.
in Django, threads had their own table (a good normalization to have).
a thread from TikiWiki mapped to a thread + a post in Django

Because of those (and other) differences, we couldn’t run a SQL-only import as reliably as a python/PHP conversion script. You could try doing a SQL-only import, but you’re gonna hit some key issues you wish SQL could solve:

while + if/else if/else structures. Much too often, data is corrupted or columns contain unexpected values - MySQL’s conditional construct doesn’t do much.
colliding unique keys, or integrity errors due to foreign key references are very difficult to prevent against in SQL.

Converting data

Here are some general observations of problems you might run into when converting data, based on our problems:

Frequently, normalization/denormalization changes are worth the effort:
- More often than not, normalization is the way to go.
- Long-term benefits are:
  - Data integrity a huge issue.
  - Usually, more flexibility to add functionality later on.
  - Easier migration (should you need to do it again).
  - And more!
Syntax changes are the worst:
- If you have to change wiki syntax, consider going straight to HTML: this way, you can use the rendered output of the documents from the old platform.
- Can’t get away with just HTML? Try regex conversion! I’m not saying that using a proper parser is a bad idea – but if you have to write a parser, it’s probably not worth the effort.
- Consider a simple PDA instead of a parser, it might just work.
Dates and times are timezone-sensitive. When converting, make sure to test that you’re taking timezones into account.
- In our case, we were moving over from a basically timezone-ignorant platform (as many PHP applications are), to a timezone-sensitive application (Django).
- Some columns were integers (UNIX timestamps), which are universally based on UTC, and so less tricky to convert. MySQL’s FROM_UNIXTIME can be useful here.
- Others were mysql dates/datetimes/timestamps, recorded at the server’s timezone (for us, US/Pacific, or PDT).
Be prepared for lots and lots of edge cases:
- Syntax changes are a nightmare of edge cases, as the old syntax probably has different rules and a different parser than the new one. One common problem for us was whitespace handling.
- If foreign keys were not (well) enforced in the old system, there will be numerous exceptions to catch in the new one.

Data integrity and storage

One of the most valuable lessons to learn from migrating data is integrity. Another one is storing data properly encoded and consistently. I’ll highlight just a few things that help with respect to data migration:

Use foreign keys and any other constraints as much as technically accurate
- E.g. you will have fewer exceptions to handle if you can rely on foreign keys to refer to valid rows
Storing text in one encoding is better than multiple; preferably unicode.
Store dates and times in the same timezone; preferably UTC.
Use a database that supports deferred foreign key checks, and is generally better about ACID. That’s not MySQL.

Testing

Although testing some temporary code that you’ll use once may seem like a waste, if you’re dealing with large amounts of data that can change, you will be better off writing tests.

Cover the basics - migrating a typical piece of content.
Cover some, if not all of the edge cases.
(somewhat related to testing) Write some helper scripts (e.g. bash) to make it easy to run the migration and spot check.

I worked on migrating the Knowledge Base over several weeks, and sometimes changes I made to the script broke tests – if it wasn’t for those tests, lots of content could have been migrated poorly, or not at all.

Django and AJAX image uploads

2010-08-20T00:00:00-07:00

Table of contents:

Demo
Summary
Server side (Django)
Client side
Graceful degradation

Demo

Screencast
Screenshots:
- The upload form, empty and ready for action:
- Browsing for an image:
- Uploading the image (in progress):
- The image is uploaded:
- Deleting the image would show a similar progress block as uploading:

Summary

In this post, we’ll go through how to get AJAX uploads to work with Django, including:

csrf protection with Django’s forms
graceful degradation (see also unobtrusive JavaScript)
uploading files in Django
thumbnail generation with PIL
cross-browser uploading of files through AJAX

Note: I’m planning to add upload progress. If you can’t wait for that post (understandably), there are several ways to go about it:

If your site isn’t using multiple webheads, you can just ask the webhead to get you the size of what’s been uploaded so far. Since Django can read in chunks, it can tell you how much has been processed. See this post for implementation ideas.
Or, regardless of the server setup, you can use the File API (in Firefox and Chrome) - easier, cleaner, no server-side interaction required.
Other multi-webhead approaches: writing progress to a file shared among them, or saving directly to a shared folder and e.g. returning the size uploaded so far.

Server side (Django)

First, we’ll look at how the server handles files sent to it.

Model

I created an app called upload with an ImageAttachment model, like so:

apps/upload/models.py:

from django.conf import settings
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes import generic
from django.db import models


class ImageAttachment(models.Model):
    """A tag on an item."""
    file = models.ImageField(upload_to=settings.IMAGE_UPLOAD_PATH)
    thumbnail = models.ImageField(upload_to=settings.THUMBNAIL_UPLOAD_PATH)
    creator = models.ForeignKey(User, related_name='image_attachments')
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()

    content_object = generic.GenericForeignKey()

    def __unicode__(self):
        return self.file.name

This represents an image attached to a piece of content (using a generic foreign key). Pretty basic stuff. The form is ridiculously simple:

Form

apps/upload/forms.py:

from django import forms


class ImageUploadForm(forms.Form):
    """Image upload form."""
    image = forms.ImageField()

View (uploading image, saving to disk)

The view is a bit more complicated, so I won’t go into the details. But you can have a look at the entire app and contact me if you have questions. Basically, the view does the file upload as you see in Django’s documentation. The function create_image_attachment deals with the part about saving a file to disk.

Generating the thumbnail with PIL

There is also a task for generating thumbnails, which is offloaded from the web server thread to improve performance. If you don’t need that, you can just call generate_thumbnail directly, it’s defined here.

Client side

Now, the magical JavaScript!

We’re using jQuery on SUMO, so I wrote two jQuery extensions:

jQuery.fn.ajaxSubmitInput(options) – wraps an <input type="file"> in a <form> and creates an <iframe> to which that form posts. To get around Django’s csrf protection, it also copies the csrfmiddlewaretoken hidden input into the form. You can’t clone a file input for security reasons (nor can you change or access its value), so you need to wrap it in a form.
jQuery.fn.wrapDeleteInput(options) – wraps an input<type="submit"> in a <form> and creates an <iframe> to which that form posts.

These two pretty much summarize the process:

when the user changes the value of the file input, post the form
- show some progress while the file is uploading
once the file is done uploading, show a thumbnail of the image
- also create the delete input and wrap it in the form using wrapDeleteInput()
when the user clicks on the delete button, post the action
- show some progress while the file is being deleted

A note about graceful degradation

To degrade gracefully, you want to post the file input to whatever view you’re including it to. And you can just do something like:

def some_view(request):
    # ...
    # NOJS: upload image
    if 'upload_image' in request.POST:
        upload_images(request, obj)
    # ...

Thanks for reading, hope it helps!

Full time @ Mozilla, yay!

2010-08-04T00:00:00-07:00

I’ve been meaning to blog about this for about a month now! Finally, here it is: I started my full time envolvement with Mozilla on July 12, right after an amazing Summit.

What’s it like?

First of all I must say that working here is amazing. It’s more than just the things we do here – web browser, mail client, lots of websites and serious community engagement (and many many other things, some of which you can read about on planet, spread firefox, or mozilla.org). It’s about the spirit of Mozilla. It goes hand in hand with our manifesto – our mission is to help make a better internet for everyone (e.g.). To strangers of the Mozilla culture, that may sound like a catchy advertisement, but those who have been in contact with the Mozilla community know the dedication and excitement that drives us, to help drive toward a world in which it’s all about real people being able to have and create the world that they want, privately, securely, freely.

As a side note, one of the things a lot of people don’t know about Mozilla is that it’s a non profit. Yes, that’s right. And we’re open and always looking for help :)

What changed?

I’d been contracting for Mozilla for the past 2 years, while in college. Working full time is better in many ways, but here are just some that have made a difference for me:

I graduated UCSC, woot! (still waiting for that official diploma, though!)
More focus. Having just one job to focus on, as opposed to other side things that come along with contracting
Oficially, more responsibility and say in decisions, always a good thing!
And a phone line (which I don’t use, but it makes me feel important!)

I was never a volunteer, but I was an intern – that’s how I started. I had no idea that I was going to enjoy working here so much. Really, though, it doesn’t feel like work.

What’s in store for me (and for you)?

Mostly, SUMO. Some things I’m helping with, past and future:

migrating legacy data (which I’ll blog about)
a very smooth process for uploading images, with awesome UX thanks to chowse (blog about that too)
a live chat system based on node.js (you guessed! blog post coming)
Getting a Knowledge Base in Django, with cool Wiki and non-Wiki features (see also this).

The end

Thank you, Mozilla, for being so awesome! I’m glad to have the fortune of working here.

If you would like to know more about Mozilla, web development (here and in general), and what I do here, don’t hesitate to ask.

May 17-21 status

2010-05-26T00:00:00-07:00

Last week (May 3-7)

a few reviews
activate admin in forums app
implement ACL after serious research
migrate user passwords from Tiki on login
migrate users and discussion-forum related groups from Tiki
assist IT with Friday Pac Man issues
help out new team members, lots of discussion
started work on forms validation and error messages and adding a 403 handler

This week (May 17-21)

posting this late, so I’ve already reviewed some bugs from James
finish 403 handler, form validation and sphinx config for discussion forums as well as the advanced search UI for it
fix up the wiki parser bugs I meant to get to last week: this and this
start on some 2.2 bugs

Access control in Django, and django-authority

2010-05-22T00:00:00-07:00

The past week I’ve been working with Django and permissions. I’ll talk here about the available options and go into details about the one Kitsune is using.

Here are the options I looked at:

Django’s default permissions
AMO’s access app
django-authority

Django’s default permissions

Read more about this in Django’s documentation.

Django provides a convenient and easy to use permissions system. The documentation covers it all, so I’m just going to summarize the pros and cons:

Pros:

Comes with Django (=> robust, well-written and stable)
Permission functions attached to the user object
Per-group permissions and per-user permissions
Model-level (a.k.a. per content_type) permissions (can add custom model-level permissions as well)
Work with Jinja templates, simply use user.has_perm('perm_name')

Cons:

No per-object permissions

AMO’s access app

AMO’s access app adds some flexibility on assigning permissions with wildcards and also stores the user’s groups in request.groups.

Pros:

Flexible way to assign all permissions in a certain group
Does not store permissions in the models Meta class

Cons:

Replaces django’s permission system (either use one or the other)
No per-object permissions

To use this, you can do something like this in your views:

from access import acl

# ...

def some_view_action():
    acl.action_allowed(request, 'Group', 'PermissionName')

django-authority

django-authority was the most promising of the three. It adds per-object permissions and wraps Django’s default permissions system without forcing an either-or choice. It also has a decent documentation, and although still in draft at the time of this writing, it was enough for me to figure out how to use.

Pros:

Per-object permissions
Wraps Django’s default permissions system, so you can use both
Custom permissions
Add permissions to appname/permissions.py, so they are completely separate from the models (unlike Django’s permissions)

Cons:

Needs a bit of work to hook in with Jinja templates
Admin functionality not yet complete - the admin form generated by the edit_permissions admin action does not work at the time of this writing. The alternative is not too bad however, you can manually add to the Permission model using that model’s admin.

django-authority on Kitsune

We are currently focused on rewriting SUMO’s discussion forums. For Kitsune, we have multiple forums and we wanted the ability to have different forum moderators per forum. We’re also using Jinja. django-authority comes with default Django template support, so if you’re using that, you don’t need to write your own template functions/filters.

For Jinja support, just add this to some app’s helpers.py

import jinja2
from jingo import register
import authority

# ...

@register.function
@jinja2.contextfunction
def has_perm(context, perm, obj):
    """
    Check if the user has a permission on a specific object.

    Returns boolean.
    """
    check = authority.get_check(context['request'].user, perm)
    return check(obj)

Then, in your templates, you can just use:

[% if has_perm('perm_cls.codename_model', object) %]
  {# grats, you've got the perms! #}
[% endif %]

(Couldn’t figure out how to escape braces in jekyll, so you’ll need to replace the square brackets above.)

See this link for more information on the perm_cls and codename variables.

If you’re interested, here’s the Kitsune code.

Hope that helps others who may decide to go this route. Questions and suggestions for improvement are always welcome, of course.

May 3-14 status

2010-05-19T00:00:00-07:00

Two weeks ago (May 3-7)

pushed 2.0/1.5.4 with rest of SUMOdev
- first time had problems with nfs mounts
- second time was a success, yay!
r?d nicer 404/500 handlers
r?d new wsgi script
r?d rewrite for discussion forums, which also cleaned up some of the search rewrites from previous push
r?d posting should update # of replies
threads list view

Last week (May 10-14)

r?d preliminary work on atom feeds
forums list view
wiki markup in discussion forums
filed some new bugs
added some omitted indexes to tiki
re-sync db, to test the SQL fix from above, also had IT run schematic and set up test forum
remove omniture tags from tiki
fix crc32 not working with unicode
discussion with Dave Dash about ACL on AMO

This week (May 17-21)

implement ACL after serious research
activate admin in forums app
help out new team members
fix up some of the wiki parser bugs

April 26-30 status

2010-05-03T00:00:00-07:00

Last week (April 26-30)

awesome webdev onsite
- learned about localization, the mass of krill, and cooking indian food, among other things
kitsune
- r?d use bleach to clean up sphinx results
- r?d remove didyoumean
- r?d htaccess fix for trailing slash in search url
- finish up minor bugs for 2.0
- some work on 2.1 bugs
- started converting the staging database to UTF8 with outage page
- remove YUI dependency from kitsune
- add WebTrends tracking
- investigating missing excerpts
tiki

This week (May 3-7)

kitsune
- pick up some of the bugs filed by James
- push 2.0/1.5.4 with rest of SUMOdev
  - in particular: ensure UTF8 conversion of database goes well
- team goal: handle all anonymous-access forums actions and views

April 19-25 status

2010-04-25T00:00:00-07:00

Last week (April 19-25)

kitsune
- r?d adding sumo-dev to receive useful stack traces from kitsune
- r?d improved maintenance of locales, go James!
- r?d kitsune documentation
- r?d better handling of timeouts for excerpts and merged
- r?d unicode improvements, adding sumo.utils.urlencode and merged
- r?d fixing an IE6/7 bug
- r?d case insensitive Accept-Language for Opera, Chrome, Safari
- r?d adding a search plugin
- fix pagination on IE6
- adding two new supported locales
- finished lots of tests for search filters and JSON for all
- finished the remaining 2.0 bugs with the rest of the SUMOdev team
- installed mod_wsgi locally with help of James’ docs
- assisted with IT bugs to convert the database to UTF8 according to this bug
- debugging server issues for kitsune stage
- begin work on milestone 2.1
tiki
- r?d using kitsune for search on stage-new
- filed patch for adding WebTrends to SUMO for tracking
- revert a notifications bug for articles
- add .htaccess rewrites for Fx3.7 help topic: keyboard shortcuts
- debugging screencast bug on mobile

This week (April 26-30)

WebDev onsite! Excited to hang out with the team.
kitsune
- work on forums and other 2.1 stuff
- prepare for kitsune push, potential QA bugs
tiki
- prepare for 1.5.4 push, potential QA bugs
- figure out screencast bug

April 12-16 status

2010-04-16T00:00:00-07:00

Last week (April 12-16)

kitsune
- r? cool vim indenting in templates
- r? minify bug 550515
- finish converting our current database from latin1 to utf8 after lots of painful dealing with data
- fix advanced search form logic, deals with default values, validation, cleanup. fixes a few other bugs in the process
- fix json results and add lots of tests
- fix search results sorting
- IT bugs to truncate a table, sync stage db with prod and convert the database to UTF8 according to this bug
tiki
- session fixes required for kitsune integration filed and reviewed
- fix and review some security vulnerabilities on mobile

This week (April 19-23)

kitsune
- finish the remaining kitsune bugs with the rest of the SUMOdev team
- finish UTF conversion of the database
- prepare for kitsune push, potential QA bugs
- begin work on discussion forums – milestone 2.1
- install mod_wsgi and run tiki in parallel with kitsune locally
- discuss/learn more about Test Driven Development
tiki
- finish adding WebTrends to SUMO for tracking

April 5-9 status

2010-04-11T00:00:00-07:00

Last week (April 5-9)

kitsune
- r? exclude staging articles from search, which also includes sphinx tests, reviewed again later after a fix to not require FORCE_DB
- r? l10n -> tower rename
- r? add JINJA_CONFIG() for string extraction and caching
- quick look at minify in kitsune, TODO next week
- fixed duplicate page param in paginator URL
- spent most of the week converting our current database from latin1 to utf8
tiki
- started work on adding WebTrends to SUMO for tracking
- push desktop 1.5.3 on Tuesday — see bugs

This week (April 12-16)

kitsune
- r? minify bug 550515
- r? tiki patch for django sessions
- r? cool vim indenting in templates
- finish up character encoding conversion
- finish up when to trigger advanced search
- finish getting sessions into kitsune, i.e. fix any issues spotted by davedash/jbalogh/jsocol on bug 553131
tiki
- finish adding WebTrends to SUMO for tracking

Stretch goals

search summaries are Jinja-friendly

My SUMO local development environment

2010-01-04T00:00:00-08:00

Summary

A while back, James wrote about his development environment. I wanted to summarize my approach, and go into a bit more detail about the specific setup you would need to have SUMO work locally. For more information, see also this wiki page.

My LDE: Ubuntu Linux

My LDE is slightly different: I dual boot Windows 7 and Kubuntu 9.10. As of now I’m still a student, and I work on my own machine. There are several reasons I find dual booting a great solution:

Best performance while developing. Although less and less of an issue, being in the native OS is always be faster than running a VM.
A full powered OS that I can use for other work. I do schoolwork and other contracting work, and sometimes Windows is not an option. Even a VM has its limitations when it comes to certain drivers (one example is audio – which I needed for a graphics project)
Keeps my work and pleasure separate – having dual boot is basically like having two machines. Typically, when I’m in Linux, I do work, otherwise I’m in Windows to do personal things.

The only downside is some additional wait time for rebooting if you need to switch to the other OS. Since I work on Linux only, I rarely need to switch while working. However, everything works fine for me on Linux – sound, video, even webcam and VPN (will blog later about all of these).

LAMP

For web development in general, you’ll need to have the LAMP stack installed. Fortunately, Ubuntu provides an easy way to do this in one line. From a terminal, run:

sudo tasksel

Select LAMP server from the list, and choose OK (TAB + ENTER). Follow the steps to choose a MySQL password and any other configuration there may be. Once that’s done, you should be able to go to http://localhost and see the “It works!” text. Here are some useful paths:

Web root: /var/www/
Hosts file: /etc/hosts
Apache sites: /etc/apache2/sites-available/ and /etc/apache2/sites-enabled/
PHP ini: /etc/php5/apache2/php.ini
MySQL ini: /etc/php5/conf.d/mysql.ini

SUMO setup

What you’ll need:

LAMP stack – see above
A host (optional) – see below
SSL – see this guide
Sphinx – see this guide
Memcached – I just install the package and that’s enough: sudo apt-get install memcached php5-memcache
Virtualbox (optional) – for testing: sudo apt-get install virtualbox
A copy of our database – you may ask James, Laura or myself for this – see how, below
SVN – sudo apt-get install subversion
Checking out and configuring the SUMO codebase – see this guide

To set up a SUMO host, I add the entry for it in `/etc/hosts`. Here is the top of my file:

127.0.0.1   localhost
127.0.1.2   sumo

Then, add the site info to apache. Here’s my complete conf file (with SSL), in /etc/apache2/sites-available/sumo:

<VirtualHost 127.0.1.2:80>
    ServerAdmin webmaster@sumo
    ServerName sumo
    ServerAlias sumo
    DocumentRoot /var/www/trunk/webroot
    <Directory /var/www/trunk/webroot>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost _default_:443>

    ServerAdmin webmaster@sumo
    ServerName sumo
    ServerAlias sumo
    DocumentRoot /var/www/trunk/webroot
    <Directory /var/www/trunk/webroot>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
    SSLEngine on
    SSLCertificateFile /etc/apache2/ssl/apache.pem
</VirtualHost>
</IfModule>

To enable the site, just do this:

cd /etc/apache2/sites-enabled
sudo ln -s ../sites-available/sumo

Then, restart apache:

sudo /etc/init.d/apache2 restart

Testing

For testing, I also use a VM in VirtualBox, running WinXP with multiple IEs. I chose WinXP to save space – only takes up 3GB. You should allow 4GB just in case you want to install more stuff. Using multiple IEs is not the best, as quirksmode points out, so you may consider running a VM for each IE version you wish to test. However, I haven’t experienced that many problems. On WinXP I’ve also installed Firefox (of course), Chrome, and Opera. On Ubuntu I just use the defaults: Firefox and Konqueror (since I’m running Kubuntu).

Help?

I’ve tried to cover everything, but of course there may be stuff I missed. If you need additional info, you may leave a comment. To obtain a copy of our database, you can find us on IRC or contact us by email.