This post is a long time coming. I've been meaning to publicly release my Python web framework for months. There are lots of Python web frameworks already, but I think the whole point of WSGI was to make it easy for anyone to implement their own web framework. Blueberry is hosted on Google Code, GitHub, and PyPI. It's my first github project so I'm expecting to make a few mistakes committing and pushing, but that's how it goes when learning a new source control manager.

Here are some facts about Blueberry:

• It uses CherryPy's WSGI web server.
• It was inspired by web.py and Google AppEngine's web framework.
• I've been using it in production for months. This blog uses an old version of blueberry.
• It's database ORM agnostic.
• nginx is my preferred web server in front of blueberry, but it works with Apache and mod_python or mod_wsgi.

I still have a ton of work ahead of me to document blueberry, it just takes time. For now, what little information there is on blueberry can be found here on Google Code.

One of the things that's been keeping me up at night lately is this new project I've been researching and developing. SimpleStore is a way to store Python objects (in the future that will change to JSON objects) in sharded MySQL databases. The idea is that it lets developers scale their web applications, but honestly it hasn't been used on a site of any real scale yet since it's still very new.

SimpleStore is cool as a beta project for using with your own database servers. What would be even better is to take the ideas from SimpleStore and create a service that web developers can use for remote database storage. This is what's been really keeping me up at night. I've been doing research on distributed database systems and I think with my experience from writing SimpleStore I will be able to develop something pretty neat.

I won't turn SimpleStore into a cloud-based service, but the ideas from SimpleStore will help me create a truly distributed database system. If this service turns into something I can put online, I hope it creates another option for web developers looking for easy-to-use cloud database systems.

Also, I updated SimpleStore to version 0.3. That update adds MySQL sharding support. You can grab it here: simplestore-0.3.tar.gz

I've been working on a new project that has some content management features lately. It's a pretty big project and will take months to complete. I was messing with in-place editing with prototype and scriptaculous and found that I couldn't use the default editor for managing links. Normally when you're doing in-place editing you have a single piece of text/content that you want to edit. You click on it and it brings up the form. For editing links I need to edit the URL and the anchor text. I needed to have two separate fields for this and Ajax.InPlaceEditor from controls.js wasn't entirely up to the task.

I ended up inheriting from Ajax.InPlaceEditor and altering some of the behavior to handle link editing. I feel like it's pretty much a hack job but it does what I need it to do, so it seems good enough. I felt that it would be somewhat beneficial to upload the code for in-place link editing.

I only uploaded the javascript and a html demo. The server-side stuff isn't included since this will work across different web languages. I use Pylons and Mako, for example.

You can grab the demo here: http://alwaysmovefast.com/public_svn/in_place_link_editor.

To give a short example of what I did for the server-side stuff, this could be my controller/action:

class LinksController(BaseController):
    def create(self):
        c.link = {
            'url': request.params['url'],
            'anchor': request.params['anchor'],
            'id': 'some_id'
        }
        return render('/links/create.mako')

And this is what '/links/create.mako' could look like to update the page:

<a href="${c.link['url']}">${c.link['anchor']}</a>

This is a very basic example. My create.mako actually looks something like this:

<script src="/javascripts/prototype.js"></script>
<script src="/javascripts/scriptaculous.js"></script>
<script src="/javascripts/application.js"></script>

<script>
    li = $('newlink');
    li.id = '${c.link['id']}';

    li.innerHTML = '<a href="${c.link['href']}">${c.link['anchor']}</a>';

    new_link = document.createElement('li');
    new_link.id = 'newlink';
    new_link.innerHTML = 'Insert Link';

    li.parentNode.appendChild(new_link);

    new InPlaceLinkEditor('newlink', '/links/create');
</script>

Basically what it does is update 'newlink' by changing its id to the id of the newly-created link on the server side and update the innerHTML of that object. It then creates a brand new link object and appends it to the old link's parent node. I then create a new InPlaceLinkEditor for the new 'newlink'. That's about it.

Hopefully you can find this stuff useful.

There's a lack of documentation on how to use FormEncode in real projects so I've been meaning to write this article for some time now. I use FormEncode in my Pylons apps and I created a small app just for this article. You can grab the source here: http://alwaysmovefast.com/public_svn/formencode_tutorial. To make using FormEncode easier, I also created a few form helper methods that can be found in formencode_tutorial/lib/form_helpers.py. I also used a little CSS to make the form somewhat pretty.

FormEncode makes it pretty easy to do form validations and to then display any errors to your users. The way it works is that you pass a string of HTML to htmlfill.render(), along with some other options, and it returns parsed HTML for use in your pages. htmlfill.render() uses an errors option where you pass in a dict consisting of field names and error values. A typical error dict might look like: {'firstname': Invalid(u'Please enter your firstname',), 'lastname': Invalid(u'Please enter your lastname',)}.

htmlfill.render() will parse those errors and inject them into your <form:error> tag for that particular element. As well as accepting an error dict, htmlfill.render() can accept a default value dict that will inject default values into your form elements.

When injecting errors into your HTML template, it will also use an error formatter. The default error formatter looks like this:

def default_formatter(error):
    return '<span class="error-message">%s</span><br />' % html_quote(error)

You can also use custom error formatters for a little more control over the look and feel of your displayed errors. I like to use this as my formatter:

def p_error_formatter(error):
    return '<p>%s</p>' % htmlfill.html_quote(error)

The reason I use this as my formatter is because I display label elements below my form elements and above the errors. It's mostly personal preference.

I created a helper to abstract away some of the finer details of using formencode. One of them helps me create text fields with the label and form:error tags prepackaged and ready to pass to htmlfill.render():

# return HTML string that can be passed to htmlfill.render()
def tfield(name, label=None, **options):
    if label is None: label = name

    s = text_field(name, **options)

    id = ''
    if options.has_key('id'): id = options['id']
    else: options['id'] = name

    # append a label to this field for good measure
    s += '<label for="%s">%s</label>' % (id, label)

    # use a custom error formatter. when htmlfill.render() parses this html
    # it looks for the key 'p_error_formatter' and uses that formatter.
    # remember that 'p_error_formatter' is looked for in the error_formatters dict
    # defined above.
    s += '<form:error name="%s" format="p_error_formatter"></form:error>' % name
    return 

Notice the form:error tag? That's where htmlfill.render() inserts error text (if there is any) and formats it according to the 'p_error_formatter'. So how does htmlfill.render() know how to handle 'p_error_formatter'? Easy. You create a dict with your error formatter(s) and pass it to htmlfill.render(). When htmlfill.render() parses your HTML, it will take 'p_error_formatter' and do a lookup in your error formatter dict and call that function with the error text.

Here's my error_formatter dict:

error_formatter = {
    'default': htmlfill.default_formatter,
    'p_error_formatter': p_error_formatter
}

error_formatter['p_error_formatter'] is the p_error_formatter() function I defined above.

I also created my own render() function that just calls htmlfill.render() with some predetermined args:

def render(html, defaults=None, errors=None):
    return htmlfill.render(
        html,
        defaults=defaults,
        errors=errors,
        error_formatters=error_formatters,
        auto_insert_errors=False
    )

That's about it for the core htmlfill stuff. I have a couple other helper methods in form_helpers.py and some more CSS that makes the background of form elements red when there's an error.

If you're running Pylons, just go into the root formencode_tutorial directory and run 'paster serve ––reload development.ini' and check out how it's all tied together.

Feel free to leave comments if you have any questions or if I missed something. Happy hacking!

I've been working on some Pylons projects lately because I've been wanting to move into using Python for web apps instead of Ruby. It took awhile to find the right Python web framework for me. I checked out Django, TurboGears, web.py, and a couple others. I finally landed on Pylons and it was an instant hit with me.

There's still a lot to be done on the Pylons core but it's very exciting to use and I never feel like the framework is holding me back.

This is only going to be a short post until I come up with something interesting to write an article about. Just thought I'd share that Pylons is pretty amazing.

send/suspend/dispatch

send/suspend/dispatch takes a function as its argument just like send/suspend. The difference is that this page-generating function takes another function as its argument, conventionally called embed/url. embed/url is used to embed continuations into a URL. To generate continuation URLs with embed/url, you pass it a function that takes the HTTP request object as its argument. When that continuation URL is requested, either by clicking a link or posting a form, the continuation is resumed with the HTTP request object being passed to it.

This article's example will be a simple counter that lets users increment or decrement their counter by clicking the appropriate links. Each link will use embed/url to dispatch to separate functions in the servlet. The two increment functions are really simple. They take the current value of the counter and return its new result.

(define (increment-counter counter)
  (+ counter 1))

(define (decrement-counter counter)
  (- counter 1))

Most of the work is done in the show-counter function. It shows the current value of the counter and generates links for incrementing and decrementing the counter. I also added a little styling to the links that just separates them with a little whitespace.

(define (show-counter counter)
  (send/suspend/dispatch
    (lambda (embed/url)
      `(html (head (title "counter"))
             (body
               (h3 ,(number->string counter))
               (span (a ([href ,(embed/url (lambda _ (increment-counter counter)))]) "++"))
               (span ([style "margin-left: 10px;"])
                     (a ([href ,(embed/url (lambda _ (decrement-counter counter)))]) "--")))))))

This code should be pretty self-explanatory after my little introduction to send/suspend/dispatch at the top of the article. And if you've followed along with my previous posts you should know what all the supporting pieces of code do.

Everytime the increment and decrement functions are called, show-counter returns control back to the start function of the servlet. This only allows us to increment or decrement once, so we have to loop each time show-counter returns.

(define (start request)
  (let loop ([counter 0])
    (loop (show-counter counter))))

When show-counter returns, it returns the new result of the counter. The loop then calls show-counter again with the new counter value.

This is a pretty good introduction to send/suspend/dispatch I think and it is a building block for bigger and greater things. The code for this article can be found here.

I'm trying to decide what I want to write about next. I have a pretty cool dispatching server that lets programmers create "pretty" URLs similar to Ruby on Rails' controller/action dispatching. I may write about how to do that and provide a little proof-of-concept web application. I've also been meaning to write about keyword tracking in scheme. Keyword tracking is a pretty hot topic in the world of internet marketing. I'll come up with something to write about. Until next time, hack away at the code and learn some stuff.

send/suspend

Using continuation-based web servers is a very powerful way to develop web applications. Continuations can essentially remove problems associated with back buttons and multiple tabs/windows in web browsers. The basic idea behind web server continuations is that a continuation reference id is passed in the URL of a web request and then the continuation with that id is executed on the server. This allows web developers the ability to maintain state without using sessions. You could even think of the continuation URLs as a basic session id that's directly tied to the execution of server-side code.

The continuation ids aren't true session variables. They simply maintain the state of current execution. If you were to use two browser windows and point them at the same continuation-based website, each browser window would have its own unique execution path. On a normal website using cookies to maintain state, both of your browser windows would share data (such as a user id). Using continuations to maintain state allow both of your browser sessions to be completely independent of eachother. Keep in mind that there's nothing stopping you from using cookies with continuation-based web servers like PLT Web Server.

If you followed along with my previous posts here and here, you should have scheme installed and ready to go. In your /usr/plt/collects/web-server/default-web-root/servlets/examples/ directory there are a bunch of scheme servlet examples. One of the more famous examples used for introducing people to continuation web programming is the add.ss servlet in the examples directory.

I'm including that add.ss example in this article's source code so all you have to do to test it out is require "main.ss" in your mzscheme prompt and visit http://localhost:3000/servlets/add.ss. When you're submitting numbers to the servlet, take notice of the strange ids in the URL that look something like "2*1*34452322". The first number is the servlet instance id, the second number is the continuation id, and the third number is a randomly-generated id that helps prevent users from guessing continuation URLs.

All the code for this article can be found under svn: http://alwaysmovefast.com/public_svn/article_src/two.

Like the simple.ss example, add.ss has a start function where the initial request is passed. When a request for add.ss comes into the server, the start function is executed. Before the start function returns, it calls (request-number "first"), followed by (request-number "second"), and then adds the result of those two function calls.

When (request-number "first") is called, it runs the build-request-page function inside of a send/suspend call. send/suspend captures the current continuation and binds it to a URL (k-url) that's passed to the function returned by (build-request-page which-number). Once the lambda function is called with k-url, it's expected to generate an HTTP response. That response is the HTML form you see on the page.

When you use send/suspend with the continuation-bound URL (k-url), you can use that in your links and form actions. That's exactly what build-request-page is doing here:

(form ([action ,k-url] [method "post"])
      "Enter the " ,which-number " number to add: "
      (input ([type "text"] [name "number"] [value ""]))
      (input ([type "submit"] [name "enter"] [value "Enter"])))

The responses generated by (start) and (build-request-page) are inside a quasi-quote (`). This means that these functions return code as data that will be evaluated later. When you quasi-quote an expression, you're able to unquote expressions and variables such as k-url and which-number. By unquoting k-url and which-number, we're telling the interpreter to evaluate them immediately.

When you enter the first number in the form and post it to the continuation URL, the continuation is invoked and the request is returned from send/suspend. The call to send/suspend is wrapped in a request-bindings call. request-bindings takes an HTTP request and returns the variable bindings. The variable bindings can then be extracted. In this case, the code uses extract-binding/single to request a single variable from the request bindings. The text field of the form is named "number", so in extract-binding/single, we pass 'number as the binding to be extracted from the request-bindings call. Once the number value is extracted, we have to convert it from a string to an integer by using string->number. The number is then returned as the last evaluated expression of request-number.

After (request-number "first") returns with an integer, (request-number "second") is called and follows the same path as (request-number "first"). When (request-number "second") returns, both values are added together and then converted from an integer to a string in the start function.

The start function then returns its HTTP response with the sum of the two numbers.

There is a major limitation when using send/suspend: you can't easily generate multiple links and forms from one continuation URL unless you're content with using URL query strings. This means that all your requests will essentially go to one function and then you will have to dispatch based on the query string values. It's definitely not very elegant. One of the solutions to this problem is by using send/suspend/dispatch, which I'll be covering in my next blog post.

Although send/suspend is somewhat limited, you should still hack around with it and create some cool things to get used to it.

Building the Server

Web application development in Scheme is something I've just recently gotten into. It's been a challenging and exciting journey and I'm happy to begin sharing what I've learned. I figured I would start with writing a custom web server built from PLT Web Server that serves files and does some basic servlet dispatching.

If you don't yet have PLT Scheme installed, read my post on Installing PLT Scheme from Source.

PLT Web Server technology allows developers to essentially create their own web servers in scheme. I'm going to start with a basic web server that's built off of the /usr/plt/collects/web-server/run.ss web server. That will allow us to evolve the code over the course of these articles as we get into some more advanced techniques.

All of the code for this article can be found here: http://alwaysmovefast.com/public_svn/article_src/one/

Go ahead and download it so you can follow along easier and actually run the software on your machine. To run the server:

1. cd into the directory 2. mzscheme 3. (require "main.ss")

The server should be started now so point your web browser to http://localhost:3000/ and you should see the index.html file.

At this point, I'm not going to go in-depth with what the server is actually doing. For now I'm only going to be talking about building servlets so you shouldn't have to worry about the server for now.

Go ahead and point your browser at http://localhost:3000/servlets/simple.ss now. The code for this servlet is pretty short:

(module simple mzscheme
  (provide (all-defined))
  (define interface-version 'v1)
  (define timeout +inf.0)

  (define (start request)
    `(html (head (title "Hello World"))
           (body (h4 "Hello World")))))

All you really need to know about this module is that its start function is called when the web server receives a request for it and the request is passed to the start function. The start function then passes back a response to the browser as html.

As a little exercise you could implement your own servlets in the servlets/ directory so you can get a feel for scheme web development.

I'm going to be writing a bunch of follow-up posts on different aspects of scheme web development. The next article is going to be on continuation-based web development. For reference, I'll be talking about send/suspend and send/suspend/dispatch, so feel free to read up a little on those beforehand.