Disembrangling Programming

Embrangler: Moving to Jekyll

2010-03-21T00:00:00-07:00

It took only three days to migrate my blog from Wordpress to Jekyll. For anyone who plans to do this, I’m summarizing the whole process below.

Table of contents:

Why change?
Why Jekyll?
Migration process
Conclusion

Why change?

While I realize Wordpress the most popular blogging platform for a reason, I was bothered by the workflow involved with having a WP blog. Some of the (arguable) inconveniences I found were:

the WYSIWYG editor doesn’t handle some of the markup I would like (headings, tables aren’t great either), so I would constantly switch between the Visual and HTML tabs to format things the way I wanted to. While I realize this could be fixed by customizing the editor, why should I have to do that? Plus, I have long given up hope of the ideal WYSIWYG editor. Besides not handling markup very well, it would sometimes reformat some of my plain HTML and break it.
creating posts is pretty much the only thing I do. I don’t use categories, tags, or any other features. Yet I had this admin area cluttered with features.
I’m not too fond of WP’s version control. I would often see the message that “a more recent version of this post is available” erroneously, I’m still not sure why. While changing versions worked well enough, I find the idea of an actual version control system more appealing.
My blog could use a redesign. For a while, I was using the Carrington Text theme, which was ok but not great. I wanted to make it feel more like my blog – no sidebar, larger font, a better home page to name a few.

Why Jekyll?

A few weeks before I decided to switch off of Wordpress, I found out about static site-generator engines alternatives that would use available version control software such as git or svn. After looking around I hesitated to switch to any of these because they were mostly Ruby-based, and I was unfamliar with how they work. However, all other alternatives seem to be heading towards full-fledged CMS’s.

A quick note on static sites: they have virtually no security holes (well, in theory), because there is no server-side handling of data submission. I’m using a VPS too, so I was fond of having less CPU usage on the server (due to not having to process, say, PHP scripts for Wordpress).

Some of the alternatives to Jekyll that I considered are Toto, StaceyApp, Chyrp, Subtext and Typo. I picked Jekyll for two main reasons:

popularity – looking at the project’s github watches/forks
documentation and examples – there is actually a Sites page, leading to other sites and their github hosted sources, immensely helpful when starting out

Migration process

I followed these instructions:

… and started by looking at Tom Preston-Werner’s blog and its source on github.

Migrating posts

After I got the redesign to look good on Jekyll, the next step was migrating my Wordpress posts. Fortunately, there is documentation for that too.

Unfortunately, the automatic migration did not convert some of the markup very well, such as LaTex, images with captions, most lists and links. Yet I had less than 15 posts, so I just went through them and checked everything. Because I’m using LaTeX in some of my posts, I used maruku (instead of rdiscount) to parse them. I couldn’t find LaTeX support for rdiscount, which is a faster parser.

I’m also using lsi for related posts, and python-pygments for syntax highlighting. The latter is simply a wonderful tool, and I recommend it to anyone posting code on the web. Of course, all of these are documented in the Jekyll documentation, so it was easy to get it all working.

The code for this site is up on github as well, so you can check it out there to see how it works.

Migrating comments

This took me about 20 minutes :), following these instructions.

Server setup and more

I actually didn’t want to install jekyll on my server, and preferred to do all the generating locally. My server only works with plain html this way. The one thing I did do, however, was use github’s post-receive-hook service. I ended up having something similar to this:

github posts data to a php file on my server
this file verifies the posted data, logs the commit, and runs a C script.
the C script runs git pull

Here’s the code for all of this:

php script

Update: James pointed out that my security measures weren’t good enough, so I updated the script. If you don’t care much for security, you may prefer automated publishing. See my second script, below.

<?php

$check = auth();
if (!$check) die;

echo '<pre>';
echo exec('/path/to/site/pull_script');
echo '</pre>';

function auth() {
    // do some parameter checking here
    // and return true when matches
}

To make things even more awesome, I bookmarked this URL using Firefox’s keywords, so I only need to type one character to publish ;)

Here is my second script, which does automatic publishing.

<?php

if (!$_POST['payload']) {
    header('HTTP/1.0 403 Forbidden');
    exit;
}

define('HOOKLOG', '../logs/hooks.log');
$fh = fopen(HOOKLOG, 'w') or die("Can't open file");
$data = '';

$hook = json_decode($_POST['payload']);

if ($hook->repository->owner->name != 'pcraciunoiu') {
    header('HTTP/1.0 403 Forbidden');
    exit;
}

$cs = $hook->commits;

foreach ($cs as $c) {
    $data .= $c->timestamp . "\n";
    $data .= $c->author->name
        . ' (' . $c->author->email . ')'
        . ' pushed to ' . $hook->repository->url . "\n"
        . 'View commit at ' . $c->url . "\n"
        . "\n\nCommit message was:\n" . $c->message . "\n\n"
    ;
}

fwrite($fh, $data);
fclose($fh);

// set this to your path
exec('/path/to/site/pull_script');

There are two important security measures in the first script:

it not write to a file, so if it gets hit by someone trying to find the secret codes, my server’s disk doesn’t perform intensive IO
it does not hint in any way at parameter names, number of parameters that must be submitted, or whether they should be submitted through GET or POST – if you don’t get the right values, you don’t see anything

the c file

#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>

int main(void) {
    execl("/usr/bin/git", "git", "pull", "origin", "master",
        (const char *) NULL);

    return EXIT_FAILURE;
}

Compile this with

gcc pull_script.c -o pull_script

So now, every time I push to github, the server automatically updates. Really cool way of publishing!

Conclusion

I enjoy using git, and having a static site. My comments are offloaded to a separate server, and I write plain text files using the markdown syntax. My blog gets published when I visit a bookmarked URL.

Goodbye Wordpress!

Hacking Wordpress: check contact messages for spam using Wordpress' built-in Akismet spam for comments

2009-11-14T00:00:00-08:00

This article explains how to easily create a contact form with spam filtering in Wordpress.

Duration: 30 minutes
Demo: http://awesomemath.org/ (see footer)
Software: Wordpress 2.8.6 (latest as of this writing)

Don’t you wish there was a nice and easy Wordpress plugin to add a contact form anywhere on your site, with the benefit of spam filtering without Captchas? I do. And that’s what prompted me to do things this way.

Why not use Captchas? Hey, don’t take my word for it: other people have spent time showing why. In short, Captchas are not user friendly, and, frankly, to me, they are frustrating. Every time I want to send a comment or submit some data to a website - bam, I have to spend that extra few to stare at some deformed text. I hate Captchas.

So what instead? Well, you’re in luck: Wordpress offers a free, well-integrated spam service called Akismet, which works really well, and is integrated into Wordpress by default.

This article shows you how to quickly integrate Akismet into a contact form, and it’s all built from scratch.

Step 1: The frontend (HTML + PHP + jQuery)

I needed my contact form to be on every page (in the footer), so I created a widget. I used the ExecPHP plugin to include PHP code in it. I also limit submission to 1 every so often, based on a cookie. I also use jQuery to submit this form with AJAX for a snappy sense of response ;)

Step 1a: The HTML and PHP:

<div id="contact_thanks" <?php
  if (!$_GET['contacted'] && !$_COOKIE['examplesite_contact'])
    echo 'style="display:none;"';
?>>
    Thank you for contacting example site!
</div>
<?php if (!$_GET['contacted'] && !$_COOKIE['examplesite_contact']) { ?>
<form id="contact_form" method="post" action="">
    <div id="contact_msgbox" style="display:none;">
        Put contact information here.
    </div>
    <div id="contact_error" style="display: none">
        <a href="#" class="contact_error_close" "Dismiss message">x</a>
        We couldn't submit your form for one or more of the following reasons:
        <ul>
            <li>All fields must be longer than 5 characters</li>
            <li>You must provide a valid email</li>
        </ul>
        Please resolve the above problems for the highlighted fields
        and try again.
    </div>
    <div id="contact_info" style="display: none">
        <a href="#" class="contact_error_close" "Dismiss message">x</a>
        <p><strong>E-mail:</strong> <a href="#" id="contact_email_addr">
        <img src="/photos/email.png" alt="email" /></a><br/>
    </div>
    <label id="contact_message_label">Message:
        <br/>
        <textarea rows="5" cols ="25" name="contact_message"
            id="contact_message" tabindex="3" ></textarea>
    </label>
    <label>Name:
        <br/>
        <input type="text" name="contact_name" tabindex="1"
            id="contact_name" />
    </label>
    <br/>
    <label>Email:
        <br/>
        <input type="text" name="contact_email" tabindex="2"
           id="contact_email" />
    </label>
    <div class="btn-panel">
        <a class="btn btn-brown" href="#" id="contact_form_info">
            Contact Info
        </a>
        <input type="submit" name="contact" value="Submit"
            class="btn btn-green" />
    </div><!-- btn-panel -->
</form>
<?php } ?>

That may look a bit complicated, but let’s strip this down to understand what’s going on. At first, we wish to display a thank you message if the form was submitted – either recently (through a cookie), or right after submission (using the URL parameter “contacted”). So, we hide the thank you message in any other situation. If the form hasn’t been submitted recently, we show it. This, too, breaks down into parts:

I included a contact information box, too, but you may find that unnecessary. See the demo to get the picture.
I also include an error message box, to help validate the submitted contact information.
Finally, the inputs themselves, Name, Email and Message, each using tabindex attributes to ensure that the user fills them out in this order. Since, in my layout, the <textarea> comes before the two <input>, I need to use tabindex. I’m using this layout because I intend to float the <textarea> to the right. Then, the submit button and contact info button are grouped together in &div class="btn-panel">

Step 1b: The jQuery:

var MIN_FIELDLENGTH = 5;
var EMAIL_PREFIX = 'me';
var EMAIL_END = 'examplesite.com';
jQuery(document).ready(function($) {
    function is_valid_email_address(email_address) {
        var pattern = new RegExp(/^(("[\w-\s]+")|([\w-]+(?:\.[\w-]+)*)|("[\w-\s]+")([\w-]+(?:\.[\w-]+)*))(@((?:[\w-]+\.)*\w[\w-]{0,66})\.([a-z]{2,6}(?:\.[a-z]{2})?)$)|(@\[?((25[0-5]\.|2[0-4][0-9]\.|1[0-9]{2}\.|[0-9]{1,2}\.))((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[0-9]{1,2})\.){2}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[0-9]{1,2})\]?$)/i);
        return pattern.test(email_address);
    }
    function validate_submission(field) {
        if (field) {
            var fields = [field];
        }
        else {
            var fields = ['name', 'email', 'message'];
        }
        var returnval = true;
        var text;
        for (var i in fields) {
            text = $('#contact_'+fields[i]).val();
            minlen = MIN_FIELDLENGTH;
            if (fields[i] == 'message') minlen *= 10;
            if ((text.length < minlen)
                || ((fields[i] == 'email')
                && !is_valid_email_address($('#contact_'+fields[i]).val())))
            {
                $('#contact_'+fields[i]).attr('class', 'chighlight');
                returnval = false;
            }
            else {
                $('#contact_'+fields[i]).attr('class', '');
            }
        };
        return returnval;
    }
    $('#contact_name').keypress(function() {
        validate_submission('name');
    });
    $('#contact_email').keypress(function() {
        validate_submission('email');
    });
    $('#contact_message').keypress(function() {
        validate_submission('message');
    });

    $('#contact_form').submit(function () {
        if (validate_submission()) {
            var formInput = $(this).serialize();
            $.post('/contact_form.php', formInput, function(data){
                $('#contact_form').hide();
                $('#contact_thanks').show();
            });
        }
        else if (showing_message != 1) {
            showing_message = 1;
            $('#contact_msgbox').html($('#contact_error').html());
            $('#contact_msgbox').addClass('error')
                                .removeClass('message').show();
            if (msg_timeout) clearTimeout(msg_timeout);
            msg_timeout = setTimeout(function(){ close_msgbox(); }, 10000);
        }
        return false;
    });

    $('#contact_form_info').click(function () {
        if (showing_message == 2) {
            close_msgbox();
            return false;
        }
        $('#contact_msgbox').html($('#contact_info').html());
        $('#contact_msgbox').removeClass('error')
                            .addClass('message').show();
        showing_message = 2;
        if (msg_timeout) clearTimeout(msg_timeout);
        msg_timeout = setTimeout(function(){ close_msgbox(); }, 10000);
        return false;
    });

    function close_msgbox() {
        $('#contact_msgbox').hide();
        showing_message = 0;
        return false;
    }

   function email_link() {
        window.location = "mailto:" + EMAIL_PREFIX + "@" + EMAIL_END;
        return false;
    }
    $('#contact_msgbox a:first-child').live('click', close_msgbox);
    $('#contact_email_addr').live('click', function() {
      return email_link();
    });
});

Okay. First we have a constant for the minimum length of each field. In the validation function, validate_submission(), fields get checked against this length. There’s a special case for the <textarea>, which has a minimum length of the constant value * 10 (times ten). Then two constants making up the email address for clicking on the image – this is nice, it keeps the functionality of the mailto: link, but protects it from spam. I doubt bots are smart enough to do string concatenation for every site they visit :)

is_valid_email_address() validates an email using a regex. There’s tons of these on the web. I wanted mine to be not too long, but long enough :)
validate_submission() either valides a certain field, or validates the entire form (latter if no parameters are passed
The next three are jQuery events for onkeypress in the form fields. They just call the validation function for each field. To make this even better (but a bit more CPU intensive), you may consider additional events on blur or change.
Next up, the submit function. This validates the entire form and submits data through ajax, using jQuery’s built-in $.post call. Ideally, you should expect some kind of data response after posting, and show a thank you message based on that (or error otherwise) to ensure best functionality for the user, but I didn’t bother.
Finally, we have the close button functionality, with a timeout. At the end, we add events for email and bindings for the close button. You can read about all of these in jQuery’s wonderful documentation.

Step 2: The backend (PHP)

This is fairly short :)

<?php
define('DEFAULT_POST_ID', post_id_here);
/* Contact form handling here */
if ($_POST['contact']) {
    require_once( '/wp-load.php' );
    $contact_invalid = array();
    $contact_data = array();
    $contact_data['name'] = filter_var($_POST['contact_name'],
        FILTER_SANITIZE_STRING);
    $contact_data['message'] = filter_var($_POST['contact_message'],
        FILTER_SANITIZE_STRING);
    $contact_data['email'] = filter_var($_POST['contact_email'],
        FILTER_VALIDATE_EMAIL);
    foreach ($contact_data as $contact_key => $contact_field) {
        if (!$contact_field) {
            $contact_invalid[$contact_key] = true;
        }
    }

    if (!$contact_invalid && !$_COOKIE['awesomemath_contact']) {
        $comment_post_ID = DEFAULT_POST_ID;
        $comment_author = $contact_data['name'];
        $comment_author_email = $contact_data['email'];
        $comment_author_url = '';
        $comment_content = $contact_data['message'];
        $comment_type = '';
        $comment_parent = 0;
        $user_ID = 0;

        $commentdata = compact('comment_post_ID', 'comment_author',
            'comment_author_email', 'comment_author_url', 'comment_content',
            'comment_type', 'comment_parent', 'user_ID');
        $comment_id = wp_new_comment( $commentdata );
        $comment_approved = $wpdb->get_results(
           "SELECT (comment_approved = 'spam')
                AS spam
            FROM wp_comments
            WHERE comment_ID = '{$comment_id} LIMIT 1;'"
        );
        // allow contact again after one hour
        setcookie('awesomemath_contact', 1, time() + 3600, '/');
        if ($comment_id && !$comment_approved->spam) {
            // not spam!
            $contact_headers = 
            "From: {$contact_data['name']} <{$contact_data['email']}>\r\n\\";
            wp_mail(get_option('admin_email'),
                'Examplesite Contact Form Message'
                , $contact_data['message'], $contact_headers);
            $contact_headers =
                "From: Examplesite <me@examplesite.com>\r\n\\";
            wp_mail("{$contact_data['name']} <{$contact_data['email']}>",
                'Thank you for contacting Examplesite!',
                'Thank you for contacting us. We will get back to you shortly.

Below is a copy of your message.
If for any reason we do not get back to you soon, simply reply to this email.

------------------------------

' . $contact_data['message'], $contact_headers);
            // redirect to prevent resubmission
            header("Location: {$_SERVER['HTTP_REFERER']}?contacted=1");
            die;
        }
    }
}

See? That wasn’t so bad. Let me summarize the above. First, we validate data once again by using a simple php filter. See this easy tutorial for the basics of php filters.

Then, we use Wordpress’ built-in comment system to post contact messages as comments. We then check to see if the comments were potentially marked as spam. If they weren’t, we send an email to the site admin (it can be any email, really), with the contact message. Make sure to set DEFAULT_POST_ID to a post against which you can file these “comments”.

I should add that this contact_form.php file is also included in my theme to support no-JS submission. But, really, nowadays, unless you’re on your phone, 99.9% of users have Javascript enabled in their web browsers.

If I had time, I would turn this into a plugin with options and such. Wishlist:

A separate panel for administering contact form messages, similar to the built-in WP comments.
A built-in easy-to-customize widget
Custom fields

All-in-all, this was a quick and easy way to safely implement a contact form on a site without the hassle of using multiple plugins, having Captchas, or the inflexibility of requiring a subject line for the contact message (yet another reason I chose to implement my own contact form from scratch).

Hope you found this article useful. Feedback welcome!