Samrat Man Singh

Hi! I'm Samrat, a programmer from Nepal.

Building a database-backed Clojurescript app

2012-10-17


In my previous post, I gave a pretty quick introduction to Clojurescript. If you haven’t already, I recommend you read through that post. This post assumes that you have some Clojure knowledge and already have Leiningen running.

In this post, I’ll show how to create a SQL database-backed Clojurescript app(you were expecting NoSQL, weren’t you?). For the lack of a better idea, I’m going to walk you through building a trivial app that helps keep track of books you’ve read. You can view the source code for the app on Github.

The Setup

We’ll use Noir as the back-end(with Hiccup generating the HTML); on the front-end besides using Clojurescript we’ll also use a Clojurescript library called Fetch, which makes client-server communication(as in AJAX) really easy and another one called enfocus for DOM manipulation(mainly stuff like event-handling). For dealing with the database we’ll use clojure.java.jdbc. To compile our Clojurescript we’ll use a Leiningen plugin called lein-cljsbuild.

So, first create a Noir project called books(I’m assuming you’re using Leiningen 2):

lein new noir books

Now, let’s add some dependencies and some Clojurescript-specific settings to our project.clj:

(defproject books "0.1.0-SNAPSHOT"
            :description "Books- A database-backed Clojurescript app."
            :dependencies [[org.clojure/clojure "1.4.0"]
                           [noir "1.3.0-beta3"]
                           [fetch "0.1.0-alpha2"]
                           [org.clojure/java.jdbc "0.2.3"]]
            :plugins [[lein-cljsbuild "0.2.8"]]
            :cljsbuild {
                        :builds [{
                                  :source-path "src/cljs"
                                  :compiler {
                                             :output-to "resources/public/js/main.js"
                                             :optimizations :whitespace
                                             :pretty-print true}}]}
            :main books.server)

If you’ve gone through the first post, this should be pretty self-explanatory.

Adding a database

The first thing we’re going to do is set up our database. For the sake of simplicity, in this post I’ll use SQLite, however I think its safe to advise you guys not to use SQLite in production. Anyway, you’ll also need to add [org.xerial/sqlite-jdbc "3.7.2"] to the list of dependencies.

Pull in the newly added dependency using lein deps, then create a file in src/books/models called db.clj. To that file add:

(ns books.models.db
  (:require [clojure.java.jdbc :as sql]))

(def db
  {:classname   "org.sqlite.JDBC"
   :subprotocol "sqlite"
   :subname     "db/database.db"
   })

(defn init-db []
  (try
    (sql/with-connection db
      (sql/create-table
       :books
       [:title "varchar(250)"]
       [:review "varchar(500)"]))
    (catch Exception ex
      (.getMessage (.getNextException ex)))))

We’ve set the path of the SQLite database to db/database.db, so you’ll need to create a folder called db at the root of the project. Now, to initialize the database, run lein repl then

=> (use 'books.models.db)
=> (init-db)

If you don’t get an error the database file should have been created. You can check if its present inside db/. The database will have a table called :books with just two fields- :title and :review.

Now, we’ll add some helper functions to db.clj to make our dealings with our database a lot simpler:

(defn add-book
  [book]
  (sql/with-connection
    db
    (sql/insert-record :books book)))

(defn db-read-all
  []
  (sql/with-connection db
    (sql/with-query-results result
      ["SELECT * FROM books"]
      (into [] result))))

The add-book function does exactly what you’d expect and the code should be pretty easy to understand. The argument to the function should be a Clojure hash-map, so a call to that function would look like:

(add-book {:title "Clojure Programming" :review "Great book. I really need to work on completing this one, though."})

The db-read-all function pulls all entries from the :books table and returns a vector of the entries.

Views

Now, we’ll work on our views. Open src/books/views/welcome.clj to edit it. This is what it should look like:

(ns books.views.welcome
  (:require [books.views.common :as common])
  (:use [noir.core :only [defpage]]
        books.models.db
        noir.fetch.remotes
        hiccup.form))

(defpage "/" []
         (common/layout
          [:h1 "Books"]
          [:div
           (label {} "title" "Title: ")
           (text-field {:class "title"} "title")
           [:br]
           (label {} "review" "Review: ")
           (text-area  {:class "review"} "review")
           [:br]
           [:button {:class "submit"} "Submit"]]))

(defremote add-book-to-db [book]
  (println book)
  (add-book book))

The most important part of this is the defremote definition. Its defining a fetch remote, which simply calls the add-book function from the books.models.db namespace that we defined above. The little println call is simply there to help us see in a short while whether our program is working.

Client-side

Now, we finally get to writing some Clojurescript code. Create a new file inside src/cljs/main.cljs and into it type in the following:

(ns books.cljs.main
  (:require [enfocus.core :as ef]
            [fetch.remotes :as remotes])
  (:require-macros [enfocus.macros :as em]
                   [fetch.macros :as fm]))

(defn get-book-title []
  (em/from (em/select ["#title"]) (em/get-prop :value)))

(defn get-book-review []
  (em/from (em/select ["#review"]) (em/get-prop :value)))

(defn get-book-data []
  {:title (get-book-title)
   :review (get-book-review)})

(defn push-book []
  (fm/remote (add-book-to-db (get-book-data))))

(em/defaction setup []
  [".submit"] (em/listen :click push-book))

(set! (.-onload js/window) setup)

In the namespace declaration you’ll notice that we’re bringing in stuff into our namespace from the Clojurescript libraries that we talked about in the beginning- Fetch and Enfocus. You’ve already seen how the server-side of our Fetch remote works, now you’ll see how the other half of it, the client-side works.

Starting from the top, the two functions get-book-title and get-book-review use enfocus to extract the value of the “title” and “review” fields in the browser. Read the enfocus docs to find out exactly how that works.

The function get-book-data simply puts the title and review into a Clojure map and returns it. push-book then pushes this map to the remote function we defined in our welcome.clj file.

The next block of code sets up a listener that calls the push-book function if the submit button is clicked. And the last line loads this listener when the web page loads.

Compile the Javascript using lein cljsbuild once and make sure you’ve added the Javascript file to your template(in common.clj). If you visit the browser now, you should see the form as expected. Fill in the title and review and hit “Submit”. And what happens? Nothing! Well, actually something does happen. If everything worked fine, the little println call in our remote function should have printed out some text in the process where you’re running the Noir server. Also, if you try running the db-read-all function we defined, you should see that a book was in fact added when you hit “Submit”.

Congratulations! You’ve created a Clojurescript application backed by a database. I know its a really trivial app, silly even but I do hope this post helped at least a few people get started with Clojurescript. And if you are interested in moving forward with this app, here are a few thoughts:


Getting started with Clojurescript

2012-10-14


There doesn’t seem to be much written about running Clojurescript, especially considering how great a tool it really is. I know there is a book that’s coming out soon, but I had some trouble getting started with Clojurescript so I decided to put together this post, that hopefully at least some of you will find useful. This post does assume that you have some knowledge of Clojure and that you’ve got Leiningen already running.

To those not familiar with Clojurescript, its a Clojure compiler that targets Javascript. This simply means that it turns Clojure code into Javascript. It’s like Coffeescript. To find out why you might want to use Clojurescript(and Clojure) check out this talk.

Getting started

As I said, you need to have Leiningen installed. For this post, I’ll use Noir as the backend for a really simple app that doesn’t do much. However, I’ll show how you can have the app’s client and server side communicate with each other, which’ll make use of Noir. So, we’ll just start off with a Noir project:

If you’re using Lein 1:

lein plugin install lein-noir 1.3.0-beta3
lein noir new cljsintro

And if you’re running Lein 2:

lein new noir cljsintro

Great! Now if you cd into your Noir project and do lein run your app should run and you should be able to see the default Noir page, when you visit http://localhost:8080 on your browser. Nothing special there. To be able to have your Clojurescript compile, we’ll use the lein-cljsbuild plugin. To do that, you need to add a couple of things to your project.clj:

(defproject cljsintro "0.1.0-SNAPSHOT"
             :description "A short intro to Clojurescript"
             :dependencies [[org.clojure/clojure "1.4.0"]
                            [noir "1.3.0-beta3"]]

             ;; Add lein-cljsbuild plugin
             :plugins [[lein-cljsbuild "0.2.8"]]

             ;; config. for cljsbuild
             :cljsbuild {
                         :builds [{
                                   :source-path "src/cljs"
                                   :compiler {
                                              :output-to "resources/public/js/main.js"
                                              :optimizations :whitespace
                                              :pretty-print true}}]}
             :main cljsintro.server)

We’ve added 2 main things to the default project.clj: :plugins and :cljsbuild. The :plugins part is pretty self-explanatory- we just added the lein-cljsbuild plugin to our project. The second thing that we added, :cljsbuild gives the plugin the configuration necessary to compile our Clojrurescript code. Let’s take a look at the configuration. Our :builds sequence contains only one map which means that we want all our code to compile with the same settings. Inside :builds, the :source-path tells the compiler where to look for the Clojurescript source files. And the :output-to tells the compiler where to put the compiled Javascript file.

Before talking about optimizations, lets tackle off :pretty-print- its pretty simple, setting it to true will cause the resulting JS file to have pretty-printed code, and setting-it to false will not. Now, to talk about optimizations- Clojurescript is compatible with with something called Google Closure(don’t confuse yourself between Closure and Clojure), which optimizes Javascript code. I’m really not familiar with Google Closure, but apparently, its really powerful and will help your code load and run faster. You can set :optimizations to three possible values: :whitespace, :simple and :advanced. Here, we have set it to :whitespace which is the most basic level of optimization but you can set it to :simple and :advanced when pushing code to production.

Clojurescript-ing

We’ve told the compiler that all our Clojurescript is to be found at src/cljs, so you’ll need to make that directory. Also, before writing any Clojurescript, let’s make a few changes to the Noir app. Open common.clj inside /src/cljsintro/views and make a few edits:

(ns cljsintro.views.common
  (:use [noir.core :only [defpartial]]
        [hiccup.page :only [include-css include-js html5]]))

(defpartial layout [& content]
        (html5
          [:head
           [:title "cljsintro"]
           (include-css "/css/reset.css")]
          [:body
           [:div#wrapper
            content]
           (include-js "/js/main.js")]))

I’ve made two changes to the default template- on line 3 I’ve added include-js, which we used on the last line to use main.js on our HTML files. Note that you didn’t have to type in the resources/public where the js folder lies in because Noir is already looking there for static files.

Now, finally lets create a file inside the cljs directory called main.cljs and add the following:

(ns cljs.main)

(js/alert "Hey, there")

That’s the Javascript equivalent of just alert("Hey There");. To compile it run

lein cljsbuild once

which will compile the code just once. Alternatively if you do lein cljsbuild auto, the compiler will watch for changes in the source-path and re-compile when a change is made.

Run the Noir app with lein run and if you visit http://localhost:8080/welcome you should see an alert box. Cool.

DOM

A lot of people use Javascript for manipulating the DOM- that is, adding effects like making things happen when buttons get clicked. You can do all of that stuff with Clojurescript. There are a couple of libraries available like jayq(which is a jQuery wrapper), domina and enfocus. I’ve personally used enfocus because its better documented compared to the other two. These are pretty easy to use.

Go, fetch

At the beginning I talked about making the client and server sides of our app talk to each other. Now, let’s do that using a neat library called fetch.

The first thing we’ll need to do is add fetch as a dependency. Strangely enough, fetch’s Github Readme page doesn’t tell what the latest version is and I have to go to project.clj to find it out. At the time of this writing its “0.1.0-alpha2”, so add [fetch "0.1.0-alpha2"] to :dependencies.

Then, add these two lines to your Clojurescript file’s namespace declaration:

(:require [fetch.remotes :as remotes])
(:require-macros [fetch.macros :as fm])

And let’s create a call to a function that’s on the server-side.

(fm/remote (adder 2 3 4) [result]
  (js/alert result))

This calls a remote function on the server which looks like this:

;; Add (:use noir.fetch.remotes) to the namespace

(defremote adder [& nums]
  (apply + nums))

Now recompile the Clojurescript code and refresh your browser, and you should be able to see the result of adder applied to the numbers we provided in a JS alert box. This is nothing special, as we could have defined adder in the Clojurescript code itself, but the same principle can be applied to use with functions that needs to be run on the server.

Hope you found this post useful; you can shoot out any questions on Twitter @samratmansingh or email me. Some resources that you might want to check out:


Fighting an N-headed monster with recursion

2012-10-07


A while back, I came upon this problem:

“You need to kill an N-headed monster. To do that, you have two swords. The first sword(A) cuts cutA heads, however, in case the monster doesn’t die(ie no. of heads > 0), it will grow growA heads. The second sword works the same way, except that it’ll cut cutB heads and in case the monster is still alive, it’ll grow growB heads. If a sword is used to result in the no. of monster heads being less than 0, you die.”

The problem is to find the shortest combination of swords that can be used to kill the monster(without killing yourself).

This is a paraphrase of the original question(so the question might have sounded a bit awkward). Here’s my solution to it, in Scheme(Racket):

(define cutA 7)
(define growA 3)
(define cutB 5)
(define growB 2)

(define (attack-monster heads)
  (use-sword heads 0 '()))

(define (use-sword n grow s)
  (cond ((< n 0) '())
         ((= n 0)
          (list s))
         (else
          (append
           (use-sword (- (+ n grow) cutA) growA (append s (list 'a)))
           (use-sword (- (+ n grow) cutB) growB (append s (list 'b)))))))

(define (shortest-way heads)
  (first (sort (attack-monster heads) (lambda (x y) (< (length x) (length y))))))

Here’s how to use it:

> (shortest-way 23)
'(a a a a a)

Thanks to skeeto on Reddit for helping me out with this, and more importantly for not showing me his code :)

Also, I’d love to see how you guys do this in a more efficient and more elegant ways.


A first look at provisioning with Puppet(on a Vagrant box)

2012-06-04


In my previous post, I talked about deploying a Flask app on a Vagrant box using Gunicorn and Nginx. The response I got was mind-blowing, so I’ve decided to write about another neat tool that’s awesome for deploying web apps- Puppet. Vagrant actually encourages its users to use it, and you should use it. There’s also an alternative to Puppet called Chef; some people prefer that over Puppet, so you might want to check it out.

Hopefully I’ll be able to demonstrate what Puppet does and why its awesome in this post. But please note that this isn’t meant as a comprehensive tutorial, you should check out Puppet’s docs for that. Also even though the Puppet docs asks you to get the Learning Puppet VM, I found it much more comfortable to use vagrant ssh for learning Puppet, so if you already have Vagraant installed, you might want to try that out too- just try running puppet inside the virtual machine.

What Puppet does is something called provisioning- that means that it makes computers do what they are supposed to do. In other words, it does the configuring for you. To understand what that means, let’s see it in action.

First create a Vagrant box,

::shell
mkdir codebase_with_puppet
cd codebase_with_puppet

vagrant init

You should now see Vagrantfile. Open it with a text editor, then uncomment the lines

::puppet
config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "manifests"
    puppet.manifest_file  = "base.pp"
end

Now, create base.pp inside a folder called manifests, and add the following to it.

::puppet
package {"nginx":
    ensure => present,
}

Now, run vagrant up. You’ll notice that Vagrant automatically installs nginx after it boots the VM. You should get a message like

::shell
notice: /Stage[main]//Package[nginx]/ensure: ensure changed 'purged' to 'present'

This can become a real treasure as this way, you won’t have to memorize what you need to install, to get the app running. All the system needs is to have puppet installed, after that puppet with the right manifests will handle everything.

Now, let’s do something different with Puppet- instead of installing another package, we’ll use it to configure nginx.

First, create a file in your local machine, inside codebase_with_puppet called codebase_nginx. To that file add the following

::nginx
server {
    location / {
        proxy_pass http://127.0.0.1:8000;
    }
}

If you’ve gone through the previous post you’ll notice that it’s the same configuration that we had used.

Now, we’ll use Puppet to make sure that the configuration file is placed where its supposed to be. To your base.pp file, add

::puppet 
file {"rm-nginx-default":
    path => '/etc/nginx/sites-enabled/default',
    ensure => absent,
    require => Package['nginx'],
}

file {"setup-nginx-codebase":
    path => '/etc/nginx/sites-enabled/codebase_nginx',
    ensure => present,
    require => Package['nginx'],
    source => "/vagrant/codebase_nginx",
}

Run vagrant reload and you’re done with the nginx configuration. Besides removing the repetitiveness for you, Puppet is also wonderful when you’re working on a team or on an open-source project. Now, all you need to do is write the manifests and once you share them you can rest assured that the entire team has the exact same environment.


Flask + Nginx + Gunicorn(on a Vagrant box)

2012-05-27


I had some difficulty in grasping how exactly to set up a server when I tried to do so recently, so I decided to write a tutorial that will guide you through the process. Hopefully, this post will help you avoid at least some of the confusion that I encountered.

We’ll be using Nginx + Gunicorn to host a simple Flask app. Many of you may not have access to a server but don’t worry, we’ll use Vagrant, which makes use of a VirtualBox VM to emulate a server.

###The Flask App

Because this is a post about deployment more than development, we’ll make the web app super-simple. If you’re not familiar with Flask, please check it out, its awesome and really easy to learn. You’ll also probably want to develop the app inside virtualenv- it makes things a lot neater. Make a folder in your local machine(we’re not working with the virtual-machine yet) for your app, I’ll call it codebase. Create two folders called static and templates, and a Python file called app.py. codebase should now look like this:

.
├── app.py
├── static
└── templates

Now, open app.py with a text editor and add the following:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "Hello world!"

if __name__ == '__main__':
    app.run()

At this point, if you run app.py with python app.py, you should be able to open http://localhost:5000/ and see a “Hello World!” printed. Now, freeze your requirements with

pip freeze > requirements.txt

Great, now we’ll start working on the actual server.

Vagrant

As I said before, Vagrant allows you to work with server-like environments on your local machine. It’s absolutely great. To get Vagrant up and running:

# first make sure Virtualbox is installed, then,
gem install vagrant
vagrant box add base http://files.vagrantup.com/precise32.box
vagrant init
vagrant up

If nothing went wrong, you should now see a file called Vagrantfile inside codebase- that’s Vagrant’s configuration file. Open the file, we’ll need to make a few changes to the file.

First, uncomment the line:

config.vm.network :hostonly, "192.168.33.10"
If nothing went wrong, you should now see a file called `Vagrantfile` inside `codebase`- that's Vagrant's configuration file. Open the file, we'll need to make a few changes to the file.

and change “192.168.33.10” to “33.33.33.33”. This will enable the host-machine(that is your computer) to access the webserver running on the VM.

That way we should be able to access a web app running in the VM’s localhost, on our machine.

Because, we did a vagrant up the Vagrant box should already be running. Now, run

vagrant reload

so that the changes we made to the Vagrantfile take place.

After the VM restarts, run

vagrant ssh

This allows you to run commands into the VM. Once inside the VM, we’ll need to get some things installed. Run

apt-get install nginx
pip install virtualenv

Now let’s create a folder inside the VM where we’ll keep the application

cd /home/vagrant
mkdir www
cd www
virtualenv --no-site-packages .
mkdir codebase

And let’s grab the application from our local machine

cp /vagrant/* /home/vagrant/www/codebase/

Note that while I used cp, its always a better idea to use git or some other version-control system. For more on that, I recommend that you read this post.

Then, activate the virtualenv we created.

cd /home/vagrant/www
source bin/activate

Install gunicorn with pip

pip install gunicorn

Also install the other Python dependencies your app has with

pip install -r requirements.txt

That will grab and install your app’s required dependencies like Flask.

Now, if you run

gunicorn /home/vagrant/www/codebase/app.py:app -b 127.0.0.1:8000

you’ll have your app running but if you try opening it from your browser you’ll find that you can’t actually see the “Hello World” message that we were expecting. That’s where nginx comes in.

Nginx

First of all, you need to start nginx with

/etc/init.d/nginx start

Then

rm /etc/nginx/sites-enabled/default
touch /etc/nginx/sites-available/codebase
ln -s /etc/nginx/sites-available/codebase  /etc/nginx/sites-enabled/codebase

To /etc/nginx/sites-enabled/codebase add

server {
        location / {
                proxy_pass http://127.0.0.1:8000;
        }
}

And restart nginx with

/etc/init.d/nginx restart

Now, from inside codebase run

gunicorn app:app -b localhost:8000

If everything went right, if you visit http://33.33.33.33/ you should now see the “Hello World!” message. Congratulations! You’ve successfully set up your own server.

Update- I’ve written a follow-up to this post which covers Puppet, a really handy tool that’s comes packaged with Vagrant- you can find the post here.