Eric Florenzano’s Blog

Server/Client With React, Part 3: Frontend Server

Apr 11, 2014

In the past two posts, we started writing the client code, and then made it build. What's stopping us from loading it in our browser? It's not being served yet! Let's fix that, starting with installing Connect -- the middleware we'll use to build our http server.

npm install --save connect

Why not some other thing like Koa, hapi, or mach? No real reason. Our needs are simple and any one of those would work well. I chose Connect because it's popular, it'd been around a while, and it seemed to work well enough.

Now let's create a file server.js right at the root of our project, starting with the basics, and we'll fill in more as we go:

var connect = require('connect');

// Set up the application and run it
var server = connect();
  .use(connect.static(__dirname + '/build'))
  .use(connect.logger())
  .use(connect.csrf())
  .use(connect.urlencoded())
  .use(connect.query())
  .use(connect.json())
  .listen(5000);

Running this file will start up a server listening on port 5000, serving any static files that are found in /build, which knows how to parse querystrings, form submissions, and json, and is protected against CSRF attacks. So far, so easy. Now let's add cookie sessions, where I've found that maxAge needs to be set per-request in Connect for some reason:

var connect = require('connect');

var IRLMOJI_COOKIE_SECRET = process.env['IRLMOJI_COOKIE_SECRET'];

function fixConnectCookieSessionHandler(req, res, next) {
  req.session.cookie.maxAge = 365 * 24 * 60 * 60 * 1000;
  return next();
}

// Set up the application and run it
var server = connect();
  .use(connect.static(__dirname + '/build'))
  .use(connect.logger())
  .use(connect.cookieParser())
  .use(connect.cookieSession({
    secret: IRLMOJI_COOKIE_SECRET,
    cookie: {maxAge: 365 * 24 * 60 * 60 * 1000, proxy: true}
  }))
  .use(fixConnectCookieSessionHandler)
  .use(connect.csrf())
  .use(connect.urlencoded())
  .use(connect.query())
  .use(connect.json())
  .listen(5000);

Now we're up and running with a cookie-based session system, but we're not yet using it. In fact, we're not using any of this yet, because we're not rendering or serving the main site yet. We can write that now:

// Note that we're importing from the build directory
var makeRouter = require('./build/javascript/router').makeRouter;
var routes = require('./build/javascript/routes');

function reactHandler(req, res, next) {
  // Render implemented here so it can capture req/res in its closure
  function render(reactComp, opts) {
    // We'll implement this next
  }

  var app = {
    render: render,
    isServer: function() {
      return true;
    },
    getUrl: function() {
      var proto = req.headers['x-forwarded-proto'] || 'http';
      return proto + '://' + req.headers.host + req.url;
    },
    getPath: function() {
      return req.url;
    }
  };

  var router = makeRouter(
    routes.getRoutes(app),
    routes.getNotFound(app)
  );

  router.go(app.getPath());
}

// ...

// Set up the application and run it
var server = connect();
  .use(connect.static(__dirname + '/build'))
  .use(connect.logger())
  // ...
  .use(reactHandler)
  .listen(5000);

The basic idea here is to build an app object that exactly mimics the functionality available on the app object in frontend/javascript/client.js that we built in part 1. To do so, we create this object on-the-fly using the information available to us from the request. Then we import that same simple router we used before, and tell the router to route and render its contents by calling the go function with the current path as a parameter.

How do we actually render it though? We left that function blank. Before we work on that, we need some sort of HTML template to work from. Let's build our basic HTML page template in frontend/page.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta content="IE=edge,chrome=1" http-equiv="X-UA-Compatible">
  <meta name="description" content="Take a pic that looks like an emoji!">
  <meta name="author" content="IRLMoji">
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <meta content="yes" name="apple-mobile-web-app-capable">
  <title>{{ PAGE_TITLE }}</title>
  <script src="//cdnjs.cloudflare.com/ajax/libs/es5-shim/2.2.0/es5-shim.min.js"></script>
  <script src="//cdnjs.cloudflare.com/ajax/libs/es5-shim/2.2.0/es5-sham.min.js"></script>
  <link href="//cdnjs.cloudflare.com/ajax/libs/font-awesome/4.0.3/css/font-awesome.min.css" rel="stylesheet">
  <link href="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
  <link href="{{ STYLE_PATH }}" rel="stylesheet" media="screen">
</head>
<body class="{{ BODY_CLASS }}">
  <div id="react-root">{{ BODY_CONTENT }}</div>
  <input style="display: none" type="hidden" id="csrftoken" name="csrf" value="{{ CSRF_TOKEN }}" />
  <script src="{{ SCRIPT_PATH }}"></script>
</body>
</html>

This "template" has no real logic (it's just a frame) so we can use basic variable substitution -- in this case, in the style of Django templates. So that's what our render function will have to do: determine what should be inserted for e.g. BODY_CONTENT and PAGE_TITLE, render the template with that content, and serve it up to the user. Here's a first stab at it:

var fs = require('fs');
var _ = require('lodash/dist/lodash.underscore');

var NODE_ENV = process.env['NODE_ENV'];
var PROD = NODE_ENV === 'production';

// Read the whole template into memory, no need to re-read it every request
var PAGE_TEMPLATE = fs.readFileSync('frontend/page.html');

function render(reactComp, opts) {
    opts = opts || {};

    // Render the React component to a string
    var bodyContent = React.renderComponentToString(reactComp);

    // Build up the list of variable substitutions
    var sub = {
      BODY_CLASS: opts.bodyClass || '',
      BODY_CONTENT: bodyContent,
      CSRF_TOKEN: req.csrfToken(),
      SCRIPT_PATH: '/javascript/compiled' + (PROD ? '.min' : '') + '.js',
      STYLE_PATH: '/styles/main' + (PROD ? '.min' : '') + '.css',
      PAGE_TITLE: opts.title || 'IRLMoji'
    };

    // Create a regex out of the variable substituion object
    var re = new RegExp('{{ (' + _.keys(sub).join('|') + ') }}', 'g');

    // Start the response
    res.writeHead(opts.statusCode || 200, {'Content-Type': 'text/html'});

    // Substitute all the variables and write it to the response to finish
    res.end(('' + PAGE_TEMPLATE).replace(re, function(m) {
      return sub[m.substring(3, m.length - 3)];
    }));
}

We could have used any templating language, but as you can see, most of what's going on happens inside of react-root, so this is all we'll need.

Hey, we're live! If we start up our server by running:

gulp watch

We'll see build directory cleaned up, then the files generated, then the server will start up. Cool! Let's open the browser to http://127.0.0.1:5000, and it should say "Hello, World!", as that's what we have in our handleIndex() function in frontend/javascript/routes.js.

You can check out what the fully-built demo site is doing in server.js by visiting the code on github.

What is happening here?

Now that we've got the basic structure of our site set up, what all is happening?

  • The server looks at the URL, routes to the right React component, and renders our hello world component to a string.
  • Then it interpolates that string into the html page template and servers it up to the user.
  • In that template, we've told the browser to load a script which is the browserified (and potentially minified) version of client.js, which is an implementation of the app that was used to render the page. (Whoa.)
  • The browser downloads and executes that script, which in turn runs its router on the client side and routes to the same component.
  • React does a fast checksum and notices that, hey, the markup we just generated on the client matches what was just served from the server, so it doesn't change the DOM.

So now we've loaded a javascript implementation of the website frontend, and attached it to the existing markup that was served down the wire. Pretty cool, but right now we're not taking advantage of that. Soon we will :)

What's Next?

  • Build the communications layer between the frontend and the API
  • Ensure that the client re-uses the same data the server used when it rendered
  • Build a basic IRLMoji timeline
  • Implement camera upload by interfacing with non-React JavaScript Dropzone.js
  • Finish building the app and deploy it

Server/Client With React, Part 2: The Build System

Apr 10, 2014

In the last post, we started writing the client glue code and our first React component. But we're using things like require() for something that should be run on the client, how will that work? Browserify can take a look at our client.js, walk the dependencies, and spit out something that browsers can actually execute. Let's set that up!

But first a bit about we'll structure the app. This is how things look currently:

frontend/
    javascript/
        client.js
        routes.js
        components/
            common.js

But these JavaScript files need processing. Some of them have JSX in them, which needs to be converted to pure JavaScript. It all needs to be transformed to work in the browser. Additionally, we may have other non-JS files that we want to preprocess. To address this, we'll do all this processing and save it in a directory adjacent to frontend named build.

Here's the basic idea of how things will look after this blog post:

build/
    images/
    javascript/
        compiled.js
        compiled.min.js
        client.js
        client.min.js
        routes.js
        routes.min.js
        components/
            common.js
            common.min.js
    styles/
        main.js
        main.min.js
frontend/
    images/
    javascript/
        client.js
        routes.js
        components/
            common.js
    styles/
        main.less
    tests/
gulpfile.js
package.json

First make sure you have a package.json file. If you're not familiar with this file, here's some light reading about it. We're going to use Gulp to do our processing, and we need a number of plugins to make that work. These commands should get you started:

npm install --save react react-tools lodash
npm install --save-dev browserify envify gulp gulp-util gulp-uglify \
    gulp-clean gulp-rename gulp-browserify gulp-less gulp-react \
    gulp-minify-css gulp-nodemon

Now that we have the dependencies installed, let's create a file named gulpfile.js which will describe to Gulp how to build the app:

var gulp = require('gulp');
var clean = require('gulp-clean');

gulp.task('clean', function() {
  return gulp.src(['build/*'], {read: false}).pipe(clean());
});

So far it's not that impressive, all we're doing is deleting anything inside the build/, directory, which doesn't even exist. We noted earlier that some of the JavaScript had JSX in it, and would need to be processed, so let's set that up by adding this to gulpfile.js:

var react = require('gulp-react');
var uglify = require('gulp-uglify');
var rename = require('gulp-rename');

// Parse and compress JS and JSX files

gulp.task('javascript', function() {
  // Listen to every JS file in ./frontend/javascript
  return gulp.src('frontend/javascript/**/*.js')
    // Turn React JSX syntax into regular javascript
    .pipe(react())
    // Output each file into the ./build/javascript/ directory
    .pipe(gulp.dest('build/javascript/'))
    // Optimize each JavaScript file
    .pipe(uglify())
    // Add .min.js to the end of each optimized file
    .pipe(rename({suffix: '.min'}))
    // Output each optimized .min.js file into the ./build/javascript/ dir
    .pipe(gulp.dest('build/javascript/'));
});

Now let's set up browserify to transform the require() calls into JavaScript the browser understands. You'll notice this matches the structure of the previous code block almost exactly, except we're swapping out a call to browserify() instead of a call to react():

var browserify = require('gulp-browserify');

gulp.task('browserify', ['javascript'], function() {
  return gulp.src('build/javascript/client.js')
    .pipe(browserify({transform: ['envify']}))
    .pipe(rename('compiled.js'))
    .pipe(gulp.dest('build/javascript/'))
    .pipe(uglify())
    .pipe(rename({suffix: '.min'}))
    .pipe(gulp.dest('build/javascript/'));
});

And like I mentioned, there are things other than JavaScript which need to be processed too!

var less = require('gulp-less');
var minifycss = require('gulp-minify-css');

gulp.task('styles', function() {
  return gulp.src('frontend/**/*.less')
    .pipe(less())
    .pipe(gulp.dest('build/'))
    .pipe(minifycss())
    .pipe(rename({suffix: '.min'}))
    .pipe(gulp.dest('build/'));
});

But no gulpfile.js is complete without a default task that will run when the user types just gulp in their console:

gulp.task('default', ['clean'], function() {
  return gulp.start('browserify', 'styles');
});

This says to run clean as a dependency, then to run the browserify and styles tasks in parallel, and finish when everything returns. So just to hammer this point home, all you need to do to build everything is to open a terminal, navigate to your project directory and type:

gulp

Local Development

Doing local development is slightly more tricky, but not so bad to set up. What we'd like is to be able to run a command, and then have it watch for changes and automatically rebuild the parts that have changed. Let's add a watch task which does this:

var nodemon = require('gulp-nodemon');

gulp.task('watch', ['clean'], function() {
  var watching = false;
  gulp.start('browserify', 'styles', function() {
    // Protect against this function being called twice. (Bug?)
    if (!watching) {
      watching = true;

      // Watch for changes in frontend js and run the 'javascript' task
      gulp.watch('frontend/**/*.js', ['javascript']);

      // Run the 'browserify_nodep' task when client.js changes
      gulp.watch('build/javascript/client.js', ['browserify_nodep']);

      // Watch for .less file changes and re-run the 'styles' task
      gulp.watch('frontend/**/*.less', ['styles']);

      // Start up the server and have it reload when anything in the
      // ./build/ directory changes
      nodemon({script: 'server.js', watch: 'build'});
    }
  });
});

Most of this is fairly self-explanatory except two things:

  • What is 'browserify_nodep'?
  • We haven't built server.js yet.

The latter we'll tackle in the next post, but the reason for 'browserify_nodep' is that we don't need/want all the javascript to be rebuilt every time just client.js changes. We have something already watching for that. So let's modify our browserify task and split it into browserify and browserify_nodep:

function browserifyTask() {
  return gulp.src('build/javascript/client.js')
    .pipe(browserify({
      transform: ['envify']
    }))
    .pipe(rename('compiled.js'))
    .pipe(gulp.dest('build/javascript/'))
    .pipe(uglify())
    .pipe(rename({suffix: '.min'}))
    .pipe(gulp.dest('build/javascript/'));
}

gulp.task('browserify', ['javascript'], browserifyTask);
gulp.task('browserify_nodep', browserifyTask);

This is why it's awesome that Gulpfiles are just javascript files: we can break out common functionality into a function, and apply it to different tasks. If you ever want to see the fully finished gulpfile.js for the final project, you can check it out here.

What's Next?

  • Write the server.js that mimics the client.js we've been building and acts as http server.
  • Build the communications layer between the frontend and the API
  • Ensure that the client re-uses the same data the server used when it rendered
  • Oh yeah, write our app :)

Server/Client With React, Part 1: Getting Started

Apr 09, 2014

A few months ago I wrote about how React.js has made it possible (and relatively easy) to write code that renders markup quickly on the server, and also can be executed on the client to perform all page updates. Now that we know what the goal is, we can explore how to put all these pieces together, or how I've managed to do it anyway.

Let's Build Something!

It's easiest to talk about concepts if we can also work on something concrete, so in the next few posts we'll build a little example site called IRLMoji, which asks users to post pictures that look like emoji. All credit goes to @dwiskus for originally starting this meme on Twitter. But we're going to turn it into its own app for demo purposes, which will look something like this:

A screenshot of what we are building

It's definitely not pretty (sorry), but it's got all the stuff we need to learn about React including auth, content creation, server communication, integration with third-party JS libraries, etc. If you'd like to follow along and see the code for the finished site, it's all up on GitHub: https://github.com/ericflo/irlmoji.

Let's Begin

There will be two entry points to our app: server.js, which will sit at the project's top level, and frontend/javascript/client.js. Let's start with the client, because it's a bit simpler. First let's start small and build a render() function, which will take a React component and some options, and render it to the page:

function render(reactComp, opts) {
  opts = opts || {};
  var elt = document.getElementById('react-root');
  React.renderComponent(reactComp, elt);
}

All we're doing here is calling React.renderComponent on the component that was passed in, rendering the component into that #react-root DOM element. We'll end up making this function do a bit more work in a future post, but for now let's move on to writing some handlers, and hooking them up to URL routes.

In the past I've used director to handle URL routing, but eventually I wrote a small router which escews hashbang-style URLs in favor of HTML5 pushState. If needed, we can always fall back to full page reloads, rather than resort to hashbangs. Now we've got this router to work with, let's put it to use, starting with a new file at frontend/javascript/routes.js:

/** @jsx React.DOM */

// Lodash is a fast/light underscore replacement that works better for me
var _ = require('lodash/dist/lodash.underscore');
var common = require('./components/common');

// Renders the index page, for now just "Hello, World!"
function handleIndex(app) {
  app.render(<p>Hello, World!</p>, {
    title: 'Welcome',
  });
}

// Prepares a handler for use. For now all we're doing is making sure
// that 'app' is passed as the first argument to the handler function.
function prepareHandler(app, handler) {
  return _.partial(handler, app);
}

// Gets a list of routes to be passed to the router
function getRoutes(app) {
  return [
    ['/', prepareHandler(app, handleIndex)]
  ];
}

// Gets a function that should be called when no routes match
function getNotFound(app) {
  return function() {
    // At the time this post was written, JSX does not allow namespacing,
    // so we need to grab a reference to the NotFound component class.
    var NotFound = common.NotFound;
    app.render(<NotFound />, {statusCode: 404});
  };
}

module.exports = {
  getRoutes: getRoutes,
  getNotFound: getNotFound
};

"Wait a minute," you may be thinking. What is this app thing that every function seems to refer to? The app is what I've chosen to call the object that is used to tie everything together. It's because of this object that we're able to use one codebase for both server and client. Right now all we're using is the render function that we created earlier, but through the interface of this app object. Let's go back to client.js and create a basic app object.

var app = {
  render: render,
  isServer: function() {
    return false;
  },
  getUrl: function() {
    return '' + window.location;
  },
  getPath: function() {
    return window.location.pathname + window.location.search;
  }
};

Here we've got a basic app object, with access to a render function and a few helpers like the ability to get the current path or get the full URL or to detect whether we're on the server or the client. We're building this in client.js, so we know we're not on the server, and can just return false.

Now that we have our app, and our routes, let's tie them together:

// Import the routes we created earlier
var routes = require('./routes');
// ...and the simple router we're using
var makeRouter = require('./router').makeRouter;

app.router = makeRouter(routes.getRoutes(app), routes.getNotFound(app));
app.router.start();

To finish up this part, we still have to create that NotFound React component, so let's create a new file in frontend/javascript/components/common.js with this as its content:

/** @jsx React.DOM */

var React = require('react/addons');

var NotFound = React.createClass({
  render: function() {
    return <p>That page could not be found.</p>;
  }
});

module.exports = {NotFound: NotFound};

What's Next?

It would be great if we could fire up our browsers now and see in action what we've built so far. Unfortunately, however, we haven't built the server yet. Here are some of the high level things that we're going to cover next:

  • Set up Gulp and Browserify to compile our node JavaScript into Browser JS
  • Write the server.js that mimics the client.js we've been building and acts as http server.
  • Build the communications layer between the frontend and the API
  • Ensure that the client re-uses the same data the server used when it rendered
  • Oh yeah, write our app :)

React: Finally, a great server/client web stack

Jan 23, 2014

We started with static websites, simple pages that could link between each other. Then we started constructing those pages on demand, customizing the contents of the page for each individual user. Next we started using Javascript to add more interactivity, and these pages started to become more like applications. But now we have XMLHttpRequest, and most new projects are moving towards building their pages entirely in the browser.

But when we do it purely in the browser, we have to make quite a few concessions: search engines have a hard time indexing that content, for example. It also takes longer for users to see the rendered content (not in every case, but in many, many cases.) Also if you're using a lot of plugins for your favorite library, or a heavyweight MVC framework, sometimes it can slow down interactivity.

I've thought for a long time (and blogged about it previously) that the ideal solution would fully render the markup on the server, deliver it to the client so that it can be shown to the user instantly. Then it would asynchronously load some Javascript that would attach to the rendered markup, and invisibly promote the page into a full app that can render its own markup. This code would also have to be fast enough to meet or exceed the performance of other modern solutions, or it won't be worth it.

What I've discovered is that enough of the pieces have come together, that this futuristic-sounding web environment is actually surprisingly easy to do now with React.js. React isn't a full-fledged framework, it's a library, so you'll have to build your own foundation and write some glue code. I've been rewriting an existing website in React (usually a terrible idea, we'll see about this time), and now that the code is settling down a little bit, I'd like to share how it works. This post will just be an overview, but up next I'll share some of my glue code and lessons learned etc.

React.js

First, a bit about React. It was released by Facebook and was apparently a collaboration between the Instagram and Facebook web engineers. It looks similar in shape but not in size to Angular.js, but even though it looks similar, it has a very different philosophy underneath. React has a few key ideas: a virtual DOM, a component framework, and a Javascript syntax extension.

Virtual DOM

The key innovation of React is that it has a pure-Javascript reimplementation of the browser's DOM. It builds the DOM tree in Javascript and then has two options for output: a root browser element, or a string. How can it output to a root browser element without having horrible performance? Core to React is a tree diffing algorithm that can look at the in-memory virtual DOM, and the real browser DOM, and build the optimal diff. Then it applies that diff to the browser's DOM.

The second option for output is equally interesting to me though. If you can walk this virtual DOM tree and render to a string, then you can do that all on the server and then send it down to the client. And since it's a tree, React simply stores a checksum in a data attribute on the root node, and if those checksums match, then React will simply attach its event handlers and do nothing more.

Component Framework

React's component framework is how you construct that virtual DOM. Each component has a render() function which returns a virtual DOM node with any number of children and event handlers attached. So at its core, React components are widgets that can render themselves and react to user input. At render time, components should only access two instance objects: props, and state.

Props are passed-in and are immutable, whereas state is assigned and updated entirely by the component itself. Callback functions can be passed from parent node to child node, as props, and attached directly to the virtual DOM nodes in the render() function. In fact, that's the primary way that components communicate with each other.

These components have a lifecycle, and ways to hook into state and prop changes. They also support mixins, which can hook into the component lifecycle and take care of things easily, like canceling timers when a component is removed from the DOM, for example.

JSX (A Javascript syntax extension)

JSX lets you write components without pulling your hair out. Instead of writing tedious code like this:

React.DOM.a({href: '/about'}, 'About this site')

It allows you to just write this right alongside your javascript:

<a href="/about">About this site</a>

The transformer is smart enough to not need any special indicators about where the markup blocks are, but it also allows you to drop into Javascript, like so:

<a href={this.props.aboutUrl}>About this site</a>

You can choose not to use this, but after getting over my initial distaste for it ("Ack! Who got markup in my code!"), I could never go back to not using it.


So fix it!

Jul 01, 2013

"Uhh, why is this single controller method 500 lines of code?" - Me, a few months into my first job after college. I remember saying it out loud, expecting to hear a faint chuckle from my co-workers, as if it were a joke. "We should really break this up into smaller functions," I said, confident that I was right.

What my coworker said in response surprised me in the best possible way, and what I learned then is something I still remind myself of to this day. He said, "You're right, that's too long. So fix it!" I responded with, "yeah, we really should." To which he responded, "You are one of us, so fix it!"

This became somewhat of a refrain. Someone would complain about something and another person would shout back at them, "So fix it!"

It's interesting how easy it is for us to associate with a group so strongly that it becomes part of our identity--part of who we are--yet in some situations it's easy to diffuse that responsibility and dismiss it as not being our own.

But, of course, he was right--I was one of "we", and "we" should fix it, so I should fix it! If you're running a small startup with just a few people, this mentality is easy to have. There are so few people contributing that you don't have much choice in the matter.

As a company grows, people start to specialize, to carve out their area, and to take some ownership. This can be a good thing! By allowing people to specialize, and hiring more people, we can build something bigger. But everyone's still in it together, and the individual is still part of "we". If you ever find yourself thinking "that's not my job," then it's a red flag that you've let things go too far.

All this being said, you have to find a balance. You can't fix every code smell you find, or you'd never get anything done. In fact there may be code that you recognize as bad, but you just don't have the knowledge or resources to fix it. That's just the name of the game, and that's fine, but you also can't fix nothing, or your tech debt will grow unbounded and that will end in disaster as well.

Anyway, it's been almost a year since I made the jump from working at a company of 2, to working at a company that's much much larger, and this early lesson is one thing that I often find myself thinking and reminding myself of.