Use SSH ForwardAgent for fun and profit

Yesterday I discovered something amazing. With just one line of code you can bypass the annoyance of not being able to access private web services (i.e. GitHub) while SSHed into a machine. The method is called SSH agent forwarding and it’s changed my deployment workflow.

Here’s the quick version. For every Host in your ~/.ssh/config file you can add a new line that allows you to use your own local SSH keys even while SSHed into the remote server. Example:

Host myhost
HostName X.X.X.X
ForwardAgent yes # <-- This line is key

See that last line? That is all it takes to turn on agent forwarding whenever you SSH into the server called myhost. Cool right?

The workflow

So, you might be wondering why this matters. The answer is ease of deployment. For small and even mid-size projects it’s probably not too much to ask to SSH in to the actual server whenever you need to deploy new code. This may not be the best or easiest deployment workflow, but it not difficult and it’s quite simple.

Furthermore, if your deployment process ever breaks it will not be difficult to troubleshoot because you will understand every step as opposed to using some automated deployment method that makes everything ten times as complex.

Now that you have ForwardAgent turned on, you can do something like this:

  1. Change the codebase. Commit changes
  2. git push
  3. ssh into production server. cd into app directory
  4. git pull
  5. Restart your app, now with up-to-date code

I love this workflow, mainly because step 4 is much easier than it was before I discovered ForwardAgent. Since you can easily pull code from any private repo that you have SSH access to you can now use Git as your primary means of deployment.

Step 5 will depend on your setup and is up to you, but hopefully it’s nothing too onerous. If it is, then I suggest you check out Docker. It may just make your deployment life a whole lot better.

Managing state and controlled form fields with React

UPDATE (2016-07-28): As of React v15 valueLink and checkedLink are officially deprecated. I heavily advocate the use of both of these APIs in this article. If you want to see my thoughts on the deprecation see this comment below. For the official deprecation see the docs article here.


With React, you basically get two different ways to deal with forms:

  • Standard input elements that can be modified by the user
  • “Controlled” input elements that can only be modified programatically

The first type is pretty straightforward in React, you simply don’t provide a value prop:

React.createClass({
render (
<input type='text' />
);
});

The value of that <input> element can be edited by the user in a browser, just like one would expect in a normal we app.

The other type of input—a “Controlled” input—looks almost the same, but has a value property:

Read More

In search of the ultimate tooling workflow

Lately I’ve been exploring the various options out there for front-end tooling. I’m about to start a greenfield project next week and I figure I might as well do it right. The goal is not to look back in two months and wish I had implemented a better workflow. The tooling can always change, but in practice it usually doesn’t so it’s best to get it right the first time.

In short, here’s what I’ve decided on:

Webpack

Webpack is amazing. I never thought Browserify would be replaced in my workflow, but it has been. Webpack essentially does everything Browserify did and more. It let’s you require anything, including images, Markdown files, fonts and probably anything else you would need in a website.

So what?

Webpack let’s you truly componentize your applications by letting you keep everything organized in a single directory. For example, if you wanted to componentize your website’s header you could put the HTML, CSS, JS and all associated images in the same directory. No more fumbling with paths in you’re image src attribute.

Read More

Solving problems is the biggest win

As I ramp up my job search, I’ve recently become more introspective about why I like programming. It’s not uncommon during an initial phone screen to be asked some variation of the following “soft” interview question:

What are you passionate about?

or…

Why do you like software development?

As simple and possibly over-used as these questions are, they’re still good ones. So am I really passionate about software development? Hell yes. But that’s not the answer to the underlying question of why—why am I passionate about it?

It’s this line of thinking that has led me to do some introspection recently, and I’ve come to the conclusion that what it really boils down to is solving problems.

Solving problems is the biggest win

When you solve a problem for yourself it feels great. Not only that, but you then have one less problem. The satisfaction of solving problems certainly isn’t exclusive to programming, but programming is one of the best tools to solve an extremely wide array of problems, which is why I love it.

But as satisfying as solving one’s own problems can be, the biggest win in my mind is to …

Solve other people’s problems

This realization really solidified in my mind a number of months ago as I was combing through my GitHub profile to clean out old repositories I no longer use. It was when I first saw one of my repos getting starred by other people 😀.

Alfred Maestro Repo

Other people were using something I created, and apparently they liked it. The level of satisfaction I got from that project remains very high to this day.

The best part is, solving other people’s problems is not only valuable but it’s also motivating.

Maintaining motivation

Alfred Maestro (my most-starred repo) is a project I wrote over a year ago, using my least favorite language (PHP) but I’ve still maintained it and responded to any user issues throughout because other people were using it. It’s that motivation that has kept that little project going. That motivation is really powerful.

In this way solving someone’s problem can become a positive feedback loop: The more positive feedback you get about your project and the way it’s solving problems, the more you’re motivated to continue to solve that problem.

It’s a beautiful thing.

A New Direction

For the past few years I’ve been living in Taiwan running my own freelance business. It’s been an amazing, transformative experience and I’ve had a great time doing it. Entering 2015, I was planning to continue growing my business into the foreseeable future, but after a recent revelation I feel it’s time for a change.

It’s time to get a real job.

At my last “job” I was my own boss, I worked my own hours, I traveled often and lived however I wanted. So, why then would I choose to give all that up? The answer is simple, it’s because…

I’m in my comfort zone.

I woke up one morning and realized: I’ve acclimated. I’m no longer challenged on a daily basis. This was a hard realization to come by, but it’s true. The projects I had been working on might have been fun or interesting but they were ultimately well within my current skill set, which means I had stopped pushing.

“Never stop pushing.”

I once heard that quote somewhere and it really resonated with me. I never want to stop improving or stop iterating on my own talents. So I’ve decided to join a startup and find something I’m truly passionate about working on. I want to learn and grow at the same time as I provide value to a company I admire.

There are a number of companies I’m already very passionate about and likely many others I haven’t heard of yet, so…

The search is on.

For the next few months I’ll be staying in San Francisco learning, improving, meeting new people and looking for the right company. I don’t expect it to be easy, but it will be worthwhile.

Express Basics

A quick guide to using Express, and the basics of Express middleware

Intro

All source files for this article can be found in the GitHub Repo: iansinnott/express-middleware-lecture.

Outline

  1. Intro (Simplifying Express)
  2. What is middleware?
  3. The relationship between Express and middleware
    • Explanation (it’s just a function)
    • Build a simple logger
  4. Important points about middleware
    • MUST call next()
    • Order matters
    • Middleware can be localized to various routes
  5. Routes are also middleware
  6. Handling errors
    • 404
    • Internal server error (500)
  7. Conclusion
    • Express is just a stack of middleware
    • Common and useful middleware you will undoubtedly see
    • More resources

Simplifying Express

So first things first, rather than starting with everything and figuring out what it all does, let’s start with a base Express configuration and add things as we need them.

A standard Express app configuration using the express-generator NPM module will produce something like this:

var express = require('express');
var path = require('path');
var favicon = require('serve-favicon');
var logger = require('morgan');
var cookieParser = require('cookie-parser');
var bodyParser = require('body-parser');
var routes = require('./routes/index');
var users = require('./routes/users');
var app = express();
// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');
// uncomment after placing your favicon in /public
//app.use(favicon(__dirname + '/public/favicon.ico'));
app.use(logger('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
app.use('/', routes);
app.use('/users', users);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
var err = new Error('Not Found');
err.status = 404;
next(err);
});
// error handlers
if (app.get('env') === 'development') {
app.use(function(err, req, res, next) {
res.status(err.status || 500);
res.render('error', {
message: err.message,
error: err
});
});
}
app.use(function(err, req, res, next) {
res.status(err.status || 500);
res.render('error', {
message: err.message,
error: {}
});
});
module.exports = app;

There’s a lot going on there, and for anyone without experience using Express that may look daunting if all we really want is a super simple application.

Simplify all of the things!

The above configuration file also makes assumptions about what you will need in your project. This can be useful but it is quite detrimental to learning the ins and outs of Express, because it does too much automatically. Let’s start from a bare-bones Express app and expand as needed. Here’s our configuration file:

var express = require('express');
var app = express();
app.get('/', function(req, res) {
res.send("hello express");
});
app.listen(3000);

This file will responds to GET requests at /—the root URL. Any other requests will cause an Express error, because there is no request handler for any other route or method.

If you open your browser and go to localhost:3000 you should see the text “hello express”, or you could use CURL:

$ curl :3000
hello express

Now that we have a super simple Express app up and running, let’s move on to middleware.

Middleware

Middleware, in the context of Express is just a fancy term for a function that gets run on your server after a request has been received from a client and before a response is sent back. In other words, it’s a function that gets run in the middle of the requeset-response cycle.

So what does that look like in practice? Let’s take a look:

// Define your middleware
function uselessMiddleware(req, res, next) {
// Do some stuff...
next();
}
// "Use" your middlware
app.use(uselessMiddleware);

The above function (middleware) is in fact useless as implied by its name because it doesn’t do anything at all. It simply calls the next middleware in the middleware stack (more on that below).

When writing simple middleware, we often use an anonymous function, so let’s refactor the above as so:

app.use(function(req, res, next) {
// Do some stuff...
next();
});

Now let’s make our middleware do something.

app.use(function(req, res, next) {
console.log('hey nice middleware');
next();
});

This middleware is simply going to log a string to the console. To make sure it’s working, and to see how middleware works, let’s send a request to our server. Use either your browser or CURL from the command line:

$ curl :3000
hello express

Regardless of how you made the requeset, if you now look in your terminal you should see that our middleware was indeed run because “hey nice middleware” will have been printed to the command line.

Express and Middleware

The one thing I want you to take away from this lecture is that an Express app is simply a stack of middleware. In other words, as the Express website says:

An Express application is essentially a series of middleware calls.

To see how true this is, let’s look at a standard Express route:

app.get('/hello', function(req, res) {
res.send('Hello route');
});

This is a super simple Express route that will respond to a GET request at /hello with the text string “hello route”.

But what if we want some bit of code to be run on all requests instead of just GET requests to /hello? For instance, what if we wanted to implement our own logger that would log the method and path of every request to the command line? That’s where middleware comes in.

When we call app.use we are saying, “Every time a new request comes in run this function.”

Building a simple logger

To see this in action let’s built a simple logger that will output strings of the form:

[METHOD] [ROUTE] [DATE]

Example:

GET / Sun Jan 25 2015 22:27:07 GMT-0800 (PST)
GET /some-route Sun Jan 25 2015 22:27:07 GMT-0800 (PST)
POST /contact Sun Jan 25 2015 22:27:07 GMT-0800 (PST)

Using express middleware, we can accomplish this pretty easily:

app.use(function(req, res, next) {
var date = new Date();
console.log(req.method + ' ' + req.url + ' ' + date);
next();
});

Now in just five lines of code we’ve created a rudimentary logging system.

Important points about middleware

Middleware is super flexible and easy to use, but there are still a few important points that need to be noted.

Always call next

As you’ve seen in all the middleware examples so far, I make sure to call next at the end of the function. What would happen if we didn’t call next? Try it out. Remove next from our logging middleware above and run the app. Here’s the full source code for app.js:

var express = require('express');
var app = express();
app.use(function(req, res, next) {
var date = new Date();
console.log(req.method + ' ' + req.url + ' ' + date);
// WE DIDN'T CALL `next` HERE!!
});
app.get('/', function(req, res) {
res.send("hello express");
});
app.listen(3000);

Run the app, then make a request:

node app.js
curl :3000

Now your terminal will just sort of hang and do nothing. You will also see that the logging function was still run, because there will be a log entry in the terminal. This is because your app (app.js) received a request, called the logging middleware, logged the output and then… did nothing. The app is waiting for us to either send a response back to the client or call next. Since we didn’t do either, it just hangs until we hit ctrl-c to stop our server.

So the point is, always call next, otherwise you’re server will not only not respond but also not give you an error. It will simply do nothing, and leave the browser waiting for a response.

Order Matters

As the documentation says:

Middleware functions are executed sequentially, therefore the order of middleware inclusion is important.

This is nothing overly complex, but it’s important to be aware of. Here’s a quick example, again using our logging middleware from above:

var express = require('express');
var app = express();
// Route
app.get('/', function(req, res) {
res.send("hello express");
});
// Logging middleware added after route
app.use(function(req, res, next) {
var date = new Date();
console.log(req.method + ' ' + req.url + ' ' + date);
next();
});
app.listen(3000);

As before, run the app and send a request to the root URL at localhost:3000. The request that comes back from the server will have the text “hello express” as expected, but if you look at the console you will notice that our middleware did not run. This is because we added it after our route.

What happened?

Even though we added the logging middleware after the route, you might think it should still get run after our route is hit, but that’s not the case. Calling res.send actually ends the request-response cycle, because it sends a response to the client. After the req-res cycle has ended, no more middleware will be called, so our logging middleware never gets executed.

Middleware can be localized to routes

Global middleware is really useful, but sometimes it would be unecessary to run certain middleware for all routes and methods. That’s why Express let’s you pass an optional string as the initial argument to app.use. Imagine we have an admin area on our website at /admin. We want to check to make sure a user is logged in before they access the page. A simple way to accomplish this would be:

app.use('/admin', function(req, res, next) {
if (!req.user)
res.redirect('/login');
else
next();
});

This example assumes that req.user will be set if the user is already logged in, as is the case when using Passport JS. If there is no user present on the request then the browser will be redirected to login. If there is a user however then the middleware simply calls next and allows the process to continue as normal.

Routes are just more middleware

Usually when we talk about Express we talk about routes and middleware as separate, because conceptually they are. A “route” typically means the end of the middleware stack, the last response handler that is called and that decides ultimately how to respond to the request. However, to understand Express it’s important to understand that routes themselves are also “middleware”—simply functions that get called under certain circumstances.

As an example, take a look at these two completely equivalent “routes”. One is written as you would normally write a route, using app.get while the other is written as middleware, but the effect is exactly the same:

app.get('/', function(req, res) {
res.send('hello express');
});
app.use('/', function(req, res, next) {
if (req.method === 'GET')
res.send('hello express');
else
next();
});

Using middlware syntax and manually checking the request method is of course more verbose, and certainly not what you want to do in practice. But it’s important to realize that Express routes and Express middleware ware actually on in the same. This is why, as mentioned in the docs and above…

An Express application is essentially a series of middleware calls.

Error Handling

Until you are comfortable with Express as a “stack” of middleware that get called in the order they were added, it may not be obvious how errors are normally handled. There is no app.error method. The way errors are handled is again, through middleware:

// The rest of the app...
app.use(function(req, res, next) {
res.status(404).send("Oh no, there was an error...");
});

This middleware matches any route and any method, and all it does is send an error message to the browser. If you added this at the top of all your middleware your site would continually send this error message to the client. However, by putting this at the very end of our app configuration we can allow all routes that weren’t matched by previous middleware to effectively fall through and be caught here.

However, since not every error is a 404, it’s important to allow for other errors with other status codes. To accomplish this, Express recognizes middleware with four arguments instead of three as being an error handler.

app.use(function(err, req, res, next) {
res.status(err.status || 500);
res.send("Oh no, there was an error...");
});

In the eyes of Express, this middleware is clearly an error handler because it expects four arguments, the first of which is an error object. Note: If you don’t manually set a status code 200 will still be the default, which would of course be incorrect for responding with an error. If there is no status set on the err object then we default to the generic 500 status code.

So, now we have an error handler but if you restart your server and navigate to a URL that we know there’s no router for (ex: “/something”) you will get a standard Express error, NOT the custom error we just created. What happened?

Since we aren’t using any third party middleware yet, we know every piece of code that’s being executed when a request hits the server. As a result, we can see that at no point is an error generated. This is crucial, because errors in Express are not automatic. JavaScript errors will happily bring down your server, but if you want to handle them with easy you will need to manually call the next function and pass an error argument. That will let Express execute our custom error handling code as scene above. So, to make sure all 404’s do indeed generate an error, we can add a catch-all middleware right above our error handler:

app.use(function(req, res, next) {
var err = new Error("Page not found!");
err.status = 404;
next(err); // Call `next` with an argument
});
app.use(function(err, req, res, next) {
res.status(err.status || 500);
res.send("Oh no, there was an error...");
});

Now if we restart our server we will get the result we want: All errors, 404 or otherwise, will be directed to our final error handler and we will get a response containing the string “Oh no, there was an error…”.

A quick note on triggering errors

In the example above we called next with a single Error object as the lone argument. To generate an Express error and jump directly to error handling middleware you need only call next with a truthy argument. For example:

app.use(function(req, res, next) {
// All of these would trigger error handling middleware
next('hey there');
next(40);
next(true);
});

The only exception to the above is calling next('route'). Calling next with the special string 'route' will not cause an error. See the docs here for more on this.

Conclusion

So, once again to reiterate:

An Express application is essentially a series of middleware calls.

If you remember one thing from this lecture remember that, because it will give you a clear understanding of how Express works and more importantly how to extend it to suite your needs.

That being said, there is currently a plethora of useful middleware in the wild that you will most likely encounter at one point or another if you stick with Express.

Useful & Common Middleware

Body Parser

This is one you’re almost certain to see in most Express apps, as it’s useful for handling forms as well as AJAX requests.

Serve Static

This one makes it easy to statically serve a directory. For example, if you have a /public and you would like to put all your static files there (images, css, scripts) then you would want to use the static middleware to field requests for all resources under public. This is so common that the serve-static middleware is actually bundled with Express:

app.use(express.static(__dirname + '/public')));

Morgan

This is flexible logging software that can be used in development or production. Even though as we saw it’s quite simple to write your own logging software, it’s a better idea to use something standard like Morgan because you can be sure your logs will be well formatted, and it’s also one lest thing for you to test.

There’s a ton of middleware out there, so whenever you run into a feature that Express doesn’t have but you would like chances are there’s a middleware module for it.

Outro

As I mentioned in the intro, this guide and all JS source files can be found in the repo on GitHub: iansinnott/express-middleware-lecture.

This write-up was the content of an interview and guest lecture I had at Makersquare, a JavaScript bootcamp in San Francisco. The Markdown file for the presentation I gave can be found here. The presentation was created for use with Deckset, a Mac application for creating slideshows with Markdown.

Use RedCarpet to Render Markdown

Most of the markdown I write ends up on GitHub, so when I preview my markdown I on my computer I like it to look exactly like it will once I push it live to some repository. I also like to use the excellent Marked app to preview my markdown. Unfortunately, even though Marked claims to support Github Flavored Markdown (GFM) it leaves some features out. I’m not going to blame Marked for the incomplete implementation of GFM, they outsource the processing to Discount, so it’s probably not their fault. But certain features of GFM, like nested lists, are really useful so I decided to switch to RedCarpet, since that is the Gem GitHub itself uses to parse READMEs.

So, for anyone else who runs into the issue of Marked’s incomplete GFM support, simply install redcarpet and use it as a custom processor:

sudo gem install redcarpet

In the “Advanced” section of Marked’s preferences you can find the custom processor section:

Custom Marked Processor

Note: You might have to click the “Update Permissions” button since Marked may not have read access to wherever you’re redcarpet executable is located.

Once you’ve added the custom processor you should get a little nondescript button near the lower right-hand side of your Marked windows:

Custom processor button in Marked

If you click that button you should now get markdown rendered through your custom processor! Nice 😎

Hope that helps anyone who ran into the same problem I did.

Back to the JavaScript Basics

Today I was interviewed for a developer position at a San Francisco startup and I was caught off guard when I was asked to create a simple implementation of “inheritance” in JS. Inheritance is in quotes because, well, this is JS we’re talking about. There is no real inheritance in the classical sense, but we can certainly make it work. The interview question went something like this:

Create a class Animal with a walk method. Then create a class Dog that inherits from Animal.

Simple enough, right? Here’s what I wrote:

var _ = require('lodash');
function Animal() {}
Animal.prototype.walk = function() {
console.log("Hey there I'm an Animal walking");
};
function Dog() {
Animal.call(this);
}
_.extend(Dog.prototype, Animal.prototype);

This is certainly how I would have solved the inheritance problem in any of my projects, but as it turns out this isn’t the best solution in the context of someone evaluating your JS prowess. Firstly, my solution explicitly depends on Lo-Dash (Underscore.js would also work just as well). These days it’s hardly outlandish to assume that a given project would depend on either Lo-Dash or Underscore, so I don’t see anything wrong with this but it still wasn’t optimal in the given setting.

Then there’s the larger assumption my solution makes, which is that you’re compiling your source through Browserify. Since I now use Browserify religiously on every project it didn’t even occur to me to mention this implicit dependency. If the interviewer wasn’t familiar with Browserify himself he might have thought I was just crazy—requiring modules in a browser 😕. Anyway, the answer they were going for was to use Object.create for defining the prototype of the subclass:

function Animal() {}
Animal.prototype.walk = function() {
console.log("Hey there I'm an Animal walking");
};
function Dog() {
Animal.call(this);
}
Dog.prototype = Object.create(Animal.prototype);
Dog.prototype.constructor = Dog;

Both solutions accomplish the same thing, but the latter can simply be pasted into a browser console while the former would have to be run through Browserify and have Lo-Dash installed through NPM.

Back to the basics

What this brief experience made me realize is that there are still some basic JS methods that I’m not familiar with. Of course now it’s unlikely that I will forget Object.create. This was a very healthy realization, because despite the fact that I am unlikely ever to use Object.create in any project of my own it’s still good to know the ins and outs of the language you write your software in.

So, now that I’m quite familiar with Object.create I’m still going right back to my complex build process complete with browserify, stylus, jade and numerous Gulp tasks 😏. For anyone interested, my whole build process is implemented in a Slush generator I use for new Node projects. The readme could use some work, but it should give you a gist of how I do my development.

New Site Is Live

Yup, the new site is up and running! It’s been a bit of a long time coming, but my new website is finally live online. If you’re interested you can check it out at my home page here: iansinnott.com.

So what?

The big reason I’m super excited about this iteration of my home site is that it was designed entirely from scratch. The previous iterations of my site were all influenced heavily by designer friends of mine, but this latest work is my own. So you’ll have to excuse me if I’m a little more excited than usual about launching a new site.

Note quite done

All that being said, there is still work to be done. The new site isn’t entirely consistent with this blog. It’s not a very big deal, but I really like consistency so updating this site to match the new visual style is now on my to-do list. The new site is also thoroughly untested, so if you stumble upon a part of it that seems broken please don’t hesitate to get in touch and let me know.

New Site in the Works

One of my personal projects recently has been completely redesigning my website. Well, I’m excited to say it’s coming along quite nicely. Here is a preview of the new look:

New website mockup

If you have any feedback I’d love to hear it. Feel free to comment bellow.

Motivation

My motivation for redesigning the site pretty simple: The site hasn’t changed in over two years. Under “Work”, the old site lists a number of personal projects that don’t even exist anymore. The “Skills” section is likewise outdated, listing PHP, MySQL and Photoshop. These days those have all been replaced by Node.js/Ruby, MongoDB and Sketch respectively.

If I’m being honest with myself I don’t think the old site is that bad, but it’s time for a change. I’ll be launching sometime before Christmas. 😎