Archive for the ‘Scripting’ Category

Node.js Asynchronicity and Callback Nesting

July 26, 2014

Just a heads up before we get started on this… Chrome DevTools (v35) now has the ability to show the full call stack of asynchronous JavaScript callbacks. As of writing this, if you develop on Linux you’ll want the dev channel. Currently my Linux Mint 13 is 3 versions behind. So I had to update to the dev channel until I upgraded to the LTS 17 (Qiana).

All code samples can be found at GitHub.

Deep Callback Nesting

AKA callback hell, temple of doom, often the functions that are nested are anonymous and often they are implicit closures. When it comes to asynchronicity in JavaScript, callbacks are our bread and butter. In saying that, often the best way to use them is by abstracting them behind more elegant APIs.

Being aware of when new functions are created and when you need to make sure the memory being held by closure is released (dropped out of scope) can be important for code that’s hot otherwise you’re in danger of introducing subtle memory leaks.

What is it?

Passing functions as arguments to functions which return immediately. The function (callback) that’s passed as an argument will be run at some time in the future when the potentially time expensive operation is done. This callback by convention has it’s first parameter as the error on error, or as null on success of the expensive operation. In JavaScript we should never block on potentially time expensive operations such as I/O, network operations. We only have one thread in JavaScript, so we allow the JavaScript implementations to place our discrete operations on the event queue.

One other point I think that’s worth mentioning is that we should never call asynchronous callbacks synchronously unless of course we’re unit testing them, in which case we should be rarely calling them asynchronously. Always allow the JavaScript engine to put the callback into the event queue rather than calling it immediately, even if you already have the result to pass to the callback. By ensuring the callback executes on a subsequent turn of the event loop you are providing strict separation of the callback being allowed to change data that’s shared between itself (usually via closure) and the currently executing function. There are many ways to ensure the callback is run on a subsequent turn of the event loop. Using asynchronous API’s like setTimeout and setImmediate allow you to schedule your callback to run on a subsequent turn. The Promises/A+ specification (discussed below) for example specifies this.

The Test

var assert = require('assert');
var should = require('should');
var requireFrom = require('requirefrom');
var sUTDirectory = requireFrom('post/nodejsAsynchronicityAndCallbackNesting');
var nestedCoffee = sUTDirectory('nestedCoffee');

describe('nodejsAsynchronicityAndCallbackNesting post integration test suite', function (done) {
   // if you don't want to wait for the machine to heat up assign minutes: 2.
   var minutes = 32;
   this.timeout(60000 * minutes);
   it('Test the ugly nested callback coffee machine', function (done) {

      var result = function (error, state) {
         var stateOutCome;
         var expectedErrorOutCome = null;
         if(!error) {
            stateOutCome = 'The state of the ordered coffee is: ' + state.description;
            stateOutCome.should.equal('The state of the ordered coffee is: beautiful shot!');
         } else {
            assert.fail(error, expectedErrorOutCome, 'brew encountered an error. The following are the error details: ' + error.message);
         }
         done();
      };

      nestedCoffee().brew(result);
   });
});
lets test

The System Under Test

'use strict';

module.exports = function nestedCoffee() {

   // We don't do instant coffee ####################################

   var boilJug = function () {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
   };
   var addInstantCoffeePowder = function () {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('Crappy instant coffee powder is being added.');
   };
   var addSugar = function () {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('Sugar is being added.');
   };
   var addBoilingWater = function () {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('Boiling water is being added.');
   };
   var stir = function () {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('Coffee is being stirred. Hmm...');
   };

   // We only do real coffee ########################################

   var heatEspressoMachine = function (state, callback) {
      var error = undefined;
      var wrappedCallback = function () {
         console.log('Espresso machine heating cycle is done.');
         if(!error) {
            callback(error, state);
         } else
            console.log('wrappedCallback encountered an error. The following are the error details: ' + error);
      };
      // Flick switch, check water.
      console.log('Espresso machine has been turned on and is now heating.');
      // Mutate state.
      // If there is an error, wrap callback with our own error function

      // Even if you call setTimeout with a time of 0 ms, the callback you pass is placed on the event queue to be called on a subsequent turn of the event loop.
      // Also be aware that setTimeout has a minimum granularity of 4ms for timers nested more than 5 deep. For several reasons we prefer to use setImmediate if we don't want a 4ms minimum wait.
      // setImmediate will schedule your callbacks on the next turn of the event loop, but it goes about it in a smarter way. Read more about it here: https://developer.mozilla.org/en-US/docs/Web/API/Window.setImmediate
      // If you are using setImmediate and it's not available in the browser, use the polyfill: https://github.com/YuzuJS/setImmediate
      // For this, we need to wait for our huge hunk of copper to heat up, which takes a lot longer than a few milliseconds.
      setTimeout(
         // Once espresso machine is hot callback will be invoked on the next turn of the event loop...
         wrappedCallback, espressoMachineHeatTime.milliseconds
      );
   };
   var grindDoseTampBeans = function (state, callback) {
      // Perform long running action.
      console.log('We are now grinding, dosing, then tamping our dose.');
      // To save on writing large amounts of code, the callback would get passed to something that would run it at some point in the future.
      // We would then return immediately with the expectation that callback will be run in the future.
      callback(null, state);
   };
   var mountPortaFilter = function (state, callback) {
      // Perform long running action.
      console.log('Porta filter is now being mounted.');
      // To save on writing large amounts of code, the callback would get passed to something that would run it at some point in the future.
      // We would then return immediately with the expectation that callback will be run in the future.
      callback(null, state);
   };
   var positionCup = function (state, callback) {
      // Perform long running action.
      console.log('Placing cup under portafilter.');
      // To save on writing large amounts of code, the callback would get passed to something that would run it at some point in the future.
      // We would then return immediately with the expectation that callback will be run in the future.
      callback(null, state);
   };
   var preInfuse = function (state, callback) {
      // Perform long running action.
      console.log('10 second preinfuse now taking place.');
      // To save on writing large amounts of code, the callback would get passed to something that would run it at some point in the future.
      // We would then return immediately with the expectation that callback will be run in the future.
      callback(null, state);
   };
   var extract = function (state, callback) {
      // Perform long running action.
      console.log('Cranking leaver down and extracting pure goodness.');
      state.description = 'beautiful shot!';
      // To save on writing large amounts of code, the callback would get passed to something that would run it at some point in the future.
      // We would then return immediately with the expectation that callback will be run in the future.

      // Uncomment the below to test the error.
      //callback({message: 'Oh no, something has gone wrong!'})
      callback(null, state);
   };
   var espressoMachineHeatTime = {
      // if you don't want to wait for the machine to heat up assign minutes: 0.2.
      minutes: 30,
      get milliseconds() {
         return this.minutes * 60000;
      }
   };
   var state = {
      description: ''
      // Other properties
   };
   var brew = function (onCompletion) {
      // Some prep work here possibly.
      heatEspressoMachine(state, function (err, resultFromHeatEspressoMachine) {
         if(!err) {
            grindDoseTampBeans(state, function (err, resultFromGrindDoseTampBeans) {
               if(!err) {
                  mountPortaFilter(state, function (err, resultFromMountPortaFilter) {
                     if(!err) {
                        positionCup(state, function (err, resultFromPositionCup) {
                           if(!err) {
                              preInfuse(state, function (err, resultFromPreInfuse) {
                                 if(!err) {
                                    extract(state, function (err, resultFromExtract) {
                                       if(!err)
                                          onCompletion(null, state);
                                       else
                                          onCompletion(err, null);
                                    });
                                 } else
                                    onCompletion(err, null);
                              });
                           } else
                              onCompletion(err, null);
                        });
                     } else
                        onCompletion(err, null);
                  });
               } else
                  onCompletion(err, null);
            });
         } else
            onCompletion(err, null);
      });
   };
   return {
      // Publicise brew.
      brew: brew
   };
};

What’s wrong with it?

  1. It’s hard to read, reason about and maintain
  2. The debugging experience isn’t very informative
  3. It creates more garbage than adding your functions to a prototype
  4. Dangers of leaking memory due to retaining closure references
  5. Many more…

What’s right with it?

  • It’s asynchronous

Closures are one of the language features in JavaScript that they got right. There are often issues in how we use them though.   Be very careful of what you’re doing with closures. If you’ve got hot code, don’t create a new function every time you want to execute it.

Resources

  • Chapter 7 Concurrency of the Effective JavaScript book by David Herman

 

Alternative Approaches

Ranging from marginally good approaches to better approaches. Keeping in mind that all these techniques add value and some make more sense in some situations than others. They are all approaches for making the callback hell more manageable and often encapsulating it completely, so much so that the underlying workings are no longer just a bunch of callbacks but rather well thought out implementations offering up a consistent well recognised API. Try them all, get used to them all, then pick the one that suites your particular situation. The first two examples from here are blocking though, so I wouldn’t use them as they are, they are just an example of how to make some improvements.

Name your anonymous functions

  1. They’ll be easier to read and understand
  2. You’ll get a much better debugging experience, as stack traces will reference named functions rather than “anonymous function”
  3. If you want to know where the source of an exception was
  4. Reveals your intent without adding comments
  5. In itself will allow you to keep your nesting shallow
  6. A first step to creating more extensible code

We’ve made some improvements in the next two examples, but introduced blocking in the arrays prototypes forEach loop which we really don’t want to do.

Example of Anonymous Functions

var boilJug = function () {
   // Perform long running action
};
var addInstantCoffeePowder = function () {
   // Perform long running action
   console.log('Crappy instant coffee powder is being added.');
};
var addSugar = function () {
   // Perform long running action
   console.log('Sugar is being added.');
};
var addBoilingWater = function () {
   // Perform long running action
   console.log('Boiling water is being added.');
};
var stir = function () {
   // Perform long running action
   console.log('Coffee is being stirred. Hmm...');
};
var heatEspressoMachine = function () {
   // Flick switch, check water.
   console.log('Espresso machine is being turned on and is now heating.');
};
var grindDoseTampBeans = function () {
   // Perform long running action
   console.log('We are now grinding, dosing, then tamping our dose.');
};
var mountPortaFilter = function () {
   // Perform long running action
   console.log('Portafilter is now being mounted.');
};
var positionCup = function () {
   // Perform long running action
   console.log('Placing cup under portafilter.');
};
var preInfuse = function () {
   // Perform long running action
   console.log('10 second preinfuse now taking place.');
};
var extract = function () {
   // Perform long running action
   console.log('Cranking leaver down and extracting pure goodness.');
};

(function () {
   // Array.prototype.forEach executes your callback synchronously (that's right, it's blocking) for each element of the array.
   return [
      'heatEspressoMachine',
      'grindDoseTampBeans',
      'mountPortaFilter',
      'positionCup',
      'preInfuse',
      'extract',
   ].forEach(
      function (brewStep) {
         this[brewStep]();
      }
   );
}());

anonymous functions

Example of Named Functions

Now satisfies all the points above, providing the same output. Hopefully you’ll be able to see a few other issues I’ve addressed with this example. We’re also no longer clobbering the global scope. We can now also make any of the other types of coffee simply with an additional single line function call, so we’re removing duplication.

var BINARYMIST = (function (binMist) {
   binMist.coffee = {
      action: function (step) {

         return {
            boilJug: function () {
               // Perform long running action
            },
            addInstantCoffeePowder: function () {
               // Perform long running action
               console.log('Crappy instant coffee powder is being added.');
            },
            addSugar: function () {
               // Perform long running action
               console.log('Sugar is being added.');
            },
            addBoilingWater: function () {
               // Perform long running action
               console.log('Boiling water is being added.');
            },
            stir: function () {
               // Perform long running action
               console.log('Coffee is being stirred. Hmm...');
            },
            heatEspressoMachine: function () {
               // Flick switch, check water.
               console.log('Espresso machine is being turned on and is now heating.');
            },
            grindDoseTampBeans: function () {
               // Perform long running action
               console.log('We are now grinding, dosing, then tamping our dose.');
            },
            mountPortaFilter: function () {
               // Perform long running action
               console.log('Portafilter is now being mounted.');
            },
            positionCup: function () {
               // Perform long running action
               console.log('Placing cup under portafilter.');
            },
            preInfuse: function () {
               // Perform long running action
               console.log('10 second preinfuse now taking place.');
            },
            extract: function () {
               // Perform long running action
               console.log('Cranking leaver down and extracting pure goodness.');
            }
         }[step]();
      },
      coffeeType: function (type) {
         return {
            'cappuccino': {
               brewSteps: function () {
                  return [
                     // Lots of actions
                  ];
               }
            },
            'instant': {
               brewSteps: function () {
                  return [
                     'addInstantCoffeePowder',
                     'addSugar',
                     'addBoilingWater',
                     'stir'
                  ];
               }
            },
            'macchiato': {
               brewSteps: function () {
                  return [
                     // Lots of actions
                  ];
               }
            },
            'mocha': {
               brewSteps: function () {
                  return [
                     // Lots of actions
                  ];
               }
            },
            'short black': {
               brewSteps: function () {
                  return [
                     'heatEspressoMachine',
                     'grindDoseTampBeans',
                     'mountPortaFilter',
                     'positionCup',
                     'preInfuse',
                     'extract',
                  ];
               }
            }
         }[type];
      },
      'brew': function (requestedCoffeeType) {
         var that = this;
         var brewSteps = this.coffeeType(requestedCoffeeType).brewSteps();
         // Array.prototype.forEach executes your callback synchronously (that's right, it's blocking) for each element of the array.
         brewSteps.forEach(function runCoffeeMakingStep(brewStep) {
            that.action(brewStep);
         });
      }
   };
   return binMist;

} (BINARYMIST || {/*if BINARYMIST is falsy, create a new object and pass it*/}));

BINARYMIST.coffee.brew('short black');

named functions


Web Workers

I’ll address these in another post.

Create Modules

Everywhere.

Legacy Modules (Server or Client side)

AMD Modules using RequireJS

CommonJS type Modules in Node.js

In most of the examples I’ve created in this post I’ve exported the system under test (SUT) modules and then required them into the test. Node modules are very easy to create and consume. requireFrom is a great way to require your local modules without explicit directory traversal, thus removing the need to change your require statements when you move your files that are requiring your modules.

NPM Packages

Browserify

Here we get to consume npm packages in the browser.

Universal Module Definition (UMD)

ES6 Modules

That’s right, we’re getting modules as part of the specification (15.2). Check out this post by Axel Rauschmayer to get you started.


Recursion

I’m not going to go into this here, but recursion can be used as a light weight solution to provide some logic to determine when to run the next asynchronous piece of work. Item 64 “Use Recursion for Asynchronous Loops” of the Effective JavaScript book provides some great examples. Do your self a favour and get a copy of David Herman’s book. Oh, we’re also getting tail-call optimisation in ES6.

 


EventEmitter

Still creates more garbage unless your functions are on the prototype, but does provide asynchronicity. Now we can put our functions on the prototype, but then they’ll all be public and if they’re part of a process then we don’t want our coffee process spilling all it’s secretes about how it makes perfect coffee. In saying that, if our code is hot and we’ve profiled it and it’s a stand-out for using to much memory, we could refactor EventEmittedCoffee to have its function declarations added to EventEmittedCoffee.prototype and perhaps hidden another way, but I wouldn’t worry about it until it’s been proven to be using to much memory.

Events are used in the well known Ganf Of Four Observer (behavioural) pattern (which I discussed the C# implementation of here) and at a higher level the Enterprise Integration Publish/Subscribe pattern. The Observer pattern is used in quite a few other patterns also. The ones that spring to mind are Model View Presenter, Model View Controller. The pub/sub pattern is slightly different to the Observer in that it has a topic/event channel that sits between the publisher and the subscriber and it uses contractual messages to encapsulate and transmit it’s events.

Here’s an example of the EventEmitter …

The Test

var assert = require('assert');
var should = require('should');
var requireFrom = require('requirefrom');
var sUTDirectory = requireFrom('post/nodejsAsynchronicityAndCallbackNesting');
var eventEmittedCoffee = sUTDirectory('eventEmittedCoffee');

describe('nodejsAsynchronicityAndCallbackNesting post integration test suite', function () {
   // if you don't want to wait for the machine to heat up assign minutes: 2.
   var minutes = 32;
   this.timeout(60000 * minutes);
   it('Test the event emitted coffee machine', function (done) {

      function handleSuccess(state) {
         var stateOutCome = 'The state of the ordered coffee is: ' + state.description;
         stateOutCome.should.equal('The state of the ordered coffee is: beautiful shot!');
         done();
      }

      function handleFailure(error) {
         assert.fail(error, 'brew encountered an error. The following are the error details: ' + error.message);
         done();
      }

      // We could even assign multiple event handlers to the same event. We're not here, but we could.
      eventEmittedCoffee.on('successfulOrder', handleSuccess).on('failedOrder', handleFailure);

      eventEmittedCoffee.brew();
   });
});

The System Under Test

'use strict';

var events = require('events'); // Core node module.
var util = require('util'); // Core node module.

var eventEmittedCoffee;
var espressoMachineHeatTime = {
   // if you don't want to wait for the machine to heat up assign minutes: 0.2.
   minutes: 30,
   get milliseconds() {
      return this.minutes * 60000;
   }
};
var state = {
   description: '',
   // Other properties
   error: ''
};

function EventEmittedCoffee() {

   var eventEmittedCoffee = this;

   function heatEspressoMachine(state) {
      // No need for callbacks. We can emit a failedOrder event at any stage and any subscribers will be notified.

      function emitEspressoMachineHeated() {
         console.log('Espresso machine heating cycle is done.');
         eventEmittedCoffee.emit('espressoMachineHeated', state);
      }
      // Flick switch, check water.
      console.log('Espresso machine has been turned on and is now heating.');
      // Mutate state.
      setTimeout(
         // Once espresso machine is hot event will be emitted on the next turn of the event loop...
         emitEspressoMachineHeated, espressoMachineHeatTime.milliseconds
      );
   }

   function grindDoseTampBeans(state) {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('We are now grinding, dosing, then tamping our dose.');
      eventEmittedCoffee.emit('groundDosedTampedBeans', state);
   }

   function mountPortaFilter(state) {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('Porta filter is now being mounted.');
      eventEmittedCoffee.emit('portaFilterMounted', state);
   }

   function positionCup(state) {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('Placing cup under portafilter.');
      eventEmittedCoffee.emit('cupPositioned', state);
   }

   function preInfuse(state) {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('10 second preinfuse now taking place.');
      eventEmittedCoffee.emit('preInfused', state);
   }

   function extract(state) {
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      console.log('Cranking leaver down and extracting pure goodness.');
      state.description = 'beautiful shot!';
      eventEmittedCoffee.emit('successfulOrder', state);
      // If you want to fail the order, replace the above two lines with the below two lines.
      // state.error = 'Oh no! That extraction came out far to fast.'
      // this.emit('failedOrder', state);
   }

   eventEmittedCoffee.on('timeToHeatEspressoMachine', heatEspressoMachine).
   on('espressoMachineHeated', grindDoseTampBeans).
   on('groundDosedTampedBeans', mountPortaFilter).
   on('portaFilterMounted', positionCup).
   on('cupPositioned', preInfuse).
   on('preInfused', extract);
}

// Make sure util.inherits is before any prototype augmentations, as it seems it clobbers the prototype if it's the other way around.
util.inherits(EventEmittedCoffee, events.EventEmitter);

// Only public method.
EventEmittedCoffee.prototype.brew = function () {
   this.emit('timeToHeatEspressoMachine', state);
};

eventEmittedCoffee = new EventEmittedCoffee();

module.exports = eventEmittedCoffee;

With using raw callbacks, we have to pass them (functions) around. With events, we can have many interested parties request (subscribe) to be notified when something that our interested parties are interested in happens (the event). The Observer pattern promotes loose coupling, as the thing (publisher) wanting to inform interested parties of specific events has no knowledge of it’s subscribers, this is essentially what a service is.

Resources


Async.js

Provides a collection of methods on the async object that:

  1. take an array and perform certain actions on each element asynchronously
  2. take a collection of functions to execute in specific orders asynchronously, some based on different criteria. The likes of async.waterfall allow you to pass results of a previous function to the next. Don’t underestimate these. There are a bunch of very useful routines.
  3. are asynchronous utilities

Here’s an example…

The Test

var assert = require('assert');
var should = require('should');
var requireFrom = require('requirefrom');
var sUTDirectory = requireFrom('post/nodejsAsynchronicityAndCallbackNesting');
var asyncCoffee = sUTDirectory('asyncCoffee');

describe('nodejsAsynchronicityAndCallbackNesting post integration test suite', function () {
   // if you don't want to wait for the machine to heat up assign minutes: 2.
   var minutes = 32;
   this.timeout(60000 * minutes);
   it('Test the async coffee machine', function (done) {

      var result = function (error, resultsFromAllAsyncSeriesFunctions) {
         var stateOutCome;
         var expectedErrorOutCome = null;
         if(!error) {
            stateOutCome = 'The state of the ordered coffee is: '
               + resultsFromAllAsyncSeriesFunctions[resultsFromAllAsyncSeriesFunctions.length - 1].description;
            stateOutCome.should.equal('The state of the ordered coffee is: beautiful shot!');
         } else {
            assert.fail(
               error,
               expectedErrorOutCome,
               'brew encountered an error. The following are the error details. message: '
                  + error.message
                  + '. The finished state of the ordered coffee is: '
                  + resultsFromAllAsyncSeriesFunctions[resultsFromAllAsyncSeriesFunctions.length - 1].description
            );
         }
         done();
      };

      asyncCoffee().brew(result)
   });
});

The System Under Test

'use strict';

var async = require('async');
var espressoMachineHeatTime = {
   // if you don't want to wait for the machine to heat up assign minutes: 0.2.
   minutes: 30,
   get milliseconds() {
      return this.minutes * 60000;
   }
};
var state = {
   description: '',
   // Other properties
   error: null
};

module.exports = function asyncCoffee() {

   var brew = function (onCompletion) {
      async.series([
         function heatEspressoMachine(heatEspressoMachineDone) {
            // No need for callbacks. We can just pass an error to the async supplied callback at any stage and the onCompletion callback will be invoked with the error and the results immediately.

            function espressoMachineHeated() {
               console.log('Espresso machine heating cycle is done.');
               heatEspressoMachineDone(state.error);
            }
            // Flick switch, check water.
            console.log('Espresso machine has been turned on and is now heating.');
            // Mutate state.
            setTimeout(
               // Once espresso machine is hot, heatEspressoMachineDone will be invoked on the next turn of the event loop...
               espressoMachineHeated, espressoMachineHeatTime.milliseconds
            );
         },
         function grindDoseTampBeans(grindDoseTampBeansDone) {
            // Perform long running action, delegating async tasks passing callback and returning immediately.
            console.log('We are now grinding, dosing, then tamping our dose.');
            grindDoseTampBeansDone(state.error);
         },
         function mountPortaFilter(mountPortaFilterDone) {
            // Perform long running action, delegating async tasks passing callback and returning immediately.
            console.log('Porta filter is now being mounted.');
            mountPortaFilterDone(state.error);
         },
         function positionCup(positionCupDone) {
            // Perform long running action, delegating async tasks passing callback and returning immediately.
            console.log('Placing cup under portafilter.');
            positionCupDone(state.error);
         },
         function preInfuse(preInfuseDone) {
            // Perform long running action, delegating async tasks passing callback and returning immediately.
            console.log('10 second preinfuse now taking place.');
            preInfuseDone(state.error);
         },
         function extract(extractDone) {
            // Perform long running action, delegating async tasks passing callback and returning immediately.
            console.log('Cranking leaver down and extracting pure goodness.');
            // If you want to fail the order, uncomment the below line. May as well change the description too.
            // state.error = {message: 'Oh no! That extraction came out far to fast.'};
            state.description = 'beautiful shot!';
            extractDone(state.error, state);

         }
      ],
      onCompletion);
   };

   return {
      // Publicise brew.
      brew: brew
   };
};

Other Similar Useful libraries


Adding to Prototype

Check out my post on prototypes. If profiling reveals you’re spending to much memory or processing time creating the objects that contain the functions that are going to be used asynchronously you could add the functions to the objects prototype like we did with the public brew method of the EventEmitter example above.


Promises

The concepts of promises and futures which are quite similar, have been around a long time. their roots go back to 1976 and 1977 respectively. Often the terms are used interchangeably, but they are not the same thing. You can think of the language agnostic promise as a proxy for a value provided by an asynchronous actions eventual success or failure. a promise is something tangible, something you can pass around and interact with… all before or after it’s resolved or failed. The abstract concept of the future (discussed below) has a value that can be mutated once from pending to either fulfilled or rejected on fulfilment or rejection of the promise.

Promises provide a pattern that abstracts asynchronous operations in code thus making them easier to reason about. Promises which abstract callbacks can be passed around and the methods on them chained (AKA Promise pipelining). Removing temporary variables makes it more concise and clearer to readers that the extra assignments are an unnecessary step.

JavaScript Promises

A promise (Promises/A+ thenable) is an object or sometimes more specifically a function with a then (JavaScript specific) method.
A promise must only change it’s state once and can only change from either pending to fulfilled or pending to rejected.

Semantically a future is a read-only property.
A future can only have its value set (sometimes called resolved, fulfilled or bound) once by one or more (via promise pipelining) associated promises.
Futures are not discussed explicitly in the Promises/A+, although are discussed implicitly in the promise resolution procedure which takes a promise as the first argument and a value as the second argument.
The idea is that the promise (first argument) adopts the state of the second argument if the second argument is a thenable (a promise object with a then method). This procedure facilitates the concept of the “future”

We’re getting promises in ES6. That means JavaScript implementers are starting to include them as part of the language. Until we get there, we can use the likes of these libraries.

One of the first JavaScript promise drafts was the Promises/A (1) proposal. The next stage in defining a standardised form of promises for JavaScript was the Promises/A+ (2) specification which also has some good resources for those looking to use and implement promises based on the new spec. Just keep in mind though, that this has nothing to do with the EcmaScript specification, although it is/was a precursor.

Then we have Dominic Denicola’s promises-unwrapping repository (3) for those that want to stay ahead of the solidified ES6 draft spec (4). Dominic’s repo is slightly less specky and may be a little more convenient to read, but if you want gospel, just go for the ES6 draft spec sections 25.4 Promise Objects and 7.5 Operations on Promise Objects which is looking fairly solid now.

The 1, 2, 3 and 4 are the evolutionary path of promises in JavaScript.

Node Support

Although it was decided to drop promises from Node core, we’re getting them in ES6 anyway. V8 already supports the spec and we also have plenty of libraries to choose from.

Node on the other hand is lagging. Node stable 0.10.29 still looks to be using version 3.14.5.9 of V8 which still looks to be about 17 months from the beginnings of the first sign of native ES6 promises according to how I’m reading the V8 change log and the Node release notes.

So to get started using promises in your projects whether your programming server or client side, you can:

  1. Use one of the excellent Promises/A+ conformant libraries which will give you the flexibility of lots of features if that’s what you need, or
  2. Use the native browser promise API of which all ES6 methods on Promise work in Chrome (V8 -> Soon in Node), Firefox and Opera. Then polyfill using the likes of yepnope, or just check the existence of the methods you require and load them on an as needed basis. The cujojs or jakearchibald shims would be good starting points.

For my examples I’ve decided to use when.js for several reasons.

  • Currently in Node we have no native support. As stated above, this will be changing soon, so we’d be polyfilling everything.
  • It’s performance is the least worst of the Promises/A+ compliant libraries at this stage. Although don’t get to hung up on perf stats. In most cases they won’t matter in context of your module. If you’re concerned, profile your running code.
  • It wraps non Promises/A+ compliant promise look-a-likes like jQuery’s Deferred which will forever remain broken.
  • Is compliant with spec version 1.1

The following example continues with the coffee making procedure concept. Now we’ve taken this from raw callbacks to using the EventEmitter to using the Async library and finally to what I think is the best option for most of our asynchronous work, not only in Node but JavaScript anywhere. Promises. Now this is just one way to implement the example. There are many and probably many of which are even more elegant. Go forth explore and experiment.

The Test

var should = require('should');
var requireFrom = require('requirefrom');
var sUTDirectory = requireFrom('post/nodejsAsynchronicityAndCallbackNesting');
var promisedCoffee = sUTDirectory('promisedCoffee');

describe('nodejsAsynchronicityAndCallbackNesting post integration test suite', function () {
   // if you don't want to wait for the machine to heat up assign minutes: 2.
   var minutes = 32;
   this.timeout(60000 * minutes);
   it('Test the coffee machine of promises', function (done) {

      var numberOfSteps = 7;
      // We could use a then just as we've used the promises done method, but done is semantically the better choice. It makes a bigger noise about handling errors. Read the docs for more info.
      promisedCoffee().brew().done(
         function handleValue(valueOrErrorFromPromiseChain) {
            console.log(valueOrErrorFromPromiseChain);
            valueOrErrorFromPromiseChain.errors.should.have.length(0);
            valueOrErrorFromPromiseChain.stepResults.should.have.length(numberOfSteps);
            done();
         }
      );
   });

});

The System Under Test

'use strict';

var when = require('when');
var espressoMachineHeatTime = {
   // if you don't want to wait for the machine to heat up assign minutes: 0.2.
   minutes: 30,
   get milliseconds() {
      return this.minutes * 60000;
   }
};
var state = {
   description: '',
   // Other properties
   errors: [],
   stepResults: []
};
function CustomError(message) {
   this.message = message;
   // return false
   return false;
}

function heatEspressoMachine(resolve, reject) {
   state.stepResults.push('Espresso machine has been turned on and is now heating.');
   function espressoMachineHeated() {
      var result;
      // result will be wrapped in a new promise and provided as the parameter in the promises then methods first argument.
      result = 'Espresso machine heating cycle is done.';
      // result could also be assigned another promise
      resolve(result);
      // Or call the reject
      //reject(new Error('Something screwed up here')); // You'll know where it originated from. You'll get full stack trace.
   }
   // Flick switch, check water.
   console.log('Espresso machine has been turned on and is now heating.');
   // Mutate state.
   setTimeout(
      // Once espresso machine is hot, heatEspressoMachineDone will be invoked on the next turn of the event loop...
      espressoMachineHeated, espressoMachineHeatTime.milliseconds
   );
}

// The promise takes care of all the asynchronous stuff without a lot of thought required.
var promisedCoffee = when.promise(heatEspressoMachine).then(
   function fulfillGrindDoseTampBeans(result) {
      state.stepResults.push(result);
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      return 'We are now grinding, dosing, then tamping our dose.';
      // Or if something goes wrong:
      // throw new Error('Something screwed up here'); // You'll know where it originated from. You'll get full stack trace.
   },
   function rejectGrindDoseTampBeans(error) {
      // Deal with the error. Possibly augment some additional insight and re-throw.
      if(state.errors[state.errors.length -1] !== error.message)
         state.errors.push(error.message);
      throw new CustomError(error.message);
   }
).then(
   function fulfillMountPortaFilter(result) {
      state.stepResults.push(result);
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      return 'Porta filter is now being mounted.';
   },
   function rejectMountPortaFilter(error) {
      // Deal with the error. Possibly augment some additional insight and re-throw.
      if(state.errors[state.errors.length -1] !== error.message)
         state.errors.push(error.message);
      throw new Error(error.message);
   }
).then(
   function fulfillPositionCup(result) {
      state.stepResults.push(result);
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      return 'Placing cup under portafilter.';
   },
   function rejectPositionCup(error) {
      // Deal with the error. Possibly augment some additional insight and re-throw.
      if(state.errors[state.errors.length -1] !== error.message)
         state.errors.push(error.message);
      throw new CustomError(error.message);
   }
).then(
   function fulfillPreInfuse(result) {
      state.stepResults.push(result);
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      return '10 second preinfuse now taking place.';
   },
   function rejectPreInfuse(error) {
      // Deal with the error. Possibly augment some additional insight and re-throw.
      if(state.errors[state.errors.length -1] !== error.message)
         state.errors.push(error.message);
      throw new CustomError(error.message);
   }
).then(
   function fulfillExtract(result) {
      state.stepResults.push(result);
      state.description = 'beautiful shot!';
      state.stepResults.push('Cranking leaver down and extracting pure goodness.');
      // Perform long running action, delegating async tasks passing callback and returning immediately.
      return state;
   },
   function rejectExtract(error) {
      // Deal with the error. Possibly augment some additional insight and re-throw.
      if(state.errors[state.errors.length -1] !== error.message)
         state.errors.push(error.message);
      throw new CustomError(error.message);

   }
).catch(CustomError, function (e) {
      // Only deal with the error type that we know about.
      // All other errors will propagate to the next catch. whenjs also has a finally if you need it.
      // Todo: KimC. Do the dealing with e.
      e.newCustomErrorInformation = 'Ok, so we have now dealt with the error in our custom error handler.';
      return e;
   }
).catch(function (e) {
      // Handle other errors
      e.newUnknownErrorInformation = 'Hmm, we have an unknown error.';
      return e;
   }
);

function brew() {
   return promisedCoffee;
}

// when's promise.catch is only supposed to catch errors derived from the native Error (etc) functions.
// Although in my tests, my catch(CustomError func) wouldn't catch it. I'm assuming there's a bug as it kept giving me a TypeError instead.
// Looks like it came from within the library. So this was a little disappointing.
CustomError.prototype = Error;

module.exports = function promisedCoffee() {
   return {
      // Publicise brew.
      brew: brew
   };
};

Resources


Testing Asynchronous Code

All of the tests I demonstrated above have been integration tests. Usually I’d unit test the functions individually not worrying about the intrinsically asynchronous code, as most of it isn’t mine anyway, it’s C/O the EventEmitter, Async and other libraries and there is often no point in testing what the library maintainer already tests.

When you’re driving your development with tests, there should be little code testing the asynchronicity. Most of your code should be able to be tested synchronously. This is a big part of the reason why we drive our development with tests, to make sure your code is easy to test. Testing asynchronous code is a pain, so don’t do it much. Test your asynchronous code yes, but most of your business logic should be just functions that you join together asynchronously. When you’re unit testing, you should be testing units, not asynchronous code. When you’re concerned about testing your asynchronicity, that’s called integration testing. Which you should have a lot less of. I discuss the ratios here.

As of 1.18.0 Mocha now has baked in support for promises. For fluent style of testing promises we have Chai as Promised.

Resources


There are plenty of other resources around working with promises in JavaScript. For myself I found that I needed to actually work with them to solidify my understanding of them. With Chrome DevTool async option, we’ll soon have support for promise chaining.

Other Excellent Resources

And again all of the code samples can be found at GitHub.

Exploring JavaScript Prototypes

June 28, 2014

Not to be confused with the GoF Prototype pattern that defines a lot more than the simple JavaScript prototype. Although the abstract concept of the prototype is the same.

My intention with this post is to arm our developers with enough information around JavaScript prototypes to know when they are the right tool for the job as opposed to other constructs when considering how to create polymorphic JavaScript that’s performant and easy to maintain. Often performant code and easy to maintain code are in conflict with each other. I.E. if you want code that’s fast, it’s often hard to read and if you want code that’s really easy to read, it “may” not be as fast as it could/should be. So we make trade-offs.

Make your code as readable as possible in as many places as possible. The more eyes that are going to be on it, generally the more readable it needs to be. Where performance really matters, we “may” have to carefully sacrifice some precious readability to achieve the essential performance required. This really needs measuring though, because often we think we’re writing fast code that either doesn’t matter or that just isn’t fast. So we should always favour readability, then profile your running application in an environment as close to production as possible. This removes the guess work, which we usually get wrong anyway. I’m currently working on a Node.js performance blog post in which I’ll attempt to address many things to do with performance. What I’m finding a lot of the time is that techniques that I’ve been told are essential for fast code are all to often incorrect. We must measure.

Some background

Before we do the deep dive thing, lets step back for a bit. Why do prototypes matter in JavaScript? What do prototypes do for us? Where do prototypes fit into the design philosophy of JavaScript?

What do JavaScript Prototypes do for us?

Removal of Code Duplication (DRY)

Excellent for reducing unnecessary duplication of members that will need garbage collecting

Performance

Prototypes also allow us to maximise economy of memory, thus reducing Garbage Collection (GC) activity, thus increasing performance. There are other ways to get this performance though. Prototypes which obtain re-use of the parent object are not always the best way to get the performance benefits we crave. You can see here under the “Cached Functions in the Module Pattern” section that using closure (although not mentioned) which is what modules leverage, also gives us the benefit of re-use, as the free variable in the outer scope is baked into the closure. Just check the jsperf for proof.

The Design Philosophy of JavaScript and Prototypes

Prototypal inheritance was implemented in JavaScript as a key technique to support the object oriented principle of polymorphism. Prototypal inheritance provides the flexibility of being able to choose what the more specific object is going to inherit, rather than in the classical paradigm where you’re forced to inherit all the base class’s baggage whether you want it or not.

Three obvious ways to achieve polymorphism:

  1. Composition (creating an object that composes a contract to another object)(has-a relationship). Learn the pros and cons. Use when it makes sense
  2. Prototypal inheritance (is-a relationship). Learn the pros and cons. Use when it makes sense
  3. Monkey Patching courtesy of call, apply and bind
  4. Classical inheritance (is-a relationship). Why would you? Please don’t try this at home in production ;-)

Of course there are other ways and some languages have unique techniques to achieve polymorphism. like templates in C++, generics in C#, first-class polymorphism in Haskell, multimethods in Clojure, etc, etc.

Diving into the Implementation Details

Before we dive into Prototypes…

What does Composition look like?

There are many great examples of how composing our objects from other object interfaces whether they’re owned by the composing object (composition), or aggregated from independent objects (aggregation), provide us with the building blocks to create complex objects to look and behave the way we want them to. This generally provides us with plenty of flexibility to swap implementation at will, thus overcoming the tight coupling of classical inheritance.

Many of the Gang of Four (GoF) design patterns we know and love leverage composition and/or aggregation to help create polymorphic objects. There is a difference between aggregation and composition, but both concepts are often used loosely to just mean creating objects that contain other objects. Composition implies ownership, aggregation doesn’t have to. With composition, when the owning object is destroyed, so are the objects that are contained within the owner. This is not necessarily the case for aggregation.

An example: Each coffee shop is composed of it’s own unique culture. Each coffee shop has a different type of culture that it fosters and the unique culture is an aggregation of its people and their attributes. Now the people that aggregate the specific coffee shop culture can also be a part of other cultures that are completely separate to the coffee shops culture, they could even leave the current culture without destroying it, but the culture of the specific coffee shop can not be the same culture of another coffee shop. Every coffee shops culture is unique, even if only slightly.

Programmer Show Pony
programmer show pony

Following we have a coffeeShop that composes a culture. We use the Strategy pattern within the culture to aggregate the customers. The Visit function provides an interface to encapsulate the Concrete Strategy, which is passed as an argument to the Visit constructor and closed over by the describe method.

// Context component of Strategy pattern.
var Programmer = function () {
   this.casualVisit = {};
   this.businessVisit = {};
   // Add additional visit types.
};
// Context component of Strategy pattern.
var ShowPony = function () {
   this.casualVisit = {};
   this.businessVisit = {};
   // Add additional visit types.
};
// Add more persons to make a unique culture.

var customer = {
   setCasualVisitStrategy: function (casualVisit) {
      this.casualVisit = casualVisit;
   },
   setBusinessVisitStrategy: function (businessVisit) {
      this.businessVisit = businessVisit;
   },
   doCasualVisit: function () {
      console.log(this.casualVisit.describe());
   },
   doBusinessVisit: function () {
      console.log(this.businessVisit.describe());
   }
};

// Strategy component of Strategy pattern.
var Visit = function (description) {
   // description is closed over, so it's private. Check my last post on closures for more detail
   this.describe = function () {
      return description;
   };
};

var coffeeShop;

Programmer.prototype = customer;
ShowPony.prototype = customer;

coffeeShop = (function () {
   var culture = {};
   var flavourOfCulture = '';
   // Composes culture. The specific type of culture exists to this coffee shop alone.
   var whatWeWantExposed = {
      culture: {
         looksLike: function () {
            console.log(flavourOfCulture);

         }
      }
   };

   // Other properties ...
   (function createCulture() {
      var programmer = new Programmer();
      var showPony = new ShowPony();
      var i = 0;
      var propertyName;

      programmer.setCasualVisitStrategy(
         // Concrete Strategy component of Strategy pattern.
         new Visit('Programmer walks to coffee shop wearing jeans and T-shirt. Brings dog, Drinks macchiato.')
      );
      programmer.setBusinessVisitStrategy(
         // Concrete Strategy component of Strategy pattern.
         new Visit('Programmer brings software development team. Performs Sprint Planning. Drinks long macchiato.')
      );
      showPony.setCasualVisitStrategy(
         // Concrete Strategy component of Strategy pattern.
         new Visit('Show pony cycles to coffee shop in lycra pretending he\'s just done a hill ride. Struts past the ladies chatting them up. Orders Chai Latte.')
      );
      showPony.setBusinessVisitStrategy(
         // Concrete Strategy component of Strategy pattern.
         new Visit('Show pony meets business friends in suites. Pretends to work on his macbook pro. Drinks latte.')
      );

      culture.members = [programmer, showPony, /*lots more*/];

      for (i = 0; i < culture.members.length; i++) {
         for (propertyName in culture.members[i]) {
            if (culture.members[i].hasOwnProperty(propertyName)) {
               flavourOfCulture += culture.members[i][propertyName].describe() + '\n';
            }
         }
      }

   }());
   return whatWeWantExposed;
}());

coffeeShop.culture.looksLike();
// Programmer walks to coffee shop wearing jeans and T-shirt. Brings dog, Drinks macchiato.
// Programmer brings software development team. Performs Sprint Planning. Drinks long macchiato.
// Show pony cycles to coffee shop in lycra pretending he's just done a hill ride. Struts past the ladies chatting them up. Orders Chai Latte.
// Show pony meets business friends in suites. Pretends to work on his macbook pro. Drinks latte.

Now for Prototype

EcmaScript 5

In ES5 we’re a bit spoilt as we have a selection of methods on Object that help with prototypal inheritance.

Object.create takes an argument that’s an object and an optional properties object which is a EcmaScript 5 property descriptor like the second parameter of Object.defineProperties and returns a new object with the first argument passed as it’s prototype and the properties described in the property descriptor (if present) added to the returned object.

prototypal inheritance
// The object we use as the prototype for hobbit.
var person = {
   personType: 'Unknown',
   backingOccupation: 'Unknown occupation',
   age: 'Unknown'
};

var hobbit = Object.create(person);

Object.defineProperties(person, {
   'typeOfPerson': {
      enumerable: true,
      value: function () {
         if(arguments.length === 0)
            return this.personType;
         else if(arguments.length === 1 && typeof arguments[0] === 'string')
            this.personType = arguments[0];
         else
            throw 'Number of arguments not supported. Pass 0 arguments to get. Pass 1 string argument to set.';
      }
   },
   'greeting': {
      enumerable: true,
      value: function () {
         console.log('Hi, I\'m a ' + this.typeOfPerson() + ' type of person.');
      }
   },
   'occupation': {
      enumerable: true,
      get: function () {return this.backingOccupation;},
      // Would need to add some parameter checking on the setter.
      set: function (value) {this.backingOccupation = value;}
   }
});

// Add another property to hobbit.
hobbit.fatAndHairyFeet = 'Yes indeed!';
console.log(hobbit.fatAndHairyFeet); // 'Yes indeed!'
// prototype is unaffected
console.log(person.fatAndHairyFeet); // undefined

console.log(hobbit.typeOfPerson()); // 'Unknown '
hobbit.typeOfPerson('short and hairy');
console.log(hobbit.typeOfPerson()); // 'short and hairy'
console.log(person.typeOfPerson()); // 'Unknown'

hobbit.greeting(); // 'Hi, I'm a short and hairy type of person.'

person.greeting(); // 'Hi, I'm a Unknown type of person.'

console.log(hobbit.age); // 'Unknown'
hobbit.age = 'young';
console.log(hobbit.age); // 'young'
console.log(person.age); // 'Unknown'

console.log(hobbit.occupation); // 'Unknown occupation'
hobbit.occupation = 'mushroom hunter';
console.log(hobbit.occupation); // 'mushroom hunter'
console.log(person.occupation); // 'Unknown occupation'

Object.getPrototypeOf

console.log(Object.getPrototypeOf(hobbit));
// Returns the following:
// { personType: 'Unknown',
//   backingOccupation: 'Unknown occupation',
//   age: 'Unknown',
//   typeOfPerson: [Function],
//   greeting: [Function],
//   occupation: [Getter/Setter] }

 

EcmaScript 3

One of the benefits of programming in ES 3, is that we have to do more work ourselves, thus we learn how some of the lower level language constructs actually work rather than just playing with syntactic sugar. Syntactic sugar is generally great for productivity, but I still think there is danger of running into problems when you don’t really understand what’s happening under the covers.

So lets check out what really goes on with….

Prototypal Inheritance

What is a Prototype?

All objects have a prototype, but not all objects reveal their prototype directly by a property called prototype. All prototypes are objects.

So, if all objects have a prototype and all prototypes are objects, we have an inheritance chain right? That’s right. See the debug image below.

All properties that you may want to add to an objects prototype are shared through inheritance by all objects sharing the prototype.

So, if all objects have a prototype, where is it stored? All objects in JavaScript have an internal property called [[Prototype]]. You won’t see this internal property. All prototypes are stored in this internal property. How this internal property is accessed is dependant on whether it’s object is an object (object literal or object returned from a constructor) or a function. I discuss how this works below. When you dereference an object in order to find a property, the engine will first look on the current object, then the prototype of the current object, then the prototype of the prototype object and so on up the prototype chain. It’s a good idea to try and keep your inheritance hierarchies as shallow as possible for performance reasons.

Prototypes in Functions

Every function object is created with a prototype property, whether it’s a constructor or not. The prototype property has a value which is a constructor property which has a value that’s actually the function. See the below example to help clear it up. ES3 and ES5 spec 13.2 say pretty much the same thing.

var MyConstructor = function () {};
console.log(MyConstructor.prototype.constructor === MyConstructor); // true

and to help with visualising, see the below example and debug. myObj and myObjLiteral are for the two code examples below the debug image.

var MyConstructor = function () {};
var myObj = new MyConstructor();
var myObjLiteral = {};

Accessing JavaScript Prototypes

 

Up above in the composition example on line 40 and 41, you can see how we access the prototype of the constructor. We can also access the prototype of the object returned from the constructor like this:

var MyConstructor = function () {};
var myObj = new MyConstructor();
console.log(myObj.constructor.prototype === MyConstructor.prototype); // true

We can also do similar with an object literal. See below.

Prototypes in Objects that are Not Functions

Every object that is not a function is not created with a prototype property (All objects do have the hidden internal [[Prototype]] property though). Now sometimes you’ll see Object.prototype talked about. Even MDN make the matter a little confusing IMHO. In this case, the Object is the Object constructor function and as discussed above, all functions have the prototype property.

When we create object literals, the object we get is the same as if we ran the expression new Object(); (see ES3 and ES5 11.1.5)
So although we can access the prototype property of functions (that may or not be constructors), there is no such exposed prototype property directly on objects returned by constructors or on object literals.
There is however conveniently a constructor property directly on all objects returned by constructors and on object literals (as you can think of their construction procedure producing the same result). This looks similar to the above debug image:

var myObjLiteral = {};
            // ES3 ->                              // ES5 ->
console.log(myObjLiteral.constructor.prototype === Object.getPrototypeOf(myObjLiteral)); // true

I’ve purposely avoided discussing the likes of __proto__ as it’s not defined in EcmaScript and there’s no valid reason to use something that’s not standard.

Polyfilling to ES5

Now to get a couple of terms used in web development well defined before we start talking about them:

  • A shim is a library that brings a new API to an environment that doesn’t support it by using only what the older environment supports to support the new API.
  • A polyfill is some code in the form of a function, module, plugin, etc that provides the functionality of a later environment (ES5 for example) if it doesn’t exist for an older environment (ES3 for example). The polyfill often acts as a fallback. The programmer writes code targeting the newer environment as though the older environment doesn’t exist, but when the code is pulled into the older environment the polyfill kicks into action as the new language feature isn’t yet implemented natively.

If you’re supporting older browsers that don’t have full support for ES5, you can still use the ES5 additions so long as you provide ES5 polyfills. es5-shim is a good choice for this. Checkout the html5please ECMAScript 5 section for a little more detail. Also checkout Kangax’s ECMAScript 5 compatibility table to see which browsers currently support which ES5 language features. A good approach and one I like to take is to use a custom build of a library such as Lo-Dash to provide a layer of abstraction so I don’t need to care whether it’ll be in an ES5 or ES3 environment. Then for anything that the abstraction library doesn’t provide I’ll use a customised polyfill library such as es5-shim to fall back on. I prefer to use Lo-Dash over Underscore too, as I think Lo-Dash is starting to leave Underscore behind in terms of performance and features. I also like to use the likes of yepnope.js to conditionally load my polyfills based on whether they’re actually needed in the users browser. As there’s no point in loading them if we have browser support now is there?

Polyfilling Object.create as discussed above, to ES5

You could use something like the following that doesn’t accommodate an object of property descriptors. Or just go with the following next two choices which is what I do:

  1. Use an abstraction like the lodash create method which takes an optional second argument object of properties and treats them the same way
  2. Use a polyfill like this one.
if (typeof Object.create !== 'function') {
   (function () {
      var F = function () {};
      Object.create = function (proto) {
         if (arguments.length > 1) {
            throw Error('Second argument not supported');
         }
         if (proto === null) {
            throw Error('Cannot set a null [[Prototype]]');
         }
         if (typeof proto !== 'object') {
            throw TypeError('Argument must be an object');
         }
         F.prototype = proto;
         return new F();
      };
   })();
};

Polyfilling Object.getPrototypeOf as discussed above, to ES5

  1. Use an abstraction like the lodash isPlainObject method (source here), or…
  2. Use a polyfill like this one. Just keep in mind the gotcha.

 

EcmaScript 6

I got a bit excited when I saw an earlier proposed prototype-for (also seen with the name prototype-of) operator: <| . Additional example here. This would have provided a terse syntax for providing an object literal with an object to use as its prototype. It looks like it must have lost traction though as it was removed in the June 15, 2012 Draft.

There are a few extra methods in ES6 that deal with prototypes, but on trawling the EcmaScript 6 draft spec, nothing at this stage that really stands out as revolutionising the way I write JavaScript or being a mental effort/time saver for me. Of course I may have missed something. I’d like to hear from anyone that has seen something interesting to the contrary?

Yes we’re getting class‘s in ES6, but they are just an abstraction giving us a terse and declarative mechanism for doing what we already do with functions that we use as constructors, prototypes and the objects (or instances if you will) that are returned from our functions that we’ve chosen to act as constructors.

Architectural Ideas that Prototypes Help With

This is a common example that I often use for domain objects that are fairly hot that use one set of accessor properties added to the business objects prototype, as you can see on line 13 of my Hobbit module (Hobbit.js) below.

First a quick look at the tests/spec to drive the development. This is being run using mocha with the help of a Makefile in the root directory of my module under test.

  • Makefile
# The relevant section.
unit-test:
	@NODE_ENV=test ./node_modules/.bin/mocha \
		test/unit/*test.js test/unit/**/*test.js
  • Hobbit-test.js
var requireFrom = require('requirefrom');
var assert = require('assert');
var should = require('should');
var shire = requireFrom('shire/');

// Hardcode $NODE_ENV=test for debugging.
process.env.NODE_ENV='test';

describe('shire/Hobbit business object unit suite', function () {
   it('Should be able to instantiate a shire/Hobbit business object.', function (done) {
      // Uncomment below lines if you want to debug.
      //this.timeout(444000);
      //setTimeout(done, 444000);

      var Hobbit = shire('Hobbit');
      var hobbit = new Hobbit();

      // Properties should be declared but not initialised.
      // No good checking for undefined alone, as that would be true whether it was declared or not.

      hobbit.should.have.property('id');
      (hobbit.id === undefined).should.be.true;
      hobbit.should.have.property('typeOfPerson');
      (hobbit.typeOfPerson === undefined).should.be.true;
      hobbit.should.have.property('greeting');
      (hobbit.greeting === undefined).should.be.true;
      hobbit.should.have.property('occupation');
      (hobbit.occupation === undefined).should.be.true;
      hobbit.should.have.property('emailFrom');
      (hobbit.emailFrom === undefined).should.be.true;
      hobbit.should.have.property('name');
      (hobbit.name === undefined).should.be.true;      

      done();
   });

   it('Should be able to set and get all properties of a shire/Hobbit business object.', function (done){
      // Uncomment below lines if you want to debug.
      this.timeout(444000);
      setTimeout(done, 444000);

      // Arrange
      var Hobbit = shire('Hobbit');
      var hobbit = new Hobbit();      

      // Act
      hobbit.id = '32f4d01e-74dc-45e8-b3a8-9aa24840bc6a';
      hobbit.typeOfPerson = 'short and hairy';
      hobbit.greeting = {
         intro: 'Hi, I\'m a ',
         outro: ' type of person.'};
      hobbit.occupation = 'mushroom hunter';
      hobbit.emailFrom = 'Bilbo.Baggins@theshire.arn';
      hobbit.name = 'Bilbo Baggins';

      // Assert
      hobbit.id.should.equal('32f4d01e-74dc-45e8-b3a8-9aa24840bc6a');
      hobbit.typeOfPerson.should.equal('short and hairy');
      hobbit.greeting.should.equal('Hi, I\'m a short and hairy type of person.');
      hobbit.occupation.should.equal('mushroom hunter');
      hobbit.emailFrom.should.equal('Bilbo.Baggins@theshire.arn');
      hobbit.name.should.eql('Bilbo Baggins');

      done();
   });
});
  • Now the business object itself Hobbit.js
    -
    Now what’s happening here is that on instance creation of new Hobbit, the empty members object you see created on line 9 is the only instance data. All of the Hobbit‘s accessor properties are defined once per export of the Hobbit module which is assigned the constructor function object. So what we store on each instance are the values assigned in the Hobbit-test.js from lines 47 through 54. That’s just the strings. So very little space is used for each instance of the Hobbit function returned by invoking the Hobbit constructor that the Hobbit module exports.
// Could achieve a cleaner syntax with Object.create, but constructor functions are a little faster.
// As this will be hot code, it makes sense to favour performance in this case.
// Of course profiling may say it's not worth it, in which case this could be rewritten.
var Hobbit = (function () {
   function Hobbit (/*Optionally Construct with DTO and serializer*/) {
      // Todo: Implement pattern for enforcing new.
      Object.defineProperty (this, 'members', {
         value: {}
      });
   }

   (function definePublicAccessors (){
      Object.defineProperties(Hobbit.prototype, {
         id: {
            get: function () {return this.members.id;},
            set: function (newValue) {
               // Todo: Validation goes here.
               this.members.id = newValue;
            },
            configurable: false, enumerable: true
         },
         typeOfPerson: {
            get: function () {return this.members.typeOfPerson;},
            set: function (newValue) {
               // Todo: Validation goes here.
               this.members.typeOfPerson = newValue;
            },
            configurable: false, enumerable: true
         },
         greeting: {
            get: function () {
               return this.members.greeting === undefined ?
                  undefined :
               this.members.greeting.intro +
                  this.typeOfPerson +
                  this.members.greeting.outro;
            },
            set: function (newValue) {
               // Todo: Validation goes here.
               this.members.greeting = newValue;
            },
            configurable: false, enumerable: true
         },
         occupation: {
            get: function () {return this.members.occupation;},
            set: function (newValue) {
               // Todo: Validation goes here.
               this.members.occupation = newValue;
            },
            configurable: false, enumerable: true
         },
         emailFrom: {
            get: function () {return this.members.emailFrom;},
            set: function (newValue) {
               // Todo: Validation goes here.
               this.members.emailFrom = newValue;
            },
            configurable: false, enumerable: true
         },
         name: {
            get: function () {return this.members.name;},
            set: function (newValue) {
               // Todo: Validation goes here.
               this.members.name = newValue;
            },
            configurable: false, enumerable: true
         }
      });

   })();
   return Hobbit;
})();

// JSON.parse provides a hydrated hobbit from the DTO.
//    So you would call this to populate this DO from a DTO
// JSON.stringify provides the DTO from a hydrated hobbit

module.exports = Hobbit;
  • Now running the test
lets test

 

Flyweights using Prototypes

A couple of interesting examples of the Flyweight pattern implemented in JavaScript are by the GoF and Addy Osmani.

The GoF’s implementation of the FlyweightFactory makes extensive use of closure to store its flyweights and uses aggregation in order to create it’s ConcreteFlyweight from the Flyweight. It doesn’t use prototypes.

Addy Osmani has a free book “JavaScript Design Patterns” containing an example of the Flyweight pattern, which IMO is considerably simpler and more elegant. In saying that, the GoF want you to buy their product, so maybe they do a better job when you give them money. In this example closure is also used extensively, but it’s a good example of how to leverage prototypes to share your less specific behaviour.

Mixins using Prototypes

Again if you check out the last example of Mixins in Addy Osmani’s book, there is quite an elegant example.

We can even do multiple inheritance using mixins, by adding which ever properties we want from what ever objects we want to the target objects prototype.

This is a similar concept to the post I wrote on Monkey Patching.

Mixins support the Open/Closed principle, where objects should be able to have their behaviour modified without their source code being altered.

Keep in mind though, that you shouldn’t just expect all consumers to know you’ve added additional behaviour. So think this through before using.

Factory functions using Prototypes

Again a decent example of the Factory function pattern is implemented in the “JavaScript Design Patterns” book here.

There are many other areas you can get benefits from using prototypes in your code.

Prototypal Inheritance: Not Right for Every Job

Prototypes give us the power to share only the secrets of others that need to be shared. We have fine grained control. If you’re thinking of using inheritance be it classical or prototypal, ask yourself “Is the class/object I’m wanting to provide a parent for truly a more specific version of the proposed parent?”. This is the idea behind the Liskov Substitution Principle (LSP) and Design by Contract (DbC) which I posted on here. Don’t just inherit because it’s convenient In my “javascript object creation patterns” post I also discussed inheritance.

The general consensus is that composition should be favoured over inheritance. If it makes sense to compose once you’ve considered all options, then go for it, if not, look at inheritance. Why should composition be favoured over inheritance? Because when you compose your object from another contract of an object, your sub object (the object doing the composing) doesn’t inherit anything or need to know anything about the composed objects secrets. The object being composed has complete freedom as to how it minds it’s own business, so long as it provides a consistent contract for consumers. This gives us the much loved polymorphism we crave without the crazy tight coupling of classical inheritance (inherit everything, even your fathers drinking problem :-s).

I’m pretty much in agreement with this when we’re talking about classical inheritance. When it comes to prototypal inheritance, we have a lot more flexibility and control around how we use the object that we’re deriving from and exactly what we inherit from it. So we don’t suffer the same “all or nothing” buy in and tight coupling as we do with classical inheritance. We get to pick just the good parts from an object that we decide we want as our parent. The other thing to consider is the memory savings of inheriting from a prototype rather than achieving your polymorphic behaviour by way of composition, which has us creating the composed object each time we want another specific object.

So in JavaScript, we really are spoilt for choice when it comes to how we go about getting our fix of polymorphism.

When surveys are carried out on..

Why Software Projects Fail

the following are the most common causes:

  • Ambiguous Requirements
  • Poor Stakeholder Involvement
  • Unrealistic Expectations
  • Poor Management
  • Poor Staffing (not enough of the right skills)
  • Poor Teamwork
  • Forever Changing Requirements
  • Poor Leadership
  • Cultural & Ethical Misalignment
  • Inadequate Communication

You’ll notice that technical reasons are very low on the list of why projects fail. You can see the same point mentioned by many of our software greats, but when a project does fail due to technical reasons, it’s usually because the complexity got out of hand. So as developers when focusing on the art of creating good code, our primary concern should be to reduce complexity, thus enhance the ability to maintain the code going forward.

I think one of Edsger W. Dijkstra’s phrases sums it up nicely. “Simplicity is prerequisite for reliability”.

Stratification is a design principle that focuses on keeping the different layers in code autonomous, I.E. you should be able to work in one layer without having to go up or down adjacent layers in order to fully understand the current layer you’re working in. Its internals should be able to move independently of the adjacent layers without effecting them or being concerned that a change in it’s own implementation will affect other layers. Modules are an excellent design pattern used heavily to build medium to large JavaScript applications.

With composition, if your composing with contracts, this is exactly what you get.

References and interesting reads

 

Exploring JavaScript Closures

May 31, 2014

Just before we get started, we’ll be using the terms lexical scope and dynamic scope a bit. In computer science the term lexical scope is synonymous with static scope.

  • lexical or static scope is where name resolution of “part of a program” depends on the location in the source code
  • dynamic scope is whether name resolution depends on the program state (dependent on execution context or calling context) when the name is encountered.

What are Closures?

Now establishing the formal definition has been quite an interesting journey, with quite a few sources not quite getting it right. Although the ES3 spec talks about closure, there is no formal definition of what it actually is. The ES5 spec on the other hand does discuss what closure is in two distinct locations.

  1. “11.1.5 Object Initialiser” section under the section that talks about accessor properties This is the relevant text: (In relation to getters): “Let closure be the result of creating a new Function object as specified in 13.2 with an empty parameter list (that’s getter specific) and body specified by FunctionBody. Pass in the LexicalEnvironment of the running execution context as the Scope.
  2. “13 Function Definition” section This is the relevant text: “Let closure be the result of creating a new Function object as specified in 13.2 with parameters specified by FormalParameterList (which are optional) and body specified by FunctionBody. Pass in funcEnv as the Scope.

Now what are the differences here that stand out?

  1. We see that 1 specifies a function object with no parameters, and 2 specifies some parameters (optional). So from this we can establish that it’s irrelevant whether arguments are passed or not to create closure.
  2. 1 also mentions passing in the LexicalEnvironment, where as 2 passes in funcEnv. funcEnv is the result of “calling NewDeclarativeEnvironment passing the running execution context‘s LexicalEnvironment as the argument“. So basically there is no difference.

Now 13.2 just specifies how functions are created. Given an optional parameter list, a body, a LexicalEnvironment specified by Scope, and a Boolean flag (for strict mode (ignore this for the purposes of establishing a formal definition)). Now the Scope mentioned above is the lexical environment of the running execution context (discussed here in depth) at creation time. The Scope is actually [[Scope]] (an internal property).

The ES6 spec draft runs along the same vein.

Lets get abstract

Every problem in computer science is just a more specific problem of a problem we’re familiar with in the natural world. So often it helps to find the abstract problem that we are already familiar with in order to help us understand the more specific problem we are dealing with. Patterns are an example of this. Before I was programming as a profession I was a carpenter. I find just about every problem I deal with in programming I’ve already dealt with in physical carpentry and at a higher level still with physical architecture.

In search of the true formal definition I also looked outside of JavaScript at the language agnostic term, which should just be an abstraction of the JavaScript closure anyway. Yip… Wikipedias definition “In programming languages, a closure (also lexical closure or function closure) is a function or reference to a function together with a referencing environment—a table storing a reference to each of the non-local variables (also called free variables or upvalues) of that function. A closure—unlike a plain function pointer—allows a function to access those non-local variables even when invoked outside its immediate lexical scope.

My abstract formal definition

A closure is a function containing a reference to the lexical (static) environment via the function objects internal [[Scope]] property (ES5 spec 13.2.9) that it is defined within at creation time, not call time (ES5 spec 13.2.1). The closure is closed over it’s parent lexical environment and all of it’s properties. You can access these properties as variables, but not as properties, because you don’t have access to the internal [[Scope]] property directly in order to reference it’s properties. So this example fails. More correctly (ES5 spec 8.6.2) “Of the standard built-in ECMAScript objects, only Function objects implement [[Scope]].

var outerObjectLiteral = {

   x: 10,

   foo: function () {
      console.log(x); // ReferenceError: x is not defined obviously
   },
   invokeMe: function (funArg) {
      var x = 20;
      funArg();
   }
};

outerObjectLiteral.invokeMe(outerObjectLiteral.foo);

See here for an explanation on the differences between properties and variables. That’s basically it. Of course there are many ways we can use a closure and that’s often where confusion creeps in about what a closure actually is and is not. Feel free to bring your perspective on this in the comments section below.

When is a closure born?

So lets get this closure closing over something. JavaScript addresses the funarg problem with closure.

var x = 10;

var outerObjectLiteral = {   

   foo: function () {
      // Because our internal [[Scope]] property now has a property (more specifically a free variable) x, we can access it directly.
      console.log(x); // Writes 10 to the console.
   },
   invokeMe: function (funArg) {
      var x = 20;
      funArg();
   }
};

outerObjectLiteral.invokeMe(outerObjectLiteral.foo);

The closure is created on line 13. Now at line 9 we have access to the closed over lexical environment. When we print x on line 7, we get 10 which is the value of x on [[Scope]] that our closure was statically bound to at function object creation time (not the dynamically scoped x = 20). Now of course you can change the value of the free variable x and it’ll be reflected where ever you use the closed over variable because the closure was bound to the free variable x, not the value of the free variable x.

This is what you’ll see in Chrome Dev Tools when execution is on line 10. Bear in mind though that both foo and invokeMe closures were created at line 13.

Closure

Now I’m going to attempt to explain what the structure looks like in a simplified form with a simple hash. I don’t know how it’s actually implemented in the varius EcmaScript implementations, but I do know what the specification (single source of truth) tells us, it should look something like the following:

////////////////
// pseudocode //
////////////////
foo = closure {
   FormalParameterList: {}, // Optional
   FunctionBody: <...>,
   Environment: { // ES5 10.5 VariableEnvironment's Environment record. This is actually the internal [[Scope]] property (set to the outer lexical environment).
      x: 10
   }
}

The closure is born when the function is created (“the result of creating a new Function object” as stated above). Not when it’s returned by the outer function (I.E. upwards funarg problem) and not when it’s invoked as Angus Croll mentioned here under the “The [[Scope]] property” section.

Angus quotes the ES5 spec 10.4.3.5-7. On studying this section I’m pretty sure it is meant for the context of actually creating the function object rather than invoking an existing function object. The clauses I’ve detailed above (11.1.5 Object Initialiser and 13 Function Definition), confirm this.

The ES6 spec draft “14.1.22 Runtime Semantics: Evaluation” also confirms this theory. Although it’s titled Runtime Semantics, it has several points that confirm my theory… The so called runtime semantics are the runtime semantics of function object creation rather than function object invocation. As some of the steps specified are FunctionCreate, MakeMethod and MakeConstructor (not FunctionInvoke, InvokeMethod or InvokeConstructor). The ES6 spec draft “14.2.17 Runtime Semantics: Evaluation” and also 14.3.8 are similar.

Why do we care about Closure?

Without closures, we wouldn’t have the concept of modules which I’ve discussed in depth here.

Modules are used very heavily in JavaScript both client and server side (think NPM), and for good reason. Until ES6 there is no baked in module system. In ES6 modules become part of the language. The entire Node.js ecosystem exists to install modules via the CommonJS initiative. Modules on the client side most often use the Asynchronous Module Definition (AMD) implementation RequireJS to load modules, but can also use the likes of CommonJS via Browserify, which allows us to load node.js packages in the browser.

As of writing this, the TC39 committee have looked at both the AMD and CommonJS approaches and come up with something completely different for the ES6 module draft spec. Modules provide another mechanism for not allowing secrets to leak into the global object.

Modules are not new. David Parnas wrote a paper titled “On the Criteria To Be Used in Decomposing Systems into Modules” in 1972. This explores the idea of secrets. Design and implementation decisions that should be hidden from the rest of the programme.

Here is an example of the Module pattern that includes both private and public methods. my.moduleMethod has access to private variables outside of it’s VariableEnvironment (the current scope) via the Environment record which references the outer LexicalEnvironment via it’s internal [[Scope]] property.

Information hiding: state and implementation. In JavaScript we don’t have access modifiers, but we don’t need them either. We can hide our secrets with various patterns. Closure is a key concept for many of these patterns. Closure is a key building block for helping us to programme against contract rather than implementation, helping us to form consistent abstractions, giving us the ability to engage with a concept while safely ignoring some of its details. Thus hiding unnecessary complexity from consumers.

I think Steve McConnell explains this very well in his classic “Code Complete” book. Steve uses the house abstraction as his metaphor. “People use abstraction continuously. If you had to deal with individual wood fibers, varnish molecules, and steel molecules every time you used your front door, you’d hardly make it in or out of your house each day. Abstraction is a big part of how we deal with complexity in the real world. Software developers sometimes build systems at the wood-fiber, varnish-molecule, and steel-molecule level. This makes the systems overly complex and intellectually hard to manage. When programmers fail to provide larger programming abstractions, the system itself sometimes fails to make it through the front door. Good programmers create abstractions at the routine-interface level, class-interface level, and package-interface level-in other words, the doorknob level, door level, and house level-and that supports faster and safer programming.

Encapsulation: you can not look at the details (the internal implementation, the secrets).

Partial function application and Currying: I have a set of posts on this topic. Closure is an integral building block of these constructs. Part 1, Part 2 and Part 3.

Functional JavaScript relies heavily on closure.

Are there any Costs or Gotchas of using Closures?

Of course. You didn’t think you’d get all this expressive power without having to think about how you’re going to use it did you? As we’ve discussed, closures were created to address the funarg problem. In doing that, the closure references the lexical (static) scope of the outer scope. So even once the free variables are out of scope, closure will still reference them if they were saved at function creation time. They can not be garbage collected until the function that references (is closed over) the outer scope has fallen out of scope. I.E. the reference count is 0.

var x = 10;
var noOneLikesMe = 20;
var globalyAccessiblePrivilegedFunction;

function globalyScopedFunction(z) {

  var noOneLikesMeInner = 40;

  function privilegedFunction() {
    return x + z;
  }

  return privilegedFunction;

}

// This is where privilegedFunction is created.
globalyAccessiblePrivilegedFunction = globalyScopedFunction(30);

// This is where privilegedFunction is applied.
globalyAccessiblePrivilegedFunction();

Now only the free variables that are needed are saved at function creation time. We see that when execution arrives at line 7, the currently scoped closure has the x free variable saved to it, but not z, noOneLikesMe, or noOneLikesMeInner.

noOneLikesMe

When we enter innerFunction on line 10, we see the hidden [[Scope]] property has both the outer scope and the global scope saved to it.

TwoClosures

Say for example execution has passed the above code snippet. If the closed over variables can still be referenced by calling globalyAccessiblePrivilegedFunction again, then they can not be garbage collected. This is a frequently abused problem with the upwards funarg problem. If you’ve got hot code that is creating many functions, make sure the functions that are closed over free variables are dropped out of scope as soon as you no longer have a need for them. This way garbage collection can deallocate the memory used by the free variables.

Looking at how the specification would look simplified, we can see that each Environment record inherits what it knows it’s going to need from the Environment record of its lexical parent. This chaining inheritance goes all the way up the lexical hierarchy to the global function object as seen below. In this case the family tree is quite short. Remember this structure is formed at function creation time, not invocation time. the free variables (not their values) are statically baked.

////////////////
// pseudocode //
////////////////
globalyScopedFunction = closure {
   FormalParameterList: { // Optional
      z: 30 // Values updated at invocation time.
   },
   FunctionBody: {
      var noOneLikesMeInner = 40;

      function privilegedFunction() {
         return x + z;
      }

      return privilegedFunction;
   },
   Environment: { // ES5 10.5 VariableEnvironment's Environment record. This is actually the internal [[Scope]] property (set to the outer lexical environment).
      x: 10 // Free variable saved because we know it's going to be used in privilegedFunction.
   },
   privilegedFunction: = closure {
      FormalParameterList: {}, // Optional
      FunctionBody: {
         return x + z;
      },
      Environment: { // ES5 10.5 VariableEnvironment's Environment record. This is actually the internal [[Scope]] property (set to the outer lexical environment).
         x: 10 // Free variable inherited from the outer Environment.
         z: 30 // Formal parameter saved from outer lexical environment.
      }
   }
}

Scope

I discuss closure here very briefly and how it can be used to create block scoped variables prior to block scoping with the let keyword in ES6, supposed to be officially approved by December 2014. I discuss scoping here in a little more depth.

Closure misunderstandings

Closures are created when a function is returned

A closure is formed when one of those inner functions is made accessible outside of the function in which it was contained” found here is simply incorrect. There are also a lot of other misconceptions found at that link. I’d advise to read with a bag of salt.

Now we’ve already addressed this one above, but here is an example that confirms that the closure is in fact created at function creation time, not when the inner function is returned. Yes, it does what it looks like it does. Fiddle with it?

(function () {

   var lexicallyScopedFunction = function () {
      console.log('We\'re in the lexicalyScopedFunction');
   };

   (function innerClosure() {
      lexicallyScopedFunction();
   }());

}());

On line 8, we get to see the closure that was created from the execution of line 11.

lexicallyScopedFunction

Closures can create memory leaks

Yes they can, but not if you let the closure go out of scope. Discussed above.

Values of free variables are baked into the Closure

Also untrue. Now I’ve put in-line comments to explain what’s happening here. Fiddle with the below example?

var numberOfFunctionsRequired = 3;
var getLoopPrinter = function () {
   var loopCountingFunctions = new Array(numberOfFunctionsRequired);
   for (var i = 0; i < numberOfFunctionsRequired; i++) {
      loopCountingFunctions[i] = (function printLoopCount() {
         // What you see here is that each time this code is run, it prints the last value of the loop counter i.
         // Clearly showing that for each new printLoopCount function created and saved to the loopCountingFunctions array,
         // the variable i is saved to the Environment record, not the value of the variable i.
         console.log(i);
      });
   }
   return loopCountingFunctions;
};

var runLoopPrinter = getLoopPrinter();
runLoopPrinter[0](); // 3
runLoopPrinter[1](); // 3
runLoopPrinter[2](); // 3

An aside… getLoopPrinter is a global function. Once execution is on line 3 you get to see that global functions also have closure… supporting my comments above

global functions have closure too

Now in the above example, this is probably not what you want to happen, so how do we give each printLoopCount function it’s on value? Well by creating a parameter for each iteration of the loop, each with the new value. Fiddle with the below example?

var numberOfFunctionsRequired = 3;
var getLoopPrinter = function () {
   var loopCountingFunctions = new Array(numberOfFunctionsRequired);
   for (var i = 0; i < numberOfFunctionsRequired; i++) {
      (function (i) {
         // Now what happens here is each time the above loop runs this code,
         // inside this scope (the scope of this comment) i is a new formal parameter which of course
         // gets statically saved to each printLoopCount functions Environment record (or more simply each closure of printLoopCount).
         loopCountingFunctions[i] = (function printLoopCount() {
            console.log(i);
         });
      })(i)
   }
   return loopCountingFunctions;
};

var runLoopPrinter = getLoopPrinter();
runLoopPrinter[0](); // 0
runLoopPrinter[1](); // 1
runLoopPrinter[2](); // 2

As always, let me know your thoughts on this post, any thing you think I may have the wrong handle on, or anything that otherwise stood out.

References and interesting reads

Evaluation of AngularJS, EmberJS, BackboneJS + MarionetteJS

December 28, 2013

This post will continue to be modified for at least a month from the publish date. I just didn’t want to wait another month before publishing, so people can start to get some use out of it early. If you have resources, comments, anything you think that could be useful to others, please add a comment and if it makes sense, I may add it to the post. This will also be used as a resource for the attendees to the CHC.JS MV* Battle Royale meet-up.

Recently I’ve undertaken the task of reviewing some JavaScript MV* frameworks to help organise/structure the client side code within an application I’m currently working on. This is about the third time I’ve done this. Each time has been for a different type of application with completely different requirements, frameworks and libraries to consider.
Unlike Angular and Ember, Backbone is a small library. Marionette adds quite a lot of extra functionality and provides some nice abstractions on top . All mentioned frameworks/libraries are free and open source.

I found a useful tool for helping with the selection process about a year ago. It’s called TodoMVC and it contains a generous collection of applications all satisfying the requirements of a single specification (a small web app that allows the person using it to add todo notes etc.). So basically they all do the same thing, but use a different JavaScript framework or library to do it. It’s still being maintained. Addy Osmani’s blog post on the project is here.

The idea is that you can work through a decent size selection of applications that all do the same thing.
This assists the R&D developer or architect to make informed decisions on which JavaScript framework or library will suite their purposes, if any.
There are also a couple of Todo apps (vanillajs and jquery) that don’t use a framework at all.
There’s a template to use as a starting point, so you can create your own.

Just bear in mind though, that the TodoMVC app doesn’t really show case what Ember and Angular has to offer.

On Addy’s post There are a collection of good points on how to create your selection criterion under the heading “Our Suggested Criteria For Selecting A Framework”.

I’ve heard a few times that “all you really need to do in order to make an informed decision on which framework or library to go with is just write a small app for each of the frameworks, do a bit of reading and maybe watch a few screen casts. Shouldn’t take more than a day”. I disagree with this. I don’t think there is any way you can learn all or most of the pros and cons of each framework in a day or even two. Depending on how much time you have, my recommended approach would be to go through the following activities in the following order (give or take). Spending as little or as much time as you have, ideally in a few iterations, for each of the offerings you’re investigating.

  1. Listen to a pod-cast (say, on your way to/from a clients or even in your sleep. Good time savers.)
  2. Read some of the documentation
  3. Watch a screen-cast on each one
  4. Play with some examples
  5. Evaluate on features you (definitely or may) want verses features available. Features need to be learned. If you don’t need them, you will probably be better going with the offering that doesn’t have the features you don’t need, but has the architecture to add them (thinking Backbone) if/when you do need them.
  6. Are the features implemented in an architecture that you believe is good (I.E. are the layers muddied)?
  7. Read some blog posts, tutorials.
  8. Read some opinions and evaluate for yourself.
  9. Start testing it’s limits
  10. Decide whether you like its opinions imposed
  11. Does it impose enough or to many opinions for you and your team

As the JavaScript MV* landscape is constantly and very quickly changing, the outcome of your evaluation will have a short use by date.

This is my attempt to distil the attributes of the discussed offerings. I’ve attempted to come at this with an open mind. Hopefully this will help save some work for those that come after me. lists are sorted in the order of most useful to me. I make no apologies for the abundance of links, as I’ve also used this for a resource collection point and hope that this post will fall into the category of a “one stop shop” for what I consider to “currently” be the top three contenders in the client side MV* line-up. In saying that though, there are other strong contenders like Meteor not discussed here, as it’s more than just a client side MV* framework. Without further ado, here they are…

Angular.js

AngularJS

Intro

Opinionated framework that has Models, Views and Controllers, but does not conform to the MVC pattern.

Core Team

Igor Minár, Miško Hevery, Vojta Jína.
All work at google.

Backed by the commercial giant Google (you decide whether that’s a good thing).

Community

Conferences

  1. ng-conf

Statistics

  • Version: 1.2.5
  • Payload Size: Depends on handlebars development version 85kb
    1. development version 716.7kb
    2. minified 99.8kb
    3. minified and compressed < 36kb
  • Age: Initial Github commits: January 2010

Performance

See Backbone Performance below.

Documentation

Pod-casts

  1. Angular.js

Screen-casts

  1. AngularJS on YouTube
  2. EggHead.io Lessons

Blog Posts, Tutorials, etc.

  1. Learning AngularJS in one day
  2. Angular docs Tutorial

Features

  • Directives: used via Non HTML compliant tags, attributes, comment and class names. Although there are options to make it compliant:  via the class (not recommended) and data attributes.
  • scope. The first half of this video shows how the scope may be confusing to those new to Angular. If I can not tell how code will work without running it, it violates the Principle of Least Astonishment (PoLA). It seems quite clunky to me.

Positive

  1. Good for long running and complex applications with deep nested view hierarchies
  2. Two-way data binding
  3. All tests run against IE8 (good for those that are locked into legacy MS)
  4. Test driven (and more vocal about it than Ember)
  5. Payload is about 1/3 smaller than Ember

Negative

  1. Steep learning curve compared to Backbone, but not as steep as Ember.
  2. Dirty checking to keep views and models in sync is costly. Ember keeps sync in a more elegant way. Possible perceived downside to this is Ember models have to inherit from DS.Model (next point addresses this as a positive though). Also discussed here under the “Performance issues” heading.
  3. Models are Plain Old JavaScript Objects (POJO’s). Doesn’t have to be anything special. Now there’s an argument here that attempts to explain this as being a selling point of Angular, but in reality what happens is a violation of the Uniform Access Principle, thus creating tight coupling. How’s that? Well now the view needs to know too much about the model’s members. Discussed in more detail here. For example if one of the models properties is a function, the view has to know this. So you see this sort of thing in the view {{area()}} (so we’re pulling our JavaScript into our view.) where as with Ember because it’s models are well defined and you can use computed properties on them, all the view needs to specify is an identifier, then you’ll see this sort of thing in the view {{area}}. The Ember model then creates a computed property with the same name. The opposing view is that in ES5 you can just hide your functions etc. behind property getters and setters. Most developers take the path of least resistance, so I think most will be doing it the wrong way.

Interesting Plug-ins

  1. ?

Useful Tools

  • “AngularJS Batarang” for Chrome browser (it’s an extension)

Ember.js

EmberJS

Intro

Opinionated framework that has Models, Views and Controllers, but does not conform to the MVC pattern.

Core Team

Yehuda Katz, formerly of Rails and SproutCore projects.
Tom Dale, Peter Wagenet, Trek Glowacki, Erik Bryn, Kris Selden, Stefan Penner, Leah Silber, Alex Matchneer.

Backed by the JavaScript community.

Community

Conferences

Statistics

  • Version: 1.2.0
  • Payload Size:
    1. development version 1.1MB
    2. production version 1.0MB
    3. minified and GZipped 67kb
  • Age: Initial Github commits: April 2011

Performance

See Backbone Performance below.

Documentation

Pod-casts

  1. JavaScript Jabber Ember Tools
  2. JavaScript Jabber Ember.js (also covers some backbone)
  3. JavaScript Jabber Ember.js & Discourse
  4. EmberWatch

Screen-casts

  1. Building an Ember.js Application
  2. Ember101
  3. EmberWatch
  4. tutsplus
  5. EmberWatch 

Blog Posts, Tutorials, etc.

Features

  • the ember-application class gets added to the root element (body) in the ember JavaScript file. I was wondering how this class was magically added to the markup. Couldn’t find any documentation on it, so had to look through the JavaScript.

Positive

  1. Good for long running and complex applications with deep nested view hierarchies
  2. Aggregates model data changes and update the DOM late in the RunLoop.
  3. Well defined models and computed properties (See Angular negative point around this).
  4. Test driven

Negative

  1. Steepest learning curve out of the three. Why? Because there’s more in it. If you need it, great! Maybe you don’t. If not, is the extra learning worth using it? Part of the “more in it” may also be around the elegant way things have been designed, I.E. more constraints to push the users down the right path, thus higher chance of less friction and pain in the future of your application, that is of course if your application does things they way Ember says they must be done. I’m seeing some of these things in the likes of the well defined models and computed properties.
  2. Payload is the largest out of all three.

Interesting Plug-ins

  1. ?

Useful Tools

  • “Ember Inspector” for Chrome browser (it’s an extension)
  • ember-tools. Listen to and/or read the pod-cast linked to above. Provides file organisation, scaffolding, template pre-compilation, generators, CommonJS (that’s node.js style) modules. and other goodies. Useful for setting up your project to conform to the Ember conventions, so you don’t end up fighting them.

Angular versus Ember views

Pod-casts

  1. Angular vs Ember Cage Match NDC

Screen-casts

  1. Angular vs Ember Cage Match NDC

Blog Posts, Tutorials, etc.

  1. Evil Trout Ember versus Angular (possible bias toward Ember)
  2. Why AngularJS beat EmberJS

My Thoughts

Both Frameworks Appear to be Targeting a Similar Problem Space

Don’t believe everything you read. Test it before you buy it. I’ve come across quite a few articles that are just incorrect. Even by reputable people. Sometimes because the frameworks have changed how they do things and/or their documentation has changed. So don’t just take it all at face value. The concept of MVC has changed over the past decade. Although concepts have changed, a pattern doesn’t change, that’s why it’s a pattern. Something everybody familiar with a pattern understands. If an implementation starts to change, then it no longer conforms to the pattern and should not be named after the pattern, as this just brings confusion. Microsoft’s ASP.NET MVC framework is a perfect example of this. It does not follow the MVC pattern (documented in 1979) and so should never have been named MVC. Ask me in the comments to explain if your not aware of how this is. In the MVC pattern Models are not injected into views by Controllers. With the MVC pattern, Views listen to events from a Model (The View is actually oblivious to the Model) which the Controller has hooked up, since the Controller knows about both the View and the Model. This may not be your understanding of MVC? More than likely this is due to certain frameworks being labelled as MVC when they are not, thus bringing the confusion. The following image provided by Gang-Of-Four depicts the MVC pattern.

MVC

Angular Doesn’t Pretend JavaScript has Real Classes

Personally I find both frameworks have opinions that make me nauseous.
Like Angular’s scope and Embers class-hierarchy abstraction. Yes Harmony will have pseudo classes for the classical programmers that struggle with JavaScripts declarative prototypal inheritance. (disclaimer: my roots are in classical OO languages) The way I feel about it: Say a whole lot of JavaScript programmers start using a classical OO language and decide they don’t like the way it does classical inheritance, so the classical object oriented language authorities decide to add syntactic sugar on top of the language to make it’s classical inheritance “look” more like prototypal inheritance for those that struggle with the classical paradigm. Now seriously, why would you muddy the language to cater for those that are not prepared to spend the time learning how it works?

Another and probably the most obvious reason why JavaScript didn’t have classes, is so that object hierarchies could be built up via composition (only inheriting what is actually needed) rather than having to inherit every member needlessly from a base class (essentially knowing far more than is actually needed).  Once you have to re-factor your way out of a code base that has abused inheritance thus creating very tightly coupled code by violating one of the object-oriented design principles (information hiding), the perils of over using inheritance will become very clear.

I’m open to exploring what the other client side JavaScript frameworks and libraries have to offer and I’d love to hear from everyone that’s had experience with them.

Angular and Ember do a Lot For You

With all the bells and whistles, both frameworks impose strong opinions that you must follow in order to make the magic (in a lot of the cases convention) work. Once you’ve learnt Angular and/or Ember, productivity is maximised. But… you must be building your application the way the framework creators want you to. At this stage, I’m not supper comfortable with that. This is where Backbone and friends comes in to its own.

Backbone.jsMarionette.js

BackboneJS + MarionetteJS

Intro

Backbone is an unopinionated library that has Models, Views but no Controllers out of the box. That’s right, a library rather than a framework because your code needs to know about it, rather than it knowing about and executing your code. It does not follow the MVC, MVP or MVVM patterns. It’s views and routers act similarly to a controller. Marionette brings the controller to Backbone (if you want or need it), thus you can keep your router doing what it should be doing (just routing, with no controller logic).

What I find strange is that a Backbone view contains a model. I’m not sure I’d even call this a MV* library, as it may introduce confusion.

Backbone’s sweet spot is providing the user with brief and casual interaction. Doesn’t provide help or guidance with deallocating memory and detaching events. Assumptions are that the user isn’t going to be using this application all day without closing the browser window. Although in saying that, there are many applications that use Backbone for this type of thing, but they must provide explicit code to release event handlers. Marionette provided some help here for older versions of Backbone. and Backbone has improved things with newer versions. You will still need to keep in mind that event handlers need to be released though (Backbone’s view.remove takes care of this now). Marionette provides abstractions to deal with these like the close method which provides a place to add clean-up code and then calls Backbone’s remove. Failing to remove event handlers are the largest cause of memory leaks in Backbone.

Core Team

Backbone: Jeremy Ashkenas

Marionette: Derick Bailey

Community

IRC: #marionette on FreeNode. Little activity.

Conferences

  1. BackboneConf

Statistics

  • Version: 1.1.0
  • Payload size: Depends on Underscore development version 43kb or minified and gzipped 4.9kb
    1. Backbone development version 59kb
    2. Backbone minified and gzipped 6.4kb
  • Age: Backbone: Initial Github commits: September 2010

Performance

The second half of this video shows the difference between Backbone and Ember performance. What I’ve seen to date, is that in terms of performance, Backbone leads, second is Ember, third is Angular. You need to decide how much performance matters to your situation and whether or not it’s “good enough” for the framework/library you choose.

Documentation

Pod-casts

  1. Marionette.js
  2. JavaScript Jabber Ember.js (also covers some backbone)
  3. Backbone.js

Screen-casts

  1. How to build modular Backbone applications using MarionetteJS
  2. Tuts+ Intro to Marionette
  3. Plugging in MarionetteJS. This resource is about adding Marionette to a MongoDB document explorer. Also features source code.
  4. Github
  5. BackboneConf 2013 Talks

Blog Posts, Tutorials, etc.

  1. Github
  2. backbone and ember
  3. Marionette Wiki

Books

  1. Backbone Fundamentals

Features

  • ?

Positive

  1. Free to use any templating engine. You can use underscore as it’s the only dependency of backbone, or any other of your choosing.
  2. A lot of excellent documentation
  3. Very flexible in how you may want to use it
  4. Minimalist library
  5. Easy to learn (not a lot of it).
  6. Payload including dependencies is the smallest out of all three. About 9 times smaller than Ember.

Negative

  1. No two way data-binding. Although if you want/need it, you could use the likes of the data binding offerings below in the Interesting Plug-ins section.
  2. No provision for handling nested views. This is where the likes of Marionette’s Backbone.BabySitter comes in
  3. More work required to build large scale applications than the likes of Angular or Ember (just a library after all).
  4. If your large complex application is written in Backbone, chances are you have added a lot of boiler plate code. Any new developers coming onto the project will have to get up to speed on this code. If your large complex application uses Angular or Ember and the new developers coming onto the project have worked with these frameworks, they more than likely won’t have to learn the boiler plate code that they would have to with the likes of Backbone, because it’s part of the framework.

Interesting Plug-ins

  1. There is a similar offering: backbone.layoutmanager which I haven’t really looked into, but according to Derick Bailey (Marionette BDFL) is more of a framework where as Marionette is a library.
  2. Two way data binding with Rivets.jsKnockback.jsbackbone.stickit
    NYTimes backbone.stickit “is a Backbone data binding plug-in that binds Model attributes to View elements with a myriad of options for fine-tuning a rich application experience”. What looks to be nice about this is that unlike most model binding plug-ins I’ve seen, it doesn’t require you to add any extra tags like Angular to your view. In fact your views are not contaminated at all.
  3. Backbone.routefilter plug-in allows you to add behaviour that will be executed immediately before and/or after a route (Backbone.Router or Marionette.AppRouter) executes.

Useful Tools

  • “Backbone Debugger” for Chrome browser (it’s an extension)
  • Frameworks that leverage backbone and provide more functionality
    1. chaplinJS
    2. thoraxJS (adds handlebars integration plus other functionality)

Now a few more concepts that I think are important to know about if your serious about using a client side JavaScript MV* framework/library and in regards to module loading, this applies to the server side also.

Templating

Blog Posts etc.

  1. net tuts+ Best Practices When Working With JavaScript Templates
  2. net tuts+ An Introduction to Handlebars

Some Offerings

I covered some of the template engines here under “Templating Engines evaluated”, or just use the likes of the Template Engine Chooser

Coupling Domain with Framework

As Boris Smus has said and I think it’s right on the money (although I disagree with his comments around JavaScript class as per my comments above):
Once you bite the bullet and decide to invest in a framework, you often have no easy way to move your code out of it.
If you pick Backbone, but decide mid-cycle that it’s not for you, you are in for a world of hurt:
If you have core functionality that you want to release, release it in pure JavaScript, not as a jQuery plug-in, or some MV* module.

Because there are so many JavaScript frameworks coming and going, and we don’t want to invest to heavily into any one of them,
we really need to keep our investment separate from the library/framework code.

To avoid library/framework and class-system lock-in, a good approach in regards to JavaScript MV* libraries/frameworks,
Is to keep the core functionality separate from the user interface code, thus giving us two separate layers.
This gives us flexibility to swap user interfaces as they come and go, yet still keep the majority of our code in an API layer.
The API layer being a logical single layer, but can be modularised, and loaded as needed, AMD style.
With this separation, we can implement the two layers in the following manner.

1) Build the base layer using pure JavaScript prototypal inheritance.
This is the part you write with the intention of keeping and possibly using parts for other projects also.
This base layer will implement an API that you will want to spend a bit of time getting right.
This is the code that will make the most use of unit tests.
To get the separation clear in your head, think of the user interface code as a client that uses this API as if it was service API sitting on the server.
This way you can avoid creating leaky abstractions.

2) Use an MV* library/framework to implement the user interface, and call into the base layer directly.
This lets you move quickly and focus entirely on writing the user interface.
This architecture should facilitate building your user interface on a solid foundation and avoid investing heavily into an offering that you may want to swap out further down the track.

Modules

In most browsers, just including a script tag will cause the rest of the page to stop rendering until the script has loaded then executed.
Which is why if loading scripts synchronously, they should be concatenated, minified, compressed and included at the bottom.
Loading scripts asynchronously don’t block, which is why you can load multiple scripts in parallel where ever you want (any more than 2-3 concurrently and performance will degrade). Make sure to concatenate your scripts though.

What we see as our projects get larger, is that scripts start to have many dependencies in a way that may overlap and nest.

The simplest way to load asynchronously is to create a script tag and inject it into an existing DOM element on your page.
Because the DOM element already exists, the rendering is not blocked.
See the first code example here

// Create a new script element.
var script = document.createElement('script');

// Find an existing script element on the page (usually the one this code is in).
var firstScript = document.getElementsByTagName('script')[0];

// Set the location of the script.
script.src = "http://example.com/myscript.min.js";

// Inject with insertBefore to avoid appendChild errors.
firstScript.parentNode.insertBefore( script, firstScript );

If you want or need to get serious about script loading (which you’re probably going to have to do at some stage), use a best-of-breed script loader. This will also push you down the path of defining modular JavaScirpt (AKA modules).

Next we look at employing script loaders to load our modules…

Formats available for Writing and Using Modular JavaScript

Asynchronous Module Definition (AMD)

For writing modular JavaScript in the browser. To save re-writing what’s already been done… http://addyosmani.com/writing-modular-js/ see “AMD” section, explains it well. What does AMD actually give us? http://requirejs.org/docs/whyamd.html#amd Separation of Concerns, essentially placing value on interface rather than implementation. Mapping of module IDs to different paths. Lots more. Allows asynchronous loading of modules and their dependencies, which is something we need on the client side, but is not generally a requirement for the server side. For getting started, see “Getting Started With Modules” under the AMD section here. Also check out the AMD specification and of course the most common AMD implementation: RequireJS. Then at some stage you’re probably going to want to concatenate and minify your modules and that’s where the likes of r.js comes in. r.js also has a node.js adapter which allows you to use node’s implementation of  require.

Tom Dale (core team member on Ember) also has some interesting ideas around why he thinks AMD is not the answer.

CommonJS API (Optimised for the server)

Although we have the likes of browserify a CommonJS module implementation that can run in the browser or browser-build… makes CommonJS modules available in the browser and is very fast. Ryan Florence discusses module loaders in the pod-cast listed above “JavaScript Jabber Ember Tools” where he decided to move to CommonJS rather than RequireJS for his Ember Tools mostly due to speed. So it’s horses for courses. Decide what your requirements are, then decide which module loader satisfies the most of them. Also see “writing modular js” under the “CommonJS” section.
Providing a rich standard library. The intention is that an application developer will be able to write an application using the CommonJS APIs and then run that application across different JavaScript interpreters and host environments. With CommonJS-compliant systems, you can use JavaScript to write:

  • Server-side JavaScript applications
  • Command line tools
  • Desktop GUI-based applications
  • Hybrid applications (Titanium, Adobe AIR)

Why it doesn’t excel in the browser “out of the box”: http://requirejs.org/docs/whyamd.html#commonjs
ES Harmony (Modules implemented in the language. were not quite there yet, but the current offerings look to be a pretty good step in the right direction).

http://addyosmani.com/writing-modular-js/ (specifically “ES Harmony” section) discusses where TC39 are going in regards to implementing modules in ES.next.

So AMD and CommonJS can be used on server side or client side. In some cases one will work better for you than the other. You’ll need to do your homework as to what to use in which scenarios. Both have advantages and disadvantages that may work for or against you.

I’m keen to get a discussion going here on peoples experiences with the three MV* offerings mentioned. Especially those that have experience with two or more.

Up and Running with Express on Node.js … and friends

July 27, 2013

This is a result of a lot of trial and error, reading, notes taken, advice from more knowledgeable people than myself over a period of a few months in my spare time. This is the basis of a web site I’m writing for a new business endeavour.

Web Frameworks evaluated

  1. ExpressJS Version 3.1 I talked to quite a few people on the #Node.js IRC channel and the preference in most cases was Express. I took notes around the web frameworks, but as there were not that many good contenders, and I hadn’t thought about pushing this to a blog post at the time, I’ve pretty much just got a decision here.
  2. Geddy Version 0.6

MV* Frameworks evaluated

  1. CompoundJS (old name = RailwayJS) Version 1.1.2-7
  2. Locomotive Version 0.3.6. built on Express

At this stage I worked out that I don’t really need a server side MV* framework, as Express.js routes are near enough to controllers. My mind may change on  this further down the track, if and when it does, I’ll re-evaluate.

Templating Engines evaluated

  1. jade Version 0.28.2, but reasonably mature and stable. 2.5 years old. A handful of active contributors headed by Chuk Holoway. Plenty of support on the net. NPM: 4696 downloads in the last day, 54 739 downloads in the last week, 233 570 downloads in the last month (as of 2013-04-01). Documentation: Excellent. The default view engine when running the express binary without specifying the desired view engine. Discussion on LinkedIn. Discussed in the Learning Node book. Easy to read and intuitive. Encourages you down the path of keeping your logic out of the view. The documentation is found here and you can test it out here.
  2. handlebars Version 1.0.10 A handful of active contributors. NPM: 191 downloads in the last day, 15 657 downloads in the last week, 72 174 downloads in the last month (as of 2013-04-01). Documentation: Excellent: nettuts. Also discussed in Nicholas C. Zakas’s book under Chapter 5 “Loose Coupling of UI Layers”.
  3. EJS Most of the work done by the Chuk Holoway (BDFL). NPM: 258 downloads in the last day, 13 875 downloads in the last week, 56 962 downloads in the last month (as of 2013-04-01). Documentation: possibly a little lacking, but the ASP.NET syntax makes it kind of intuitive for developers from the ASP.NET world. Discussion on LinkedIn. Discussed in the “Learning Node” book by Shelley Powers. Plenty of support on the net. deoxxa from #Node.js mentioned: “if you’re generating literally anything other than all-html-all-the-time, you’re going to have a tough time getting the job done with something like jade or handlebars (though EJS can be a good contender there). For this reason, I ended up writing node-ginger a while back. I wouldn’t suggest using it in production at this stage, but it’s a good example of how you don’t need all the abstractions that some of the other libraries provide to achieve the same effects.”
  4. mu (Mustache template engine for Node.js) NPM: 0 downloads in the last day, 46 downloads in the last week, 161 downloads in the last month (as of 2013-04-01).
  5. hogan-express NPM: 1 downloads in the last day, 183 downloads in the last week, 692 downloads in the last month (as of 2013-04-01). Documentation: lacking

Middleware AKA filters

Connect

Details here https://npmjs.org/package/connect express.js shows that connect().use([takes a path defaulting to '/' here], andACallbackHere) http://expressjs.com/api.html#app.use the body of andACallbackHere will only get executed if the request had the sub directory that matches the first parameter of connect().use

Styling extensions etc evaluated

  1. less (CSS3 extension and (preprocessor) compilation to CSS3) Version 1.4.0 Beta. A couple of solid committers plus many others. runs on both server-side and client-side. NPM: 269 downloads in the last day, 16 688 downloads in the last week, 74 992 downloads in the last month (as of 2013-04-01). Documentation: Excellent. Wiki. Introduction.
  2. stylus (CSS3 extension and (preprocessor) compilation to CSS3) Worked on since 2010-12. Written by the Chuk Holoway (BDFL) that created Express, Connect, Jade and many more. NPM: 282 downloads in the last day, 16 284 downloads in the last week, 74 500 downloads in the last month (as of 2013-04-01).
  3. sass (CSS3 extension and (preprocessor) compilation to CSS3) Version 3.2.7. Worked on since 2006-06. Still active. One solid committer with lots of other help. NPM: 12 downloads in the last day, 417 downloads in the last week, 1754 downloads in the last month (as of 2013-04-01). Documentation: Looks pretty good. Community looks strong: #sass on irc.freenode.net. forum. - less, stylus, sass comparison on nettuts. -
  • rework (processor) Version 0.13.2. Worked on since 2012-08. Written by the Chuk Holoway (BDFL) that created Express, Connect, Jade and many more. NPM: 77 downloads in the last week, 383 downloads in the last month (as of 2013-04-01). As explained and recommended by mikeal from #Node.js its basically a library for building something like stylus and less, but you can turn on the features you need and add them easily.  No new syntax to learn. Just CSS syntax, enables removal of prefixes and provides variables. Basically I think the idea is that rework is going to use the likes of less, stylus, sass, etc as plugins. So by using rework you get what you need (extensibility) and nothing more.

Responsive Design (CSS grid system for Responsive Web Design (RWD))

There are a good number of offerings here to help guide the designer in creating styles that work with the medium they are displayed on (leveraging media queries).

Keeping your Node.js server running

Development

During development nodemon works a treat. Automatically restarts node when any source file is changed and notifies you of the event. I install it locally:

$ npm install nodemon

Start your node app wrapped in nodemon:

$ nodemon [your node app]

Production

There are a few modules here that will keep your node process running and restart it if it dies or gets into a faulted state. forever seems to be one of the best options. forever usage. deoxxa’s jesus seems to be a reasonable option also, ningu from #Node.js is using it as forever was broken for a bit due to problems with lazy.

Reverse Proxy

I’ve been looking at reverse proxies to forward requests to different process’s on the same machine based on different domain names and cname prefixes. At this stage the picks have been node-http-proxy and NGinx. node-http-proxy looks perfect for what I’m trying to do. It’s always worth chatting to the hoards of developers on #Node.js for personal experience. If using Express, you’ll need to enable the ‘trust proxy’ setting.

Adding less-middleware

I decided to add less after I had created my project and structure with the express executable.
To do this, I needed to do the following:
Update my package.json in the projects root directory by adding the following line to the dependencies object.
“less-middleware”: “*”

Usually you’d specify the version, so that when you update in the future, npm will see that you want to stay on a particular version, this way npm won’t update a particular version and potentially break your app. By using the “*” npm will download the latest package. So now I just copy the version of the less-middleware and replace the “*”.

Run npm install from within your project root directory:

my-command-prompt npm install
npm WARN package.json my-apps-name@0.0.1 No README.md file found!
npm http GET https://registry.npmjs.org/less-middleware
npm http 200 https://registry.npmjs.org/less-middleware
npm http GET https://registry.npmjs.org/less-middleware/-/less-middleware-0.1.11.tgz
npm http 200 https://registry.npmjs.org/less-middleware/-/less-middleware-0.1.11.tgz
npm http GET https://registry.npmjs.org/less
npm http GET https://registry.npmjs.org/mkdirp
npm http 200 https://registry.npmjs.org/mkdirp
npm http 200 https://registry.npmjs.org/less
npm http GET https://registry.npmjs.org/less/-/less-1.3.3.tgz
npm http 200 https://registry.npmjs.org/less/-/less-1.3.3.tgz
npm http GET https://registry.npmjs.org/ycssmin
npm http 200 https://registry.npmjs.org/ycssmin
npm http GET https://registry.npmjs.org/ycssmin/-/ycssmin-1.0.1.tgz
npm http 200 https://registry.npmjs.org/ycssmin/-/ycssmin-1.0.1.tgz
less-middleware@0.1.11 node_modules/less-middleware
├── mkdirp@0.3.5
└── less@1.3.3 (ycssmin@1.0.1)

So you can see that less-middleware pulls in less as well.
Now you need to require your new middleware and tell express to use it.
Add the following to your app.js in your root directory.

var lessMiddleware = require('less-middleware');

and within your function that you pass to app.configure, add the following.

app.use(lessMiddleware({
   src : __dirname + "/public",
   // If you want a different location for your destination style sheets, uncomment the next two lines.
   // dest: __dirname + "/public/css",
   // prefix: "/css",
   // if you're using a different src/dest directory, you MUST include the prefix, which matches the dest public directory
   // force true recompiles on every request... not the best for production, but fine in debug while working through changes. Uncomment to activate.
   // force: true
   compress : true,
   // I'm also using the debug option...
   debug: true
}));

Now you can just rename your css files to .less and less will compile to css for you.
Generally you’ll want to exclude the compiled styles (.css) from your source control.

The middleware is made to watch for any requests for a .css file and check if there is a corresponding .less file. If there is a less file it checks to see if it has been modified. To prevent re-parsing when not needed, the .less file is only reprocessed when changes have been made or there isn’t a matching .css file.
less-middleware documentation

Bootstrap

Twitters Bootstap is also really helpful for getting up and running and comes with allot of helpful components and ideas to get you kick started.
Getting started.
Docs
.

Bootstrap-for-jade

As I decided to use the Node Jade templating engine, Bootstrap-for-Jade also came in useful for getting started with ideas and helping me work out how things could fit together. In saying that, I came across some problems.

ReferenceError: home.jade:23

body is not defined
    at eval (eval at <anonymous> (MySite/node_modules/jade/lib/jade.js:171:8), <anonymous>:238:64)
    at MySite/node_modules/jade/lib/jade.js:172:35
    at Object.exports.render (MySite/node_modules/jade/lib/jade.js:206:14)
    at View.exports.renderFile [as engine] (MySite/node_modules/jade/lib/jade.js:233:13)
    at View.render (MySite/node_modules/express/lib/view.js:75:8)
    at Function.app.render (MySite/node_modules/express/lib/application.js:506:10)
    at ServerResponse.res.render (MySite/node_modules/express/lib/response.js:756:7)
    at exports.home (MySite/routes/index.js:19:7)
    at callbacks (MySite/node_modules/express/lib/router/index.js:161:37)
    at param (MySite/node_modules/express/lib/router/index.js:135:11)
GET /home 500 22ms

I found a fix and submitted a pull request. Details here.

I may make a follow up post to this titled something like “Going Steady with Express on Node.js … and friends'”

JavaScript Object Creation Patterns

July 6, 2013

What are the differences in creating an object by way of simple function invocation, vs using a constructor vs creating an object using the object literal notation vs function application?

To make sure we’re all on the same page, a quick refresher of what an object actually is in JavaScript…

What is an object in JavaScript?

  • An object is an unordered mutable keyed collection of properties. Each property is either a named data property, a named accessor property, or an internal property. I discussed JavaScript properties in depth here.
  • The ECMAScript language types are Undefined, Null, Boolean, Number, String and Object.
  • The simple types (primitives) of JavaScript are members of one of the following built-in types: Undefined, Null, Boolean (true and false), Number, and String.
  • All other values are objects. Function, String, Number, RegExp etc all indirectly inherit Object via their prototype property, which has a hidden link to Object. Try not to get confused about the fact that we can have for example a String primitive which isn’t an object and we can have a String object by calling the String constructor. See the ES3 and ES5 spec 15.5.
  • All objects have a prototype. I explain JavaScript prototypes here.
  • An object created from a function has a “prototype” property (an Object) (seen below in the red box) (whether invoked as a constructor or function). It has a property which is a constructor function (seen below in blue box) and a hidden property (a link) to the actual Object.prototype (seen below in the pink box).
  • An object created by means of an object literal inherits straight from (is linked to) Object.prototype. So the “prototype” property doesn’t exist, but there are other ways to access it. Problem is… how to access it varies from browser to browser.

internals of a function object

internalsOfAfunctionobject

Every function is also created with two additional hidden properties: the functions context and the code that implements the functions behaviour.

Object creation

On invocation, every function receives 2 hidden parameters. this and the arguments array (which provides access to all the arguments that were supplied to the function on invocation). The value assigned to this is determined by how the function was invoked. We’ll look at this in the following sections.

Object creation via function invocation

Pros

Best suited for creation of one-time on-demand objects.

Cons

If a function is not the property of an object literal, when you invoke it, this will be bound to the global object. Often not what you’re expecting. A mistake in the design of the language. As you can see in the following code, the this of the local scope (kimsGlobalFunction)

aFunctionInItsOwnScope

correct this is global

Now here you can see when we invoke the local function, the this is bound to the global object. That’s a mistake in the language. When innerFunction is invoked, the this of that function is also bound to the global object.


InsideaFunctionsOwnScope

Line 18 above alerts undefined, because that doesn’t belong to the global object.


NotCorrectThisIsGlobal

-

Object creation via constructor

What is a constructor in JavaScript?

It’s a function, nothing more. It’s how it is invoked that determines it as a constructor.

What does it look like?

(function () {
   // Create a constructor function called MyFunc.
   var MyFunc = function (aString) {
      // The following variable is private.
      var privateString = aString;
      // Access it with a privileged method.
      this.publicString = function () {
         return privateString;
      }
   };

   // Prefixing with new means we're now using the function as a constructor.
   // So we use PascalCase rather than camelCase, so users of MyFunc don't invoke without the new prefix.
   var myFunc = new MyFunc('sponge bob');
   alert(myFunc.publicString());
}());

Pros

Great for re-use. Creating a constructor and assigning members to it’s prototype, mean that every time you create an object from the constructor using the new prefix, the new object uses the same prototypes members. This can save big time on memory if you are creating many objects with the constructor.

Cons

Con 1

What happens when someone invokes a constructor function directly without the new prefix?
The this of the function will not be bound to the new object, but rather to the global object.
so instead of augmenting your new myFunc object, you will be clobbering the global object.
myFunc‘s this would refer to the global object

What counter measures do we have at our disposal to make sure this doesn’t happen?
Naming conventions.
Enforcing new is used with JavaScript constructor functions? pg 45 – 46 of Stoyan Stefanov’s JavaScript Patterns book addresses this. Problem is, those patterns all have significant flaws. So, you really need to weigh up the pros and cons. Maturity of your development team should also play a role in your decision here. It may be worth taking a safer route and employing an object literal if developers are likely to omit the new prefix on an intended constructor invocation.

Object creation via object literal

What does it look like?

Line 12 executes the creation and return of an object with a property called publicString.

// In its simplest form:
var myObject = {};

(function () {
   var myFunc = (function () {
      // private members
      var privateString = 'Spong Bob';
      // implement the public part
      return {
         publicString: function () {
            return privateString;
         }
      };
   }());

   alert(myFunc.publicString());
}());

Pros

Pro 1

The this is bound to where you’d expect it to be (adherence to Principle of least astonishment (POLA).
When a function is stored as a property of an object literal, it’s a method. When a method is invoked with this pattern, this is bound to the object. In the below section of code, when execution is on line 05

(function kims() {

   var myObjectLiteral = {
      myProp: 'a property value',
      myFunc: function () {
         alert(this.myProp);
      }
   };
   myObjectLiteral.myFunc();
}());

Here you can see that the value of myFunc‘s this argument is in fact the myObjectLiteral.

this value

In which case because a function is an object, and a variable is a property, then the same must apply to invoking a function of a function? No. The vital word here is “literal”… As above… “When a function is stored as a property of an object literal

Pro 2

Best suited for creation of one-time on-demand objects.

Cons

As with storing members in a function, be it a function you intend using new with (a constructor), or just by invoking the function, members will not be shared between instances of the prototype. This means that if you create 200’000 myObj object literals as I did in the test above, you will have 200’000 separate add functions in memory. The same goes for adding the add function to the constructor without adding it to the constructors prototype.

Object creation via function application

There is one more object creation pattern. Function application. I’ve discussed this in depth in the following three posts. Check them out.

http://blog.binarymist.net/2012/04/29/extending-currying-and-monkey-patching-part-1/

http://blog.binarymist.net/2012/05/14/extending-currying-and-monkey-patching-part-2/

http://blog.binarymist.net/2012/05/27/extending-currying-and-monkey-patching-part-3/

Speed Testing

Object creation is significantly slower using constructors with no prototype than it is using object literals. Now I did some more testing around this and got some surprising results. Using Chromium, V8 is doing some severe optimisation with object creation using constructors. The more members I added to my test constructor and object literal, the more it became noticeable.

var runTestTimes = function (iterations) {
   var test = function () {
      var constructorIterations = 1000000;
      var objLiteralIterations = 1000000;
      var constructorStart;
      var objLiteralStart;
      var constructorTime;
      var objLiteralTime;
      var MyFunc = function () {
         var simpleString = 'simple string';
         var add = function () {
            return 1+1;
         };
         add();
      };

      constructorStart = new Date();
      while (constructorIterations--) {
         var myFunc = new MyFunc();
      }
      constructorTime = new Date - constructorStart;

      objLiteralStart = new Date();
      while (objLiteralIterations--) {
         var myObj = {
            simpleString: 'simple string',
            add: function () {
               return 1+1;
            }
         };
         myObj.add();
      }
      objLiteralTime = new Date - objLiteralStart;

      console.log('constructor: ' + constructorTime + ' object literal: ' + objLiteralTime);
   };
   while (iterations--) {
      test();
   }
};
runTestTimes(10);

Yields the following results:

constructor: 32 object literal: 19
constructor: 25 object literal: 18
constructor: 25 object literal: 17
constructor: 25 object literal: 18
constructor: 26 object literal: 17
constructor: 25 object literal: 18
constructor: 25 object literal: 17
constructor: 25 object literal: 17
constructor: 25 object literal: 18
constructor: 25 object literal: 17

By simply adding another function to the constructor and the object literal, the results in chromium swung to favour the constructor.

// add this function to the constructor.
var subtract = function () {
   return 1-1;
};

// add this function to the object literal.
subtract: function () {
   return 1-1;
}

Reducing the number of iterations from 1’000’000 to 200’000 because the same code run in Firefox crashed it… Yielded the following in Chromium:-

constructor: 5 object literal: 6
constructor: 3 object literal: 5
constructor: 3 object literal: 5
constructor: 2 object literal: 5
constructor: 3 object literal: 4
constructor: 2 object literal: 5
constructor: 2 object literal: 5
constructor: 3 object literal: 4
constructor: 3 object literal: 5
constructor: 2 object literal: 5-

and yielded the following using the Firefox JavaScript engine SpiderMonkey:-

constructor: 701 object literal: 21
constructor: 729 object literal: 17
constructor: 705 object literal: 17
constructor: 721 object literal: 18
constructor: 727 object literal: 20
constructor: 723 object literal: 18
constructor: 726 object literal: 18
constructor: 727 object literal: 20
constructor: 728 object literal: 17
constructor: 736 object literal: 18-

When I moved the add and subtract functions from the constructor to the constructors prototype, the speed results in chromium didn’t yield any noticeable difference. in Firefox

the average went from 854ms to 836ms. The change looked like the following:

var MyFunc = function () {
   var simpleString = 'simple string';
};

MyFunc.prototype.add = function () {
   return 1+1;
};

MyFunc.prototype.subtract = function () {
   return 1-1;
};

I decided to create some tests on jsperf to provide repeatable results. What’s interesting is you don’t have to change a lot to get completely different results, so if you’re concerned about performance, it really pays to test it. I think the tests on jsperf are probably a bit more truthful. here they are.

Summary

There are many more pros and cons of each invocation pattern. I’ve listed the ones that I think are the most important to understand. There is no right or wrong pattern to use for everything. Consider your target audience and what the majority of them may be using in terms of browsers. Consider the maturity of your development team. Benchmark the different approaches, but don’t fall into the trap of micro optimisation, or optimising for a single browser or JavaScript engine unless that’s all your users are using. Choose the pattern that provides the most wins for the given situation. I didn’t test in I.E, but as you can see, the JavaScript engines I did test with do things very differently.

Tools like

  1. Google Page Speed
  2. Google Speed Tracer
  3. Chromiums Profiler

Will help you focus on the areas that matter most.

There are of course many other areas to look at when it comes to “is your app delivering an acceptable user experience”.
Take a few steps back from your situation. You only have so much time. Spend it wisely.
There are many good wins to be had for little cost. Yahoos YSlow has a bunch.
Many books also address this in depth:

  1. Even Faster Web Sites by Steve Souders
  2. High Performance Web sites by Steve Souders
  3. High Performance JavaScript by Nicholas Zakas

There is also a good read on how V8’s full and optimising JIT compilers optimise JavaScript.
I’ve found that most of it’s intuitive and if your using good design and coding principles, in “most” cases your safe, but it’s still worth the read.

  1. As developers we try not to change classes on the fly. deleting or adding properties to hot objects in JavaScript negatively effects the optimising compiler.
  2. We don’t use floats when we only need ints.
  3. In JavaScript, use Arrays when the property names are small sequential integers. Otherwise, use an object. JavaScript Arrays are not like arrays in most other languages. They are simply objects with some array like characteristics.
  4. Assign your array elements as early as possible. 
  5. Don’t delete elements in an array, or leave elements empty.

JavaScript is a very dynamic language. Use its dynamic nature cautiously if you want performant code. Most importantly, favour read time convenience over write time. Your code is going to be read many more times than it’s written.

At this stage, V8 is way ahead of the other JavaScript engines in terms of performance. Node.js uses the V8 engine and enjoys the same incredible performance.

Inheritance

Prototypal inheritance is more OO than classical inheritance.
With prototypal inheritance, a child object only needs to inherit the parent objects specific properties pertinent to it.
With classical inheritance, a child object inherits all the parent objects members, even the ones that it should have no knowledge of.

Hopefully most of us already know to favour composition (aggregation) over inheritance. Be it classical or prototypal. I’ve explained some techniques of how this can be done effectively in JavaScript in the above three posts.

Angus Croll is a master at explaining these concepts, so be sure to check out his post here.

Even the Java creator James Gosling says he’d do away with classes or classical inheritance if he could write the language again.
Inheritance can be an anti pattern as it’s tight coupling. Sub classes inherit everything no matter what. Prototypal is opt-in.
One of the Fluent Conference talks by Eric Elliott on why we should steer away from classical inheritance goes to say the following:

Classical Inheritance is Obsolete
“Those who are unaware they are walking in darkness will never seek the light.” —Bruce Lee
In “Design Patterns”, the Gang of Four recommend two important principles of object oriented design:
1) Program to an interface, not an implementation.
2) Favour object composition over class inheritance.
In a sense, the second principle could follow from the first, because classical inheritance exposes the parent class to all child classes. The child classes are all programming to an implementation rather than an interface. Classical inheritance breaks the principle of encapsulation, and tightly couples the child classes to its ancestors.
Why is the seminal work on Object Oriented design so distinctly anti-inheritance? Because classical inheritance causes several problems:
Tight coupling. Classical inheritance is the tightest coupling available in OO design. Descendant classes have an intimate knowledge of their ancestor classes.
Inflexible hierarchies. Single parent hierarchies are rarely capable of describing all possible use cases. Eventually, all hierarchies are “wrong” for new uses—a problem that necessitates code duplication.
Complicated multiple inheritance. It’s often desirable to inherit from more than one parent. That process is inordinately complex and its implementation is inconsistent with the process for single inheritance, which makes it harder to read and understand.
Brittle architecture. Because of tight coupling, it’s often difficult to refactor a class with the “wrong” design, because much existing functionality depends on the existing design.
The Gorilla / Banana problem. Often there are parts of the parent that you don’t want to inherit. Subclassing allows you to override properties from the parent, but it doesn’t allow you to select which properties you want to inherit.

Additional Sources:
ECMA-262 edition 5.1
JavaScript The Good Parts.
JavaScript The Definitive Guide.
Eric Elliott’s talk at the fluent conference May 28-30, 2013.

Software Engineer Interview Quick Question Set

May 11, 2013

Ice breakers

  • Tell us a little bit about yourself and what drives you?
  • Ask a question from their CV that is positive, ‘what was your greatest success in your current or last role’
  • What’s your ideal job?
  • Can you give us one thing you really enjoyed in your last job?
  • What about one thing that you didn’t enjoy as much?
    How did you solve that?

Testing

  • How can you implement unit testing when there are dependencies between a business layer and a data layer, or the presentation layer and the business layer?
  • The development team is getting near release date. They start saying things like, we’re going to need a sprint to test. What would your reaction be?

Maintenance

  • What measures have you taken to make your software products more easily maintainable?
  • What is the most expensive part of the SDLC?
    (hint: reading others code)

Design and architecture

  • Can you explain some design patterns, and where you have used them?

Scrum

  • Have you used scrum before? (If the answer is no, move on)
  • If you were taken on as a team member and the team was failing Sprint after Sprint. What would you do?
  • What would you do if you were part of a Scrum Team and your manager asked you to do a piece of work not in the Scrum Backlog?
    (hint: manager needs to consult PO. Something has to be removed from Sprint backlog in order for something to be added)

Construction

  • When do you use an abstract class and when do you use an interface?
  • How do you make sure that your code is both safe and fast?
  • Can you describe the process you use for writing a piece of code, from requirements to delivery?

Software engineering questions

  • What are the benefits and drawbacks of Object Orientated Design?
    (hint: polymorphism inheritance encapsulation)
  • What books have you read on software engineering that you thought were good?
  • Explain the terms YAGNI, DRY, SOLID?
    (hint You Aint Gonna Need It. Build what you need as you need it, aggressively refactoring as you go along; don’t spend a lot of time planning for grandiose, unknown future scenarios. Good software can evolve into what it will ultimately become. Every piece of code is code we have to test. If the code is not needed, why are we spending time on it?)

Functional design questions

  • Which controls would you use when a user must select multiple items from a big list, in a minimal amount of space?
  • How would you design editing twenty fields for a list of 10 items? And editing 3 fields for a list of 1000 items?

Specific technical requirements

  • When, where and how do you optimize code?

Web questions

  • How would you mitigate SQL injection?
    (hint: looking for multi layered sanitisation. parameterised SQL. Least privileged account for data access)
  • Have you used XSS and can you provide us an example?
  • What JavaScript libraries have you used?
  • What are some of the irritating limitations of CSS?
  • How would you remove the ASP.NET_SessionId cookie from a MVC controllers Response?
    (hint: Response.Cookies["ASP.NET_SessionId"].Expires = DateTime.Now;)

JavaScript

  • How does JavaScript implement inheritance?
    (hint: via Object’s prototype property)

Service Oriented

  • What are the 3 things a WCF end point must have, or what is the ABC of a WCF service?
    (hint:
    Address – where the WCF service is hosted.
    Binding – that specifies the protocol and its myriad of options.
    Contract – service contract defines what service operations are available to the client for consumption.
    )

C# / .Net questions

  • What’s the difference between public, private, protected and internal modifiers?
  • What are the main differences between the .NET 2.0 and 4.0 garbage collector?
    (hint: background GC was introduced)
  • Describe the different ways arguments can be passed in C#
    (hint: pass val by val, pass val by ref, pass ref by val, pass ref by ref)
  • We have a Base class, we have a child class that inherits BaseClass. Does the child class inherit the base class’s private members?
    (hint: this is normally good for a laugh)
  • Have you ever worked with a deadlock and how did it occur?
  • When should locks be used in concurrent programming?
    (hint:
    when synchronization cannot be performed in any other way. This is rare. With careful thought and planning, there is just about always a better way. There are many ways to synchronise without using locks. System.Threading.Interlocked class generally supported by the processor
    )
  • What are some of your favourite .NET features?

Finally, this question is from Google; can you quickly tell us something that we don’t know anything about? It can be anything.

Software Engineer Interview Process and Questions

April 27, 2013

A short time ago, I was tasked with finding the right software engineer/s for the organisation I was working for. I settled on a process, a set of background questions,  a set of practical programming exercises and a set of verbal questions. Later on I cut the set of verbal questions down to a quicker set. In this post, I’ll be going over the process and the full set of verbal questions. In a subsequent post I’ll go over the quicker set.

The Process

  1. We sent them an email with a series of questions.
    Technical and non-technical.
    They have two days to reply with answers.
    The programming exercises are not covered here.
    If they passed this…
  1. We would get them in for an interview.
    Technical and non-technical questions would be asked.
    They would be put on the spot and asked to speak to the development team about a technical subject that they were familiar with.
    The development team would quiz them on whatever comes to mind.
    Once the candidate had left, the development team would collaborate on what they thought of the candidate and whether or not they would be a good fit for the team.
    The team would take this feedback and discuss whether the candidate should be given a trial. 
    Step 2 could be broken into two parts depending on how many questions and their intensity, you wanted to drill the candidate with.

The following set of tests will confirm whether the candidate satisfies the points we have asked for in the job description.

The non functional (soft) qualities listed on the Job add would need to be kept in mind during the interview events.

Qualities such as:

  • Quality focus
  • Passion
  • Personality
  • Commitment to the organisations needs
  • A genuine sense of excitement about the technologies we work with

Email test

  1. Send Screening.pdf
  2. Send InterviewQuestions.doc

Now with the following questions, with many of them there is not necessarily a right or wrong answer. Many of them are just to gauge how the candidate thinks and whether or not they hold the right set of values.

Ice breakers

  • Would you like to be the team leader or team member?
  • Tell me about a conflict at a previous job and how you resolved it.
  • (Summary personality item: Think to yourself, “If we hire this person, would I want to spend four hours driving in a car with them?”)

Design and architecture

  • What’s the difference between TDD and BDD and why do they matter?
  • What is Technical Debt. How do you deal with it once in it? How do you stay out of it?
  • How would you deal with a pair when reviewing their code, when they have not followed good design principles?
  • What would you do if a fellow team member reviewed your code and suggested you change something you had designed that followed good design principles, to something inferior?
  • Can you explain how the Composite pattern works and where you would use it?
  • Can you describe several class construction techniques?
    What are two design patterns that are focused on class construction, and how do they work?
    (hint: Builder, Factory Method).
  • How would you model the animal kingdom (with species and their behaviour) as a class system?
    (hint GoF design pattern. Abstract Factory)
  • Can you name a number of non-functional (or quality) requirements?
  • What is your advice when a customer wants high performance, high usability and high security?
  • What is your advice when a customer wants high performance, Good design, Cheap?
    (hint: pick 2)
  • What do low coupling and high cohesion mean? What does the principle of encapsulation mean to you?
  • Can you think of some concurrency patterns?
    (hint: Asynchronous Results, Background Worker, Compare/Exchange pattern via Interlocked.CompareExchange)
  • How would you manage conflicts in a web application when different people are editing the same data?
  • Where would you use the Command pattern?
  • Do you know what a stateless business layer is? Where do long-running transactions fit into that picture?
    (hint: if you have long-running transactions, you are going to have to manage state somehow. How would you do this?)
  • What kinds of diagrams have you used in designing parts of an architecture, or a technical design?
  • Can you name the different tiers and responsibilities in an N-tier architecture?
    (hint: presentation, business, data)
  • Can you name different measures to guarantee correctness and robustness of data in an architecture?
    (hint: for example transactions, thread synchronisation)
  • What does the acronym ACID stand for in relation to transactions?
    (hint: atomicity, consistency, isolation, durability)
  • Can you name any differences between object-oriented design and component-based design?
    (hint: objects vs services or documents)
  • How would you model user authorization, user profiles and permissions in a database?(hint: Membership API)

Scrum questions

  • Have you used Scrum before? (If the answer is no, not much point in asking the rest of these questions).
  • If you were taken on as a team member and the team was failing Sprint after Sprint. What would you do?
  • What are the Scrum events and the purpose of them?
    (hint: Daily Scrum, Sprint Planning Meetings 1 & 2, Sprint Review and Sprint Retrospective)
  • What would you do if you were part of a Scrum Team and your manager asked you to do a piece of work not in the Scrum Backlog?
  • Who decides what Product Backlog Items should be pulled into a Sprint?
  • What is the DoD and what is it useful for?
  • Where and how do changing requirements fit into scrum?

Construction questions

  • How do you make sure that your code can handle different kinds of error situations?
    (hint: TDD, BDD, testing…)
  • How do you make sure that your code is both safe and fast?
  • When would you use polymorphism and when would you use delegates?
  • When would you use a class with static members and when would you use a Singleton class?
  • Can you name examples of anticipating changing requirements in your code?
  • Can you describe the process you use for writing a piece of code, from requirements to delivery?
  • Explain DI / IoC. Are there any differences between the two? If so, what are they?
    (hint: DI is one method of following the Dependency Inversion Principle (DIP) or IoC)

Software engineering skills

  • What is Object Oriented Design? What are the benefits and drawbacks?
    (hint: polymorphism inheritance encapsulation)
  • What is the role of interfaces in design?
  • What books have you read on software engineering that you thought were good?
  • What are important aspects of GUI design?
  • What Object Relational Mapping tools have you used?
  • What are the differences between Model-View-Controller, Model-View-Presenter and Model-View-ViewModel
    Can you draw MVC and MVP?
    (hint: doted lines are pub/sub)

MVCM-V-VM

  • What is the difference between Mocks, Stubs, Fakes and Dummies?
  • (hint:
    Mocks are objects pre-programmed with expectations which form a specification of the calls they are expected to receive. Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test.
    Stubs may also record information about calls, such as an email gateway stub that remembers the messages it ‘sent’, or maybe only how many messages it ‘sent’.
    Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
    Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.)
  • Describe the process you would take in setting up CI for our company?
  • We’re going to design the new IMDB.
    On the whiteboard, what would the table that holds the movies look like?
    Every movie has actors, how would the Actors table look?
    Actors star in many movies, any adjustments?
    We need to track Characters also. Any adjustments to the schema?

Relational Database

  • What metrics, like cyclomatic complexity, do you think are important to track in code?

Functional design questions

  • What are metaphors used for in functional design? Can you name some successful examples?
    (hint: Partial Function Application, Currying)
  • How can you reduce the user’s perception of waiting when some routines take a long time?
  • Which controls would you use when a user must select multiple items from a big list, in a minimal amount of space?
  • How would you design editing twenty fields for a list of 10 items? And editing 3 fields for a list of 1000 items?
  • Can you name some limitations of a web environment vs. a Windows environment?

Specific technical requirements

  • What software have you used for bug tracking and version control?
  • Which branching models have you used?
    (hint: No Branches, Release, Maintenance, Feature, Team)
  • What have you used for unit testing, integration testing, UA testing, UI testing?
  • What build tools are you familiar with?
    (hint: Nant, Make, Rake, PSake)

Web questions

  • Would you use a black list or white list? Why?
  • Can you explain XSS and how it works?
  • Can you explain CSRF? and how it works?
  • What is the difference between GET and POST in web forms? How do you decide which to use?
  • What do you know about HTTP.
    (hint: Application Layer of OSI model (layer 7), stateless)
  • What are the HTTP methods sometimes called verbs?
    (hint: there are 9 of them. HEAD, GET, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH)
  • How do you get the current users name from an MVC Controller?
    (hint: The controller has a User property which is of type IPrinciple which has an Identity property of type IIdentity, which has a Name property)
  • What JavaScript libraries have you used?
  • What is the advantage of using CSS?
  • What are some of the irritating limitations of CSS?

JavaScript questions

  • How does JavaScript implement inheritance?
    (hint: via Object’s prototype property)
  • What is the difference between "==" and "===", "!=" and "!=="?
    (hint: If the two operands are of the same type and have the same value, then “===” produces true and “!==” produces false. The evil twins do the right thing when the operands are of the same type, but if they are of different types, they attempt to coerce the values. The rules by which they do that are complicated and unmemorable.
    If you want to use "==", "!=" be sure you know how it works and test well.
    By default use “===” and “!==“. )
    These are some of the interesting cases:
'' == '0'          // false
0 == ''            // true
0 == '0'           // true
false == 'false'   // false
false == '0'       // true
false == undefined // false
false == null      // false
null == undefined  // true
' \t\r\n ' == 0    // true
  • On the whiteboard, could you show us how to create a function that takes an object and returns a child object?
if (typeof Object.create !== ‘function’) {
   Object.create = function (o) {
      var F = function () {};
      F.prototype = o;
      return new F();
   };
}
var child = Object.create(parent);
  • When is “this” bound to the global object?
    (hint: When the function being invoked is not the property of an object)
  • With the following code, how does myObject.pleaseSetValue set myObject.value?
var myObject = {
	value: 0
};

myObject.setValue = function () {
	var that = this; // don’t show this

	var pleaseSetValue = function () {
		that.value = 10; // don’t show this
	};
	pleaseSetValue ();
}
myObject.setValue();
document.writeln(myObject.value); // 10

Service Oriented questions

  • Can you think of any Advantages and Disadvantages in using SOA over an object oriented n-tier model?
  • What’s the simplest way to make a service call from within a web page and how many lines could you do this in?
  • What scales better, per-call services or per-session and why?
    (hint: maintaining service instances (maintaining state) in memory or any entities for that matter quickly blows out memory and other resources.)
  • What is REST’s primary objective?
  • How many ways can you create a WCF proxy?
    (hint:
    Add Service Reference via Visual Studio project
    Using svcutil.exe
    Create proxy on the fly with… new ChannelFactory<IMyContract>().CreateChannel();
    )
  • What do you need to turn on on the service in order to create a proxy?
    (hint: enable an HTTP-GET behaviour, or MEX endpoint)

C# / .Net questions

  • What’s the difference between public, private, protected and internal modifiers?
    Which ones can be used together?
  • What’s the difference between static and non-static methods?
  • What’s the most obvious difference in IL with static constructors?
    (hint: static method causes compiler to not mark type with beforefieldinit, thus giving lazy initialisation.)
  • How have you used Reflection?
  • What does the garbage collector clean up?
    (hint: managed resources, not unmanaged resources. Such as files, streams and handles)
  • Why would you implement the the IDisposable interface?
    (hint: clean up resources deterministically. Clean up unmanaged resources.)
  • Where should the Dispose function be called from?
    (hint: the objects finalizer)
  • Where is an objects finalizer called from?
    (hint: the GC)
  • If you call an objects Dispose method, what System method should you also make sure is called?
    (hint: System.GC.SuppressFinalize)
  • Why should System.GC.SuppressFinalize be called?
    (hint: finalization is expensive)
  • Are strings mutable or immutable?
    (hint: immutable)
  • What’s the most significant difference between struct’s and class’s?
    (hint: struct : value type, class : reference type)
  • What are the other differences between struct’s and class’s?
    (hint: struct’s don’t support inheritance (all value types are sealed) or finalizers)
    (hint: struct’s can have the same fields, methods, properties and operators)
    (hint: struct’s can implement interfaces)
  • Where are reference types stored? Where are value types stored?
    (hint:
    bit of a trick question. Ref on the heap, val on the stack (generally)
    The reference part of reference type local variables is stored on the stack.
    Value type local variables also on the stack.
    Content of reference type variables is stored on the heap.
    Member variables are stored on the heap.
    )
  • Where is the yield key word used?
    (hint: within an iterator)
  • What are some well known interfaces in the .net library that iterators provide implementation for?
    (hint: IEnumerable<T> )
  • Are static methods thread safe?
    (hint: a new stack frame is created with every method call. All local variables are safe… so long as they are not reference types being passed to another thread or being passed to another thread by ref.)
  • What is the TPL used for?
    (hint: a set of API’s in the System.Threading and System.Threading.Tasks namespaces simplifying the process of adding parallelism and concurrency to applications.)
  • What rules would you consider when choosing a lock object?
    (hint: keep the scope as tight as possible (private), so other threads cannot change its value, thus causing the thread to block.
    Declare as readonly, as its value should not be changed.
    Must not be a value type.
    If the lock keyword is used on a value type, the compiler will report an error.
    If used with System.Threading.Monitor, an exception will occur at runtime, because Monitor.Exit receives a boxed copy of the original variable.
    Never lock on “this”.)
  • Why would you declare a field as volatile?
    (hint: So that the order of the operations performed on the variable are not optimised to a different order.)
  • Are reads and writes to a long (System.Int64) atomic? Are reads and writes to a int (System.Int32) atomic?
    (hint: The runtime guarantees that a type whose size is no bigger than a native integer will not be read or written only partially. This is in the CLI spec and the C# 4.0 spec.)
  • Before invoking a delegate instance just before the null check is performed, What’s a good way to make sure no other threads can set your delegate to null between when the check occurs and when you invoke it?
    (hint:
    assign reference to heap allocated memory to stack allocated implements thread safety.
    Assign your delegate instance to a second local delegate variable.
    This ensures that if subscribers to your delegate instance are removed (by a different thread) between checking for null and firing the invocation, you won’t fire a NullReferenceException.)
void OnCheckChanged(EventArgs e) {
	// assign reference to heap allocated memory to
	// stack allocated implements thread safety

	// CheckChanged is a member declared as…  public event EventHandler CheckChanged;
	EventHandler threadSafeCheckChanged = CheckChanged;
	if (threadSafeCheckChanged != null)  {
		// fire the event off
		foreach(EventHandler handler in threadSafeCheckChanged.GetInvocationList()) {
			try {
				handler(this, e);
			}
			catch(Exception e) {
				// handling code
			}
		}
	}
}
  • What is a deadlock and how does one occur? Can you draw it on the white board?
    (hint: two or more threads wait for each other to release a synchronization lock.
    Example:
    Thread A requests a lock on _sync1, and then later requests a lock on _sync2 before releasing the lock on _sync1.
    At the same time,
    Thread B requests a lock on _sync2, followed by a lock on _sync1, before releasing the lock on _sync2.
    )
  • How many ways are there to implement an interface member, and what are they?
    (hint: two. Implicit and explicit member implementation)
  • How do I declare an explicit interface member?
    (hint: prefix the member name with the interface name)
public class MyClass : SomeBaseClass ,IListable, IComparable {
    // …
    public intCompareTo(object obj) {
        // …
    }

    #region IListable Members
    string[] Ilistable.ColumnValues {

        get {
            // …
            return values;
        }
    }
    #endregion
}
  • Write the above on a white board, then ask the following question. If I want to make a call to an explicit member implementation like the above, How do I do it?
string[] values;
    MyClass obj1, obj2;

    // ERROR:  Unable to call ColumnValues() directly on a contact
    // values = obj1.ColumnValues;

    // First cast to IListable.
    values = ((IListable)obj2).ColumnValues;
  • What is wrong with the following snippet?
    (hint: possibility of race condition.
    If two threads in the program both call GetNext simultaneously, two threads might be given the same number. The reason is that _curr++ compiles into three separate steps:
    1. Read the current value from the shared _curr variable into a processor register.
    2. Increment that register.
    3. Write the register value back to the shared _curr variable.
    Two threads executing this same sequence can both read the same value from _curr locally (say, 42), increment it (to, say, 43), and publish the same resulting value. GetNext thus returns the same number for both threads, breaking the algorithm. Although the simple statement _curr++ appears to be atomic, this couldn’t be further from the truth.)
// Each call to GetNext should hand out a new unique number
static class Counter {
    internal static int _curr = 0;
    internal static int GetNext() {
        return _curr++;
    }
}
  • What are some of your favourite .NET features?

Data structures

  • How would you implement the structure of the London underground in a computer’s memory?
    (hint: how about a graph. The set of vertices would represent the stations. The edges connecting them would be the tracks)
  • How would you store the value of a colour in a database, as efficiently as possible?
    (hint: assuming we are measuring efficiency in size and not retrieval or storage speed, and the colour is 16^6 (FFFFFF), store it as an int)
  • What is the difference between a queue and a stack?
  • What is the difference between storing data on the heap vs. on the stack?
  • What is the number 21 in binary format? And in hex?
    (hint: 10101, 15)
  • What is the last thing you learned about data structures from a book, magazine or web site?
  • Can you name some different text file formats for storing unicode characters?
  • How would you store a vector in N dimensions in a datatable?

Algorithms

  • What type of language do you prefer for writing complex algorithms?
  • How do you find out if a number is a power of 2? And how do you know if it is an odd number?
  • How do you find the middle item in a linked list?
  • How would you change the format of all the phone numbers in 10,000 static html web pages?
  • Can you name an example of a recursive solution that you created?
  • Which is faster: finding an item in a hashtable or in a sorted list?
  • What is the last thing you learned about algorithms from a book, magazine or web site?
  • How would you write a function to reverse a string? And can you do that without a temporary string?
  • In an array with integers between 1 and 1,000,000 one value is in the array twice. How do you determine which one?
  • Do you know about the Traveling Salesman Problem?

Testing questions

  • It’s Monday and we’ve just finished Sprint Planning. How would you organize testing?
  • How do you verify that new changes have not broken existing features?
    (hint: regression test)
  • What can you do reduce the chance that a customer finds things that he doesn’t like during acceptance testing?
  • Can you tell me something that you have learned about testing and quality assurance in the last year?
  • What sort of information would you not want to be revealed via Http responses or error messages?
    (hint: Critical info about the likes of server name, version, installed program versions, etc)
  • What would you make sure you turned off on an app or web server before deployment?
    (hint: directory listing?)

Maintenance questions

  • How do you find an error in a large file with code that you cannot step through?
  • How can you make sure that changes in code will not affect any other parts of the product?
  • How can you debug a system in a production environment, while it is being used?

Configuration management questions

  • Which items do you normally place under version control?
  • How would you manage changes to technical documentation, like the architecture of a product?

Project management

  • How many of the three variables scope, time and cost can be fixed by the customer?
  • Who should make estimates for the effort of a project? Who is allowed to set the deadline?
  • Which kind of diagrams do you use to track progress in a project?
  • What is the difference between an iteration and an increment?
  • Can you explain the practice of risk management? How should risks be managed?
  • What do you need to be able to determine if a project is on time and within budget?
    (hint: Product Backlog burn-down)
  • How do you agree on scope and time with the customer, when the customer wants too much?

Candidate displays how they communicate / present to a group of people about a technical topic they are passionate and familiar about.

References I used

If any of these questions or answers are not clear, or you have other great ideas for questions, please leave comments.

Running Wireshark as non-root user

April 13, 2013

As part of my journey with Node.js I decided I wanted to see exactly what was happening on the wire. I decided to use Burp Suite as the Http proxy interceptor and Wireshark as the network sniffer (not an interceptor). Wireshark can’t alter the traffic, it can’t decrypt SSL traffic unless the encryption key can be provided and Wireshark is compiled against GnuTLS.

This post is targeted at getting Wireshark running on Linux. If you’re a windows user, you can check out the Windows notes here.

When you first install Wireshark and try to start capturing packets, you will probably notice the error “You didn’t specify an interface on which to capture packets.”

When you try to specify an interface from which to capture, you will probably notice the error “There are no interfaces on which a capture can be done.”

You can try running Wireshark as root: gksudo wireshark

Wireshark as root

This will work, but of course it’s not a good idea to run a comprehensive tool like Wireshark (over 1’500’000 lines of code) as root.

So what’s actually happening here?

We have dumpcap and we have wireshark. dumpcap is the executable responsible for the low level data capture of your network interface. wireshark uses dumpcap. Dumpcap needs to run as root, wireshark does not need to run as root because it has Privilege Separation.

If you look at the above suggested “better way” here, this will make a “little” more sense. In order for it to make quite a lot more sense, I’ll share what I’ve just learnt.

Wireshark has implemented Privilege Separation which means that the Wireshark GUI (or the tshark CLI) can run as a normal user while the dumpcap capture utility runs as root. Why can’t this just work out of the box? Well there is a discussion here on that. It doesn’t appear to be resolved yet. Personally I don’t think that anybody wanting to use wireshark should have to learn all these intricacies to “just use it”. As the speed of development gets faster, we just don’t have time to learn everything. Although on the other hand, a little understanding of what’s actually happening under the covers can help in more ways than one. Anyway, enough ranting.

How do we get this to all “just work”

from your console:

sudo dpkg-reconfigure wireshark-common

You’ll be prompted:

Configuring wireshark-common

Respond yes.

The wireshark group will be added

If the Linux Filesystem Capabilities are not present at the time of installing wireshark-common (Debian GNU/kFreeBSD, Debian GNU/Hurd), the installer will fall back to set the set-user-id bit to allow non-root users to capture packets. Custom built kernels may lack Linux Capabilities.

The help text also warns about a security risk which isn’t an issue because setuid isn’t used. Rather what actually happens is the following:

addgroup --quiet --system wireshark
chown root:wireshark /usr/bin/dumpcap
setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap

You will then have to manually add your user to the wireshark group.

sudo adduser kim wireshark # replacing kim with your user

or

usermod -a -G wireshark kim # replacing kim with your user

log out then back in again.

I wanted to make sure that what I thought was happening was actually happening. You’ll notice that if you run the following before and after the reconfigure:

ls -liah /usr/bin/dumpcap | less

You’ll see:

-rwxr-xr-x root root /usr/bin/dumpcap initially
-rwxr-xr-x root wireshark /usr/bin/dumpcap after

And a before and after of my users and groups I ran:

cat /etc/passwd | cut -d: -f1
cat /etc/group | cut -d: -f1

-

Alternatively to using the following as shown above, which gives us a nice abstraction (if that’s what you like):

sudo dpkg-reconfigure wireshark-common

We could just run the following:

addgroup wireshark
sudo chgrp wireshark /usr/bin/dumpcap
sudo chmod 750 /usr/bin/dumpcap
sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/dumpcap

The following will confirm the capabilities you just set.

getcap /usr/bin/dumpcap

What’s with the setcap?

For full details, run:

man setcap
man capabilities

setcap sets the capabilities of each specified filename to the capabilities specified (thank you man ;-))

For sniffing we need two of the capabilities listed in the capabilities man page.

  1. CAP_NET_ADMIN Perform various network-related operations (e.g., setting privileged socket options, enabling multicasting, interface configuration, modifying routing tables). This allows dumpcap to set interfaces to promiscuous mode.
  2. CAP_NET_RAW Use RAW and PACKET sockets. Gives dumpcap raw access to an interface.

For further details check out Jeremy Stretch’s explanation on Linux Filesystem Capabilities and using setcap. There’s also some more info covering the “eip” in point 2 here and the following section.

man capabilities | grep -A24 "File Capabilities"

Lets run Wireshark as our usual low privilege user

Now that you’ve done the above steps including the log off/on, you should be able to run wireshark as your usual user and configure your listening interfaces and start capturing packets.

Also before we forget… Ensure Wireshark works only from root and from a user in the “wireshark” group. You can add a temp user (command shown above).

Log in as them and try running wireshark. You should have the same issues as you had initially. Remove the tempuser:

userdel -r tempuser

Setup of Chromium, Burp Suite, Node.js to view HTTP on the wire

March 30, 2013

As part of my Node.js development I really wanted to see what was going over the wire from chromium-browser to my Node.js web apps.

I have node.js installed globaly, express installed locally, a very simple express server listening on port 3000

var express = require('express');
var app = express();

app.get('/', function (request, response) {
   response.send('Welcome to Express!');
});

app.listen(3000);

Burp Suite setup in my main menu. Added the command via System menu -> Preferences -> Main Menu

Burp Suite Command

The Command string looks like the following.

java -jar -Xmx1024m /WhereTheBurpSuiteLives/burpsuite_free_v1.5.jar

Setting up Burp Suite configuration details are found here. I’ve used Burp Suite before several times. Most notably to create my PowerOffUPSGuests library which I discuss here. In that usage I reverse engineered how the VMware vSphere client shuts down it’s guests and replicated the traffic in my library code. For a simple setup, it’s very easy to use. You can spend hours exploring Burps options and all the devious things you can use it for, but to get started it’s simple. Set it up to listen on localhost and port 3001 for this example.

Burp Suite Proxy Listeners

Run the web app

to start our express app from the directory where our above server is located, from a console, run:

node index.js

Where index.js is the name of the file that contains our JavaScript.

To test that our express server is active. We can browse to http://localhost:3000/ or we can curl it:

curl -i  http://localhost:3000/

Should give us something in return like:


HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 19
Date: Sun, 24 Mar 2013 07:53:38 GMT
Connection: keep-alive

Welcome to Express!

Now for the Proxy interception (Burp Suite)

Now that we’ve got end to end comms, lets test the interceptor.

Run burpsuite with the command I showed you above.

Fire the Http request at your web app via the proxy:

curl -i --proxy http://localhost:3001 http://localhost:3000/

Now you should see burps interceptor catch the request. On the Intercept tab, press the Forward button and curl should show a similar response to above.

Burp Suite Proxy Intercept

If you look at the History tab, you can select the message curl sent and also see the same Response that curl received.

Burp Suite Proxy History

Now you can also set Burp to intercept the server responses too. In fact Burp is extremely configurable. You can also pass the messages to different components of Burp to process how ever you see fit. As you can see in the above image looking at all the tabs that represent burp tools. These can be very useful for penetration testing your app as you develop it.

I wanted to be able to use chromium normally and also be able to open another window for browsing my express apps and viewing the HTTP via Burp Suite. This is actually quite simple. Again with your app running locally on port 3000 and burp listening on port 3001, run:

chromium-browser --temp-profile --proxy-server=localhost:3001

For more chromium options:

chromium-browser -help

Now you can just browse to your web app and have burp intercept your requests.

chromium proxied via burp

You may also want to ignore requests to your search provider, because as your typing in URL’s chromium will send searches when you pause. Under Proxy->Options tab you can do something like this:

Ignore Client Requests


Follow

Get every new post delivered to your Inbox.

Join 212 other followers