Extending, Currying and Monkey Patching. part 2

May 14, 2012

Currying

Currying got it’s name from Haskell Curry,

Haskell Curry

originally discovered by Moses Schönfinkel.

schonfinkel

The programming language Haskell named after Haskell Curry is a purely functional language.
So the concept of Currying has it’s roots in functional programming.

It seems that the concepts of Currying and Partial Function Application are often used interchangeably, although they are different.
Lets try and shed some light on the confusion.

Function Application

In carrying on from part 1 of this topic “Extending, Currying and Monkey Patching”
When it comes to functional programming languages, strictly speaking we don’t so much use the terms calling or invoking a function.
We apply functions.
As you can see in the next example, calling or applying a function has the same effect.
Lines 36, 40 and 45 all do the same thing.

// In JavaScript
var coffeeAttributes;
var BINARYMIST;

BINARYMIST = (function (binMist) {
   return binMist;
} (BINARYMIST || {}));

BINARYMIST = (function (binMist) {

  var capitaliseFirstChar = function (target) {
    return target.replace(/^\w/, function (character) { return character.toUpperCase(); });
  };

   binMist.coffeeAttributes = function (myPoison) {
      try {
        // If String.prototype.toLowerCase fails, myPoison isn't a string
        var poison = (myPoison || "").toLowerCase();
      } catch (e) {
        return "Looks like the argument you passed wasn't a string.";
      }

      if(poison === "real coffee")
        return capitaliseFirstChar(poison) + " provides great taste and a good buzz.";
      if(poison === "instant coffee")
        return capitaliseFirstChar(poison) + " provides terrible taste and minimal buzz.";
      if(poison)
        return "I don't recognise " + poison + ". It's attrbutes could be anyones guess.";

      return "well it depends on what type of coffee we're talking about.";
  };
  return binMist;
}(BINARYMIST || {}));

// Call or invoke your function.
coffeeAttributes = BINARYMIST.coffeeAttributes( new String("Real coffee"));
document.writeln(coffeeAttributes + '<br>');
// Real coffee provides great taste and a good buzz.

coffeeAttributes = BINARYMIST.coffeeAttributes('instant Coffee');
document.writeln(coffeeAttributes + '<br>');
// Instant coffee provides terrible taste and minimal buzz.

// Apply your function.
coffeeAttributes = BINARYMIST.coffeeAttributes.apply(null, ["pseudo coffee"]);
document.writeln(coffeeAttributes + '<br>');
// I don't recognise pseudo coffee. It's attrbutes could be anyones guess.

In essence, how we code  a function call / invocation is syntactic sugar for the slightly longer form “function application”.

Partial Application

Or partial function application…

  • returns a function which behaves like the function you pass in but which takes fewer arguments, because the others have been bound into the returned function.
  • takes a function and from it builds a function which takes fewer arguments.

Implementing Partial Application

Now we could continue with the same theme as above, but for the purpose of providing easy understanding, I’m going to use a simpler theme.
I’ve seen the add function used a few times to explain the topic, so I’m going to use it as well.
We’re going to look at the examples in both JavaScript and C#.
Don’t get to caught up on the C# examples, as I’ve never had a need to curry in C#.
They’re just there to provide some contrast.

// In JavaScript
var BINARYMIST;
var fullApplicationResult;
var add9;
var partialApplicationResult;

BINARYMIST = (function (binMist) {
   return binMist;
} (BINARYMIST || {}));

BINARYMIST = (function (binMist) {
  binMist.add = function (firstParam, secondParam) {
    var firstLocal = (typeof firstParam === 'number') ? firstParam : 0;
    var secondLocal = (typeof secondParam === 'number') ? secondParam : 0;
    return firstLocal + secondLocal;
  }
  return binMist;
}(BINARYMIST || {}));

BINARYMIST.partialApply = (function () {
  return function (localFunction, param2) {
    return function (param3) {
      return localFunction(param2, param3);
    }
  }
}());

fullApplicationResult = BINARYMIST.add.apply(null, [9, 6]);
document.writeln('Full Function Application results: ' + fullApplicationResult + '<br>');
// Full Function Application results: 15

add9 = BINARYMIST.partialApply.apply(null, [BINARYMIST.add, 9]);
partialApplicationResult = add9(6);
document.writeln('Partial Function Application results: ' + partialApplicationResult + '<br>');
// Partial Function Application results: 15

// In C#
using System;

namespace BinaryMist {
    class Program {

        static string Add(int firstParam, int secondParam) {
            return (firstParam + secondParam).ToString();
        }

        static Func<T2, TResult> PartialApply<T1, T2, TResult>(Func<T1, T2, TResult> localFunction, T1 param2) {
            return (param3) => localFunction(param2, param3);
        }

        static void Main(string[] args) {
            Func<int, int, string> kimsAdd = Add;

            Func<int, string> add9 = PartialApply(kimsAdd, 9);
            string partialApplicationResult = add9(6);
            Console.WriteLine("Partial Function Application results: {0}", partialApplicationResult);
            // Partial Function Application results: 15
        }
    }
}

Now with an add method that takes 3 arguments:

// In JavaScript
var BINARYMIST;
var add9;
var partialApplicationResult;

BINARYMIST = (function (binMist) {
   return binMist;
} (BINARYMIST || {}));

BINARYMIST = (function (binMist) {
  binMist.add = function (firstParam, secondParam, thirdParam) {
    var firstLocal = (typeof firstParam === 'number') ? firstParam : 0;
    var secondLocal = (typeof secondParam === 'number') ? secondParam : 0;
    var thirdLocal = (typeof thirdParam === 'number') ? thirdParam : 0;
    return firstLocal + secondLocal + thirdLocal;
  }
  return binMist;
}(BINARYMIST || {}));

BINARYMIST.partialApply = (function () {
  return function (localFunction, param2) {
    return function (param3, param4) {
      return localFunction(param2, param3, param4);
    }
  }
}());

add9 = BINARYMIST.partialApply.apply(null, [BINARYMIST.add, 9]);
partialApplicationResult = add9(6, 2);
document.writeln('Partial Function Application results: ' + partialApplicationResult + '<br>');
// Partial Function Application results: 17

// In C#
using System;

namespace BinaryMist {
    class Program {

        static string Add(int firstParam, int secondParam, int thirdParam) {
            return (firstParam + secondParam + thirdParam).ToString();
        }

        static Func<T2, T3, TResult> PartialApply<T1, T2, T3, TResult>(Func<T1, T2, T3, TResult> localFunction, T1 param2) {
            return (param3, param4) => localFunction(param2, param3, param4);
        }

        static void Main(string[] args) {
            Func<int, int, int, string> kimsAdd = Add;

            Func<int, int, string> add9 = PartialApply(kimsAdd, 9);
            string partialApplicationResult = add9(6, 2);
            Console.WriteLine("Partial Function Application results: {0}", partialApplicationResult);
            // Partial Function Application results: 17
        }
    }
}

and again, but broken out into stages:

// In JavaScript
var BINARYMIST;
var add9;
var add6;
var add2;
var partialApplicationResult;

BINARYMIST = (function (binMist) {
   return binMist;
} (BINARYMIST || {}));

BINARYMIST = (function (binMist) {
  binMist.add = function (firstParam, secondParam, thirdParam) {
    var firstLocal = (typeof firstParam === 'number') ? firstParam : 0;
    var secondLocal = (typeof secondParam === 'number') ? secondParam : 0;
    var thirdLocal = (typeof thirdParam === 'number') ? thirdParam : 0;
    // Time to un-wind the call stack
    return firstLocal + secondLocal + thirdLocal;
  }
  return binMist;
}(BINARYMIST || {}));

BINARYMIST.partialApply = (function () {
  // Because of JavaScript's dynamic typing, we don't need to overload our methods.
  return function (localFunction, param2) {
    return function (param3, param4) {
      return localFunction(param2, param3, param4);
    }
  }
}());

add9 = BINARYMIST.partialApply.apply(null, [BINARYMIST.add, 9]);
add6 = BINARYMIST.partialApply.apply(null, [add9, 6]);
add2 = BINARYMIST.partialApply.apply(null, [add6, 2]);
// Time to wind the call stack up.
partialApplicationResult = add2();
document.writeln('Partial Function Application results: ' + partialApplicationResult + '<br>');

 

// In C#
using System;

namespace BinaryMist {
    class Program {

        static string Add(int firstParam, int secondParam, int thirdParam) {
            // Time to un-wind the call stack.
            return (firstParam + secondParam + thirdParam).ToString();
        }

        // Because of C#'s static typing, we need 3 overloaded methods.
        static Func<T2, T3, TResult> PartialApply<T1, T2, T3, TResult>(Func<T1, T2, T3, TResult> localFunction, T1 param2) {
            return (param3, param4) => localFunction(param2, param3, param4);
        }

        // Because of C#'s static typing, we need 3 overloaded methods.
        static Func<T2, TResult> PartialApply<T1, T2, TResult>(Func<T1, T2, TResult> localFunction, T1 param2) {
            return param3 => localFunction(param2, param3);
        }

        // Because of C#'s static typing, we need 3 overloaded methods.
        static Func<TResult> PartialApply<T1, TResult>(Func<T1, TResult> localFunction, T1 param2) {
            return () => localFunction(param2);
        }

        static void Main(string[] args) {
            Func<int, int, int, string> kimsAdd = Add;

            Func<int, int, string> add9 = PartialApply(kimsAdd, 9);
            Func<int, string> add6 = PartialApply(add9, 6);
            Func<string> add2 = PartialApply(add6, 2);
            // Time to wind the call stack up.
            string partialApplicationResult = add2();
            Console.WriteLine("Partial Function Application results: {0}", partialApplicationResult);
            // Partial Function Application results: 17
        }
    }
}

Currying

  • is creating a function that understands partial application and implements it.
  • builds functions which take multiple arguments by composition of functions which each take a single argument.

Implementing Currying

Now the function that takes the single argument is on line 48.
The composition of functions is performed in the same function where we return that (which is the original add applied to the parameter of 9 for the first test).
So in this function we actually return add applied to 9 concatenated to 6.
The original 9 is held in argumentsArray.
I’ve also modified the add method to be a little more flexible and deal with n number of arguments.

// In JavaScript
// Setup our global
var BINARYMIST;

// Variables for testing
var add9;
var sixteen;
var addOne;
var fortyOne;
var addSix;

// First up we add a method to Function's prototype.
// In JavaScript every object (incl Function) is linked to a prototype object
// This allows us to add any function to Function's prototype
Function.prototype.method = (function (name, func) {
  // Add the function (func) as a property of Function's prototype object.
  this.prototype[name] = func;
  return this;
});

// This is our add method repeated from above.
BINARYMIST = (function (binMist) {
  binMist.add = function () {
    var addend;
    var sum = 0;
    for (arg in arguments) {
      if (arguments.hasOwnProperty(arg)) {
        addend = arguments[arg];
        sum += (typeof addend === 'number') ? addend : 0
      }
    }
    return sum;
  }
  return binMist;
}(BINARYMIST || {}));

// Second we add a function called curry to the global Functions prototype.
Function.method('curry', function ( ) {
  // Because we want to concatenate 2 lots of the arguments, we need arguments to be of type Array.
  // Because the arguments parameter is not a real array, we need to borrow the slice method from Array.prototype.
  var slice = Array.prototype.slice;
  // Create an array from arguments.
  var argumentsArray = slice.apply(arguments);
  // In our case, this is bound to add because it's add's prototype's curry method that we invoked.
  var that = this;
  return function ( ) {
    // We now apply add (which is referenced by that) to the current scope, which is the curried function (add9, in this case).
    return that.apply(null, argumentsArray.concat(slice.apply(arguments)/*Create a real array from arguments.*/)/*Concatenate it with the curry parameter*/);
  };
});

Testing curry

// In JavaScript

// Execute the curry method which is a member of add's prototype.
add9 = BINARYMIST.add.curry(9);
document.writeln("Curry results: " + add9(6) + '<br>'); // Curry results: 15

// Single-step currying. Works with any number of arguments.
addSixteen = BINARYMIST.add.curry(1, 2, 3)(5, 5);
document.writeln("Curry results: " + addSixteen + '<br>'); // Curry  results: 16

// Two-step currying.
addOne = BINARYMIST.add.curry(1);
addSix = BINARYMIST.add.curry(3, 1, 2);
fortyOne = addOne(10, 10, 10, 10);
six = addOne(2, 3);
sixteen = addSix(5, 5);

As you can see from the C# example below, dynamic languages such as JavaScript, make currying less verbose and provide considerably more flexibility.

Currying in C# feels a little awkward.

// In C#
using System;

namespace BinaryMist {
    class Program {

        static int Add(int firstParam, int secondParam, int thirdParam) {
            // Time to un-wind the call stack.
            return firstParam + secondParam + thirdParam;
        }

        static Func<T1, Func<T2, Func<T3, TResult>>> Curry<T1, T2, T3, TResult>(Func<T1, T2, T3, TResult> function) {
            return a => b => c => function(a, b, c);
        }

        // Testing curry
        static void Main(string[] args) {
            Func<int, int, int, int> kimsAdd = Add;

            string curryResult = "Curry results: {0}";

            // Single-step currying.
            var curried = Curry(kimsAdd);
            Console.WriteLine(curryResult, curried(9)(6)(3));
            // Curry results: 18

            // Two-step currying.
            Func<int, Func<int, Func<int, int>>> curryAdd = Curry(kimsAdd);
            Func<int, Func<int, int>> add9 = curryAdd(9);
            Func<int, int> add15 = add9(6);
            Console.WriteLine(curryResult, add15(3));
            // Curry results: 18
        }
    }
}

Why / where would we want to Curry

If you are making calls to a particular function with some of the same arguments every time.
You could dynamically create a function that holds on to the previously repeated parameters, thus saving you the effort of passing them every time.
This function will supply the original function with it’s saved parameters, so the original function has the full set.
Currying is not really needed in C#. The examples provided are for academic purposes only.

To wrap up:

Although I don’t think there is much use for explicit Partial Application or Currying in C#,
Jon Skeet has an interesting blog post on Partial Application and Currying in C#, with:

  1. some sample code useful for stepping through
  2. a good controversial and thought provoking conversation following

This post also provided some clarification.

Other Insights obtained from the excellent:

  1. Doug Crockfords JavaScript The Good Parts
  2. Stoyan Stefanov’s JavaScript Patterns

Extending, Currying and Monkey Patching. part 1

April 29, 2012

Extending

The JavaScript Function.prototype.call and Function.prototype.apply methods allow us to extend an object with additional functionality.
The first argument to Function.prototype.call and Function.prototype.apply is the object on which the function is to be:
temporarily added to, invoked, then removed again.
The first argument is bound to this for the current scope (within the current function body).
Be wary of what this is bound to. I talk a little more about this below.
If you have no arguments to pass to the applied method, call or apply will achieve the same result.

The way I remember how the call and apply functions work, is like the following:

BINARYMIST.extensions.minutes.remaining applied to (BINARYMIST.coffee)
Method on the left is applied to the object on the right.

First we create our single global object (line 01) that will be used as a namespace
On line 05 we declare the extensions object and augment it to our single BINARYMIST object.
You can see on line 07, this is how we create extension methods in JavaScript.
If we replaced the apply method from line 34 with call,  the outcome would be exactly the same.
You’ll also notice on line 34 and 35, that the remaining method of BINARYMIST.extensions.minutes returns a memberMinutes property.
Once BINARYMIST.extensions.minutes.remaining is applied to BINARYMIST.coffee, return this.memberMinutes now returns a function that returns privateMinutes.

javascript binding callback

You can copy past the below code:

var BINARYMIST = (function (binMist) {
   return binMist;
} (BINARYMIST || {/*if BINARYMIST is falsy, create a new object and pass it*/}));

BINARYMIST.extensions = (function () {
   var localExtensions = {}; // private
   localExtensions.minutes = {
      memberMinutes: 0,
      remaining: function () {
         // this is bound to the object that "remaining" is a member of at runtime, because "remaining" is a property of the object "localExtensions.minutes".
         // If "remaining" was not the property of an object, it would be invoked as a function rather than a method. "this" would be bound to the global object.
         return this.memberMinutes;
      }
   };
return localExtensions;
}());

var BINARYMIST = (function (binMist) {
   var privateMinutes = 5;
   binMist.coffee = {
      type: "ShortBlack",
      minutes: function () {
         return privateMinutes;
      }
   };
   return binMist;
}(BINARYMIST || {}));
// Function.call, Function.apply are interchangeable if your only using a single parameter.
// The first argument of call or apply can also be a primitive value, null or undefined.
//   Either way it will be bound to the this object.
//   A word of warning here:
//   When a function is not the property of an object, then it is invoked as a function.
//   The this object is then bound to the global object (One of the poor design decisions in JavaScript).
var deliveryTime = BINARYMIST.extensions.minutes.remaining.apply(BINARYMIST.coffee);
alert("Your coffee will be ready in " + deliveryTime() + " minutes");

In continuing with the code comment immediately above:
If the first parameter to apply or call is null, then the this is bound to the global object upon invocation.
When you invoke a function that is not a method, this is exactly what happens.

You may have noticed, I’m using the module pattern in the above example.
This pattern is effective for getting our objects nicely nested into a single namespace (or more correctly an object ) that sits in the global object.
This stops us from littering the global space with every object we create.
You can also see from the following image, what’s private and what’s public.
So we can easily define accessibility.

JavaScript global scope abatement

What are the differences between apply and call then?

call

You can think of Function.prototype.call as being syntax sugar on top of Function.prototype.apply.
If you only have a single parameter after the first, your better off to use Function.prototype.call as it’s more efficient than creating an array for a single value.

The difference is in the parameters after the first invocation context argument.
With call, an arbitrary number of arguments can be passed.
Following the first argument, each argument will be passed to the extension method. In our case remaining (line 09 above).
So we could make the following changes to the above code.
You’ll notice on line 8 below once applied to BINARYMIST.coffee, no longer returns a function, because we explicitly set this.memberMinutes to a number… 3.
Now in the below example on line 16, if we fail to pass an argument to BINARYMIST.extensions.minutes.remaining, by default our coffee will be ready in 10 minutes.

BINARYMIST.extensions = (function () {
   var localExtensions = {}; //private
   localExtensions.minutes = {
      memberMinutes: 0,
      remaining: function (numberOfMinutes) {

         var defaultMinutes = 10;
         this.memberMinutes = numberOfMinutes || defaultMinutes;

         return this.memberMinutes;
      }
   };
return localExtensions;
}());

var deliveryTime = BINARYMIST.Extensions.minutes.remaining.call(BINARYMIST.coffee, 3);
alert("Your coffee will be ready in " + deliveryTime + " minutes");

Better still, lets pass a couple of arguments..
While we’re at it, lets tidy things up a bit.
Lets put the code we use to test the extensions on our coffee into a self invoking function expression,
also known as an anonymous closure.
You can see this on line 36.

On line 15 we assign localSpent the second argument we passed in (minutesSpent),
but only if we’re sure it’s a number,
else we assign the undefined value.

var BINARYMIST = (function (binMist) {
   return binMist;
} (BINARYMIST || {/*if BINARYMIST is falsy, create a new object and pass it*/}));

BINARYMIST.extensions = (function () {
   var localExtensions = {}; // private
   localExtensions.minutes = {
      memberMinutes: 0,
      remaining: function (numberOfMinutes) {
         var defaultMinutes = 10;

         // check our numbers are actually numbers.
         var isNumber = function isNumber(num) { return typeof num === 'number' && isFinite(num);}
         var localRemaining = isNumber(numberOfMinutes) ? numberOfMinutes : defaultMinutes;
         var localSpent = isNumber(arguments[1]) ? arguments[1] : undefined;

         this.memberMinutes = {remaining: localRemaining, spent: localSpent}
         return this.memberMinutes;
      }
  };
return localExtensions;
}());

// lets add some coffee
var BINARYMIST = (function (binMist) {
   var privateMinutes = 5;
   binMist.coffee = {
      type: "ShortBlack",
      memberMinutes: function () {
         return privateMinutes;
      }
   };
   return binMist;
}(BINARYMIST || {}));

(function () {
  var minutesRemaining = 3;
  var minutesSpent = 7;
  var deliveryTime = BINARYMIST.extensions.minutes.remaining.call(BINARYMIST.coffee, minutesRemaining, minutesSpent);
  var minutesSinceOrder = deliveryTime.spent || 'a number of';
  alert("You placed your order " + minutesSinceOrder + " minutes ago. Your coffee will be ready in " + deliveryTime.remaining + " minutes.");
}());

On line 39 we pass in a couple of arguments.
Now to show another cool feature of JavaScript,
because I hate to miss a good opportunity.
All functions come with an arguments array.
This parameter is populated with all the arguments that were supplied to the function on invocation.
This includes any arguments that were not assigned to parameters.
As you can see below, arguments holds both values we passed in.

JavaScript argument's array

Now if we only pass in the first argument to the specified parameter, when line 41 is executed, we get…

apply

The difference with Function.apply is that it only takes 2 arguments.
The second being an array of arbitrary length.
Function.apply works with array-like objects as well as true arrays.


// show a new BINARYMIST.extension.minutes that takes an array

var minutesRemaining = 3;
var minutesSpent = 7;
var minutes = [minutesRemaining, minutesSpent];
var deliveryTime = BINARYMIST.Extensions.minutes.remaining.call(BINARYMIST.coffee, minutes);

Bind

In response to Mike Wilcox’s comments on the JavaScript Linked-in group
I’ve added some info on Function.prototype.bind introduced in Ecma 262 (I think).

This function is very similar to Function.prototype.call.
In fact the only difference I can see is that bind essentially returns a reference to the function that applies the function we want to apply to a target function.
Essentially this allows us to re-use an applied function, rather than create one each time we want to execute.
bind has the same function signature (has the same parameters) as Function.prototype.call.
This following example was taken from Mike West’s post here with some small changes and comments added.

This is tested and works as you would expect.

var first_object = {
  num: 42
};

var second_object = {
  num: 24
};

function multiply(mult) {
  return this.num * mult;
}

Function.prototype.bind = function (obj) {
  // When a function is not the property of an object, then it is invoked as a function:
  // Now because bind is a property of Function.prototype
  // and multiply inherits the bind property (which is a function of course) because multiply's prototype is Function's prototype.
  // this is bound to the bind properties object... which in this case is multiply.
  var method = this
  var temp = function () {
    return method.apply(obj, arguments);
  };
  // temp holds a reference to our bind functionality
  return temp;
}

var first_multiply = multiply.bind(first_object);
document.writeln(first_multiply(5) + '
'); // returns 42 * 5

var second_multiply = multiply.bind(second_object);
document.writeln(second_multiply(5) + '
'); // returns 24 * 5

Supporting multiple sites with a single SSL Certificate

April 9, 2012

There are a couple of ways I’m aware of you can support multiple web sites with a single SSL certificate using the same port.

  1. Wild card certificate
    Useful for when your collection of sites are on the same domain.
    For example:
    mysane.site.com, myinsane.site.com, mycrazy.site.com
  2. Unified Communications Certificate (UCC) / Subject Alternative Name (SAN) / MultiDomain
    Useful for when your collection of sites are on different domains.
    For example:
    mysanesite.com, myinsanesite.com, mycrazysite.com

You can choose to purchase a SSL cert,
you can use convergence (check out Moxie Marlinspikes talk on the subject),
or you can create a self signed one.

If you chose to create a self signed certificate

IIS 7.x

Click on the root machine node in the left tree view of IIS (7.x) manager.
Then double click the “Server Certificates” icon in the Features View.

Server Certificates

This will show you all the certificates currently registered on the server.
You will be able to see in the Actions pane,
that you can Import or create your own certificate.
To create the self signed wild card certificate,
chose “Create Self-Signed Certificate…”.
Give it the friendly name *.site.com
Ok.
The certificate will be registered on you machine.

Server Certificates

Now for each site you want to use the certificate for,
right click -> Edit Bindings… -> Add.
Select the Type to be https,
and select the certificate you just created from the SSL certificate drop down menu.
Ok -> Close.
Repeat these steps for the rest of the sites you want to share the certificate.

Using the appcmd utility

We now add the https binding and host information to our sites that need to share the wild card certificate.

Run a command prompt as administrator and

cd to %WINDIR%\system32\inetsrv

The format of the command looks like the following:

appcmd set site /site.name:"<your website name>" /+bindings.[protocol='https',bindingInformation='*:443:<your ssl domain>']

For our above three sites we wanted to use the same certificate,
mysane.site.com, myinsane.site.com, mycrazy.site.com
They may be named respectively:
mysane, myinsane, mycrazy
So for example,
we’d run the following commands:

appcmd set site /site.name:"mysane" /+bindings.[protocol='https',bindingInformation='*:443:mysane.site.com']

You should get feedback similar to the following:

SITE object "mysane.site.com" changed

if all goes well

appcmd set site /site.name:"myinsane" /+bindings.[protocol='https',bindingInformation='*:443:myinsane.site.com']

You should get feedback similar to the following:

SITE object "myinsane.site.com" changed

if all goes well

appcmd set site /site.name:"mycrazy" /+bindings.[protocol='https',bindingInformation='*:443:mycrazy.site.com']

You should get feedback similar to the following:

SITE object "mycrazy.site.com" changed

if all goes well

Although I normally keep it simple and name my sites the same as the URL (your ssl domain) I want to use.

IIS 6

Now this is a bit more work than with IIS 7.

If it’s not already installed, you’ll need the SelfSSL tool.
You can get this from the SSL Diagnostics Kit or the IIS 6.0 Resource Kit which contains lots of other stuff.
Once installed, run IIS.

Create the self signed wildcard certificate

You’ll need to generate the certificate for one existing IIS site.
For the first site take note of the site idendifier.
You can see this in the right pane when you select Web Sites from the server node in the IIS manager.
Open a command prompt, you’ll need to run the SelfSSL app.
Actually I think the easiest way to run this is Start menu -> All Programs -> IIS Resources -> SelfSSL -> SelfSSL.
The command string looks like this:

selfssl /n:cn=<your wild card domain> /s:<first website identifier> /P:<port you want to use> /v:<number of days to expiration>

So for example, we’d run the following command:

selfssl /n:cn=*.site.com /s:1 /P:443 /v:365

Options for SelfSSL

selfssl /?

some of them are:

/N: – This specifies the common name of the certificate. The computer name is used if there is no common name specified.
/K: – This specifies the key length of the certificate. The default is length 1024.
/V: – This specifies the amount of time the certificate will be valid for, calculated in days. The default setting is seven days.
/S: – This specifies the Identifier of the site, which we obtained earlier. The default will always be 1, which is the Default Web Site in IIS.

Assign the certificate to the sites that need it

Have a look at the site properties in IIS Manager -> Directory Security tab -> Server Certificate button.
This will start the IIS wizard.
Click Next -> Assign an existing certificate -> Next.
You should see the wild card certificate you created.
Select it, click next, and make sure you assign it the same port that was assigned to the first site.

Configure the SecureBindings

In order for IIS to use the host headers with SSL and secure the certificate as we did with appcmd,
you’ll need to run the following command for each of the sites that require it.
My adsutil is found in C:\Inetpub\AdminScripts\
It’s probably not in your path, so you’ll have to run it from location.
cscript adsutil.vbs set /w3svc/<website identifier>/SecureBindings ":443:<your ssl domain>"

So for example, we’d run the following command:
cscript adsutil.vbs set /w3svc/1/SecureBindings ":443:mysane.site.com"

That should be it.

Now if you need to remove a certificate from your store

Run mmc.exe
File menu -> Add/Remove Snap-in… -> Add… -> select Certificates -> Add -> select Computer account -> Next -> select Local computer -> Close -> Ok.
Select the Certificates node, expand Personal, Certificates.
Now in the right window pane, you can manage the certificates.
Delete, Renew etc.

copying with scp

March 25, 2012

I was having some trouble today copying a file (1.5GB .iso) from a notebook to a file server.
The notebook I was using was running Linux Ubuntu.
The server FreeBSD.
I was trying to copy this file using SMB/CIFS via Nautilus.
I tried several times, it failed each time.
Then I thought, what are you doing… drop to the command line.

scp to the rescue

The command I used:

From the directory on my local machine I was copying the file from

scp -P MyPortNumberHere MyFile.iso MyUserName@MyServer:/Path/To/Where/I/Want/MyFile/ToGo/MyFile.iso

This also took about half  the time to copy that SMB took, and SMB didn’t even complete.

Some additional resources

http://www.linuxtutorialblog.com/post/ssh-and-scp-howto-tips-tricks

http://amath.colorado.edu/computing/software/man/scp.html

Also don’t forget to check the man page out ;-)

man scp

How to optimise your testing effort

March 24, 2012

I recently wrote a post for the company I currently work for around the joys of doing TDD.
You can check it out here.

What is your current approach to testing?
How can you spend the little time you have on the most important areas?

I thought I’d share some thoughts around where I see the optimal areas to invest your test effort.
I got to thinking last night, and when I was asleep.
We are putting too much effort into our UI, UA and system tests.
We are writing to many of them, thus we’re creating a top heavy test structure that will sooner or later topple.
These tests have their sweet spot, but they are slow, fragile and time consuming to write.

We should have a small handful for each user story to provide some UA, and the rest should be without the UI and database (the slow and fragile bits).
We need to get our mind sets lower down the test triangle.

test triangle

I’ll try and explain why we should be doing less Manual tests, followed by GUI tests, followed by UA tests, followed by integration tests, followed by Unit tests.

Try not to test the UI with the lower architectural layers included in the tests.
UI tests should have the lower layers mocked and / or stubbed.
Check out Dummy vs Fake vs Stub vs Mock
Full end to end system tests are not required to validate UI field constraints.
Dependency injection really helps us here.

When you are explicitly testing the upper levels of the test triangle, the lower / immediate lower layers are implicitly being tested.
So you might think, cool, if we invest in the upper layers, we implicitly cover the lower layers.
That’s right, but the disadvantages of the higher level tests outweigh the advantages.
UI tests and especially ones that go from end to end, should be avoided, or very few in number,
as they are fragile and incur high maintenance costs.
If we create to many of these, confidence in their value diminishes.
Read on and you’ll find out why.

Lets look at cost vs value to the business.

Some tests cost a lot to create and modify.
Some cost little to create and modify.
Some yield high value.
Some yield low value.
We only have so much time for testing,
so lets use it in the areas that provide the greatest value to the business.
Greatest value of course, will be measured differently for each feature.
There is no stock standard answer here, only guidelines.
What we’re aiming for is to spend the minimum effort (cost) and get the maximum benefit (value).
Not the other way around…
With the following set of scales, we’ve spent to much in the wrong areas, yielding suboptimal value.

cost verse business value

It’s worth the effort to get under the UI layer and do the required setup incl mocking the layers below.
It’s also not to hard to get around the likes of the HttpContext hierarchy of classes (HttpRequest, HttpResponse, and so on) encountered in ASP.NET Web Forms and MVC.

Beware

  • the higher level tests get progressively more expensive to create and maintain.
  • They are slower to run, which means they don’t run as part of CI, but maybe the nightly build.
    Which means there is more latency in the development cycle.
    Developers are less likely to run them manually.
  • When  they break, it takes longer to locate the fault, as you have all the layers below to go through.

Unreliable tests are a major cause for teams ignoring or losing confidence in automated tests.
UI, Acceptance, followed by integration tests are usually the culprits for causing this.
Once confidence is lost, the value initially invested in the automated tests is significantly reduced.
Fixing failing tests and resolving issues associated with brittle tests should be a priority to remove false positives.

Planning the test effort

This is usually the first step we do when starting work on a user story,
or any new feature.
We usually create a set of Test Conditions (Given/When/Then) for Product Backlog items where there are enough use cases for it to be worth doing.
Where we don’t create Test Conditions, we have a Test Condition workshop.
In the workshop we look at the What, How, Who and Why in that order.
The test quadrant (pictured below) assists us in this.
In the workshop, we write the Acceptance Criteria on a board (the What) and discuss the most effective way to verify that the conditions are meet (the How)
With the how we look at the test triangle and the test quadrant and decide where our  time is most effectively spent.

Test condition workshop

With the test condition workshop,
when we start on a user story (generally a feature in the sprint backlog),
we plan where we are going to spend our test resource.
Think about What, and sometimes Who, but not How.
The How comes last.

Unit tests are the developers bread and butter.
They are cheap to create and modify,
and consistently yield not only good value to the developers,
but implicitly good value to most / all other areas.
This is why they sit at the bottom of the test triangle.
This is why TDD is as strong as it is today.
test quadrant

The hierarchy of criteria that we use to help us

  1. Release Criteria
    Ultimately controlled by the Product backlog owner or release manager.
  2. Acceptance Criteria
    Also owned by the Product backlog owner.
    Attached to each user story, or more correctly… product backlog item.
    The Development team must meet these in order to fulfill the Definition of Done.
  3. Test Conditions
    When executable, confirm the development team have satisfied the requirements of the product backlog item.

Write your tests first

TDD is  not about testing, it’s about creating better designs.
This forces us to design better software. “Testable”, “Modular”, separating concerns, Single responsibility principle.
This forces us down the path of SOLID Principles.

red green refactor

  1. Write a unit test
    Run it and watch it fail (because the production code is not yet written)
  2. Write just enough production code to make the test pass
  3. Re-run the test and watch it pass

This podcast around TDD has lots of good info.

Continuous Integration

Realise the importance of setting up CI and nightly builds.
The benefit of having your unit (fast running) tests automatically executed regularly are great.
You get rapid feedback, which is crucial to an agile team completing features on time.
Tests that are not being run regularly have the risk that they may be failing.
The sooner you find a failing test, the easier it is to fix the code.
The longer it’s left unattended, the more technical debt you accrue and the more effort is required to hunt down the fault.
Make the effort to get your tests running on each commit or push.

Nightly Builds

The slower running tests (that’s all the automated tests above unit tests on the triangle), need to be run as part of a nightly build.
We can’t have these running as part of the CI because they are just too slow.
If something gets in the way of a developers work flow, it won’t get done.

Pair Review

Don’t forget to pair review all code written.
In my current position we’ve been requesting reviews verbally and responding with emails, comments on paper.
This is not ideal and we’re currently evaluating review software, of which there are many offerings.

Professional Scrum Master

March 23, 2012

Hi all.

Looking forward to attending the PSM course on Monday 26/03.
Shortly after I’ll be going for the exam.

I’ve been mostly working in a scrum environment since around 2007.
Now I’m looking at solidifying some of that experience and knowledge, and gaining a little more hopefully?

Here’s the outline.

Scrum.org has designed the Professional Scrum Master (PSM) program to have the utmost rigor. The program’s courses, assessments, and certifications give participants the knowledge they need to use Scrum effectively and the credentials they need to communicate this ability in the marketplace.

Audience

The audience of the PSM course includes those that help lead the software development process in an organization. PSM is specifically targeted at the role of the Scrum Master, but the lessons are applicable to anyone in a role that supports a software development team’s efficiency, effectiveness, and continual improvement.

The Course

The Professional Scrum Master course is the first significant update of the Certified ScrumMaster (CSM) course that Ken Schwaber first created in 2002. This course covers Scrum basics, including the framework, mechanics, and roles of Scrum. But it also teaches how to use Scrum to optimize value, productivity, and the total cost of ownership of software products. Students learn through instruction and team-based exercises, and they are challenged to think on their feet to better understand what to do when they return to their workplaces.

Scrum.org maintains a defined curriculum for the Professional Scrum Master courses and selects only the most qualified instructors to deliver them. Each instructor brings his or her individual experiences and areas of expertise to bear, but all students learn the same core course content. This improves their ability to pass the Professional Scrum Master assessments and apply Scrum in their workplaces.

The Professional Scrum Master course (previously known as the Scrum In Depth course) covers Scrum basics, including the framework, mechanics, and roles of Scrum. But it also teaches how to use Scrum how to optimize value, productivity, and the total cost of ownership of software products. Students learn through instruction and team-based exercises, and they are challenged to think on their feet to better understand what to do when they return to their workplaces.

The course curriculum covers:

  • Scrum Basics. What is Scrum and how has it evolved?
  • Scrum Theory. Why does Scrum work and what are its core principles? How are the Scrum principles different from those of more traditional software development approaches, and what is the impact?
  • Scrum Framework and Meetings. How Scrum theory is implemented using time-boxes, roles, rules, and artifacts. How can these be used most effectively and how can they fall apart?
  • Scrum and Change. Scrum is different: what does this mean to my project and my organization? How do I best adopt Scrum given the change that is expected?
  • Scrum and Total Cost of Ownership. A system isn’t just developed, it is also sustained, maintained and enhanced. How is the Total Cost of Ownership (TCO) of our systems or products measured and optimized?
  • Scrum Teams. Scrum Teams are self-organizing and cross-functional; this is different from traditional development groups. How do we start with Scrum teams and how do we ensure their success?
  • Scrum Planning. Plan a project and estimate its cost and completion date.
  • Predictability, Risk Management, and Reporting. Scrum is empirical. How can predictions be made, risk be controlled, and progress be tracked using Scrum.
  • Scaling Scrum. Scrum works great with one team. It also works better than anything else for projects or product releases that involve hundreds or thousands of globally dispersed team members. How is scaling best accomplished using Scrum?

Prerequisites

The Professional Scrum Master course is primarily targeted at those responsible for the successful use and/or rollout of Scrum in a project or enterprise. Attendees will be able to make the most of the class if they:

  • Have attended the Professional Scrum Foundations course
  • Understand the basics of project management.
  • Understand requirements and requirements decomposition.
  • Have been on or closely involved with a project that builds or enhances a product.
  • Have studied the Scrum Guide.
  • Have read one of the Scrum books.
  • Want to know more about how Scrum works, how to use it, and how to implement it in an organization.

Assessment and Certification

As a matter of principle, Scrum.org feels that certification should be available to all those who possess a particular level of knowledge — not only to those who have taken a class. As a result, they offer the option of Professional Scrum Master I and II assessments to the public — not only to those who have taken the Professional Scrum Master course. The Professional Scrum Master program features two assessments and two levels of certification.

Keeping your events thread safe

March 11, 2012

An area I’ve noticed where engineers often forget to think about synchronization is where firing events.
Now I’m going to go over a little background on C# delegates quickly just to refresh what we learnt or should have learnt years ago at the beginnings of the C# language.

It seems to be a common misconception, that all that is needed to keep synchronisation,
is to check the delegate (technically a MulticastDelegate, or in architectural terms the publisher of the publish-subscribe pattern (more commonly known as the observer pattern)) for null.

Defining the publisher without using the event keyword

public class Publisher {
   // ...

   // Define the delegate data type
   public delegate void MyDelegateType();

   // Define the event publisher
   public MyDelegateType OnStateChange {
      get{ return _onStateChange;}
      set{ _onStateChange = value;}
   }
   private MyDelegateType _onStateChange;

   // ...
}

When you declare a delegate, you are actually declaring a MulticastDelegate.
The delegate keyword is an alias for a type derived from System.MulticastDelegate.
When you create a delegate, the compiler automatically employs the System.MulticastDelegate type rather than the System.Delegate type.
When you add a method to a multicast delegate, the MulticastDelegate class creates a new instance of the delegate type, stores the object reference and the method pointer for the added method into the new instance, and adds the new delegate instance as the next item in a list of delegate instances.
Essentially, the MulticastDelegate keeps a linked list of Delegate objects.

It’s possible to assign new subscribers to delegate instances, replacing existing subscribers with new subscribers by using the = operator.
Most of the time what is intended is actually the += operator (implemented internally by using System.Delegate.Combine()).
System.Delegate.Remove() is what’s used when you use the -+ operator on a delegate.

class Program {
   public static void Main() {

      Publisher publisher = new Publisher();
      Subscriber1 subscriber1 = new Subscriber1();
      Subscripber2 subscripber2 = new Subscripber2();

      publisher.OnStateChange = subscriber1.OnStateChanged;

      // Bug: assignment operator overrides previous assignment.
      // if using the event keyword, the assignment operator is not supported for objects outside of the containing class.
      publisher.OnStateChange = subscriber2.OnStateChanged;

   }
}

Another short coming of the delegate is that delegate instances are able to be invoked outside of the containing class.

class Program {
   public static void Main() {
      Publisher publisher = new Publisher();
      Subscriber1 subscriber1 = new Subscriber1();
      Subscriber2 subscriber2 = new Subscriber2();

      publisher.OnStateChange += subscriber1.OnStateChanged;
      publisher.OnStateChange += subscriber2.OnStateChanged;

      // lack of encapsulation
      publisher.OnStateChange();
   }
}

C# Events come to the rescue

in the form of the event keyword.
The event keyword address’s the above problems.

The modified Publisher looks like the following:

public class Publisher {
   // ...

   // Define the delegate data type
   public delegate void MyDelegateType();

   // Define the event publisher
   public event MyDelegateType OnStateChange;

   // ...
}

Now. On to synchronisation

The following is an example from the GoF guys with some small modifications I added.
You’ll also notice, that the above inadequacies are taken care of.
Now if the Stock.OnChange is not accessed by multiple threads, this code is fine.
If it is accessed by multiple threads, it’s not fine.
Why I hear you ask?
Well, between the time the null check is performed on the Change event
and when Change is fired, Change could be set to null, by another thread.
This will of course produce a NullReferenceException.

The code on lines 59,60 is not atomic.

using System;
using System.Collections.Generic;

namespace DoFactory.GangOfFour.Observer.NETOptimized {
    /// <summary>
    /// MainApp startup class for .NET optimized
    /// Observer Design Pattern.
    /// </summary>
    class MainApp {
        /// <summary>
        /// Entry point into console application.
        /// </summary>
        static void Main() {
            // Create IBM stock and attach investors
            var ibm = new IBM(120.00);

            // Attach 'listeners', i.e. Investors
            ibm.Attach(new Investor { Name = "Sorros" });
            ibm.Attach(new Investor { Name = "Berkshire" });

            // Fluctuating prices will notify listening investors
            ibm.Price = 120.10;
            ibm.Price = 121.00;
            ibm.Price = 120.50;
            ibm.Price = 120.75;

            // Wait for user
            Console.ReadKey();
        }
    }

    // Custom event arguments
    public class ChangeEventArgs : EventArgs {
        // Gets or sets symbol
        public string Symbol { get; set; }

        // Gets or sets price
        public double Price { get; set; }
    }

    /// <summary>
    /// The 'Subject' abstract class
    /// </summary>
    abstract class Stock {
        protected string _symbol;
        protected double _price;

        // Constructor
        public Stock(string symbol, double price) {
            this._symbol = symbol;
            this._price = price;
        }

        // Event
        public event EventHandler<ChangeEventArgs> Change;

        // Invoke the Change event
        private void OnChange(ChangeEventArgs e) {
            // not thread safe
            if (Change != null)
                Change(this, e);
        }

        public void Attach(IInvestor investor) {
            Change += investor.Update;
        }

        public void Detach(IInvestor investor) {
            Change -= investor.Update;
        }

        // Gets or sets the price
        public double Price {
            get { return _price; }
            set {
                if (_price != value) {
                    _price = value;
                    OnChange(new ChangeEventArgs { Symbol = _symbol, Price = _price });
                    Console.WriteLine("");
                }
            }
        }
    }

    /// <summary>
    /// The 'ConcreteSubject' class
    /// </summary>
    class IBM : Stock {
        // Constructor - symbol for IBM is always same
        public IBM(double price)
            : base("IBM", price) {
        }
    }

    /// <summary>
    /// The 'Observer' interface
    /// </summary>
    interface IInvestor {
        void Update(object sender, ChangeEventArgs e);
    }

    /// <summary>
    /// The 'ConcreteObserver' class
    /// </summary>
    class Investor : IInvestor {
        // Gets or sets the investor name
        public string Name { get; set; }

        // Gets or sets the stock
        public Stock Stock { get; set; }

        public void Update(object sender, ChangeEventArgs e) {
            Console.WriteLine("Notified {0} of {1}'s " +
                "change to {2:C}", Name, e.Symbol, e.Price);
        }
    }
}

At least we don’t have to worry about the += and -= operators. They are thread safe.

Ok. So how do we make it thread safe?
Now I’ll do my best not to make your brain hurt.
We can assign a local copy of the event and then check that instead.
How does that work you say?
The Change delegate is a reference type.
You may think that  threadSafeChange references the same location as Change,
thus any changes to Change would also be reflected in threadSafeChange.
That’s not the case though.
Change += investor.Update does not add a new delegate to Change, but assigns it a new MulticastDelegate,
which has no effect on the original MulticastDelegate that threadSafeChange also references.

The reference part of reference type local variables is stored on the stack.
A new stack frame is created for each thread with every method call
(whether its an instance or static method).
All local variables are safe…
so long as they are not reference types being passed to another thread or being passed to another thread by ref.
So, only one thread can access the threadSafeChange instance.

private void OnChange(ChangeEventArgs e) {
   // assign reference to heap allocated memory to stack allocated implements thread safety
   EventHandler<ChangeEventArgs> threadSafeChange = Change;
   if ( threadSafeChange != null)
      threadSafeChange(this, e);
}

Now for a bit of error handling

If one subscriber throws an exception, any subscribers later in the chain do not receive the publication.
One way to get around this problem, is to semantically override the enumeration of the subscribers.
Thus providing the error handling.

private void OnChange(ChangeEventArgs e) {
   // assign reference to heap allocated memory to stack allocated implements thread safety
   EventHandler<ChangeEventArgs> threadSafeChange = Change;
   if ( threadSafeChange != null) {
      foreach(EventHandler<ChangeEventArgs> handler in Change.GetInvocationList()) {
         try {
            //if subscribers delegate methods throw an exception, we'll handle in the catch and carry on with the next delegate
            handler(this, e);
            // if we only want to allow a single subscriber
            if (Change.GetInvocationList().Length > 1)
               throw new Exception("Too many subscriptions to the Stock.Change" /*, provide a meaningful inner exception*/);
         }
         catch (Exception exception) {
            // what we do here depends on what stage of development we are in.
            // if we're in early stages, pre-release, fail early and hard.
         }
      }
   }
}

Bare-metal Hypervisor Setup Evaluation

January 23, 2012

The views expressed in this post are my own and don’t reflect the views of my employer.

Recently I had the opportunity for work, to carry out some research on what’s in the market in regards to bare-metal hypervisors.

The following is the result of an in depth research and deployment project of the following bare-metal hyper-visors.
This will enable us to trial the hypervisors out for performance, ease of setup, ease of administration, and ease of use.

I’ve also looked at hardware costs, but first it needs to be decided which hypervisor we are going to go with.
As this would be a team decision, I thought the best way to go about this was to record some of my existing experience with further research into some of the product leaders offerings.

I haven’t used KVM before.
I knew it existed, but when I was last in the market comparing hypervisors, KVM was an infant.
Now it appears to have grown up and is comparable with it’s commercial rivals.
This pretty much sums up the KVM vs VMware battle
This pretty much sums up the Xen vs KVM battle


ESX(i)

I’ve used these extensively and am well aware of their pros and cons.
Supports iscsi.
I prefer not to have to pay for a product if there are FOS (Free & Open Source) offerings that get the job done just as well.
In looking at the likes of KVM and Xen, the cons of ESX/ESXi really stand out, not to mention the fact that KVM is completely free, more efficient and has a faster pace of growth.
With the free version, that’s ESXi, you get (as of version 5) 32GB vRAM, and that’s only because the community kicked up such a fuss about paying per CPU for a product that was originally free.
VMware keep changing the rules and pricing strategies when users go else where. I’d prefer not to pay at all.
I’m not going to spend time recording the pros and cons of VMware at this stage, as I think the other contenders have more to offer, and ask for less or nothing in return.
If we find that there are un-foreseen hurdles in the other products, we should look at ESXi as a backup.

Management

vSphere client (only runs on windows).
vSphere CLI (read-only, unless you pay for license)
Have very limited access to the hypervisor

Migration

  • General
  • Potential migration of KVM to VMware.
    Although this link says  the above won’t work, but has some other suggestions.

UPS

See my blog posts.


Citrix XenServer

XenServer support for iscsi

Xen is a type 1 bare-metal hypervisor. This means it runs as close to the hardware as possible.
To take full advantage of it’s speed, you have to run paravirtualised (modified OS’s).
Since most of our work at this stage would be on Windows, there would be no benefit here for us.
Runs in a small custom Linux system.
Intel VT-x or AMD-V is required to run full hardware virtualisation (HVM) rather than paravirtualised.

Licensing for XenServer Express

Be aware, Citrix can change their licensing structure at any time.
Features and current licensing model
XenServer Licensing FAQ
XenCenter can only connect to a single instance of XenServer at any one time.
XenServer currently free
XenCenter free
http://www.citrix.com/English/NE/news/news.asp?newsID=1687130

FAQ

Management

Migration

ESX(i) to XenServer

Seemed to have struggles (windows guest).
Seemed to be a little more successful (windows guest).

UPS

Integrating XenServer and APC PowerChute. Also see this.
Using apcupsd as KVM can.

Installation Stage

The getting started page. You can find the quick installation guide here.

The full installation guide.
The Administrators guide.

Download and install XenServer on your host.
Download and install XenCenter on your management box.

You’ll need the following details:

  1. Hostname
  2. Host IP and mask
  3. Gateway
  4. DNS Server
  5. NTP Address

This was a very straight forward install.
I was expecting some trouble, but there wasn’t any.


KVM

KVM has support for iscsi.
Expected to run all production OS’s.
Why will KVM be the leader amongst hypervisors?

Interesting articles:

Is completely free.
Considerably more resource efficient than the alternatives
There are no resource constraints. We pay for nothing and get an enterprise level product with a huge community.

KVM on Debian

Management

Web based KVM management offerings of which ProxMox VE seems to be the stand-out.
Many of these can also be used for Xen. Also see this.

ProxMoxVE

ProxMox is a commercial company.
ProxMox VE Looks Good.
From what I’ve seen, looks easier to setup than Archipel.
Proxmox VE is licensed under GPLv2 (Open source).
My understanding of the GPLv2 license, is that the suplier of the GPL’d software can decide to charge a fee for download at any time.
As far as I’m aware, Proxmox are within their rights to do so at any time.
Correct me if I’m wrong?
The ISO installer is packaged with Debian, although you can install on top of Debian.
Looks User friendly, has Web interface (multi platform). No installs required.
Support: incl free community and paid for. See here and here.
The wiki
Looks like what ever you can do on a Debian system, you can do on a ProxMox system.
See this link. Also includes ESXi comparisons.
Proxmox VE is free to use and open source.
Easy backups and restores.
Video tutorials here and here.

Archipel

Archipel Also looks good.
Free and Open Source, licensed under AGPL (which more specifically targets distributed applications).
Team of 6 voluntary developers. Lots of info here.
Supports all libvirt-supported virtualisation engines like KVM, Xen, VMware
The install on first appearance, looks more work than ProxMox.
Documentation, IRC channel (members are very helpful), etc.
The Archipel client is JavaScript, which is run locally.

Industry support

KVM is supported by major industry players such as…

  1. IBM
  2. Cisco
  3. Intel
  4. AMD
  5. Redhat
  6. Novell amongst others.

Migration

Looks like migration of guests from most platforms to KVM is covered.
VMware to Proxmox, XenServer to Proxmox.

UPS

Can be shutdown by an APC Smart-UPS
using the APCUPSD daemon This will shutdown immediately.
Or better, by using PCNS for Linux.
Using PCNS we can specify when to shutdown and all sorts of other things.

Installation Stage Archipel

Links found useful for the Debian setup

http://www.debian-tutorials.com/virtualization/kvm-virtualization-on-debian-squeeze-server

http://wiki.debian.org/KVM

http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_physical_device.22.29

http://wiki.kartbuilding.net/index.php/KVM_Setup_on_Debian_Squeeze

Setting up Debian

Download Debian Wheezy from here
Install it.
Give it a hostname. For example “vmhost” without the quotes.
When prompted, select the SSH Server option.
Update your package index and install the necessary packages.

As root, run:

apt-get update
apt-get install qemu-kvm libvirt-bin virtinst virt-top

virtinst is for virt-install tools etc.
qemu-kvm is the new name for the kvm package in squeeze
libvirt-bin is what will control kvm and start guests on boot etc.
virt-top is a ‘top’-like utility for virtualisation stats

Add user to groups

Add the currently logged in user that will be using the associated programmes.

usermod -a -G libvirt myusername
usermod -a -G kvm myusername

Then check that the user was added to the groups.

groups myusername

or

id myusername

or view all users in all groups

cat /etc/group | less

Setup networking

Your /etc/network/interfaces needs to have a similar section:
As root, run the following…

vi /etc/network/interfaces

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
   address 192.168.1.20
   netmask 255.255.255.0
   gateway 192.168.1.254
   broadcast 192.168.1.255

Now restart your interface:

ifdown eth0

ifup eth0

Check that the changes have taken affect:

ip addr show

Setup Bridged networking

You also need to set up a network bridge on our server.
Rather than use NAT based connectivity, we need bridge networking.

install the package bridge-utils.

apt-get install bridge-utils

I’ve yet to set the bridge up.
Will add this once done

Setting up Archipel

Links I found helpful:

FAQ and supported browsers
https://github.com/primalmotion/Archipel/wiki/General%3A-FAQ&nbsp;&nbsp

Install ejabberd

apt-get install ejabberd

According to this, which is linked if you follow the install guide through,
we will need to update the path to the tls certificate.
Not sure where that is, but will have to find out.
the sample file contains the ejabberd configuration needed for Archipel.
It is not ready for production, so will need some modification. Yet to find out what.
Change all occurrences of FQDN to vmhost.mydomain.local and follow the other directions.

Once the ejabberd.cfg file is modified as suggested, download pscp.exe from here.
Put both the pscp.exe file and the ejabberd.cfg in the same folder (just to save typing paths and adding environment variables).
The help page is here if you get stuck.
Run a cmd prompt from the directory you have the 2 previous mentioned files within.
Then run:

pscp ejabberd.cfg myusername@192.168.1.20:ejabberd.cfg

Enter your password when prompted.
The file will be securely copied via SSH to your ~ dir.
You can’t copy directly to the /etc/ejabberd/ directory as you would need to be root of the destination machine.
Now go to the Debian box. cd into ~.
and move the config file to where it belongs.

su root

Enter your password when prompted.

mv ejabberd.cfg /etc/ejabberd/ejabberd.cfg

Then check that the move was successful.

Start the jabber server if it’s not already.
As root:

/etc/init.d/ejabberd start

Wait a few seconds and run:

/usr/sbin/ejabberdctl status

And you should get a result of running, with the version details.

You need to register a XMPP admin account (if you want archipel to work out of the box, just name it admin):

ejabberdctl register admin vmhost.mydomain.local MyCrazyPassWordHere

You should get something like:

User admin@vmhost.mydomain.local successfully registered.

Although I didn’t the last time because I wasn’t running as root.

Continue with the Archipel installation

The client is easy, just fetch and un-compress and your ready to go.

The agent, you will need to install qemu-utils if it’s not already.
It was for me.

As root, run:

apt-get install python-setuptools python-imaging python-numpy python-libvirt

python-libvert is Python bindings for the libvirt library which was already installed.

I also installed subversion:

apt-get install subversion

Now… as root, I chose to install the published packages on Pypi.
I ran:

easy_install archipel-agent

Post installation formalities

Finalise the installation:

archipel-initinstall

Follow the additional output instructions on the screen.

Now as root:

Create the pubsub nodes
archipel-tagnode --jid=admin@vmhost.mydomain.local --password=MyCrazyPassWordHere --create
archipel-rolesnode --jid=admin@vmhost.mydomain.local --password=MyCrazyPassWordHere --create
archipel-adminaccounts --jid=admin@vmhost.mydomain.local --password=MyCrazyPassWordHere --create
archipel-vmparkingnode --jid=admin@vmhost.mydomain.local --password=MyCrazyPassWordHere --create

The last two commands were, introduced after beta 4, so they didn’t exist on the binary I installed.

You can now start the archipel agent.

/etc/init.d/archipel start

The logs are printed to /var/log/archipel/archipel.log

To be completely sure Archipel is up and your hypervisor is connected you can run:

ejabberdctl connected_users

If you choose to just dump the archipel client somewhere and browse to the index.html,
you will have to use Safari as the browser.
Alternatively, you can use Chrome,
but you need to pass the argument… –disable-web-security
Or the better way is to just uncompress the archive into a HTTP server directory,
and access it with your browser.
I’ve been told nginx works well with serving Archipel.
At this stage I just set the client up on IIS locally.
In saying that, I’m getting the index.html,
but I’m getting 404′s for Info.plist and main.j
I need to look into this.

Using Archipel

https://github.com/primalmotion/Archipel/wiki/User-manual

Once you have the page in your browser, enter the following details into the dialog.

Jabber ID: admin@vmhost.mydomain.local
Password: MyCrazyPassWordHere
BOSH service: http://vmhost.mydomain.local:5280/http-bind

If you can’t access vmhost, try navigating to http://vmhost.mydomain.local:5280/http-bind in your browser.

You should get something like the following:

If you don’t,
try pinging vmhost.mydomain.local.
If the IP works but the host.FQDN doesn’t, it’s a dns issue.
I checked the /etc/hosts file and it had the host name as expected.

127.0.1.1   vmhost.mydomain.local   vmhost

For some reason, the Debian box’s hostname wasn’t getting registered on the DNS server.
The way around this is to add the following entry to the hosts file of the machine you have your client running from.

192.168.1.20    vmhost.mydomain.local

OpenSSH from Linux to Windows 7 via tunneled RDP

December 27, 2011

I recently acquired a new second hand Asus laptop from my work,
that will be performing a handful of responsibilities on one of my networks.

This is the process I took to set up OpenSSH on Cygwin running on the Windows 7 box.

I won’t be going over the steps to tunnel RDP as I’ve already done this in another post

Make sure your LAN Manager Authentication Level is set as high as practical.
Keeping in mind, that some networked printers using SMB may struggle with these permissions set to high.

  1. Windows Firewall -> Allowed Programs -> checked Remote Desktop.
  2. System Properties -> Remote tab -> turn radio button on to at least “Allow connections from computers running any version of Remote Desktop”
    If you like, this can be turned off once SSH is set-up, or you can just turn the firewall rule off that lets RDP in.

CopSSH which I used on my last set of Linux to Windows RDP via SSH set-ups is no longer free.
So I’m not paying for something I can get for free, but with a little extra work involved.

So I looked at some other Windows SSH offerings

  1. freeSSHd which looked like a simple set-up, but it didn’t appear to be currently maintained.
  2. OpenSSH the current latest version of 5.9 released September 6, 2011
    A while back OpenSSH wasn’t being maintained. Looks like that’s changed.

OpenSSH is part of Cygwin, so you need to create a
c:\cygwin directory and download setup.exe into it.

    1. Right click on c:\cygwin\setup.exe and select “Run as Administrator”.
      Click Next.
    2. If Install from Internet is not checked, check it. Then click Next.
    3. Accept the default “Root Directory” of C:\cygwin. Accept the default for “Install For” as All Users.
    4. Accept the default “Local Package Directory” of C:\cygwin.
    5. Accept the default “Select Your Internet Connection” of “Direct Connection”. Click Next.
    6. Select the closest mirror to you. Click Next.
    7. You can expand the list by clicking the View button, or just expand the Net node.
    8. Find openssh and click the Skip text, so that the Bin check box for the item is on.
    9. Find tcp_wrappers and click the Skip text, so that the Bin check box for the item is on.

If you selected tcp_wrappers and get the “ssh-exchange-identification: Connection closed by remote host” error,
you’ll need to edit /etc/hosts.allow and add the following two lines before the PARANOID line.

ALL: 127.0.0.1/32 : allow
 ALL: [::1]/128: allow

These lines were already in the /etc/hosts.allow

(optional) find the package “diffutils”, click on the word “skip” so that an x appears in Column B,
find the package “zlib”, click on the word “skip” (it should be already selected) so that an x appears in Column B.

Click Next to start the install.
Click Next again to… Resolving Dependencies, keep default “Select required packages…” checked.
At the end of the install, I got the “Program compatibility Assistant” stating… This program might not have installed correctly.
I clicked This program installed correctly.

Add an environment variable to your Systems Path variable.
Edit the Path and append ;c:\cygwin\bin

Right click the new Cygwin Terminal shortcut and Run as administrator.
Make sure the following files have the correct permissions.

/etc/passwd -rw-r–r–
/etc/group -rw-r–r–
/var drwxr-xr-x

Create a sshd.log file in /var/log/

touch /var/log/sshd.log
chmod 664 /var/log/sshd.log

Run ssh-host-config

  1. Cygwin will then ask Should privilege separation be used? Answer Yes
  2. Cygwin will then ask Should this script create a local user ‘sshd’ on this machine? Answer Yes
  3. Cygwin will then ask Do you want to install sshd as service? Answer Yes
  4. Cygwin will then ask for the value of CYGWIN for the daemon: []? Answer ntsec tty
  5. Cygwin will then ask Do you want to use a different name? Answer no
  6. Cygwin will then ask Please enter a password for new user cyg_server? Enter a password twice and remember it.

replicate your Windows user credentials with cygwin

mkpasswd -cl > /etc/passwd
mkgroup --local > /etc/group

I think (although I haven’t tried it yet) when you change your user password, which you should do regularly,
you should be able to run the above 2 commands again to update your password.
As I haven’t done this yet, I would take a backup of these files before I ran the commands.

to start the service, type the following:

net start sshd

Test SSH

ssh localhost

When you make changes to the /etc/sshd_config,
because it’s owned by cyg_server, you’ll need to make any changes as the owner.
I added the following line to the end of the file:

Ciphers blowfish-cbc,aes128-cbc,3des-cbc

As it sounds like Blowfish runs faster than the default AES-128

There are also a collection of changes to be made to the /etc/sshd_config

for example:

  • Change the LoginGraceTime to as small as possible number.
  • PermitRootLogin no
  • Set PasswordAuthentication to no once you get key pair auth set-up.
  • PermitEmptyPasswords no
  • You can also setup AllowUsers and DenyUsers.

The options available are here in the man page.
This is also helpful, I used this for my CopSSH setup.

Open firewalls TCP port 22 and close the RDP port once SSH is working.

As my blog post says:
ssh-copy-id MyUserName@MyWindows7Box

I already had a key pair with pass phrase, so I used that.
Now we should be able to ssh without being prompted for a password, but instead using key pair auth.

The links I found helpful:

http://pigtail.net/LRP/printsrv/cygwin-sshd.html
http://www.petri.co.il/setup-ssh-server-vista.htm
http://www.scottmurphy.info/open-ssh-server-sshd-cygwin-windows

JavaScript Reserved Words

December 19, 2011

Funnily enough, most of these are not used in the language.
They cannot be used to name variables or parameters.

Usage of keywords in red should be avoided.

Reserved Keyword Comments
abstract  no
boolean  no
break  yes
byte  no  No type of byte in JavaScript
case  yes
catch  yes
char  no  JavaScript doesn’t have char. Use string instead
class  no  technically JavaScript doesn’t have class
const  no  no const, but read-only can be implemented
continue  yes
debugger  yes
default  yes
delete  yes
do  yes
double  no  JavaScript only has number (64 bit floating point)
else  yes
enum  no
export  no
extends  no
false  no
final  no
finally  yes
float  no  JavaScript only has number (64 bit floating point)
for  yes
function  yes
goto  no
if  yes
implements  no  JavaScript uses prototypal inheritance
import  no
in  yes
instanceof  no
int  yes  JavaScript only has number (64 bit floating point)
interface  no  technically no interfaces, but they can be implemented
long  no  JavaScript only has number (64 bit floating point)
native  no
new  yes  use in moderation
null  no
package  no
private  no  access is inferred
protected  no  JavaScript has privileged, but it’s inferred
public  no  access is inferred
return  yes
short  no  JavaScript only has number (64 bit floating point)
static  no
super  no
switch  yes
synchronized  no
this  yes
throw  yes
throws  no
transient  no
true  no
try  yes
typeof  yes
var  yes
volatile  no
void  yes
while  yes
with  yes

-

When reserved words are used as keys in object literals,
they must be quoted.
They cannot be used with the dot notation,
so it is sometimes necessary to use the bracket notation instead.
Or better, just don’t use them for your names.

 
var method;                  // ok
var class;                   // illegal
object = {box: value};       // ok
object = {case: value};      // illegal
object = {'case': value};    // ok
object.box = value;          // ok
object.case = value;         // illegal
object['case'] = value;      // ok

-
JavaScript also predefines a number of global variables and functions
that you should also avoid using their names for your own variables and functions.
Here’s a list:

  • arguments
  • Array
  • Boolean
  • Date
  • decodeURI
  • decodeURIComponent
  • encodeURI
  • encodeURIComponent
  • Error
  • eval
  • EvalError
  • Function
  • Infinity
  • isFinite
  • isNaN
  • JSON
  • Math
  • NaN
  • Number
  • Object
  • parseFloat
  • parseInt
  • RangeError
  • ReferenceError
  • RegExp
  • String
  • SyntaxError
  • TypeError
  • undefined
  • URIError

Follow

Get every new post delivered to your Inbox.

Join 99 other followers